llms.txt and the Future of AI Discoverability

A new standard for talking to AI

The llms.txt file is an emerging specification that helps AI systems understand your website. Created by the team behind fast.ai and documented at llmstxt.org, it's a plain-text Markdown file at your website's root (/llms.txt) that provides a structured summary of what your site offers.

Think of it as robots.txt for the AI era. Where robots.txt tells crawlers what to access, llms.txt tells AI models what your site is about and where to find the important content.

Why llms.txt matters now

AI search engines — ChatGPT, Perplexity, Claude — generate answers by synthesizing information from multiple sources. When these models need to answer "what's the best website audit tool for agencies?", they look for clear, structured information they can cite confidently.

A well-crafted llms.txt gives them exactly that. Instead of forcing the model to parse your entire site and guess what matters, you provide a curated summary: product description, pricing, key features, and links to important pages.

The specification

The llms.txt file follows Markdown format with specific conventions:

• H1 heading — Your product/company name
• Blockquote — A one-sentence description
• H2 sections — Organized by topic (Product, Pricing, Blog, etc.)
• Bullet points with links — Key pages with short descriptions
• Optional section — Links that are helpful but not essential

What to include

For a SaaS product:

• Product name and one-line description
• Key features and capabilities
• Pricing tiers with specifics
• Links to documentation, blog, and tools
• FAQ answers
• API information (if applicable)

For a local business:

• Business name, location, and description
• Services offered with links
• Hours and contact information
• Service areas
• Common questions and answers

llms.txt vs llms-full.txt

The specification supports two files:

llms.txt — A concise directory (~50–100 lines). Lists key pages with links and one-line descriptions. AI models use this for quick orientation.

llms-full.txt — The complete documentation in a single file. Contains the full text of all important pages concatenated into one Markdown document. AI models and developer tools (like Cursor's @Docs feature) use this for deep context.

Both serve different use cases. llms.txt is for quick reference. llms-full.txt is for comprehensive understanding.

How to create yours

Step 1: Inventory your content

List every important page on your site:

• Homepage, pricing, features
• Blog posts (published ones only)
• Tool/resource pages
• Comparison pages
• Industry/vertical pages
• FAQ and support content

Step 2: Write the summary

Create the Markdown file following the spec format. Focus on facts, not marketing. "Our tool runs audits in under 30 seconds across 5 categories" is useful. "We're the best audit tool ever" is not.

Step 3: Deploy as a route

The file should be accessible at yoursite.com/llms.txt. You can serve it as a static file in your public/ directory or generate it dynamically from your data (which keeps it in sync with your content automatically).

Step 4: Keep it updated

The llms.txt file should reflect your current content. If you add new blog posts, features, or pages, update the file. If you generate it dynamically from your CMS or data files, this happens automatically.

Verifying AI visibility

After deploying your llms.txt, check your overall AI search readiness with the AI Visibility Grader. The tool checks for:

• AI crawler permissions in robots.txt
• Presence of llms.txt
• Structured data that AI models can extract
• Content structure and clarity

The agency opportunity

Fewer than 5% of websites have an llms.txt file today. For agencies, this is a quick-win service:

1. Create the llms.txt file for the client (30 minutes)
2. Update their robots.txt to allow AI search crawlers (5 minutes)
3. Add structured data to key pages (1–2 hours)
4. Monitor AI search citations over time

Charge $200–$500 as a one-time setup, or include it in your monthly retainer. The work is minimal, the value proposition is forward-looking ("prepare for AI search before your competitors do"), and the deliverable is tangible.

What's next

The llms.txt specification will likely evolve as AI search matures. But the core principle will remain: make your content easy for machines to understand and cite. The sites that do this now will have a compounding advantage as AI search adoption grows.

Run a free audit to check any site's AI visibility signals today.