What is llms.txt and How to Add It to Your Site
AI systems are crawling the web constantly, but most websites give them nothing to work with. The llms.txt file is a simple plain-text file at your site root that tells AI crawlers exactly what your site does, who it is for, and where to find the important pages. Think of it as a README for machines.
Why llms.txt exists
When an AI assistant like ChatGPT or Perplexity visits your site, it has to parse your HTML, read marketing copy, and guess what matters. Most of that content is designed for humans: hero sections, testimonials, animations. None of that helps a machine understand your product.
llms.txt solves this by giving AI systems a clean, structured summary they can parse in milliseconds. It lists what your product does, your pricing, your API endpoints, and links to documentation. One file, no ambiguity.
The format specification
llms.txt uses Markdown-like syntax. The format is intentionally simple so any text parser can read it. Here are the rules:
- Line 1:
# Product Nameas an H1 heading. - Line 2: A blockquote (
>) with a one-sentence description. - Sections: Use
## Headingfor each section. Common sections: What it does, Who it is for, Pricing, Key pages, API, Contact. - Links: Use standard Markdown links
[text](url)or plain bullet lists with URLs. - Length: Keep llms.txt under 50 lines. Use llms-full.txt for extended documentation.
Real examples
Here is what Anthropic and Cursor publish. Notice how they are concise and link to deeper documentation rather than trying to include everything inline.
anthropic.com/llms.txt (simplified)
# Anthropic > Anthropic is an AI safety company that builds reliable, > interpretable, and steerable AI systems. ## Docs - [API documentation](https://docs.anthropic.com) - [Prompt engineering](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview) - [Model Context Protocol](https://modelcontextprotocol.io) ## API - Base URL: https://api.anthropic.com - Auth: x-api-key header
cursor.com/llms.txt (simplified)
# Cursor > The AI-first code editor. ## Docs - [Documentation](https://docs.cursor.com) - [Changelog](https://cursor.com/changelog) ## Features - Tab completion - Chat with codebase context - Composer for multi-file edits - MCP server support
Step-by-step: create your llms.txt
1. Start with the template
Copy this template and replace the placeholder content with your actual product details.
# Product Name > One-sentence description of what the product does. ## What it does Plain English explanation of the core functionality. No marketing language. Describe features in 2-3 sentences. ## Who it's for Specific user type and use case. Be precise, not broad. ## Pricing - Free: what's included (if applicable) - Starter: $X/month or one-time, what's included - Pro: $X/month, what's included ## Key pages - Homepage: https://yoursite.com - Docs: https://yoursite.com/docs - API reference: https://yoursite.com/api-docs - Pricing: https://yoursite.com/pricing - Blog: https://yoursite.com/blog - Changelog: https://yoursite.com/changelog ## API Base URL: https://yoursite.com/api/v1/ Auth: Bearer token in Authorization header. Full machine-readable reference: https://yoursite.com/skills.md ## Contact Support: hello@yoursite.com Twitter: @yourhandle
2. Place the file at your site root
The file must be accessible at yoursite.com/llms.txt. In different frameworks:
- Next.js: save to
public/llms.txt - Astro: save to
public/llms.txt - WordPress: upload to your web root via FTP, or use a plugin that lets you edit files in the root directory
- Static hosting (Netlify, Vercel, Cloudflare Pages): place in your build output directory
3. Verify it works
After deploying, visit yoursite.com/llms.txt in your browser. You should see the plain text content. If you get a 404, the file is not in the right directory.
4. Optional: add llms-full.txt
If you have API documentation, detailed guides, or integration examples, create an llms-full.txt alongside your llms.txt. This extended file can be as long as needed. Reference it from your llms.txt with a link in the API or Docs section. AI agents that need deep context will fetch the full version automatically.
Tips for a good llms.txt
- Write for machines, not marketing. Skip superlatives and buzzwords. State facts.
- Include pricing. AI assistants frequently answer 'how much does X cost' questions.
- Link to documentation and API references. AI agents follow these links for deeper context.
- Keep it under 50 lines. If you need more, use llms-full.txt.
- Update it when your product changes. Stale information is worse than no information.
- Reference it in your robots.txt sitemap or HTML head so crawlers discover it reliably.
FAQ
Is llms.txt an official web standard?
It is a community-driven convention, not a W3C or IETF standard. It was proposed in late 2024 and has been adopted by companies like Anthropic, Cursor, and many developer tool sites. AI crawlers increasingly look for it, making it a practical standard even without formal ratification.
Where do I put the llms.txt file?
Place it at the root of your domain so it is accessible at yoursite.com/llms.txt. In a Next.js project, put it in the public/ folder. In other frameworks, place it alongside your index.html or in the static files directory.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short summary (under 50 lines) that gives AI systems a quick overview of your site. llms-full.txt is an extended version with complete documentation, API references, and detailed content. AI agents that need deep context fetch llms-full.txt, while those doing quick lookups use llms.txt.
Does llms.txt replace robots.txt?
No. They serve different purposes. robots.txt controls crawling permissions (which bots can access which paths). llms.txt provides content for AI systems to understand your site. You need both. robots.txt controls access, llms.txt provides context.
Check if your llms.txt is working
SEOLint scans your site and flags missing or malformed llms.txt, along with 40+ other SEO and AI visibility issues.