Do You Need an llms.txt File?

llms.txt is a plain markdown file you put at yourdomain.com/llms.txt that summarizes your site for AI models. It's worth adding, it takes an afternoon, and it costs you nothing. But don't confuse it with an actual AI visibility strategy: no major AI product has confirmed it reads the file, and the things that really determine whether your site gets crawled and cited by AI answers are the same things that determine whether Google can crawl it. Where llms.txt came from The spec was proposed in late 2024 by Jeremy Howard at Answer.AI, modeled loosely on robots.txt: a single, predictable location where a site tells automated readers what matters. The idea is that a language model has a limited context window and can't (or shouldn't have to) crawl your entire site to answer a question about it, so you hand it a curated index instead: your product pages, your docs, your pricing, in one short file with links and one-line descriptions. The full spec is at llmstxt.org. It's a reasonable idea. The problem is adoption. As more than one developer has pointed out publicly this year, nobody has confirmed their crawler actually fetches it. OpenAI, Anthropic, and Google have not published documentation committing to parse llms.txt as part of how ChatGPT, Claude, or Gemini answer questions about your product. That doesn't mean it's useless, it means you should size the investment to match the uncertainty. What it does and doesn't do What it does: gives you a clean, low-effort way to hand-curate what an AI system sees if it does decide to look. It costs nothing to maintain if your site doesn't change often. It's a reasonable hedge, the same way you'd add a sitemap.xml even though most of your traffic doesn't come from crawlers reading it directly. What it doesn't do: - It's not a ranking signal for Google or any traditional search engine. - It doesn't override or supplement your actual page content, if the linked pages are JavaScript-rendered client-side with no server-rendered HTML, an AI crawler that does try to follow the links will hit the same wall a search engine crawler does. We've written about this tradeoff before in the context of server-side rendering and SEO. - It doesn't fix a slow, bloated site. If your Core Web Vitals are bad, an AI agent trying to fetch and parse your pages within a reasonable timeout will bail the same way a human does. The decision framework If you're a founder or CTO deciding whether to spend engineering time on this, here's the honest breakdown: Worth doing - You have a docs site, developer product, or API that AI coding assistants (Claude Code, Cursor, Copilot) might reference when a user asks "how do I integrate X." - Your marketing site already has clean, server-rendered pages, so llms.txt is additive, not a patch over a broken foundation. - You can generate it once and regenerate it on deploy with a small script rather than hand-maintaining it forever. Not worth prioritizing - Your core pages are client-side rendered with no SSR, meaning the links in your llms.txt point to content a crawler can't actually read anyway. Fix that first. - You're treating it as an SEO strategy. It isn't one. Google has not indicated llms.txt affects ranking at all. - You'd need to build custom tooling to keep it in sync with a fast-changing site. At that point the maintenance cost exceeds the unconfirmed upside. What actually gets you cited by AI answers If the real goal is "when someone asks an AI assistant about a problem we solve, we want to show up," the leverage is almost entirely in things that predate llms.txt: - Server-rendered, crawlable content. Every major AI crawler (GPTBot, ClaudeBot, PerplexityBot) behaves like a search crawler: it fetches HTML, and if your content only appears after a client-side fetch, it often sees an empty shell. - Clean semantic structure. Real headings, real paragraph text, schema.org markup where relevant. This is the same discipline that makes a page rank well and the same discipline that makes it easy for an LLM to extract a clean answer. - Fast, stable pages. Crawlers time out. A bloated bundle that takes eight seconds to become interactive gets skipped the same way a slow page gets a lower crawl budget from Google. This lines up with something we've seen directly in our own tooling, not on the SEO side but on the consumption side. We run an outreach engine that scrapes each prospect's site with a self-hosted Firecrawl instance and a local model to draft a tailored email per company. The pattern that actually worked reliably was a single call that extracts the facts we need and drafts the email in one pass, not a multi-step chain that tries to crawl a sitemap, summarize each page, then synthesize. The lesson translates directly to llms.txt: a clean, well-structured page that an LLM can read in one pass beats a curated index pointing at pages the model still has to fight to parse. If you're weighing similar tradeoffs in your own AI features, we've written more on single-call versus agent-chain design and on RAG for founders if the underlying question is really "how do we make our content retrievable by an LLM," which is the same problem from a different angle. How to actually implement it If you've decided it's worth the afternoon: - List your 10-20 most important pages: product, pricing, docs, key integration guides. - Write one honest sentence per page, no marketing copy, just what's there. - Serve it as a static file at /llms.txt using the format from the spec (H1 title, blockquote summary, H2 sections with linked bullet points). - Regenerate it as part of your build if your page set changes often, don't let it go stale. - Move on. Don't build a dashboard for it. llms.txt is a cheap insurance policy, not a strategy. Spend real effort on server-rendered, fast, well-structured pages, that's what both search engines and AI crawlers actually need to cite you, and it's work you should be doing regardless of whether any model ever reads your llms.txt file. If you're trying to figure out whether your site or product is actually AI-crawlable, and not just checking a box, let's talk. Originally published on the Pykero blog. Top comments (0)