Make AI Agents See Your Website

The Problem: Docs Written for People, Not Agents

AI coding agents are now part of the developer workflow. Whether we like that shift or hate it, users ask Codex, Cursor, Claude Code, GitHub Copilot, and other tools to install packages, wire examples, migrate code, and explain APIs. For JavaScript data grids, that often means asking an agent to build columns, editors, filters, Pivot views, or Gantt timelines.

We ran into the obvious problem: the docs were written for people. A person can use the sidebar, search, breadcrumbs, examples, and product context. An agent often starts with a URL and a vague instruction. If it cannot find the right entry point quickly, it guesses.

There are several ways to make a site easier for agents to use: an MCP server for live retrieval, skill bundles for structured on-demand context, and llms.txt as a public discovery layer. That is where llms.txt helps. It is a public, text-first entry point that tells AI agents where the important documentation lives: installation guides, API reference, examples, migration notes, troubleshooting, full text exports, and any richer machine-readable bundles.

The goal is not to "optimize for bots" at the expense of people. The goal is to make sure that when a user asks an agent to build with RevoGrid, the agent can find the same source material a careful developer would read first.

Is llms.txt Actually Used?

llms.txt began as a proposal, published by Answer.AI co-founder Jeremy Howard in September 2024. There is meaningful adoption among major developer platforms:

• OpenAI publishes an llms.txt index with Markdown versions of its API documentation and a combined llms-full.txt export.
• Anthropic publishes an llms.txt index for its developer documentation.
• Cloudflare documents its AI-consumability strategy and publishes both per-product and site-wide llms.txt and llms-full.txt files.
• Stripe and Vercel expose public documentation indexes in the same format.
• Mintlify automatically generates llms.txt and llms-full.txt for documentation sites hosted on its platform.

What llms.txt Is For

Use it to answer these questions quickly:

• What is this product or library?
• Where are the most important getting started docs?
• Where is the API reference?
• Where are examples, migration notes, and troubleshooting pages?
• Is there a full text export or machine-readable bundle?
• Is there an MCP server or agent-specific package?

Example shape:

# Product Documentation

## Primary Docs
- [Installation](https://rv-grid.com/guide/installation): Install and configure the package.
- [API Reference](https://rv-grid.com/guide/api/revoGrid): Public TypeScript API.
- [Migration Guide](https://rv-grid.com/guide/migration): Upgrade notes.

How to Add llms.txt to a Documentation Site

1. Choose the canonical site origin, for example https://rv-grid.com.
2. Generate a Markdown index at /llms.txt.
3. Link to the highest-signal docs first.
4. Include the full docs corpus if you publish one.
5. Include agent-specific formats such as .skill bundles or MCP instructions.
6. Link to it from your human-facing AI or agent documentation, and optionally include it in your sitemap.
7. Make sure auth middleware allows public access.
8. Regenerate it whenever docs change.

For most docs sites, llms.txt can be generated from the same content collection, sidebar, or route manifest that powers the website. That is important. If the AI-facing docs drift from the real docs, agents will confidently produce stale answers.

What to Include in llms.txt

• Installation and package names
• Framework-specific setup for React, Vue, Angular, Svelte, or plain JavaScript
• API reference and TypeScript types
• Examples and live demos

If your product has paid, enterprise, or advanced modules, say so directly. Agents need to know whether a feature is core, Pro, Enterprise, experimental, or deprecated. Ambiguity here turns into broken code later.

Split Documentation Skill

For large docs, a split skill is usually better than asking an agent to read one giant file. The RevoGrid skill gives agents a SKILL.md index and one reference file per docs page. This structure is no longer limited to one agent.

Anthropic introduced Agent Skills as folders containing instructions, scripts, and resources, and later released the format as an open standard. OpenAI's Codex documentation also recommends a SKILL.md file with optional references/, scripts/, and assets/ directories. Both describe progressive disclosure: the agent sees a small description first and loads the full skill or individual references only when the task needs them.

The format and current ecosystem are documented at Agent Skills. This is the best format when:

• The docs are large
• Pages have similar headings
• The agent needs exact page-level context
• Examples need surrounding explanation
• Users work in tools that support skill-style uploads or local reference folders

Better for Large Docs: Split Reference Files

For coding agents, the better structure is closer to how a human reads docs:

product-docs/
├── SKILL.md
└── references/
    ├── getting-started/
    │   └── installation.md
    ├── api-reference/
    │   └── grid.md
    └── guides/
        └── migration.md

SKILL.md is the index and operating guide. It explains when to use the docs and links to focused reference files. This mirrors the progressive-disclosure model documented by both OpenAI and the Agent Skills specification: advertise a small amount of metadata, then load detailed instructions and references only when relevant.

Each file under references/ should represent one real documentation page. The agent can read the index, choose two or three relevant files, and avoid loading a massive corpus into context. This improves answer quality because the model sees complete local context for the page it selected: title, URL, description, source path, examples, and surrounding explanation. It also makes failures easier to debug because you can see which reference file the agent used.

Recommended Reference File Format

Each split reference file should be self-contained:

# Page Title
URL: https://rv-grid.com/guide/page/
Source: src/content/docs/guide/page.mdx
Description: Short page summary.

Page content starts here...

Keep the original code examples. Remove interactive-only wrappers that do not help in text form. If your docs use MDX components, unwrap or remove them carefully so the remaining Markdown still reads naturally.

Why Write About This Publicly

This is not only for bots. Developers are already trying to understand how their docs should work with agents, and there is not much practical writing from teams that have actually wired this into a docs pipeline. That is the reason to write about it. Not because another SEO page needs to exist, but because someone else will hit the same problem: they publish a full text export, point an agent at it, and still get an answer that misses the important part.

The useful searches are real: "How do I make my docs work with AI agents?" "What should be in llms.txt?" "Why is llms-full.txt ignored by coding agents?" "How do I package documentation as a skill?" "How should API docs be structured for Cursor, Codex, or Claude Code?" The page should answer those questions directly. If it does that, search visibility is a side effect of being useful, not the whole point.

Mistakes We Made

Making llms.txt Too Large

If llms.txt becomes another full corpus, it stops being useful as an index. Keep it short and link out.

Relying Only on llms-full.txt

A full export is helpful, but it is not enough for large docs. Agents often grep it and miss important context.

Splitting Without an Index

Reference files need a good SKILL.md or manifest. Without an index, the agent still has to guess which file to open.

Publishing Stale AI Docs

Do not write a separate AI manual by hand unless you can maintain it. Generate from the same docs source whenever possible.

Treating llms.txt as robots.txt

llms.txt provides content and navigation; it does not grant or deny crawler access. Keep using robots.txt and the documented controls for each crawler when access policy matters.

Promising an AI Ranking Boost

There is no reliable evidence that llms.txt alone improves AI citations or search visibility. The Ahrefs study, critical field reports, and practitioner discussion on Reddit all show why the claim remains disputed. Measure requests in server logs and evaluate agent answers against real tasks instead of assuming an effect.

A Practical Checklist

• Keep llms.txt short and navigational.
• Keep llms-full.txt as a full fallback.
• Add split reference files for large documentation sets.
• Add MCP support when your users work in tools that can retrieve live context.
• Include canonical URLs in every generated page.
• Preserve code fences exactly.
• Use stable slugs and deterministic output.
• Make the files public and easy to fetch.
• Mention the machine-readable docs from your human docs.
• Regenerate AI docs from the same source as the website.
• Test whether agents can answer representative questions with the published files.
• Monitor server logs separately for ordinary pages, Markdown pages, and llms.txt.
• Keep crawler access policy in robots.txt, not llms.txt.

Sources and Further Reading

Specifications and First-Party Guidance

• llms.txt proposal - the original purpose, suggested Markdown structure, and examples.
• OpenAI API documentation index - a live first-party llms.txt implementation with Markdown page links and a full export.
• Cloudflare: AI consumability - a detailed implementation covering indexes, full exports, per-page Markdown, and token efficiency.
• Mintlify: llms.txt - automatic generation, discovery headers, authentication behavior, and file structure.
• Anthropic: Equipping agents with Agent Skills - the motivation and architecture behind reusable skill folders.
• OpenAI: Build skills - Codex skill structure, progressive disclosure, installation locations, and authoring practices.
• Agent Skills open specification - the portable SKILL.md format and validation rules.
• Anthropic: Writing effective tools for agents - guidance on tool descriptions, agent-friendly documentation, response design, and evaluation.

Adoption, Analysis, and Debate

• Anthropic's public llms.txt, Stripe's public llms.txt, and Vercel's public llms.txt - examples from widely used developer platforms.
• llms.txt Hub: Anthropic - a third-party directory view that makes the Anthropic implementation easier to inspect and discover.
• Ahrefs: What Is llms.txt, and Should You Care About It? - adoption and request data plus a skeptical assessment of SEO claims.
• Kai Spriestersbach: "The llms.txt is dead. More precisely: a dud." - a forceful counterargument based on observed bot requests.
• Reddit r/SEO practitioner discussion - community experiences and the continuing disagreement about measurable AI visibility benefits.

Final Thought

The goal is not to feed the model more text. The goal is to help the agent choose the right text. For a small project, llms.txt plus clean Markdown pages may be enough. For larger libraries, frameworks, and enterprise components, add levels: a short index, a full fallback corpus, split reference files, and MCP for live retrieval.

That is the model we use for RevoGrid: make the docs easy for humans to browse and easy for AI agents to discover, select, and read. The fact that major platforms publish llms.txt and formally support Skills gives this approach credibility; honest measurement and multiple retrieval paths keep it grounded.