llms.txt on a 4-language static site: how we generate it, validate it and keep it honest

AI assistants now answer "who does X in Y" questions directly, and they build those answers from what they can read about you. For a small team, the cheapest thing you control is your own site: what crawlers are allowed to read, and how clearly your facts are stated.

On veilvex.com (an Astro static site in English, Russian, Azerbaijani and German) we treat this as part of the build, not a one-off file. Here is the setup and the rules we ended up with.

1. robots.txt: say yes explicitly

Many sites block AI crawlers by accident: a template, a CDN toggle, a security plugin. We generate robots.txt at build time and list the crawlers by name, even though User-agent: * already allows everything:

User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /

Why list them? Because the next person who adds a "block bots" rule will see the intent, and because a test can assert that every agent in the list stays allowed.

The Content-Signal line follows contentsignals.org and states how the content may be used. Also check your CDN: on Cloudflare, "Block AI bots" and bot-fight settings can override a perfectly good robots.txt. We verified with real requests that GPTBot, ClaudeBot and PerplexityBot get a 200 and the full HTML.

2. llms.txt: a fact sheet, generated from the same source as the pages

llms.txt is a proposed convention: a short Markdown file at the site root that tells language models what the site is and where the important pages are. It is not a ranking factor and no engine promises to use it. We still ship it because it costs nothing and removes ambiguity.

The important decision: never write it by hand. Ours is built by a script from the same i18n content files that render the pages. So the price on the services page and the price in llms.txt cannot drift apart.

What goes in (English, compact):

• one-sentence brand line (the same sentence is used in the footer, the Organization.description in JSON-LD and external profiles);
• an About block: name spelled out letter by letter (search engines like to "correct" unusual brand names), legal status stated plainly, official contacts only;
• every service with its "from" price and a link to its own page;
• service names in the other languages, linked to the localized pages;
• a "what we do not promise" section;
• a page list per language.

Next to it there is llms-full.txt with the full text of every page in all four languages (about 400 KB for us), for tools that want everything in one fetch.

3. The generator fails the build

This is the part that keeps it honest. The script throws, and the deploy stops, when:

• llms.txt grows over 15 KB (a fact sheet nobody reads in full is not a fact sheet);
• a price in the file does not match the single price source (prices.ts);
• the text contains words we decided never to publish: guarantees, "official", leftover template markup like [[...]] or {{...}}.

A simplified version of the price check:

for (const s of services) {
  const expected = formatFrom(PRICES[s.id]);
  if (!s.line.includes(expected)) {
    throw new Error(`llms.txt: ${s.id} shows a price that is not ${expected}`);
  }
}

Models repeat what they read. If your site says "guaranteed first place" in one corner, that is what an assistant may quote back to a customer. A build-time check is cheaper than finding it later.

4. Structured data that agrees with the text

Each service page carries Service + Offer and FAQPage JSON-LD built from the same content object as the visible text, with stable @ids across languages. The organization node uses the same brand line as llms.txt. One source, three outputs (HTML, JSON-LD, llms.txt), so a crawler sees the same facts whichever it reads.

5. Sitemaps with honest dates and IndexNow

<lastmod> is the date of the last git commit that touched the page's source files, not the build time. A build date on every URL tells crawlers nothing.

After a deploy, a script compares the new sitemap with the live one and sends only new or changed URLs to IndexNow (Bing, Yandex and others).

What this does not do

None of this makes an assistant recommend you. It removes the reasons not to: blocked crawlers, conflicting prices, vague descriptions. Being named still depends on being mentioned in places models trust: directories, reviews, articles, discussions. We check monthly by asking ChatGPT, Perplexity and Gemini the same set of queries and logging who they name.

Checklist

• [ ] AI crawlers allowed in robots.txt and not blocked by your CDN
• [ ] One page per service with price range, area and FAQ in plain text
• [ ] llms.txt generated from your content, not typed by hand
• [ ] Build fails on price mismatch or forbidden claims
• [ ] JSON-LD built from the same source as the visible text
• [ ] Real lastmod dates, IndexNow after deploy
• [ ] A fixed list of 10-20 queries you re-check every month

You can see the live files at veilvex.com/llms.txt and veilvex.com/robots.txt. Questions about the setup are welcome in the comments. We are Veilvex, a remote team working on GEO (visibility in AI answers), SEO, websites and AI automation.