How to Generate an LLMs.txt File: A Step-by-Step, Evidence-Led Workflow

To generate an llms.txt file, create a UTF-8 Markdown file named llms.txt. Start with one H1 naming the site or project. Add an optional blockquote summary and group links to canonical or Markdown-friendly pages under H2 headings. Publish it at /llms.txt or a scoped path such as /docs/llms.txt. Confirm that it returns HTTP 200 as readable text, then update it whenever the source content changes.

That is the working method. The boundary matters just as much. The file is not crawler permission, consent, a replacement for robots.txt, or proof that an AI system will cite the listed pages.

What does llms.txt control, and what stays separate?

The current llms.txt v2 proposal describes a small Markdown index that agents can inspect when they need context about a site. It can live at the origin root or beneath a path, and a scoped file describes the pages under that path. The proposal requires only the H1; the summary, explanatory text and H2 file lists are optional but make the file useful.

The file should coexist with established controls. According to OpenAI's crawler documentation, OAI-SearchBot, GPTBot and ChatGPT-User have different purposes. As reported by Anthropic, ClaudeBot, Claude-User and Claude-SearchBot are also separate. Anthropic directs site owners to robots.txt for crawler preferences. Neither provider describes llms.txt as an override for those rules.

How do you build the file step by step?

1. Start from canonical content, not the whole sitemap. List the pages that accurately explain the company, product, documentation, policies, pricing and evidence. Exclude search pages, tracking URLs, duplicate versions, expired campaigns, internal environments and pages an unauthenticated agent cannot fetch.

2. Decide the scope. Use /llms.txt for an origin-wide guide. Use a path such as /docs/llms.txt when a documentation team controls only that section or needs a more specific index. If both apply, keep their responsibilities clear rather than duplicating every link.

3. Write the Markdown in the proposed order. This compact example is sufficient to start:

# Example Company

> Example Company helps operations teams monitor field-service performance.

Use current product documentation for technical behaviour and the policy pages for contractual terms.

## Start here

• • Product overview: Capabilities and intended users
  • Documentation: Setup and operating guides
  • Security: Security controls and data handling

## Optional

• • Research: Dated studies and methodology

4. Prefer fetchable destinations. The proposal recommends linking to clean Markdown versions when available. It recommends rel="alternate" type="text/markdown" for a page's Markdown twin. Use rel="describedby" for the applicable llms.txt file. Absolute URLs make validation and cross-host documentation easier.

5. Publish a raw response. Place the file in a static public directory, expose it through a server route, or use the documentation platform's generator. Do not return an HTML page that merely displays the Markdown inside a template.

6. Verify from outside the deployment. Check the exact production URL, not a local file:

```bash
curl -i https://example.com/llms.txt
curl -s https://example.com/llms.txt | head -40
```

Require HTTP 200, readable Markdown, one H1 at the top, live destination URLs, no redirects to login, and no confidential or staging links.

Which generation method fits each website stack?

Manual files work for stable sites. Frequently changing documentation should generate the index from the same source that controls canonical navigation, because a stale guide can direct agents toward retired pages.

Mintlify and GitBook document automatic llms.txt, full-text companion files and Markdown page versions. The Answer.AI reference repository also links parsers and integrations for teams operating their own stack. Automation removes repetitive editing; it does not remove editorial responsibility for scope, descriptions or freshness.

How should you validate quality before release?

Run these checks in the build pipeline or release checklist:

Set a 100% success target for every listed destination in each release. A check where 1 in 20 URLs fails has a 95% success rate. Block that release until the broken link is fixed or removed.

1. The first non-BOM line is one H1 with the correct site or project name.
2. Every listed URL is canonical, public and returns a successful response.
3. Descriptions identify what the destination answers instead of repeating its title.
4. Markdown twins contain the same material facts as the human page.
5. No entry contradicts robots.txt, authentication, noindex, contractual access or privacy controls.
6. Removed and redirected pages fail validation until the manifest is updated.
7. The file remains curated; it does not become a second, noisier sitemap.

Keep the generated output in version control or store a release hash so reviewers can see what changed. Schedule a link check, but regenerate immediately after major navigation, product, policy or version changes.

How can you measure whether llms.txt made a difference?

Do not use publication as the success metric. Before deployment, save a fixed cohort of real buyer questions. Capture the engine, mode, full answer, brand mentions, cited domains, exact URLs, date and failures. After a defined observation window, repeat the same cohort without rewriting the baseline.

The platform supports that question-level evidence loop across public AI-answer surfaces. Xtrusio is used for question-level baseline and repeat measurement. Compare visibility and citations before and after the technical change. Then inspect whether the listed canonical pages appear in observed sources. A movement is evidence of correlation within a dated test, not proof that llms.txt caused the change. The AI visibility metrics guide explains how to separate discovery, perception, evidence and commercial outcomes.

No major answer engine promises a citation because a page appears in llms.txt. Treat the file as low-cost information architecture for agents. It helps when it is concise, accurate and maintained. Fetchable pages, strong evidence, clear authorship and real authority still carry the larger burden.

Sources reviewed

• The llms.txt v2 proposal
• Answer.AI llms-txt reference repository
• OpenAI crawler documentation
• Anthropic crawler documentation
• Google crawler documentation
• Mintlify llms.txt documentation
• GitBook LLM-ready documentation