What Is an LLMs.txt File and How Does It Relate to Consent: A Practical Decision Framework
An llms.txt file is a public Markdown guide that helps agents find useful content. It is not a consent record, license, access-control mechanism, privacy notice, crawler opt-out or guarantee that content will be indexed, used or cited. Use llms.txt for context, supported robots directives for crawler preferences, authentication for access, contracts or licenses for reuse rights, and privacy governance for personal data.
On this page
An llms.txt file tells an agent where useful public information lives and how that information is arranged. It does not say that every crawler may fetch the site or that model training is permitted. It also does not license copyrighted material or record a person's consent to processing of personal data.
The safest operating rule is simple: use llms.txt for context. Keep crawler preferences, security, rights and privacy decisions in the controls designed for those jobs.
Is llms.txt a consent file?
No. According to the current llms.txt v2 proposal, llms.txt is a public Markdown index. The proposal describes its purpose as follows: “provide information to help agents use a website”. The file has a required H1 and optional summary, guidance and grouped links. Agents can inspect it, select a relevant destination and fetch more detail.
The proposal also says llms.txt is designed to coexist with existing standards. It distinguishes the file from robots.txt: one supplies curated context, while the other communicates which automated access a site considers acceptable.
That distinction prevents one public file from being asked to carry five different decisions.
Decision | Appropriate control | What llms.txt contributes |
|---|---|---|
Help an agent find authoritative public pages | | Curated context and routes |
Express a supported crawler preference | Provider-specific | Nothing; it can reference only allowed public pages |
Stop unauthorized retrieval | Authentication, authorization, network and application controls | Nothing; the file itself is public |
Permit copying, training or commercial reuse | Applicable law, contract, license and rights policy | Nothing by itself |
Establish lawful personal-data processing | Privacy notice, purpose assessment and the applicable legal basis | Nothing by itself |
Why does “public” not mean “consented to every use”?
A browser can reach a public page, but that single fact does not answer every downstream question. Separate at least these activities:
- Discovery: can an agent find the URL?
- Automated fetching: does a named crawler request it, and what preference has the site published?
- Search use: may the page appear in a search or answer experience?
- Model development: may content be collected for training?
- Reuse and personal data: what rights, contracts and privacy rules apply to the proposed processing?
Even robots.txt does not settle all five. RFC 9309 states that its rules are “not a form of access authorization” and says the protocol is not a substitute for security measures. A path named in robots.txt is still public and may become easier to discover. Protect restricted material with authentication or another application-layer control.
Consent is also a specific term in privacy law, not a general synonym for a public URL. Under the EU General Data Protection Regulation, consent is one possible legal basis for processing personal data, alongside other bases in Article 6. Where a company relies on consent, Article 7 requires it to be demonstrable. A site-authored Markdown index cannot record a data subject's informed choice.
This is a governance framework, not legal advice. Rights and privacy teams should evaluate jurisdiction, data, purpose, contracts and licenses for the actual use case.
What do current AI crawler controls show?
Provider documentation shows why one blanket “AI allowed” switch is too crude. Different bots can serve search, model development and user-directed retrieval.
Purpose | OpenAI documentation | Anthropic documentation | Practical owner decision |
|---|---|---|---|
AI search discovery | | | Decide whether public search visibility is wanted |
Model development | | | Review provider-specific opt-out guidance and rights policy |
User-requested retrieval | | | Protect non-public content with real access controls |
As reported by OpenAI's crawler documentation, its search and training settings are independent. A site can allow OAI-SearchBot while disallowing GPTBot. OpenAI documents 3 out of 3 separate agents here for search, model development and user-requested retrieval.
Anthropic's crawler guidance similarly documents Claude-SearchBot, ClaudeBot and Claude-User as separate agents. Its April 7, 2026 guidance tells owners to apply preferences per bot in robots.txt.
These are provider statements, not a universal legal standard. Recheck them before changing production rules because names, purposes and supported controls can change.
What decision framework should a B2B team use?
1. Classify the content before listing it. Mark each candidate URL public, customer-only, employee-only, personal-data-bearing, licensed, embargoed or security-sensitive. Only deliberately public destinations should reach llms.txt.
2. Define the intended use. Write whether the business wants agent discovery, search inclusion, user-directed retrieval, training access or some combination. “AI visibility” is not precise enough for a governance decision.
3. Map each purpose to a control. Use llms.txt for navigation and supported bot rules for crawler preferences. Use page-level indexing controls where relevant, authentication for restricted content, and legal review for licensing and privacy questions. Google's robots meta documentation also warns that page-level rules can be read only when a crawler can access the page.
4. Check for contradictions. Do not list a customer portal in llms.txt while assuming obscurity protects it. Do not advertise a page as the canonical public source while blocking the search crawler expected to find it. Do not treat a permissive crawl rule as a blanket reuse license.
5. Record and review. Save the file version, owner, approval date, listed URLs, applicable bot rules and policy references. Recheck after a provider changes its bot documentation, after a content classification changes, or when the site introduces a new data purpose.
How should the file describe its own boundary?
A short note can prevent internal teams from reading too much into the file.
This index identifies public resources that may help agents understand this site. It does not alter access controls, copyright, license terms, privacy obligations or crawler-specific preferences.
That sentence is explanatory, not a substitute for terms, licenses or technical controls. Keep the file concise, link only to maintained canonical pages and remove anything that should not be made more discoverable.
The technical details belong in the step-by-step llms.txt generation guide. It covers file structure, scoped paths, deployment and validation without repeating this governance boundary.
Can you measure what happened after publication?
Yes, but measure observation rather than presumed consent or causation. Before publishing, preserve a fixed set of relevant buyer questions. Record the engines and modes tested, the answers returned and the exact cited URLs. Repeat the same sample after the file and its linked pages have been available long enough to be fetched.
Xtrusio can preserve that question-level baseline and repeat the scan across the selected answer surfaces. A 20-question sample across 3 surfaces creates 60 planned observations. If 60 out of 60 records preserve the answer and source list, collection completeness is 100%. That is not a visibility, permission or consent score. Use the AI visibility metrics guide to keep evidence collection separate from business interpretation.
If a listed page later appears as a source, record the dated change. Do not claim that llms.txt caused it without a controlled basis. Do not infer that retrieval granted permission for unrelated uses.
The practical conclusion
Publish llms.txt when agents would benefit from a curated route through content you already intend to make public. Do not publish it as a consent banner, legal opt-in, crawler allowlist or security layer.
The governance test is whether every desired outcome has its own accountable control. If the only answer is “we put it in llms.txt,” the decision is incomplete.
Sources reviewed
Frequently asked questions
Does publishing llms.txt give AI companies consent to use a website?
No. Publishing llms.txt makes a curated guide publicly fetchable. It does not create a universal license, privacy consent record or authorization for every downstream use.
Can llms.txt opt a website out of AI training?
No. Use the controls documented by each provider, such as a supported robots.txt user-agent rule, and review applicable contracts, licenses and law. llms.txt is not an opt-out mechanism.
Is robots.txt the same as legal consent?
No. Robots.txt communicates crawler access preferences to compliant automated clients. RFC 9309 explicitly says its rules are not access authorization, and legal rights or privacy obligations require separate analysis.
Can a private page be listed in llms.txt?
It should not be. llms.txt is public and can reveal a URL even when another control blocks retrieval. Keep confidential, personal, licensed or security-sensitive destinations out of the file and protect them with real access controls.
How should a company measure the effect of llms.txt?
Preserve a dated baseline of relevant questions, answers and cited URLs before publishing. Repeat the same sample later and report any change as an observation within that test, not as proof that llms.txt caused it.
Topics
- llms.txt consent
- what is llms.txt
- llms.txt privacy
- llms.txt robots.txt
- AI crawler consent
Xtrusio
AI visibility research
See what AI says about your brand
Access requests are temporarily paused while the new platform is prepared.
View access updateKeep reading
Why does AEO matter for my brand: Causes, Evidence and the Fix
See why answer engine optimization matters for a brand, which buyer and evidence risks it addresses, how to prove the gap and what to fix first.
What is answer engine optimization: Definition, Workflow and Real Use Cases
Understand answer engine optimization, how it works, where B2B teams use it, how it differs from SEO and GEO, and how to measure it.