What Is an LLMs.txt File and How Does It Relate to Consent: A Practical Decision Framework

The practical rule is to use llms.txt for context and keep permissions in the controls designed for them. A file that points to your documentation should not become an accidental company-wide declaration that every use of every page is permitted.

What does llms.txt actually do?

The llms.txt v2 proposal describes a Markdown guide containing a site or project name, optional explanatory text and links to further information. Markdown is plain text with simple formatting. The guide helps an agent choose what to read next; it is not a permissions database.

An agent still needs to fetch the linked content. The file does not authenticate the requester, unlock a customer portal or prove that a provider has agreed to your terms. Nor does publishing it guarantee that a search system will use or cite it.

“Public” answers whether material is available. It does not settle every question about how someone may use it. Equally, a notice cannot by itself technically prevent a non-compliant visitor from fetching an unprotected page.

Why is a public file not a consent record?

Consent in privacy law concerns a person's choice about processing their data. It is different from a publisher deciding which documentation to expose to an agent.

For example, the UK's ICO guidance on valid consent explains the need for a freely given, specific, informed and unambiguous choice, with records demonstrating consent. A website owner's Markdown index does not demonstrate that choice by each individual whose information appears on the site. Consent is not the only possible legal basis for processing.

The ICO page is under review following UK legislative changes, so check the current guidance when making an actual decision. This article provides a practical governance framework, not a legal opinion. Your privacy or rights adviser should assess the jurisdiction, data, purpose and proposed reuse.

Even robots.txt is not a security system. RFC 9309 says its rules are not access authorization. Naming a private path in a public file may make the path easier to discover. Protect the content itself rather than relying on an obscure URL or a request that crawlers stay away.

Can you allow search while restricting training?

Some providers document separate controls for those purposes. Check each provider's current policy instead of applying one generic “AI allowed” decision.

OpenAI's crawler documentation separates search from model-development controls. A site can allow its search crawler while disallowing GPTBot. OpenAI also says robots.txt rules may not apply to ChatGPT-User requests because those actions are initiated by a user.

Anthropic's documentation describes a different user-request policy: it says its bots honor robots.txt and that disabling Claude-User prevents retrieval in response to a user query. Do not copy OpenAI's user-fetch exception into your Anthropic policy.

These are documented provider behaviors, not universal legal permission rules. A supported training opt-out is also not evidence that previously collected content has been removed from a model. Keep that question separate from future crawl preferences.

Search indexing needs a separate check too. Google's page-level robots documentation explains that a crawler must be able to access a page to read its directives. Blocking access can prevent it from seeing a noindex instruction. Have the responsible engineer check the combination rather than stacking contradictory rules.

What should an actual approval decision look like?

Suppose a B2B software company wants agents to understand its public product documentation while keeping customer records private. The following is a fictional planning example, not a compliance assessment of a real company.

The team has four content types. Listing all four because they belong to the same domain would be the wrong starting point.

Notice that an approved public guide can still have a separate training preference. “Include in the index” is not the same decision as “allow every crawler.” The index decision also does not authorize future uses of a customer quote beyond the permission actually obtained.

Record the result in a short approval note. For this fictional company, it could read:

Purpose: help agents find public integration documentation. Scope: the approved documentation URLs only. Owner: documentation lead. Access review: security confirms that customer tickets and attachments require authorization. Crawler preferences: engineer records each provider's search, training and user-fetch decision. Rights review: customer stories and licensed material stay out until their owners clear them. Review trigger: any change in content classification, reuse purpose or provider controls.

Save the approved URL list, file version and decision date with that note. Recheck linked pages and any generated Markdown copies: keeping the original page private is not enough if a public export contains the same material.

Should the file explain these limits?

A brief note can help colleagues avoid treating the index as a permission statement. For example:

“This index points to public resources. It does not replace access controls, license terms, privacy decisions or provider-specific crawler preferences.”

That is suggested explanatory wording, not a tested legal clause. If you intend to impose reuse conditions, have the appropriate owner approve those terms and where they belong. A sentence in llms.txt cannot substitute for technical enforcement or resolve rights you do not hold.

Keep the distinction when removing material too. Removing a link from llms.txt does not delete the destination page, its copies or material already fetched elsewhere. If the page should no longer be public, address its actual access and publication state.

What can monitoring prove after publication?

Use different records for different questions. An authorized server log can show a request for the file; a saved answer can show a cited page; a consent record can document a person's choice. None automatically proves the other two.

Xtrusio's saved source records help with the visibility question. In this existing record, two entries point to variants of the public pitch-deck URL. The history area says only one published report includes the question. It is inspectable evidence of what that saved response contains, not evidence that llms.txt was read or that permission was granted.

Open this image at full size

The record was inspected in September 2026; private client paths are blurred. Generated capability claims visible in the response are not a compliance finding or a promise that those claims are correct. The useful evidence here is the question, cited URLs and limit on the available history.

If you want to test visibility after an approved change, preserve the baseline questions, settings, answers and source URLs. Keep later observations comparable and log other page changes. A new citation after publication is an observation, not proof that the file caused it or that reuse was authorized.

Publish llms.txt when a curated guide would help agents use information you intend to make public. Keep the permission and security decisions with their accountable owners. If you need help checking how those public pages appear in AI answers, ask Xtrusio for a scoped free evaluation. Consent and rights approval remain separate from that visibility review.