In plain English
llms.txt is a proposed Markdown file that lists and describes important website resources for language-model use. It is not an access-control file and is not required by Google Search, which says it ignores the file. A business may test it as a maintained directory, but should first fix crawlability, indexable content, structured facts and official profiles.
What does the llms.txt proposal actually specify?
The llms.txt project describes a Markdown file placed at /llms.txt or within a subpath to give language models and agents a curated route into useful resources. In its current format, an H1 naming the site or project is the only required element. A summary blockquote, explanatory text and H2 sections containing named links and optional notes can provide the additional structure.
The proposal positions the file as a small overview whose links lead to more detailed, LLM-friendly material. It also says llms.txt and robots.txt have different purposes: robots.txt communicates crawl permissions, while llms.txt supplies context and directions. Calling it “robots.txt for AI” therefore creates the wrong operational expectation. Listing a page does not grant access, and omitting a page is not a privacy, indexing or training control. The proposal remains a convention, not a permission protocol.
Sources for this section: The /llms.txt file, v2.
Do major search providers require llms.txt?
Google's July 2026 guidance is explicit: Google Search does not use llms.txt, and maintaining one neither helps nor harms visibility or rankings in Google Search. The same guide says there is no special Schema.org markup required for generative Search. That makes an llms.txt project a poor substitute for indexable pages, unique content, clear technical structure and maintained local-business information.
OpenAI's current publisher FAQ focuses on public accessibility, OAI-SearchBot and robots.txt for inclusion in ChatGPT summaries and snippets. It separately describes GPTBot controls for potential training. The cited guidance does not present llms.txt as a requirement. Follow each provider's published crawler and content controls directly, and recheck them over time. Document the guidance date used for each decision. Do not assume that one experimental file governs search retrieval, agent access and model training across companies.
Sources for this section: Optimizing your website for generative AI features on Google Search, Publishers and Developers - FAQ.
When might an llms.txt experiment be reasonable?
Consider an experiment when a known tool or customer workflow consumes the proposal, or when a documentation-heavy site can cheaply generate and maintain a curated map. Define the consumer, expected behaviour, test method and removal condition before building it. A small service website with ten clear pages is unlikely to justify manual duplication while its services, profiles or internal links remain inaccurate.
Treat the file as another published interface. It may help a compatible agent navigate selected resources, but that outcome must be observed in the intended system. Do not report adoption, citations or improved rankings without evidence. The cost includes monitoring broken links, synchronising changed descriptions and preventing stale or private material from entering the list. If no accountable owner can maintain it, a current sitemap and clear human-facing navigation are usually the safer priorities.
Sources for this section: The /llms.txt file, v2, Optimizing your website for generative AI features on Google Search, Bing Webmaster Guidelines.
How can a business implement the file safely?
If the test has a defined consumer, serve plain text at the agreed path and follow the proposal's order: site name, concise summary, optional context, then grouped links with accurate titles and notes. Link only to public, authoritative resources that the business is comfortable helping an agent retrieve. Use canonical HTTPS URLs, keep the file small enough to curate, and return an ordinary successful response that monitoring can check.
Generate the file from an approved content inventory where practical, then test every link and compare descriptions with the live pages. Do not place secrets, private proposals, personal data or hidden instructions in it. Keep robots.txt, noindex, authentication and provider-specific controls responsible for their actual jobs. Add the file to change management so removals and service updates cannot drift. Record the experiment separately from SEO performance because Google says it ignores llms.txt.
Sources for this section: The /llms.txt file, v2, Optimizing your website for generative AI features on Google Search, Publishers and Developers - FAQ.
Official references
- The /llms.txt file, v2 (llms.txt project)
- Optimizing your website for generative AI features on Google Search (Google Search Central)
- Publishers and Developers - FAQ (OpenAI Help Center)
- Bing Webmaster Guidelines (Microsoft Bing Webmaster Tools)