Does llms.txt Do Anything? What the AI Companies Have Said

By May 2026, every Shopify store was serving an llms.txt file. Shopify switched it on by default, and All in One SEO does the same on WordPress. The funny thing is, no AI company has said its assistant reads this file, and Google says its search ignores it.

As at 14 October 2026, OpenAI, Anthropic and Perplexity each explain on their own sites how they collect pages from websites, and none of the three says it reads llms.txt, a text file written for AI assistants that your store may already serve. You can check in a browser whether your store already serves one, and robots.txt is the file worth checking instead.

What llms.txt is and who proposed it

llms.txt is a proposal written by one person. Jeremy Howard published it on llmstxt.org on 3 September 2024, and the page’s author line lists only him, with no company.

The file sits at /llms.txt on a website. In the proposal’s words, it means “adding a /llms.txt markdown file to websites to provide LLM-friendly content”. Markdown is what its creator John Gruber calls an “easy-to-read, easy-to-write plain text format”.

Jeremy calls it “A proposal to standardise on using an /llms.txt file to provide information to help agents use a website.” An agent here is an AI assistant that uses a website for a person, such as one that shops on a customer’s behalf. Version 2 of the proposal, last modified on 10 August 2026, adds the line that agents “are expected to view or search llms.txt to find the information they need, then follow the relevant links”. That sentence is the author’s expectation of how agents will behave.

The proposal also says “robots.txt and llms.txt have different purposes”. In Google’s definition, a crawler is any program that automatically finds and scans websites, and Google describes a robots.txt file as one that “tells search engine crawlers which URLs the crawler can access on your site”.

What OpenAI, Anthropic, Perplexity and Microsoft have said

OpenAI, Anthropic and Perplexity each point site owners to robots.txt on their crawler pages. OpenAI’s crawler page says it “uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.” OAI-SearchBot and GPTBot are the names of two of OpenAI’s crawlers. Anthropic’s help article says its crawlers respect “do not crawl” signals by following the rules in robots.txt. Perplexity’s crawler page recommends “allowing PerplexityBot in your site’s robots.txt file” if you want to appear in its search results.

Anthropic’s article does not mention llms.txt at all. OpenAI and Perplexity mention it only as the address of their own documentation index. We found no statement from Microsoft or Bing on whether Bing or Copilot reads llms.txt.

That index line points to something you will see quoted as proof the file works. OpenAI, Anthropic, Perplexity, Google’s Gemini API team and Cloudflare each publish an llms.txt for their own developer documentation. The proposal cites the AI companies’ files as adoption. Each of those files is an index of the company’s own documentation. It shows the company publishing a file and tells you nothing about whether it reads yours.

What Google has said

Google has said the most, and from multiple teams. Its guide to generative AI in Search was announced on the Search Central blog on 15 May 2026 and last updated on 10 July 2026. It says you do not need AI text files to appear in Google Search “(including its generative AI capabilities), as Google Search itself doesn’t use them.”

The same guide is relaxed about stores that publish one anyway. “It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files,” Google writes, and doing so “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” Google also lists “creating unnecessary AI text files (like llms.txt)” among the tactics you can ignore for Google Search.

Another Google team signals something different. Chrome’s Lighthouse, a Google tool that runs checks (called audits) on web pages, has an llms.txt audit in an experimental category for “agentic browsing”, meaning browsing done by agents. The audit page calls llms.txt an emerging convention and warns that “Without this file, agents may spend more time crawling the site to understand its high-level structure and primary content.” A site with no file is marked “Not Applicable”, because “providing the file is optional at the moment.” Lighthouse’s scoring page describes the check as looking for a summary written for software, placed at the site’s main address.

John Mueller of Google has commented on it in public more than once. On Bluesky on 17 June 2025, he wrote “FWIW no AI system currently uses llms.txt”. That was a personal post, and it is now more than a year old. On Google’s Search Off the Record podcast, episode 111 of 15 June 2026, he said an LLM system “by design, can’t trust what is here as a way of differentiating between different websites.” He added that if “someone is already on your website, maybe some kind of automated system is helpful.”

Your store may already have one

On Shopify, the platform has already switched the file on for every store. Shopify’s developer changelog of 28 May 2026 says every store includes a default agents.md file at /agents.md, and that the store’s /llms.txt address points to the same content by default. Shopify’s agents.md documentation says agents.md tells “AI agents and shopping assistants” how to find what a store offers and how to buy from it, and that the llms.txt address mirrors its content.

We fetched the llms.txt of three Australian Shopify stores on 14 October 2026. That check covered three stores on one day. All three returned Shopify’s default document, titled “Agent Instructions”, which tells an AI agent it can browse products and shop at the store “on your user’s behalf”.

WordPress and WooCommerce stores get the file through their SEO plugin:

• All in One SEO says its llms.txt feature “is enabled by default” and can be disabled.
• Yoast offers it free in Yoast SEO, Yoast SEO Premium, Yoast WooCommerce SEO and Yoast SEO for Shopify. Yoast added it to Yoast SEO for Shopify on 31 March 2026.
• Rank Math says “The llms.txt is currently a proposal and not an official standard” and leaves its file empty until you configure it.
• SEOPress puts it inside an “Agent Readiness” toggle in SEOPress PRO 9.8.

Yoast says “llms.txt helps AI tools give better answers about your site”, and Rank Math says the file “can help AI assistants better understand your site”.

What three studies counted

Ahrefs, which sells SEO tools, studied the server logs (the record a website’s server keeps of each request it gets) and live traffic of 137,210 websites in its web analytics product during May 2026, and published the results on 15 June 2026. Ahrefs says those customers “skew more technical and SEO-aware than the web at large”. Of the sites with an llms.txt, 97% got no requests for it at all that month.

Among the other 3%, about 1,100 sites, OpenAI’s GPTBot crawler was the top fetcher among what Ahrefs calls “named AI tools”. Ahrefs warns that “fetched doesn’t mean read”. Googlebot, the crawler Google uses for search, also fetched the files, which Ahrefs attributes to its normal crawling of any address it finds on a site.

SE Ranking, which also sells SEO tools, looked at nearly 300,000 domains in a study published on 7 November 2025. Its statistical tests and machine learning model showed “no effect of LLMs.txt on how often a domain is cited by LLMs.” That study came out six months before Google’s guide, so it describes the web of late 2025.

Common Crawl runs repeated crawls of the web. For its July 2026 crawl, it requested /llms.txt from 5,167,831 addresses, drawn at random from sites it can fetch, which is not the whole web. It also requested a longer variant of the file from a smaller sample. On 31 August 2026, it reported that 68.27% of all the files it examined were built from a standard template, the kind a plugin or platform writes.

What to do with your store’s llms.txt

Start by finding out whether your store already serves the file. Type your store’s address followed by /llms.txt into a browser. If a page of text loads, you have one.

On Shopify, leave the default alone. Shopify’s llms.txt documentation says “in most cases” not to add your own version to your theme, because its managed default keeps /llms.txt in line with /agents.md.

In WordPress or WooCommerce, check your SEO plugin’s settings. In Yoast, the llms.txt toggle “will be purple and show a checkmark when it is set to on.” In All in One SEO, it is on unless someone turned it off. If your plugin leaves the file off, or empty as Rank Math does until you configure it, you can add one or leave it.

We think llms.txt deserves none of your budget or time today, because none of OpenAI, Anthropic, or Perplexity says its assistant reads the file. That changes if one of them says its assistant does. Until then, if an app or a consultant charges extra for an llms.txt, ask them to name the AI assistant that reads it and show you where its maker says so.

The file worth your attention is robots.txt, the one those crawler pages do name. We found our own robots.txt “telling the major AI crawlers not to read our site”, and wrote it up in our article on AI crawlers. Once your robots.txt lets in the crawlers you want, the useful question is what AI assistants say about your store, and neither file can tell you that.

To see where your store appears in AI answers today, ask us for a free AI Search report. We write 50 questions a buyer might ask and run them through five AI tools, ChatGPT, Gemini, Microsoft Copilot, Google AI Overviews and Google AI Mode. We typically respond within one business day.

References

1. OpenAI, “Overview of OpenAI Crawlers”. https://developers.openai.com/api/docs/bots
2. Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”. https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
3. Perplexity, “Perplexity Crawlers”. https://docs.perplexity.ai/guides/bots
4. Google Search Central, AI optimization guide, last updated 10 July 2026. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
5. Jeremy Howard, “The /llms.txt file, v2”, llmstxt.org, published 3 September 2024, modified 10 August 2026. https://llmstxt.org/
6. John Gruber, “Markdown”, Daring Fireball. https://daringfireball.net/projects/markdown/
7. llmstxt.org, Changes. https://llmstxt.org/changes.html
8. Google Search Central, “Introduction to robots.txt”. https://developers.google.com/search/docs/crawling-indexing/robots/intro
9. Google Search Central, “Overview of Google crawlers and fetchers”. https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
10. OpenAI, API documentation llms.txt. https://developers.openai.com/api/llms.txt
11. Anthropic, developer documentation llms.txt. https://platform.claude.com/llms.txt
12. Perplexity, API documentation llms.txt. https://docs.perplexity.ai/llms.txt
13. Google, Gemini API documentation llms.txt. https://ai.google.dev/gemini-api/docs/llms.txt
14. Cloudflare, developer documentation llms.txt. https://developers.cloudflare.com/llms.txt
15. Google Search Central Blog, “A new resource for optimizing for generative AI in Google Search”, 15 May 2026. https://developers.google.com/search/blog/2026/05/a-new-resource-for-optimizing
16. Chrome for Developers, Lighthouse llms.txt audit, last updated 5 May 2026. https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt
17. Chrome for Developers, Lighthouse agentic browsing scoring. https://developer.chrome.com/docs/lighthouse/agentic-browsing/scoring
18. John Mueller, Bluesky post, 17 June 2025. https://bsky.app/profile/johnmu.com/post/3lrshm4gggs2v
19. Google Search Relations, Search Off the Record, episode 111 transcript, 15 June 2026. https://search-off-the-record.libsyn.com/should-i-use-markdown-for-my-site-transcript
20. Shopify developer changelog, 28 May 2026. https://shopify.dev/changelog/customize-llmstxt-llms-fulltxt-and-agentsmd
21. Shopify.dev, agents.md.liquid. https://shopify.dev/docs/storefronts/themes/architecture/templates/agents-md-liquid
22. All in One SEO, “How to create an llms.txt using All in One SEO”. https://aioseo.com/docs/how-to-create-an-llms-txt-using-all-in-one-seo/
23. Yoast, llms.txt feature page. https://yoast.com/features/llms-txt/
24. Yoast, Yoast SEO for Shopify release, 31 March 2026. https://yoast.com/yoast-seo-for-shopify-march-31-2026/
25. Rank Math, llms.txt knowledge base article. https://www.rankmath.com/kb/llms-txt/
26. SEOPress, Agent Readiness guide. https://www.seopress.org/support/guides/agent-readiness/
27. Ahrefs (Louise Linehan), llms.txt study, 15 June 2026. https://ahrefs.com/blog/llmstxt-study/
28. SE Ranking, llms.txt study, 7 November 2025. https://seranking.com/blog/llms-txt/
29. Common Crawl, “A content analysis of llms.txt files from the July 2026 crawl archive”, 31 August 2026. https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive
30. Shopify.dev, llms.txt.liquid. https://shopify.dev/docs/storefronts/themes/architecture/templates/llms-txt-liquid
31. Yoast help centre, “How to enable llms.txt”. https://yoast.com/help/enable-llmstxt/
32. Duelling Pixels, “Some AI Bots Train on You. Others Bring You Customers.”, 28 July 2026. https://duellingpixels.com/blog/business-online/some-ai-bots-train-on-you-others-bring-you-customers/
33. Duelling Pixels, AI Search Optimisation & GEO, new site build, 21 October 2026. https://duellingpixels.com/ai-search-optimisation/
34. Duelling Pixels, contact page (reply-time wording). https://duellingpixels.com/contact