Last week we checked our website’s robots.txt file properly for the first time in a couple of weeks. To our surprise, it was telling the major AI crawlers not to read our site. 

We never wrote this rule.

Investigating further, we found a setting in Cloudflare rewrites your robots.txt for you, and we had switched it on without clarifying what it turned away. Then we did what most busy businesses do, we assumed the file said what we thought it said and never double-checked.

The useful part is not the mistake. It is what the mistake taught us.

Blocking AI crawlers sounds like one decision. It is not. Some of those bots are collecting content to train models, which may or may not be something you want. Others are only there because a customer has just asked an assistant a question about your business, and something needs to read your page to answer them. Those are opposite things wearing similar names, and a single setting had quietly been chosen for us.

Here is what we found, what it cost us, and how to check your own site in about thirty seconds. There is also a date worth putting in your calendar, because Cloudflare is changing the default for new domains on 15 September 2026.

The file you see is not always the file being served

Your robots.txt is a plain text file at the root of your domain that tells automated visitors which parts of your site they may read. On a WordPress site, your SEO plugin usually generates it, and you can view what it produced from the plugin settings.

That is the trap, the settings screen shows you what WordPress generated. It cannot show you what your CDN did to that file on the way out.

Ours looked completely ordinary in WordPress. Twenty-two lines, blocking the admin area, some feeds, some search URLs. Nothing unusual.

The file being served to the internet was eighty-three lines long. Cloudflare had added its own block at the top, and that block said this:

User-agent: GPTBot            Disallow: /

User-agent: ClaudeBot         Disallow: /

User-agent: CCBot             Disallow: /

User-agent: Google-Extended   Disallow: /

User-agent: Bytespider        Disallow: /

User-agent: Amazonbot         Disallow: /

User-agent: Applebot-Extended Disallow: /

User-agent: meta-externalagent Disallow: /

Along with a machine-readable declaration that our content could be indexed for search, but not used for AI training.

To be clear, this was our doing.

Cloudflare did not switch it on behind our backs. Someone here enabled the managed robots.txt setting at some point, most likely during a security tidy-up, and the description of it reads reasonably enough. It is the list underneath that nobody checks. More than 3.8 million domains are running this same setting, and over a million Cloudflare customers have used the one-click AI blocking option. We would guess most of them have not read the list either.

What it actually cost us, and what it did not

This is where it gets more interesting than a simple mistake, because the block was doing something more specific than “keeping AI out”.

Every one of these visits is recorded in our server logs, which means we can separate two very different things. A crawler asking for robots.txt is checking the rules. A crawler asking for a page is reading our content. Here are the eighteen days before we changed anything.

The last two rows are the ones we had blocked, and they tell the story better than any statistic we could have borrowed from someone else.

Anthropic’s ClaudeBot came to our site 174 times in eighteen days. On 165 of those visits, it asked for one file, robots.txt, read the line telling it to go away, and left without reading a single page. Amazon’s crawler did the same seven times out of seven. They knocked, they read the sign, they went home. Politely, repeatedly, for a fortnight.

Meanwhile, everything we had not blocked was reading us constantly. Two hundred and sixty-five times, somebody asked ChatGPT a question, and ChatGPT came and read one of our pages to answer them. Anthropic’s search crawler read us over a thousand times. None of that was ever affected.

So the block worked exactly as designed. It was never stopping AI systems from finding us or citing us. What we had opted out of was having our content used to train the models in the first place.

A detour, because we nearly published something false

You may have noticed GPTBot is missing from that table. It should be there, and the reason it is not is worth more than the table itself.

Our logs showed GPTBot making twenty-four page requests during the period we were blocking it, and never once asking for robots.txt. Read plainly, that says OpenAI ignored our rules and helped itself. It is a good story. We were about to write it.

Then we looked at where those requests came from. One address claimed to be both GPTBot and Common Crawl’s CCBot. Another claimed to be GPTBot, CCBot and Meta’s crawler, all three at once. OpenAI publishes the list of addresses GPTBot genuinely uses, and not one of the addresses in our logs appears on it.

They were impostors. Something was wearing GPTBot’s name to look respectable, which is trivially easy, because a user agent is just a line of text and anyone can type anything they like into it. Cloudflare, which verifies bot identity rather than believing the label, recorded genuine GPTBot activity exactly once in the same period.

There is a second lesson sitting alongside the main one. Do not trust the settings screen, trust the served file. And do not trust the user agent, trust verified identity. If you go and search your own logs after reading this, some of what you find will be lying to you, and the more striking the finding looks, the harder you should check it.

Which brings us back to the distinction that matters most here, and it is worth spelling out properly.

There are three kinds of AI bot and they are not interchangeable

Most writing on this subject treats “AI crawlers” as one thing. They are not. Using OpenAI as the example, because their documentation is the clearest:

• GPTBot collects content to train models. Blocking it keeps you out of the training data.
• OAI-SearchBot decides whether you can be shown in ChatGPT search results. Blocking this one removes you from answers.
• ChatGPT-User fetches a page on demand when a user asks about it. It is not used to decide whether you appear in search.

We were blocking the first one. We had never blocked the second or third, which is why we kept appearing in ChatGPT answers throughout. If we had blocked OAI-SearchBot instead, the outcome would have been considerably worse and much harder to notice.

Anthropic is blunter. ClaudeBot serves more than one purpose, so blocking it is a heavier decision than blocking GPTBot. Our data shows Claude-SearchBot arriving 98 times while ClaudeBot sat at zero, which is the split working itself out in practice.

The one that actually stung was Google-Extended. That token governs whether your content can be used for Gemini and Google’s AI Overviews. We had it disallowed. We have spent months writing about being cited in AI answers while quietly excluding ourselves from Google’s.

Worth knowing, and almost nobody gets this right, Google-Extended is not a crawler at all. It has no user agent and never appears in your server logs. It is a control token that governs what Google does with content it has already collected through Googlebot. You cannot see its effect by watching traffic. You can only see it by reading your robots.txt.

What changes on 15 September 2026

Until now, all of this has been opt-in. Cloudflare has never switched AI blocking on for an existing site without being asked.

That changes in September.

From 15 September 2026, newly onboarded Cloudflare domains will have Training and Agent crawlers blocked by default on pages that display ads, with Search crawlers still allowed.

Two things follow from that. If you register a domain and put it behind Cloudflare after that date, you will start with restrictions you did not choose, and you should check what they are rather than assume. And if you already have a domain there, nothing changes automatically, which means anything currently blocking your site is something someone chose, possibly years ago, possibly not you.

There is a further wrinkle worth understanding. Cloudflare now sorts crawlers into Search, Agent and Training categories, but plenty of real crawlers do more than one job. When a bot spans categories, it gets blocked according to the broadest restriction that applies to it. Block Training, and you can lose Search on any crawler that does both. That is how a reasonable-sounding choice produces an outcome nobody intended.

How to check your own site

This takes about thirty seconds and does not require any tools.

• Open your browser and go to yourdomain.com/robots.txt. Use the real file, not the preview inside your SEO plugin.
• Look for a section headed BEGIN Cloudflare Managed content, or any Disallow: / sitting under a named AI bot.
• If you find one, go to your Cloudflare dashboard, open AI Crawl Control, and look at the Managed robots.txt setting along with the per-crawler list under Security.

If you are not on Cloudflare, the same principle applies to any CDN, security plugin or managed host that can rewrite files on the way out. The rule is simple. Trust the served file, never the settings screen.

Then make an actual decision. Blocking AI training is a legitimate choice, and for a publisher whose words are the product, it is often the right one. What is not a good idea is doing it by accident and then wondering why you never appear in AI answers.

What we changed

We turned the managed robots.txt setting off. Our served file went from eighty-three lines back to twenty-two, and every one of those AI crawler blocks is gone. We left two genuine blocks in place, because we had reasons for those, and reasons are the point.

We are now counting, every day, how long it takes the training crawlers to come back.

That measurement nearly did not happen. Cloudflare keeps only twenty-four hours of crawler data on the free plan, with no way to look further back, so our first instinct was that we would have to wait weeks to have anything worth reporting. Our own server logs turned out to hold eighteen days of history that Cloudflare had already discarded, which is where every number in this article came from. If you are ever trying to measure something about your own site, check what you already have before you start waiting.

We will publish what we find, including if the answer turns out to be that nobody came back for a month. If you want us to look at what your site is currently telling AI systems, that check is part of our free site audit.