Did Cloudflare Just Hide Your Site From ChatGPT? How to Check

Did Cloudflare just hide your site from ChatGPT? If your site is on Cloudflare, it might have. On 15 September 2026 Cloudflare changed its default rules for AI crawlers. Your robots.txt did not change and your pages did not change, but what a crawler receives when it asks for them may have. Here is what changed, why robots.txt cannot tell you, and a two-minute check.

What Cloudflare changed on 15 September

Cloudflare announced the change in July, in its Content Independence Day post:

• On pages that display ads, the new default blocks crawlers used for AI training and AI agents, and allows crawlers used for search.
• A crawler that does more than one job falls under the strictest rule that applies to it. Cloudflare named Googlebot, Applebot and Bingbot as crawlers that mix search with training.
• The default applies to domains joining Cloudflare from that date. Existing customers were asked to confirm their settings beforehand.

If your site was already on Cloudflare and nobody opened the AI crawler settings, you may or may not be affected, depending on your plan and whether you carry ads. Cloudflare's post does not settle every case. The check below does.

Why the mixed-use rule matters

Most crawlers behind AI answers do more than one job. OpenAI runs GPTBot for training, OAI-SearchBot for its search index and ChatGPT-User for fetches a person triggers in a chat. Anthropic and Perplexity run similar sets. Under the strictest-rule logic, "block training, allow search" can quietly become "block the crawler", which removes you from answers you assumed you were still in.

Why robots.txt will not tell you

robots.txt is a request. Well-behaved crawlers honour it, but it enforces nothing.

Cloudflare sits in front of your server and applies its bot rules at the edge, before the request reaches you. It decides whether to pass the request, challenge it or return a 403, without consulting your robots.txt. So your file can say User-agent: GPTBot and Allow: / while GPTBot receives a 403 on every page.

The two-minute check

Fetch your homepage as the crawler would:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
  https://your-domain.com/

A 200 means GPTBot got a page. A 403 means it was refused. A 503, or a page that is mostly a JavaScript challenge, means it was asked to prove it is a browser, which a crawler cannot do.

Repeat with the Perplexity and Claude user agents, because Cloudflare's rules can differ per operator:

Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Mozilla/5.0 (compatible; ClaudeBot/1.0; [email protected])

If you would rather not run commands, the AI readiness audit in Presence Scout does the same thing. It reads your robots.txt, fetches your homepage as GPTBot, ClaudeBot and PerplexityBot, and shows both results side by side, so you can see when what your site says and what a crawler gets disagree. It is on every plan and uses no credits. We ran it on our own site before writing this, for the same reason we published a five-minute test for AI visibility tools: a claim you cannot check is not worth much.

What to change if you are blocked

Fix things in the order the request travels, from the edge inwards.

1. Cloudflare first. The AI crawler controls are under Security in the Cloudflare dashboard. You can allow specific crawlers, or set training, search and agent use separately. If you want to be named in AI answers, allow the crawlers behind the engines your buyers use, and accept that some of them also collect training data.

2. Then robots.txt. Name the crawlers you want rather than relying on a User-agent: * wildcard, which is a default rather than a decision:

User-agent: GPTBot
Allow: /
Disallow: /account

User-agent: PerplexityBot
Allow: /
Disallow: /account

Repeat your disallow rules inside every named group. A crawler that finds a group with its own name ignores the wildcard entirely, so naming GPTBot without repeating your private paths hands them over.

To state your intent for AI use as well, Cloudflare's Content Signals adds one line per group, such as Content-signal: search=yes, ai-input=yes, ai-train=no. search covers indexing, ai-input covers being read and cited in answers, and ai-train covers model training. Our own site says yes to all three, because we sell being named by AI assistants. Your answer may differ. What matters is that you decided.

3. Check again. Rerun the curl command or the audit. Edge settings can take a few minutes to apply, and the rule you changed is not always the one that was firing.

If your AI mentions dropped in September

When mentions in ChatGPT or Perplexity fall, the usual suspect is content: a competitor published a better comparison page or earned a citation you lost. An access change looks similar, but it hits every prompt at once, and it empties your citations too, because no answer can cite a page a crawler cannot fetch.

So before you rewrite anything, check whether the crawlers can still get in. It takes two minutes, and access is the only part of AI visibility that a single setting can switch off.