Cloudflare was 403-ing ChatGPT, Perplexity and Claude on my site, and my logs never knew For three weeks I wrote content aimed squarely at answer engines. An llms.txt, FAQPage JSON-LD on two pages, a
By Coderz Club · 2026-08-09 · Tags: ai, go
Cloudflare was 403-ing ChatGPT, Perplexity and Claude on my site, and my logs never knew
For three weeks I wrote content aimed squarely at answer engines. An llms.txt, FAQPage JSON-LD on two pages, a comparison page built to be quotable, a "how it works" page with HowTo markup. The site is a small dating app I run on my own server, loviam.com, Symfony and PostgreSQL on a single box behind Cloudflare. Referrals stayed at zero. The only external referrer my analytics table had ever recorded, over its whole history, was chatgpt.com: three visits, one day in July, never seen again. I assumed the content was not good enough. The robots had never read it. The five minute probe that should have been the first thing I did Nothing in my usual instruments could see the problem, so I stopped looking at them and asked the site directly, once per user agent, at its real public address: for ua in \ "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \ "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \ "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \ "Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +claudebot@anthropic.com)" ; do code=$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://www.loviam.com/) echo "$code $ua" done 200 Googlebot 403 OAI-SearchBot 403 PerplexityBot 403 Claude-SearchBot Same result for ChatGPT-User, Claude-User and MistralAI-User. Meanwhile bingbot, DuckDuckBot, Applebot, YandexBot and an ordinary browser all got a 200. The 403 body was eight bytes long, blocked., with server: cloudflare in the headers, and there was no matching line in the Apache access log. The refusal happened at the edge. My origin never heard about it. Why every instrument I had was blind to this This is the part worth stealing, because the failure mode generalises. Application logs cannot record a request that never arrives. Obvious once said, easy to forget when you are grepping the access log for PerplexityBot and concluding "it never came". First party analytics is worse than blind, it is reassuring. My beacon fires from the page. A robot that gets a 403 never gets the page, so it never appears, so the dashboard looks exactly like "nobody is interested". My deployment smoke tests talked to the wrong server. They run with curl --resolve www.loviam.com:443:127.0.0.1, which is the right call for a deployment gate: you want to test the code you just shipped, not the CDN cache. But it means the whole test suite speaks to Apache directly and Cloudflare is invisible to it by construction. Thirty-eight green checks, every deploy, on a path no visitor uses. And the managed robots.txt actively pointed the wrong way. Cloudflare injects its own block above yours. Mine named GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, meta-externalagent, Google-Extended and Applebot-Extended, all of them training crawlers. Blocking those is a defensible editorial decision and I stand by it. But the HTTP filter was blocking something else: the search and on-demand reading robots, which were not named in that file at all. So a perfectly obedient crawler read "you are allowed", requested the page, and got a 403. That mismatch between the two settings is the tell. It is not a policy, it is a misconfiguration, and reading either file alone will never show it to you. The fix is one toggle, and it is not the one named after crawlers Cloudflare dashboard, zone level: Security → Settings → Block AI bots. Not "AI Crawl Control", which is where I looked first and where the interesting per-crawler table lives. The switch that returns the 403 is the plain one in Security Settings, and it treats "AI bot" as a single category: the crawler that builds ChatGPT's search index and the crawler that scrapes you for training data are the same thing to it, even though for a publisher they are close to opposites. One sends you traffic. The other does not. Also worth checking afterwards: the managed robots.txt block, so it does not still name a robot you have just decided to let through. The guard I put in, and why it is a timer and not a test What broke here is not my code. It is an edge configuration that can change without a deploy, from a dashboard, possibly by a provider default I never chose. A test in CI would only prove it was fine at build time. So it is a daily command instead, app:seo:check-crawlers, which: probes the real public URL, through Cloudflare, never --resolve; walks sixteen user agents in three families: search, answer engines, training; checks two independent things per robot, the actual HTTP status and whether the served robots.txt names it in a Disallow, because those two disagreeing is the exact signature I missed; keeps an ordinary browser user agent as a control, so that "the robots are blocked" is never confused with "the site is down"; fails for search and answer engines, and never for training crawlers, which stay blocked on purpose. A systemd timer
For three weeks I wrote content aimed squarely at answer engines. An llms.txt, FAQPage JSON-LD on two pages, a comparison page built to be quotable, a "how it works" page with HowTo markup. The site is a small dating app I run on my own server, loviam.com, Symfony and PostgreSQL on a single box behind Cloudflare. Referrals stayed at zero. The only external referrer my analytics table had ever recorded, over its whole history, was chatgpt.com: three visits, one day in July, never seen again. I assumed the content was not good enough. The robots had never read it. The five minute probe that should have been the first thing I did Nothing in my usual instruments could see the problem, so I stopped looking at them and asked the site directly, once per user agent, at its real public address: for ua in \ "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \ "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \ "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \ "Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +claudebot@anthropic.com)" ; do code=$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://www.loviam.com/) echo "$code $ua" done 200 Googlebot 403 OAI-SearchBot 403 PerplexityBot 403 Claude-SearchBot Same result for ChatGPT-User, Claude-User and MistralAI-User. Meanwhile bingbot, DuckDuckBot, Applebot, YandexBot and an ordinary browser all got a 200. The 403 body was eight bytes long, blocked., with server: cloudflare in the headers, and there was no matching line in the Apache access log. The refusal happened at the edge. My origin never heard about it. Why every instrument I had was blind to this This is the part worth stealing, because the failure mode generalises. Application logs cannot record a request that never arrives. Obvious once said, easy to forget when you are grepping the access log for PerplexityBot and concluding "it never came". First party analytics is worse than blind, it is reassuring. My beacon fires from the page. A robot that gets a 403 never gets the page, so it never appears, so the dashboard looks exactly like "nobody is interested". My deployment smoke tests talked to the wrong server. They run with curl --resolve www.loviam.com:443:127.0.0.1, which is the right call for a deployment gate: you want to test the code you just shipped, not the CDN cache. But it means the whole test suite speaks to Apache directly and Cloudflare is invisible to it by construction. Thirty-eight green checks, every deploy, on a path no visitor uses. And the managed robots.txt actively pointed the wrong way. Cloudflare injects its own block above yours. Mine named GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, meta-externalagent, Google-Extended and Applebot-Extended, all of them training crawlers. Blocking those is a defensible editorial decision and I stand by it. But the HTTP filter was blocking something else: the search and on-demand reading robots, which were not named in that file at all. So a perfectly obedient crawler read "you are allowed", requested the page, and got a 403. That mismatch between the two settings is the tell. It is not a policy, it is a misconfiguration, and reading either file alone will never show it to you. The fix is one toggle, and it is not the one named after crawlers Cloudflare dashboard, zone level: Security → Settings → Block AI bots. Not "AI Crawl Control", which is where I looked first and where the interesting per-crawler table lives. The switch that returns the 403 is the plain one in Security Settings, and it treats "AI bot" as a single category: the crawler that builds ChatGPT's search index and the crawler that scrapes you for training data are the same thing to it, even though for a publisher they are close to opposites. One sends you traffic. The other does not. Also worth checking afterwards: the managed robots.txt block, so it does not still name a robot you have just decided to let through. The guard I put in, and why it is a timer and not a test What broke here is not my code. It is an edge configuration that can change without a deploy, from a dashboard, possibly by a provider default I never chose. A test in CI would only prove it was fine at build time. So it is a daily command instead, app:seo:check-crawlers, which: probes the real public URL, through Cloudflare, never --resolve; walks sixteen user agents in three families: search, answer engines, training; checks two independent things per robot, the actual HTTP status and whether the served robots.txt names it in a Disallow, because those two disagreeing is the exact signature I missed; keeps an ordinary browser user agent as a control, so that "the robots are blocked" is never confused with "the site is down"; fails for search and answer engines, and never for training crawlers, which stay blocked on purpose. A systemd timer