SEO · 2026-11-14

Is Your CDN Blocking Google and AI Search? How to Check

Bot protection that challenges every automated visitor also stops Googlebot and AI search crawlers. Allow verified bots, target the bots you dislike, and check monthly.

Short answer: Yes, it can, and it usually happens silently. When a CDN or firewall challenges every automated visitor, Googlebot and AI search crawlers such as OAI-SearchBot and PerplexityBot get a challenge page instead of your content, and your pages slowly drop out of Google and AI answers. The fix is to allow verified search crawlers, block only the specific bots you do not want, and check your security logs after every CDN or security change.

Google flagged this exact problem in a Search Central post on 11 November 2026 about moving to a CDN without blocking Googlebot. This guide explains why it happens, gives you a 20 minute check, and lists the settings we recommend for businesses in the UK, US, Europe, Australia and the Gulf.

Why does moving to a CDN change how Google crawls you?

Two things happen at once when a site goes behind a CDN.

First, your site suddenly answers much faster. Google sets how hard it crawls a site partly by how quickly and reliably the server responds, so a faster site can earn a higher crawl rate. Google describes this as its crawlers seeing what looks like a hardware upgrade and ramping up to match. Its crawl budget documentation explains the same capacity logic.

Second, the CDN's caches start empty. Until the edge servers have a copy of each page, both people and crawlers are still fetched from your origin server. A crawl spike on a cold cache can hit a small origin hard on the first days.

Google's suggestion is to warm the cache before or right after the switch: request your highest traffic pages, seasonal or promotional pages and most crawled URLs through the CDN at a controlled pace. If your origin really is overwhelmed, return 429 or 503 for a short window, a few hours or one to two days at most, rather than a 403 wall or a challenge page. Google's page on HTTP status codes and network errors explains how its crawlers treat each code.

How does bot protection block search and AI crawlers?

Modern CDNs ship with bot protection that is easy to switch on and easy to forget. The trouble starts when a rule treats all automation as hostile.

  • Blanket challenges. A managed challenge or CAPTCHA on every automated request catches crawlers too. Google warns that leaving blanket bot protection on without exempting verified crawlers leaves them stuck on an error page, and that pages and shopping listings will then begin to drop out of Search.
  • Challenge pages that return 200. Some setups serve the challenge with a normal success code. To a crawler that looks like your page has been replaced by a near empty one, which is worse than an honest error.
  • One switch for "AI bots". Many teams block every crawler with "AI" in its description. That removes them from ChatGPT search, Claude and Perplexity answers as well as from training. Our earlier guide on robots.txt for AI crawlers covers the difference between training and search crawlers.
  • Settings you cannot make exceptions to. On Cloudflare's free plan, Bot Fight Mode cannot be skipped with custom rules. If it challenges a crawler you want, your options are to turn it off or move to a plan with Super Bot Fight Mode, which supports exceptions.

Cloudflare's AI crawler and agent policies show how subtle this gets. Policies are set separately for Search, Agents and Training, and Cloudflare notes that Training actions also apply to mixed purpose crawlers used for both search and training. A training block can therefore reach further than you meant.

A 20 minute check: is your CDN blocking the crawlers you want?

Run this on your homepage, one service or product page and one blog post.

  1. Test as Google. In Google Search Console, run a live test in the URL Inspection tool. If the rendered page shows a challenge, an error or a blank shell, Googlebot is not seeing your content.
  2. Read Crawl Stats. Open Settings, then Crawl Stats. Look at host status and the breakdown by response code. A rise in 403, 429 or 5xx responses after a CDN or security change is the signal.
  3. Search your CDN security events. Filter events by user agent for Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot. On Cloudflare, challenges from Bot Fight Mode are labelled as such in the Service field of Security Events.
  4. Open the AI crawler screen. On Cloudflare this is AI Crawl Control. Its Crawlers tab lists each AI crawler with allowed and unsuccessful requests, so you can see whether search crawlers are being turned away.
  5. Compare with robots.txt. Make sure your CDN rules and robots.txt say the same thing. Our free AI robots.txt generator builds a file that blocks training crawlers and keeps search crawlers.

Which crawlers should get through?

Crawler Operator What it does Let it through? How to verify it
Googlebot Google Google Search crawling and indexing Yes Reverse DNS or Google's published IP ranges (guide)
Bingbot Microsoft Bing and Copilot search Yes Reverse DNS to search.msn.com
OAI-SearchBot OpenAI Surfaces sites in ChatGPT search Yes, if you want ChatGPT visibility OpenAI's published IP ranges (docs)
ChatGPT-User OpenAI Fetches a page when a user asks Usually yes OpenAI's published IP ranges
PerplexityBot Perplexity Indexes for Perplexity answers Yes, if you want Perplexity visibility Perplexity's published IP ranges (docs)
GPTBot, ClaudeBot OpenAI, Anthropic Model training Your choice Block by name in robots.txt or a targeted CDN rule

Never trust the user agent string alone. Anyone can call their scraper Googlebot, so allow rules should rely on your CDN's verified bot detection or on published IP ranges.

What settings keep you protected and visible?

  • Turn on the verified bot allowance. Most CDNs and firewalls have a toggle or managed rule for known good bots. On Cloudflare, custom rules can use the known bots field, available on all plans, as in its example rule to allow traffic from search engine bots.
  • Target, do not blanket. Block the specific bots you object to by name, in robots.txt and, for bots that ignore it, in a CDN rule. Google recommends this over blocking all automated clients.
  • Put challenges where attacks happen. Login, checkout, account, search and form endpoints deserve rate limits and challenges. Your public marketing and content pages usually do not.
  • Use honest status codes. Short spells of 429 or 503 tell crawlers to come back later. A permanent 403 or a challenge page tells them the page is gone.
  • Recheck after every change. Security settings drift as plugins, presets and people change. Add the check above to your monthly report.

When should you not loosen protection?

Opening the door to verified crawlers is low risk. Loosening everything is not. Keep strict protection if you are under an active attack, if your origin genuinely cannot take more load (fix capacity or caching first, and return 503 briefly in the meantime), or if a page holds private data. Robots.txt and crawler rules are not security: anything private belongs behind a login. And if your legal or content team has decided to keep content out of AI products altogether, block those crawlers deliberately and document the trade off, rather than letting a default switch decide.

Key takeaways

  • A CDN can make Google crawl you harder and, at the same time, bot protection can stop it crawling you at all.
  • Allow verified search crawlers, block named bots you object to, and never rely on user agent strings alone.
  • Use 429 or 503 for short overload windows; avoid 403 walls and challenge pages for crawlers.
  • AI crawler switches can be broader than they look, because training policies may cover mixed purpose crawlers.
  • Check URL Inspection, Crawl Stats and your CDN security events after every CDN or security change, then monthly.

FAQ

Can Cloudflare block Googlebot?

It can if a rule challenges or blocks traffic without exempting known good bots, for example a country block or a blanket challenge. Cloudflare's own firewall FAQ warns that block and challenge rules based on path, ASN or country can unintentionally affect search crawlers. Use the known bots field in your rules and check Security Events for crawler user agents.

Will blocking AI training crawlers remove me from ChatGPT or Perplexity answers?

Not if you block only the training crawlers. GPTBot and ClaudeBot are training crawlers, while OAI-SearchBot, Claude-SearchBot and PerplexityBot power search answers. Be careful with one click AI blocks, because they may cover search and mixed purpose crawlers too.

How long can my site return 503 before Google drops pages?

Google's November 2026 guidance suggests 429 or 503 only for a short window, a few hours or one to two days at most. Longer spells risk crawlers backing off and URLs dropping from the index. If overload lasts longer than that, fix caching or server capacity instead.

How do I know a visitor claiming to be Googlebot is real?

Run a reverse DNS lookup on the IP, check that it resolves to a Google domain, then do a forward lookup to confirm it matches, or compare the IP with Google's published crawler IP ranges. Many CDNs do this for you through their verified bot detection.

Not sure what your CDN, firewall and robots.txt are telling search and AI crawlers right now? Ask our SEO team to run this check for you.