Cloudflare blocking AI crawlers is one of those technical problems that can exist without anyone on the marketing team noticing.
Your website can look completely normal.
Google Analytics can still collect traffic.
Forms can still work.
Your rankings may even appear stable.
But somewhere between an AI crawler requesting a page and your server returning it, a security layer can decide that the request looks automated and block, challenge or restrict it.
That security layer might sit in:
- Cloudflare
- another CDN
- a web application firewall
- the hosting environment
- a WordPress security plugin
- custom server rules
- robots.txt
This matters because AI search platforms use their own crawlers.
OpenAI says publishers that want their site content to be discoverable and included in ChatGPT Search summaries should not block OAI-SearchBot. It also separates OAI-SearchBot from GPTBot, which is associated with potential training use.
That distinction is useful.
You may want search visibility without opening every AI crawler for every purpose.
The job is not:
Allow every bot.
It is:
Know which automated visitors you are blocking, why you are blocking them and whether those rules conflict with your visibility strategy.
Table of Contents
- Where AI crawler blocking actually happens
- Why Cloudflare deserves a separate check
- Search bots and training bots are not the same thing
- How to check whether Cloudflare is blocking AI crawlers
- Why an “Allow” setting may still not be enough
- What response codes should you look for?
- How to allow useful AI crawlers safely
- When blocking AI crawlers is the right decision
- What about SiteGround?
- What about WP Rocket?
- Frequently Asked Questions About Cloudflare and AI Crawlers
- What to do next
Where AI crawler blocking actually happens
robots.txt gets most of the attention because it is easy to inspect.
But it is only one layer.
A crawler might be allowed in robots.txt and still fail to retrieve the page.
The path can look something like this:
AI crawler → CDN → WAF → host → security plugin → website
A rule at any one of those layers can interrupt the request.
For example:
CDN
A content delivery network can identify automated traffic and apply bot-management rules.
Web application firewall
A WAF can block or challenge traffic according to IP address, country, user agent, bot score or custom rules.
Hosting environment
A host can operate its own anti-bot systems or server-level firewall.
Security plugin
WordPress tools can add their own blocking, rate-limiting or firewall rules.
Server configuration
Rules inside .htaccess, Nginx configuration or another server layer can explicitly deny particular user agents.
That is why checking only:
robots.txt
is not enough.
Why Cloudflare deserves a separate check
Cloudflare now gives site owners unusually granular control over AI crawler access.
Its AI Crawl Control product can show which AI crawlers are requesting content and lets the site owner choose whether individual crawlers should be allowed or blocked. Cloudflare says the feature is available across its plans.
Cloudflare AI Crawl Control documentation
Cloudflare also distinguishes AI traffic by behaviour.
Its current bot controls separate categories including:
- Search
- Agent
- Training
This is a useful development because the commercial decision is not necessarily binary.
You may want a crawler that supports search discovery.
You may not want a crawler whose purpose is model training.
As of September 2026, Cloudflare’s newer defaults for newly added domains treat those categories differently rather than relying on one universal AI-bot switch. Search traffic remains allowed under those new defaults, while some Training and Agent traffic can be blocked on ad-supported pages.
So do not rely on an old screenshot or tutorial.
Check the settings that actually exist in your account today.
Search bots and training bots are not the same thing
This is one of the most important distinctions in AI crawler configuration.
For OpenAI:
OAI-SearchBot supports ChatGPT Search discovery.
GPTBot is associated with potential model-training use.
OpenAI’s current publisher guidance specifically says not to block OAI-SearchBot if you want content to be discoverable and included in ChatGPT Search summaries and snippets. It separately tells publishers to block GPTBot on pages they want excluded from potential training.
That means:
Blocking GPTBot does not automatically mean you must block OAI-SearchBot.
Cloudflare’s crawler reference also classifies OAI-SearchBot as AI Search, while GPTBot is classified separately as an AI crawler. It similarly distinguishes Claude’s search crawler and Perplexity’s search crawler.
This is why broad rules such as:
Block anything AI-related
can be too crude for a business that actually wants AI search visibility.
How to check whether Cloudflare is blocking AI crawlers
If your domain uses Cloudflare, start inside AI Crawl Control.
Cloudflare’s current workflow lets you view crawler activity and filter by crawler, operator, hostname, status code and path.
The practical process is:
1. Open AI Crawl Control
Select the relevant domain.
2. Review the Crawlers section
Look for relevant search crawlers.
Examples can include:
- OAI-SearchBot
- Claude-SearchBot
- PerplexityBot
Cloudflare’s current bot reference identifies those as search-oriented AI crawlers.
3. Look at successful and unsuccessful requests
Do not focus only on request volume.
Look for patterns in failed requests.
4. Check the action
Is the crawler:
Allow
or:
Block
5. Review the Security / WAF rules
This part is important.
An AI crawler may show as allowed inside AI Crawl Control while another firewall rule still blocks it.
Why an “Allow” setting may still not be enough
This is where Cloudflare configuration gets more interesting.
Cloudflare explains that AI Crawl Control works alongside the WAF and other bot-management systems.
An existing WAF rule can still affect an AI crawler even if that crawler is permitted in AI Crawl Control. Cloudflare specifically recommends reviewing custom WAF rules when an intended crawler is still failing to access the site.
Cloudflare guidance on AI Crawl Control and WAF rules
For example, imagine you have:
AI Crawl Control:
OAI-SearchBot → Allow
but also:
Custom WAF rule:
Challenge all automated traffic
The second rule can still become the problem depending on how the rules are configured and ordered.
So when troubleshooting, check both:
AI crawler policy
and
general firewall policy
What response codes should you look for?
The most obvious signal is often:
403 Forbidden
That means access was denied.
OpenAI specifically notes that CDN, firewall and bot-protection systems can block legitimate crawler requests and that a 403 response can prevent access.
Other signals can include:
- managed challenges
- repeated redirects
- rate limiting
- unusual 4xx responses
- unsuccessful crawler requests
A browser test from your own laptop is not enough.
Your browser and the crawler do not necessarily receive the same response.
Server or CDN logs are far more useful.
How to allow useful AI crawlers safely
Do not switch off all security protection.
That is not the solution.
Use the narrowest rule that achieves the intended result.
Step 1: Decide which crawler you actually want
For ChatGPT Search visibility, the obvious OpenAI crawler to evaluate is:
OAI-SearchBot
OpenAI specifically says this crawler should not be blocked if you want content included in ChatGPT Search summaries and snippets.
Step 2: Allow the crawler in AI Crawl Control
Cloudflare provides individual allow and block actions for detected crawlers.
Step 3: Check competing WAF rules
Make sure another broad bot rule is not overriding the intention.
Step 4: Check robots.txt
Allowing a crawler through the firewall does not help if your robots.txt still says:
User-agent: OAI-SearchBot
Disallow: /
Step 5: Test again
Do not assume a settings change worked.
Review the crawler activity and response status after the change.
When blocking AI crawlers is the right decision
Not every bot deserves access.
There are legitimate reasons to block a crawler.
For example:
- you do not want particular content used for training
- the crawler has no commercial value to you
- it ignores your stated crawling preferences
- it generates unacceptable load
- sensitive or licensed content should not be retrieved
- the crawler’s behaviour conflicts with your content strategy
Cloudflare itself positions AI Crawl Control as a way to make those decisions crawler by crawler rather than automatically opening everything.
The strategic question is:
What value does this crawler provide, and what access does it actually need?
That is better than:
AI bot = block.
What about SiteGround?
Do not assume SiteGround automatically blocks the AI search crawlers you want.
SiteGround currently publishes a list of AI crawlers it allows on its servers. That list includes:
- OAI-SearchBot
- ChatGPT-User
- Claude-SearchBot
- PerplexityBot
among others.
SiteGround’s current allowed AI crawler list
SiteGround also documents how customers can manually block specific AI user agents through .htaccess.
So if a SiteGround site has an AI-access problem, check for:
- old .htaccess rules
- custom security configuration
- WordPress security plugins
- another CDN such as Cloudflare sitting in front of SiteGround
Do not automatically blame the hosting platform.
What about WP Rocket?
WP Rocket is primarily a performance and caching plugin.
It is not where I would start looking for an AI crawler allow/block setting.
The more relevant relationship is the opposite: security tools can block WP Rocket’s own external services.
WP Rocket’s documentation specifically notes that Cloudflare Bot Fight Mode and firewall rules can incorrectly block its own requests.
That is a useful reminder of the broader principle:
Automated traffic can be blocked even when it is legitimate.
If you use WP Rocket alongside Cloudflare, Wordfence or hosting-level bot protection, review the actual firewall layer rather than searching WP Rocket for an AI-search toggle.
Frequently Asked Questions About Cloudflare and AI Crawlers
Common questions about Cloudflare bot controls, AI crawler access, SiteGround, WP Rocket and the difference between search and training crawlers.
Does Cloudflare blocking AI bots affect Google?
AI-specific controls and Googlebot are not necessarily the same thing.
Cloudflare classifies Googlebot as a search-engine bot and OAI-SearchBot as AI Search.
However, a broad custom firewall rule that blocks or challenges automated traffic could potentially affect legitimate search crawlers if it is poorly configured.
Cloudflare recommends allowing verified bots where appropriate.
Check the exact rule rather than assuming all bot controls behave identically.
Will allowing AI crawlers increase my bandwidth bill?
Potentially. Any crawler creates requests and consumes some resources.
Whether that materially affects cost depends on:
- Crawl frequency
- Page size
- Hosting plan
- CDN pricing
- Caching
- Traffic volume
Do not guess.
Cloudflare AI Crawl Control provides crawler request data that can help you measure activity before deciding whether the volume is commercially significant.
Which Cloudflare setting should I check first?
Start with AI Crawl Control.
Then review:
- Security and bot policies
- WAF custom rules
Do not stop after seeing an AI crawler set to Allow. Another WAF rule may still affect it.
Which SiteGround setting should I change?
There is no universal SiteGround switch that every site should change.
SiteGround currently lists several AI search and user crawlers as allowed.
If access is failing, inspect:
- .htaccess rules
- Security plugins
- CDN rules
- Any custom bot blocks
Check those layers before changing hosting-level protection.
Which WP Rocket setting allows AI crawlers?
WP Rocket is not normally the layer controlling AI crawler access.
Check your CDN, firewall, host and security plugin first.
If Cloudflare is also installed, Cloudflare’s bot or WAF configuration is more relevant to this particular issue.
Should I allow GPTBot for ChatGPT Search?
Do not treat GPTBot and OAI-SearchBot as interchangeable.
OpenAI’s current publisher documentation identifies OAI-SearchBot as the crawler relevant to ChatGPT Search discovery.
GPTBot is documented separately for pages publishers may want excluded from potential model training.
Your decision on each crawler can therefore be different.
What to do next
AI visibility can fail before the crawler ever reads a headline.
That is why this problem is easy for marketing teams to miss.
Your:
- content
- schema
- internal links
- case studies
- service pages
can all be perfectly good.
But if the crawler receives:
403 Forbidden
or gets caught in a challenge before reaching the page, the content itself is not the first problem.
So check the access path.
Start with:
- robots.txt
- Cloudflare AI Crawl Control
- general bot settings
- WAF custom rules
- hosting protection
- WordPress security plugins
- server logs
Then make the smallest change required.
Allow the crawlers that support the visibility you actually want.
Keep blocking traffic that does not.
That is a much stronger security and GEO strategy than switching everything on or everything off.
If you want a wider check of crawler access, rendering and AI-search visibility:
Get your free AI visibility audit
