Should I block AI crawlers? It is the right question, but the answer depends less on your opinion of AI and more on how your website makes money.
If you are a publisher, research business or content company whose articles, data or intellectual property are the product, unrestricted AI crawling can create an obvious commercial concern.
If you run a cleaning company, consultancy, property business, law firm or other service business, your website usually has a different job. Its content exists to help potential customers discover you, understand what you do and make an enquiry.
Blocking the systems those buyers increasingly use to research suppliers may protect the content while reducing its commercial value.
That is why there is no honest universal answer.
For businesses that depend on being discovered, blanket blocking can carry a visibility cost. For businesses whose content itself has direct commercial value, restricting training crawlers may be entirely rational.
And increasingly, you do not have to choose between “allow everything” and “block everything”.
Table of contents
- Why are businesses blocking AI crawlers?
- Who genuinely benefits from blocking AI crawlers?
- Who can lose by blocking them?
- When discoverability is part of the business model
- Search crawling and AI training are different decisions
- Should I block AI crawlers if I run a service business?
- What Cloudflare may already be doing
- Can you charge AI companies instead of blocking them?
- A ten-minute AI crawler decision framework
- A sensible selective-access policy
- Marcus’s view
- Frequently asked questions
- Visibility should be a commercial decision
Why are businesses blocking AI crawlers?
There are legitimate reasons.
Some organisations spend heavily producing journalism, original research, specialist databases, photography, technical documentation or other material that has standalone value.
If an automated system can repeatedly collect that material without sending meaningful traffic, creating a commercial relationship or paying for access, the publisher may reasonably ask what it receives in return.
That is not an SEO question.
It is a content economics question.
Cloudflare now explicitly separates AI crawler management into behaviours such as Search, Agent and Training, recognising that different crawler activity has different commercial implications. Site owners can allow, block and, through its Pay Per Crawl programme, potentially charge selected crawler operators.
Cloudflare’s guidance on managing AI crawlers
The useful question is therefore not:
Should AI be allowed to crawl websites?
It is:
What value does this particular crawler receive, and what value does my business receive in return?
Who genuinely benefits from blocking AI crawlers?
Blocking makes the strongest commercial sense when the content itself is what customers are paying for.
Think about the distinction between two businesses.
A research publisher spends months building a proprietary dataset and charges customers to access it.
A facilities management company publishes a guide answering a buyer’s question because it wants that buyer to discover the company and request a quote.
Both have content.
But that content does completely different jobs.
For the publisher, unrestricted AI content scraping could substitute for something the company normally monetises.
For the service business, being found is often the point.
Businesses therefore have a stronger reason to investigate restricting a crawler when:
- the content has standalone licensing or subscription value
- access to the information is normally paid
- the material contains proprietary research or original datasets
- the crawler offers little clear discovery or referral benefit
- contractual or intellectual-property considerations require tighter control
- aggressive automated access creates meaningful infrastructure costs
This does not mean every publisher should block every AI crawler.
It means the commercial case for tighter control is easier to make.
Who can lose by blocking AI crawlers?
Businesses whose websites exist primarily to generate demand should approach blanket blocking more carefully.
If your site contains service pages, location pages, case studies, FAQs and educational content designed to persuade a potential buyer, wider discovery can have commercial value.
That principle is not unique to AI.
Lobo Digital’s commercial cleaning work provides a useful example from traditional organic search.
A UK commercial cleaning company began with no meaningful organic revenue. Lobo Digital built its SEO, content and technical foundations around commercially valuable searches. According to the published case study, the organic channel grew from zero to more than £1 million per year in revenue, generating around 30 to 50 organic enquiries per month at steady state.
Read the commercial cleaning SEO case study
That is not an AI crawler case study, and it should not be presented as one.
The relevant lesson is commercial.
When website content exists to create enquiries, discoverability has economic value.
AI search is a different channel, but the underlying calculation is similar. If prospective customers increasingly use AI-assisted search to research suppliers, voluntarily removing your public content from useful discovery mechanisms deserves more thought than simply ticking “block AI”.
Search crawling and AI training are different decisions
One of the biggest mistakes in this debate is treating every AI crawler as though it does the same thing.
It does not.
Some crawlers support search and retrieval.
Others collect content that may be used for model training.
Some companies operate separate controls for those activities.
OpenAI, for example, documents OAI-SearchBot for ChatGPT Search discovery and GPTBot for content that may be used to train its generative AI foundation models. OpenAI explicitly states that the two controls are independent, so a site can allow search crawling while blocking GPTBot training access.
OpenAI’s official crawler documentation
That means a business can take a position such as:
We want our public pages discoverable in AI search, but we do not want selected training crawlers collecting them.
That is a much more precise policy than either:
Allow every AI bot.
or:
Block every AI bot.
For the technical rules and individual user-agents, see Lobo Digital’s guide to AI crawlers and robots.txt.
AI crawlers and robots.txt guide
Should I block AI crawlers if I run a service business?
For most public-facing service businesses, start with a different question:
What am I actually trying to prevent?
If the answer is:
We do not want our blog used for model training.
then investigate the relevant training crawler.
Do not automatically block search and retrieval crawlers at the same time.
If the answer is:
We have private information that AI systems should never access.
robots.txt is not enough.
Private content should be protected using proper authentication and access controls.
And if the answer is simply:
AI scraping sounds bad.
that is not yet a commercial strategy.
Before blocking anything, identify:
- which crawler it is
- what that crawler does
- which pages it can access
- what commercial value you receive from that access
- what you potentially give up by blocking it
This matters particularly for companies investing in Generative Engine Optimisation.
Lobo Digital Generative Engine Optimisation services
There is little value in making your public site easier for answer engines to understand while accidentally preventing useful discovery crawlers from reaching it.
What Cloudflare may already be doing without your knowledge
This is worth checking even if nobody on your team remembers editing robots.txt.
Cloudflare introduced more granular AI bot controls in 2026, separating Search, Agent and Training behaviour.
For new domains onboarding to Cloudflare from 15 September 2026, its documented default is to block bots classified as Training or Agent on pages displaying ads while allowing Search. Cloudflare also says mixed-purpose crawlers combining Search and Training can be affected by training-block policies.
The important detail is new domains.
This should not be interpreted as meaning Cloudflare automatically changed every existing website to the same policy.
Your effective crawler policy can also be influenced by controls outside robots.txt.
Cloudflare states that its WAF and Bot Management rules can override Pay Per Crawl access. If another security rule blocks the crawler first, a separate crawler setting does not necessarily restore access.
So do not check only:
yourdomain.co.uk/robots.txt
Also review:
- Cloudflare AI bot policies
- AI Crawl Control
- WAF rules
- Bot Management
- CDN security settings
- server and CDN logs
- 403 responses
- rate limiting
- crawler verification
Can you charge AI companies instead of blocking them?
Potentially.
This is one of the more interesting developments on the publisher side of the debate.
Cloudflare’s Pay Per Crawl gives participating site owners the ability to control and monetise crawler access.
A crawler configured for paid access can either provide payment intent and receive the page successfully or receive an HTTP 402 Payment Required response containing pricing information. Cloudflare currently describes Pay Per Crawl as a closed beta and acts as merchant of record for the payment infrastructure.
Cloudflare Pay Per Crawl documentation
The broader commercial choice therefore becomes:
Allow. Block. Or potentially charge.
For a publisher selling access to proprietary material, that is an interesting development.
For a local cleaning company or consultancy whose public website exists to produce leads, charging crawlers to access ordinary service pages is a much harder commercial case to make.
Again:
Business model first. Crawler policy second.
Will blocking AI crawlers hurt your Google rankings?
It depends entirely on which crawler or control you block.
Google says Google-Extended is a standalone robots.txt control relating to certain Gemini training and grounding uses. Google also states that Google-Extended does not affect inclusion in Google Search and is not used as a Google Search ranking signal.
Google’s official Google-Extended documentation
Blocking Googlebot is a different matter.
Google says that to be eligible as a supporting link in AI Overviews or AI Mode, a page needs to be indexed and eligible to appear in Google Search with a snippet. Google specifically recommends ensuring crawling is allowed in robots.txt and through CDN or hosting infrastructure.
Google guidance for AI features and website visibility
So:
Will blocking AI crawlers hurt Google rankings?
is too broad a question.
First ask:
Which crawler am I actually blocking?
How to decide in ten minutes
You do not need a philosophical position on artificial intelligence.
You need answers to five commercial questions.
1. Is the content itself the product?
If people normally pay for the information, stronger controls deserve serious consideration.
2. Or is the content there to sell something else?
If your guides and service pages exist to create enquiries, discoverability has direct commercial value.
3. Is the crawler used for training, search or user retrieval?
Do not treat those as the same permission.
4. What do you receive in return?
Consider:
- citations
- referral traffic
- enquiries
- visibility
- licensing arrangements
- paid crawler access
- no obvious value at all
5. Can you make a narrower decision?
Before blocking an entire class of crawlers, ask whether you can:
- block only training crawlers
- allow search crawlers
- restrict particular paths
- protect premium content separately
- use authentication for genuinely private material
- restrict abusive traffic
- monetise access where commercially appropriate
That is normally a better decision than copying a universal block from somebody else’s robots.txt file.
A sensible selective-access approach
For many lead-generation businesses, a practical starting framework looks like this:
| Content or crawler situation | Starting consideration |
|---|---|
| Public service and location pages | Keep discoverable |
| Search-oriented crawlers | Usually preserve access if AI visibility matters |
| Training crawlers | Make a separate commercial and IP decision |
| Premium research or subscriber content | Consider restricting or monetising |
| Private or customer-only information | Protect with authentication |
| Unknown or abusive automated traffic | Investigate and restrict where justified |
This is not a universal policy.
It is a way of making the decision based on what the website is supposed to achieve.
Frequently Asked Questions About Blocking AI Crawlers
Common questions about AI crawler access, ChatGPT Search, AI model training, Google rankings and whether businesses should allow or block AI bots.
Will blocking AI crawlers hurt my Google rankings?
Not automatically.
It depends on what you block.
Google says Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. Blocking Googlebot is different because ordinary Google crawling and indexing underpin eligibility for Search and Google’s AI search features.
Can I block ChatGPT from my website?
You can control different types of OpenAI access separately.
OpenAI documents OAI-SearchBot for automatic ChatGPT Search discovery and GPTBot for content that may be used in model training. It says those controls are independent.
That means you do not necessarily have to make one universal “block ChatGPT” decision.
Can I allow ChatGPT Search but block AI training?
Yes. With platforms that provide separate crawler controls, you can make those decisions independently.
OpenAI specifically documents an example where OAI-SearchBot can be allowed for Search while GPTBot is disallowed for training purposes.
Can I charge AI companies to crawl my content?
There are emerging mechanisms for doing so.
Cloudflare’s Pay Per Crawl allows participating websites to monetise successful AI crawler access, although Cloudflare currently describes the programme as a closed beta.
What does pay per crawl mean?
Rather than only allowing or blocking automated access, a site can require payment.
In Cloudflare’s implementation, an AI crawler that does not provide the required payment intent can receive an HTTP 402 Payment Required response containing pricing information.
Should a small service business block AI training crawlers?
That is a separate decision from whether the business should remain discoverable in AI search.
A service business may decide that it does not want selected training access while still allowing search-oriented crawlers to access public pages.
The important thing is to avoid treating every AI crawler as the same thing.
Visibility should be a commercial decision
There is no prize for allowing every crawler.
There is no prize for blocking every crawler either.
The right policy depends on what your website is supposed to produce.
If your content is the product, restricting access, negotiating licensing arrangements or experimenting with paid crawler access may protect genuine commercial value.
If your website exists to generate enquiries, the equation changes.
A page that cannot be discovered cannot influence a buyer.
That does not mean handing every crawler unrestricted access.
It means separating search, retrieval and training, understanding what each one contributes and deciding deliberately.
Crawler access is only one part of GEO and AEO, but it is one of the first technical barriers worth checking.
If you do not know which AI platforms can currently access your website, whether your company is appearing in AI-generated answers or which competitors are being surfaced instead, start with Lobo Digital’s free AI visibility audit.
Get your free AI visibility audit
