AI crawlers and robots.txt rules now play an important role in how platforms such as ChatGPT, Claude, Perplexity and Google access your website. Your robots.txt file may already contain instructions controlling which of these crawlers can reach your pages, even if nobody in the business has reviewed those rules recently.
A developer may have added a rule years ago. A security platform may be blocking bots automatically. A plugin may have changed the file. Or somebody may have copied a blanket “block AI” configuration without checking what each crawler actually does.
That can create an expensive distinction.
Some AI crawlers are used to collect content that may contribute to model training. Others help AI search products find and retrieve current information from your website. Blocking one does not necessarily have the same effect as blocking another.
The short answer: if AI search visibility matters to your business, search and retrieval crawlers generally need access to your public content. Training crawlers are a separate commercial decision.
For example, OpenAI treats GPTBot and OAI-SearchBot independently. A website can allow OAI-SearchBot for ChatGPT Search while blocking GPTBot for model-training purposes.
That training-versus-retrieval distinction is one of the first technical checks in a wider Generative Engine Optimisation strategy.
Table of contents
- What is an AI crawler?
- Which AI crawlers should you allow?
- GPTBot vs OAI-SearchBot
- What about Claude and Perplexity?
- Does Google-Extended affect AI Overviews?
- Why Bingbot, CCBot and Applebot matter
- How to check your robots.txt
- Example robots.txt configurations
- Is robots.txt legally binding?
- Should you block AI crawlers?
- Frequently asked questions
- robots.txt is only one part of AI visibility
What is an AI crawler?
An AI crawler is an automated system that requests content from websites.
That sounds similar to a traditional search engine crawler because technically it is. The important difference is what happens after the content has been collected.
For this discussion, there are three useful categories:
Search and discovery crawlers find and index information that may later be surfaced in an AI-powered search result.
User-triggered retrieval agents access a website because somebody has asked an AI assistant to retrieve or use information from that page.
Training crawlers collect public web content that may contribute to developing future AI models.
Those purposes should not be treated as interchangeable.
OpenAI, Anthropic and Perplexity now publish separate user-agents for different functions, giving website owners more control over what they allow.
Which AI crawlers should you allow?
There is no sensible universal rule that says every AI crawler should be allowed or every AI crawler should be blocked.
The first question should be:
What does this crawler actually do?
| Crawler | Owner | Primary purpose | Visibility consideration |
|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT Search | Allow if ChatGPT Search visibility matters |
| GPTBot | OpenAI | Model development and training | Separate commercial decision |
| ChatGPT-User | OpenAI | User-triggered retrieval | Separate from Search indexing |
| Claude-SearchBot | Anthropic | Claude search | Allow if Claude search visibility matters |
| ClaudeBot | Anthropic | Model development | Separate commercial decision |
| Claude-User | Anthropic | User-triggered retrieval | May affect user-directed retrieval |
| PerplexityBot | Perplexity | Search discovery | Allow if Perplexity visibility matters |
| Googlebot | Google Search | Important for Google Search and its AI features | |
| Google-Extended | Gemini training and grounding controls | Does not control Google Search inclusion | |
| Bingbot | Microsoft | Bing crawling and indexing | Normally allow for search discovery |
| CCBot | Common Crawl | Open web dataset | Separate data-use decision |
| Applebot | Apple | Apple search experiences | Allow if Apple search visibility matters |
| Applebot-Extended | Apple | Training-use control | Can be blocked separately from Apple search |
This is why simply searching your file for the word “AI” is not enough.
The commercial question is whether you want the specific activity that crawler represents.
GPTBot vs OAI-SearchBot: what is the difference?
This is one of the most important distinctions in AI crawler management.
GPTBot is not the same thing as OAI-SearchBot.
OpenAI says GPTBot is used to crawl content that may be used to improve and train its generative AI foundation models.
OAI-SearchBot, meanwhile, is used to surface websites in ChatGPT’s search features. OpenAI explicitly allows website owners to make these decisions independently.
That means this is possible:
OAI-SearchBot
AllowUser-agent: OAI-SearchBot Allow: /
GPTBot
DisallowUser-agent: GPTBot Disallow: /
In plain English:
Allow my public website to be considered for ChatGPT Search, but do not use GPTBot to collect my content for model training.
That does not guarantee your business will appear in a ChatGPT answer. It simply removes one potential technical barrier to search discovery.
You can review the current user-agent definitions in OpenAI’s official crawler documentation.
Does blocking GPTBot remove you from ChatGPT?
Not by itself.
OpenAI says OAI-SearchBot controls automatic crawling for ChatGPT Search, while GPTBot relates to potential training use. ChatGPT-User is different again and may access a page when a user specifically requests it.
This is exactly why blanket rules can be misleading.
The question is not:
“Do we allow OpenAI?”
It is:
“Which OpenAI activity are we allowing?”
What about ClaudeBot and Claude-SearchBot?
Anthropic has a similar separation.
ClaudeBot collects web content that could contribute to model development.
Claude-SearchBot navigates the web to improve Claude search results. Anthropic states that blocking Claude-SearchBot may reduce a site’s visibility and accuracy in those results.
Claude-User supports user-initiated retrieval when someone asks Claude to access information from the web.
A selective policy could therefore look like:
Claude-SearchBot
AllowUser-agent: Claude-SearchBot Allow: /
ClaudeBot
DisallowUser-agent: ClaudeBot Disallow: /
Again, whether training access should be allowed is a business decision. It should not be confused with the decision to make your public website discoverable through AI search.
Should you allow PerplexityBot?
If visibility in Perplexity matters to you, there is a clear reason to allow it.
Perplexity says PerplexityBot is designed to surface and link websites in its search results and is not used to crawl content for AI foundation-model training. It recommends allowing the bot if you want your website to appear in its search results.
Perplexity also operates Perplexity-User for user-triggered requests.
That makes PerplexityBot closer to a search crawler than a training crawler.
Does Google-Extended affect AI Overviews?
No.
This is one of the easier settings to misunderstand.
Google says Google-Extended controls whether content crawled by Google may be used for certain Gemini training and grounding purposes. It does not affect inclusion in Google Search and is not a Google Search ranking signal.
Google’s AI Overviews and AI Mode are part of Search.
Google states that a page must be indexed and eligible to appear in normal Search with a snippet to be eligible as a supporting link in AI Overviews or AI Mode. It also specifically recommends making sure crawling is allowed in robots.txt and through CDN or hosting infrastructure.
So:
Google-Extended
DisallowUser-agent: Google-Extended Disallow: /
does not mean:
Keep my website out of AI Overviews.
For AI features inside Google Search, Googlebot and normal Search eligibility still matter.
See Google’s official guidance on AI features and websites.
Why Bingbot, CCBot and Applebot matter
Not every crawler fits neatly into the same category.
Bingbot is Microsoft’s search crawler. Microsoft recommends keeping important content discoverable and using robots.txt to manage Bingbot access as part of building an effective Bing index.
CCBot belongs to Common Crawl, a non-profit organisation that maintains an openly accessible web crawl dataset. Common Crawl respects robots.txt and documents how publishers can block CCBot.
There is no promise that allowing CCBot will make your business appear in a particular AI product. Treat it as a separate decision about participation in a widely used open-web dataset.
Apple separates its controls too.
Applebot crawls content used in Apple search experiences including Siri, Spotlight and Safari. Apple says publishers can separately block Applebot-Extended to opt out of certain foundation-model training uses while still remaining eligible for search results.
The pattern should now be clear.
Search visibility and AI training are not always the same permission.
How to check your robots.txt in 60 seconds
Visit:
yourdomain.co.uk/robots.txtThen search for crawler names such as:
Also look for broad rules such as:
Wildcard crawler rule
All user-agentsUser-agent: * Disallow:
Under the Robots Exclusion Protocol, crawlers use Allow and Disallow rules associated with matching user-agent groups to determine which paths they may request.
But do not stop at robots.txt.
Your CDN, hosting platform or Web Application Firewall can also stop legitimate crawlers before they reach the page. Perplexity, for example, publishes specific WAF guidance because security rules can block its crawlers even when robots.txt allows them.
And crawler access alone does not mean the page is easy for an AI system to process. Sites that rely heavily on client-side rendering can create another layer of difficulty.
See our guide to why a website can be invisible in AI search because of JavaScript.
Example visibility-first robots.txt configuration
A simplified visibility-first approach might include:
Visibility-first crawler configuration
Allow search accessUser-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: Applebot Allow:
Do not copy this directly over an existing production robots.txt file.
Your site may have important rules for admin areas, internal search pages, ecommerce filters, staging directories or other URL patterns.
Review the existing configuration first.
Example search-visible, training-restricted configuration
A company that wants search discovery but prefers to restrict selected training use could consider a configuration such as:
Search-visible, training-restricted configuration
Example policy# OpenAI search User-agent: OAI-SearchBot Allow: / # OpenAI training User-agent: GPTBot Disallow: / # Anthropic search User-agent: Claude-SearchBot Allow: / # Anthropic training User-agent: ClaudeBot Disallow: / # Perplexity search User-agent: PerplexityBot Allow: / # Google Search User-agent: Googlebot Allow: / # Google AI training and grounding control User-agent: Google-Extended Disallow: / # Apple search User-agent: Applebot Allow: / # Apple model-training control User-agent: Applebot-Extended Disallow:
This is an example policy, not a universal recommendation.
Your decision should reflect your priorities around search visibility, intellectual property, training use, security and server resources.
Is robots.txt legally binding or a security tool?
robots.txt is not a security system.
The formal Robots Exclusion Protocol states that its rules are not access authorisation. It also warns that robots.txt should not replace genuine content security controls.
That matters particularly for staging websites.
If a development site contains private information, unreleased content or duplicated pages that should not be publicly accessible, use proper authentication or password protection.
Do not depend solely on:
User-agent: *
Disallow: /
A compliant crawler may respect it. That does not make the website private.
Should you block AI crawlers?
Possibly some. Probably not all if AI search visibility matters to your business.
The better approach is to decide crawler by crawler.
Ask:
Is this bot for search?
Is it for user-triggered retrieval?
Is it primarily a model-training control?
Then make the commercial decision.
A blanket block can feel safer because it is simple. It can also remove your public content from the discovery systems you are actively trying to appear in.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT Search?
No. OpenAI manages GPTBot and OAI-SearchBot separately. GPTBot relates to potential model training, while OAI-SearchBot is used for ChatGPT Search discovery.
Does Google-Extended affect AI Overviews?
No. Google states that Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode use Google’s Search infrastructure and normal Search eligibility.
Should I allow PerplexityBot?
If appearing in Perplexity search results matters to your business, Perplexity recommends allowing PerplexityBot.
Is robots.txt legally binding?
It is a standardised protocol that compliant crawlers are expected to follow, but it is not access authorisation or a security mechanism.
Should my staging website use robots.txt?
Yes, it can be an additional crawler instruction, but sensitive or non-public staging sites should also use genuine access controls such as authentication.
robots.txt is only one part of AI visibility
A perfect robots.txt file does not guarantee that ChatGPT, Claude, Perplexity, Gemini or another AI system will recommend your business.
It solves a narrower problem:
Can the relevant crawler access the content it needs?
After that come harder questions.
Can the platform clearly understand what your business does? Is the important information available in readable text? Do your entities and business details agree across the web? Is there enough evidence for an AI system to cite or recommend you confidently?
Being crawlable is not the same as being recommendable.
If you do not know where your business currently stands, start with Lobo Digital’s free AI visibility audit.
Get your free AI visibility audit.
