SEO · 2026-11-09

Robots.txt for AI Crawlers: Block Training, Keep Visibility

Block the training crawlers (GPTBot, ClaudeBot, Google-Extended) and allow the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot).

Short answer: Most AI companies now run separate crawlers for model training and for search, so you can say no to one and yes to the other. To opt out of training while staying visible in AI answers, disallow GPTBot, ClaudeBot and Google-Extended, and keep OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot allowed. Blocking everything with an AI name is the most common mistake we see, and it quietly removes a site from ChatGPT search, Claude and Perplexity answers.

The rules changed again at the end of October 2026, when Google, Anthropic and other AI providers backed a shared training signal for robots.txt. This guide explains which bot does what, gives you a copy and adapt robots.txt, and covers what robots.txt cannot do.

Which AI crawlers train models and which power search?

Every major provider documents its crawlers. The table below summarises what each company says about its own bots in OpenAI's crawler overview, Anthropic's crawler article, Perplexity's crawler docs and Google's crawling overview.

Company Training control Search and answer crawler Fetches a page when a user asks
OpenAI GPTBot OAI-SearchBot (ChatGPT search) ChatGPT-User
Anthropic ClaudeBot Claude-SearchBot Claude-User
Google Google-Extended (a robots.txt token, not a separate crawler) Googlebot (Search, AI Overviews, AI Mode) Listed separately as user-triggered fetchers in Google's docs
Perplexity None listed; Perplexity says PerplexityBot is not used to train foundation models PerplexityBot Perplexity-User

Three details matter more than the names:

  1. The settings are independent. OpenAI and Perplexity both state that each crawler setting works on its own, so blocking GPTBot does not block OAI-SearchBot.
  2. Google-Extended does not touch Search. Google states that using Google-Extended "doesn't affect a site's inclusion in Search, nor do we use Google-Extended as a ranking signal in Search." It controls whether your content helps train future Gemini models, among other things.
  3. Blocking the search bots has a cost. Anthropic warns that disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results", and gives a similar warning for Claude-User.

How do you block AI training but stay in AI search?

Here is the pattern our team starts from. Adapt the private paths and the sitemap URL to your site.

User-agent: *
Disallow: /admin/
Disallow: /account/

# Opt out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Stay visible in AI search and answers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

Prefer not to edit it by hand? Our free AI robots.txt generator builds this file from a few choices and repeats your private paths for you.

Notice that the private paths are repeated in the search bot group. That is deliberate, and it is the gotcha behind most broken AI robots.txt files.

The group rule that catches almost everyone

A crawler follows only the most specific group that names it. If you add a group for OAI-SearchBot, that bot ignores everything under User-agent: *. Google's robots.txt specification describes the same behaviour for its crawlers: only one group applies to a given crawler.

So a file that says Disallow: /admin/ for everyone, then adds User-agent: OAI-SearchBot with Allow: /, has just opened /admin/ to that bot. Repeat any rule you still need inside every named group.

Changes are not instant

OpenAI and Perplexity both say it can take up to about 24 hours for their systems to reflect a robots.txt change. Test the file, publish it, then check again the next day rather than assuming it applied straight away.

What is the new shared AI training signal?

On 30 October 2026, Google announced it will support a shared AI training control for robots.txt, based on IETF AIPREF work, with the aim of supporting it "along with other AI providers in 2027". The idea is one rule that tells every participating provider your training preference, instead of one rule per company.

Google's crawler documentation, as reported by Search Engine Roundtable, says it intends to support the signal through the user agent __usage_ai-training in early 2027. Google also said that if a site opts out with this signal and does not name Google-Extended, it will treat that as a Google-Extended disallow. Company specific rules keep working alongside it.

Anthropic's crawler article says that as of 30 October 2026 it has begun recognising the signal, rolling out over time, and that a rule naming ClaudeBot takes priority over the general signal.

Our advice for now:

  • Keep your named bot rules. The shared signal is still rolling out and the underlying proposal is an IETF draft, not a final standard.
  • You can add the shared group today. It does no harm, and it will cover providers as they adopt it:
User-agent: __usage_ai-training
Disallow: /
  • Review the file again in early 2027, when Google's support is due, and check each provider's documentation for the exact syntax it reads.

What robots.txt cannot do

Robots.txt is a polite request, not a lock. Plan around these limits:

  • User-triggered fetchers may ignore it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt because a person asked for the page.
  • It does not protect private content. Anything that must stay private needs authentication, not a Disallow line. A Disallow line also tells everyone where the interesting folders are.
  • Blocking by IP is fragile. Anthropic says blocking its IP addresses may not reliably keep you opted out, and recommends robots.txt instead.
  • It does not control how Google shows you in AI Overviews. Google-Extended has no effect there. Google's page on controlling what you share lists the real levers: noindex, robots meta tags, data-nosnippet, X-Robots-Tag and a site-wide Search generative AI control in Search Console. Use them with care, because opting out of AI features also removes the traffic and impressions those features send.

Should your business block AI training at all?

There is no universal right answer, and anyone who claims to have hard numbers on the effect of training opt-outs should show their source. It is a business decision. Here is how we frame it with clients:

Your situation Training crawlers Search crawlers
Agency, SaaS or local service business that wants to be recommended Usually allow Always allow
Publisher whose content is the product (paid articles, research, data) Often block Allow, unless you have a licensing deal that says otherwise
Site with original creative work you want to protect Block Allow, so people can still find you
Members area, portal or internal tool Block Block the private paths, and use authentication

We allow every crawler on neztdigital.com, because our goal is to be known and recommended. A publisher selling subscriptions is right to choose differently. What nobody should do is block search crawlers by accident while trying to stop training.

If you want the bigger picture on earning AI citations, start with our guide to getting cited by AI search.

Key takeaways

  • AI companies separate training crawlers from search crawlers, so you can opt out of one and keep the other.
  • Block GPTBot, ClaudeBot and Google-Extended to opt out of training. Keep OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot allowed to stay visible.
  • A bot with its own group ignores the User-agent: * rules. Repeat your private paths in every group.
  • The shared __usage_ai-training signal is coming, with Google aiming for early 2027. Add it, but keep the named rules.
  • Robots.txt is not security. Use authentication for anything private.

FAQ

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler that surfaces sites in ChatGPT search, and says each setting is independent. Blocking GPTBot while allowing OAI-SearchBot keeps you eligible for ChatGPT search results.

Will blocking Google-Extended hurt my Google rankings or AI Overviews?

Google says Google-Extended does not affect inclusion or ranking in Search. AI Overviews and AI Mode are part of Search, so they are governed by Googlebot and Google's Search controls, not by Google-Extended. Google has said that sites allowing Google-Extended can appear in grounded responses in the Gemini apps, so blocking it may affect that surface.

Do I need to change my robots.txt because of the new AIPREF signal?

Not urgently. Your existing named rules keep working, and Google's support is planned for 2027. Adding a __usage_ai-training group now is a sensible extra layer, as long as you keep the provider specific rules in place.

How do I check which AI bots are visiting my site?

Look at your server or CDN logs and filter by the user agent names in the table above. Many CDNs also show verified bot traffic in their dashboards. Check that the search crawlers are getting normal responses, not challenges or errors from a firewall rule.

Is a robots.txt file enough to stop AI tools from using my content?

No. It is a request that the major providers say they honour for their crawlers, but user-triggered fetchers may not follow it, and it does nothing against scrapers that ignore the rules. Content that must stay private belongs behind a login.

Not sure what your robots.txt, CDN and firewall are telling AI crawlers right now? Talk to our SEO team and we will check it for you.