Free tool

Free AI Robots.txt Generator: Block Training, Keep AI Search

Choose how AI training crawlers, AI search crawlers and user triggered fetchers should treat your site, add your private paths, and copy a robots.txt that gets the group rules right. Everything runs in your browser; nothing you type is sent anywhere.

Default output: opt out of AI training, stay in AI search

# robots.txt made with the free NezT Digital AI robots.txt generator
# https://neztdigital.com/tools/ai-robots-txt-generator
# Bot names checked against provider documentation, November 2026. Review before publishing.

# Everyone else, including Googlebot and Bingbot (Google Search, AI Overviews, AI Mode, Bing)
User-agent: *
Disallow: /admin/
Disallow: /account/

# AI model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /

# AI search and answer engines: welcome to read and cite (private paths repeated, because a named group ignores the * group)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/
Disallow: /account/

# Fetchers that run when a person asks for a page (private paths repeated, because a named group ignores the * group)
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /admin/
Disallow: /account/

# Shared AI training opt-out (IETF AIPREF draft). Anthropic has begun recognising it; Google plans support in 2027.
User-agent: __usage_ai-training
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

How to use it

  1. Pick a setting for each crawler type. The default opts out of AI training and keeps AI search open.
  2. List the private paths you never want crawled, one per line (for example /admin/ or /checkout/).
  3. Add your sitemap URL, then copy or download the file.
  4. Upload it as robots.txt at the root of your domain (https://yourdomain.com/robots.txt) and check it again the next day.

Which AI crawler does what?

Checked against provider documentation, November 2026.

User agentCompanyTypeWhat it does
GPTBotOpenAIAI model trainingTraining crawler for OpenAI models.
ClaudeBotAnthropicAI model trainingTraining crawler for Claude models.
Google-ExtendedGoogleAI model trainingA robots.txt token, not a separate crawler. Controls Gemini training and grounding in the Gemini apps; Google says it does not affect Search.
Applebot-ExtendedAppleAI model trainingA robots.txt token that lets you opt out of Apple using your pages to train its foundation models.
Meta-ExternalAgentMetaAI model trainingMeta crawler used for purposes that include training AI models.
CCBotCommon CrawlAI model trainingBuilds the open Common Crawl archive, which is widely used as AI training data.
OAI-SearchBotOpenAIAI search and answer enginesSurfaces sites in ChatGPT search results.
Claude-SearchBotAnthropicAI search and answer enginesImproves search results inside Claude.
PerplexityBotPerplexityAI search and answer enginesIndexes sites for Perplexity answers; Perplexity says it is not used to train foundation models.
ChatGPT-UserOpenAIUser triggered fetchersFetches a page when a ChatGPT user asks. OpenAI says robots.txt rules may not apply.
Claude-UserAnthropicUser triggered fetchersFetches a page when a Claude user asks.
Perplexity-UserPerplexityUser triggered fetchersFetches a page when a Perplexity user asks; Perplexity says it generally ignores robots.txt.

Read the full guide: Robots.txt for AI crawlers: block training, keep visibility.

FAQ

Will opting out of AI training remove my site from ChatGPT, Claude or Perplexity answers?

No, as long as you keep the search crawlers allowed. OpenAI, Anthropic and Perplexity each run separate crawlers for search, and OpenAI and Perplexity state that each setting works independently. This generator keeps those crawlers in their own group.

Why does the file repeat my private paths in every group?

A crawler follows only the most specific group that names it and ignores the rules under User-agent: *. Google's robots.txt specification describes this. If you name OAI-SearchBot and only write Allow: /, it can crawl your admin area, so the private paths are repeated in each group.

Should I block Googlebot to stay out of AI Overviews?

No. Googlebot powers Google Search, AI Overviews and AI Mode together, so blocking it removes you from Google entirely. Google points to noindex, nosnippet, data-nosnippet and a Search generative AI control in Search Console as the levers for its AI features.

What is the __usage_ai-training line?

It is a shared training opt-out based on IETF AIPREF work, announced in late October 2026. Anthropic says it has begun recognising it and Google plans support in 2027. It is still a draft, so the generator adds it alongside the named rules, never instead of them.

Is my data sent anywhere when I use this tool?

No. The tool runs entirely in your browser. Nothing you type is stored or sent to NezT Digital or anyone else.

Sources

Talk to our SEO team