Free tool
Free AI Robots.txt Generator: Block Training, Keep AI Search
Choose how AI training crawlers, AI search crawlers and user triggered fetchers should treat your site, add your private paths, and copy a robots.txt that gets the group rules right. Everything runs in your browser; nothing you type is sent anywhere.
Default output: opt out of AI training, stay in AI search
# robots.txt made with the free NezT Digital AI robots.txt generator # https://neztdigital.com/tools/ai-robots-txt-generator # Bot names checked against provider documentation, November 2026. Review before publishing. # Everyone else, including Googlebot and Bingbot (Google Search, AI Overviews, AI Mode, Bing) User-agent: * Disallow: /admin/ Disallow: /account/ # AI model training: opted out User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: CCBot Disallow: / # AI search and answer engines: welcome to read and cite (private paths repeated, because a named group ignores the * group) User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Disallow: /admin/ Disallow: /account/ # Fetchers that run when a person asks for a page (private paths repeated, because a named group ignores the * group) User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User Disallow: /admin/ Disallow: /account/ # Shared AI training opt-out (IETF AIPREF draft). Anthropic has begun recognising it; Google plans support in 2027. User-agent: __usage_ai-training Disallow: / Sitemap: https://www.example.com/sitemap.xml
How to use it
- Pick a setting for each crawler type. The default opts out of AI training and keeps AI search open.
- List the private paths you never want crawled, one per line (for example /admin/ or /checkout/).
- Add your sitemap URL, then copy or download the file.
- Upload it as robots.txt at the root of your domain (https://yourdomain.com/robots.txt) and check it again the next day.
Which AI crawler does what?
Checked against provider documentation, November 2026.
| User agent | Company | Type | What it does |
|---|---|---|---|
| GPTBot | OpenAI | AI model training | Training crawler for OpenAI models. |
| ClaudeBot | Anthropic | AI model training | Training crawler for Claude models. |
| Google-Extended | AI model training | A robots.txt token, not a separate crawler. Controls Gemini training and grounding in the Gemini apps; Google says it does not affect Search. | |
| Applebot-Extended | Apple | AI model training | A robots.txt token that lets you opt out of Apple using your pages to train its foundation models. |
| Meta-ExternalAgent | Meta | AI model training | Meta crawler used for purposes that include training AI models. |
| CCBot | Common Crawl | AI model training | Builds the open Common Crawl archive, which is widely used as AI training data. |
| OAI-SearchBot | OpenAI | AI search and answer engines | Surfaces sites in ChatGPT search results. |
| Claude-SearchBot | Anthropic | AI search and answer engines | Improves search results inside Claude. |
| PerplexityBot | Perplexity | AI search and answer engines | Indexes sites for Perplexity answers; Perplexity says it is not used to train foundation models. |
| ChatGPT-User | OpenAI | User triggered fetchers | Fetches a page when a ChatGPT user asks. OpenAI says robots.txt rules may not apply. |
| Claude-User | Anthropic | User triggered fetchers | Fetches a page when a Claude user asks. |
| Perplexity-User | Perplexity | User triggered fetchers | Fetches a page when a Perplexity user asks; Perplexity says it generally ignores robots.txt. |
Read the full guide: Robots.txt for AI crawlers: block training, keep visibility.
FAQ
Will opting out of AI training remove my site from ChatGPT, Claude or Perplexity answers?
No, as long as you keep the search crawlers allowed. OpenAI, Anthropic and Perplexity each run separate crawlers for search, and OpenAI and Perplexity state that each setting works independently. This generator keeps those crawlers in their own group.
Why does the file repeat my private paths in every group?
A crawler follows only the most specific group that names it and ignores the rules under User-agent: *. Google's robots.txt specification describes this. If you name OAI-SearchBot and only write Allow: /, it can crawl your admin area, so the private paths are repeated in each group.
Should I block Googlebot to stay out of AI Overviews?
No. Googlebot powers Google Search, AI Overviews and AI Mode together, so blocking it removes you from Google entirely. Google points to noindex, nosnippet, data-nosnippet and a Search generative AI control in Search Console as the levers for its AI features.
What is the __usage_ai-training line?
It is a shared training opt-out based on IETF AIPREF work, announced in late October 2026. Anthropic says it has begun recognising it and Google plans support in 2027. It is still a draft, so the generator adds it alongside the named rules, never instead of them.
Is my data sent anywhere when I use this tool?
No. The tool runs entirely in your browser. Nothing you type is stored or sent to NezT Digital or anyone else.