Start free

Free tool · no sign-up

robots.txt generator for AI crawlers

Choose which AI crawlers may read your site, by what they feed: AI answers, or model training. Copy or download the file. Everything runs in your browser.

Start from
Crawlers that fetch pages for AI answers
Crawlers that collect training data
robots.txt
# AI crawlers, named so access is a decision rather than a default.

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: DuckAssistBot
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /account/
Content-signal: search=yes, ai-input=yes, ai-train=yes

# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /account/
Content-signal: search=yes, ai-input=yes, ai-train=yes

Questions

Should I block AI training but allow AI answers?
It is a common choice: your pages can still be read and cited in ChatGPT, Perplexity and Claude answers, while GPTBot, ClaudeBot and similar crawlers stay out. Google-Extended covers both Gemini training and grounding, so blocking it can also keep you out of Gemini's answers.
Why repeat my private paths for every crawler?
A crawler that finds a group with its own name follows only that group and ignores the User-agent: * group. Private paths listed only under * are open to every crawler you named.
Does robots.txt actually stop crawlers?
It is a request that well-behaved crawlers follow. A CDN or firewall rule decides what a crawler really receives. Check both with our free AI crawler checker.

Published it? Check what each crawler really gets with the AI crawler checker.