Robots.txt AI Policy
Which AI crawlers a site allows or blocks (robots.txt).
Enter a value above to run Robots.txt AI Policy
e.g. example.com
Quick answer
Robots.txt AI Policy reads a site's robots.txt and lists which AI crawlers (such as GPTBot, ClaudeBot, and Google-Extended) are allowed or blocked. "Allow" means the bot may crawl the site; "Disallow" means it is asked to stay out.
This tool fetches a site's /robots.txt and reports which AI crawlers it allows or blocks. It looks for the specific user-agent tokens that AI companies publish for training and answer-engine bots, such as GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, and Bytespider, then shows the Allow or Disallow rules that apply to each one.
Site owners use it to confirm that an opt-out they intended is actually in the file, and to audit competitors or their own multiple domains without opening each robots.txt by hand.
How it works
The tool requests https://yourdomain/robots.txt and parses the plain-text rules. robots.txt is organized into groups, each starting with one or more User-agent lines followed by Allow and Disallow paths. For every known AI bot token, the tool finds the most specific matching group and evaluates whether the site root is permitted. If no group names the bot, it falls back to the wildcard User-agent: * group, exactly as a real crawler would.
How to read the results
Each AI bot is listed as allowed or blocked. "Blocked" means a group for that agent contains Disallow: / , so the crawler is asked to skip the whole site. "Allowed" means no such rule applies. Remember robots.txt is voluntary: it signals intent but does not enforce anything. Google-Extended and Applebot-Extended only govern AI training and answer features, not normal search indexing, so blocking them does not remove you from search results.
Common problems & fixes
A frequent mistake is blocking a bot in a specific group while a broad User-agent: * group above it looks stricter; crawlers use only the single best-matching group, not both, so rules do not stack. Also check the token spelling exactly, GPTBot and ClaudeBot are case-insensitive but must match. If robots.txt returns a 404 or HTML error page, no directives exist and every crawler treats the site as fully allowed.
Frequently asked questions
- Does blocking AI bots in robots.txt actually stop them?
- It stops well-behaved crawlers that read and honor robots.txt, including GPTBot, ClaudeBot, and Google-Extended. It is a request, not a firewall. Bots that ignore the standard, or content that is scraped through third parties and datasets, are not stopped. For hard enforcement you need server-side blocking by user-agent or IP.
- What is the difference between GPTBot and Google-Extended?
- GPTBot is a distinct crawler that fetches pages for OpenAI. Google-Extended is not a crawler at all; it is a control token that tells Google whether your already-indexed content may be used for Gemini and AI training. Disallowing Google-Extended does not affect Googlebot or your normal search ranking in any way.
- How do I block a specific AI crawler after checking?
- Add a group to robots.txt at your site root, for example: User-agent: GPTBot then a new line with Disallow: / . Repeat with separate groups for each bot such as ClaudeBot, CCBot, and PerplexityBot. Deploy the file, then re-run this tool to confirm the rules are live and parsed correctly.