robots.txt AI bot checker: GPTBot, ClaudeBot, PerplexityBot

Check a live site

Or paste the file

Answer bots and training bots are different

Answer bots fetch a page because a person asked an assistant about it, or to build that assistant's search results. Block them and ChatGPT, Claude or Perplexity cannot quote or link your page:

• OAI-SearchBot: OpenAI, ChatGPT search results
• ChatGPT-User: OpenAI, opens a page a ChatGPT user asked about
• Claude-SearchBot: Anthropic, Claude search results
• Claude-User: Anthropic, opens a page a Claude user asked about
• PerplexityBot: Perplexity, search index
• Perplexity-User: Perplexity, opens a page a user asked about

Training bots collect pages to train models. Blocking them does not remove you from AI answers, and it is a fair choice either way, so this checker reports them but never counts them as a problem:

• GPTBot: OpenAI, model training
• ClaudeBot: Anthropic, model training
• Google-Extended: Google, Gemini training (not Google Search)
• CCBot: Common Crawl, open dataset used for training
• Applebot-Extended: Apple, AI training (not Siri/Spotlight search)
• Meta-ExternalAgent: Meta, AI training

A common mistake: one User-agent: GPTBot block that was meant to stop training, copied together with ChatGPT-User and OAI-SearchBot, which hides the site from ChatGPT answers too.

How the matching works (RFC 9309)

1. A bot uses the group whose User-agent line names it. Only if none does, it uses the User-agent: * group. It never combines the two, so a named group with only Allow: / overrides everything under *.
2. Inside that group the longest matching Allow/Disallow path wins; on a tie, Allow wins.
3. * matches any characters and $ ends the path. An empty Disallow: allows everything.
4. No robots.txt (or an HTML page at /robots.txt) means nothing is blocked.

Set Page path to test to check a deeper page, such as /pricing/ or /blog/post.

Example: answer bots in, training bots out

The Try the example button checks this file:

# Let AI answer bots read the site, keep training bots out of everything
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Disallow: /account/
Allow: /

Sitemap: https://example.com/sitemap.xml

robots.txt is one part

Bots also meet firewalls and captchas that robots.txt does not show. The free Agent-Ready Check fetches your pages as an agent and reports bot walls, captchas on sign-up and contact pages, llms.txt, text that only appears after JavaScript, and JSON-LD. See also the llms.txt checker.