AI crawler checker: GPTBot, ClaudeBot, PerplexityBot and more
An AI crawler checker tells you whether AI tools can reach your website. It reads your robots.txt for rules that allow or block AI user agents such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. A good checker also tests your firewall, because a host can refuse AI bots even when robots.txt allows them. Silkpath does both in every crawl.
The AI crawlers Silkpath checks
| User agent | Company | What it is used for |
|---|---|---|
GPTBot | OpenAI | Collects content that may be used to train OpenAI models |
OAI-SearchBot | OpenAI | Finds pages to show and link in ChatGPT search |
ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks for it |
ClaudeBot | Anthropic | Collects content for Anthropic's Claude models |
PerplexityBot | Perplexity | Indexes pages to show and cite in Perplexity answers |
Google-Extended | A robots.txt token that controls use of content for Gemini; not used for Search ranking | |
Applebot-Extended | Apple | A robots.txt token that controls use of content for Apple's AI models |
CCBot | Common Crawl | An open web archive that many AI models are trained on |
Bytespider | ByteDance | Collects content for ByteDance's AI models |
Meta-ExternalAgent | Meta | Collects content for Meta's AI models |
Companies add new agents from time to time (Anthropic, for example, also uses Claude-SearchBot and Claude-User), so check each company's own documentation before writing strict rules.
Example robots.txt rules
Allow AI search tools, block model training
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /Welcome every crawler
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlWatch the order of the groups. A crawler follows the most specific group that names it. If a group for GPTBot exists, GPTBot ignores the rules under User-agent: *.
Why the firewall test matters
Many hosts and CDNs have bot protection that refuses unknown or AI user agents with a 403 error or a challenge page. Your robots.txt can say "welcome" while the firewall says "no". Silkpath requests the site with an AI user agent and reports the answer it gets, so you see what the AI tools see.
How to check your site with Silkpath
- Open Silkpath and crawl your domain.
- In the report, open AI Search, then GEO.
- Read the AI crawlers in robots.txt check: it says "All allowed" or names every AI agent that robots.txt blocks. The Firewall test below it says whether the server refuses AI user agents.
- Check the same section for llms.txt, entity schema and content that needs JavaScript.
Questions and answers
Should I block AI crawlers?
It depends on your goals. Blocking training crawlers such as GPTBot keeps content out of future training, while allowing search agents such as OAI-SearchBot and PerplexityBot lets those tools show and cite your pages. Many sites allow search agents and decide case by case about training.
Does robots.txt really stop AI crawlers?
The major AI companies say their crawlers respect robots.txt. It is a request, not a lock: a firewall rule is the only way to enforce a block.
Will blocking GPTBot remove my site from ChatGPT search?
Not by itself. ChatGPT search uses OAI-SearchBot, so a site can block GPTBot and still allow OAI-SearchBot.
Audit a website with Silkpath
Enter a website address and Silkpath crawls every page, then reports SEO, AEO and GEO problems page by page, with PDF, Excel and CSV exports.
Read next
llms.txt Checker
What llms.txt is, an example file, and how Silkpath checks it.
What is GEO
Generative engine optimization: how to be read and cited by ChatGPT, Perplexity and AI Overviews.
Features
Every check Silkpath runs, section by section, and what you get out of a crawl.