AI crawler directory · Checked 20 October 2026
Every AI crawler that reads your website
Who runs it, what it does, whether you should let it in, and the robots.txt lines to allow or block it. With the share of the world's top sites that block each one, from our own study.
Search engines
Keep these open. Blocking them takes the site out of Google or Bing, including Google's AI Overviews and AI Mode.
AI search and answers
Keep these open if you want ChatGPT, Claude and Perplexity to find and cite your pages in their answers.
| Crawler | Run by | What it does | Top sites blocking it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds websites to show in ChatGPT search answers. | 8.1% world · 2.8% India |
| Claude-SearchBot | Anthropic | Indexes pages to improve Claude's search answers. | 8.6% world · 3.3% India |
| PerplexityBot | Perplexity | Finds and links websites in Perplexity answers. Not used for model training. | 10.7% world · 3.4% India |
Visits a person asked for
These fetch a page when someone asks an assistant about it. OpenAI and Perplexity say robots.txt may not apply to their user visits, so a block there may not work.
| Crawler | Run by | What it does | Top sites blocking it |
|---|---|---|---|
| ChatGPT-User | OpenAI | Opens a page when a ChatGPT user's question needs it. | 9.1% world · 3.5% India |
| Claude-User | Anthropic | Opens a page when a Claude user's question needs it. | 9.1% world · 3.5% India |
| Perplexity-User | Perplexity | Opens a page when a Perplexity user's question needs it. | 9.9% world · 3.7% India |
AI model training
Your choice. Blocking these keeps your future content out of AI training data. It does not remove you from AI search answers.
| Crawler | Run by | What it does | Top sites blocking it |
|---|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train OpenAI's models. | 13.5% world · 5.6% India |
| ClaudeBot | Anthropic | Collects content that may be used to train Anthropic's models. | 13.1% world · 5.5% India |
| Google-Extended | A control, not a separate crawler. Covers Gemini model training and grounding in the Gemini apps. It does not affect Google Search or AI Overviews. | 12.3% world · 5% India | |
| CCBot | Common Crawl | Builds the Common Crawl dataset, which many AI models are trained on. | 14.4% world · 5.6% India |
Block shares: share of the world's top 1,000 and India's top 1,000 websites whose robots.txt blocks the crawler from the home page, read on 20 October 2026. See the full study and data.