Skip to main content
DiwaliFree checklist

AI crawlers · AI training crawler · Common Crawl

CCBot

CCBot is Common Crawl's AI training crawler. Builds the Common Crawl dataset, which many AI models are trained on. Your choice. Blocking it keeps your future content out of that company's AI training. It doesn't remove you from AI search answers, which use separate crawlers.

By Praveen Kumar, Vectura Digital · Checked against Common Crawl's documentation on 20 October 2026.

14.4%

of the world's top 1,000 sites block CCBot

5.6%

of India's top 1,000 sites block CCBot

From our study of 2,000 sites, 20 October 2026.

Allow or block CCBot in robots.txt

robots.txt is a plain text file at the top of your site (yoursite.com/robots.txt). Add one of these groups. If CCBot isn't mentioned at all, it follows your rules for all crawlers.

To allow it

User-agent: CCBot
Allow: /

To block it

User-agent: CCBot
Disallow: /
Check if your site blocks CCBotMake a full robots.txt

Questions

What is CCBot?

CCBot is run by Common Crawl. Builds the Common Crawl dataset, which many AI models are trained on.

Should I block CCBot?

Your choice. Blocking it keeps your future content out of that company's AI training. It doesn't remove you from AI search answers, which use separate crawlers.

How do I block CCBot?

Add "User-agent: CCBot" followed by "Disallow: /" to the robots.txt file at the top of your site. To allow it, use "Allow: /" instead, or simply don't mention it.

Other crawlers of this kind: GPTBot, ClaudeBot, Google-Extended. All crawlers.

Source