13.1%
of the world's top 1,000 sites block ClaudeBot
5.5%
of India's top 1,000 sites block ClaudeBot
From our study of 2,000 sites, 20 October 2026.
Allow or block ClaudeBot in robots.txt
robots.txt is a plain text file at the top of your site (yoursite.com/robots.txt). Add one of these groups. If ClaudeBot isn't mentioned at all, it follows your rules for all crawlers.
To allow it
User-agent: ClaudeBot Allow: /
To block it
User-agent: ClaudeBot Disallow: /
Not the same as Claude-SearchBot. Claude-SearchBot indexes pages to improve Claude's search answers. Blocking one doesn't block the other. See Claude-SearchBot.
Questions
What is ClaudeBot?
ClaudeBot is run by Anthropic. Collects content that may be used to train Anthropic's models.
Should I block ClaudeBot?
Your choice. Blocking it keeps your future content out of that company's AI training. It doesn't remove you from AI search answers, which use separate crawlers.
How do I block ClaudeBot?
Add "User-agent: ClaudeBot" followed by "Disallow: /" to the robots.txt file at the top of your site. To allow it, use "Allow: /" instead, or simply don't mention it.
Is ClaudeBot the same as Claude-SearchBot?
No. They're separate. Blocking one doesn't block the other, so check both lines in your robots.txt.
Other crawlers of this kind: GPTBot, Google-Extended, CCBot. All crawlers.