Skip to main content
DiwaliFree checklist

AI crawlers · AI search crawler · Anthropic

Claude-SearchBot

Claude-SearchBot is Anthropic's AI search crawler. Indexes pages to improve Claude's search answers. Allow it if you want to be found and cited in AI answers. Blocking it keeps your pages out of that assistant's search answers. It doesn't change your Google rankings.

By Praveen Kumar, Vectura Digital · Checked against Anthropic's documentation on 20 October 2026.

8.6%

of the world's top 1,000 sites block Claude-SearchBot

3.3%

of India's top 1,000 sites block Claude-SearchBot

From our study of 2,000 sites, 20 October 2026.

Allow or block Claude-SearchBot in robots.txt

robots.txt is a plain text file at the top of your site (yoursite.com/robots.txt). Add one of these groups. If Claude-SearchBot isn't mentioned at all, it follows your rules for all crawlers.

To allow it

User-agent: Claude-SearchBot
Allow: /

To block it

User-agent: Claude-SearchBot
Disallow: /

Not the same as ClaudeBot. ClaudeBot collects content that may be used to train Anthropic's models. Blocking one doesn't block the other. See ClaudeBot.

Check if your site blocks Claude-SearchBotMake a full robots.txt

Questions

What is Claude-SearchBot?

Claude-SearchBot is run by Anthropic. Indexes pages to improve Claude's search answers.

Should I block Claude-SearchBot?

Allow it if you want to be found and cited in AI answers. Blocking it keeps your pages out of that assistant's search answers. It doesn't change your Google rankings.

How do I block Claude-SearchBot?

Add "User-agent: Claude-SearchBot" followed by "Disallow: /" to the robots.txt file at the top of your site. To allow it, use "Allow: /" instead, or simply don't mention it.

Is Claude-SearchBot the same as ClaudeBot?

No. They're separate. Blocking one doesn't block the other, so check both lines in your robots.txt.

Other crawlers of this kind: OAI-SearchBot, PerplexityBot. All crawlers.

Source