Skip to main content
DiwaliFree checklist

AI crawlers · Fetches pages for a user · Anthropic

Claude-User

Claude-User is Anthropic's crawler that fetches pages for a user. Opens a page when a Claude user's question needs it. Usually leave it allowed. It only visits when a person asks the assistant about a page. Some companies say robots.txt may not apply to these visits.

By Praveen Kumar, Vectura Digital · Checked against Anthropic's documentation on 20 October 2026.

9.1%

of the world's top 1,000 sites block Claude-User

3.5%

of India's top 1,000 sites block Claude-User

From our study of 2,000 sites, 20 October 2026.

Allow or block Claude-User in robots.txt

robots.txt is a plain text file at the top of your site (yoursite.com/robots.txt). Add one of these groups. If Claude-User isn't mentioned at all, it follows your rules for all crawlers.

To allow it

User-agent: Claude-User
Allow: /

To block it

User-agent: Claude-User
Disallow: /
Check if your site blocks Claude-UserMake a full robots.txt

Questions

What is Claude-User?

Claude-User is run by Anthropic. Opens a page when a Claude user's question needs it.

Should I block Claude-User?

Usually leave it allowed. It only visits when a person asks the assistant about a page. Some companies say robots.txt may not apply to these visits.

How do I block Claude-User?

Add "User-agent: Claude-User" followed by "Disallow: /" to the robots.txt file at the top of your site. To allow it, use "Allow: /" instead, or simply don't mention it.

Other crawlers of this kind: ChatGPT-User, Perplexity-User. All crawlers.

Source