2.6%
of the world's top 1,000 sites block Googlebot
2%
of India's top 1,000 sites block Googlebot
From our study of 2,000 sites, 20 October 2026.
Allow or block Googlebot in robots.txt
robots.txt is a plain text file at the top of your site (yoursite.com/robots.txt). Add one of these groups. If Googlebot isn't mentioned at all, it follows your rules for all crawlers.
To allow it
User-agent: Googlebot Allow: /
To block it
User-agent: Googlebot Disallow: /
Not the same as Google-Extended. Google-Extended a control, not a separate crawler. Covers Gemini model training and grounding in the Gemini apps. It does not affect Google Search or AI Overviews. Blocking one doesn't block the other. See Google-Extended.
Questions
What is Googlebot?
Googlebot is run by Google. Crawls for Google Search, including AI Overviews and AI Mode.
Should I block Googlebot?
Keep it allowed. Blocking it removes your pages from that search engine, and from the AI answers built on it.
How do I block Googlebot?
Add "User-agent: Googlebot" followed by "Disallow: /" to the robots.txt file at the top of your site. To allow it, use "Allow: /" instead, or simply don't mention it.
Is Googlebot the same as Google-Extended?
No. They're separate. Blocking one doesn't block the other, so check both lines in your robots.txt.
Other crawlers of this kind: Bingbot. All crawlers.