The short answer
Your robots.txt file tells each crawler which pages it may read. Block the wrong one and your site drops out of Google, or out of ChatGPT and Perplexity answers, without any warning.
A common mistake is a copied rule meant to stop AI training (GPTBot) that also blocks AI search (OAI-SearchBot). They are separate crawlers, so you can say no to one and yes to the other.
What each crawler does
From each company's own documentation, checked on 19 October 2026.
| Crawler | Company | What it does | Our advice |
|---|---|---|---|
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode. | Keep open | |
| Bingbot | Microsoft | Crawls for Bing search, which Microsoft Copilot answers draw on. | Keep open |
| OAI-SearchBot | OpenAI | Finds websites to show in ChatGPT search answers. | Keep open |
| Claude-SearchBot | Anthropic | Indexes pages to improve Claude's search answers. | Keep open |
| PerplexityBot | Perplexity | Finds and links websites in Perplexity answers. Not used for model training. | Keep open |
| ChatGPT-User | OpenAI | Opens a page when a ChatGPT user's question needs it. | Leave open |
| Claude-User | Anthropic | Opens a page when a Claude user's question needs it. | Leave open |
| Perplexity-User | Perplexity | Opens a page when a Perplexity user's question needs it. | Leave open |
| GPTBot | OpenAI | Collects content that may be used to train OpenAI's models. | Your choice |
| ClaudeBot | Anthropic | Collects content that may be used to train Anthropic's models. | Your choice |
| Google-Extended | A control, not a separate crawler. Covers Gemini model training and grounding in the Gemini apps. It does not affect Google Search or AI Overviews. | Your choice | |
| CCBot | Common Crawl | Builds the Common Crawl dataset, which many AI models are trained on. | Your choice |
How to change robots.txt on your platform
WordPress
With Yoast SEO: Yoast SEO, then Tools, then File editor. With Rank Math: Rank Math, then General Settings, then Edit robots.txt. Without a plugin, the file sits in your site's main folder, and your host's file manager can edit it.
Shopify
Online Store, then Themes, then Edit code. Add a new template, choose robots, and put your rules in robots.txt.liquid.
Wix
Open your SEO settings and use the Robots.txt editor. Keep the lines already there and add only what you need.
Any other site
robots.txt must sit at the top of your domain, for example yourbusiness.com/robots.txt. Ask your developer or host to edit it, and send them the lines from your result above.
If you use Cloudflare
Check its bot settings too. Cloudflare can block AI crawlers at the firewall even when robots.txt lets them in, and this tool can't see that.
Questions
Should I block GPTBot?
It depends on whether you mind your content being used to train future AI models. Blocking GPTBot does not take you out of ChatGPT search. That is OAI-SearchBot's job, so keep OAI-SearchBot open if you want ChatGPT to find and cite you.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not control AI features in Search and is not a ranking signal. AI Overviews and AI Mode follow your Googlebot rules, like the rest of Google Search.
Everything is allowed, but AI still doesn't mention my business. Why?
Access is only the first step. Assistants pick pages that answer the question clearly, and brands other websites talk about. Also check whether a firewall such as Cloudflare blocks AI bots even when robots.txt allows them. The full SEARCH ONE audit looks at the page itself, not just robots.txt.
I don't have a robots.txt file. Is that a problem?
No. Without a robots.txt file nothing is blocked. A short file is still worth having, because it can point crawlers to your sitemap.
How long until a change works?
OpenAI and Perplexity both say changes can take about 24 hours to reach their systems. Search engines re-read robots.txt on their own schedule, usually within a day or so.
Does this tool change anything on my website?
No. It reads your public robots.txt file, the same file any crawler reads. Nothing is saved to your site and you don't need to sign in.