Skip to main content
DiwaliFree checklist

Free tool from SEARCH ONE

Make a robots.txt for the AI era

Choose who may read your site. Stay findable in Google, Bing, ChatGPT search and Perplexity, and keep your content out of AI training if you want to.

1. Start with a preset

2. Adjust any crawler

Search engines
AI search and answers
Visits a person asked for
AI model training

3. Folders to keep private (optional)

Your robots.txt

# robots.txt made with the Vectura Digital AI robots.txt generator (https://www.vecturadigital.com/tools/ai-robots-txt-generator)

# Crawlers asked not to read this site
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

# Every other crawler, including search engines and AI search
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout

Upload it so it opens at yoursite.com/robots.txt, then check it with the AI crawler checker.

Click-by-click steps: WordPress · Shopify · Wix · Squarespace

Why this matters

In our study of 2,000 top websites, 61% of the world's sites that block AI training crawlers also block AI search crawlers, which keeps them out of answers in ChatGPT search, Perplexity and Claude. Many likely copied one list of "AI bots to block". This generator keeps training and search separate.

Questions

Will blocking GPTBot remove my site from ChatGPT?

Not from ChatGPT search. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. This generator keeps them separate so you can block one and allow the other.

Does robots.txt stop all bots?

No. It's a request that well-behaved crawlers follow, and the major search and AI companies say theirs do. It doesn't stop bad bots, and it doesn't hide pages from people.

Where do I put the file?

At the top of your site, so it opens at yoursite.com/robots.txt. On WordPress, Shopify and Wix you edit it from the dashboard; the platform guides below show where. Squarespace has a single AI crawler setting instead.

Should I block AI training?

It's your choice. Blocking training crawlers keeps future content out of those companies' training data. It doesn't change your Google rankings or remove you from AI search answers.

What does the preset I picked do?

Each preset only changes which crawlers are blocked. Your own disallowed folders and sitemap are added for every crawler.

Read next

Share this tool on your own site or blog

Paste this into any page. It's a plain link in a small box, with no script and no tracking, so it can't slow your page down.

<p style="border:1px solid #d4d8e0;border-radius:12px;padding:14px 16px;font-family:system-ui,sans-serif;line-height:1.5">
  <strong>Free AI robots.txt generator</strong><br>
  <a href="https://vecturadigital.com/tools/ai-robots-txt-generator" target="_blank" rel="noopener">Check your website with the free AI robots.txt generator by SEARCH ONE</a>
</p>