Skip to main content
DiwaliFree checklist

Research · Data collected 20 October 2026

Who blocks AI search?

AI crawler access on the world's top 1,000 websites, compared with India's top 1,000.

11% of the world's top 1,000 websites block at least one AI search crawler, rising to 17.6% of the top 100. And 60.9% of sites that block AI training crawlers also block the crawlers that put sites into ChatGPT search, Perplexity and Claude answers. India's top 1,000 sites are far more open: 3.7% block AI search.

By Praveen Kumar, Vectura Digital. We read each site's public robots.txt file twice, on 20 October 2026. Nothing is estimated. How we did it.

11%

of the world's top 1,000 sites block an AI search crawler

60.9%

of sites that block AI training also block AI search

3.7%

of India's top 1,000 sites block AI search, three times fewer

Finding 1

1 in 9 of the world's top sites block AI search; in India it's 1 in 27

Across the world's top 1,000 sites, 11% block at least one of the three main AI search crawlers (OpenAI's OAI-SearchBot, PerplexityBot and Anthropic's Claude-SearchBot). Among India's top 1,000 .in sites it's 3.7%. The very biggest sites are the most restrictive: 17.6% of the world's top 100 block one.

  • Block at least one AI search crawler

    India
    3.7%
    World
    11%

  • Block at least one AI training crawler

    India
    7.6%
    World
    17.9%

  • Block Googlebot from the home page

    Mostly infrastructure and government domains that aren't meant to be searched.

    India
    2%
    World
    2.6%

Bars use one scale from 0 to 20%. Share of sites that gave a clear answer: 762 in India, 717 worldwide.

Finding 2

Blocking AI training often blocks AI search too

Sites have good reasons to stop their content being used to train AI models. But training and search use different crawlers. OpenAI, for example, says GPTBot collects content that may be used for training, while OAI-SearchBot finds sites to show in ChatGPT search.

Of the sites that block at least one training crawler, 48.3% in India and 60.9% worldwide also block at least one AI search crawler. For a business that wants customers to find it in AI answers, that's worth checking. Some do it on purpose. Others may simply have copied a list of “AI bots to block”.

Two contrasting examples from the data: Amazon India blocks both GPTBot and OAI-SearchBot. Tripadvisor India blocks GPTBot and other training crawlers, but lets OAI-SearchBot, ChatGPT-User and PerplexityBot in.

Finding 3

Crawler by crawler

Share of sites whose robots.txt asks each crawler not to read the home page. Bars use one scale from 0 to 20%.

  • Googlebot (Google)

    Search

    India
    2%
    World
    2.6%

  • Bingbot (Microsoft)

    Search

    India
    2.4%
    World
    2.8%

  • OAI-SearchBot (OpenAI)

    AI search

    India
    2.8%
    World
    8.1%

  • PerplexityBot (Perplexity)

    AI search

    India
    3.4%
    World
    10.7%

  • Claude-SearchBot (Anthropic)

    AI search

    India
    3.3%
    World
    8.6%

  • ChatGPT-User (OpenAI)

    Fetch for a user

    India
    3.5%
    World
    9.1%

  • GPTBot (OpenAI)

    Training

    India
    5.6%
    World
    13.5%

  • ClaudeBot (Anthropic)

    Training

    India
    5.5%
    World
    13.1%

  • Google-Extended (Google)

    Training control

    India
    5%
    World
    12.3%

  • CCBot (Common Crawl)

    Training

    India
    5.6%
    World
    14.4%

  • Bytespider (ByteDance)

    Training

    India
    6.3%
    World
    15.5%

  • meta-externalagent (Meta)

    Training

    India
    4.9%
    World
    12.3%

Finding 4

Almost half of India's top sites have no crawler rules at all

48% of India's top sites either have no robots.txt file or return a web page instead of one, against 30.5% worldwide. That isn't a problem in itself: with no rules, crawlers may read everything. It does mean those sites haven't made a choice about AI crawlers either way.

Finding 5

Few sites use llms.txt or describe their business in code

  • Have a valid llms.txt file

    55 Indian sites and 99 worldwide. llms.txt is optional; Google says its search doesn't need it.

    India
    5.5%
    World
    9.9%

  • Home page has Organization or business structured data

    Of home pages we could read: 503 in India, 472 worldwide.

    India
    17.9%
    World
    21.4%

  • Home page has a meta description

    India
    59.6%
    World
    65.7%

What it means for a small business

  • Check your own robots.txt. If you block training crawlers, make sure you haven't blocked OAI-SearchBot, PerplexityBot or Claude-SearchBot by accident.
  • Decide on purpose. No rules means everything is allowed. That's fine for most small businesses, but it should be a choice.
  • Add the basics that few sites have yet: business details in structured data, and a clear description on your home page.
Check your site's crawler access

How we did it

  • Which sites. The top 1,000 domains ending in .in and the top 1,000 domains overall in the Tranco research ranking, list GQG5K, downloaded on 20 October 2026. No sites were added or removed by hand.
  • What we read. Each site's robots.txt, fetched twice with two different methods a few seconds apart; its llms.txt; and its home page, only where robots.txt allowed us. Our crawler identified itself as VecturaResearchBot with a link to this page.
  • How we decided “blocked”. We parsed robots.txt with Protego, the parser used by Scrapy, and asked whether each crawler may read the home page. A missing file (404) or a web page in its place counts as no rules, as Google treats them.
  • What we left out. Sites that didn't respond, returned errors, or gave different rules to our two requests (238 in India, 283 worldwide). Large sites often protect themselves from unknown crawlers, so the very biggest sites are slightly under-represented.
  • Limits. robots.txt is a request, not a lock. A site can also block crawlers at its firewall, which this study doesn't measure. Rankings measure popularity, not quality, so the lists include government, infrastructure and some low-quality domains. Results are for one day.

Use the data

The full site-by-site data is free to use under CC BY 4.0. In the file, 1 means the crawler is allowed, 0 means blocked, and blank means we couldn't tell.

Download the data (CSV)Summary (JSON)

How to cite this study

Vectura Digital (2026). Who blocks AI search? AI crawler access on the world's top 1,000 websites and India's top 1,000. Data collected 20 October 2026. https://www.vecturadigital.com/research/ai-crawler-access-2026

Journalists and researchers: write to info@vecturadigital.com for questions or a comment from Praveen Kumar.

Questions

Which websites were checked?

The top 1,000 domains ending in .in, and the top 1,000 domains overall, from the Tranco research ranking (list GQG5K). We didn't hand-pick or remove any, so the lists include government, infrastructure and some low-quality domains.

What does "blocks" mean here?

The site's robots.txt tells that crawler not to read its home page. It's a request, not a technical barrier: well-behaved crawlers follow it, and the major AI companies say theirs do.

Why are some sites missing from the percentages?

Some sites didn't answer our request, returned an error, or gave different rules to two different requests made seconds apart. We left those out rather than guess. 762 Indian and 717 worldwide sites gave a clear answer.

Does blocking GPTBot remove a site from ChatGPT?

Not from ChatGPT search. OpenAI says GPTBot collects content that may be used for training, while OAI-SearchBot finds sites to show in ChatGPT search. A site can block one and allow the other.

Can I use these figures?

Yes. Quote them with a link to this page. The full site-by-site data is free to download below.

Sources