Skip to main content
DiwaliFree checklist

Free tool from SEARCH ONE

Do AI bots actually visit your site?

Analytics can't see crawlers. Your server's access log can. Open it here to see which AI crawlers came, what they read and when they were last there.

Why check the log at all

A robots.txt file says who may visit. It doesn't tell you who actually came. The log does: if OAI-SearchBot or PerplexityBot have never read your services page, ChatGPT search and Perplexity have little to quote from it. In our study of top websites, many sites block AI search crawlers without meaning to; the log shows the effect of that choice page by page.

Questions

Why can't Google Analytics show AI bots?

Analytics tools count visits by running a script in the browser. Crawlers read the page's code without running that script, so they never appear in analytics. Your server's access log records every request, bots included.

Where do I get my access log?

Most hosts let you download it from the control panel. In cPanel it's under Metrics, then Raw Access. Other hosts call it access logs or raw logs. If you can't find it, ask your host's support for the raw access log for your domain. Managed platforms such as Wix, Squarespace and Shopify don't give out these logs.

Is my file uploaded anywhere?

No. The file is read by your own browser and the results are worked out on your computer. Nothing is sent to us.

Can a bot pretend to be GPTBot?

Yes. Any scraper can copy a crawler's name. This checker goes by the name each request gives. To be certain, the big companies publish the IP addresses their crawlers use, which your developer can compare against the log.

None of the AI search bots came. What should I do?

First check they aren't blocked, with our free AI crawler checker. Then make sure your pages are in your sitemap and linked from your home page. New or small sites can take weeks to be visited.

Read next

Share this tool on your own site or blog

Paste this into any page. It's a plain link in a small box, with no script and no tracking, so it can't slow your page down.

<p style="border:1px solid #d4d8e0;border-radius:12px;padding:14px 16px;font-family:system-ui,sans-serif;line-height:1.5">
  <strong>Free AI bot log checker</strong><br>
  <a href="https://vecturadigital.com/tools/ai-bot-log-checker" target="_blank" rel="noopener">Check your website with the free AI bot log checker by SEARCH ONE</a>
</p>

Keep going