The technical SEO checklist for AI search: what matters, and what is hype

Google, ChatGPT, Gemini, Perplexity and Copilot all work the same way at the start: a crawler fetches your page, a system reads the text, and an index stores it. Only then can an AI answer use it.

Google's documentation for AI Overviews and AI Mode says it directly: to be shown as a supporting link, a page must be indexed and eligible to appear with a snippet, and there are "no additional technical requirements". That cuts both ways. There is nothing magic to add, and there is no way around a page that cannot be crawled or indexed.

Work through the checklist below in order. Each item is something you or your developer can check in an afternoon.

Open yourdomain.com/robots.txt in a browser. This small file tells crawlers what they may visit. Problems here are common: a staging rule that was never removed, a security plugin that blocks "bots", or a copied template that blocks AI crawlers wholesale.

There are now two kinds of AI crawler, and the difference matters:

• Search and answer crawlers fetch pages so an assistant can cite them: OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User, Claude-SearchBot and Claude-User (Anthropic), plus Googlebot and Bingbot.
• Training crawlers collect content to train future models: GPTBot (OpenAI), ClaudeBot (Anthropic), and the Google-Extended control.

OpenAI states that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers", while GPTBot is used for training. You can block training and still appear in answers.

Training crawlers also take far more than they give back. Cloudflare compared how many pages each AI company's crawlers fetched with how many visitors those companies sent back to websites in August 2025. Turned into visitors per 10,000 pages crawled, the gap is stark:

Our recommendation for most businesses: allow search and answer crawlers, keep Googlebot and Bingbot allowed, and make a deliberate business choice about training crawlers.

Search engines and AI tools get confused when the same page appears at several addresses. Check:

• One version of your domain. yourdomain.com and www.yourdomain.com should not both show the site. One should permanently redirect (a 301) to the other.
• HTTPS everywhere, with http addresses redirecting.
• A canonical tag on each page pointing to its preferred address.
• No accidental noindex on pages you want found. It is a common leftover from a website launch.
• An XML sitemap listing your real pages, linked from robots.txt and submitted in Google Search Console and Bing Webmaster Tools.

In Search Console, the Pages report shows which pages are indexed and why others are not. Start with your home page and your top three service pages.

AI systems quote text. If your prices are inside an image, your menu is a PDF scan, or your service descriptions only appear after a script runs, many crawlers will miss them.

Quick checks:

1. 1Right-click your page and choose View page source. Search for your main heading and your phone number. If they are not in the source, they may not be visible to every crawler.
2. 2Make sure headings follow the content: one main heading, then a heading for each topic or question.
3. 3Give every meaningful image alt text that describes it ("Bridal kundan necklace set, 22 karat"), not "image1".
4. 4Link to your important pages from your menu and from related pages. Pages nobody links to are hard to discover.

Google lists exactly these basics in its guidance for AI features: allow crawling, use internal links, and keep important content in textual form.

Speed does not decide whether an assistant cites you, but it decides whether the person who clicks stays. Google measures real-world experience with three Core Web Vitals, judged on the 75th percentile of visits:

• Largest Contentful Paint (LCP): the main content should load within 2.5 seconds.
• Interaction to Next Paint (INP): the page should respond to taps within 200 milliseconds.
• Cumulative Layout Shift (CLS): the layout should not jump around; aim for 0.1 or less.

Most websites still do not pass all three on mobile.

The usual fixes for small business sites are simple: compress and resize images, remove unused plugins and chat widgets, and avoid huge sliders at the top of the page. PageSpeed Insights will show your scores, and SEARCH ONE includes the same Google data in its report.

Structured data (schema markup) is a small block of code that labels facts on your page: this is a business, this is its address, these are its opening hours, this is an FAQ.

Useful types for most businesses:

• LocalBusiness (or a more specific type such as Dentist or JewelryStore) on your home or contact page.
• Organization with your logo and official profiles.
• Product or Service where relevant.
• FAQPage on pages that genuinely show questions and answers.
• Article with author information on your guides.

The rule that matters most: the markup must match what visitors can see. Google's AI features guidance asks for structured data that is consistent with the visible text. Marking up reviews, prices or FAQs that are not on the page is a spam risk, not a shortcut. Test your pages with Google's Rich Results Test.

An llms.txt file is a plain text summary of your site, with links, written for AI tools. It is a community proposal, not a standard that the major search engines have adopted.

Google's position is clear: you do not need new machine-readable files, AI text files or special markup to appear in AI Overviews or AI Mode. We publish an llms.txt on our own site because it takes ten minutes and does no harm. But if someone sells you llms.txt as a ranking solution, that is hype.

The same goes for "AI-only" pages, hidden text written for language models, or prompt-like instructions in your HTML. Hidden content has been against search engine spam policies for decades, and AI tools built on search indexes inherit that.

If you only do one thing after reading this, do this:

1. 1Open robots.txt and confirm Googlebot, Bingbot, OAI-SearchBot and PerplexityBot are not blocked.
2. 2Type your domain with and without www, and with http. Each should land on the same https address.
3. 3In Search Console, inspect your home page and top service page. Both should be indexed.
4. 4View source on your top service page and find your heading, phone number and price in the text.
5. 5Run the page through PageSpeed Insights on mobile.

Or run it through our free SEARCH ONE audit, which checks crawler access, indexing signals, structured data, answer-readiness and Google PageSpeed results for one page. For the content side, see Google AI Overviews and AI Mode: what they mean for your website.