By Evan Weber, Founder, AEO Improvement · Updated October 20, 2026
Start with the same job your customer has: find a reliable answer and decide what to do next. Make that answer accessible, clear, and supported by facts. Then measure whether search and AI systems send useful visitors.
Imagine a fictional equipment-rental business receiving the question, "Can I pick this up on Saturday and return it on Monday?" Its page describes flexible service but does not give the relevant hours or rental-period rules.
The first useful edit is to publish the verified pickup window, return arrangement, applicable pricing explanation and a clear way to confirm availability. Keep the answer on the existing relevant page and link it from the equipment listing.
After publishing, check the actual URL and the relevant search-access rules. If you sample the question in different AI services, keep each response and its context separate. One service citing the page does not establish that another service has seen it or that a customer made a booking.
Each engine has different crawlers, different trust signals, and different citation behavior. Check the providers your customers actually use and keep their crawler rules separate.
ChatGPT
OpenAI
Crawler: OAI-SearchBot (search), GPTBot (training)
Key signal: Search retrieval and accessible page content
Keep search and training crawler policies separate.
Claude
Anthropic
Crawler: Claude-SearchBot, Claude-User (search and retrieval)
Key signal: Web search and user-initiated retrieval
ClaudeBot is a separate model-development crawler.
Gemini
Crawler: Googlebot (Google Search and AI features)
Key signal: Search indexing and snippet eligibility
Google-Extended controls certain Gemini uses outside Google Search.
Perplexity
Perplexity AI
Crawler: PerplexityBot
Key signal: Search retrieval
Check current Perplexity guidance before changing access rules.
Traditional search foundations still matter, especially crawl access, useful content, and clear links.
Start with the search crawlers relevant to your customers: Googlebot for Google's AI search features, OAI-SearchBot for ChatGPT search, Claude-SearchBot and Claude-User for Claude, and PerplexityBot for Perplexity. Check each provider's current documentation, then inspect robots.txt and any CDN restrictions. GPTBot and ClaudeBot serve separate model-development purposes. Google-Extended controls some Gemini uses outside Search.
# Example only: review each provider's current policy
User-agent: Googlebot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
Check the initial HTML and the rendered page for the main answer and useful links. If content is missing from the initial HTML, use Search Console's URL Inspection tool to see what Google rendered. Google can render JavaScript, so a missing sentence in raw HTML alone does not prove a page is invisible or require a framework change.
Put a clear answer near the relevant heading, then add the detail needed to make it useful. Descriptive headings, concise summaries, and verifiable facts help readers. There is no required number of sentences or ideal answer length for AI search.
An llms.txt file is an optional content map. Google says no special AI text file is required for its AI search features. Prioritize useful pages, search eligibility, and accurate internal links first.
Give readers clear information about the company, its products, limitations, and contact options. Keep facts consistent across the pages and profiles you control. Independent coverage can add context, but it does not guarantee inclusion in an AI answer.
Check the page before adding content
A blocked crawler, incorrect canonical, or missing main content can limit discovery. Check those issues first, then improve the answer and next step for the visitor. Missing schema by itself does not erase the value of useful page content.
Make the page publicly accessible, check OAI-SearchBot access for ChatGPT search, and answer a question people actually ask. Clear, useful content can improve eligibility, but no change guarantees that ChatGPT will cite your page.
No. Search providers use different systems and crawler controls. Google AI Overviews and AI Mode rely on Google Search eligibility and Googlebot access. ChatGPT search uses OAI-SearchBot. Claude has separate search and user retrieval bots. Check each provider's current documentation before editing robots.txt.
More pages alone are not a strategy. Start with pages that answer real customer questions. Improve access, clarity, original detail, and the next step for the reader before publishing another page on the same topic.
No. Do not create a Wikipedia page unless the topic independently meets Wikipedia's rules. Google says there is no special schema or AI text file required for its AI search features. Accurate structured data can still help Search understand content when it matches the visible page.
There is no reliable timeline. Crawl and indexing can take time, and an eligible page may never be selected for a particular answer. Track search queries, qualified visits, conversions, and repeated AI answer samples instead of expecting a result within a fixed number of days.
Provider documentation, research and practitioner guidance have different roles. Expert surveys express opinions, not confirmed ranking factors. Read the original source and its scope before applying a recommendation.