AI crawler simulator: see your page the way an AI crawler does.

Why a page can rank in Google and still be invisible to AI

Google and the AI assistants read the web through different doors. Googlebot fetches a page, then comes back to render its JavaScript. The AI crawlers, as far as any vendor documents, fetch the HTML and stop there. A page that builds its content in the browser can rank well on Google and still hand an AI crawler a near-empty document.

Two other doors close more quietly. robots.txt can block a crawler for one path and not another, so a site that allows GPTBot on the homepage can still block it on /pricing/. And many CDNs now ship bot-management rules that challenge or block AI user agents by default, which means the server answers a real browser with the page and an AI crawler with a 403. Neither shows up when you open the site yourself.

The simulator requests the page once per crawler and puts the three answers side by side: what robots.txt says, what the server sent back, and what text was actually in the response.

Training crawlers and retrieval crawlers are different decisions

Not every AI crawler does the same job. Training crawlers, such as GPTBot, ClaudeBot and Google-Extended, collect pages to train models. Retrieval crawlers, such as OAI-SearchBot, ChatGPT-User and PerplexityBot, fetch a page because someone just asked a question it might answer.

Blocking a training crawler is a position on how your work is used. Blocking a retrieval crawler removes the page from AI answers, and it is usually done by accident, with a “block all AI bots” rule copied from somewhere. When the simulator shows a block, check which kind it is before you decide whether to fix it. The robots.txt generator writes the split version, training blocked and retrieval allowed, if that is the policy you want.

Reading the result

• Blocked in robots.txt. The crawler is asked not to fetch the page. Well-behaved crawlers comply, so nothing else in the row applies. Fix it in robots.txt if the block is not deliberate.
• Allowed, but a 403, 429 or 503. The request reached something that refused it. Look at your CDN’s bot settings and firewall rules for the named user agent.
• Allowed, 200, but little text. The server answered, but the content is not in the HTML. That usually means client-side rendering. Server-side rendering or static generation fixes it.
• Allowed, 200, and the text you expect. That crawler can read the page. Whether it cites it is a separate question, and one about the content and the site’s authority rather than access.

Access is step one of nine

Being fetchable is the prerequisite, not the goal. An assistant also needs to find the page, parse what it is about and trust it enough to quote. The free AI-readiness check tests a whole domain against nine of those signals, including the robots.txt verdict for eight named crawlers, and gives you a shareable result. Run the simulator on the page, and the check on the site.

Once a site can be read, the work is what it says. AltoRank plans and drafts that content every week, and a person approves each piece before it publishes.