Access is the prerequisite for everything else
All the content quality, structured data, and entity-building work in GEO is wasted if AI crawlers cannot reach your pages. Access is the literal prerequisite: before a model can retrieve your content, a crawler must have fetched it. The access layer has three components — your robots.txt directives, your rendering approach, and your llms.txt file — and each one can silently eliminate you from every AI answer engine simultaneously.
robots.txt: the first gate
robots.txt is a text file at the root of your domain that tells crawlers which paths they are and are not allowed to access. It was designed for search engine bots, but AI crawlers obey it too — and there are now more than a dozen major AI crawlers, each with its own user-agent name. An overly restrictive robots.txt written before AI crawlers existed may be blocking all of them. The default behaviour when a crawler encounters an ambiguous or missing rule varies by crawler, but most err on the side of caution. If you have not audited your robots.txt for AI crawler compatibility, you likely have unknown access gaps.
The render gap: JavaScript content that bots never see
No major AI crawler executes JavaScript. They send an HTTP request, receive the HTML in the response body, and process that — nothing more. Any content your site assembles in the browser after page load is invisible to them: client-rendered product listings, lazy-loaded descriptions, dynamically injected structured data, chat widgets, and personalised recommendations. This is the render gap. It is particularly damaging for e-commerce sites where the most valuable, specific content — product names, prices, specifications — is often the content most likely to be rendered client-side.
llms.txt: the opt-in signal
llms.txt is an emerging standard: a plain-text file at /llms.txt that gives AI systems a curated, human-authored summary of your site's content and purpose — something optimised for language models rather than for human browsers. Think of it as a structured introduction: here is who we are, here are our key pages, here is what an AI should know before summarising us. Fewer than 1% of websites have published one, which means publishing llms.txt is currently a differentiating signal rather than a baseline expectation. It does not guarantee citation, but it signals intentionality to AI systems that look for it.