Phase 1 · Quick Wins · L0 → L1

Fix crawler access first.
Everything else depends on it.

No code required. Fix the biggest access and signal gaps this week. AI crawlers are blocked by missing robots.txt, JS-only rendering, and absent meta signals before they even evaluate your content.

L0 → L1Beginner~1 weekNo code required

·

Why this matters

TL;DR

Fix access first. AI crawlers are blocked by missing robots.txt, JS-only rendering, and absent meta signals before they even evaluate your content. Phase 1 clears these blockers in under a week with no developer required — setting the foundation every later phase builds on.

These fixes unblock AI crawlers entirely. Without them, every subsequent phase is irrelevant — a site that cannot be read cannot be cited.

Most sites are at L0 not because their content is bad, but because they have never declared they want to be found. robots.txt is the digital equivalent of posting an “Open” sign — without it, crawlers are legally and technically uncertain whether they can enter.

Major AI and search crawlers — JS execution and robots.txt behaviour
BotOperatorTypeExecutes JSHonors robots.txt
GPTBotOpenAIAINoYes
ClaudeBotAnthropicAINoYes
PerplexityBotPerplexityAINoYes
GooglebotGoogleSearchPartialYes
BingbotMicrosoftSearchPartialYes
meta-externalagentMetaAINoYes
ApplebotAppleAI+SearchPartialYes
Quick check

Test your site in 30 seconds

Enter your domain to open the three Phase 1 files directly — see exactly what AI crawlers see. Then run a full audit for a complete readiness score.

No http:// needed — e.g. example.com

Phase 1 steps

Work through these in priority order — P0 items block everything else.

P0Quick win

Audit and publish robots.txt

AI crawlers rely on robots.txt to know they're allowed. Missing or restrictive rules silently remove you from AI search results — crawlers default to denial on absence.

Impact
Very high
Effort
Low
Time
1–2 hours
Owner
Dev / Marketing
What the data says
How to
  1. Check your current robots.txt at yourdomain.com/robots.txt — if it returns a 404, you have no file and all crawlers are using default rules.
  2. Verify AI bots are not blocked. Look for blanket Disallow: / rules that could block GPTBot, ClaudeBot, PerplexityBot, and others.
    DIRECTIVE
    # Permissive robots.txt for AI crawlers
User-agent: *
Allow: /

# Explicitly allow major AI crawlers
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml
  3. Deploy the file at the domain root (not a subdirectory). It must be accessible at exactly https://yourdomain.com/robots.txt.
  4. Verify with: curl -I https://yourdomain.com/robots.txt — you should see HTTP/2 200 and text/plain content-type.
AI bot policy deep-dive →
P0Quick win

Add /llms.txt — the AI-specific robots.txt

The emerging standard for AI-specific directives. Without it, LLMs have no structured way to understand what you want them to use. Sites with llms.txt signal readiness and get better indexing.

Impact
High
Effort
Low
Time
1–3 hours
Owner
Dev / Content
What the data says
How to
  1. Create a plain-text file at /llms.txt in your site root (same level as robots.txt and sitemap.xml).
  2. Write the file using the llms.txt spec format — a markdown-like structure with a title, summary, and links to key content.
    TEXT
    # YourBrand

> YourBrand is a [brief description of what you do].

## Key pages

- [Homepage](https://yourdomain.com/)
- [About](https://yourdomain.com/about)
- [Products](https://yourdomain.com/products)
- [API docs](https://yourdomain.com/docs)

## Optional: full content index

- [llms-full.txt](https://yourdomain.com/llms-full.txt)
  3. Validate it's accessible: curl https://yourdomain.com/llms.txt — should return 200 with text/plain or text/markdown.
  4. Optionally create /llms-full.txt with your complete content index for LLMs that want deeper context.
llms.txt guide →
P1

Audit meta descriptions and Open Graph tags

Agents use meta descriptions as the first signal for page intent. Missing OG tags mean social sharing and link-preview agents get no structured context about your pages.

Impact
Medium
Effort
Medium
Time
2–4 hours
Owner
Dev / Marketing
How to
  1. Audit your top 10 pages. For each, check: meta description present and under 160 chars, og:title, og:description, og:image (at least 1200×630px), og:url.
    HTML
    <!-- Minimal required OG tags -->
<meta name="description" content="Clear, keyword-rich description under 160 chars." />
<meta property="og:title" content="Page Title — Brand Name" />
<meta property="og:description" content="Same or similar to meta description." />
<meta property="og:image" content="https://yourdomain.com/og/page-image.png" />
<meta property="og:url" content="https://yourdomain.com/page-path" />
<meta property="og:type" content="website" />
  2. Use the Open Graph Debugger (developers.facebook.com/tools/debug/) to validate any page URL.
  3. Priority pages: homepage, product pages, blog posts, and any page you want AI systems to surface.
P0

Check for JavaScript-only content (the render gap)

AI crawlers do not execute JavaScript. If your core content — product names, prices, descriptions — only renders client-side, LLM crawlers miss it entirely. This is the most common hidden blocker.

Impact
Very high
Effort
Medium–High
Time
1 day – 1 week
Owner
Developer
What the data says
How to
  1. Test any page by disabling JavaScript in your browser (DevTools → Settings → Disable JavaScript) and reloading. If your main content disappears, AI crawlers see the same blank page.
  2. Alternatively, curl the raw HTML and check for content: curl -s https://yourdomain.com | grep -o "<h1>[^<]*"
  3. If content is missing in the no-JS view: implement Server-Side Rendering (SSR) or Static Site Generation (SSG). On Next.js: use getServerSideProps or getStaticProps. On other frameworks: enable SSR mode.
  4. At minimum: ensure product names, descriptions, and prices are in the initial HTML payload — even if other features are client-rendered.
The render gap explained →
P1Quick win

Verify canonical URL consistency

A canonical tag pointing to a different domain silently redirects LLM crawl credit to that domain. Misconfigured canonicals are surprisingly common after CMS migrations or domain changes.

Impact
Medium
Effort
Low
Time
1–2 hours
Owner
Dev / SEO
How to
  1. Check your canonical tag: curl -s https://yourdomain.com | grep -i "canonical"
    HTML
    <!-- Correct: points to your own domain -->
<link rel="canonical" href="https://yourdomain.com/page-path" />

<!-- Wrong: points to a different domain (often a CMS or staging URL) -->
<link rel="canonical" href="https://staging.yourdomain.com/page-path" />
  2. Common mistakes to check: staging URLs in production, www vs non-www mismatches, HTTP vs HTTPS mismatches, old domain from a rebrand.
  3. Verify your top 5–10 pages individually — canonical errors are often page-specific, not sitewide.

What AI actually sees

This is what an AI-generated answer looks like when a site is blocked vs. when it's accessible. Note how the accessible competitor gets cited, while the blocked site is mentioned only vaguely.

AI-generated answersimulated

[GPTBot blocked — no robots.txt] When looking for product recommendations for enterprise PIM software, I found Akeneo PIM, which according to their site offers native AI enrichment and a product cloud for B2B and B2C commerce. Their documentation clearly outlines integration options and supported formats.

YourBrand appears to offer similar capabilities — however, I couldn't retrieve detailed feature comparisons. Their product pages may not be fully indexed.

Signal analysis
  1. BLOCKED
    Crawl blockedGPTBot denied in robots.txt — site cannot appear in OpenAI-powered answers.
  2. PASS
    Competitor citedAkeneo is cited because their site is accessible, server-rendered, and has clear meta signals.
  3. THIN
    Thin signalYour site appears but with hedged language ("appears to") — no structured data, weak content signals.
Simulated AI answer — illustrates the citation gap between a crawlable and a blocked site

Key takeaways

After completing Phase 1, you should have:
  • AI crawlers can access and index your site (robots.txt declared, no blanket blocks).
  • /llms.txt is published and links to your key pages.
  • Core content is server-rendered HTML — no render gap blocking LLM crawlers.
  • Meta descriptions and canonical tags are consistent across key pages.
← Playbook hubPhase 2 — Schema & Feeds →