The 50-site golden corpus: implementation grades and citation predictions

The previous post in this series identified four measurement sins in GEO auditing: N=1 LLM calls, unpinned models, truncated transcripts, and — the hardest one — no ground truth. We built a 50-site corpus, then found that its grade and citation labels answered different questions. This correction records those questions separately.

What the corpus is

The corpus is a labeled fixture: 50 domains, each with an expected_implementation_grade (A–F), a separate expected_citation_behavior, a tier classification, and metadata about robots policy, llms.txt, and schema richness. The implementation label is the answer key for the audit grade; the citation label is not.

The four tiers represent distinct positions in the 'LLM citation distribution':

The 'well-cited' tier and expected_citation_behavior are calibrated to what LLMs are predicted to cite organically — not to what scores well on our implementation rubric. Stripe, Vercel, and GitHub are in this tier because if you ask an LLM what payment API to use, it may name Stripe. Their expected_implementation_grade separately records the signals the audit expects to observe.

Original verification: 37 of 50 sites, live-audited

In the original 14 May 2026 run, 13 of the 50 corpus sites were unreachable from Cloudflare's edge network via standard HTTP (marked cf_edge_reachable: false in the fixture). Those sites either blocked CF IPs, required JavaScript-rendered responses, or sat behind WAFs that rejected non-browser UAs. The remaining 37 were batch-verified through the live Hidden Layer API.

Three findings from verification that changed the corpus:

• nextjs.org scored F/36 — not D as initially estimated. robots.txt returns 404 (robots_present: fail), content is Next.js-rendered (low content_efficiency), and no explicit AI bot rules. A canonical developer documentation site in the invisible tier would score D; nextjs.org has two compounding failures that push it to F.
• developer.mozilla.org scored D/56 — lower than its 'well-cited' tier implies. MDN has massive LLM citation presence (geo_cold_recall: pass, hn_mentions: pass) but thin structured data and no llms.txt. The gap between 'LLMs know this site' and 'this site has GEO signals in place' is wider here than anywhere else in the corpus.
• Social and cloaking sites (facebook.com, linkedin.com, yelp.com, glassdoor.com) scored poorly on the implementation audit. Facebook robots.txt blocks every AI crawler explicitly. The robots.txt is actually a useful model: complete, deliberate, explicit policy for every named bot.

Correction: 34.7% exact implementation-grade match on 49 scored sites

The original comparison treated disagreement with a citation-oriented answer key as grading error. On 12 September 2026 we renamed that answer key, ran the real runAudit engine across all 50 corpus domains, and obtained grades for 49. The corrected implementation-to-implementation comparison produced 17 exact matches: 17/49, or 34.7%. One domain failed inside the engine and remains recorded outside the denominator.

That 34.7% is an exact-match rate for a live implementation snapshot, not a citation-prediction accuracy figure. The same report retains citation prediction as a separate score: wikipedia.org, for example, scored 50 for implementation while its curated citation prediction is 100. A site can be highly cited because of authority and history while implementing few of the signals this audit checks.

The surprising finding: twitter_card_tags at 0.67 correlation

The original study computed Pearson correlation between each check's pass/fail result and the then-ambiguous grade field (converted to ordinal: A=4, B=3, C=2, D=1, F=0). Those historical correlations are reproduced below, but they must not be read as citation effects.

twitter_card_tags at 0.67 correlation was the biggest surprise in that historical run. We had it at weight 2 — 'low: Twitter/X cards; minor AI preview signal'. Established brands with B/A labels almost universally had complete Twitter Card metadata. This was a proxy for teams that maintain structured metadata, not proof that Twitter Cards cause citations.

The negative correlations for agent_integration checks (a2a_agent_card: -0.21, oauth_protected_resource: -0.22) are expected and confirm the design decision to keep those weights low. Only developer-tool companies have these — and developer-tool companies tend to cluster in the mid tier (C/D), not at the top. Weighting them heavily would punish media companies and fashion brands for not having MCP servers.

The bot access paradox

The highest-correlation group is bot access rules (gpt_bot_rule: +0.79, cc_bot_rule: +0.78, claude_bot_rule: +0.77). This looks like it validates the current weighting — except it's mostly spurious correlation.

Well-known brands in the well-cited tier (Stripe, Vercel, Cloudflare, GitHub) tend to have explicit robots.txt entries for each AI bot because they have dedicated developer-relations teams that update infrastructure documentation. Less well-known brands (mid and invisible tier) often have default robots.txt files that haven't been touched since 2019.

The correlation isn't 'explicit bot rules cause high grade'. It's 'having explicit bot rules is a proxy for being the kind of company that actively maintains its web infrastructure'. That's a real signal — but it means bot access rules are partly measuring team maturity, not just GEO signal compliance. Something to watch as the corpus grows.

The calibration limit: N=38 is too small for per-weight precision

With 57 weights and 38 samples, the grid search is operating at roughly 1.5 samples per parameter. Any individual weight change is noise-sensitive. Our approach was conservative: only change weights with |correlation| ≥ 0.25 and cap changes at ±2 points. We changed 8 weights.

The right version of this analysis uses N=200+. The corpus expansion plan:

1. Current: 50 sites, 37 verified (this post)
2. Phase 4: 200 sites, 150+ verified — sufficient for per-category weight calibration
3. Phase 5: 500 sites with actual citation-frequency data from ChatGPT/Claude/Perplexity API queries — this is the ground truth that replaces 'our rubric' with 'actual AI citation behavior'

Phase 5 is the version that makes GEO audit scores meaningful in the same way that click-through rate data makes SEO audit scores meaningful. We're building toward it.

What the corpus revealed about 'canonical' sites

The most interesting finding isn't in the numbers — it's in the gap between 'well-cited by LLMs' and 'scores well on our audit'. Vercel and Stripe are the canonical examples: they're in the well-cited tier because LLMs cite them unprompted, yet both score C on the current audit.

Why? The recalibration we shipped in Phase 3 expanded the agent_integration and ai_visibility categories. Vercel and Stripe don't have llms.txt (surprisingly), don't have agent-card.json, and have modest structured data for their homepage. The checks they fail are the ones added most recently — the 2025-era signals.

This creates a useful heuristic: if a site is well-cited by LLMs but scores C on our audit, it's usually because it earned its training corpus presence before these signals existed. It's a legacy moat. The question for those brands is whether they'll maintain that moat as newer sites adopt the signals their audits now check for.

The corpus is public

The 50-site fixture is published at /api/research/dataset.json (JSON) and /api/research/dataset.csv (CSV). Each row contains: domain, tier, expected_implementation_grade, robots_policy, llms_txt, schema_org, expected_citation_behavior, and notes including the live audit result where verified.

If you use the corpus to evaluate your own GEO audit tool and find a site we've miscategorized, file an issue at the project repository. The corpus is a living document — we'll expand and recalibrate it as more verified data comes in.

Next in the series: 400 queries × 4 models × 30 days — what ChatGPT, Claude, Perplexity, and Gemini actually cite when asked category questions. That citation-frequency data can replace expected_citation_behavior with a measured citation rate while leaving the implementation grade intact.