Hidden Layer · Research
Field notes on GEO, agent-readiness, and the AI web — every statistic adjudicated by our truth engine, with receipts.
From September 15 the network in front of about a fifth of the web blocks AI training and agent traffic by default on ad-supported pages, for every domain onboarding after that date. Existing settings are untouched, which is the part worth reading twice.
Google now tells you how to optimize for AI search. The honest reading: Google declared GEO to be SEO and told you to stop buying AI files and special schema. That advice is also self-serving, so we audit Google’s claims the same way we audit everyone’s.
A source link in an AI answer is not the same as a brand recommendation. As of 2026-06-18, the cleaner metric is ghost citation rate: cited without being named.
Not every category needs the same GEO investment. The first question is not "how do we rank in AI?" It is whether your buyer uses AI at the moment a purchase decision is being made.
Free-tier AI relies more on training memory. Paid-tier AI adds retrieval, shopping feeds, and agentic actions. The same brand needs two tracks, not one GEO checklist.
AI search does not keyword-match. It resolves entities, reads constraints, maps intent, and chooses sources. A prompt is a decision object, not a string.
Schema markup gates inclusion. Prose attributes earn citations. They are not two artifacts — they are two renderings of the same Golden Record. Here is what the evidence says about which layer does which job, and what the practical implications are for product content strategy.
What the evidence actually says — 21 findings verified across independent sources, 9 popular claims refuted, and the open questions nobody has settled. Every assertion is grounded in an adjudicated fact, with its receipts.
What the evidence actually says about AI crawler infrastructure — 14 findings verified across independent sources, 1 popular claim(s) refuted, and the open questions nobody has settled. Every assertion is grounded in an adjudicated fact.
AI shopping agents parse structured product data — they do not browse. Whether ChatGPT, Perplexity, or Google AI Mode surfaces your products comes down to the completeness and accuracy of the records your PIM maintains. This piece connects the mechanics of product data quality to AI inclusion, and the gaps that silently delete products from consideration.
JSON-LD markup tracks at ~0% direct lift in AI citations across ChatGPT, AI Overviews, and Google AI Mode. The marginal hour spent on schema is worth less than one good statistic.
Three acronyms, three objectives, one question: where is your content actually being read? A decision-useful breakdown of search engine optimization, answer engine optimization, and generative engine optimization — what each optimizes for, what the evidence says moves the needle, and where the centre of gravity is now.
Three protocols now govern how AI agents discover, price, and transact with merchants. This reference maps who backs what, what each surface does, and which to implement first — grounded in specification dates and adoption data through May 2026.
A research blog that tells you how to be discovered by AI should be discoverable by AI. Here is exactly how we apply our own findings to this site — and how to verify every claim yourself.
We audited Nike, Allbirds, Patagonia, Glossier, and Shopify for product schema depth. Here's what property coverage reveals about AI-readiness across categories.
We ran our own auditor against itself and found five measurement sins that make every published score a coin flip dressed as data. Here's what we fixed — and what every GEO tool should disclose.
We hand-labeled 50 domains across 4 tiers, live-verified 37 of them, and ran a grid search against the results. Here's what we learned — and why twitter_card_tags (0.67 correlation) is the most underweighted check in every GEO audit you've ever seen.
In March 2026, Wix became the first platform to ship NLWeb by default — giving every eligible site an agent-queryable MCP endpoint. We tested 10 of them. Every premium site returned HTTP 401. Wix.com itself doesn't run NLWeb.
Discoverability 78%, agent integration 13%. We audited 128 domains across 13 industries. The foundation is mostly solid. The agentic web is not.
Two audits of the same domain, three weeks apart, return different GEO scores. Did the site improve — or did the model change? Without version pinning, you cannot tell the difference.
Agent integration = 0 across all 118 domains. The agentic web is coming but zero companies have started building for it.
Hidden Layer audited 118 domains across 12 industries. Every single one scored 0/13 on agent integration. The frontier is completely open.
A transparent look at the 8 categories, 229 base points, and why a B doesn't mean you're ready and an F doesn't mean you're invisible.
AI shopping agents don't browse — they parse. Product JSON-LD is table stakes for being read correctly, but it is not how you get into ChatGPT Shopping: that runs on merchant feeds. Here's the difference, and why it matters for where you spend the hour.
We ask a language model — no tools, no search, just training knowledge — to describe your domain. 15 points. The highest-weight single check in the audit. Here's why it matters and how to move the needle.
Your homepage loads beautifully in a browser. An AI crawler sees a skeleton. Here's why — and how to measure the gap for your own domain.
The Robots Exclusion Protocol became an IETF standard in 2022, and that standard says its rules are not access authorization. Two major AI operators document fetchers that ignore robots.txt by design. Here is what the file still does, and what has to sit behind it.
Shopify has one. Samsung has one. Anthropic and OpenAI don't. A practical look at what llms.txt does, what format AI models actually process, and whether it's worth your time.
A field guide to every major AI crawler: what company runs it, what it does with your content, whether it respects robots.txt, and what UA string to put in your rules.