Who allows which AI crawlers, at scale

Cross-domain research on robots.txt policy toward AI crawlers and agents: who allows, who blocks, and who stays silent — aggregated across every audited domain.

• Allow / Block / Silent breakdown per AI crawler, across all domains
• Spot policy trends before they show up in any single audit
• Backed by the same audit data set as the leaderboard