CrawlCheck

Most sites were configured before AI crawlers existed. A single Disallow: / can make you invisible across ChatGPT, Perplexity and Claude at once - and you would never know.

CrawlCheck is a free, browser-based GEO audit toolkit. Paste your robots.txt and it checks it against 60+ AI crawlers - GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended, Bytespider, CCBot and more - showing exactly which are blocked and where a path-level rule cuts off your best pages.

It also audits the signals AI engines quote from: JSON-LD structured data, llms.txt, noindex tags and extractable text. Every issue comes with a copy-paste fix, plus a generator with presets for 'allow AI search only' and 'block training'.

Everything runs locally in your browser. Nothing is uploaded, logged or tracked.

Key features

• ✓Audits robots.txt against 60+ AI crawlers at 8 path levels
• ✓Live crawler fetch test - catches server-level blocks and 403s
• ✓Generates AI-ready robots.txt, llms.txt and JSON-LD snippets