AI crawler & llms.txt checker

Check whether your robots.txt allows or blocks the crawlers that power ChatGPT, Claude, Perplexity, Google's AI features, and more — then generate a starter llms.txt file, a proposed standard for pointing AI systems at your most important pages.

Website URL

Which crawlers this checks

GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI) · ClaudeBot, Claude-User, Claude-SearchBot (Anthropic) · Google-Extended (Google's AI features, separate from regular Googlebot) · PerplexityBot (Perplexity) · CCBot (Common Crawl — used as training data by many LLMs, not just one) · Bytespider (ByteDance) · Amazonbot (Amazon) · Applebot-Extended (Apple Intelligence) · meta-externalagent (Meta AI). Blocking a bot here doesn't remove you from that engine entirely — some products (like ChatGPT's live browsing) also fetch pages directly regardless of the training crawler's status — but it's the main lever you actually control.

Frequently Asked Questions

What is the AI Crawler & llms.txt Checker and how does it work? +

The AI Crawler & llms.txt Checker is a powerful, browser-based utility designed to streamline your SEO workflow. It processes your inputs entirely in your browser or through secure edge relays, delivering instant, accurate results without storing your data.

How do I use the AI Crawler & llms.txt Checker? +

Simply enter your URL or required data into the input field above and click the action button. The AI Crawler & llms.txt Checker will automatically analyze your input and generate a comprehensive, exportable report or code snippet in seconds.

Is the AI Crawler & llms.txt Checker completely free and unlimited? +

Yes! Like all tools on Crawlie, the AI Crawler & llms.txt Checker is 100% free to use. There are no paywalls, no login requirements, and absolutely no strict usage limits. You can run as many scans and generate as many reports as you need.

Is my data safe and secure? +

Absolutely. The AI Crawler & llms.txt Checker runs its core logic locally in your browser. We do not store your website data, track your internal links, or keep a database of your reports.