Can AI search read your site?
Check robots.txt rules and live access for ChatGPT, Claude, Perplexity, Google and Bing crawlers, plus llms.txt, in about 15 seconds.
| Crawler | Used for | robots.txt | Live access |
|---|
What this checks
robots.txt rules. For each crawler we find the user-agent group that applies (its own name, or *) and the longest matching Allow or Disallow rule for the page you entered.
Live access. We request the page with each crawler's user-agent, once normally and once with a cache-busting parameter so the request reaches your server rather than a CDN cache. A 403, 429, 503 or a challenge page counts as blocked or challenged. Many sites look fine on cached pages but block bots at the origin.
llms.txt. We look for a plain-text /llms.txt file that summarizes your key pages for language models.
Limits
Tests are sent from Cloudflare's network, not from each crawler's real IP addresses. Firewalls that verify crawler IPs may treat the real bots differently. Google-Extended and Applebot-Extended are robots.txt tokens only, so they have no live test.
Read more: How to check whether AI crawlers can read your site.