Checkers & Validators

Robots.txt Checker

Fetch and parse any robots.txt, then test whether a specific URL is allowed for Googlebot, GPTBot, ClaudeBot, PerplexityBot, and other agents.

URL to test

We fetch /robots.txt from that domain and test this exact path against each agent.

How the matching works

This follows the rules Google documents: the most specific User-agent group wins, and inside that group the longest matching path wins, with Allow breaking an exact tie. Wildcards (*) and end-anchors ($) are supported.

Things worth knowing

A missing robots.txt (404) allows everything. A 5xx is treated by Google as disallow-all, so an outage on that one file can quietly deindex a site.
robots.txt controls crawling, not indexing. A blocked URL can still appear in results without a snippet. Use noindex to keep a page out.
Rules listed before any User-agent line apply to nobody. They show up here as uninterpretable lines.

Compare your policy with the public rules used by 50 AI, SEO, and CMS companies in the AI Crawler Access Benchmark.