Only 23% of readable robots.txt files named an AI agent.

What the 50-site snapshot found

1. Explicit AI controls were the exception. Eleven of 47 readable files named at least one studied AI agent. Thirty-six readable files relied on wildcard rules or had no matching rules for those agents.
2. GPTBot was the most frequently named training crawler. Eight readable files named GPTBot. Seven named OAI-SearchBot, showing that many teams that configure OpenAI controls distinguish training from search retrieval.
3. Search and user-directed access stayed open at the root. None of the readable files blocked OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, or Perplexity-User at the root path under the rules we parsed.
4. Selective blocks concentrated on training and control tokens.GPTBot, ClaudeBot, CCBot, and Applebot-Extended were each blocked at the root by one readable file. Google-Extended was blocked by two.
5. Three domains could not be classified. Similarweb failed during collection, while OtterlyAI and AEO Engine returned HTTP 403. We report those checks as unavailable instead of assuming access.

Training, search, and user-directed agents are different controls

A single AI-crawler score hides the most important distinction. OpenAI, Anthropic, and Perplexity publish separate agents for different uses. A site may permit search discovery while declining model-training access. Site owners should make those choices independently.

Perplexity states that Perplexity-User generally ignores robots.txt because the fetch is user requested. Its row is still useful for detecting whether publishers name the token, but the rule should not be treated as proof of enforcement.

The 50-site AI crawler access dataset

“Allowed” below means the applicable robots.txt group did not block the domain root, /, for that token. It does not prove that every path is crawlable, that the crawler will visit the site, or that content will be indexed, trained on, or cited.

Methodology and reproducibility

The sample contains 50 known companies across AI platforms, SEO platforms, content optimization, AI visibility, local SEO, and CMS software. The list was selected for relevance to the search and AI-discovery market. It was not randomly drawn.

1. Request https://domain/robots.txt with a named RadiusRank research user agent and a 15-second timeout.
2. Follow redirects and record the final URL, HTTP status, content type, and byte size.
3. Reject HTTP errors, network failures, and HTML responses rather than interpreting them as valid robots files.
4. Parse user-agent groups, Allow, Disallow, wildcards, end anchors, and Sitemap declarations.
5. Choose the most specific matching agent group, then apply the longest matching path rule at /, with Allow winning exact ties.
6. Store the SHA-256 hash of every readable source so future snapshots can detect changes without republishing third-party files.

The downloadable generator uses Node.js built-ins and the source domain list in this repository. Run it again with a new date to create a fresh CSV and JSON snapshot.

What this benchmark does not prove

• Root access is not whole-site access. A site may allow / while blocking private paths, search pages, parameters, media, or specific directories.
• robots.txt is a declared preference. The benchmark does not verify crawler IP addresses, actual requests, compliance, indexing, model training, citations, or rankings.
• A 403 is not classified as blocked by robots.txt. It may come from a firewall, bot defense, authentication, or another server policy.
• The web changes. Results describe the collected snapshot. Source hashes and the collection date are included so changes can be audited.
• The sample is intentionally narrow. These results describe selected technology brands and cannot estimate behavior across the whole web.

What site owners should do

1. Test the effective rules. Use the free robots.txt checker on production content paths, not only the home page.
2. Separate policy decisions. Decide independently whether to allow training, search retrieval, product controls, and user-requested visits.
3. Generate narrowly scoped rules. The robots.txt generator includes separate AI-agent controls.
4. Check server enforcement. Review CDN and WAF logs because a correct robots file cannot fix a firewall that blocks the real crawler.
5. Monitor changes. Store dated copies or hashes of robots.txt so accidental policy changes are visible.

Crawler access is only one part of answer engine optimization. Read the practical AEO guide or see how RadiusRank handles technical eligibility, evidence, and authority in its answer engine optimization service.

Primary documentation

• OpenAI crawler documentation: GPTBot, OAI-SearchBot, and ChatGPT-User
• Anthropic crawler documentation: ClaudeBot, Claude-SearchBot, and Claude-User
• Perplexity crawler documentation: PerplexityBot and Perplexity-User
• Google crawler documentation: Google-Extended
• Google Search Central: introduction to robots.txt

You may cite the benchmark with the snapshot date and a link to this page. For a custom breakdown or a correction supported by a public source, contact team@radiusrank.com.