RadiusRank
Menu

Generators

Robots.txt Generator

Build a robots.txt with explicit rules for search crawlers and AI agents like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.

Allow rules override a broader Disallow for the same agent.

Google ignores this. Bing and others honour it.

Block search crawlers

Block AI crawlers

robots.txt

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /*?sort=

Training, search, and user-directed agents

These are two different decisions and they are constantly confused.

  • Training crawlers such as GPTBot, ClaudeBot, and CCBot collect content that may contribute to model or public training corpora.
  • Search retrieval agents such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot support discovery for search and generated answers. Blocking them can reduce eligibility for those uses.
  • User-directed agents include ChatGPT-User, Claude-User, and Perplexity-User. These fetch pages in response to a user action, and published enforcement behavior differs by provider.
  • Product control tokens such as Google-Extended and Applebot-Extended govern specified AI uses of content collected by their associated crawlers.

A blanket block can create a different outcome than a narrow training opt-out. Review each provider's current documentation before publishing.

Before you ship it

  • robots.txt controls crawling, not indexing. To keep a page out of results, use noindex. A blocked page can still be listed without a snippet.
  • Save it at the domain root and test with the Robots.txt Checker.

See how 50 technology companies configure these tokens in the AI Crawler Access Benchmark.