robots.txt tester
Pick a crawler and paste a page address to see whether your robots.txt lets it in, and the exact rule that decides.
What it checks
- Which group applies
- A crawler follows the group whose User-agent names it most specifically, and falls back to User-agent: *.
- Which rule wins
- The longest matching Allow or Disallow path wins; on a tie, Allow wins. * matches anything and a trailing $ anchors the end.
- AI crawlers
- Training crawlers (GPTBot, ClaudeBot, CCBot) and AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are separate names, so you can allow one kind and block the other.
It follows Google's reading of robots.txt (RFC 9309). Other crawlers mostly match it, but not always. robots.txt controls crawling, not indexing.
