Does it give the same result as Googlebot?

It uses the same code. Google published on GitHub the library Googlebot uses to read robots.txt files, and I compile it to WebAssembly without touching it, so the typos it accepts, the percent-encoding and the tie-break between rules are Google's own. What the library doesn't do is the inheritance between Google crawlers: Googlebot-Image follows Googlebot's rules if it has no group of its own, and that is decided by the crawler, not by the parser.

What about ChatGPT, Claude or Perplexity?

The standard (RFC 9309) is the same for everyone, and OpenAI, Anthropic and Perplexity say they respect robots.txt, so the verdict is a good approximation. Each company uses its own parser, and in rare cases (typos, lines without a colon) they may read something different from Google.

What happens if my robots.txt returns an error?

If it returns a 4xx, Google understands there are no restrictions and crawls everything. If it returns a 5xx or doesn't respond, it stops crawling the site while the problem lasts. When you load from a URL, the tool tells you which of the two cases you're in.

Is what I test stored?

No. The parser runs in your browser. The only thing that goes through my server is the robots.txt download when you load it from a URL, and it isn't stored.

robots.txt Tester with Google's Official Parser

Why another robots.txt tester

Google retired the robots.txt tester from Search Console in 2023. Most of the ones left rewrite the rules their own way, and they fail in the odd cases: the longest rule against the one that allows, a user-agent: Google that doesn't apply to Googlebot, a typo like disalow that Google does accept, or a URL with accents.

This one uses the library Google published at github.com/google/robotstxt, the one that reads robots.txt files in production, compiled to WebAssembly. The line report comes from its reporting library, the one that feeds the robots.txt report in Search Console.

How to test a change before you deploy it

Load your site's robots.txt, edit the rules in the box and enter the URLs you care about: the ones you want to block and, above all, the ones you don't. Test with Googlebot and the AI bots at the same time to see whether a rule meant for GPTBot affects anything else.

To check whether the server or firewall also lets each bot through, use the bot access checker.

robots.txt tester with Google's parser

Result

What Google makes of your robots.txt

Why another robots.txt tester

How to test a change before you deploy it

Frequently asked questions

Tell me about your project