Enter your sitemap or your site and review what you're asking Google to index: duplicate URLs, URLs from another domain or blocked by robots.txt, plus the HTTP status code, noindex and canonical of a sample requested as Googlebot.
If you enter just the site, I look for the sitemaps in its robots.txt (or at /sitemap.xml).
Each URL is requested as Googlebot and its redirects are followed. Large sitemaps get a random sample.
How to use it
The sitemap tells Google which URLs you consider important. If it mixes in redirects, errors or pages with noindex, the signal loses value and in Search Console you'll see "Submitted" URLs that never get indexed. Ideally every URL returns a 200, is indexable and is its own canonical.
If you have to rebuild it, the XML sitemap generator creates it from a list of clean URLs.
FAQ
Can't find your question? Email me at [email protected] and I'll reply within 24 hours.
The ones you don't want Google to index: redirects, 4xx and 5xx errors, URLs with noindex, URLs canonicalized to another URL, and URLs blocked by robots.txt. A sitemap with those URLs sends contradictory signals, and Google ends up trusting it less.
Because requesting every URL takes time and a sitemap can have hundreds of thousands. A random sample of 50 or 100 URLs is enough to spot a systematic problem (for example, every URL of one type redirecting). To check them all, run them through a crawler.
Yes. It reads .xml and .xml.gz files, follows sitemap indexes up to 50 files and 200,000 URLs, and if you give it just the site, it looks for the sitemaps in its robots.txt.
The auditor downloads the site's robots.txt and tests every sitemap URL with Google's official parser, the same one Googlebot uses. To test rules by hand, there's the robots.txt tester.
Let's talk
Tell me what you want to achieve and I'll tell you honestly whether I can help.
What happens next
Prefer to talk? Book a 30-min call
Or email me at [email protected]