Two emails from Search Console arrived the same morning. The first was the one that worried me:
New reasons prevent pages in a sitemap from being indexed on site utilorax.com
Excluded by 'noindex' tag
Read that carefully. Pages in a sitemap — pages I am explicitly asking Google to index — are being skipped because they carry a noindex. That is a contradiction, and it is the kind that quietly costs you: the sitemap says "index this", the page says "don't", and Google believes the page.
Fifty-one of them. My site has around 6,000 URLs across five languages, so 51 was small enough to be a real bug and large enough to matter.
Here is how I found out whether it was one, without opening 51 URLs by hand.
The method: sample by URL shape, not at random
Random sampling is the wrong tool here. A noindex bug is almost never random — it comes from a template, a route, or a language variant, so it affects a whole class of URLs. If you sample 50 URLs out of 6,000 at random you will probably miss a class of 51. If you sample one URL from every distinct shape, you cannot.
So:
1. Collect every URL the sitemaps actually claim. Not what you think you publish — what the XML says today.
curl -s https://example.com/sitemap.xml \
| grep -o '<loc>[^<]*</loc>' | sed 's|</\?loc>||g' > maps.txt
while read m; do
curl -s --retry 2 "$m" | grep -o '<loc>[^<]*</loc>' | sed 's|</\?loc>||g'
done < maps.txt | sort -u > allurls.txt
That gave me 5,881 URLs across 54 sitemap files.
2. Reduce them to shapes. Strip the language prefix, keep the first path segment, mark anything deeper with a wildcard. /es/convert/miles-to-km and /es/convert/kg-to-lb are the same shape; /es/blog/some-post is a different one.
5,881 URLs collapsed into 101 distinct shapes — en /convert/*, es /convert/*, fr /tools/*, en /about, and so on.
3. Fetch one of each and check both places a noindex can hide. This is the step people get wrong: they check the HTML meta tag and forget the header, or the reverse.
while read u; do
H=$(curl -s -m 30 -D - -o body.html "$u")
XR=$(echo "$H" | grep -i "^x-robots-tag")
MR=$(grep -o '<meta name="robots" content="[^"]*"' body.html | head -1)
if echo "$XR $MR" | grep -qi "noindex"; then echo "HIT: $u"; fi
done < sample.txt
Result: 101 out of 101 returned 200, and not one carried a noindex — in the header or in the markup. No contradiction. The sitemap and the robots directives agreed everywhere.
Ten minutes, and the scary email was answered.
So what were the 51?
Once the sitemap was cleared, the answer had to be pages Google found some other way — through internal links, old crawls, or the wider web. Grepping my own routing for every place that sets a noindex gave the list immediately:
-
/api/*— JSON endpoints -
/cron/*— scheduled jobs -
/dl/*— file downloads -
/developers/*— logged-in account pages - and eight pages I had to work out: the
es,fr,ptandzhversions of/refundand/terms, which have no translation and are deliberatelynoindexoutside English until someone writes one
Every one of them is noindex on purpose. Google was not reporting a bug. It was reporting a fact, in a message worded strongly enough to look like a bug.
That is the part worth keeping. "Excluded by 'noindex' tag" is not an error. It is Search Console telling you what it found, and on any site with an admin area, an API or logged-in pages, a non-zero number there is correct and healthy. The number to react to is not the count — it is whether any of those URLs are ones you wanted indexed.
The other number I had been reading wrong for a week
While I was in there, the Page Indexing report finished processing for the first time since launch. It said:
Indexed 3,040
Not indexed 3,237
Total known 6,277 -> 48.4% indexed
My notes said 12%.
That figure had come from the HTTPS report, which showed 728 URLs at a time when the indexing report was stuck on "processing". I had divided 728 by my sitemap total and written down "only 12% is indexed, so indexing is the bottleneck" — and then let that sentence drive my priorities for a week.
The HTTPS report counts URLs whose HTTPS status Google evaluated. It is a sample. It was never the index size, and it never claimed to be. I had made it answer a question it was not asked.
Half of a twelve-day-old site being indexed is not a bottleneck. It is fine. The actual bottleneck was somewhere else entirely, and I had spent a week not looking at it.
The one that was genuinely worth opening
The bigger bucket was "Crawled — currently not indexed": 215 pages. That one means Google fetched the page, read it, and decided against it — which is a judgement, not a mechanic, and worth understanding.
I pulled the full list and counted it rather than eyeballing the examples:
| Non-English | 97% (es 38%, fr 35%, pt 23%) |
/convert/* pages |
71% |
So: non-English converter pages, almost exclusively.
The obvious theory is thin or machine-translated content. I tested it by comparing a Spanish page against its English counterpart word by word — they share only 17–21% of their vocabulary and each runs about 1,000 words. The translations are real.
What I did find was worse in a subtler way: three places built an English sentence and dropped translated nouns into the gaps. The breadcrumb read "Caballos de Fuerza to Vatios". Related cards read "Convert Caballos de Fuerza to Kilovatios instantly." Eleven English fragments per page, on every converter, in four languages.
That is a genuine quality bug and I fixed it. But I want to be honest about what fixing it does, because the temptation is to declare victory: it is almost certainly not why those pages were skipped. A twelve-day-old domain with zero external links does not get unlimited index budget, and non-English pages of a site nobody has ever linked to are the first thing dropped. That is an authority problem. No amount of copy editing fixes it — links and time do.
What I would tell myself a week ago
- Sample by URL shape, not at random. Template bugs affect classes, so test classes.
-
Check the header and the meta tag. A
noindexcan live in either, and the report does not tell you which. - "Excluded by noindex" is usually correct. Verify it is only hitting things you meant.
- Never infer one report's number from another's. The HTTPS report is not the index count. A number that is roughly the right shape is the easiest kind of wrong to keep believing.
- When you find a real bug during an investigation, do not let it become the answer. It felt like the cause. It was not.
I build Utilorax, a set of free browser-based tools. Everything runs client-side, which produces a steady supply of problems like this one.
Top comments (10)
This is the right kind of Search Console response: verify the actual page state before treating the report as a mystery. I would check source HTML, rendered DOM, headers, canonical signals, sitemap inclusion, and whether templates are applying noindex conditionally.
Your list is the right one, and two items on it did the work here.
Headers as well as HTML: an X-Robots-Tag does the same job invisibly, and checking only the markup is how people end up deciding the report is wrong.
And sitemap inclusion, which turned out to be the entire answer. All 101 URL shapes in my sitemaps came back 200 with no noindex anywhere, header or markup. The 51 were pages Google had reached by other routes — /api/, /cron/, /dl/*, account pages, plus the untranslated legal pages that are deliberately noindex outside English. Every one correctly excluded.
The conditional-template case you mention is the one I was braced for, and it wasn't that. Worth saying out loud, because "the report is right and your site is fine" is a real outcome and nobody writes that one up.
That distinction between sitemap URLs and discovered URLs is important. Search Console can look wrong when it is really reporting a crawl path you did not mean to expose. Checking headers, markup, status, and sitemap membership separately gives you a much cleaner answer.
"A crawl path you did not mean to expose" is a better description of it than the one I used in the post, and it points at the part I still cannot answer.
The sitemap was clean — I checked all 101 URL shapes and none carried a noindex. So Google reached /api/, /cron/ and /dl/ some other way, and I do not actually know which. They are not linked from any public page. Not from a sitemap, not from navigation.
My guesses in order: an old crawl from before those routes were locked down, a stray link in something I have forgotten about, or plain URL discovery from somewhere outside my control.
None of those are things I can rule in or out from my side, which is the mildly uncomfortable part — the report was right, my pages were right, and the interesting question turned out to be one Search Console does not answer at all: not "why is this excluded" but "how did you find it".
Have you found a reliable way to work out the discovery path? That is the one piece of this I would still like a method for.
That unknown crawl path is the interesting part. I would log it like a local SEO crawl incident: URL shape, first-seen date, referrer if available, sitemap presence, internal-link proof, robots state, and server logs around Googlebot. The fix is not only noindex cleanup; it is understanding which discovery surface made Google think those routes existed.
That list is the answer, and the item I would have skipped is the one that would have settled it: server logs around Googlebot. Everything else describes the page's current state, which is what I checked and why I got stuck — none of it records how the URL was first reached.
The referrer and first-seen date are the only two fields that point backwards, and they are exactly the two I do not have, because I went looking after the report rather than logging before it. Which is the honest shape of the problem: it was answerable, and it stopped being answerable at the moment Google crawled it.
Turning it into a standing record rather than an investigation is the part I am taking from this. A crawl-incident row written when the URL first appears costs nothing and is worth more than any amount of checking after the fact.
That is the part I would turn into a standing check now: when Search Console shows an excluded URL class, capture first-seen, referrer if available, sitemap membership, internal links, and server-log hits before changing anything. Once you fix the routes, the evidence trail gets cleaner but the origin story disappears.
That unknown crawl path is exactly why I like first-seen logs. Once the route is cleaned up, current inspection can prove the present state, but it cannot reconstruct the original doorway. The prevention move is boring: log discovery evidence before it becomes a mystery.
ngl i always struggle with finding which exact pages are triggering that noindex warning in GSC. thanks for showing the workflow.
The thing that made it tractable was not sampling randomly. A noindex almost always comes from a template or a route, so it hits a whole class of URLs at once — 50 random URLs out of 6,000 can miss a class of 51 completely.
Collapsing 5,881 URLs into 101 shapes (strip the language prefix, keep the first path segment, wildcard the rest) meant one fetch per shape covered everything. 101 requests instead of 5,881, and it structurally can't miss a class.
The other half is checking both places the tag can live — X-Robots-Tag in the response headers as well as the meta tag. Grep only the HTML and you'll come away convinced the report is lying to you.