a single source, nothing corroborates it

Never use robots.txt to hide a page

A disallowed URL can still be indexed and listed with no description under it, because a single inbound link is enough. Worse, blocking a page then adding a noindex tag guarantees the tag is never read. Use the file for what it is good at, keeping crawlers off tools that spin out near-identical URLs, manage it from exactly one place, and export the affected URLs when the warning fires.

The method

1. 01

   Disallow crawling of any on-site tool that spins out large numbers of near-identical URLs, so crawlers are not spending their time on them.

2. 02

   Do not use robots.txt to keep a page out of Google's results, because a disallowed URL can still be indexed and shown with no description underneath it.

3. 03

   Never block a page in robots.txt and then add a noindex tag to it: allow the crawl first, because a crawler that cannot fetch the page will never read the directive telling it to drop the page.

4. 04

   Fix an indexed though blocked warning by exporting the affected URLs from Search Console, deleting the disallow rules covering any you actually want crawled, then returning to validate the fix.

5. 05

   Manage robots.txt from exactly one place, either the file in your root directory or an SEO plugin, never both at once.

What it returned

• Detailed's robots.txt blocks /tweets and /links, two tools that generate many URLs with similar content.

• Detailed disallows /links/ and /tweets/, and /links/ is indexed anyway, listed with a note that no information is available. John Mueller is quoted saying a random inbound link is enough for Google to index a blocked URL with no content.

• Google flags this as important in its own documentation.

• The status means Google has surfaced a URL it was never allowed to crawl, which happens when other sites link to a blocked page.

Sources

• Resolving the 'Crawled - Currently Not Indexed' Page Indexing Status - Detailed.comGlen Allsopp · detailed.com · Technical Issues · 2023-10-23
• What Is the Indexed, Though Blocked By Robots.txt Message? - Detailed.comGlen Allsopp · detailed.com · The Quick Fix · 2023-10-17
• Robots.txt SEO Best Practices & Examples (Updated for 2023) - DetailedGlen Allsopp · detailed.com · Table of Contents · 2018-11-21