Page MenuHomeMiraheze

Google Search Console Team Coverage issues warnings
Closed, DuplicatePublic

Description

I get this email regarding https://abj.miraheze.org/wiki/Special:Categories

From: Google Search Console Team <sc-noreply@google.com>
Subject: New Coverage issue detected for site https://abj.miraheze.org/

Coverage issues detected on https://abj.miraheze.org/
To the owner of https://abj.miraheze.org/:
Search Console has identified that your site is affected by 1 Coverage
issues:

Top Warnings

Warnings are suggestions for improvement. Some warnings can affect
your appearance on Search; some might be reclassified as errors in the
future. The following warnings were found on your site:

Indexed, though blocked by robots.txt

We recommend that you fix these issues when possible to enable the
best experience and coverage in Google Search.

OK, so it is

Disallow: /w/index.php?title=Special:
Disallow: /wiki/Special:

That are causing it. So you need to put a special Allow for Special:Categories, before these lines. See
https://support.google.com/webmasters/answer/6062596

Event Timeline

Jidanni triaged this task as Normal priority.Feb 2 2020, 23:06
Jidanni created this task.

Indexing special pages are generally disallowed, as special pages are user interfaces for several functions rather than actual content pages. Wikimedia wikis also disallow indexing them, and example in Manual:Robots.txt on MediaWiki.org also explains about it. I also don't think it should be allowed (at least by default).

By the way, projects should be added as tags rather than subscribers.

OK, it seems even
<meta name="robots" content="noindex,nofollow"/>
isn't enough to prevent "Indexed, though blocked by robots.txt"
See https://yoast.com/x-robots-tag-play/

And here we see that it can't be in robots.txt, else Google can't read the <meta name="robots" content="noindex,nofollow"/> ...
https://webmasters.stackexchange.com/questions/117744/how-to-resolve-google-indexed-though-blocked-by-robots-txt

Sorry for the delay in answering. I don't quite understand what the issue here is, have you set this in Special:ManageWiki/settings?

Oh yes, I forgot that users couldn't change that. Would you like me to change that value?

No!
I am saying,
for all sites,
you have a problem.
Google will send a message to all people who use its Webmaster Tools about the problem.
The problem you can read about in the top of this bug report.

I see, though we will probably keep the default for all our sites so would you like us to make Special:Categories for your wiki indexable then?

Better keep it the same, lest this become a maintenance headache for you guys.
Let's see if other users run into this problem first.

Reception123 claimed this task.

Okay, if you would like that I will decline this for now.

Please reopen and let me know if you encounter users who have the same problem, or if it's important for you.

Here's what they look like:

5591-1.jpg (948×1 px, 101 KB)

All I know is anybody who indexes their site with Google should be getting these

To the owner of https://*.miraheze.org/

emails.

We blocked indexing of all special pages, this was fixed months back.

Jidanni renamed this task from robots.txt should not block Special:Categories to Google Search Console Team Coverage issues warnings.Sep 19 2020, 17:13

We blocked indexing of all special pages, this was fixed months back.

Yes but the problem is "Indexed, though blocked by robots.txt"

Apparently the only real solution is:
https://webmasters.stackexchange.com/a/117746/100701

Yes we did that in mid July. I'm just not blacklisting the ~6000 entires manually it would take to resolve the error in search console. Google shouldn't be indexing them any more. They're old errors.

Yes but with robots.txt:

# Disallow API and special pages
User-agent: *
Disallow: /w/api.php
Disallow: /w/index.php?title=Special:
Disallow: /wiki/Special:

according to https://webmasters.stackexchange.com/a/117746/100701 Google will never be able to read any of those meta tags embedded within the pages...

Hmm, I'd need to ask whoever asked that rule to see if we can switch it.