Page MenuHomeMiraheze

Let each wiki control their robots.txt via MediaWiki:Robots.txt
Open, LowPublic

Description

For example: https://en.wikipedia.org/robots.txt transcludes https://en.wikipedia.org/wiki/MediaWiki:Robots.txt after the global Robots.txt. Can we do same thing for Miraheze?

Thanks!

Event Timeline

Dmehus triaged this task as Low priority.Sun, Feb 21, 19:36
Dmehus added a subscriber: Dmehus.

Possibly would be normal priority, but will triage as low based on triaging of similar requests

Technically the location of the file means little and moving it to be in MirahezeMagic would seem out of scope - as it's a root-level file. Keeping it within puppet seems easiest as theres no gain from moving it to mw-config.

@John: Oh makes sense. Any objections to implementing this through the puppet robots.php though?

In T6881#135656, @John wrote:

Technically the location of the file means little and moving it to be in MirahezeMagic would seem out of scope - as it's a root-level file. Keeping it within puppet seems easiest as theres no gain from moving it to mw-config.

Ah, true. Makes sense. I'm not fussed about which repo it's in.

If or when this is implemented, the global robots.txt's content should be added to the robots.txt file even if wikis do not keep them in their MediaWiki:Robots.txt as there may be global incidents caused by web crawlers otherwise.