Looks like someone wrote an override to the MediaWiki:Uploadtext system message using a template that only exists on meta. Should be fixed here.

The writeapi permission is also a required component for certain features and extensions, some of which may not be immediately obvious. This includes VisualEditor and DiscussionTools, though other extensions such as Comments may also not work without that permission. That being said, it's hard to determine the entirety of what may be affected given that some of these extensions don't list the requirement.

I fixed the sitenotice, it was a simple fix, but php error messages aren't very helpful at identifying the cause.

Possibly a conflict with Lingo and Wikibase:

Seems to be a problem with moderation:

Slightly different issue (I'd raise it elsewhere but I have a meeting I need to get to), but it appears that restricting view access to require "delete" permissions does not work, only restricting access to "createwiki" (I have not checked os).

I'm uncertain right now (I simply don't have the availability to investigate anything), but I believe this may have just happened again on db121. High CPU load and memory usage caused mariadb to get oom-killed at around 1:51 UTC tonight. Ordinarily, I would create a new task for this, but I'm not sure if we have the proper logs to verify the cause of this crash, so I'll comment it here for someone else to determine.

It's a curious situation here, as I'm not sure what is specifically breaking the queries. I've checked each individual table (using a simple SELECT * LIMIT 10), but have not found anything indicating a problem. If I had more time, I'd try and rebuild a query piece by piece to see what breaks it, but unfortunately I'm playing way too close to some professional deadlines to risk spending any more time on this today.

See P455 for examples of slow running queries for both wikis.

I suspect some relation to T8849

Seems related to some revisions (unidentified) on lgbtawiki AND recent changes on mogaiwiki.

Append ?safemode=1 (or &safemode=1 if there is already a ? in the URL) to the URL to disable the local changes to JS/CSS. From there you should be able to reverse your changes.

This appears to be a bug with the skin. I'd recommend changing the skin you use on your wiki, as other skins do not appear to be having the same problems.

I believe I've fixed it now @Wikicryptos please reopen this task if that is not the case.

All right, seems really broken then, I'll see if I can take a look.

Lowering priority to normal as not all pages are broken.

Per above. Seems possible the values inserted into the table may not match what was needed for this wiki?

RhinosF1 already has PRs opened for this. We are just waiting a bit before merging/deploying them. Thank you though!

Stack trace:

Question: Do we think pointing Parsoid/RESTBase configuration (wgVirtualRestConfig) to localhost might improve the performance of these curl requests? We'll probably have to do some tracing to verify, but my suspicions are that our current config allows for the request to leave the DC, before reentering through a cache proxy (that can't even cache the request), and then being passed on to a MediaWiki server. I might be completely wrong about this, but any thoughts?

In T8511#172061, @Universal_Omega wrote:

This extensions configuration is to advanced/complex to make it customisable via ManageWiki

I decided to run rebuildall.php because if an error prevented it from fully running before, it probably needs to be fully run now to fix other issues. However, I encountered a problem that appears to be Upstream, an instance of https://phabricator.wikimedia.org/T274706 or https://phabricator.wikimedia.org/T231827

In T8507#171804, @Universal_Omega wrote:

Just to note, this could potentially be caused if the wiki was recently imported with the --no-updates option, and without later (successfully) running rebuildall.php or more specifically rebuildtextindex.php. And actually more likely this would be caused if the script was run, but at something rebuildtextindex.php failed. This is because if ALTER TABLE $searchindex DROP INDEX si_title, DROP INDEX si_text from the script is ran then the script fails before ALTER TABLE $searchindex ADD FULLTEXT $field ($field) is ran then the above error would likely be given.

Hmm, this looks like a persistent issue from upstream with cargo:

Also reproduced, and I've pulled the logs:

Stack trace:

Hmm, would it be possible to simply put the values of $wgNamespaceProtection into $wgAvailableRights? Or would this not work because this config would be loaded after everything else that depends on $wgAvailableRights?

oop, meant to leave open as per above

The core problem has been fixed, though the problem raised in T8448 is still unfixed in core. I'm leaving this open so the MediaWiki SRE team can determine which emergency measures can be disabled.

Doing now because of T8448, no guarantee we've fixed all possible problems short of updating.

The security announcement was made about an hour ago. Am I understanding correctly that we've deployed the LS fix and are currently working to upgrade?

Transferred over to the Miraheze account and renewed. DNS settings look appropriate, though let me know if that is not the case. I'll be opening a new task shortly to discuss registrar access and details, as we should definitely replace SPF's details that is currently being used as the contact.

Not sure when labster and I will be in the same place at the same time. If I'm not around, it might be best to have him reach out directly to me (either via email or whatever else he prefers) so we can coordinate somehow.

Thanks for the further analysis Paladox! I admit I've been exceptionally busy since late August, and haven't had the time to work on this, or much of anything really.

The extension in question is FancyBoxThumbs, and the version we are using technically isn't tracked upstream. It's also easier to point folks here instead of sending them out to figure out where they need to open a task, especially when it is not obvious where that is.

Done!

This has now been done, sorry for the delay!

Logs on cp3 show:

I don't personally have time to review current entries (or new ones) until next week. I don't know if we should create a new task for that, but I'm un-assigning for now so someone else can look at it.

I18n should very likely have been rebuilt already. If the issue is still present it may be unrelated to other steps taken so far. It may be worth posting the stack trace to the error on this task (and preferably updating the description with more accurate details to the problem), so that it can be investigated more easily by others, and potentially be referred to upstream for assistance.

I've updated https://staff.miraheze.org/wiki/CSP_Whitelist_Policy and will go live with that if no other changes are necessary.

I just need to make a clarification update to the draft on staffwiki based on my messages above, and then I think it should be good to go. I'll draft that now (locally), so that it can hopefully be good to go sometime tonight after the database maintenance is done.

IMO, not really a security problem. We can require that the creation of a local account requires a login instead of a page view, but I'd rather boot this down to the community before making such a change.

@Slspencer Would you mind creating a new task so that this can be tracked better? Thanks!
EDIT: Probably not necessary, looks like Paladox is handling

This issue appears to be caused by $wgVisualEditorAvailableNamespaces getting set to only custom namespaces (or others). Simply going to Special:ManageWiki/namespaces and toggling the relevant setting seems to fix this.

Seems to be a fairly common issue lately that the Visual Editor edit tabs disappear from wikis where it is enabled. We should look into why this is occurring and see if we can figure out a fix or workaround better than just slapping ?veaction=edit into a URL.

Later redis issues appear unrelated to the original task. I'd suggest we open a new task for whatever this new problem is.

Both cp12 and cp13 OOM'd tonight and didn't restart cleanly. Logs suggest this issue isn't fixed.

Resolved through local hack, for upstream task, see T285504.

The cause for this is the same as https://phabricator.wikimedia.org/T285504, however, we've already fixed that issue for wiki pages. Instead, the problem we are having is caused due to the image URL being set as https://static.miraheze.org/whatever. We need to tell the file repo, (or perhaps make some other change) to strip https from this url and replace it with http so the purge request can be properly proxied to the server.

Standard OVH is done, but not ca.ovh.com or RamNode (though I believe I can do both). Twitter will still need to be done.

Standard OVH is done, but not ca.ovh.com or RamNode (though I believe I can do both). Twitter will still need to be done.