Not a security issue, original task was fine (T9924). You just needed to wait until we did the configuration on our side.

In T9500#193867, @Universal_Omega wrote:

https://github.com/miraheze/RemovePII/pull/60

https://github.com/miraheze/RemovePII/pull/60

@Dmehus I've discussed this with @Universal_Omega a little and it seems like what we'd need to do is remove mentions from the page text? It doesn't seem possible without deleting all boards but potentially we could attempt to delete all boards which are sent by a user? Though that seems quite complex

In T9366#189952, @John wrote:

It does also now cross my mind that ProxMox is installed on the HDDs of the server, maybe not too relevant here but we might want to look at some work to move it over to the SSDs

It does also now cross my mind that ProxMox is installed on the HDDs of the server, maybe not too relevant here but we might want to look at some work to move it over to the SSDs

The best bet in my opinion is the reboot them during the MW upgrade as users already expect broken.

@Paladox is this something you can facilitate?

In T9366#189949, @RhinosF1 wrote:

Cloud servers can't be done without downtime can they?

Cloud servers can't be done without downtime can they?

It feels like hosts should have been done based on cloud server rather than individually as we need to reboot the physical hosts as well.

Upgraded graylog121

Upgraded phab121, ldap111, bast101, bast121 and mail121.

Upgraded matomo101, prometheus101, mon111 and puppet111.

mw*, mwtask111, test101, and jobchron121 are now done.

In T9366#189724, @RhinosF1 wrote:

I filed the task because Icinga alerted again. It was only released this morning.

And yes they were numerous issues in how the upgrades were done early this morning

I filed the task because Icinga alerted again. It was only released this morning.

I think this would've been done yesterday by Reception123, so just needs a reboot on servers that weren't rebooted. Some major ones had to be because of outage. db* was rebooted, mon111 was rebooted, phab121 was rebooted, a single mw server was (by me) and test101 was. So I think all those are already done.

On a related note post-resolution (after several days' delay): Subsequent conversions to Scribunto/Lua have still led to similar problems on the Tovasala-English pages whose titles begin with "S"; instances of the recently launched {{Find}} module in the {{Entry}} system are causing the Position-component system and rhyme-page links to go awry:

This should now be fixed. Apologies for the issue.

https://github.com/miraheze/ManageWiki/pull/359 should hopefully fix this. The issue is not as severe as I initially thought since autopromote still is functional, it just gets overriden if group is saved again, since the form defaults for the autopromote groups is incorrect.

I am able to reproduce with 100% reproduction. (Every single time)

@John: it says email us and ask if you don't have a sponsor?

Early Warning has been signed up to.

https://www.ncsc.gov.uk/information/cyber-security-information-sharing-partnership--cisp- & https://acdhub.service.ncsc.gov.uk/

In T9061#184224, @Dmehus wrote:

In T9061#184218, @Universal_Omega wrote:

In T9061#184216, @Naleksuh wrote:

I was directed to this task over IRC. It appears to already be closed, and have little relevance to me at all. What is going on here?

You must've been directed to the wrong task, I'd assume? T9071 is probably what they meant to direct you to, I'm assuming, based off conversation I have observed. But that task is currently private.

No, I thought Naleksuh might be interested in the task, so sent him this link.

In T9061#184218, @Universal_Omega wrote:

In T9061#184216, @Naleksuh wrote:

I was directed to this task over IRC. It appears to already be closed, and have little relevance to me at all. What is going on here?

You must've been directed to the wrong task, I'd assume? T9071 is probably what they meant to direct you to, I'm assuming, based off conversation I have observed. But that task is currently private.

In T9061#184216, @Naleksuh wrote:

I was directed to this task over IRC. It appears to already be closed, and have little relevance to me at all. What is going on here?

I was directed to this task over IRC. It appears to already be closed, and have little relevance to me at all. What is going on here?

In T9061#183835, @Samwilson wrote:

I mentioned the CreateRedirect error on its talk page (sorry, I didn't realise this was a hidden security task! I shouldn't've advertised it publicly), and it looks like the issue has been fixed: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CreateRedirect/+/780567

In T9061#184209, @Universal_Omega wrote:

In T9061#184207, @Dmehus wrote:

In T9061#183836, @RhinosF1 wrote:

No problem.

@Dmehus: any issues to making public?

No objections if @John and @Owen have no issues making it public

It already is now.

In T9061#183765, @RhinosF1 wrote:

@Samwilson: It looks like create redirect is at fault. I can move the Main_Page without being able to edit it and the user used that. It looks like you are one of 2 project members. Can you look into this?

In T9061#184207, @Dmehus wrote:

In T9061#183836, @RhinosF1 wrote:

No problem.

@Dmehus: any issues to making public?

No objections if @John and @Owen have no issues making it public

In T9061#183836, @RhinosF1 wrote:

No problem.

@Dmehus: any issues to making public?

As to what might have caused those 500s/502s all along:

In T8866#183960, @Routhwick wrote:

Which leaves only Regex Fun--or Scribunto/Lua--as our only options from here. Too bad it had to come down to this--but for what it's worth, I implemented RegexFunctions on the basis of good faith four months ago (as a means of code curbing). Never expected this trouble to arise just from RgxF, but then again...

Which leaves only Regex Fun--or Scribunto/Lua--as our only options from here. Too bad it had to come down to this--but for what it's worth, I implemented RegexFunctions on the basis of good faith four months ago (as a means of code curbing). Never expected this trouble to arise just from RgxF, but then again...

In T8866#183951, @Routhwick wrote:

In T8866#183941, @Universal_Omega wrote:

I think we should consider permanent removal of RegexFunctions.

But not before bringing up this advice/pro-tip from RegexFunctions developer "Skizzers" himself, which may help me prevent future catastrophes of such like:

RegexFunctions will not block you from using a terrible regex that causes all sorts of backtracking and uses up a ton of resources. Either optimize your regexes or move to a solution like Scribunto (and lua's pattern matching, which is a lot lighter-weight than regex). If you want to go the former route, there is plenty of information online on how to avoid regex patterns that cause excessive backtracking.

Bolded emphasis mine--and for starters, Jan Goyvaerts of RegExp.info has been there before. That said, I'll do some testing of the trouble spot(s) at ExpandTemplates and remind you on how it's shaping up.

In T8866#183941, @Universal_Omega wrote:

I think we should consider permanent removal of RegexFunctions.

Is their a reason this is a security task still? It is not an issue that users can reproduce themselves so see no reason why.

Made public since RegexFunctions was disabled for us, which mitigated the issue for you, so considering this task resolved, as it is less issue now. I think we should consider permanent removal of RegexFunctions.

No objectionw

Is this task ok to be made public?

Extension patched upstream, and updated for us, I will do another full review of the extension, and then hopefully re-enable it.

As possible I strongly encourage interfacing with the local wikis. I suspect a lot of traffic that would benefit from an announcement, simply does not pass through Meta or its CN.

Technical mishap ;)

In T9061#183845, @Reception123 wrote:

My view is that if RedirectManager is very similar to CreateRedirect and does not present the issues that CR does we should replace it.

In T8866#183800, @Universal_Omega wrote:

Hi, can you please see if it is better now?

My view is that if RedirectManager is very similar to CreateRedirect and does not present the issues that CR does we should replace it.

Note: T1140#25572 is original review

@Universal_Omega: I'd appreciate a security review / opinion on the below as this doesn't fill me with great confidence.

I filed https://phabricator.wikimedia.org/T306174 for getting a CVE

No problem.

I mentioned the CreateRedirect error on its talk page (sorry, I didn't realise this was a hidden security task! I shouldn't've advertised it publicly), and it looks like the issue has been fixed: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CreateRedirect/+/780567

Given the risk with the currently layout and how much nicer @Samwilson's looks, if it passed security review, I'd consider replacing CreateRedirect with it.

I'm not a member, just watching that project.

RegexFunctions has been disabled as it's causing OOMs.

@Samwilson: It looks like create redirect is at fault. I can move the Main_Page without being able to edit it and the user used that. It looks like you are one of 2 project members. Can you look into this?

nothing to do with GlobalBlock at all

Complete outage as we saw on 7/8 April has not occurred since database backups were disabled so lowering from UBN to High as this is not currently impacting us anymore

In T9061#183551, @Dmehus wrote:

In T9061#183509, @Reception123 wrote:

The issue will have to be tested on a test wiki by applying a global block to an IP and identifying potential extensions and them being disabled/enabled until we can conclude which one is causing it.

Yeah... I wouldn't prefer to do extension testing on Public Test Wiki, though. This is one of those times when it'd be really still be helpful to have an SRE testing wiki within the existing CentralAuth-linked production wikis.