Page MenuHomeMiraheze

GlobalNewFiles disabled as caused too many connections to db11 (CVE-2021-32722)
Closed, ResolvedPublic

Description

@Reception123: Load is way too high. This looks like a DOS. Can you check mysql logs etc to see what was happening around when it started?

Might be worth restarting mysql to kill connections and see if it resumes.

Event Timeline

@RhinosF1 I checked cp* and it didn't seem like there were too many queries (other than Cyberpower's bot). Regarding MySQL I'm not sure how safe restarting is and I'd rather not risk it without someone who knows more than me around.

MySQL error is

2021-06-26 11:53:53 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)

which I also saw on Icinga

and also

2021-06-26 11:51:56 1127084718 [Warning] Aborted connection 1127084718 to db: 'commonswiki' user: 'mediawiki' host: '2001:41d0:800:1bbd::8' (Got an error reading communication packets)

That's weird. What's that ip resolve too

It's mw10. Though that's just an example of an error, I'm sure there's other errors with other mw*s

Connections shouldn't be that high. Can you see what queries are being executed via bin logs?

GlobalNewFiles was causing a large amount of processes on db11. Disabled for the time being and connections have gone down so I hope things should stay calm.

Suggestion from Martin Urbanec is to move it to a job rather than on load assuming it is actually caused by people moving files too quick. He's sent a PR to include METHOD so it's easier to debug

Reception123 renamed this task from 26 June Incident to GlobalNewFiles caused too many connections to db11.Jun 26 2021, 13:13
Reception123 changed the visibility from "Custom Policy" to "Public (No Login Required)".
Reception123 changed the edit policy from "Custom Policy" to "All Users".
Reception123 lowered the priority of this task from High to Normal.Jun 26 2021, 13:56

Moving down to normal as the incident is now over and what is left to do is to modify GlobalNewFiles in a way that will avoid someone moving a lot of files from doing this to the database.

RhinosF1 renamed this task from GlobalNewFiles caused too many connections to db11 to GlobalNewFiles disabled as caused too many connections to db11.Jun 26 2021, 14:41
RhinosF1 added a project: GlobalNewFiles.
RhinosF1 renamed this task from GlobalNewFiles disabled as caused too many connections to db11 to GlobalNewFiles disabled as caused too many connections to db11 (CVE-2021-32722).Jun 28 2021, 20:19

I will re-enable the extension once my patches are tested and merged. (And it is deemed safe again)

Fixes deployed; re-enabled.