@Reception123: Load is way too high. This looks like a DOS. Can you check mysql logs etc to see what was happening around when it started?
Might be worth restarting mysql to kill connections and see if it resumes.
Description
Description
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Resolved | Reception123 | T7531 503 Backend fetch failed errors lasting a few minutes | ||
| Resolved | Universal_Omega | T7532 GlobalNewFiles disabled as caused too many connections to db11 (CVE-2021-32722) | ||
| Resolved | Universal_Omega | T7533 Consider index on gnf_files tables | ||
| Resolved | Universal_Omega | T7534 Move global new files updates to background | ||
| Declined | Universal_Omega | T7535 Consider use of MediaWiki PoolCounter | ||
| Declined | Universal_Omega | T7542 Create & run a script to recover missing global files data | ||
| Declined | Universal_Omega | T7543 Monitor size of database tables | ||
| Declined | Universal_Omega | T7544 Monitor performance of DB Queries | ||
| Invalid | Universal_Omega | T7545 Ensure data from deleted wikis is purged from global tables |
Event Timeline
Comment Actions
@RhinosF1 I checked cp* and it didn't seem like there were too many queries (other than Cyberpower's bot). Regarding MySQL I'm not sure how safe restarting is and I'd rather not risk it without someone who knows more than me around.
Comment Actions
MySQL error is
2021-06-26 11:53:53 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
which I also saw on Icinga
Comment Actions
and also
2021-06-26 11:51:56 1127084718 [Warning] Aborted connection 1127084718 to db: 'commonswiki' user: 'mediawiki' host: '2001:41d0:800:1bbd::8' (Got an error reading communication packets)
Comment Actions
It's mw10. Though that's just an example of an error, I'm sure there's other errors with other mw*s
Comment Actions
Connections shouldn't be that high. Can you see what queries are being executed via bin logs?
Comment Actions
GlobalNewFiles was causing a large amount of processes on db11. Disabled for the time being and connections have gone down so I hope things should stay calm.
Comment Actions
Suggestion from Martin Urbanec is to move it to a job rather than on load assuming it is actually caused by people moving files too quick. He's sent a PR to include METHOD so it's easier to debug
Comment Actions
Moving down to normal as the incident is now over and what is left to do is to modify GlobalNewFiles in a way that will avoid someone moving a lot of files from doing this to the database.
Comment Actions
I will re-enable the extension once my patches are tested and merged. (And it is deemed safe again)