In T10617#213713, @BrandonWM wrote:

In T10617#213708, @John wrote:

In T10617#213706, @BrandonWM wrote:

John is pretty much threatening legal action against Miraheze right now

Where have I said this?

@John Your user page on Meta where you stated you were taking action against some users with NDAs. If I’m mistaken, I apologize, but that’s what it seemed to imply.

In T10617#213706, @BrandonWM wrote:

John is pretty much threatening legal action against Miraheze right now

In T10564#212732, @Void wrote:

Even if John is available to refresh the key tomorrow, it is still most likely worth changing the owner of the consumer to a shared staff account, such as the existing Miraheze Operations account.

https://github.com/miraheze/puppet/commit/bedbbf259236895187b13d9dde21e980787117bd temporary solution until we have more disk space to expand.

Change deployed locally

Will take a look over this later tonight

So the idea itself is invalid - backups don't remain locally stored, the server runs out of space, causing the backup to fail.

Could this be why T10434 failed? If so, I think this task is high priority given the timeline left on Feb's SLO reporting

@Paladox half way through Feb, we really need to look into this ASAP

I've just ran that command on puppet141 25 times and the average was 0.007s (max of 0.017s) and 25 times on cloud14 which consistently gave 0.003s or 0.004s (max of 0.005s).

Grafana does not have data to suggest it is slower?

I've looked into this and the metric being used in Grafana was wildly wrong.

For January 2023 SLO Reporting - JobQueue failed the SLO for Errors.

@Collei why did you mark this Dec 2022 task as a duplicate of a Dec 2021 task?

The backups are automatically deleted - backup/dbs was full of backups from early 2022, before the backup system even existed.

The blocker on this task is actually unresolved

I do not - I haven't touched the code in about 2 years and the fact this started to happen immediately after an upgrade suggests something changed and CreateWiki wasn't tested correctly.

Boldly going to mark as declined due to concerns raised above.

https://grafana.miraheze.org/d/pfjAbhf7k/mediawiki-slos?orgId=1&from=now-3h&to=now&viewPanel=22 now shows no data.

A dialogue box if the group contains managewiki rights is a rather easy solution to implement as all the logic would just be checking the group rights that are already exposed for the rights selection pages.

In T10230#206301, @Agent_Isai wrote:

In T10230#206299, @John wrote:

Personally, I think we should just provide a warning rather than restrict the ability entirely - a user can correctly reconfigure the wiki ecosystem and this would then generate a new type of workload of requesting stewards correctly delete a bureaucrat or sysop group thereby changing the problem, not fixing it.

Even better would be to make it so the user can't delete a user group if it would remove *their* access to ManageWiki - this would not be group specific and would solve the problem above while also resolving the core issue.

A Steward would only delete the group if an alternative exists for it. I discussed this with CosmicAlpha who said it would be harder to implement hence why this was suggested as an alternative instead. If anyone wants to give it a try then that'd be most welcome.

Personally, I think we should just provide a warning rather than restrict the ability entirely - a user can correctly reconfigure the wiki ecosystem and this would then generate a new type of workload of requesting stewards correctly delete a bureaucrat or sysop group thereby changing the problem, not fixing it.

New run schedules:

What probably doesn't help is all db servers take backups at the same time. I suggest we stagger these a bit more and then watch the next impact.

This was actually the major reason we didn't have backups for so long - this performance problem.

Suggestions welcome but the way it is is currently the best option.

This has been fixed. This was generating around 1440 failures a day - in order to meet the error threshold with these numbers, we'd need to have sent 144000 emails a day, or 100 a minute. As we don't operate at these volumes, this was always going to be the case.

Availability - having reviewed this, I am certain that the failure here is attributed to two things - one beyond our control and one where we have an open task that is blocked on MediaWiki (SRE) for a resolution.

Re-assigning as the action above identified is one for the MediaWiki team and not Infrastructure

Backup schedules defined:

• Private - weekly
• SSL Keys - weekly
• SQL - fortnightly
• mediawiki-xml - MediaWiki (SRE) - can someone propose a time frame for XML dumps please? - 3 monthly?
• Phabricator Static - fortnightly

In T9478#205493, @Reception123 wrote:

This task doesn't indicate what we'd want to consider as 'high connection' (for warning and critical).