Page MenuHomeMiraheze

cloud14 disk went read-only and is now offline
Closed, ResolvedPublic

Description

(Creating task for tracking)

Cloud14 is offline as the disk went read-only and has now been shutdown completely.

As a result some wikis are offline. With mon141 down that also takes down icinga and all irc bots.
Also with ldap141 down that means we can't use mail or matomo or graylog since we can't sign in.

This is currently being investigated.

An incident report will be needed afterword.

Event Timeline

MacFan4000 triaged this task as Unbreak Now! priority.Nov 16 2022, 16:49
MacFan4000 created this task.

Will the data in db141 be recoverable? Several of my wikis are in that database, and I'd hate to lose all the hard work I've put into them.

Will the data in db141 be recoverable? Several of my wikis are in that database, and I'd hate to lose all the hard work I've put into them.

We can't assert anything at this moment but rest assured we are working hard to resolve this issue satisfactorily.

Even if things are unrecoverable, thank you for looking into this issue.
I know you are all working hard, so i will hope for the best!

For a list of affected wikis (1400+ in total), see P473.

Paladox lowered the priority of this task from Unbreak Now! to High.
Paladox added a subscriber: Owen.

I've re-installed cloud14 onto working disks.

A initial incident report has been created by @Universal_Omega as a draft. https://meta.miraheze.org/wiki/Special:IncidentReports/54

The broken drives have been sent back to @Owen to decide how we'll proceed. Whether we can recover the data or if it's lost permanently.

Lowering the priority of the task as cloud14 is backup and running.

Assigning to Void as the next steps depend on his analysis.

Void lowered the priority of this task from High to Normal.Dec 24 2022, 03:15

Dropping priority in favor of T10188 for the recovery of db141.

The data has been recovered.
db141 still needs recovery work relating to mysql (T10188)

Does anyone else know of any data from cloud14 that may need to be recovered? @Universal_Omega has mentioned that it might be a good idea to see if the original ldap configuration can be found to compare against the new config, and I can take a look at that once db141 is back up (or someone else takes over that task). If there's nothing else, then this task can probably be closed, and I can reach out to @Owen regarding returning the drives to be replaced.

Oh dang. @JamesMarsdenasPrinceEdward saw her wiki shut down again. It is too terrible for itself to shut down. We need a cloud15 disk to get rid of it and turn it into a db142 wiki

Boldly closing following bringing db142 into production. If anything else from the old cloud14 needs to be recovered, please let me know.