The virtual disk of db141 that I recovered from the cloud14 disks has been successfully uploaded. While starting mariadb, it was necessary to set innodb_force_recovery to 3, which is not ideal. I currently have a script running on db141 to check the status of all databases. Some tables appear to be crashing mysql when attempting to access.

Documentation for innodb recovery suggests we should dump the recovered tables and re-import them. The question is where. In theory, we can simply dump each wiki, drop the database, create the database, and import the dump all on db141. On the other hand, it may be preferable to export to db131 (or another db server) and either import them there, or recreate db141 as db142 from scratch and then import the dumps from db131. I think, to minimize the travel of the data, we should attempt to perform the reconstructions on db141, but I would like to hear the thoughts of the rest of the infrastructure team (and any other voices from the community) before we decide on the proper course of action.

For reference, mediawiki should currently see db141 in read-only, and read_only is still set on db141 in mariadb. This should be left on for the time being to prevent writes until we are comfortable with the data integrity. Puppet has also been disabled by me for the duration of this task.