Page MenuHomeMiraheze
Create Task
Maniphest T8350

Redesign backup handling
Closed, ResolvedPublic
Actions

Description

Currently backups are handled by Bacula on RamNode. This requires substantial disk space and consequently cost for a service that is thoroughly unmaintained, relatively unused and rarely accessed or required.

We can produce a significant cost saving (up to 75% from rough calculations) if we utilise OVH's Public Cloud Archive service.

Given our backup policy is off-site and off-main provider, this work is unfortunately blocked on us moving away from OVH for our main service hosting.

Currently with database backups, we keep a live copy on the server for menial work (accidental data deletion or recovery of deleted databases) which covers a theoretical 99% of our backup usage needs. We could adopt a similar policy for other bacula backups (except gluster?) and use OVH's PCA to long term store backup snapshots for things like disaster recovery. We need to maintain bacula's recovery period, but we might be able to beat it in some instances.

Related Objects

Event Timeline

John triaged this task as Low priority.Nov 28 2021, 17:02
John created this task.
Herald added subscribers: Agent_Isai, Unknown Object (User), RhinosF1. · View Herald TranscriptNov 28 2021, 17:02
John moved this task from Incoming to Goals on the Infrastructure (SRE) board.Dec 22 2021, 18:57
John mentioned this in T5044: Setup centralised logging for services.Jan 1 2022, 10:45
Reception123 subscribed.Jan 14 2022, 19:28
John claimed this task.Feb 7 2022, 15:02
John raised the priority of this task from Low to Normal.

Upping to normal as we are looking to decom bacula in the very near future.

Unknown Object (User) unsubscribed.Feb 12 2022, 07:25
John added a comment.Feb 14 2022, 18:03

Started work on this using a python handler for interacting with OVH's PCA via swift.

John added a project: Goal-2022-Jul-Dec.Jun 25 2022, 13:08

Being able to run and support effective backups is looking like we need to reduce existing infrastructure strain. So this is unfortunately blocked on bigger projects in the next few months.

John moved this task from Backlog to Infrastructure on the Goal-2022-Jul-Dec board.Jun 25 2022, 13:09
John added a comment.Nov 25 2022, 14:39

Given recent events, a work around solution will be worked on and released this weekend hopefully.

Herald edited subscribers, added: Void; removed: RhinosF1. · View Herald TranscriptNov 25 2022, 14:39
Rob_Kam subscribed.Nov 25 2022, 14:43
Restricted Repository Identity mentioned this in rPUPCb4f35d28b00b: T8350: Backups using PCA (#3098).Dec 27 2022, 17:36
John added a comment.Dec 27 2022, 17:42
root@puppet141:~/private# /usr/local/bin/miraheze-backup backup private
Starting backup of 'private' for date 2022-12-27...
Completed! This took 8.501368522644043s
root@puppet141:~/private# /usr/local/bin/miraheze-backup backup sslkeys
Starting backup of 'sslkeys' for date 2022-12-27...
Completed! This took 7.49277400970459s

image.png (251×1 px, 30 KB)

Restricted Repository Identity mentioned this in rPUPC6750ddddb104: T8350: Automate private and sslkey backups weekly.Dec 27 2022, 19:03
John added a comment.EditedDec 27 2022, 19:05

Backup schedules defined:

  • Private - weekly
  • SSL Keys - weekly
  • SQL - fortnightly
  • mediawiki-xml - MediaWiki (SRE) - can someone propose a time frame for XML dumps please? - 3 monthly?
  • Phabricator Static - fortnightly
Reception123 added a comment.Dec 27 2022, 19:21

Since we've got SQL backups as well, I'd say every 3 months for XML would be reasonable

Restricted Repository Identity mentioned this in rPUPCc9d0f912ed8e: T8350: Backup Phabricator Static every 2 weeks.Dec 27 2022, 20:20
Restricted Repository Identity mentioned this in rPUPCf18b9496fba7: T8350: Add SQL backups + custom granularity for non-wiki dbs.Dec 27 2022, 20:49
Restricted Repository Identity mentioned this in rPUPCade79443ea6b: T8350: Add customer db-backups for misc on db112.Dec 27 2022, 21:04
Restricted Repository Identity mentioned this in rPUPC2d6f2c13bb3c: T8350: Backup mhglobal weekly.Dec 27 2022, 21:20
Restricted Repository Identity mentioned this in rPUPCd7684e87b47e: T8350: Add 3-monthly full MediaWiki XML backups.Dec 27 2022, 21:31
John closed this task as Resolved.Dec 27 2022, 23:07