Swift object servers' filesystem full
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	OrangeStar
	Jun 21 2023, 14:10

Description

Almost all object servers have run out of space.

Object servers currently out of space:

swiftobject111
swiftobject112
swiftobject113
swiftobject121

Only swiftobject122 still has free space.

swiftobject111

This server's disk space usage skyrocketed past 2023-06-20 22:10 UTC+2. Grafana link.

swiftobject112

This server's disk space usage skyrocketed past 2023-06-20 22:10 UTC+2. Grafana link.

swiftobject113

This server's disk space usage skyrocketed past 2023-06-20 23:00 UTC+2. Grafana link.

swiftobject121

This server has been out of space since 2023-06-04. Grafana link.

Related Objects
Search...

Status	Assigned	Task
Resolved	Paladox	T11007 Swift object servers' filesystem full
Open	Void	T10810 Attempt to fix Swift deleted, thumb, avatars and awards directories
Stalled	None	T11250 Wiki reset

Event Timeline

OrangeStar created this task.Jun 21 2023, 14:10

Herald added subscribers: Agent_Isai, Void. · View Herald TranscriptJun 21 2023, 14:10

OrangeStar merged a task: T11005: Unable to upload images .Jun 21 2023, 14:11

OrangeStar added a subscriber: Karakorum.

OrangeStar mentioned this in T11004: Database errors on "distrowiki.miraheze.org".Jun 21 2023, 14:22

UBN as this affects upload on multiple wikis.

Herald added a subscriber: BrandonWM. · View Herald TranscriptJun 21 2023, 14:44

MacFan4000 added a project: Swift.Jun 21 2023, 14:44

This should be announced at Discord and https://meta.miraheze.org/wiki/Tech:SRE_noticeboard, the later preferably by someone actually on the SRE team. The SRE noticeboard post could be something like:

==Swift outage==
Currently, it is not possible to upload files, create datadumps, or ask for a dump to be imported at Meta's Special:RequestImportDump page. This is because our swift object servers, the servers storing all our wikis' files, have run out of disk space. The Phabricator task tracking the outage is [[phab:T11007|T11007]].

OrangeStar updated the task description. (Show Details)Jun 21 2023, 18:51

OrangeStar updated the task description. (Show Details)

OrangeStar updated the task description. (Show Details)Jun 21 2023, 19:12

Redmin subscribed.Jun 21 2023, 19:45

Bob2208 subscribed.Jun 21 2023, 19:51

Void merged tasks: T11010: Can't upload new images, T11009: Why am I getting a permission error whenever I try to upload an image?, Restricted Maniphest Task.Jun 21 2023, 20:28

Void added subscribers: Lionel, Sirlance3000, Timboliu999.

a banner at the top of the pages notifying users of the error/restriction would be nice

Agent_Isai merged a task: T11011: Upload File permissions removed from Wiki.Jun 22 2023, 02:26

Agent_Isai added subscribers: Ertosi, Clarasiir.

Void added a subtask: T10810: Attempt to fix Swift deleted, thumb, avatars and awards directories.Jun 22 2023, 02:32