Page MenuHomeMiraheze

GlusterProject
ArchivedPublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

Tag to identify any tasks that affect gluster.

Recent Activity

Wed, Jan 4

John triaged T10077: Missing files on The Whisperers Wiki as Normal priority.
Wed, Jan 4, 23:09 · Gluster, Swift

Dec 15 2022

Universal_Omega closed T9821: [ACCESS REQUEST] Expanded access for mediawiki-admins, a subtask of T9708: [GOAL] Gluster -> Swift, as Invalid.
Dec 15 2022, 00:32 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Dec 12 2022

Paladox archived Gluster.
Dec 12 2022, 17:05
Paladox closed T9708: [GOAL] Gluster -> Swift as Resolved.

This is now done.

Dec 12 2022, 17:04 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Dec 6 2022

Paladox closed T10077: Missing files on The Whisperers Wiki as Resolved.
Dec 6 2022, 14:01 · Gluster, Swift
Paladox added a comment to T10077: Missing files on The Whisperers Wiki.

Your wiki is private and we have changed it so files are really private now (requires using img_auth) to access them rather then being able to access them over static.miraheze.org anonly.

Dec 6 2022, 13:02 · Gluster, Swift
DrHANK created T10077: Missing files on The Whisperers Wiki.
Dec 6 2022, 07:17 · Gluster, Swift
Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

Status update: We have now fully switched to swift. Gluster will be dismantled from the 11th of December.

Dec 6 2022, 00:51 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Nov 14 2022

Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

Due to the slow disks (sata) on cloud11 we’ve had to change things slightly. We’re migrating gluster from cloud12 to cloud11 onto the slower disks. This has been done for gluster122 but currently gluster121 is in the process of doing it. Swiftobject121 has been setup and after gluster121 is done swiftobject122 will be setup.

Nov 14 2022, 09:42 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Oct 28 2022

Universal_Omega updated the task description for T9708: [GOAL] Gluster -> Swift.
Oct 28 2022, 01:18 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Oct 27 2022

Universal_Omega added a comment to T9708: [GOAL] Gluster -> Swift.

So just to update this task a little bit now that migration is ongoing.

Oct 27 2022, 23:05 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Oct 26 2022

Universal_Omega added a subtask for T9708: [GOAL] Gluster -> Swift: T9821: [ACCESS REQUEST] Expanded access for mediawiki-admins.
Oct 26 2022, 15:35 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Oct 22 2022

John closed T9794: 503 Backend fetch failed (October 3, 2022) as Resolved.
Oct 22 2022, 17:33 · Cloud Infrastructure, Infrastructure (SRE), Gluster, Universal Omega

Oct 18 2022

Void added a comment to T9794: 503 Backend fetch failed (October 3, 2022).

Created and working on https://meta.miraheze.org/wiki/Special:IncidentReports/53, should be published for public viewing once ready.

Oct 18 2022, 01:34 · Cloud Infrastructure, Infrastructure (SRE), Gluster, Universal Omega

Oct 17 2022

Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

Happens if you do "reset" and also:

Oct 17 2022, 17:29 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

Oh and on a plain reboot it sometimes doesn't manage to boot (not able to find a drive). But if you do a cold reboot it works.

Oct 17 2022, 15:00 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

Ok, this is blocked on someone having a look at cloud11. I think it may require manual intervention. But UO said that it keeps coming up with "logical drive not found" and also it keeps having some issues with the drive fro example drive 7 showing faulty then not. Maybe the cable? Or maybe something else? @John could you have a look please as we've spent a month on this and we cannot seem to fix it so maybe you'll know.

Oct 17 2022, 14:59 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Oct 16 2022

MacFan4000 edited projects for T9794: 503 Backend fetch failed (October 3, 2022), added: Infrastructure (SRE), Cloud Infrastructure; removed MediaWiki (SRE), MediaWiki.

While MediaWiki was visibly affected, it had nothing to do with this issue, this was overallocation on Cloud Infrastructure

Oct 16 2022, 21:18 · Cloud Infrastructure, Infrastructure (SRE), Gluster, Universal Omega

Oct 4 2022

MacFan4000 reopened T9794: 503 Backend fetch failed (October 3, 2022) as "Open".

Reopening and lowering prio pending an incident report.

Oct 4 2022, 15:59 · Cloud Infrastructure, Infrastructure (SRE), Gluster, Universal Omega
Universal_Omega added a comment to T9794: 503 Backend fetch failed (October 3, 2022).

Given how long the visible outage/issues lasted there should probably be an incident report.

Oct 4 2022, 15:45 · Cloud Infrastructure, Infrastructure (SRE), Gluster, Universal Omega
MacFan4000 added a comment to T9794: 503 Backend fetch failed (October 3, 2022).

Given how long the visible outage/issues lasted there should probably be an incident report.

Oct 4 2022, 12:54 · Cloud Infrastructure, Infrastructure (SRE), Gluster, Universal Omega
Void closed T9794: 503 Backend fetch failed (October 3, 2022) as Resolved.

Here's some details on the problem:

Oct 4 2022, 04:45 · Cloud Infrastructure, Infrastructure (SRE), Gluster, Universal Omega
Universal_Omega added a comment to T9794: 503 Backend fetch failed (October 3, 2022).

Yes, Gluster is definitely at lest a part of what is causing issues.

Oct 4 2022, 01:09 · Cloud Infrastructure, Infrastructure (SRE), Gluster, Universal Omega
MacFan4000 added a project to T9794: 503 Backend fetch failed (October 3, 2022): Gluster.

it was said that gluster maybe causing some of the issues

Oct 4 2022, 01:07 · Cloud Infrastructure, Infrastructure (SRE), Gluster, Universal Omega

Oct 1 2022

Paladox added a comment to T9708: [GOAL] Gluster -> Swift.
In T9708#197874, @John wrote:

We also probably want to adjust the load check for the object servers (should have done this to gluster) since the high load is expected because of the I/o. We need to figure out what is a good number. It doesn't seem to use a lot of cpu where as the proxies do (we may have to increase it to 4 cores or even 6).

Fine tuning is stuff we need to do once in production - fine tuning without both a working model and real traffic is a pointless exercise esp for monitoring and determining resources (when we've based this already on doc recommendations)

Oct 1 2022, 16:40 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
John added a comment to T9708: [GOAL] Gluster -> Swift.

We also probably want to adjust the load check for the object servers (should have done this to gluster) since the high load is expected because of the I/o. We need to figure out what is a good number. It doesn't seem to use a lot of cpu where as the proxies do (we may have to increase it to 4 cores or even 6).

Oct 1 2022, 16:18 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

We also probably want to adjust the load check for the object servers (should have done this to gluster) since the high load is expected because of the I/o. We need to figure out what is a good number. It doesn't seem to use a lot of cpu where as the proxies do (we may have to increase it to 4 cores or even 6).

Oct 1 2022, 15:37 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Sep 18 2022

Paladox updated the task description for T9708: [GOAL] Gluster -> Swift.
Sep 18 2022, 19:07 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
John assigned T9708: [GOAL] Gluster -> Swift to Paladox.
Sep 18 2022, 18:51 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Aug 28 2022

Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

We made the above calculations with https://platform.swiftstack.com/docs/admin/hardware.html. RAM will be done on a trial and error bases.

Aug 28 2022, 18:12 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

So we’ve decided resource wise:

Aug 28 2022, 18:11 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
John lowered the priority of T9708: [GOAL] Gluster -> Swift from High to Normal.
Aug 28 2022, 15:41 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
Paladox triaged T9708: [GOAL] Gluster -> Swift as High priority.
Aug 28 2022, 15:41 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
John moved T9708: [GOAL] Gluster -> Swift from Incoming to Goals on the Infrastructure (SRE) board.
Aug 28 2022, 15:40 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
John moved T9708: [GOAL] Gluster -> Swift from Backlog to Infrastructure on the Goal-2022-Jul-Dec board.
Aug 28 2022, 15:40 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
John added a project to T9708: [GOAL] Gluster -> Swift: Gluster.
Aug 28 2022, 15:40 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Aug 15 2022

labster added a comment to T9640: Write incident report for 180 hour intermediate outages.

That's really disappointing. I understand that incidents happen, and a lot of things are out of our control once it's started. But there were no actions identified to take in the future, which means Miraheze learned nothing from a week of partial downtime. I am now significantly more worried about the future of Miraheze than I was when I was getting 503 errors half of the time.

Aug 15 2022, 23:11 · Gluster, Infrastructure (SRE)
Paladox closed T9640: Write incident report for 180 hour intermediate outages as Resolved.

That IR is not for this issue though? That was the MariaDB issue. This one was caused by Gluster.

Aug 15 2022, 21:27 · Gluster, Infrastructure (SRE)
Universal_Omega reopened T9640: Write incident report for 180 hour intermediate outages as "Open".
Aug 15 2022, 05:38 · Gluster, Infrastructure (SRE)
Universal_Omega added a comment to T9640: Write incident report for 180 hour intermediate outages.
Aug 15 2022, 05:34 · Gluster, Infrastructure (SRE)
Paladox closed T9640: Write incident report for 180 hour intermediate outages as Resolved.

https://meta.miraheze.org/wiki/Special:IncidentReports/52

Aug 15 2022, 05:32 · Gluster, Infrastructure (SRE)

Aug 10 2022

Universal_Omega closed T9581: Frequent 503s during past few days, a subtask of T9640: Write incident report for 180 hour intermediate outages, as Resolved.
Aug 10 2022, 21:21 · Gluster, Infrastructure (SRE)
Theguythatwrites reopened T9581: Frequent 503s during past few days, a subtask of T9640: Write incident report for 180 hour intermediate outages, as Open.
Aug 10 2022, 14:43 · Gluster, Infrastructure (SRE)

Aug 9 2022

Universal_Omega closed T9581: Frequent 503s during past few days, a subtask of T9640: Write incident report for 180 hour intermediate outages, as Resolved.
Aug 9 2022, 04:06 · Gluster, Infrastructure (SRE)

Aug 7 2022

John assigned T9640: Write incident report for 180 hour intermediate outages to Paladox.

As best person is position to do this.

Aug 7 2022, 18:14 · Gluster, Infrastructure (SRE)
Universal_Omega added a subtask for T9640: Write incident report for 180 hour intermediate outages: T9581: Frequent 503s during past few days.
Aug 7 2022, 16:37 · Gluster, Infrastructure (SRE)
Universal_Omega triaged T9640: Write incident report for 180 hour intermediate outages as Normal priority.
Aug 7 2022, 16:36 · Gluster, Infrastructure (SRE)

Jun 21 2022

Universal_Omega added a comment to T9427: Puppet failing on gluster*.

Was this fixed by https://github.com/miraheze/puppet/commit/1eb05e3?

Jun 21 2022, 16:19 · Infrastructure (SRE), Gluster
Reception123 closed T9427: Puppet failing on gluster* as Resolved.

Ran puppet on both gluster101 and gluster111 and there were no issues. Not sure what caused this but seems resolved now.

Jun 21 2022, 13:30 · Infrastructure (SRE), Gluster

Jun 20 2022

Universal_Omega triaged T9427: Puppet failing on gluster* as High priority.
Jun 20 2022, 17:55 · Infrastructure (SRE), Gluster