Page MenuHomeMiraheze

/mnt/mediawiki-static failures on mwx; missing thumbnails in places
Closed, ResolvedPublic

Description

Causing 5xx via mw8, missing thumbnails

CRITICAL: Puppet has 1 failures. Last run 23 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/mnt/mediawiki-static]

https://icinga.miraheze.org/monitoring/service/show?host=mw8&service=mw8%20Puppet

Event Timeline

RhinosF1 triaged this task as Unbreak Now! priority.May 11 2021, 18:29
RhinosF1 created this task.
RhinosF1 renamed this task from /mnt/MediaWiki-static failures on mw8; missing thumbnails in places to /mnt/mediawiki-static failures on mw8; missing thumbnails in places.May 11 2021, 18:29
RhinosF1 updated the task description. (Show Details)
RhinosF1 lowered the priority of this task from Unbreak Now! to High.May 11 2021, 18:32
RhinosF1 added a project: Monitoring.

Leaving open+high because no puppet fix and https://icinga.miraheze.org/monitoring/service/show?host=mw8&service=mw8%20Check%20Gluster%20Clients didn't alert.

We need to fix remount and the alert.

Would it be possible to have a puppet check for this and if /mnt/mediawiki-static is unnacessible puppet automatically does umount?

Would it be possible to have a puppet check for this and if /mnt/mediawiki-static is unnacessible puppet automatically does umount?

T7134 - It does. That didn't happen.

Noting here (for future reference) that @Paladox proposed on IRC to create a script that would run every minute to check whether the mounts are mounted.

RhinosF1 raised the priority of this task from High to Unbreak Now!.May 15 2021, 17:30

See Icinga. Check went off this time but no auto remount. Can someone?

Reception123 lowered the priority of this task from Unbreak Now! to High.May 15 2021, 17:38

Remounted, moving back.

Reception123 assigned this task to Paladox.

Closing as resolved as the gluster upgrade is expected to fix this. If this is not the case and there's another failure this can be reopened.

RhinosF1 renamed this task from /mnt/mediawiki-static failures on mw8; missing thumbnails in places to /mnt/mediawiki-static failures on mwx; missing thumbnails in places.May 26 2021, 19:45
RhinosF1 reopened this task as Open.

20:43:43 <icinga-miraheze> PROBLEM - mw9 Check Gluster Clients on mw9 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs'

@Paladox you had asked upstream for a previous OOM error, any chance you could get their help on this one? The other option is adding a cron to restart the glusterfs process periodically, which limits the impact somewhat, but it's not ideal.

Paladox lowered the priority of this task from High to Normal.May 31 2021, 14:35

Not sure if we can close this as resolved seeing as this is a hack I did but changing priority to normal.

Any objections to resolving this? Or at least lowering it to low if we want to look for a permanent / non hack version?

Reception123 lowered the priority of this task from Normal to Low.Jun 25 2021, 18:16

Moving this to low as Paladox's hack has proven to be sufficient in keeping the mounts mounted. We probably would want to look at a less hacky solution but I don't think this has to be a priority currently.

Closing since this hasn't seemed to happen for awhile I don't think. Do reopen though if there was something else that was wanted for this.