Causing 5xx via mw8, missing thumbnails
CRITICAL: Puppet has 1 failures. Last run 23 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/mnt/mediawiki-static]
https://icinga.miraheze.org/monitoring/service/show?host=mw8&service=mw8%20Puppet
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Resolved | None | T7287 /mnt/mediawiki-static failures on mwx; missing thumbnails in places | ||
| Resolved | Paladox | T7299 Upgrade to gluster 8.5 |
This shouldn't happen since T7134: Puppet cannot remount GlusterFS mount if directory exists
Leaving open+high because no puppet fix and https://icinga.miraheze.org/monitoring/service/show?host=mw8&service=mw8%20Check%20Gluster%20Clients didn't alert.
We need to fix remount and the alert.
Would it be possible to have a puppet check for this and if /mnt/mediawiki-static is unnacessible puppet automatically does umount?
Noting here (for future reference) that @Paladox proposed on IRC to create a script that would run every minute to check whether the mounts are mounted.
Closing as resolved as the gluster upgrade is expected to fix this. If this is not the case and there's another failure this can be reopened.
20:43:43 <icinga-miraheze> PROBLEM - mw9 Check Gluster Clients on mw9 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs'
@Paladox you had asked upstream for a previous OOM error, any chance you could get their help on this one? The other option is adding a cron to restart the glusterfs process periodically, which limits the impact somewhat, but it's not ideal.
I've done:
https://github.com/miraheze/puppet/commit/e73aecab43138f7c554dc6e6a0144c49a5e68281
https://github.com/miraheze/puppet/commit/03b6ffc3ee35d9108783510b561c994a110293ef
Basically a cron that checks if the mount is connected. If not it just simply unmounts and remounts.
Not sure if we can close this as resolved seeing as this is a hack I did but changing priority to normal.
Any objections to resolving this? Or at least lowering it to low if we want to look for a permanent / non hack version?
Moving this to low as Paladox's hack has proven to be sufficient in keeping the mounts mounted. We probably would want to look at a less hacky solution but I don't think this has to be a priority currently.
Closing since this hasn't seemed to happen for awhile I don't think. Do reopen though if there was something else that was wanted for this.