Page MenuHomeMiraheze
Create Task
Maniphest T5453

Alert when icinga-miraheze disconnects from IRC
Closed, ResolvedPublic
Actions

Description

During the incident today, icinga-miraheze disconnected from IRC.

We should have an alert when this happens so we don’t miss anything important.

Self-Assigning as will handle at some point this week.

Event Timeline

RhinosF1 triaged this task as Normal priority.Apr 21 2020, 13:28
RhinosF1 created this task.
Herald added a subscriber: • Examknow. · View Herald TranscriptApr 21 2020, 13:28
RhinosF1 moved this task from Radar to New Features/Services on the Site Reliability Engineering board.Apr 21 2020, 13:28
John moved this task from New Features/Services to Trivial Puppet Change on the Site Reliability Engineering board.Apr 21 2020, 13:55
• Examknow added a comment.Apr 21 2020, 13:58

SigmaBot may be able to ping staff when icinga-miraheze parts the #miraheze channel. Staff please let me know your thoughts on this.

RhinosF1 added a subscriber: John.Apr 21 2020, 13:59

@John: Is someone going to work on the puppet change?

My plan was to have ZppixBot just watch for when it quits.

Paladox added a subscriber: Paladox.Apr 21 2020, 13:59

We already have redundant monitoring.

RhinosF1 added a comment.Apr 21 2020, 14:00
In T5453#106261, @Examknow wrote:

SigmaBot may be able to ping staff when icinga-miraheze parts the #miraheze channel. Staff please let me know your thoughts on this.

My plan was to have ZppixBot set off an alert with ‘@page’ mentioned so staff can stalk.

RhinosF1 added a comment.Apr 21 2020, 14:01
In T5453#106266, @Paladox wrote:

We already have redundant monitoring.

Did it alert?

John added a comment.Apr 21 2020, 14:05

If this is a task here tagged as monitoring, it’s a puppet change. If it’s not a change you’ll be making in Miraheze, this isn’t the place to track it currently.

RhinosF1 added a comment.Apr 21 2020, 14:06
In T5453#106269, @John wrote:

If this is a task here tagged as monitoring, it’s a puppet change. If it’s not a change you’ll be making in Miraheze, this isn’t the place to track it currently.

It’s probably best out of MH’s network to reduce the chance that it goes down when we do.

John added a comment.Apr 21 2020, 14:17
In T5453#106270, @RhinosF1 wrote:
In T5453#106269, @John wrote:

If this is a task here tagged as monitoring, it’s a puppet change. If it’s not a change you’ll be making in Miraheze, this isn’t the place to track it currently.

It’s probably best out of MH’s network to reduce the chance that it goes down when we do.

This is true about all our monitoring. But we can’t afford such liberties.

RhinosF1 reassigned this task from RhinosF1 to • Examknow.Apr 21 2020, 15:25

The backup monitoring solution will:
Ping !staff when icinga-miraheze disconnects
Pings meta.miraheze.org & icinga.miraheze.org
Reports if the sites are UP or DOWN

All this should happen with seconds of icinga failing (which could be up to 300s after it stops working).

There will also be a manual !status command to check whether key services are off

RhinosF1 closed this task as Resolved.EditedApr 21 2020, 16:24

Complete.

SigmaBot will generate an alert including the ping “!sre” (stalk if you want to) when icinga-miraheze quits. It will run a short status check to indicate whether icinga and meta is up.

A full status check can be run by SRE, me or Examknow using !status but it is long as it checks multiple services so only use if you have to.

• Examknow added a comment.Apr 21 2020, 18:29
In T5453#106310, @RhinosF1 wrote:

Complete.

SigmaBot will generate an alert including the ping “!sre” (stalk if you want to) when icinga-miraheze quits. It will run a short status check to indicate whether icinga and meta is up.

A full status check can be run by SRE, me or Examknow using !status but it is long as it checks multiple services so only use if you have to.

To add to that, if SRE wants a shorter report, they can use !s and the bot will give the total amount of alive and dead services. Just felt like there needed to be a less spamy option for non-emergencies.