Uptime: I'd say 95% really at a minimum for all. That's still 1.5 days a month not working in total which is a lot.

Abandoned jobs: not much an idea, we should base on the Prometheus data.

Job Latency: We really should be handling all non expensive jobs within 1 hour of being pushed. I'm pretty sure we do this now.

Errors: Probably rate on grafana higher than 0.4 as we class that as an emergency and use the stalk if Prometheus worked.

Latency: hard to tell with no data but we should probably set it at something like 10% higher than the 30 day average. We could probably alert on this too.

Sorry about the delay here. While not my strong area I'd propose the following:

MediaWiki:
Service uptime: 97% (as RhinosF1 calculates above 95% means 1.5 days/mo not working which is too much to be reasonable IMO, so 97% is better IMO)
Errors: (Exception graph is broken so I'm unable to provide a suggestion here) - https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1&viewPanel=187
Latency: As RhinosF1 says, I find it difficult to make a suggestion here without seeing a graph first.

Jobqueue:
Availability: 97% would be reasonable? (though not with our current issues)
Errors: Without total jobs difficult to propose
Latency: same as above

Memcached:
Availability: 97% still sounds like a good number to me

In T8802#183142, @Reception123 wrote:

Sorry about the delay here. While not my strong area I'd propose the following:

MediaWiki:
Service uptime: 97% (as RhinosF1 calculates above 95% means 1.5 days/mo not working which is too much to be reasonable IMO, so 97% is better IMO)
Errors: (Exception graph is broken so I'm unable to provide a suggestion here) - https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1&viewPanel=187
Latency: As RhinosF1 says, I find it difficult to make a suggestion here without seeing a graph first.

Jobqueue:
Availability: 97% would be reasonable? (though not with our current issues)
Errors: Without total jobs difficult to propose
Latency: same as above

Memcached:
Availability: 97% still sounds like a good number to me

Are cal of there based on current data from Grafana? They need to be graphed historically before agreeing on values, otherwise you could be agreeing to things you either can’t accurately measure or things you can’t achieve.

For memcached at least uptime/availability doesn't seem to be an issue.

In T8802#183519, @Reception123 wrote:

For memcached at least uptime/availability doesn't seem to be an issue.

To formalise it, can you add it to the SLO dashboard listed above so it can be tracked and monitored in Grafana?

I got the prometheus-es-exporter working again today, for this. So we can add exceptions, etc... again, so I can do more work on this a bit later now.

I recommend we remove latency from MediaWiki also. We don't really have a way to monitor it anymore, as I removed blackbox monitoring from MediaWiki as it seemed to make hundreds of thousands of semi-expensive, otherwise unnecessary daily requests, that seemed to actually provide ortherwise unnecessary frontend load and degradation. Backend monitoring was not worth frontend performance degradation in my opinion. I don't see any other way to really monitor latency in this case.

We could store and average the backend response times variable that we post in web requests - this would be a form of inexpensive latency monitoring

@Reception123 @MacFan4000 All the monitoring for this task should be done now, but we need to make sure that the selected numbers here are agreed upon. Any other suggestions for it?

How is the abandoned jobs measurement to work as it’s not clear what percentage of jobs have failed?

The SLO for MediaWiki latency also looks extremely high based on the small data set available at 5s - is there a reason for this?

In T8802#197599, @John wrote:

How is the abandoned jobs measurement to work as it’s not clear what percentage of jobs have failed?

The SLO for MediaWiki latency also looks extremely high based on the small data set available at 5s - is there a reason for this?

For abandoned jobs, I actually just realised that right before I saw your comment. I will fix that a little later. As for latency, that is probably true, the SLO probably doesn't need to be that high. Thank you!

In T8802#197599, @John wrote:

How is the abandoned jobs measurement to work as it’s not clear what percentage of jobs have failed?

The SLO for MediaWiki latency also looks extremely high based on the small data set available at 5s - is there a reason for this?

I have now adjusted the jobs SLO and graph for data we have and to show percentage of abandoned jobs.