Page MenuHomeMiraheze
Create Task
Maniphest T6859

Huge rise in Redis server error: Could not insert <number> <job-name> job(s).
Closed, ResolvedPublic
Actions

Description

islamicdroos.miraheze.org

I get this message

https://graylog.miraheze.org/search?q=mediawiki_exception_class%3AJobQueueError&rangetype=relative&streams=5f8c6fd446640840f104b0ba&relative=7200

Event Timeline

Emad4ald created this task.Feb 17 2021, 07:35
Herald added subscribers: Unknown Object (User), RhinosF1. · View Herald TranscriptFeb 17 2021, 07:35
RhinosF1 triaged this task as High priority.Feb 17 2021, 07:50
RhinosF1 added projects: Production Error, MediaWiki (SRE).

I'll look now

Herald added a subscriber: Reception123. · View Herald TranscriptFeb 17 2021, 07:51
Emad4ald added a comment.Feb 17 2021, 07:54

This issue appear only for me

RhinosF1 raised the priority of this task from High to Unbreak Now!.Feb 17 2021, 07:59

There's a huge rise in redis errors starting 07:12

RhinosF1 renamed this task from Cannot change language in preferences to Huge rise in Redis server error: Could not insert <number> <job-name> job(s)..Feb 17 2021, 08:01
RhinosF1 updated the task description. (Show Details)
RhinosF1 added a project: MediaWiki.
Herald added subscribers: Void, NDKilla. · View Herald TranscriptFeb 17 2021, 08:01
RhinosF1 added a comment.Feb 17 2021, 08:05

jbr3&4:

ERR Error running script (call to f_3b56824083b7c4c3509163c7e76e00904014d92e): @user_script:4: @user_script: 4: -MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.

RhinosF1 added a comment.Feb 17 2021, 08:17

Looks to be out of disk space

Emad4ald added a comment.Feb 17 2021, 08:17

It has been resolved
Thanks Team

RhinosF1 added a comment.Feb 17 2021, 08:30
In T6859#135237, @Emad4ald wrote:

It has been resolved
Thanks Team

I cleared some disk space and the error rates we're seeing are much lower but it's still impacting.

RhinosF1 lowered the priority of this task from Unbreak Now! to High.Feb 17 2021, 08:53
RhinosF1 added a subscriber: Paladox.

Much lower rate but still some. We'll continue working on it. CC @Paladox to see if they know how to get redis working again.

RhinosF1 added a comment.Feb 17 2021, 08:54

Sysadmins: Use https://graylog.miraheze.org/search?q=NOT+mediawiki_exception_file%3A%5C%2Fsrv%5C%2Fmediawiki%5C%2Fw%5C%2Fextensions%5C%2FCargo%5C%2F%2A+AND+mediawiki_exception_class%3AJobQueueError&rangetype=relative&streams=5f8c6fd446640840f104b0ba&relative=7200 and https://graylog.miraheze.org/search?q=mediawiki_host%3Ajobrunner%2A&rangetype=relative&streams=5f8c6fd446640840f104b0ba&relative=7200 to see errors.

They start 07:12

Redmin subscribed.Feb 17 2021, 10:52
Paladox assigned this task to RhinosF1.Feb 17 2021, 15:22
RhinosF1 lowered the priority of this task from High to Normal.Feb 17 2021, 16:05

Only IR left

Unknown Object (User) added a comment.Feb 25 2021, 16:29

@RhinosF1: progress on IR?

RhinosF1 added a comment.Feb 25 2021, 17:07

Will do by end of week

Reception123 closed this task as Resolved.Mar 5 2021, 17:19

No need to keep open any longer since the issue itself is no longer persisting.

@RhinosF1 status on IR?