Page MenuHomeMiraheze

Production ErrorTag
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

A tag to collate all tasks which have an error log or stack trace attached from Miraheze Production.

Recent Activity

Thu, Feb 25

RhinosF1 added a comment to T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s)..

Will do by end of week

Thu, Feb 25, 17:07 · MediaWiki, MediaWiki (SRE), Production Error
Universal_Omega added a comment to T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s)..

@RhinosF1: progress on IR?

Thu, Feb 25, 16:29 · MediaWiki, MediaWiki (SRE), Production Error

Wed, Feb 17

RhinosF1 lowered the priority of T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s). from High to Normal.

Only IR left

Wed, Feb 17, 16:05 · MediaWiki, MediaWiki (SRE), Production Error
Paladox assigned T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s). to RhinosF1.
Wed, Feb 17, 15:23 · MediaWiki, MediaWiki (SRE), Production Error
RhinosF1 added a comment to T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s)..

Sysadmins: Use https://graylog.miraheze.org/search?q=NOT+mediawiki_exception_file%3A%5C%2Fsrv%5C%2Fmediawiki%5C%2Fw%5C%2Fextensions%5C%2FCargo%5C%2F%2A+AND+mediawiki_exception_class%3AJobQueueError&rangetype=relative&streams=5f8c6fd446640840f104b0ba&relative=7200 and https://graylog.miraheze.org/search?q=mediawiki_host%3Ajobrunner%2A&rangetype=relative&streams=5f8c6fd446640840f104b0ba&relative=7200 to see errors.

Wed, Feb 17, 08:54 · MediaWiki, MediaWiki (SRE), Production Error
RhinosF1 lowered the priority of T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s). from Unbreak Now! to High.

Much lower rate but still some. We'll continue working on it. CC @Paladox to see if they know how to get redis working again.

Wed, Feb 17, 08:53 · MediaWiki, MediaWiki (SRE), Production Error
RhinosF1 added a comment to T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s)..

It has been resolved
Thanks Team

Wed, Feb 17, 08:30 · MediaWiki, MediaWiki (SRE), Production Error
Emad4ald added a comment to T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s)..

It has been resolved
Thanks Team

Wed, Feb 17, 08:17 · MediaWiki, MediaWiki (SRE), Production Error
RhinosF1 added a comment to T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s)..

Looks to be out of disk space

Wed, Feb 17, 08:17 · MediaWiki, MediaWiki (SRE), Production Error
RhinosF1 added a comment to T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s)..

jbr3&4:

ERR Error running script (call to f_3b56824083b7c4c3509163c7e76e00904014d92e): @user_script:4: @user_script: 4: -MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.

Wed, Feb 17, 08:05 · MediaWiki, MediaWiki (SRE), Production Error
RhinosF1 renamed T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s). from Cannot change language in preferences to Huge rise in Redis server error: Could not insert <number> <job-name> job(s)..
Wed, Feb 17, 08:01 · MediaWiki, MediaWiki (SRE), Production Error
RhinosF1 raised the priority of T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s). from High to Unbreak Now!.

There's a huge rise in redis errors starting 07:12

Wed, Feb 17, 07:59 · MediaWiki, MediaWiki (SRE), Production Error
Emad4ald added a comment to T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s)..

This issue appear only for me

Wed, Feb 17, 07:54 · MediaWiki, MediaWiki (SRE), Production Error
RhinosF1 triaged T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s). as High priority.

I'll look now

Wed, Feb 17, 07:51 · MediaWiki, MediaWiki (SRE), Production Error
RhinosF1 added projects to T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s).: Production Error, MediaWiki (SRE).
Wed, Feb 17, 07:50 · MediaWiki, MediaWiki (SRE), Production Error

Tue, Feb 16

Universal_Omega moved T6856: DynamicPageList extension does not working from Backlog to Bugs on the MediaWiki board.
Tue, Feb 16, 15:17 · MediaWiki, Production Error, MediaWiki (SRE), Universal Omega
Universal_Omega edited projects for T6856: DynamicPageList extension does not working, added: MediaWiki; removed Extensions.
Tue, Feb 16, 15:17 · MediaWiki, Production Error, MediaWiki (SRE), Universal Omega
Universal_Omega removed a project from T6856: DynamicPageList extension does not working: Upstream.
Tue, Feb 16, 15:16 · MediaWiki, Production Error, MediaWiki (SRE), Universal Omega
Universal_Omega closed T6856: DynamicPageList extension does not working as Resolved.
Tue, Feb 16, 15:05 · MediaWiki, Production Error, MediaWiki (SRE), Universal Omega
Blackwolfe reopened T6856: DynamicPageList extension does not working as "Open".

As per Omegas request, reopened the task.

Tue, Feb 16, 14:57 · MediaWiki, Production Error, MediaWiki (SRE), Universal Omega
Universal_Omega closed T6856: DynamicPageList extension does not working as Invalid.

I maintain DynamicPageList3 at the moment, I'll fix it tomorrow, but this being an upstream extension bug is not necessarily appropriate for this phabricator. Feel free to create an issue for this at https://github.com/Universal-Omega/DynamicPageList3/issues/new however. Thanks!

Tue, Feb 16, 05:38 · MediaWiki, Production Error, MediaWiki (SRE), Universal Omega
Schwarz added a comment to T6856: DynamicPageList extension does not working.

This is an error in DynamicPageList3. The error message looks like this: [1fd7cd4921c8fba999509db0] 2021-02-16 05:32:36: Fatal exception of type "TypeError" .

Tue, Feb 16, 05:35 · MediaWiki, Production Error, MediaWiki (SRE), Universal Omega
Universal_Omega moved T6856: DynamicPageList extension does not working from Backlog to Deployed Extension Bugs on the Extensions board.
Tue, Feb 16, 05:21 · MediaWiki, Production Error, MediaWiki (SRE), Universal Omega

Sun, Feb 7

Paladox closed T6828: MySQL crashed on db11 2021-02-06 ~20:46 as Resolved.

Published at https://meta.miraheze.org/wiki/Special:IncidentReports/36

Sun, Feb 7, 19:29 · Database, Infrastructure (SRE), Production Error, MediaWiki, MediaWiki (SRE)
John assigned T6828: MySQL crashed on db11 2021-02-06 ~20:46 to Paladox.

Can you do the incident report as the primary responder?

Sun, Feb 7, 19:02 · Database, Infrastructure (SRE), Production Error, MediaWiki, MediaWiki (SRE)
John moved T6828: MySQL crashed on db11 2021-02-06 ~20:46 from Incoming to Short Term on the Infrastructure (SRE) board.
Sun, Feb 7, 19:01 · Database, Infrastructure (SRE), Production Error, MediaWiki, MediaWiki (SRE)

Sat, Feb 6

Dmehus added a comment to T6828: MySQL crashed on db11 2021-02-06 ~20:46.

Thanks for the add, @RhinosF1

Sat, Feb 6, 21:13 · Database, Infrastructure (SRE), Production Error, MediaWiki, MediaWiki (SRE)
RhinosF1 updated the task description for T6828: MySQL crashed on db11 2021-02-06 ~20:46.
Sat, Feb 6, 21:05 · Database, Infrastructure (SRE), Production Error, MediaWiki, MediaWiki (SRE)
RhinosF1 updated subscribers of T6828: MySQL crashed on db11 2021-02-06 ~20:46.
Sat, Feb 6, 21:05 · Database, Infrastructure (SRE), Production Error, MediaWiki, MediaWiki (SRE)
RhinosF1 triaged T6828: MySQL crashed on db11 2021-02-06 ~20:46 as High priority.
Sat, Feb 6, 21:04 · Database, Infrastructure (SRE), Production Error, MediaWiki, MediaWiki (SRE)

Jan 30 2021

Universal_Omega closed T6790: Awful Movies Wiki & Greatest Movies Wiki as Resolved.

Should be fixed now.

Jan 30 2021, 18:48 · MediaWiki (SRE), Universal Omega, Production Error

Jan 21 2021

Reception123 closed T6752: MediaWiki internal error. as Resolved.

I apologize for the inconvenience caused by this error. I've now fixed this for you.

Jan 21 2021, 16:32 · Database, Production Error, MediaWiki
Dmehus added a comment to T6752: MediaWiki internal error..

@Reception123 might be claiming this task.

Jan 21 2021, 16:28 · Database, Production Error, MediaWiki
Universal_Omega added a comment to T6752: MediaWiki internal error..

@Dmehus: We've seen this what feels like a million and one times before. There's a task somehwere I believe upstream.

@Paladox: can you populate the table?

Jan 21 2021, 16:01 · Database, Production Error, MediaWiki
Dmehus added a comment to T6752: MediaWiki internal error..

@RhinosF1 Ah, okay, that's helpful. Thanks, and thanks for updating the projects.

Jan 21 2021, 12:57 · Database, Production Error, MediaWiki
RhinosF1 added a project to T6752: MediaWiki internal error.: Database.

@Dmehus: We've seen this what feels like a million and one times before. There's a task somehwere I believe upstream.

Jan 21 2021, 12:54 · Database, Production Error, MediaWiki
Dmehus moved T6752: MediaWiki internal error. from Backlog to Bugs on the MediaWiki board.
Jan 21 2021, 12:53 · Database, Production Error, MediaWiki
Dmehus raised the priority of T6752: MediaWiki internal error. from Normal to High.

Thank you for reporting this, @Bimagv. A system administrator should run a full trace on this error and either (a) fix it locally or (b) fix it, or request for it to be fixed, upstream, as applicable, within the next day or two.

Jan 21 2021, 12:52 · Database, Production Error, MediaWiki

Jan 13 2021

Dmehus added a comment to T6698: Duplicate log entries being added in DataDump extension.

Yay! Looks like the screenshot helped. I'm glad I pushed you to look into fixing this. I think the fix you implemented was a reasonable one. Thanks! :)

Jan 13 2021, 22:05 · Universal Omega, Production Error, DataDump

Jan 11 2021

Universal_Omega closed T6698: Duplicate log entries being added in DataDump extension as Resolved.

Fixed with https://github.com/miraheze/DataDump/pull/17

Jan 11 2021, 08:04 · Universal Omega, Production Error, DataDump
Universal_Omega added a comment to T6698: Duplicate log entries being added in DataDump extension.

Interesting I don't see that log entry like that.

Hrm, really? On Vector?

Jan 11 2021, 05:58 · Universal Omega, Production Error, DataDump
Dmehus added a comment to T6698: Duplicate log entries being added in DataDump extension.

Interesting I don't see that log entry like that.

Jan 11 2021, 05:52 · Universal Omega, Production Error, DataDump
Universal_Omega added a comment to T6698: Duplicate log entries being added in DataDump extension.

Interesting I don't see that log entry like that.

Jan 11 2021, 05:44 · Universal Omega, Production Error, DataDump
Dmehus reopened T6698: Duplicate log entries being added in DataDump extension as "Open".

@Universal_Omega I'm reopening this task because in reference to:

Jan 11 2021, 05:41 · Universal Omega, Production Error, DataDump

Jan 8 2021

Universal_Omega closed T6698: Duplicate log entries being added in DataDump extension as Invalid.

Closing as invalid as looking at the code there is absolutely nothing that would cause this unless you reload the page a second time, without clicking the "Refresh page" link. https://github.com/miraheze/DataDump/blob/f9dc4f686221606913c207fc94589c5f78723efc/includes/specials/SpecialDataDump.php#L80.

Jan 8 2021, 01:22 · Universal Omega, Production Error, DataDump
Universal_Omega removed a project from T6698: Duplicate log entries being added in DataDump extension: Extensions.
Jan 8 2021, 01:17 · Universal Omega, Production Error, DataDump
Universal_Omega moved T6698: Duplicate log entries being added in DataDump extension from Maintenance to Bugs on the DataDump board.
Jan 8 2021, 01:16 · Universal Omega, Production Error, DataDump
Universal_Omega added a comment to T6698: Duplicate log entries being added in DataDump extension.

I can't seem to reproduce the error either.

Jan 8 2021, 00:45 · Universal Omega, Production Error, DataDump

Jan 7 2021

John closed T6701: Persistent automated wiki closure on active wiki as Resolved.

Also from a look, this is the *only* wiki affected it seems.

Jan 7 2021, 19:53 · CreateWiki, Production Error, Universal Omega
Dmehus updated subscribers of T6701: Persistent automated wiki closure on active wiki.

Adding @Lakelimbo as he was participating in the discussion on Discord and is interested in this task

Jan 7 2021, 19:53 · CreateWiki, Production Error, Universal Omega