Page MenuHomeMiraheze
Create Task
Maniphest T4995

Redis server keeps failing due to unable to allocate memory
Closed, ResolvedPublic
Actions

Description

Starting 04:08 - 07:46 - Redis was failing as unable to allocate memory leaving us with the below issues. This happened again at 08:30. redis-server was restarted both times but we need to prevent this happening again as it renders us unusable.

---Earlier description---
At 04:20, users started reporting that globally the following action were failing due to session related issues.

"There seems to be a problem with your login session; this action has been canceled as a precaution against session hacking.  Please resubmit the form."

Users can not do the following actions while logged in or log in:

  • log out
  • edit
  • any other logged action

See below for a redis error starting 04:08, let me or Reception know for more information.

Related Objects
Search...

Event Timeline

Olaroll created this task.Dec 18 2019, 04:34
Herald added a subscriber: • Examknow. · View Herald TranscriptDec 18 2019, 04:34
Olaroll updated the task description. (Show Details)Dec 18 2019, 04:38
dross triaged this task as Unbreak Now! priority.Dec 18 2019, 05:37
dross added a project: Site Reliability Engineering.
dross subscribed.

Sitewide login issues. Reputably affecting several well-known users, including self.

Herald added subscribers: CnocBride, Reception123. · View Herald TranscriptDec 18 2019, 05:37
Pmattes subscribed.Dec 18 2019, 06:17
KamovKa52 subscribed.Dec 18 2019, 07:02
RhinosF1 subscribed.Dec 18 2019, 07:35

Thanks for reporting! I’m working with @Reception123 to find a cause now.

Herald removed a subscriber: RhinosF1. · View Herald TranscriptDec 18 2019, 07:35
RhinosF1 subscribed.Dec 18 2019, 07:36
RhinosF1 renamed this task from Unable to log in to Unable to act while logged in.Dec 18 2019, 07:39
RhinosF1 updated the task description. (Show Details)
RhinosF1 added a project: MediaWiki.
Herald added subscribers: Void, NDKilla. · View Herald TranscriptDec 18 2019, 07:39
RhinosF1 renamed this task from Unable to act while logged in to Unable to act while logged in or log in.Dec 18 2019, 07:39
Reception123 added a comment.Dec 18 2019, 07:43

So far I've seen

360:M 18 Dec 2019 07:40:46.030 # Can't save in background: fork: Cannot allocate memory
8:41 AM 360:M 18 Dec 2019 07:40:52.036 * 1 changes in 60 seconds. Saving...

but don't have enough time to investigate further.

RhinosF1 closed this task as Resolved.Dec 18 2019, 07:47
RhinosF1 claimed this task.
RhinosF1 updated the task description. (Show Details)

07:46:02 <Reception123> !log restarted redis-server

RhinosF1 reassigned this task from RhinosF1 to Reception123.Dec 18 2019, 07:47
RhinosF1 reopened this task as Open.Dec 18 2019, 08:25

Reopening, looks to have hit again

RhinosF1 added a comment.Dec 18 2019, 08:27

And @Reception123 has restarted redis-server again

RhinosF1 added a comment.Dec 18 2019, 08:32

Can edit but we've noticed on bgo.miraheze.org that even though https://bgo.miraheze.org/wiki/File:SantasSleigh.png is deleted, mediawiki doesn't seem to think so.

RhinosF1 added a comment.EditedDec 18 2019, 08:33
In T4995#95039, @RhinosF1 wrote:

Can edit but we've noticed on bgo.miraheze.org that even though https://bgo.miraheze.org/wiki/File:SantasSleigh.png is deleted, mediawiki doesn't seem to think so.

REDIS does JobQueue (see 57592e8a26532c52833cfe6f in exception.log)

RhinosF1 merged a task: T4996: Not able to save pages.Dec 18 2019, 08:40
RhinosF1 added a subscriber: Timboliu999.
RhinosF1 removed Reception123 as the assignee of this task.Dec 18 2019, 08:43

Unassigning @Reception123 as not available anymore and unable to implement a more permanent fix or work out root cause.

RhinosF1 renamed this task from Unable to act while logged in or log in to Redis server keeps failing due to unable to allocate memory.Dec 18 2019, 08:45
RhinosF1 updated the task description. (Show Details)
RhinosF1 removed a project: MediaWiki.
J-Josyu subscribed.Dec 18 2019, 08:54

I can't use the social profile feature private message, is it related to this problem?

RhinosF1 added a comment.Dec 18 2019, 09:06
In T4995#95067, @J-Josyu wrote:

I can't use the social profile feature private message, is it related to this problem?

When it fails, there is very little you will be able to do logged in, please state the full error though.

RhinosF1 assigned this task to Southparkfan.Dec 18 2019, 09:37

See IRC

Southparkfan added a comment.Dec 18 2019, 09:38

10:38:22 <+SPF|Cloud> !log install cron on mics2 to restart redis each 15 minutes

RhinosF1 lowered the priority of this task from Unbreak Now! to High.Dec 18 2019, 12:35

Seems stable, Thanks @Reception123 and @Southparkfan for the response today.

Constantly restarting redis doesn't seem like a long term solution so leaving open for further action/review.

and can we get an incident report for this?

RhinosF1 closed subtask T4997: Repair failed file uploads on bgowiki as Resolved.Dec 18 2019, 14:58
Timboliu999 added a comment.Dec 18 2019, 15:19

I'm getting the same message as I got this morning. Unable to save a changed page.

RhinosF1 added a comment.Dec 18 2019, 15:28
In T4995#95101, @Timboliu999 wrote:

I'm getting the same message as I got this morning. Unable to save a changed page.

Can you clear your cache?

Paladox subscribed.Dec 18 2019, 15:33

@Timboliu999 hi, could you log back in through Special:UserLogin? Sorry for inconvenience.

RhinosF1 reassigned this task from Southparkfan to Paladox.Dec 18 2019, 15:54
RhinosF1 added a subscriber: Southparkfan.

had more reports, @Paladox is working on it.

Raising to UBN! to reflect reality.

RhinosF1 raised the priority of this task from High to Unbreak Now!.Dec 18 2019, 15:54
RhinosF1 merged a task: T4999: I can't log in.
RhinosF1 added subscribers: Pfyh, Zppix.
Timboliu999 added a comment.Dec 18 2019, 15:58

Special:UserLogin is not possible.
When I try to logout I get the following message. Log out failed due to session error. Please try again.

RhinosF1 added a comment.Dec 18 2019, 15:59
In T4995#95128, @Timboliu999 wrote:

Special:UserLogin is not possible.
When I try to logout I get the following message. Log out failed due to session error. Please try again.

We're aware and working on it.

Pfyh added a comment.Dec 18 2019, 15:59

@RhinosF1 Adding $wgCookieSecure = false; to the bottom of my localSettings.php file solved the problem. Can you do this please?

Pfyh added a comment.Dec 18 2019, 16:00

@RhinosF1 see: https://www.mediawiki.org/wiki/Topic:Tvx6bgsxmwktdye9

Wedhro subscribed.Dec 18 2019, 16:10
Pfyh added a comment.Dec 18 2019, 16:11

@RhinosF1 Now it's back.

RhinosF1 added a comment.Dec 18 2019, 16:19
In T4995#95133, @Pfyh wrote:

@RhinosF1 Now it's back.

@Paladox managed to find a solution.

Timboliu999 added a comment.Dec 18 2019, 16:21

Great work! I can save pages again. Thanks!

Pfyh added a comment.Dec 18 2019, 16:25

@RhinosF1 and @Paladox thank you so much!

RhinosF1 closed this task as Resolved.Dec 18 2019, 16:29

Seems to be fine now

RhinosF1 merged a task: T5000: All The Tropes: Spam pages and spammer accounts resisted deletion.Dec 18 2019, 21:10
RhinosF1 added a subscriber: Looney_Toons.
RhinosF1 added a comment.Dec 19 2019, 07:02

https://meta.miraheze.org/wiki/Special:IncidentReports/25/

RhinosF1 added a comment.Dec 19 2019, 07:09
In T4995#95164, @RhinosF1 wrote:

https://meta.miraheze.org/wiki/Special:IncidentReports/25/

A few things in the timeline are wrong. Would also be nice for a public more expanded reason than ‘redis had a technical issue’.