Page MenuHomeMiraheze

Error: 503 Backend fetch failed
Closed, ResolvedPublic

Description

This incident holded on for some minutes. Text of the error message:

503 Backend fetch failed

Our servers are having problems at the moment. Please try again in a few minutes. There may be additional information on our Facebook and Twitter pages.

Please provide the details below if you report this error to the system administrators via https://phabricator.miraheze.org:

Error 503 Backend fetch failed, forwarded for 83.247.102.164, 127.0.0.1 (Varnish XID 1377520) via cp1 at Fri, 07 Jul 2017 08:49:25 GMT.

Event Timeline

Kees_Langeveld added a comment.EditedJul 7 2017, 08:53

Then some minutes of normality. Then, again, the error 503:

Our servers are having problems at the moment. Please try again in a few minutes. There may be additional information on our Facebook and Twitter pages.

Please provide the details below if you report this error to the system administrators via https://phabricator.miraheze.org:

Error 503 Backend fetch failed, forwarded for 83.247.102.164, 127.0.0.1 (Varnish XID 1737850) via cp1 at Fri, 07 Jul 2017 08:52:38 GMT.

Reception123 triaged this task as High priority.
Reception123 assigned this task to Southparkfan.

Way less than before but still happening (can confirm).

And after one of two minutes back to normal.

Yes, issues are still intermittent, but still should be looked at.

Reception123 lowered the priority of this task from High to Normal.Jul 12 2017, 08:24

Have not seen any errors for 5 days

Reception123 raised the priority of this task from Normal to High.Jul 15 2017, 08:21

and they're back :(

Had this error on my wiki in the once in the past week. As Reception said the error isn't as common as it was and sometimes the wiki just magically comes back out from the 503 errors. Can be a pain sometimes but manageable.

In ITIL-terms: there is a "problem" underlying and causing the incidents. I too had some 503 errors on yesterday 15th of july. Happily, in one or two minutes Mediawiki turns back to normal functioning.

John lowered the priority of this task from High to Normal.Jul 18 2017, 20:55
John added a subscriber: John.

Happens way to infrequently to warrant more than normal priority. Potentially a capacity issue so a mw3 may be in order to be looked into.

John closed this task as Resolved.Aug 22 2017, 17:08

By virtue of above.

Reception123 reopened this task as Open.Sep 29 2017, 05:09
Reception123 raised the priority of this task from Normal to High.

I have personally not been experiencing this again, but multiple reports from users of this error being frequent again. @John said something about there not being a balance between mw1 and mw2, but I'm not sure how such balance can be achieved.

John added a comment.Sep 29 2017, 07:51

I don't know how varnish works but it naturally balances itself well. Unfortunately it's not aware of a fairly heavy process on mw1 which means varnish RR won't work.

Unsure if varnish has a weighted system.

Parzley added a subscriber: Parzley.Oct 4 2017, 11:19

I've been experiencing this for well over an hour today when attempting to access my own wiki sea.miraheze.org, or the meta wiki. The error message contains:

Error 503 Backend fetch failed, forwarded for 88.202.151.90, 127.0.0.1 (Varnish XID 3706769) via cp1 at Wed, 04 Oct 2017 11:16:23 GMT.
MacFan4000 raised the priority of this task from High to Unbreak Now!.Oct 4 2017, 12:15
MacFan4000 added a subscriber: MacFan4000.

site is down so UBN

MacFan4000 removed Southparkfan as the assignee of this task.Oct 4 2017, 12:56
revi added a comment.Oct 4 2017, 13:12

We are aware of the situation and we think we identified the cause (puppet1), but we currently have no sysadmins with access to puppet1 available, so we need to wait for one.

Error is due to db2 disk space. The temporary solution is dropping dbs in deleted.dblist.

Parzley added a comment.Oct 4 2017, 13:51

UPDATE: I can now see the wikis :-)

Reception123 closed this task as Resolved.

The permanent 503 issue was resolved by deletion of binary logs on db2
Please note that this and the occasional 503s is unrelated.

Ahmsaqib lowered the priority of this task from Unbreak Now! to Normal.Nov 25 2018, 21:42
MacFan4000 raised the priority of this task from Normal to Unbreak Now!.Nov 25 2018, 22:00