Page MenuHomeMiraheze

MediaWiki servers are overloaded, upgrades likely required
Closed, ResolvedPublic

Description

Miraheze is growing, obviously. Unfortunately that is also causing our MediaWiki servers to be overloaded, making them unusable for a few minutes. We're also risking a single server shutdown/termination by RamNode, which will turn into having the second server being even more overloaded, and then it's time to celebrate the loss of both servers.

I've even read somewhere db2 is executing 126 queries per second, I hope that's a joke.

Here are some Piwik statistics (everything between Aug 2 - Aug 6, should be ignored, the rest are all Miraheze wikis without ATT):


A clear growth can be seen here. Until now we've had 1459 unique visitors today, but remember it's 19:13 now, so there are still 6 hours left before this 'counter' will be reset. And don't forget ATT, which seems to have 600 unique visitors on average (548 now - some of them don't visit the other wikis, though). It seems today is going to become a very busy day!

mw2 has showed high load averages today:

mw1 is even worse:

For mw1, the probleem seems to have started in the end of September:

Today, I've seen load averages as high as 9.50 on mw1, and 5.55 on mw2. As an emergency measure, I've implemented https://github.com/miraheze/puppet/commit/110bbc1ec67af6e6badd17c3819226ed884c6712.

I don't think there's a way to prevent it from growing, so upgrades will be needed. I have a few options:

Stay on the SVZS plan, 1024MB SVZS

  • Give mw1 (combi webserver and jobrunner) a second CPU core (512MB SVZS -> 1024MB SVZS) and 512MB RAM extra. +$3.5/mo
  • Give mw1 and mw2 a second CPU core (512MB SVZS -> 1024MB SVZS) and 512MB RAM extra. +$7/mo

Migrate to 512MB SVZ plan

  • Give mw1 (combi webserver and jobrunner) a second CPU core. Also increase clock speed to 3.30+ GHz. +$1.5/mo
  • Give mw1 and mw2 a second CPU core. Also increase clock speed to 3.30+ GHz. +$3/mo

Migrate to 1024MB SVZ plan

  • Give mw1 (combi webserver and jobrunner) 3 extra CPU cores, and 512MB RAM extra. Also increase clock speed to 3.30+ GHz. +$6.5/mo
  • Give mw1 and mw2 3 extra CPU cores, and 512MB RAM extra. Also increase clock speed to 3.30+ GHz. +$13/mo

Other

  • Migrate mw1 to 1024MB SVZ plan (+512MB RAM, +3 CPU cores), upgrade mw2 to 1024MB SVZS (+1 CPU core, +512MB RAM). +$10/mo
  • Do nothing. +$0/mo

I'm actually voting for the plan to migrate those servers to 2x 512MB SVZ (+$3/mo). That is the cheapest way to at least double the current amount of CPU cores (and get some better single-core performance) with only little cost. We don't need more RAM at the moment anyway.

Thoughts?

Event Timeline

labster added a comment.Oct 30 2016, 18:59

At the very least, we need to migrate mw1 to an SVZ node IMO. The mediawiki servers are always going to be CPU-limited, which makes them better suited to SVZ plans than SVZS plans anyway.

Be careful when migrating mw1 and mw2. They have stuff in homedirs (that should be backupped at the very least), and a deb of mediawiki-math-texvc (which has been removed from the Debian archive).

Can we decide to move mw2 as well? Or only mw1 for now?

MacFan4000 raised the priority of this task from High to Unbreak Now!.Oct 30 2016, 20:40

Unbreak now priority as it needs to be fixed asap. Affects site performance.

This comment was removed by MacFan4000.
John added a subscriber: John.

SVZ for both is the best.

@MacFan4000 also please don't add projects which are irrelevant, it doesn't affect or interesting monitoring, varnish or nginx, it also isnt blocked on an upstream team.

labster added a comment.Oct 30 2016, 21:50

Right. Replacing/upgrading both mw1 and mw2 with 512 SVZ is my vote as well. I think we're all on the same page here, so let's go ahead and implement it.

Made a PR to add a global notice

NDKilla added a comment.Oct 31 2016, 01:58

@Southparkfan @John
I'm pretty sure everyone that matters is in agreement that both servers should be migrated to 512 MB SVZ. As I believe this involves getting new servers (not just upgrading current plans) I'm going to leave this for one of you two to do ASAP with approval from everyone.

Cool, let's go ahead and upgrade those two servers.

As I said:

Be careful when migrating mw1 and mw2. They have stuff in homedirs (that should be backupped at the very least), and a deb of mediawiki-math-texvc (which has been removed from the Debian archive).

This should be done properly. Everything needed is puppetized, but it's not just an 'install-and-repool' thing.

There are two options here:

  1. Purchase those servers as mw3 and mw4. Install them on our own
  2. For each server, depool and let RamNode deal with the migration.

Assuming RamNode could do 2, leaning towards that. Otherwise let's do 1.

Southparkfan closed this task as Resolved.Oct 31 2016, 18:49
Southparkfan claimed this task.

Both servers have been migrated by RamNode without any downtime.