Page MenuHomeMiraheze
Create Task
Maniphest T4024

Install elasticsearch
Closed, ResolvedPublic
Actions

Description

db4.miraheze.org is currently using 341G (333G for /srv/mariadb) out of the available 377G, which results in 96% thus far beyond icinga's threshold of CRITICAL disk usage.

Despite db4.miraheze.org costing about half of our budget, very close to $1k/year, this situation is not sustainable. We are working on reducing the amount of empty databases and our schema size, but there is one thing responsible for lots of data usage: MediaWiki's search functionality.

All searchindex.ibd (the actual content of the searchindex table) files take up 13.84GB and all FTS_*.ibd files (see StackExchange) take up 87.30GB, totalling over 101GB. 30.3% of our used space is going to those stupid InnoDB's FULLTEXT search indexes that still don't fully support MediaWiki.

Buying another 200GB VDS for $50/mo just to host those search indexes seems like a giant waste of money. One 1024MB CVZ (200GB disk, $10/mo) and installing Elasticsearch on it reduces disk usage on db4 to a mere 64%, improves MediaWiki search, stops us wasting money on more space and gives us more headroom to work on long-term projects such as utilising InnoDB compression.

Related Objects

Event Timeline

Southparkfan triaged this task as High priority.Jan 18 2019, 19:31
Southparkfan created this task.
Herald added subscribers: revi, CnocBride, Reception123. · View Herald TranscriptJan 18 2019, 19:31
Southparkfan updated the task description. (Show Details)Jan 18 2019, 19:31
Paladox subscribed.Jan 18 2019, 20:12
MacFan4000 subscribed.Jan 19 2019, 12:36
Herald added a project: MacFan4000. · View Herald TranscriptJan 19 2019, 12:36
NDKilla subscribed.Jan 23 2019, 01:53

why dont we already use elasticsearch? is it intensive on some other resource? I've installed and use it on the wiki at work, but it's basically no-traffic, I just wanted better search queries.

Paladox added a comment.Jan 23 2019, 02:04

Ram constraints.

John lowered the priority of this task from High to Low.Feb 7 2019, 19:42
Paladox raised the priority of this task from Low to High.Feb 26 2019, 12:44

Per chat with @Southparkfan.

Southparkfan added a comment.Feb 27 2019, 23:44

We would like to do this shortly to increase disk space on db4. Objections, anyone?

labster added a comment.Feb 28 2019, 00:02

Give it a shot. The current searches don't work well, either.

Paladox added a comment.Feb 28 2019, 15:46

This is really critical now (we missed db4 running out of space).

Southparkfan renamed this task from Evaluate Elasticsearch for search functionality to Install elasticsearch.Mar 1 2019, 00:24
Southparkfan claimed this task.
Southparkfan raised the priority of this task from High to Unbreak Now!.
Herald added a project: Amanda Catherine. · View Herald TranscriptMar 1 2019, 00:24
Herald added a subscriber: AmandaCath. · View Herald Transcript
Southparkfan added a comment.Mar 1 2019, 00:24

Time to do this right now.

Reception123 added a comment.Mar 2 2019, 10:14
In T4024#79457, @Southparkfan wrote:

Time to do this right now.

Any progress? Since this is UBN.

John lowered the priority of this task from Unbreak Now! to High.Mar 2 2019, 11:49
John subscribed.
In T4024#79545, @Reception123 wrote:
In T4024#79457, @Southparkfan wrote:

Time to do this right now.

Any progress? Since this is UBN.

Yes, not possible atm due to upstream resources + planning alternatives.

Southparkfan added a comment.EditedMar 12 2019, 23:08

elasticsearch1.miraheze.org has been bought and test1wiki is actually using CirrusSearch now. We are manually migrating a few wikis now for testing.

Steps for migration (don't change the order!):

  1. Set wmgUseCirrusSearch and wmgDisableSearchUpdate to true for wiki
  2. Run php /srv/mediawiki/w/extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki <wiki>
  3. Set wmgDisableSearchUpdate to false
  4. Run php /srv/mediawiki/w/extensions/CirrusSearch/maintenance/forceSearchIndex.php --skipLinks --indexOnSkip --wiki <wiki>
  5. Run php /srv/mediawiki/w/extensions/CirrusSearch/maintenance/forceSearchIndex.php --skipParse --wiki <wiki>
  6. Finally, set wmgSearchType to true for the wiki
NDKilla added a comment.Mar 16 2019, 14:50

I think this task as-is is is technically resolved. Should we make a new task or rename this task for migrating all wikis to use elastic search? just the largest wikis?

Void subscribed.Apr 8 2019, 17:09

We should either move on this, or figure out another way to get more space on db4, as the disk space has been pretty much stuck in critical for some time now.

Southparkfan closed this task as Resolved.Apr 9 2019, 20:19
Southparkfan mentioned this in T4259: Add elasticsearch support in CreateWiki.
Southparkfan mentioned this in T4260: Migrate all wikis to elasticsearch.

The installation has been done a while ago. Follow up in T4259 and T4260.