Discussed with Void, remaining rights will be granted during the transfer period.

Twitter done.

Already done.

@Void please provide your Twitter/FB details for access. (paladox already removed from OVH/Twitter/FB)

As I had mentioned on IRC: a change has user-visible impact (special pages outdated by X hours or disabled functionalities), be careful there. If we want to apply a change for all wikis, it'll require further discussion.

+1

In T7352#148823, @Paladox wrote:

Those actionables sound fine to me.

Thanks.

cp12 and cp13 were overwhelmed due to the increased connection & request rate. Not unexpected, given the low resources on these VMs, although we could do better. The access logs are gone, so we cannot determine the exact culprit of this load. (T5044)

One of the exceptions: 2021-06-09 17:04:03.233 +00:00 /w/index.php?title=Special:Import&action=submit Wikimedia\\Rdbms\\DBTransactionStateError from line 1501 of /srv/mediawiki/w/includes/libs/rdbms/database/Database.php: Cannot execute query from MediaWiki::preOutputCommit while transaction status is ERROR

Lots of PHP warnings here:

mediawiki_exception_message	count()
PHP Warning: XMLReader::read(): <siteinfo>	15
PHP Warning: XMLReader::read(): ^	15
PHP Warning: XMLReader::read(): Load Data before trying to read	6
PHP Warning: XMLReader::read(): </page>	4
PHP Warning: XMLReader::read(): ^	3
PHP Warning: XMLReader::read(): ^	1
PHP Warning: XMLReader::read(): '''ex 3.)''	1
PHP Warning: XMLReader::read(): ^	1

According to https://wm-bot.wmcloud.org/browser/index.php?start=05%2F24%2F2021&end=05%2F24%2F2021&display=%23miraheze-sre, issues started around 02:00 UTC

In T7420#148537, @Pfyh wrote:

@RhinosF1 , can you suggest me what values ​​i should put for:

Type: CNAME
Name: ?
Value: ?

In T7230#147468, @Southparkfan wrote:

• randomise cron times (e.g. cloud4 on first Sunday of month, cloud5 on first Tuesday of month)

The latency between db and dbbackup causes the slowness in the dump process. Moving the dbbackup VM to NL should improve the performance, but NL is much closer to UK than the US is. A disaster impacting both UK and NL is not very likely, but still...

Varnish has multiple threads: cache-main is one of them. An OOM'ing cache-main should be marked as a service failure by systemd, but if it isn't, it could explain some of the symptoms.

@Dmehus didn't you have contact regarding an XML import?

Maintaining a Phabricator fork on our own is unsustainable. On the other hand, Phabricator is essential to us and migrating to different software is a non-trivial task. If Wikimedia decides to maintain a fork, I think we could join them.

• randomise cron times (e.g. cloud4 on first Sunday of month, cloud5 on first Tuesday of month)

T7388 for decom.

Still blocked on internal issues. Backed up various logs from cp13 in my home dir.

In T6471#146147, @Universal_Omega wrote:

@Southparkfan I do not think we can backport that patch do the change to includes/GitInfo.php, unless the script would work without that change, in which case we could.

Not sure the script can be used withour the custom patch.

In T7278#147356, @RhinosF1 wrote:

In T7278#147355, @Southparkfan wrote:

Update?

I was going to ask you. Can we import this to a blank db to review charsets and stuff?

Update?

@Paladox you had asked upstream for a previous OOM error, any chance you could get their help on this one? The other option is adding a cron to restart the glusterfs process periodically, which limits the impact somewhat, but it's not ideal.

@Paladox ionice won't work on a scheduler other than cfq (removed), removing the check is too dangerous since we want to spot raid issues rather sooner than later. Proposal: randomise the run date of the check per cloud server.

There are multiple high priority tasks now, this will have to wait till more time is available.

I need to debug this.

As discussed: one cache proxy in OVHcloud London (as opposed to two), same size as cp12. Either cp10 or cp11 will be reimaged as a test cache proxy, the other freed up IP will be kept for future usage.

To-do: write IR

Same issue as T6952. Questions; is ExecStopPost (rPUPC1b7d9bd772c8d7f9a7474e0e909485e0807ca6c9) executed after a failure? And does systemd see the cache-main crash as a failure?

Scheduler has been changed on cloud5 and VMs. We will let this run for a week, after that we can do a week-by-week comparison (https://meta.miraheze.org/m/5r6 is an example).

https://phabricator.wikimedia.org/T276843#6902629
/srv/mediawiki/extensions/SyntaxHighlight_GeSHi/pygments/VERSION says we have 2.7.4 (a patched version). Patch was deployed in https://github.com/miraheze/mediawiki/pull/1373 (March 22th).

https://phabricator.wikimedia.org/T250058#6051265 suggests search will break when converting to binary?

In T7303#145159, @Dmehus wrote:

This likely will need a security review, unless it's used on at least one Wikimedia wiki. That being said, though the extension is tagged as unmaintained, that in and of itself is not necessarily a problem. This does seem to be a rather simple, uncomplicated, and benign extension, similar to a parser function extension. Additionally, I can see use cases on multiple wikis for local sysops to block spambots en masse, as an example.

In T7073#144421, @Reception123 wrote:

@Southparkfan

I could work on adding the metrics to prometheus. Which metrics would you like to collect? (a counter of <this> in unit <that>)

GitInfo::getHeadCommitDate tries to fetch the timestamp of the last commit for each extension. This can be very slow:

southparkfan@mw8:/srv/mediawiki/w/extensions/VisualEditor$ time git show -s --format=format:%ct HEAD
1620421442

In T5700#145074, @Universal_Omega wrote:

In T5700#145050, @Southparkfan wrote:

Since T7297 is considered to be a duplicate:

MediaWiki offers the HttpRequestFactory class to make HTTP calls in a standardised manner. The class ensures MediaWiki's internal logging features (e.g. 'http' log channel) and configurations settings (e.g. http_proxy) are used upon executing HTTP calls. Instead, RottenLinks uses the curl_ functions directly.
Example code (untested!):

$request = MediaWikiServices::getInstance()->getHttpRequestFactory()->create(
        $url,
	[ 
	'method' => 'HEAD', // return headers only
	'timeout' => $config->get( 'RottenLinksCurlTimeout' ),
		'userAgent' => 'RottenLinks, MediaWiki extension (https://github.com/miraheze/RottenLinks), running on ' . $config->get( 'Server' )
	],
	__METHOD__
)->execute();
return (int)$request->getStatus();

Just to mention, about to do PR for this, but the final return here is not entirely correct. It would be return (int)$request->getStatusValue()->getValue(); instead because ->execute returns a Status instance, which doesn't have getStatus(), so we use getStatusValue() to get an instance of StatusValue and then finally getValue() to get correct http response code from the status message.

Sorry, I messed up my code after rewriting it. You should not chain ->getStatus() after ->execute(). See this example:

$request = MediaWikiServices::getInstance()->getHttpRequestFactory()->create(
        $url,
	[ 
	'method' => 'HEAD', // return headers only
	'timeout' => $config->get( 'RottenLinksCurlTimeout' ),
		'userAgent' => 'RottenLinks, MediaWiki extension (https://github.com/miraheze/RottenLinks), running on ' . $config->get( 'Server' )
	],
	__METHOD__
)
$reqexec = $request->execute();
return (int)$request->getStatus();

@Reception123 See https://grafana.miraheze.org/d/3L3WYylMz/mediawiki-job-queue?orgId=1&from=now-24h&to=now for the 'insertion rate'. I have not been able to add the 'processing rate'.

In T7135#144414, @Reception123 wrote:

@Southparkfan

This does not have high priority. Can be assigned to me if you wish.

Paladox has changed the scheduler on cloud5. Let's wait for a day to see the impact on I/O performance (regular operations).

Since T7297 is considered to be a duplicate:

MediaWiki offers the HttpRequestFactory class to make HTTP calls in a standardised manner. The class ensures MediaWiki's internal logging features (e.g. 'http' log channel) and configurations settings (e.g. http_proxy) are used upon executing HTTP calls. Instead, RottenLinks uses the curl_ functions directly.
Example code (untested!):

$request = MediaWikiServices::getInstance()->getHttpRequestFactory()->create(
        $url,
	[ 
	'method' => 'HEAD', // return headers only
	'timeout' => $config->get( 'RottenLinksCurlTimeout' ),
		'userAgent' => 'RottenLinks, MediaWiki extension (https://github.com/miraheze/RottenLinks), running on ' . $config->get( 'Server' )
	],
	__METHOD__
)->execute();
return (int)$request->getStatus();

As long as you keep working on your personal development I can support this request. Your help is valuable to Miraheze.

Can't we import an XML dump instead and create user accounts (with the 'send temporary password via email' option) beforehand? Importing the SQL dump comes with multiple challenges (need to check schema, character sets, sync CentralAuth and ManageWiki tables with local users and potential exotic config).

Going to decom dbbackup2 (we'll be using dbbackup1).

In T7230#143735, @Paladox wrote:

On https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592149 it says ionice -c 3 /usr/share/mdadm/checkarray --cron --all

Test backup: mydumper -G -E -R -v 3 -t 2 -c -L "/home/dbcopy/dbbackup1-mnt/$(date +"%Y%m%d%H%M%S").log" --trx-consistency-only

• db11
  • Duration: 2095 minutes (34.9 hours)
  • Size: 14 GB
  • Tables: 204,174
• db12
  • Duration: 1615 minutes (26.9 hours)
  • Size: 26 GB
  • Tables: 156,104
• db13
  • Duration: 1359 minutes (22.7 hours)
  • Size: 35 GB
  • Tables: 125,530

OVH/RamNode done.