Page MenuHomeMiraheze
Feed Advanced Search

Thu, Jul 1

Southparkfan changed the start date for E273: SPF busy from Thu, Jul 1, 18:00 to Jun 30 2021, 22:00.
Thu, Jul 1, 17:27
Southparkfan created E273: SPF busy.
Thu, Jul 1, 17:27
Southparkfan added a comment to T7467: [Access Request] Void.

Discussed with Void, remaining rights will be granted during the transfer period.

Thu, Jul 1, 17:26 · Site Reliability Engineering

Jun 21 2021

Southparkfan added a comment to T7467: [Access Request] Void.

Twitter done.

Jun 21 2021, 20:10 · Site Reliability Engineering

Jun 18 2021

Southparkfan closed T7448: Renew miraheze.org domain name as Resolved.

Already done.

Jun 18 2021, 18:55 · Site Reliability Engineering
Southparkfan added a comment to T7447: Budget request for SRE Q3+Q4 2021.
CategoryCost
cloud4/5 + cp12/13£183 * 6 months = £1098
cloud3 (old)£83 * 4 months = £332
cloud6 (replacement for cloud3, 1yr contract)£116 * 3 months = £348
IPv4 /29 (for cloud6 + future expansion)£16.50 (one-time)
RamNode (current)(£27 * 6 months) + £12.50 = £174.50
Extra backup space (bacula upgrade)£16 * 6 months = £96
Wildcard certificate renewal£60
Utilities (testing, phone, reserves)£75
Total£2200
Jun 18 2021, 18:53 · Site Reliability Engineering
Southparkfan updated the task description for T7447: Budget request for SRE Q3+Q4 2021.
Jun 18 2021, 18:22 · Site Reliability Engineering
Southparkfan added a comment to T7467: [Access Request] Void.

@Void please provide your Twitter/FB details for access. (paladox already removed from OVH/Twitter/FB)

Jun 18 2021, 18:21 · Site Reliability Engineering

Jun 14 2021

Southparkfan added a comment to T7429: Set wgMiserMode to true on all Miraheze wikis.

As I had mentioned on IRC: a change has user-visible impact (special pages outdated by X hours or disabled functionalities), be careful there. If we want to apply a change for all wikis, it'll require further discussion.

Jun 14 2021, 11:52 · Performance, MediaWiki (SRE), Configuration, Universal Omega
Southparkfan added a comment to T7467: [Access Request] Void.

+1

Jun 14 2021, 11:17 · Site Reliability Engineering

Jun 12 2021

Southparkfan updated the task description for T7447: Budget request for SRE Q3+Q4 2021.
Jun 12 2021, 15:22 · Site Reliability Engineering
Southparkfan lowered the priority of T7352: Significant drop in cache hit rate caused all wikis to go down about 2am UK time 24 May 2021 from High to Normal.

Those actionables sound fine to me.

Thanks.

Jun 12 2021, 15:20 · Infrastructure (SRE)
Southparkfan claimed T7447: Budget request for SRE Q3+Q4 2021.
Jun 12 2021, 15:18 · Site Reliability Engineering
Southparkfan triaged T7448: Renew miraheze.org domain name as Normal priority.
Jun 12 2021, 15:18 · Site Reliability Engineering
Southparkfan triaged T7447: Budget request for SRE Q3+Q4 2021 as Normal priority.
Jun 12 2021, 15:18 · Site Reliability Engineering
Southparkfan updated the event description for E269: Southparkfan away.
Jun 12 2021, 14:41
Southparkfan created E269: Southparkfan away.
Jun 12 2021, 14:40
Southparkfan added a comment to T7352: Significant drop in cache hit rate caused all wikis to go down about 2am UK time 24 May 2021.

cp12 and cp13 were overwhelmed due to the increased connection & request rate. Not unexpected, given the low resources on these VMs, although we could do better. The access logs are gone, so we cannot determine the exact culprit of this load. (T5044)

Jun 12 2021, 12:32 · Infrastructure (SRE)

Jun 9 2021

Southparkfan added a comment to T7434: Database errors .

One of the exceptions: 2021-06-09 17:04:03.233 +00:00 /w/index.php?title=Special:Import&action=submit Wikimedia\\Rdbms\\DBTransactionStateError from line 1501 of /srv/mediawiki/w/includes/libs/rdbms/database/Database.php: Cannot execute query from MediaWiki::preOutputCommit while transaction status is ERROR

Jun 9 2021, 21:45 · Production Error, MediaWiki (SRE), MediaWiki, Database
Southparkfan added a comment to T7434: Database errors .

Lots of PHP warnings here:

mediawiki_exception_message	count()
PHP Warning: XMLReader::read(): <siteinfo>	15
PHP Warning: XMLReader::read(): ^	15
PHP Warning: XMLReader::read(): Load Data before trying to read	6
PHP Warning: XMLReader::read(): </page>	4
PHP Warning: XMLReader::read(): ^	3
PHP Warning: XMLReader::read(): ^	1
PHP Warning: XMLReader::read(): '''ex 3.)''	1
PHP Warning: XMLReader::read(): ^	1
Jun 9 2021, 21:35 · Production Error, MediaWiki (SRE), MediaWiki, Database
Southparkfan added a comment to T7352: Significant drop in cache hit rate caused all wikis to go down about 2am UK time 24 May 2021.

According to https://wm-bot.wmcloud.org/browser/index.php?start=05%2F24%2F2021&end=05%2F24%2F2021&display=%23miraheze-sre, issues started around 02:00 UTC

Jun 9 2021, 21:06 · Infrastructure (SRE)
Southparkfan added a comment to T7420: dreamsit.com.br is pointing to cp11 which has been decommissioned.
In T7420#148537, @Pfyh wrote:

@RhinosF1 , can you suggest me what values ​​i should put for:

Type: CNAME
Name: ?
Value: ?

Jun 9 2021, 20:20 · Infrastructure (SRE)

Jun 6 2021

Southparkfan created E268: Southparkfan away.
Jun 6 2021, 17:31

Jun 4 2021

Southparkfan added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.
  • randomise cron times (e.g. cloud4 on first Sunday of month, cloud5 on first Tuesday of month)
Jun 4 2021, 21:18 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance

Jun 1 2021

Southparkfan added a comment to T5877: Revise MariaDB backup strategy.

The latency between db and dbbackup causes the slowness in the dump process. Moving the dbbackup VM to NL should improve the performance, but NL is much closer to UK than the US is. A disaster impacting both UK and NL is not very likely, but still...

Jun 1 2021, 19:06 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec
Southparkfan added a comment to T7319: Varnish should clean up ramdisk after failure.

Varnish has multiple threads: cache-main is one of them. An OOM'ing cache-main should be marked as a service failure by systemd, but if it isn't, it could explain some of the symptoms.

Jun 1 2021, 19:04 · Infrastructure (SRE), Varnish

May 31 2021

Southparkfan added a comment to T7278: Help with migration.

@Dmehus didn't you have contact regarding an XML import?

May 31 2021, 19:09 · MediaWiki, MediaWiki (SRE)
Southparkfan added a comment to T7382: Phabricator is no longer maintained - decide what to do.

Maintaining a Phabricator fork on our own is unsustainable. On the other hand, Phabricator is essential to us and migrating to different software is a non-trivial task. If Wikimedia decides to maintain a fork, I think we could join them.

May 31 2021, 19:05 · Universal Omega, Site Reliability Engineering, Phabricator
Southparkfan moved T7382: Phabricator is no longer maintained - decide what to do from Radar to Discussion on the Site Reliability Engineering board.
May 31 2021, 19:04 · Universal Omega, Site Reliability Engineering, Phabricator
Southparkfan edited projects for T7382: Phabricator is no longer maintained - decide what to do, added: Site Reliability Engineering; removed Infrastructure (SRE).
May 31 2021, 19:03 · Universal Omega, Site Reliability Engineering, Phabricator
Southparkfan added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.
  • randomise cron times (e.g. cloud4 on first Sunday of month, cloud5 on first Tuesday of month)
May 31 2021, 19:02 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Southparkfan assigned T7388: Decom cp10/cp11 to Paladox.
May 31 2021, 19:01 · Infrastructure (SRE)
Southparkfan lowered the priority of T7388: Decom cp10/cp11 from High to Normal.
May 31 2021, 18:59 · Infrastructure (SRE)
Southparkfan moved T7388: Decom cp10/cp11 from Incoming to Short Term on the Infrastructure (SRE) board.
May 31 2021, 18:59 · Infrastructure (SRE)
Southparkfan closed T7323: New Server Resource Request for Cache Proxies as Resolved.

T7388 for decom.

May 31 2021, 18:59 · Varnish, Infrastructure (SRE)

May 30 2021

Southparkfan added a comment to T7352: Significant drop in cache hit rate caused all wikis to go down about 2am UK time 24 May 2021.

Still blocked on internal issues. Backed up various logs from cp13 in my home dir.

May 30 2021, 18:43 · Infrastructure (SRE)
Southparkfan added a comment to T6471: section.execute-GitInfo->getHeadCommitDate is very slow.

@Southparkfan I do not think we can backport that patch do the change to includes/GitInfo.php, unless the script would work without that change, in which case we could.

Not sure the script can be used withour the custom patch.

May 30 2021, 18:39 · MediaWiki (SRE), MediaWiki, Performance
Southparkfan added a comment to T7278: Help with migration.

Update?

I was going to ask you. Can we import this to a blank db to review charsets and stuff?

May 30 2021, 18:37 · MediaWiki, MediaWiki (SRE)
Southparkfan added a comment to T7278: Help with migration.

Update?

May 30 2021, 18:34 · MediaWiki, MediaWiki (SRE)
Southparkfan added a comment to T7287: /mnt/mediawiki-static failures on mwx; missing thumbnails in places.

@Paladox you had asked upstream for a previous OOM error, any chance you could get their help on this one? The other option is adding a cron to restart the glusterfs process periodically, which limits the impact somewhat, but it's not ideal.

May 30 2021, 18:33 · Monitoring, Production Error, MediaWiki (SRE)
Southparkfan added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

@Paladox ionice won't work on a scheduler other than cfq (removed), removing the check is too dangerous since we want to spot raid issues rather sooner than later. Proposal: randomise the run date of the check per cloud server.

May 30 2021, 18:31 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance

May 27 2021

Southparkfan claimed T7352: Significant drop in cache hit rate caused all wikis to go down about 2am UK time 24 May 2021.

There are multiple high priority tasks now, this will have to wait till more time is available.

May 27 2021, 22:51 · Infrastructure (SRE)

May 25 2021

Southparkfan claimed T7352: Significant drop in cache hit rate caused all wikis to go down about 2am UK time 24 May 2021.

I need to debug this.

May 25 2021, 21:12 · Infrastructure (SRE)
Southparkfan edited P403 SQLs / Scripts for 1.36 update.
May 25 2021, 16:33
Southparkfan edited P403 SQLs / Scripts for 1.36 update.
May 25 2021, 16:14
Southparkfan edited P403 SQLs / Scripts for 1.36 update.
May 25 2021, 16:10
Southparkfan edited P403 SQLs / Scripts for 1.36 update.
May 25 2021, 16:06

May 21 2021

Southparkfan committed rPUPCc4753526deb1: prometheus blackbox: add cp13 (authored by Southparkfan).
prometheus blackbox: add cp13
May 21 2021, 16:52
Southparkfan committed R9:4a22d81d5816: cdn: add cp13 (authored by Southparkfan).
cdn: add cp13
May 21 2021, 15:58

May 19 2021

Southparkfan updated the task description for T7323: New Server Resource Request for Cache Proxies.
May 19 2021, 19:33 · Varnish, Infrastructure (SRE)
Southparkfan placed T7323: New Server Resource Request for Cache Proxies up for grabs.

As discussed: one cache proxy in OVHcloud London (as opposed to two), same size as cp12. Either cp10 or cp11 will be reimaged as a test cache proxy, the other freed up IP will be kept for future usage.

May 19 2021, 19:32 · Varnish, Infrastructure (SRE)

May 17 2021

Southparkfan created T7323: New Server Resource Request for Cache Proxies.
May 17 2021, 20:06 · Varnish, Infrastructure (SRE)
Southparkfan moved T7319: Varnish should clean up ramdisk after failure from Incoming to Short Term on the Infrastructure (SRE) board.
May 17 2021, 19:10 · Infrastructure (SRE), Varnish

May 16 2021

Southparkfan added a comment to T7319: Varnish should clean up ramdisk after failure.

To-do: write IR

May 16 2021, 23:00 · Infrastructure (SRE), Varnish
Southparkfan lowered the priority of T7135: Ingest PHP-FPM slowlogs into Graylog from Normal to Low.
May 16 2021, 20:02 · Monitoring, MediaWiki (SRE)
Southparkfan added a comment to T7319: Varnish should clean up ramdisk after failure.

Same issue as T6952. Questions; is ExecStopPost (rPUPC1b7d9bd772c8d7f9a7474e0e909485e0807ca6c9) executed after a failure? And does systemd see the cache-main crash as a failure?

May 16 2021, 19:46 · Infrastructure (SRE), Varnish
Southparkfan triaged T7319: Varnish should clean up ramdisk after failure as Normal priority.
May 16 2021, 19:35 · Infrastructure (SRE), Varnish

May 15 2021

Southparkfan added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

Scheduler has been changed on cloud5 and VMs. We will let this run for a week, after that we can do a week-by-week comparison (https://meta.miraheze.org/m/5r6 is an example).

May 15 2021, 19:49 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Southparkfan changed the visibility for T7317: Check pygments is up to date.
May 15 2021, 19:32 · MediaWiki (SRE), MediaWiki, Security
Southparkfan closed T7317: Check pygments is up to date as Invalid.
May 15 2021, 19:32 · MediaWiki (SRE), MediaWiki, Security
Southparkfan added a comment to T7317: Check pygments is up to date.

https://phabricator.wikimedia.org/T276843#6902629
/srv/mediawiki/extensions/SyntaxHighlight_GeSHi/pygments/VERSION says we have 2.7.4 (a patched version). Patch was deployed in https://github.com/miraheze/mediawiki/pull/1373 (March 22th).

May 15 2021, 19:32 · MediaWiki (SRE), MediaWiki, Security
Southparkfan added a reverting change for rDNS66bd191fe891: Depool cp10 for performance test: rDNS6612519d40c0: Revert "Depool cp10 for performance test".
May 15 2021, 11:45
Southparkfan committed rDNS6612519d40c0: Revert "Depool cp10 for performance test" (authored by Southparkfan).
Revert "Depool cp10 for performance test"
May 15 2021, 11:45

May 14 2021

Southparkfan committed rPUPCf9de8177dadd: dbbackup: correct uid/gid for dbbackup-user on all servers (authored by Southparkfan).
dbbackup: correct uid/gid for dbbackup-user on all servers
May 14 2021, 12:59
Southparkfan added a comment to T4425: Fix all mysql tables that are using latin rather than binary.

https://phabricator.wikimedia.org/T250058#6051265 suggests search will break when converting to binary?

May 14 2021, 12:13 · Infrastructure (SRE)
Southparkfan renamed T4425: Fix all mysql tables that are using latin rather than binary from Fix all mysql tables that are using latin rather then utf8mb4 to Fix all mysql tables that are using latin rather then binary.
May 14 2021, 12:12 · Infrastructure (SRE)
Southparkfan added a comment to T7304: Temporarily remove Wikimedia Incubator from memesincubatorwiki.
[548563a69457f581a732e3ff] /wiki/Special:MultiPageEdit   InvalidArgumentException from line 2493 of /srv/mediawiki/w/includes/libs/rdbms/database/Database.php: Wikimedia\Rdbms\Database::makeList: empty input for field page_title
#0 /srv/mediawiki/w/includes/libs/rdbms/database/Database.php(1966): Wikimedia\Rdbms\Database->makeList(array, integer)
#1 /srv/mediawiki/w/includes/libs/rdbms/database/Database.php(1905): Wikimedia\Rdbms\Database->selectSQLText(array, array, array, string, array, array)
#2 /srv/mediawiki/w/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->select(array, array, array, string, array, array)
#3 /srv/mediawiki/w/includes/libs/rdbms/database/DBConnRef.php(313): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#4 /srv/mediawiki/w/includes/specialpage/QueryPage.php(475): Wikimedia\Rdbms\DBConnRef->select(array, array, array, string, array, array)
#5 /srv/mediawiki/w/includes/specialpage/QueryPage.php(655): QueryPage->reallyDoQuery(integer, integer)
#6 /srv/mediawiki/w/extensions/PageForms/specials/PF_MultiPageEdit.php(47): QueryPage->execute(NULL)
#7 /srv/mediawiki/w/includes/specialpage/SpecialPage.php(600): PFMultiPageEdit->execute(NULL)
#8 /srv/mediawiki/w/includes/specialpage/SpecialPageFactory.php(635): SpecialPage->run(NULL)
#9 /srv/mediawiki/w/includes/MediaWiki.php(307): MediaWiki\SpecialPage\SpecialPageFactory->executePath(Title, RequestContext)
#10 /srv/mediawiki/w/includes/MediaWiki.php(940): MediaWiki->performRequest()
#11 /srv/mediawiki/w/includes/MediaWiki.php(543): MediaWiki->main()
#12 /srv/mediawiki/w/index.php(53): MediaWiki->run()
#13 /srv/mediawiki/w/index.php(46): wfIndexMain()
#14 {main}
May 14 2021, 00:42 · MediaWiki (SRE), Universal Omega, Extensions
Southparkfan added a comment to T7303: Extension:BlockBatch.

This likely will need a security review, unless it's used on at least one Wikimedia wiki. That being said, though the extension is tagged as unmaintained, that in and of itself is not necessarily a problem. This does seem to be a rather simple, uncomplicated, and benign extension, similar to a parser function extension. Additionally, I can see use cases on multiple wikis for local sysops to block spambots en masse, as an example.

May 14 2021, 00:41 · Extensions, Universal Omega, MediaWiki (SRE)

May 13 2021

Southparkfan added a comment to T7073: Install prometheus-es-exporter for prometheus <-> graylog integration.

I could work on adding the metrics to prometheus. Which metrics would you like to collect? (a counter of <this> in unit <that>)

May 13 2021, 20:03 · MediaWiki (SRE), Monitoring
Southparkfan added a comment to T6471: section.execute-GitInfo->getHeadCommitDate is very slow.

GitInfo::getHeadCommitDate tries to fetch the timestamp of the last commit for each extension. This can be very slow:

southparkfan@mw8:/srv/mediawiki/w/extensions/VisualEditor$ time git show -s --format=format:%ct HEAD
1620421442
May 13 2021, 19:59 · MediaWiki (SRE), MediaWiki, Performance
Southparkfan added a comment to T5700: Make RottenLinks more effective at status detection.

Since T7297 is considered to be a duplicate:

MediaWiki offers the HttpRequestFactory class to make HTTP calls in a standardised manner. The class ensures MediaWiki's internal logging features (e.g. 'http' log channel) and configurations settings (e.g. http_proxy) are used upon executing HTTP calls. Instead, RottenLinks uses the curl_ functions directly.
Example code (untested!):

$request = MediaWikiServices::getInstance()->getHttpRequestFactory()->create(
        $url,
	[ 
	'method' => 'HEAD', // return headers only
	'timeout' => $config->get( 'RottenLinksCurlTimeout' ),
		'userAgent' => 'RottenLinks, MediaWiki extension (https://github.com/miraheze/RottenLinks), running on ' . $config->get( 'Server' )
	],
	__METHOD__
)->execute();
return (int)$request->getStatus();

Just to mention, about to do PR for this, but the final return here is not entirely correct. It would be return (int)$request->getStatusValue()->getValue(); instead because ->execute returns a Status instance, which doesn't have getStatus(), so we use getStatusValue() to get an instance of StatusValue and then finally getValue() to get correct http response code from the status message.

Sorry, I messed up my code after rewriting it. You should not chain ->getStatus() after ->execute(). See this example:

$request = MediaWikiServices::getInstance()->getHttpRequestFactory()->create(
        $url,
	[ 
	'method' => 'HEAD', // return headers only
	'timeout' => $config->get( 'RottenLinksCurlTimeout' ),
		'userAgent' => 'RottenLinks, MediaWiki extension (https://github.com/miraheze/RottenLinks), running on ' . $config->get( 'Server' )
	],
	__METHOD__
)
$reqexec = $request->execute();
return (int)$request->getStatus();
May 13 2021, 19:49 · Universal Omega, MediaWiki (SRE), RottenLinks
Southparkfan added a comment to T7127: Add more jobrunner rate tasks to Grafana.

@Reception123 See https://grafana.miraheze.org/d/3L3WYylMz/mediawiki-job-queue?orgId=1&from=now-24h&to=now for the 'insertion rate'. I have not been able to add the 'processing rate'.

May 13 2021, 19:31 · MediaWiki (SRE), Monitoring
Southparkfan added a comment to T7135: Ingest PHP-FPM slowlogs into Graylog.

This does not have high priority. Can be assigned to me if you wish.

May 13 2021, 19:10 · Monitoring, MediaWiki (SRE)
Southparkfan added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

Paladox has changed the scheduler on cloud5. Let's wait for a day to see the impact on I/O performance (regular operations).

May 13 2021, 19:10 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Southparkfan lowered the priority of T7288: Upgrade gluster to 9.2 from Normal to Low.
May 13 2021, 18:49 · Infrastructure (SRE)
Southparkfan moved T7288: Upgrade gluster to 9.2 from Incoming to Long Term on the Infrastructure (SRE) board.
May 13 2021, 18:49 · Infrastructure (SRE)
Southparkfan added a comment to T5700: Make RottenLinks more effective at status detection.

Since T7297 is considered to be a duplicate:

MediaWiki offers the HttpRequestFactory class to make HTTP calls in a standardised manner. The class ensures MediaWiki's internal logging features (e.g. 'http' log channel) and configurations settings (e.g. http_proxy) are used upon executing HTTP calls. Instead, RottenLinks uses the curl_ functions directly.
Example code (untested!):

$request = MediaWikiServices::getInstance()->getHttpRequestFactory()->create(
        $url,
	[ 
	'method' => 'HEAD', // return headers only
	'timeout' => $config->get( 'RottenLinksCurlTimeout' ),
		'userAgent' => 'RottenLinks, MediaWiki extension (https://github.com/miraheze/RottenLinks), running on ' . $config->get( 'Server' )
	],
	__METHOD__
)->execute();
return (int)$request->getStatus();
May 13 2021, 18:20 · Universal Omega, MediaWiki (SRE), RottenLinks
Southparkfan triaged T7297: RottenLinks should use HttpRequestFactory as Normal priority.
May 13 2021, 17:13 · MediaWiki (SRE), Universal Omega, RottenLinks

May 10 2021

Southparkfan added a comment to T7276: [ACCESS REQUEST] Universal Omega for MediaWiki Engineering.

As long as you keep working on your personal development I can support this request. Your help is valuable to Miraheze.

May 10 2021, 22:47 · Universal Omega, Site Reliability Engineering
Southparkfan added a comment to T7278: Help with migration.

Can't we import an XML dump instead and create user accounts (with the 'send temporary password via email' option) beforehand? Importing the SQL dump comes with multiple challenges (need to check schema, character sets, sync CentralAuth and ManageWiki tables with local users and potential exotic config).

May 10 2021, 22:42 · MediaWiki, MediaWiki (SRE)
Southparkfan committed rPUPC9de3dc50ef70: Fix dbbackup::dumper and partially resolve permissions issues (authored by Southparkfan).
Fix dbbackup::dumper and partially resolve permissions issues
May 10 2021, 10:23
Southparkfan committed rPUPC53e2192faa08: First attempt to add sshfs mounts (T5877) (authored by Southparkfan).
First attempt to add sshfs mounts (T5877)
May 10 2021, 10:02
Southparkfan committed rPUPC657404fa3ae6: Add private key for dbbackup-user on servers (T5877) (authored by Southparkfan).
Add private key for dbbackup-user on servers (T5877)
May 10 2021, 09:20

May 9 2021

Southparkfan committed rPUPC5a47dfeec066: Add MariaDB user account for database backups (T5877) (authored by Southparkfan).
Add MariaDB user account for database backups (T5877)
May 9 2021, 20:37
Southparkfan committed rPUPCebf4633610d7: Add base classes for dbbackup dump hosts & storage hosts (T5877) (authored by Southparkfan).
Add base classes for dbbackup dump hosts & storage hosts (T5877)
May 9 2021, 20:22
Southparkfan committed rDNS94d0188b5951: Remove dbbackup2, decom (T5877) (authored by Southparkfan).
Remove dbbackup2, decom (T5877)
May 9 2021, 20:16
Southparkfan committed rPUPCa5b7c02ab8c5: Remove dbbackup2 and remove replication config (T5877) (authored by Southparkfan).
Remove dbbackup2 and remove replication config (T5877)
May 9 2021, 20:11
Southparkfan committed rPUPC67a93dc535ec: dbbackup: give new uid on db hosts too (authored by Southparkfan).
dbbackup: give new uid on db hosts too
May 9 2021, 19:59
Southparkfan committed rPUPC3d7759e5dbb4: dbbackup: give new user a different uid (authored by Southparkfan).
dbbackup: give new user a different uid
May 9 2021, 19:57
Southparkfan committed rPUPC3521efab3659: dbbackup: fix syntax (authored by Southparkfan).
dbbackup: fix syntax
May 9 2021, 19:52
Southparkfan committed rPUPCe50766486f5d: Add base dbbackup module for database backups, remove old configuration (T5877) (authored by Southparkfan).
Add base dbbackup module for database backups, remove old configuration (T5877)
May 9 2021, 19:49
Southparkfan committed rPUPCfc434f57509b: Create dedicated account on servers for database backups (T5877) (authored by Southparkfan).
Create dedicated account on servers for database backups (T5877)
May 9 2021, 19:49
Southparkfan added a comment to T5877: Revise MariaDB backup strategy.

Going to decom dbbackup2 (we'll be using dbbackup1).

May 9 2021, 19:49 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

May 6 2021

Southparkfan updated the task description for T7238: Removal of access for John.
May 6 2021, 14:23 · Site Reliability Engineering
Southparkfan updated the task description for T7238: Removal of access for John.
May 6 2021, 14:18 · Site Reliability Engineering

May 5 2021

Southparkfan committed R9:1f99e83c7c25: Fix configuration for Logstash formatter (authored by Southparkfan).
Fix configuration for Logstash formatter
May 5 2021, 16:28

May 4 2021

Southparkfan added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

On https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592149 it says ionice -c 3 /usr/share/mdadm/checkarray --cron --all

May 4 2021, 16:50 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance

May 3 2021

Southparkfan added a comment to T5877: Revise MariaDB backup strategy.

Test backup: mydumper -G -E -R -v 3 -t 2 -c -L "/home/dbcopy/dbbackup1-mnt/$(date +"%Y%m%d%H%M%S").log" --trx-consistency-only

  • db11
    • Duration: 2095 minutes (34.9 hours)
    • Size: 14 GB
    • Tables: 204,174
  • db12
    • Duration: 1615 minutes (26.9 hours)
    • Size: 26 GB
    • Tables: 156,104
  • db13
    • Duration: 1359 minutes (22.7 hours)
    • Size: 35 GB
    • Tables: 125,530
May 3 2021, 22:42 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec
Southparkfan added a comment to T7238: Removal of access for John.

OVH/RamNode done.

May 3 2021, 18:58 · Site Reliability Engineering
Southparkfan updated the task description for T5044: Setup centralised logging for services.
May 3 2021, 17:54 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun