Page MenuHomeMiraheze

Paladox
Site Reliabilty Engineer

Projects

User does not belong to any projects.

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 14 2017, 11:40 (198 w, 4 d)
Availability
Available
GitHub User
Unknown
Miraheze User
Paladox [ Global Accounts ]

I live in the United Kingdom GMT/BST(UTC+0, UTC+1).

I am no longer a miraheze volunteer.

Recent Activity

Jul 4 2021

Paladox updated Paladox.
Jul 4 2021, 15:19

Jun 21 2021

PiscesKazeMGR awarded T7382: Phabricator is no longer maintained - decide what to do a Party Time token.
Jun 21 2021, 09:30 · Universal Omega, Site Reliability Engineering, Phabricator

Jun 15 2021

Paladox removed a member for DNS: Paladox.
Jun 15 2021, 19:10
Paladox removed a member for SSL: Paladox.
Jun 15 2021, 19:09
Paladox removed a member for Upstream: Paladox.
Jun 15 2021, 19:09
Paladox removed a member for Services: Paladox.
Jun 15 2021, 19:09
Paladox removed a member for DataDump: Paladox.
Jun 15 2021, 19:09
Paladox removed a member for Cloud Infrastructure: Paladox.
Jun 15 2021, 19:09

Jun 14 2021

Paladox added a comment to T7472: Extension:Lingo is incompatible with the current MediaWiki version.

Fixed with https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Lingo/+/699562

Jun 14 2021, 20:25 · Universal Omega, Extensions, Upstream, MediaWiki (SRE)
Paladox placed T4302: Deploy Apache Traffic Server up for grabs.
Jun 14 2021, 18:59 · Infrastructure (SRE)
Paladox placed T5044: Setup centralised logging for services up for grabs.
Jun 14 2021, 18:59 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Paladox placed T7241: ATS: Deploy healthchecker that depools/repools up for grabs.
Jun 14 2021, 18:59 · Infrastructure (SRE)
Paladox placed T4425: Fix all mysql tables that are using latin rather than binary up for grabs.
Jun 14 2021, 18:59 · Infrastructure (SRE)
Paladox placed T7287: /mnt/mediawiki-static failures on mwx; missing thumbnails in places up for grabs.
Jun 14 2021, 18:59 · Monitoring, Production Error, MediaWiki (SRE)
Paladox closed T7388: Decom cp10/cp11 as Resolved.
Jun 14 2021, 18:58 · Infrastructure (SRE)
Reception123 defrocked Paladox.
Jun 14 2021, 16:57
Paladox added a comment to T7467: [Access Request] Void.

I think now that’s all the approval we need to move forward?

Jun 14 2021, 12:07 · Site Reliability Engineering
Paladox closed T7460: Fatal exception of type "Error" after first edit after Update to MediaWiki 1.36.0 as Resolved.

Should be resolved now, please reopen if not.

Jun 14 2021, 02:49 · Production Error, Universal Omega, Extensions, MediaWiki (SRE)
Paladox added a comment to T7467: [Access Request] Void.

Approved 1,000%!

Jun 14 2021, 01:26 · Site Reliability Engineering

Jun 13 2021

Paladox added a comment to T7421: wiki.qadrishattari.xyz does not point at mw-lb.miraheze.org.

Bump

Jun 13 2021, 16:30 · Infrastructure (SRE)
Paladox added a comment to T7460: Fatal exception of type "Error" after first edit after Update to MediaWiki 1.36.0.

Fixed with https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ApprovedRevs/+/699522

Jun 13 2021, 12:43 · Production Error, Universal Omega, Extensions, MediaWiki (SRE)
Paladox added a comment to T7450: Evaluate what happened during the 1.36 update.

I was only around just in case. I wasn’t supposed to be involved but seems that changed…

Jun 13 2021, 02:13 · MediaWiki (SRE)

Jun 12 2021

Paladox added a comment to T7444: Special page random wiki leads to a private wiki.

Done by @Universal_Omega.

Jun 12 2021, 18:09 · WikiDiscover, MediaWiki (SRE), Universal Omega
Paladox added a comment to T7352: Significant drop in cache hit rate caused all wikis to go down about 2am UK time 24 May 2021.

Those actionables sound fine to me.

Jun 12 2021, 13:30 · Infrastructure (SRE)

Jun 9 2021

Paladox added a comment to T7421: wiki.qadrishattari.xyz does not point at mw-lb.miraheze.org.
host wiki.qadrishattari.xyz
wiki.qadrishattari.xyz has address 172.67.201.245
wiki.qadrishattari.xyz has address 104.21.60.225
wiki.qadrishattari.xyz has IPv6 address 2606:4700:3031::6815:3ce1
wiki.qadrishattari.xyz has IPv6 address 2606:4700:3030::ac43:c9f5
Jun 9 2021, 22:05 · Infrastructure (SRE)
Paladox added a comment to T7434: Database errors .

No, I just posted the log as requested.

Jun 9 2021, 20:05 · Production Error, MediaWiki (SRE), MediaWiki, Database
Paladox added a comment to T7434: Database errors .
[038812f60b8b45e95837e2b3] /w/index.php?title=Special:Import&action=submit   Wikimedia\\Rdbms\\DBTransactionStateError from line 1501 of /srv/mediawiki/w/includes/libs/rdbms/database/Database.php: Cannot execute query from MediaWiki::preOutputCommit while transaction status is ERROR
#0 /srv/mediawiki/w/includes/libs/rdbms/database/Database.php(1220): Wikimedia\Rdbms\Database->assertQueryIsCurrentlyAllowed(string, string)
#1 /srv/mediawiki/w/includes/libs/rdbms/database/Database.php(4606): Wikimedia\Rdbms\Database->query(string, string, integer)
#2 /srv/mediawiki/w/includes/libs/rdbms/database/Database.php(4574): Wikimedia\Rdbms\Database->doCommit(string)
#3 /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1712): Wikimedia\Rdbms\Database->commit(string, string)
#4 /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2150): Wikimedia\Rdbms\LoadBalancer->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#5 /srv/mediawiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1717): Wikimedia\Rdbms\LoadBalancer->forEachOpenMasterConnection(Closure)
#6 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactory.php(244): Wikimedia\Rdbms\LoadBalancer->commitMasterChanges(string, integer)
#7 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(222): Wikimedia\Rdbms\LBFactory->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer, string, array)
#8 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactory.php(246): Wikimedia\Rdbms\LBFactoryMulti->forEachLB(Closure, array)
#9 /srv/mediawiki/w/includes/libs/rdbms/lbfactory/LBFactory.php(309): Wikimedia\Rdbms\LBFactory->forEachLBCallMethod(string, array)
#10 /srv/mediawiki/w/includes/MediaWiki.php(678): Wikimedia\Rdbms\LBFactory->commitMasterChanges(string, array)
#11 /srv/mediawiki/w/includes/MediaWiki.php(648): MediaWiki::preOutputCommit(RequestContext, Closure)
#12 /srv/mediawiki/w/includes/MediaWiki.php(956): MediaWiki->doPreOutputCommit(Closure)
#13 /srv/mediawiki/w/includes/MediaWiki.php(543): MediaWiki->main()
#14 /srv/mediawiki/w/index.php(53): MediaWiki->run()
#15 /srv/mediawiki/w/index.php(46): wfIndexMain()
#16 {main}
Jun 9 2021, 19:32 · Production Error, MediaWiki (SRE), MediaWiki, Database
Paladox added a comment to T7420: dreamsit.com.br is pointing to cp11 which has been decommissioned.

See https://meta.miraheze.org/wiki/Custom_domain or https://meta.miraheze.org/wiki/Custom_domains/Cloudflare

Jun 9 2021, 12:58 · Infrastructure (SRE)
Paladox added a comment to T7420: dreamsit.com.br is pointing to cp11 which has been decommissioned.

It shouldn’t be pointing directly at a cp. it needs to be pointing at mw-lb.miraheze.org because there’s no gurentee we will have that ip in the future. Also no need to ping 4 times.

Jun 9 2021, 12:55 · Infrastructure (SRE)

Jun 7 2021

Paladox updated subscribers of T7422: lcn.zfc.id.lv keeps flapping between warning and ok causing icinga spam.

@Xymachos could you switch the cname to mw-lb.miraheze.org please?

Jun 7 2021, 14:57 · Infrastructure (SRE)
Paladox triaged T7422: lcn.zfc.id.lv keeps flapping between warning and ok causing icinga spam as High priority.
Jun 7 2021, 14:35 · Infrastructure (SRE)
Paladox triaged T7421: wiki.qadrishattari.xyz does not point at mw-lb.miraheze.org as High priority.
Jun 7 2021, 14:31 · Infrastructure (SRE)
Paladox edited P386 Resources Table.
Jun 7 2021, 14:11 · Cloud Infrastructure, Infrastructure (SRE)
Paladox edited P386 Resources Table.
Jun 7 2021, 14:08 · Cloud Infrastructure, Infrastructure (SRE)
Paladox edited P386 Resources Table.
Jun 7 2021, 14:04 · Cloud Infrastructure, Infrastructure (SRE)
Paladox updated subscribers of T7420: dreamsit.com.br is pointing to cp11 which has been decommissioned.
Jun 7 2021, 13:15 · Infrastructure (SRE)
Paladox triaged T7420: dreamsit.com.br is pointing to cp11 which has been decommissioned as High priority.
Jun 7 2021, 13:06 · Infrastructure (SRE)

Jun 6 2021

Paladox added a comment to T7416: Linter doesn’t work.

You can try it, yeh.

Jun 6 2021, 12:13 · Extensions, Universal Omega, MediaWiki (SRE)

Jun 4 2021

Paladox closed T7230: High I/O on cloud nodes affecting GlusterFS as Resolved.

https://github.com/miraheze/puppet/commit/bf0e0d579556dfcac8c2d41e627d7b8cbdea51d4

Jun 4 2021, 21:37 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Paladox closed T7404: cp12 down as Resolved.
Jun 4 2021, 14:38 · Infrastructure (SRE)

Jun 2 2021

Paladox added a comment to T4425: Fix all mysql tables that are using latin rather than binary.

Phabricator uses utf8mb4 (as it's forced).

Jun 2 2021, 12:13 · Infrastructure (SRE)
Paladox added a comment to T4425: Fix all mysql tables that are using latin rather than binary.

He also said he doesn't think it would cause corruption if one table had latin1 and the other binary but he also said it's worth checking out.

Jun 2 2021, 12:10 · Infrastructure (SRE)
Paladox added a comment to T4425: Fix all mysql tables that are using latin rather than binary.

Spoked with a DBA at wmf briefly (Marostegui). He said it uses utf8mb4 and binary. Also he said he guessed it required read only and also alter table commands.

Jun 2 2021, 12:09 · Infrastructure (SRE)

Jun 1 2021

MarioMario456 awarded T7382: Phabricator is no longer maintained - decide what to do a Like token.
Jun 1 2021, 16:17 · Universal Omega, Site Reliability Engineering, Phabricator

May 31 2021

Paladox updated the task description for T7388: Decom cp10/cp11.
May 31 2021, 18:58 · Infrastructure (SRE)
Paladox triaged T7388: Decom cp10/cp11 as High priority.
May 31 2021, 18:58 · Infrastructure (SRE)
Paladox created T7388: Decom cp10/cp11.
May 31 2021, 18:58 · Infrastructure (SRE)
Paladox committed R9:9b53d5ea315a: Merge branch 'master' into REL1_36 (authored by Paladox).
Merge branch 'master' into REL1_36
May 31 2021, 14:56
Paladox lowered the priority of T7287: /mnt/mediawiki-static failures on mwx; missing thumbnails in places from High to Normal.

Not sure if we can close this as resolved seeing as this is a hack I did but changing priority to normal.

May 31 2021, 14:35 · Monitoring, Production Error, MediaWiki (SRE)
Paladox added a comment to T7287: /mnt/mediawiki-static failures on mwx; missing thumbnails in places.

I've done:

May 31 2021, 14:34 · Monitoring, Production Error, MediaWiki (SRE)

May 29 2021

Paladox added a comment to T7382: Phabricator is no longer maintained - decide what to do.

Wikimedia previously used bugIlla and migrated to phabricator.

May 29 2021, 23:33 · Universal Omega, Site Reliability Engineering, Phabricator
Paladox added a comment to T7382: Phabricator is no longer maintained - decide what to do.

I dislike bugzilla tbh. Ugly UI and hard to use.

May 29 2021, 23:31 · Universal Omega, Site Reliability Engineering, Phabricator
Paladox updated the task description for T7382: Phabricator is no longer maintained - decide what to do.
May 29 2021, 22:42 · Universal Omega, Site Reliability Engineering, Phabricator
Paladox triaged T7382: Phabricator is no longer maintained - decide what to do as Normal priority.
May 29 2021, 22:35 · Universal Omega, Site Reliability Engineering, Phabricator
Paladox created T7382: Phabricator is no longer maintained - decide what to do.
May 29 2021, 22:34 · Universal Omega, Site Reliability Engineering, Phabricator

May 27 2021

Paladox claimed T7352: Significant drop in cache hit rate caused all wikis to go down about 2am UK time 24 May 2021.
May 27 2021, 21:48 · Infrastructure (SRE)

May 26 2021

Dmehus awarded T7372: Run createLocalAccount.php on nebulaprojectwiki a Like token.
May 26 2021, 23:35 · MediaWiki (SRE), MediaWiki
Paladox closed T7372: Run createLocalAccount.php on nebulaprojectwiki as Resolved.
May 26 2021, 23:34 · MediaWiki (SRE), MediaWiki

May 25 2021

Paladox shifted T7362: Skin Review: Anisa from the Restricted Space space to the S1 Public space.
May 25 2021, 18:46 · Configuration, Extensions, Universal Omega, MediaWiki (SRE)
Paladox committed R9:c89be3c5a56e: Merge branch 'master' into REL1_36 (authored by Paladox).
Merge branch 'master' into REL1_36
May 25 2021, 15:57
Paladox edited P403 SQLs / Scripts for 1.36 update.
May 25 2021, 15:42
Paladox added a comment to T7117: Upgrade to MediaWiki 1.36.0.

I'm also going to apply https://github.com/wikimedia/mediawiki-extensions-DiscussionTools/blob/REL1_36/sql/mysql/discussiontools_subscription.sql to metawiki. It's a new table so is a noop and required by 1.36.

May 25 2021, 15:39 · Universal Omega, MediaWiki (SRE), MediaWiki
Paladox added a comment to T7117: Upgrade to MediaWiki 1.36.0.

I'm going to run https://github.com/wikimedia/mediawiki-extensions-CentralNotice/commit/33c729fcc749b47085ef18ff6364c718591a3df1#diff-3949da3a08da523b54e5d7f4ea2d9ef2ddbd7368f6e31cd052a48bfa744bb380 and https://github.com/wikimedia/mediawiki-extensions-CentralNotice/commit/33c729fcc749b47085ef18ff6364c718591a3df1#diff-b53edff06a759d8d1a2eaea0d54ed4e599b95a3040e606671bde03a2a6a4b087 on metawikis table.

May 25 2021, 15:22 · Universal Omega, MediaWiki (SRE), MediaWiki
Paladox committed R9:63953cc014c1: Merge branch 'master' into REL1_36 (authored by Paladox).
Merge branch 'master' into REL1_36
May 25 2021, 14:25

May 24 2021

Paladox created P435 (An Untitled Masterwork).
May 24 2021, 22:06
Paladox created P434 (An Untitled Masterwork).
May 24 2021, 17:45
Paladox created P433 (An Untitled Masterwork).
May 24 2021, 17:17
Paladox created P432 (An Untitled Masterwork).
May 24 2021, 17:17
Paladox added a comment to T7352: Significant drop in cache hit rate caused all wikis to go down about 2am UK time 24 May 2021.

DB13 showed that things went down.

May 24 2021, 11:26 · Infrastructure (SRE)
Paladox added a comment to T7352: Significant drop in cache hit rate caused all wikis to go down about 2am UK time 24 May 2021.

DB11 & DB12 show higher traffic and load this morning.

May 24 2021, 11:25 · Infrastructure (SRE)

May 20 2021

Paladox closed T7299: Upgrade to gluster 8.5, a subtask of T7287: /mnt/mediawiki-static failures on mwx; missing thumbnails in places, as Resolved.
May 20 2021, 20:09 · Monitoring, Production Error, MediaWiki (SRE)
Paladox closed T7299: Upgrade to gluster 8.5 as Resolved.

Maintenance completed.

May 20 2021, 20:09 · Infrastructure (SRE)
Paladox updated the task description for T7299: Upgrade to gluster 8.5.
May 20 2021, 20:09 · Infrastructure (SRE)
Paladox updated the task description for T7299: Upgrade to gluster 8.5.
May 20 2021, 20:06 · Infrastructure (SRE)

May 18 2021

Paladox added a comment to T7323: New Server Resource Request for Cache Proxies.

We should rather than do 2 vms, we should move to one for London using 2 cores and nvme disk. Matching NA. I recommend we go with the one new vm in london and getting rid of the existing 2 vms on cloud4/5

May 18 2021, 15:28 · Varnish, Infrastructure (SRE)

May 17 2021

Paladox updated the task description for T7299: Upgrade to gluster 8.5.
May 17 2021, 19:53 · Infrastructure (SRE)
Paladox updated the task description for T7299: Upgrade to gluster 8.5.
May 17 2021, 19:52 · Infrastructure (SRE)

May 15 2021

Paladox committed rPUPC9c2038773e36: Merge branch 'master' of github.com:miraheze/puppet (authored by Paladox).
Merge branch 'master' of github.com:miraheze/puppet
May 15 2021, 14:59
Paladox committed rPUPC072c066ac388: Merge branch 'master' of github.com:miraheze/puppet (authored by Paladox).
Merge branch 'master' of github.com:miraheze/puppet
May 15 2021, 14:59
Paladox committed rPUPCfad190410889: Merge branch 'master' of github.com:miraheze/puppet (authored by Paladox).
Merge branch 'master' of github.com:miraheze/puppet
May 15 2021, 14:59
Paladox committed rPUPCbdfd1e6c8b6f: Remove unused hiera value default_lua_script (authored by Paladox).
Remove unused hiera value default_lua_script
May 15 2021, 14:59
Paladox committed rPUPCf9cd9f199c11: Remove trafficserver::get_paths (authored by Paladox).
Remove trafficserver::get_paths
May 15 2021, 14:59
Paladox committed rPUPC91053f366c5a: Disable xdebug on test4 (authored by Paladox).
Disable xdebug on test4
May 15 2021, 14:59

May 13 2021

Paladox updated the task description for T7299: Upgrade to gluster 8.5.
May 13 2021, 18:54 · Infrastructure (SRE)
Paladox updated the task description for T7299: Upgrade to gluster 8.5.
May 13 2021, 18:50 · Infrastructure (SRE)
Paladox updated the task description for T7299: Upgrade to gluster 8.5.
May 13 2021, 18:49 · Infrastructure (SRE)
Paladox moved T7299: Upgrade to gluster 8.5 from Incoming to Short Term on the Infrastructure (SRE) board.
May 13 2021, 18:48 · Infrastructure (SRE)
Paladox triaged T7299: Upgrade to gluster 8.5 as High priority.
May 13 2021, 18:48 · Infrastructure (SRE)

May 12 2021

Paladox set the icon for Trust & Safety to Group.
May 12 2021, 20:48
Paladox created Trust & Safety.
May 12 2021, 20:47

May 11 2021

Paladox triaged T7288: Upgrade gluster to 9.2 as Normal priority.
May 11 2021, 19:23 · Infrastructure (SRE)

May 10 2021

Paladox added a comment to T7276: [ACCESS REQUEST] Universal Omega for MediaWiki Engineering.

+1

May 10 2021, 16:01 · Universal Omega, Site Reliability Engineering

May 5 2021

Paladox triaged T7248: Add which db server the wiki is on to the logging configuration as Normal priority.
May 5 2021, 15:36 · MediaWiki (SRE)
Paladox added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

Ah ignore the above, having looked around I don't see how we can do it then if you cannot use ionice within the script && also using it outside it too.

May 5 2021, 12:41 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance

May 4 2021

Paladox added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

Ohh should we run with —low

May 4 2021, 22:18 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Paladox added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

I even ran it with ionice the command and the load on the disk was still high.

May 4 2021, 22:13 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance

May 3 2021

Paladox added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

The script runs with --idle which uses ionice, so it's already using it?

May 3 2021, 18:19 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Paladox added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

On https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592149 it says ionice -c 3 /usr/share/mdadm/checkarray --cron --all

May 3 2021, 18:11 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Paladox triaged T7241: ATS: Deploy healthchecker that depools/repools as Normal priority.
May 3 2021, 17:40 · Infrastructure (SRE)
Paladox triaged T7240: ATS: Review security as Normal priority.
May 3 2021, 17:39 · Infrastructure (SRE)