Page MenuHomeMiraheze

Paladox
Site Reliabilty EngineerAdministrator

User Details

User Since
Oct 14 2017, 11:40 (176 w, 7 h)
Roles
Administrator
Availability
Available
GitHub User
Unknown
Miraheze User
Paladox [ Global Accounts ]

I live in the United Kingdom GMT/BST(UTC+0, UTC+1).

I am a Site Reliabilty Engineer for Miraheze.

Recent Activity

Yesterday

Paladox closed T6870: Graylog should process the timestamp based on when the log is generated as Resolved.
Fri, Feb 26, 02:09 · Infrastructure (SRE)
Paladox added a comment to T6870: Graylog should process the timestamp based on when the log is generated.

I've created some pipeline rules which fixed this for some things (like mediawiki, also doing it for nginx).

Fri, Feb 26, 02:08 · Infrastructure (SRE)
Paladox closed T6858: Messages take a while to be sent to graylog as Resolved.

Think we've resolved this by using pipeline rules to use timestamp in the json and setting the timestamp.

Fri, Feb 26, 02:07 · Infrastructure (SRE)

Thu, Feb 25

Paladox closed T6730: Custom Access-Control-Allow-Origin as Resolved.

Error is gone now.

Thu, Feb 25, 20:09 · Infrastructure (SRE), Universal Omega, Puppet
Paladox updated subscribers of T5537: Replace SaltStack with Cumin.
Thu, Feb 25, 19:34 · Infrastructure (SRE)
Paladox closed T5537: Replace SaltStack with Cumin as Declined.

We're using salt-ssh.

Thu, Feb 25, 19:34 · Infrastructure (SRE)
Paladox closed T5537: Replace SaltStack with Cumin, a subtask of T5543: Reconfigure SaltStack or replace it with another tool, as Declined.
Thu, Feb 25, 19:34 · Security, Site Reliability Engineering
Paladox committed rPUPCe694825e6aba: Remove old puppetmaster module (authored by Paladox).
Remove old puppetmaster module
Thu, Feb 25, 19:21
Paladox claimed T5537: Replace SaltStack with Cumin.
Thu, Feb 25, 16:37 · Infrastructure (SRE)
Paladox closed T6862: Use memcache for the cache as Resolved.

We've successfully switched over, closing as resolved.

Thu, Feb 25, 16:34 · Infrastructure (SRE)
Paladox added a comment to T6873: Log locally in addition to remotely.

Talked about it with SPF and have come to the conclusion that we wont do this.

Thu, Feb 25, 11:25 · Infrastructure (SRE)
Paladox closed T6873: Log locally in addition to remotely as Declined.
Thu, Feb 25, 11:25 · Infrastructure (SRE)

Mon, Feb 22

Paladox assigned T6873: Log locally in addition to remotely to Southparkfan.

Needs a discussion with SPF.

Mon, Feb 22, 18:05 · Infrastructure (SRE)
Paladox committed rPUPC67beda16dbd0: Update symnlink for hieradata/private (authored by Paladox).
Update symnlink for hieradata/private
Mon, Feb 22, 00:53

Sat, Feb 20

Paladox added a comment to T6839: Upgrade puppet to puppet 7.
In T6839#134807, @John wrote:

Is a major software upgrade of critical services really ‘short term’?

Sat, Feb 20, 14:40 · Puppet, Infrastructure (SRE)

Fri, Feb 19

Paladox raised the priority of T6858: Messages take a while to be sent to graylog from Normal to High.
Fri, Feb 19, 23:18 · Infrastructure (SRE)
Paladox updated the task description for T6873: Log locally in addition to remotely.
Fri, Feb 19, 22:47 · Infrastructure (SRE)
Paladox updated the task description for T6873: Log locally in addition to remotely.
Fri, Feb 19, 22:37 · Infrastructure (SRE)
Paladox renamed T6873: Log locally in addition to remotely from Log locally to Log locally inaddition to remotely.
Fri, Feb 19, 22:37 · Infrastructure (SRE)
Paladox renamed T6870: Graylog should process the timestamp based on when the log is generated from Graylog should process the timestamp off the log rather then the time it processes it to Graylog should process the timestamp based on when the log is generated.
Fri, Feb 19, 22:36 · Infrastructure (SRE)
Paladox triaged T6873: Log locally in addition to remotely as High priority.
Fri, Feb 19, 22:35 · Infrastructure (SRE)
Paladox triaged T6872: Process messages instantly as High priority.
Fri, Feb 19, 22:34 · Infrastructure (SRE)
Paladox added a comment to T6870: Graylog should process the timestamp based on when the log is generated.

Technically this is a UBN.

Fri, Feb 19, 20:24 · Infrastructure (SRE)
Paladox moved T6870: Graylog should process the timestamp based on when the log is generated from Incoming to Short Term on the Infrastructure (SRE) board.
Fri, Feb 19, 20:24 · Infrastructure (SRE)
Paladox triaged T6870: Graylog should process the timestamp based on when the log is generated as High priority.
Fri, Feb 19, 20:24 · Infrastructure (SRE)
Paladox triaged T6868: Graylog: Increase max results for elasticsearch as Normal priority.
Fri, Feb 19, 19:29 · Infrastructure (SRE)
Paladox created T6868: Graylog: Increase max results for elasticsearch.
Fri, Feb 19, 19:29 · Infrastructure (SRE)

Thu, Feb 18

Paladox committed R9:a03172bcfa15: Revert "Fix default value for wgUsersNotifiedOnAllChanges (#3729)" (authored by Paladox).
Revert "Fix default value for wgUsersNotifiedOnAllChanges (#3729)"
Thu, Feb 18, 18:21
Paladox added a reverting change for R9:c4624a39516c: Fix default value for wgUsersNotifiedOnAllChanges (#3729): R9:a03172bcfa15: Revert "Fix default value for wgUsersNotifiedOnAllChanges (#3729)".
Thu, Feb 18, 18:21
Paladox committed rPUPC46b0c459f77e: php: Add wikidiff2 and remove luasandbox for php7.2 (authored by Paladox).
php: Add wikidiff2 and remove luasandbox for php7.2
Thu, Feb 18, 05:02
Paladox committed rPUPC13aff51bc965: Merge branch 'master' of github.com:miraheze/puppet (authored by Paladox).
Merge branch 'master' of github.com:miraheze/puppet
Thu, Feb 18, 00:27
Paladox committed rPUPC756d6c9be279: Introduce memcached role (authored by Paladox).
Introduce memcached role
Thu, Feb 18, 00:27

Wed, Feb 17

Dmehus awarded T6862: Use memcache for the cache a Pterodactyl token.
Wed, Feb 17, 22:37 · Infrastructure (SRE)
Paladox added a comment to T6858: Messages take a while to be sent to graylog.

I think it may be syslog-ng -> graylog. Has to be otherwise messages would be lost going from mw -> syslog-ng.

Wed, Feb 17, 22:37 · Infrastructure (SRE)
Universal_Omega awarded T6862: Use memcache for the cache a Like token.
Wed, Feb 17, 22:31 · Infrastructure (SRE)
Paladox claimed T6862: Use memcache for the cache.

Per chat with @John we're going to move ahead and put all cache in memcache. Renaming rdb4 as mc2.

Wed, Feb 17, 22:08 · Infrastructure (SRE)
Paladox closed T6861: Switch to kask for sessions as Declined.

I'm not sure its worth looking at this right now so declining based on our chat.

Wed, Feb 17, 21:26 · MediaWiki (SRE)
Paladox renamed T6862: Use memcache for the cache from Use memcache for main cache to Use memcache for the cache.
Wed, Feb 17, 21:25 · Infrastructure (SRE)
Paladox added a comment to T6862: Use memcache for the cache.

Redis is single threaded where as memcache is multi threaded This improves performance.

Wed, Feb 17, 17:20 · Infrastructure (SRE)
Paladox triaged T6862: Use memcache for the cache as Normal priority.
Wed, Feb 17, 17:19 · Infrastructure (SRE)
Paladox triaged T6861: Switch to kask for sessions as Normal priority.
Wed, Feb 17, 17:14 · MediaWiki (SRE)
Paladox triaged T6860: runJobs: json_decode() error (4): Syntax error as Normal priority.
Wed, Feb 17, 15:30 · MediaWiki, MediaWiki (SRE)
Paladox assigned T6859: Huge rise in Redis server error: Could not insert <number> <job-name> job(s). to RhinosF1.
Wed, Feb 17, 15:23 · MediaWiki, MediaWiki (SRE), Production Error

Tue, Feb 16

Paladox added a comment to T6858: Messages take a while to be sent to graylog.

the only problem is that this causes things to be out of sync, for example a message could have been sent an hour ago but we would have only just saw it now because of this. This has happened with exceptions when investigating. This impedes investigations.

Tue, Feb 16, 16:56 · Infrastructure (SRE)
Paladox triaged T6858: Messages take a while to be sent to graylog as Normal priority.
Tue, Feb 16, 16:55 · Infrastructure (SRE)

Mon, Feb 15

Dmehus awarded T6852: New Server Resource Request for bots1 a Like token.
Mon, Feb 15, 18:22 · Infrastructure (SRE)
Paladox removed a member for acl*security_reviewers: NDKilla.
Mon, Feb 15, 15:52
Paladox removed a member for acl*security_reviewers: Zppix.
Mon, Feb 15, 15:52
Paladox created T6852: New Server Resource Request for bots1.
Mon, Feb 15, 02:00 · Infrastructure (SRE)

Sun, Feb 14

Paladox closed T6849: Grafana bug CVE-2019-15043 can still be exploited despite being out of vulnerable range as Resolved.

We've sent off an email to grafana and we've blocked the url. We are running a version that is supposedly should be patched but I can still reproduce it.

Sun, Feb 14, 16:54 · Upstream, Infrastructure (SRE), Security
R4356th awarded T6840: Have topmost donation banner as image instead of text, because search engines are showing this text as snippet a Like token.
Sun, Feb 14, 04:12 · MediaWiki, MediaWiki (SRE)

Sat, Feb 13

Dmehus awarded T6840: Have topmost donation banner as image instead of text, because search engines are showing this text as snippet a Like token.
Sat, Feb 13, 15:20 · MediaWiki, MediaWiki (SRE)
Paladox added a comment to T6840: Have topmost donation banner as image instead of text, because search engines are showing this text as snippet.
<div class="cnotice" id="{{{banner}}}">
    <div id="cnotice-main">
        <div class="cnotice-logo-container">
            <img src="//static.miraheze.org/metawiki/3/35/Miraheze_Logo.svg" alt="Banner logo">
        </div>
        <div class="cnotice-message-container">
            <div class="cnotice-message">
                <p><span data-nosnippet>{{{msg-1}}} <a href="https://www.gofundme.com/f/help-miraheze-stay-online-in-2021">{{{msg-2}}}, <a href="https://paypal.me/mirahezelimited">{{{msg-3}}} or <a href="https://github.com/sponsors/miraheze">{{{msg-4}}}</a>!</span></p>
            </div>
            <a href="https://meta.miraheze.org/w/index.php?title=Special:Translate&group=Centralnotice-tgroup-{{{banner}}}" title="{{{translate}}}">
                <span data-nosnippet>{{{translate}}}</span>
            </a>
            <center><h4><small>{{{msg-5}}}</h4></center>
        </div>
    <div id="cnotice-toggle-box">
        <a href="#" title="Hide" onclick="mw.centralNotice.hideBanner();return false;"><img border="0" src="//upload.wikimedia.org/wikipedia/donate/a/ac/Close_oojs.png" alt="Hide" /></a>
    </div>
</div>
Sat, Feb 13, 14:42 · MediaWiki, MediaWiki (SRE)

Fri, Feb 12

Paladox moved T5397: Create a logbot for server actions from Short Term to Long Term on the Infrastructure (SRE) board.
Fri, Feb 12, 22:27 · Infrastructure (SRE)

Thu, Feb 11

Paladox added a comment to T6843: Uploaded image is missing (returns 404 Not Found).

Hi, we recently migrated our file storage, would you be able to re upload the file in question?

Thu, Feb 11, 19:43 · MediaWiki, MediaWiki (SRE)
Paladox triaged T6839: Upgrade puppet to puppet 7 as Normal priority.
Thu, Feb 11, 00:35 · Puppet, Infrastructure (SRE)

Tue, Feb 9

Dmehus awarded T6832: npm should not run as root on MediaWiki servers a Like token.
Tue, Feb 9, 23:32 · Infrastructure (SRE), Security
Paladox created P384 resp_time.py.
Tue, Feb 9, 23:17
Paladox created P383 read_db.py.
Tue, Feb 9, 23:16
Paladox changed the visibility for T6832: npm should not run as root on MediaWiki servers.
Tue, Feb 9, 20:21 · Infrastructure (SRE), Security
Paladox closed T6832: npm should not run as root on MediaWiki servers as Resolved.

Has all been migrated away from using root. Thus this task is resolved now.

Tue, Feb 9, 19:51 · Infrastructure (SRE), Security

Mon, Feb 8

Dmehus awarded T6828: MySQL crashed on db11 2021-02-06 ~20:46 a Like token.
Mon, Feb 8, 02:01 · Database, Infrastructure (SRE), Production Error, MediaWiki, MediaWiki (SRE)
Dmehus awarded T6826: Core Namespace translations having no effect a Like token.
Mon, Feb 8, 02:01 · MediaWiki (SRE), MediaWiki

Sun, Feb 7

Paladox updated the task description for T6564: Upgrade to Debian 10.7.
Sun, Feb 7, 23:31 · Infrastructure (SRE)
Paladox closed T6826: Core Namespace translations having no effect as Resolved.

Fixed by refreshing CreateWiki cache.

Sun, Feb 7, 22:32 · MediaWiki (SRE), MediaWiki
Paladox closed T6828: MySQL crashed on db11 2021-02-06 ~20:46 as Resolved.

Published at https://meta.miraheze.org/wiki/Special:IncidentReports/36

Sun, Feb 7, 19:29 · Database, Infrastructure (SRE), Production Error, MediaWiki, MediaWiki (SRE)

Sat, Feb 6

Paladox created P380 MySQL crash logs for db11 (2021-02-06 ~20:46).
Sat, Feb 6, 21:02
Paladox claimed T6787: Migrate cloud1 and cloud2 virtual machines to new servers.
Sat, Feb 6, 17:19 · Infrastructure (SRE)
Paladox closed T6787: Migrate cloud1 and cloud2 virtual machines to new servers as Resolved.
Sat, Feb 6, 17:19 · Infrastructure (SRE)
Paladox closed T6787: Migrate cloud1 and cloud2 virtual machines to new servers, a subtask of T6770: Renew or decommission cloud1, cloud2 and replace cp9, as Resolved.
Sat, Feb 6, 17:19 · Infrastructure (SRE)
Paladox updated the task description for T6787: Migrate cloud1 and cloud2 virtual machines to new servers.
Sat, Feb 6, 17:18 · Infrastructure (SRE)
Paladox closed T6826: Core Namespace translations having no effect as Resolved.

Rebooted the servers which fixed the issue.

Sat, Feb 6, 15:04 · MediaWiki (SRE), MediaWiki

Fri, Feb 5

Paladox created P379 (An Untitled Masterwork).
Fri, Feb 5, 17:03

Thu, Feb 4

Paladox committed R9:95a6ab6fad52: Revert "Set wgInternalServer to wgServer (#3706)" (authored by Paladox).
Revert "Set wgInternalServer to wgServer (#3706)"
Thu, Feb 4, 13:09
Paladox added a reverting change for R9:f850edc2f0c9: Set wgInternalServer to wgServer (#3706): R9:95a6ab6fad52: Revert "Set wgInternalServer to wgServer (#3706)".
Thu, Feb 4, 13:09
Paladox created P378 (An Untitled Masterwork).
Thu, Feb 4, 00:25

Wed, Feb 3

Paladox added a reverting change for rDNS8f9e193cbeb5: miraheze.wiki: failover dns for [bots|tools|db]1 (#186): rDNSa5076734a982: Revert "miraheze.wiki: failover dns for [bots|tools|db]1 (#186)".
Wed, Feb 3, 23:20
Paladox committed rDNSa5076734a982: Revert "miraheze.wiki: failover dns for [bots|tools|db]1 (#186)" (authored by Paladox).
Revert "miraheze.wiki: failover dns for [bots|tools|db]1 (#186)"
Wed, Feb 3, 23:20
Paladox lowered the priority of T6811: [Existing] Server Resource Request for gluster1 from Unbreak Now! to High.

I've increased it in the UI but it wouldn't let me extend it because no disk space.

Wed, Feb 3, 03:02 · Infrastructure (SRE)
Paladox updated the task description for T6811: [Existing] Server Resource Request for gluster1.
Wed, Feb 3, 01:13 · Infrastructure (SRE)
Paladox raised the priority of T6811: [Existing] Server Resource Request for gluster1 from Normal to Unbreak Now!.
Wed, Feb 3, 01:04 · Infrastructure (SRE)
Paladox created T6811: [Existing] Server Resource Request for gluster1.
Wed, Feb 3, 01:03 · Infrastructure (SRE)

Tue, Feb 2

Paladox added a reverting change for R9:700c2c5f646d: Disable file uploads (#3697): R9:cb1a4845dc09: Revert "Disable file uploads (#3697)".
Tue, Feb 2, 10:29
Paladox committed R9:cb1a4845dc09: Revert "Disable file uploads (#3697)" (authored by Paladox).
Revert "Disable file uploads (#3697)"
Tue, Feb 2, 10:29
Paladox added a reverting change for R9:b304fa782615: Add note to $wgEnableUploads: R9:a742cd5afd4b: Revert "Add note to $wgEnableUploads".
Tue, Feb 2, 10:29
Paladox committed R9:a742cd5afd4b: Revert "Add note to $wgEnableUploads" (authored by Paladox).
Revert "Add note to $wgEnableUploads"
Tue, Feb 2, 10:29
Paladox added a reverting change for R9:d5ed4e629ed8: remove old description for wgEnableUploads. Will be re-added when re-enabled: R9:4a14e8b605fa: Revert "remove old description for wgEnableUploads. Will be re-added when re….
Tue, Feb 2, 10:29
Paladox added a reverting change for R9:013cc22bb5ef: minor fix in help message: R9:a0e71de913ba: Revert "minor fix in help message".
Tue, Feb 2, 10:28
Paladox committed R9:4a14e8b605fa: Revert "remove old description for wgEnableUploads. Will be re-added when re… (authored by Paladox).
Revert "remove old description for wgEnableUploads. Will be re-added when re…
Tue, Feb 2, 10:28
Paladox committed R9:a0e71de913ba: Revert "minor fix in help message" (authored by Paladox).
Revert "minor fix in help message"
Tue, Feb 2, 10:28
Paladox added a reverting change for R9:f21fefa3458b: Set wgDataDumpDisableGenerate to true: R9:5ee6f86584db: Revert "Set wgDataDumpDisableGenerate to true".
Tue, Feb 2, 10:28
Paladox committed R9:5ee6f86584db: Revert "Set wgDataDumpDisableGenerate to true" (authored by Paladox).
Revert "Set wgDataDumpDisableGenerate to true"
Tue, Feb 2, 10:28
Paladox created P377 (An Untitled Masterwork).
Tue, Feb 2, 00:53

Mon, Feb 1

Paladox created P374 (An Untitled Masterwork).
Mon, Feb 1, 21:30
Paladox created P372 (An Untitled Masterwork).
Mon, Feb 1, 17:52
Paladox created P371 (An Untitled Masterwork).
Mon, Feb 1, 17:24
Paladox created P370 (An Untitled Masterwork).
Mon, Feb 1, 00:44

Sun, Jan 31

Paladox updated the task description for T6787: Migrate cloud1 and cloud2 virtual machines to new servers.
Sun, Jan 31, 23:26 · Infrastructure (SRE)
Paladox updated the task description for T6787: Migrate cloud1 and cloud2 virtual machines to new servers.
Sun, Jan 31, 23:26 · Infrastructure (SRE)
Paladox created P369 (An Untitled Masterwork).
Sun, Jan 31, 00:29