Page MenuHomeMiraheze
People Paladox

Paladox
Site Reliabilty EngineerAdministrator

Projects

Calendar

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Oct 14 2017, 11:40 (259 w, 6 d)
Roles
Administrator
Availability
Available
GitHub User
Unknown
Miraheze User
Paladox [ Global Accounts ]

I live in the United Kingdom GMT/BST(UTC+0, UTC+1).

I am a Site Reliability Engineer for miraheze.

Recent Activity
View All

Yesterday

Paladox assigned T9804: [New/Existing] Server Resource Request for swift proxy to John.
Thu, Oct 6, 11:06 · Infrastructure (SRE)
Paladox created T9804: [New/Existing] Server Resource Request for swift proxy.
Thu, Oct 6, 00:38 · Infrastructure (SRE)

Wed, Oct 5

Paladox edited P386 Resources Table.
Wed, Oct 5, 19:58 · Cloud Infrastructure, Infrastructure (SRE)
Paladox edited P386 Resources Table.
Wed, Oct 5, 19:56 · Cloud Infrastructure, Infrastructure (SRE)
Paladox edited P386 Resources Table.
Wed, Oct 5, 19:55 · Cloud Infrastructure, Infrastructure (SRE)
Paladox created T9803: [New/Existing] Server Resource Request for swiftproxy131.
Wed, Oct 5, 19:43 · Infrastructure (SRE)
Paladox edited P386 Resources Table.
Wed, Oct 5, 19:19 · Cloud Infrastructure, Infrastructure (SRE)
Paladox edited P386 Resources Table.
Wed, Oct 5, 19:17 · Cloud Infrastructure, Infrastructure (SRE)
Paladox edited P386 Resources Table.
Wed, Oct 5, 19:08 · Cloud Infrastructure, Infrastructure (SRE)
Paladox closed T9802: [New/Existing] Server Resource Request for test131 as Resolved.
Wed, Oct 5, 16:37 · Infrastructure (SRE)
Paladox assigned T9802: [New/Existing] Server Resource Request for test131 to Reception123.
Wed, Oct 5, 13:16 · Infrastructure (SRE)
Paladox updated the task description for T9802: [New/Existing] Server Resource Request for test131.
Wed, Oct 5, 12:58 · Infrastructure (SRE)
Paladox updated the task description for T9802: [New/Existing] Server Resource Request for test131.
Wed, Oct 5, 12:56 · Infrastructure (SRE)
Paladox created T9802: [New/Existing] Server Resource Request for test131.
Wed, Oct 5, 12:54 · Infrastructure (SRE)

Mon, Oct 3

Paladox committed rPUPC0000ea64e891: mediawiki: add static-new (#2895) (authored by Paladox).
mediawiki: add static-new (#2895)
Mon, Oct 3, 21:14
Paladox added a reverting change for rPUPCb583b7ec9874: mediawiki: add static-new (#2895): rPUPCc820a90a2d70: Revert "mediawiki: add static-new (#2895)".
Mon, Oct 3, 21:06
Paladox committed rPUPCc820a90a2d70: Revert "mediawiki: add static-new (#2895)" (authored by Paladox).
Revert "mediawiki: add static-new (#2895)"
Mon, Oct 3, 21:06
Paladox committed rPUPC1b1584a8387e: Update mediawiki.conf.erb (authored by Paladox).
Update mediawiki.conf.erb
Mon, Oct 3, 19:57
Paladox committed rPUPCe1b4af015cd6: Update mediawiki.conf.erb (authored by Paladox).
Update mediawiki.conf.erb
Mon, Oct 3, 19:57
Paladox committed rPUPCfeb64b4c5267: Update mediawiki.conf.erb (authored by Paladox).
Update mediawiki.conf.erb
Mon, Oct 3, 19:57
Paladox committed rPUPC12e3f81e33c4: Update mediawiki.conf.erb (authored by Paladox).
Update mediawiki.conf.erb
Mon, Oct 3, 19:57
Paladox committed rPUPCb3ea16c30a01: Update mediawiki.conf.erb (authored by Paladox).
Update mediawiki.conf.erb
Mon, Oct 3, 19:56
Paladox committed rPUPCb45c1981591e: Update rewrite.py (authored by Paladox).
Update rewrite.py
Mon, Oct 3, 19:56
Paladox committed rPUPC3b038ae80612: Update rewrite.py (authored by Paladox).
Update rewrite.py
Mon, Oct 3, 19:56
Paladox committed rPUPCeeed8ab34d11: Update rewrite.py (authored by Paladox).
Update rewrite.py
Mon, Oct 3, 19:56
Paladox committed rPUPC8749dceaa29a: Update rewrite.py (authored by Paladox).
Update rewrite.py
Mon, Oct 3, 19:56
Paladox committed rPUPC439fe6145ad9: Update proxy-server.conf.erb (authored by Paladox).
Update proxy-server.conf.erb
Mon, Oct 3, 19:56
Paladox committed rPUPCab97163e2713: mediawiki: add static-new (authored by Paladox).
mediawiki: add static-new
Mon, Oct 3, 19:56

Sat, Oct 1

Paladox added a comment to T9708: [GOAL] Gluster -> Swift.
In T9708#197874, @John wrote:
In T9708#197873, @Paladox wrote:

We also probably want to adjust the load check for the object servers (should have done this to gluster) since the high load is expected because of the I/o. We need to figure out what is a good number. It doesn't seem to use a lot of cpu where as the proxies do (we may have to increase it to 4 cores or even 6).

Fine tuning is stuff we need to do once in production - fine tuning without both a working model and real traffic is a pointless exercise esp for monitoring and determining resources (when we've based this already on doc recommendations)

Sat, Oct 1, 16:40 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

We also probably want to adjust the load check for the object servers (should have done this to gluster) since the high load is expected because of the I/o. We need to figure out what is a good number. It doesn't seem to use a lot of cpu where as the proxies do (we may have to increase it to 4 cores or even 6).

Sat, Oct 1, 15:37 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Wed, Sep 28

Paladox committed rPUPC1c9bcc6353d7: Merge remote-tracking branch 'origin/master' into add-ceph (authored by Paladox).
Merge remote-tracking branch 'origin/master' into add-ceph
Wed, Sep 28, 15:35
Paladox added a reverting change for rDNS95989c1ea2a7: Depool ca: rDNS5b94071c31f4: Revert "Depool ca".
Wed, Sep 28, 13:45
Paladox committed rDNS5b94071c31f4: Revert "Depool ca" (authored by Paladox).
Revert "Depool ca"
Wed, Sep 28, 13:45
Paladox committed rDNS95989c1ea2a7: Depool ca (authored by Paladox).
Depool ca
Wed, Sep 28, 13:33
Paladox added a reverting change for rDNSfcf323e15d94: Depool gb: rDNS456fac0cfe18: Revert "Depool gb".
Wed, Sep 28, 13:33
Paladox committed rDNS456fac0cfe18: Revert "Depool gb" (authored by Paladox).
Revert "Depool gb"
Wed, Sep 28, 13:33
Paladox committed rDNSfcf323e15d94: Depool gb (authored by Paladox).
Depool gb
Wed, Sep 28, 13:24
Paladox added a reverting change for rDNS4baf99b1a8f2: Depool CA (#361): rDNSc8d66ca63da6: Revert "Depool CA (#361)".
Wed, Sep 28, 12:59
Paladox committed rDNSc8d66ca63da6: Revert "Depool CA (#361)" (authored by Paladox).
Revert "Depool CA (#361)"
Wed, Sep 28, 12:59

Mon, Sep 26

Paladox committed rPUPCb1213b4739f3: Add ceph module (authored by Paladox).
Add ceph module
Mon, Sep 26, 19:20
Paladox committed rPUPCb9e90dab9746: Add ceph module (authored by Paladox).
Add ceph module
Mon, Sep 26, 19:07
Paladox committed rPUPC6ee2f60f25cc: Add ceph module (authored by Paladox).
Add ceph module
Mon, Sep 26, 17:52

Thu, Sep 22

Paladox committed rPUPC1831f19fcc19: swift: increase memcache_max_connections to 12 for proxy (authored by Paladox).
swift: increase memcache_max_connections to 12 for proxy
Thu, Sep 22, 20:53
Paladox committed rPUPC6f4d8c51c15d: Fix (authored by Paladox).
Fix
Thu, Sep 22, 18:30
Paladox committed rPUPCa8840fcc548d: Fix (authored by Paladox).
Fix
Thu, Sep 22, 17:48
Paladox committed rPUPCf609838e6822: swift: Fix binding ip (authored by Paladox).
swift: Fix binding ip
Thu, Sep 22, 17:40
Paladox committed rPUPC34b466287243: Fix part 2 (authored by Paladox).
Fix part 2
Thu, Sep 22, 17:13
Paladox committed rPUPC97118371ad35: Fix (authored by Paladox).
Fix
Thu, Sep 22, 17:11
Paladox committed rPUPC7c150302da95: swift: Add missing container-reconciler file (authored by Paladox).
swift: Add missing container-reconciler file
Thu, Sep 22, 17:08

Wed, Sep 21

Paladox committed rPUPC16988b70fb8a: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 23:01
Paladox committed rPUPC2392fb262d69: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 22:57
Paladox committed rPUPC92a767161b0a: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 22:56
Paladox committed rPUPC4465b881837c: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 20:30
Paladox committed rPUPCfd70d2719a48: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 20:08
Paladox committed rPUPC041f330aa91a: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 20:04
Paladox committed rPUPC33b412cf7262: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 20:02
Paladox committed rPUPC68b942da8265: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 18:13
Paladox committed rPUPCd827b9830a2f: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 18:12
Paladox committed rPUPCdc2cf5ece7bd: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 13:23

Tue, Sep 20

Paladox added a reverting change for rPUPCf3deeac46f9d: Reduce PHP-FPM childs to 16 on mw121 and mw122 (#2844): rPUPC4544b83852d2: Revert "Reduce PHP-FPM childs to 16 on mw121 and mw122 (#2844)".
Tue, Sep 20, 23:20
Paladox committed rPUPC4544b83852d2: Revert "Reduce PHP-FPM childs to 16 on mw121 and mw122 (#2844)" (authored by Paladox).
Revert "Reduce PHP-FPM childs to 16 on mw121 and mw122 (#2844)"
Tue, Sep 20, 23:20

Sun, Sep 18

Paladox updated the task description for T9708: [GOAL] Gluster -> Swift.
Sun, Sep 18, 19:07 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Fri, Sep 16

Paladox committed rPUPC867df0d26a8a: Revert "varnish: force the use of ipv6 when connecting to stunnel" (authored by Paladox).
Revert "varnish: force the use of ipv6 when connecting to stunnel"
Fri, Sep 16, 21:40
Paladox added a reverting change for rPUPCa7e9090a2ff8: varnish: force the use of ipv6 when connecting to stunnel: rPUPC867df0d26a8a: Revert "varnish: force the use of ipv6 when connecting to stunnel".
Fri, Sep 16, 21:40
Paladox added a reverting change for rDNS9ce6c5ab240d: Depool ca and repool gb temporarily (#349): rDNS0412134f45e5: Revert "Depool ca and repool gb temporarily (#349)".
Fri, Sep 16, 21:40
Paladox committed rDNS0412134f45e5: Revert "Depool ca and repool gb temporarily (#349)" (authored by Paladox).
Revert "Depool ca and repool gb temporarily (#349)"
Fri, Sep 16, 21:40

Sun, Sep 11

Paladox committed rPUPCb358b3b6f80d: Update stdlib to 8.1 (authored by Paladox).
Update stdlib to 8.1
Sun, Sep 11, 15:53

Sat, Sep 10

Paladox triaged T9740: Create a internal private network for miraheze as Normal priority.
Sat, Sep 10, 01:39 · Infrastructure (SRE)

Sep 3 2022

Paladox committed rPUPCa5af7b5a1d8d: Upgrade icinga2 module to 3.4 (authored by Paladox).
Upgrade icinga2 module to 3.4
Sep 3 2022, 14:22
Paladox committed rPUPC5749aa3a9f7f: Upgrade icinga2 module to 3.4 (authored by Paladox).
Upgrade icinga2 module to 3.4
Sep 3 2022, 14:18
Paladox committed rPUPCd344377418b4: Upgrade icinga2 module to 3.4 (authored by Paladox).
Upgrade icinga2 module to 3.4
Sep 3 2022, 14:04
Paladox committed rPUPC3b8e85394a18: Upgrade icinga2 module to 3.4 (authored by Paladox).
Upgrade icinga2 module to 3.4
Sep 3 2022, 00:51

Aug 28 2022

Paladox added a comment to T9710: [Existing] Server Resource Request for db*.

Approved

Aug 28 2022, 21:38 · Notice, Database, Infrastructure (SRE)
Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

We made the above calculations with https://platform.swiftstack.com/docs/admin/hardware.html. RAM will be done on a trial and error bases.

Aug 28 2022, 18:12 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

So we’ve decided resource wise:

Aug 28 2022, 18:11 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
Paladox triaged T9708: [GOAL] Gluster -> Swift as High priority.
Aug 28 2022, 15:41 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
Paladox created T9708: [GOAL] Gluster -> Swift.
Aug 28 2022, 15:39 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Aug 16 2022

Paladox added a reverting change for rDNSa298c5eb4302: Depool cp31: rDNSb1672d52db2c: Revert "Depool cp31".
Aug 16 2022, 21:28
Paladox committed rDNSb1672d52db2c: Revert "Depool cp31" (authored by Paladox).
Revert "Depool cp31"
Aug 16 2022, 21:28
Paladox committed rDNSa298c5eb4302: Depool cp31 (authored by Paladox).
Depool cp31
Aug 16 2022, 21:22
Paladox added a reverting change for rDNSd7d9d9d8392c: Depool cp30: rDNSa019c67bcd4e: Revert "Depool cp30".
Aug 16 2022, 21:22
Paladox committed rDNSa019c67bcd4e: Revert "Depool cp30" (authored by Paladox).
Revert "Depool cp30"
Aug 16 2022, 21:21
Paladox committed rDNSd7d9d9d8392c: Depool cp30 (authored by Paladox).
Depool cp30
Aug 16 2022, 21:17

Aug 15 2022

Paladox added a comment to T9656: db141 is down.
In T9656#195073, @Universal_Omega wrote:
In T9656#195069, @Paladox wrote:

Resolved.

Can we also have an IR for this as well?

Aug 15 2022, 21:28 · Database, Infrastructure (SRE)
Paladox closed T9640: Write incident report for 180 hour intermediate outages as Resolved.
In T9640#195328, @Universal_Omega wrote:
In T9640#195325, @Paladox wrote:

https://meta.miraheze.org/wiki/Special:IncidentReports/52

That IR is not for this issue though? That was the MariaDB issue. This one was caused by Gluster.

Aug 15 2022, 21:27 · Gluster, Infrastructure (SRE)
Paladox closed T9640: Write incident report for 180 hour intermediate outages as Resolved.

https://meta.miraheze.org/wiki/Special:IncidentReports/52

Aug 15 2022, 05:32 · Gluster, Infrastructure (SRE)

Aug 13 2022

Paladox added a comment to T9657: [ACCESS REQUEST] Expanded access for Universal Omega.

Err support sorry :)

Aug 13 2022, 21:31 · Site Reliability Engineering
Paladox added a comment to T9657: [ACCESS REQUEST] Expanded access for Universal Omega.

Approved.

Aug 13 2022, 21:30 · Site Reliability Engineering

Aug 12 2022

Paladox committed rDNS8ef7ab6a765b: Depool cp20/21 (authored by Paladox).
Depool cp20/21
Aug 12 2022, 16:47
Paladox closed T9656: db141 is down as Resolved.

Resolved.

Aug 12 2022, 15:42 · Database, Infrastructure (SRE)

Aug 11 2022

Paladox committed rDNS978e42f17013: Revert "Depool cp20/21 (#328)" (authored by Paladox).
Revert "Depool cp20/21 (#328)"
Aug 11 2022, 02:26
Paladox added a reverting change for rDNSbd7aba33f5da: Depool cp20/21 (#328): rDNS978e42f17013: Revert "Depool cp20/21 (#328)".
Aug 11 2022, 02:26

Aug 10 2022

Paladox added a reverting change for R9:2b09ff8f6bdc: Don't create any new wikis on c4 (db141) temporarily (#4865): R9:044c7d43e523: Revert "Don't create any new wikis on c4 (db141) temporarily (#4865)".
Aug 10 2022, 17:09
Paladox committed R9:044c7d43e523: Revert "Don't create any new wikis on c4 (db141) temporarily (#4865)" (authored by Paladox).
Revert "Don't create any new wikis on c4 (db141) temporarily (#4865)"
Aug 10 2022, 17:09
Paladox edited P386 Resources Table.
Aug 10 2022, 01:20 · Cloud Infrastructure, Infrastructure (SRE)

Aug 9 2022

Paladox closed T9599: [Existing] Server Resource Request for prometheus as Resolved.

Resolved, thanks!

Aug 9 2022, 16:33 · Infrastructure (SRE)
Paladox edited P386 Resources Table.
Aug 9 2022, 16:30 · Cloud Infrastructure, Infrastructure (SRE)
Paladox raised the priority of T9599: [Existing] Server Resource Request for prometheus from Normal to Unbreak Now!.

Prometheus won't start now because it keeps OOM'ing, upping the priority.

Aug 9 2022, 16:13 · Infrastructure (SRE)
Paladox added a comment to T9599: [Existing] Server Resource Request for prometheus.

It's OOM'ing:

Aug 9 2022, 16:08 · Infrastructure (SRE)

Aug 4 2022

Paladox edited P386 Resources Table.
Aug 4 2022, 21:29 · Cloud Infrastructure, Infrastructure (SRE)