Page MenuHomeMiraheze

Paladox
Site Reliabilty EngineerAdministrator

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Oct 14 2017, 11:40 (259 w, 6 d)
Roles
Administrator
Availability
Available
GitHub User
Unknown
Miraheze User
Paladox [ Global Accounts ]

I live in the United Kingdom GMT/BST(UTC+0, UTC+1).

I am a Site Reliability Engineer for miraheze.

Recent Activity

Yesterday

Paladox assigned T9804: [New/Existing] Server Resource Request for swift proxy to John.
Thu, Oct 6, 11:06 · Infrastructure (SRE)
Paladox created T9804: [New/Existing] Server Resource Request for swift proxy.
Thu, Oct 6, 00:38 · Infrastructure (SRE)

Wed, Oct 5

Paladox edited P386 Resources Table.
Wed, Oct 5, 19:58 · Cloud Infrastructure, Infrastructure (SRE)
Paladox edited P386 Resources Table.
Wed, Oct 5, 19:56 · Cloud Infrastructure, Infrastructure (SRE)
Paladox edited P386 Resources Table.
Wed, Oct 5, 19:55 · Cloud Infrastructure, Infrastructure (SRE)
Paladox created T9803: [New/Existing] Server Resource Request for swiftproxy131.
Wed, Oct 5, 19:43 · Infrastructure (SRE)
Paladox edited P386 Resources Table.
Wed, Oct 5, 19:19 · Cloud Infrastructure, Infrastructure (SRE)
Paladox edited P386 Resources Table.
Wed, Oct 5, 19:17 · Cloud Infrastructure, Infrastructure (SRE)
Paladox edited P386 Resources Table.
Wed, Oct 5, 19:08 · Cloud Infrastructure, Infrastructure (SRE)
Paladox closed T9802: [New/Existing] Server Resource Request for test131 as Resolved.
Wed, Oct 5, 16:37 · Infrastructure (SRE)
Paladox assigned T9802: [New/Existing] Server Resource Request for test131 to Reception123.
Wed, Oct 5, 13:16 · Infrastructure (SRE)
Paladox updated the task description for T9802: [New/Existing] Server Resource Request for test131.
Wed, Oct 5, 12:58 · Infrastructure (SRE)
Paladox updated the task description for T9802: [New/Existing] Server Resource Request for test131.
Wed, Oct 5, 12:56 · Infrastructure (SRE)
Paladox created T9802: [New/Existing] Server Resource Request for test131.
Wed, Oct 5, 12:54 · Infrastructure (SRE)

Mon, Oct 3

Paladox committed rPUPC0000ea64e891: mediawiki: add static-new (#2895) (authored by Paladox).
mediawiki: add static-new (#2895)
Mon, Oct 3, 21:14
Paladox added a reverting change for rPUPCb583b7ec9874: mediawiki: add static-new (#2895): rPUPCc820a90a2d70: Revert "mediawiki: add static-new (#2895)".
Mon, Oct 3, 21:06
Paladox committed rPUPCc820a90a2d70: Revert "mediawiki: add static-new (#2895)" (authored by Paladox).
Revert "mediawiki: add static-new (#2895)"
Mon, Oct 3, 21:06
Paladox committed rPUPC1b1584a8387e: Update mediawiki.conf.erb (authored by Paladox).
Update mediawiki.conf.erb
Mon, Oct 3, 19:57
Paladox committed rPUPCe1b4af015cd6: Update mediawiki.conf.erb (authored by Paladox).
Update mediawiki.conf.erb
Mon, Oct 3, 19:57
Paladox committed rPUPCfeb64b4c5267: Update mediawiki.conf.erb (authored by Paladox).
Update mediawiki.conf.erb
Mon, Oct 3, 19:57
Paladox committed rPUPC12e3f81e33c4: Update mediawiki.conf.erb (authored by Paladox).
Update mediawiki.conf.erb
Mon, Oct 3, 19:57
Paladox committed rPUPCb3ea16c30a01: Update mediawiki.conf.erb (authored by Paladox).
Update mediawiki.conf.erb
Mon, Oct 3, 19:56
Paladox committed rPUPCb45c1981591e: Update rewrite.py (authored by Paladox).
Update rewrite.py
Mon, Oct 3, 19:56
Paladox committed rPUPC3b038ae80612: Update rewrite.py (authored by Paladox).
Update rewrite.py
Mon, Oct 3, 19:56
Paladox committed rPUPCeeed8ab34d11: Update rewrite.py (authored by Paladox).
Update rewrite.py
Mon, Oct 3, 19:56
Paladox committed rPUPC8749dceaa29a: Update rewrite.py (authored by Paladox).
Update rewrite.py
Mon, Oct 3, 19:56
Paladox committed rPUPC439fe6145ad9: Update proxy-server.conf.erb (authored by Paladox).
Update proxy-server.conf.erb
Mon, Oct 3, 19:56
Paladox committed rPUPCab97163e2713: mediawiki: add static-new (authored by Paladox).
mediawiki: add static-new
Mon, Oct 3, 19:56

Sat, Oct 1

Paladox added a comment to T9708: [GOAL] Gluster -> Swift.
In T9708#197874, @John wrote:

We also probably want to adjust the load check for the object servers (should have done this to gluster) since the high load is expected because of the I/o. We need to figure out what is a good number. It doesn't seem to use a lot of cpu where as the proxies do (we may have to increase it to 4 cores or even 6).

Fine tuning is stuff we need to do once in production - fine tuning without both a working model and real traffic is a pointless exercise esp for monitoring and determining resources (when we've based this already on doc recommendations)

Sat, Oct 1, 16:40 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

We also probably want to adjust the load check for the object servers (should have done this to gluster) since the high load is expected because of the I/o. We need to figure out what is a good number. It doesn't seem to use a lot of cpu where as the proxies do (we may have to increase it to 4 cores or even 6).

Sat, Oct 1, 15:37 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Wed, Sep 28

Paladox committed rPUPC1c9bcc6353d7: Merge remote-tracking branch 'origin/master' into add-ceph (authored by Paladox).
Merge remote-tracking branch 'origin/master' into add-ceph
Wed, Sep 28, 15:35
Paladox added a reverting change for rDNS95989c1ea2a7: Depool ca: rDNS5b94071c31f4: Revert "Depool ca".
Wed, Sep 28, 13:45
Paladox committed rDNS5b94071c31f4: Revert "Depool ca" (authored by Paladox).
Revert "Depool ca"
Wed, Sep 28, 13:45
Paladox committed rDNS95989c1ea2a7: Depool ca (authored by Paladox).
Depool ca
Wed, Sep 28, 13:33
Paladox added a reverting change for rDNSfcf323e15d94: Depool gb: rDNS456fac0cfe18: Revert "Depool gb".
Wed, Sep 28, 13:33
Paladox committed rDNS456fac0cfe18: Revert "Depool gb" (authored by Paladox).
Revert "Depool gb"
Wed, Sep 28, 13:33
Paladox committed rDNSfcf323e15d94: Depool gb (authored by Paladox).
Depool gb
Wed, Sep 28, 13:24
Paladox added a reverting change for rDNS4baf99b1a8f2: Depool CA (#361): rDNSc8d66ca63da6: Revert "Depool CA (#361)".
Wed, Sep 28, 12:59
Paladox committed rDNSc8d66ca63da6: Revert "Depool CA (#361)" (authored by Paladox).
Revert "Depool CA (#361)"
Wed, Sep 28, 12:59

Mon, Sep 26

Paladox committed rPUPCb1213b4739f3: Add ceph module (authored by Paladox).
Add ceph module
Mon, Sep 26, 19:20
Paladox committed rPUPCb9e90dab9746: Add ceph module (authored by Paladox).
Add ceph module
Mon, Sep 26, 19:07
Paladox committed rPUPC6ee2f60f25cc: Add ceph module (authored by Paladox).
Add ceph module
Mon, Sep 26, 17:52

Thu, Sep 22

Paladox committed rPUPC1831f19fcc19: swift: increase memcache_max_connections to 12 for proxy (authored by Paladox).
swift: increase memcache_max_connections to 12 for proxy
Thu, Sep 22, 20:53
Paladox committed rPUPC6f4d8c51c15d: Fix (authored by Paladox).
Fix
Thu, Sep 22, 18:30
Paladox committed rPUPCa8840fcc548d: Fix (authored by Paladox).
Fix
Thu, Sep 22, 17:48
Paladox committed rPUPCf609838e6822: swift: Fix binding ip (authored by Paladox).
swift: Fix binding ip
Thu, Sep 22, 17:40
Paladox committed rPUPC34b466287243: Fix part 2 (authored by Paladox).
Fix part 2
Thu, Sep 22, 17:13
Paladox committed rPUPC97118371ad35: Fix (authored by Paladox).
Fix
Thu, Sep 22, 17:11
Paladox committed rPUPC7c150302da95: swift: Add missing container-reconciler file (authored by Paladox).
swift: Add missing container-reconciler file
Thu, Sep 22, 17:08

Wed, Sep 21

Paladox committed rPUPC16988b70fb8a: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 23:01
Paladox committed rPUPC2392fb262d69: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 22:57
Paladox committed rPUPC92a767161b0a: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 22:56
Paladox committed rPUPC4465b881837c: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 20:30
Paladox committed rPUPCfd70d2719a48: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 20:08
Paladox committed rPUPC041f330aa91a: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 20:04
Paladox committed rPUPC33b412cf7262: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 20:02
Paladox committed rPUPC68b942da8265: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 18:13
Paladox committed rPUPCd827b9830a2f: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 18:12
Paladox committed rPUPCdc2cf5ece7bd: Add swift module (authored by Paladox).
Add swift module
Wed, Sep 21, 13:23

Tue, Sep 20

Paladox added a reverting change for rPUPCf3deeac46f9d: Reduce PHP-FPM childs to 16 on mw121 and mw122 (#2844): rPUPC4544b83852d2: Revert "Reduce PHP-FPM childs to 16 on mw121 and mw122 (#2844)".
Tue, Sep 20, 23:20
Paladox committed rPUPC4544b83852d2: Revert "Reduce PHP-FPM childs to 16 on mw121 and mw122 (#2844)" (authored by Paladox).
Revert "Reduce PHP-FPM childs to 16 on mw121 and mw122 (#2844)"
Tue, Sep 20, 23:20

Sun, Sep 18

Paladox updated the task description for T9708: [GOAL] Gluster -> Swift.
Sun, Sep 18, 19:07 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Fri, Sep 16

Paladox committed rPUPC867df0d26a8a: Revert "varnish: force the use of ipv6 when connecting to stunnel" (authored by Paladox).
Revert "varnish: force the use of ipv6 when connecting to stunnel"
Fri, Sep 16, 21:40
Paladox added a reverting change for rPUPCa7e9090a2ff8: varnish: force the use of ipv6 when connecting to stunnel: rPUPC867df0d26a8a: Revert "varnish: force the use of ipv6 when connecting to stunnel".
Fri, Sep 16, 21:40
Paladox added a reverting change for rDNS9ce6c5ab240d: Depool ca and repool gb temporarily (#349): rDNS0412134f45e5: Revert "Depool ca and repool gb temporarily (#349)".
Fri, Sep 16, 21:40
Paladox committed rDNS0412134f45e5: Revert "Depool ca and repool gb temporarily (#349)" (authored by Paladox).
Revert "Depool ca and repool gb temporarily (#349)"
Fri, Sep 16, 21:40

Sun, Sep 11

Paladox committed rPUPCb358b3b6f80d: Update stdlib to 8.1 (authored by Paladox).
Update stdlib to 8.1
Sun, Sep 11, 15:53

Sat, Sep 10

Paladox triaged T9740: Create a internal private network for miraheze as Normal priority.
Sat, Sep 10, 01:39 · Infrastructure (SRE)

Sep 3 2022

Paladox committed rPUPCa5af7b5a1d8d: Upgrade icinga2 module to 3.4 (authored by Paladox).
Upgrade icinga2 module to 3.4
Sep 3 2022, 14:22
Paladox committed rPUPC5749aa3a9f7f: Upgrade icinga2 module to 3.4 (authored by Paladox).
Upgrade icinga2 module to 3.4
Sep 3 2022, 14:18
Paladox committed rPUPCd344377418b4: Upgrade icinga2 module to 3.4 (authored by Paladox).
Upgrade icinga2 module to 3.4
Sep 3 2022, 14:04
Paladox committed rPUPC3b8e85394a18: Upgrade icinga2 module to 3.4 (authored by Paladox).
Upgrade icinga2 module to 3.4
Sep 3 2022, 00:51

Aug 28 2022

Paladox added a comment to T9710: [Existing] Server Resource Request for db*.

Approved

Aug 28 2022, 21:38 · Notice, Database, Infrastructure (SRE)
Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

We made the above calculations with https://platform.swiftstack.com/docs/admin/hardware.html. RAM will be done on a trial and error bases.

Aug 28 2022, 18:12 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
Paladox added a comment to T9708: [GOAL] Gluster -> Swift.

So we’ve decided resource wise:

Aug 28 2022, 18:11 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
Paladox triaged T9708: [GOAL] Gluster -> Swift as High priority.
Aug 28 2022, 15:41 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
Paladox created T9708: [GOAL] Gluster -> Swift.
Aug 28 2022, 15:39 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Aug 16 2022

Paladox added a reverting change for rDNSa298c5eb4302: Depool cp31: rDNSb1672d52db2c: Revert "Depool cp31".
Aug 16 2022, 21:28
Paladox committed rDNSb1672d52db2c: Revert "Depool cp31" (authored by Paladox).
Revert "Depool cp31"
Aug 16 2022, 21:28
Paladox committed rDNSa298c5eb4302: Depool cp31 (authored by Paladox).
Depool cp31
Aug 16 2022, 21:22
Paladox added a reverting change for rDNSd7d9d9d8392c: Depool cp30: rDNSa019c67bcd4e: Revert "Depool cp30".
Aug 16 2022, 21:22
Paladox committed rDNSa019c67bcd4e: Revert "Depool cp30" (authored by Paladox).
Revert "Depool cp30"
Aug 16 2022, 21:21
Paladox committed rDNSd7d9d9d8392c: Depool cp30 (authored by Paladox).
Depool cp30
Aug 16 2022, 21:17

Aug 15 2022

Paladox added a comment to T9656: db141 is down.

Resolved.

Can we also have an IR for this as well?

Aug 15 2022, 21:28 · Database, Infrastructure (SRE)
Paladox closed T9640: Write incident report for 180 hour intermediate outages as Resolved.

That IR is not for this issue though? That was the MariaDB issue. This one was caused by Gluster.

Aug 15 2022, 21:27 · Gluster, Infrastructure (SRE)
Paladox closed T9640: Write incident report for 180 hour intermediate outages as Resolved.

https://meta.miraheze.org/wiki/Special:IncidentReports/52

Aug 15 2022, 05:32 · Gluster, Infrastructure (SRE)

Aug 13 2022

Paladox added a comment to T9657: [ACCESS REQUEST] Expanded access for Universal Omega.

Err support sorry :)

Aug 13 2022, 21:31 · Site Reliability Engineering
Paladox added a comment to T9657: [ACCESS REQUEST] Expanded access for Universal Omega.

Approved.

Aug 13 2022, 21:30 · Site Reliability Engineering

Aug 12 2022

Paladox committed rDNS8ef7ab6a765b: Depool cp20/21 (authored by Paladox).
Depool cp20/21
Aug 12 2022, 16:47
Paladox closed T9656: db141 is down as Resolved.

Resolved.

Aug 12 2022, 15:42 · Database, Infrastructure (SRE)

Aug 11 2022

Paladox committed rDNS978e42f17013: Revert "Depool cp20/21 (#328)" (authored by Paladox).
Revert "Depool cp20/21 (#328)"
Aug 11 2022, 02:26
Paladox added a reverting change for rDNSbd7aba33f5da: Depool cp20/21 (#328): rDNS978e42f17013: Revert "Depool cp20/21 (#328)".
Aug 11 2022, 02:26

Aug 10 2022

Paladox added a reverting change for R9:2b09ff8f6bdc: Don't create any new wikis on c4 (db141) temporarily (#4865): R9:044c7d43e523: Revert "Don't create any new wikis on c4 (db141) temporarily (#4865)".
Aug 10 2022, 17:09
Paladox committed R9:044c7d43e523: Revert "Don't create any new wikis on c4 (db141) temporarily (#4865)" (authored by Paladox).
Revert "Don't create any new wikis on c4 (db141) temporarily (#4865)"
Aug 10 2022, 17:09
Paladox edited P386 Resources Table.
Aug 10 2022, 01:20 · Cloud Infrastructure, Infrastructure (SRE)

Aug 9 2022

Paladox closed T9599: [Existing] Server Resource Request for prometheus as Resolved.

Resolved, thanks!

Aug 9 2022, 16:33 · Infrastructure (SRE)
Paladox edited P386 Resources Table.
Aug 9 2022, 16:30 · Cloud Infrastructure, Infrastructure (SRE)
Paladox raised the priority of T9599: [Existing] Server Resource Request for prometheus from Normal to Unbreak Now!.

Prometheus won't start now because it keeps OOM'ing, upping the priority.

Aug 9 2022, 16:13 · Infrastructure (SRE)
Paladox added a comment to T9599: [Existing] Server Resource Request for prometheus.

It's OOM'ing:

Aug 9 2022, 16:08 · Infrastructure (SRE)

Aug 4 2022

Paladox edited P386 Resources Table.
Aug 4 2022, 21:29 · Cloud Infrastructure, Infrastructure (SRE)