Page MenuHomeMiraheze

John (John Lewis)
Engineering Manager, InfrastructureAdministrator

Projects (26)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Apr 17 2016, 19:20 (336 w, 1 d)
Roles
Administrator
Availability
Available
IRC Nickname
JohnLewis
GitHub User
JohnFLewis
Miraheze User
John [ Global Accounts ]

Hi I'm John. I'm the Co-Founder of Miraheze and the Engineering Manager for the Infrastructure team.

Recent Activity

Yesterday

John committed rDNS9cd12f15480c: deploy cp32/33 to production (authored by John).
deploy cp32/33 to production
Sun, Sep 25, 19:36
John committed rPUPCc87e27302c51: varnish: deploy cp32/33 (authored by John).
varnish: deploy cp32/33
Sun, Sep 25, 18:27

Fri, Sep 23

John added a comment to T8802: Formalise and agree on MediaWiki SLOs.

How is the abandoned jobs measurement to work as it’s not clear what percentage of jobs have failed?

Fri, Sep 23, 18:51 · Universal Omega, Goal-2022-Jul-Dec, MediaWiki (SRE), Goal-2022-Jan-Jun

Wed, Sep 21

John closed T9696: ipmi and smart checks fail for cloud13/14 as Resolved.

New problems require new tasks, we shouldn’t be reopening tasks for two unrelated problems

Wed, Sep 21, 17:54 · Cloud Infrastructure, Infrastructure (SRE)

Tue, Sep 20

John added a comment to T8802: Formalise and agree on MediaWiki SLOs.

We could store and average the backend response times variable that we post in web requests - this would be a form of inexpensive latency monitoring

Tue, Sep 20, 18:31 · Universal Omega, Goal-2022-Jul-Dec, MediaWiki (SRE), Goal-2022-Jan-Jun

Sun, Sep 18

John assigned T9708: [GOAL] Gluster -> Swift to Paladox.
Sun, Sep 18, 18:51 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
John closed T9739: Create LDAP group for the SRE access level as Resolved.
Sun, Sep 18, 18:03 · Infrastructure (SRE)
John added a comment to T9739: Create LDAP group for the SRE access level.

To be honest, I'm thinking we should use sre-infrastructure more broadly, for Infrastructure matters so I don't see a problem really with add you to this group as you have the same access and responsibilities as anyone in sre-infra, just not the primary focus. Anything relevant for them would be relevant for you too.

Sun, Sep 18, 17:59 · Infrastructure (SRE)
John closed T9214: Allow LucidChart iframes in CSP as Resolved.
Sun, Sep 18, 17:53 · Trust & Safety, CSP Review, MediaWiki (SRE)
John closed T9252: bilibili.com CSP whitelist as Declined.

Unauthorized personally identifying information should be reported to government, they have pages that specifically handle these stuffs as I said above. I don't think Bilibili do have enough privilege to delete user data without their consent unless it's government issued.

Sun, Sep 18, 17:48 · Trust & Safety, CSP Review, MediaWiki (SRE)
John moved T9214: Allow LucidChart iframes in CSP from EM Review to Pending Addition on the CSP Review board.
Sun, Sep 18, 17:46 · Trust & Safety, CSP Review, MediaWiki (SRE)

Sat, Sep 10

John closed T9740: Create a internal private network for miraheze as Declined.

We had this conversation on setup and when doing initial testing, we decided to make use of IPv6 to prevent needing to recreate a network topology we couldn’t maintain due to cost and limited power access.

Sat, Sep 10, 06:17 · Infrastructure (SRE)

Fri, Sep 9

John assigned T9739: Create LDAP group for the SRE access level to Paladox.

Can we create a non-email based sre-roots for this please?

Fri, Sep 9, 20:17 · Infrastructure (SRE)

Sat, Sep 3

John committed rPUPC2d0995f715b5: stunnel: make monitoring less intensive (authored by John).
stunnel: make monitoring less intensive
Sat, Sep 3, 21:26
John committed rPUPC16c0d4b5c8c4: add minor stunnel config changes (authored by John).
add minor stunnel config changes
Sat, Sep 3, 18:50

Thu, Sep 1

John updated the task description for T8801: Formalise and agree on Infrastructure SLOs.
Thu, Sep 1, 18:16 · Goal-2022-Jul-Dec, Goal-2022-Jan-Jun, Infrastructure (SRE)

Tue, Aug 30

John closed T9710: [Existing] Server Resource Request for db* as Resolved.
Tue, Aug 30, 20:53 · Notice, Database, Infrastructure (SRE)
John edited P386 Resources Table.
Tue, Aug 30, 19:33 · Cloud Infrastructure, Infrastructure (SRE)

Mon, Aug 29

John added a comment to T9712: mw-config master is broken .
In T9712#196193, @John wrote:

"Was force pushed without waiting for CI by John @ 19:56"

It wasn't forced pushed - it was committed?

Clarification: CI there to point out exactly this was bypassed by merging without a PR and then when post merged failed completely ignored for about 10 minutes

Mon, Aug 29, 21:39 · Configuration, MediaWiki (SRE)
John committed rPUPC623f3c1699b6: prometheus: move varnish stats to node.d (authored by John).
prometheus: move varnish stats to node.d
Mon, Aug 29, 21:37
John committed rPUPCb79f14ee3372: add varnishrequests exporter (authored by John).
add varnishrequests exporter
Mon, Aug 29, 21:21
John added a project to T9710: [Existing] Server Resource Request for db*: Notice.

@Agent_Isai Can this be announced for tomorrow 2000-2030 please?

Mon, Aug 29, 19:22 · Notice, Database, Infrastructure (SRE)
John added a comment to T9712: mw-config master is broken .

https://github.com/miraheze/mw-config/commit/212136fee36e73f4c6d8fe1dd022a8af211dedab is where behaviour changes and likely the source of the breaking change

Mon, Aug 29, 19:21 · Configuration, MediaWiki (SRE)
John closed T9712: mw-config master is broken as Resolved.

"Was force pushed without waiting for CI by John @ 19:56"

Mon, Aug 29, 19:19 · Configuration, MediaWiki (SRE)
John moved T9710: [Existing] Server Resource Request for db* from Incoming to Short Term on the Infrastructure (SRE) board.
Mon, Aug 29, 18:52 · Notice, Database, Infrastructure (SRE)
John added a comment to T9710: [Existing] Server Resource Request for db*.

Changes have been applied, but all db servers need a reboot. Will need to schedule some time in for this.

Mon, Aug 29, 18:45 · Notice, Database, Infrastructure (SRE)
John edited P386 Resources Table.
Mon, Aug 29, 18:44 · Cloud Infrastructure, Infrastructure (SRE)
John edited P386 Resources Table.
Mon, Aug 29, 18:38 · Cloud Infrastructure, Infrastructure (SRE)

Sun, Aug 28

John created T9710: [Existing] Server Resource Request for db*.
Sun, Aug 28, 21:18 · Notice, Database, Infrastructure (SRE)
John closed T9696: ipmi and smart checks fail for cloud13/14 as Resolved.
Sun, Aug 28, 16:16 · Cloud Infrastructure, Infrastructure (SRE)
John lowered the priority of T9708: [GOAL] Gluster -> Swift from High to Normal.
Sun, Aug 28, 15:41 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
John moved T9708: [GOAL] Gluster -> Swift from Incoming to Goals on the Infrastructure (SRE) board.
Sun, Aug 28, 15:40 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
John moved T9708: [GOAL] Gluster -> Swift from Backlog to Infrastructure on the Goal-2022-Jul-Dec board.
Sun, Aug 28, 15:40 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)
John added a project to T9708: [GOAL] Gluster -> Swift: Gluster.
Sun, Aug 28, 15:40 · Gluster, Goal-2022-Jul-Dec, Infrastructure (SRE)

Aug 26 2022

John removed a project from T9697: Continue investigating 502s, 503s and intermittent slow load times: Infrastructure (SRE).

(-Infra as tasks shouldn’t be dual tagged for responsibility)

Aug 26 2022, 11:32 · Universal Omega, Performance, MediaWiki (SRE)

Aug 25 2022

John triaged T9696: ipmi and smart checks fail for cloud13/14 as Normal priority.
Aug 25 2022, 17:45 · Cloud Infrastructure, Infrastructure (SRE)

Aug 21 2022

John updated the task description for T8801: Formalise and agree on Infrastructure SLOs.
Aug 21 2022, 20:48 · Goal-2022-Jul-Dec, Goal-2022-Jan-Jun, Infrastructure (SRE)
John updated the task description for T8801: Formalise and agree on Infrastructure SLOs.
Aug 21 2022, 19:51 · Goal-2022-Jul-Dec, Goal-2022-Jan-Jun, Infrastructure (SRE)
John added a comment to T8801: Formalise and agree on Infrastructure SLOs.
  • DNS
    • SLO: Latency for DNS lookups to be below 5ms at least 99.5%
Aug 21 2022, 19:51 · Goal-2022-Jul-Dec, Goal-2022-Jan-Jun, Infrastructure (SRE)
John committed rPUPCb8527546f94e: prometheus: fix python variable (authored by John).
prometheus: fix python variable
Aug 21 2022, 19:36
John committed rPUPC40f1c1e180c6: dns: add latency to prom export (authored by John).
dns: add latency to prom export
Aug 21 2022, 19:21

Aug 20 2022

John closed T9675: Spam bios on Phabricator as Resolved.
Aug 20 2022, 19:34 · Infrastructure (SRE), Phabricator

Aug 14 2022

John awarded T9657: [ACCESS REQUEST] Expanded access for Universal Omega a Like token.
Aug 14 2022, 18:20 · Site Reliability Engineering

Aug 13 2022

John updated subscribers of T9657: [ACCESS REQUEST] Expanded access for Universal Omega.
Aug 13 2022, 20:26 · Site Reliability Engineering
John added a comment to T9195: Attempt to improve ManageWiki usability and discovery.

This feels like am infinitely defined task - where does the scope end? What's the end goal? I think this should be split up into tasks for each part so they can be tracked, fleshed out and opinions can be sought in a more defined manner.

Aug 13 2022, 18:43 · Universal Omega, Notice, ManageWiki, MediaWiki (SRE)

Aug 9 2022

John added a comment to T9599: [Existing] Server Resource Request for prometheus.

Just for the record, this was informally already approved as was communicated on IRC a few weeks ago :)

Aug 9 2022, 16:52 · Infrastructure (SRE)

Aug 7 2022

John assigned T9640: Write incident report for 180 hour intermediate outages to Paladox.

As best person is position to do this.

Aug 7 2022, 18:14 · Gluster, Infrastructure (SRE)

Aug 2 2022

John updated the task description for T8801: Formalise and agree on Infrastructure SLOs.
Aug 2 2022, 18:38 · Goal-2022-Jul-Dec, Goal-2022-Jan-Jun, Infrastructure (SRE)

Aug 1 2022

John committed rPUPCd06753bf9cc0: reports: upgrade to v6 (authored by John).
reports: upgrade to v6
Aug 1 2022, 20:37

Jul 30 2022

John committed rDNS1eea61b4d21f: add cloud13/14 (authored by John).
add cloud13/14
Jul 30 2022, 22:00
John committed rPUPCc07035919da6: cloud: add 13 and 14 (authored by John).
cloud: add 13 and 14
Jul 30 2022, 21:56

Jul 29 2022

John closed T9601: 29 July 2022 NIC cloud10<->switch link flap as Resolved.

syslog suggests the networking service on cloud10 was restarted

Jul 29 2022, 20:29 · Infrastructure (SRE), Cloud Infrastructure

Jul 28 2022

John placed T9598: [ACCESS REQUEST] Expanded access for cache-admins up for grabs.

No objections from me

Jul 28 2022, 20:33 · Site Reliability Engineering
John triaged T8911: Searching fails with hyphens as Normal priority.
Jul 28 2022, 18:38 · MediaWiki (SRE), MediaWiki
John triaged T8958: Suddenly getting Site error when trying to access and edit my wiki as Normal priority.
Jul 28 2022, 18:38 · Database, Infrastructure (SRE)
John triaged T9237: Feature request: replicate the FandomDesktop/FandomMobile skins as Normal priority.
Jul 28 2022, 18:38 · Extensions, MediaWiki (SRE)
John triaged T9568: Import request for: tlhmupaq.miraheze.org as Normal priority.
Jul 28 2022, 18:38 · MediaWiki (SRE), MediaWiki
John added projects to T9413: MSUpload shifts image names by 1: MediaWiki (SRE), MediaWiki.
Jul 28 2022, 18:37 · MediaWiki, MediaWiki (SRE)
John added projects to T8864: It's finished: MediaWiki (SRE), MediaWiki.
Jul 28 2022, 18:37 · MediaWiki, MediaWiki (SRE)
John added projects to T8995: Delete Wiki: MediaWiki (SRE), MediaWiki.
Jul 28 2022, 18:37 · MediaWiki, MediaWiki (SRE)
John added projects to T9039: ...: MediaWiki, MediaWiki (SRE).
Jul 28 2022, 18:37 · MediaWiki (SRE), MediaWiki
John added projects to T9404: Allow bureaucrats to retire without needing a Steward's help: MediaWiki, MediaWiki (SRE).
Jul 28 2022, 18:36 · MediaWiki (SRE), MediaWiki
John added projects to T9551: Request adding settings for PageImages extension: MediaWiki (SRE), Configuration.
Jul 28 2022, 18:36 · Universal Omega, Configuration, MediaWiki (SRE)
John added projects to T8945: Fatal exception of type "TypeError" in ManageWiki: MediaWiki (SRE), Configuration.
Jul 28 2022, 18:35 · Configuration, MediaWiki (SRE)
John added projects to T9066: Login Fatal Exception: MediaWiki, MediaWiki (SRE).
Jul 28 2022, 18:34 · MediaWiki (SRE), MediaWiki
John added projects to T9325: URL Decouple: MediaWiki (SRE), MediaWiki.
Jul 28 2022, 18:34 · MediaWiki, MediaWiki (SRE)
John triaged T8882: Clear notifications for User:Magogre as Normal priority.
Jul 28 2022, 18:33 · MediaWiki, MediaWiki (SRE)
John triaged T8906: Internal Error (?) as Normal priority.
Jul 28 2022, 18:33 · MediaWiki, MediaWiki (SRE)
John triaged T9034: Make my wiki private/invite only as Normal priority.
Jul 28 2022, 18:33 · MediaWiki, MediaWiki (SRE)
John triaged T9035: Html not working on this page as Normal priority.
Jul 28 2022, 18:33 · MediaWiki, MediaWiki (SRE)
John triaged T9082: Update on importing as Normal priority.
Jul 28 2022, 18:32 · MediaWiki, MediaWiki (SRE)
John triaged T9134: Multilingual Wiki as Normal priority.
Jul 28 2022, 18:32 · MediaWiki (SRE), MediaWiki
John triaged T9436: Special:Preference page throws DomainException as Normal priority.
Jul 28 2022, 18:32 · MediaWiki (SRE), MediaWiki
John triaged T9593: I cannot access a wiki as Normal priority.
Jul 28 2022, 18:31 · MediaWiki, MediaWiki (SRE)
John triaged T9596: I can't access various pages on Greatest Movies Wiki as Normal priority.
Jul 28 2022, 18:31 · Production Error, MediaWiki, MediaWiki (SRE)
John moved T8470: Rack equipment cloud1[012] and switch from Cloud to SCSVG on the Cloud Infrastructure board.
Jul 28 2022, 17:49 · Cloud Infrastructure, Infrastructure (SRE)
John moved T8583: Investigate cloud12 reboot from Cloud to SCSVG on the Cloud Infrastructure board.
Jul 28 2022, 17:49 · Infrastructure (SRE), Cloud Infrastructure
John moved T9478: Add monitoring for high MariaDB connections from Incoming to Short Term on the Infrastructure (SRE) board.
Jul 28 2022, 17:48 · Monitoring, Infrastructure (SRE), Database
John moved T9481: Phabricator loads slowly from Incoming to Short Term on the Infrastructure (SRE) board.
Jul 28 2022, 17:48 · Infrastructure (SRE), Phabricator
John edited P386 Resources Table.
Jul 28 2022, 17:44 · Cloud Infrastructure, Infrastructure (SRE)
John edited P386 Resources Table.
Jul 28 2022, 17:43 · Cloud Infrastructure, Infrastructure (SRE)
John closed T9597: [New/Existing] Server Resource Request for [Service] as Resolved.

Approved as a transitionary measure

Jul 28 2022, 17:42 · Infrastructure (SRE)

Jul 27 2022

John merged T9591: Error 503 Backend Fetch Failed when I try to access Requests for Comment/Global bans into T9581: Frequent 503s during past few days.
Jul 27 2022, 18:28 · Performance, MediaWiki (SRE), MediaWiki
John merged task T9591: Error 503 Backend Fetch Failed when I try to access Requests for Comment/Global bans into T9581: Frequent 503s during past few days.
Jul 27 2022, 18:28 · Site Reliability Engineering

Jul 25 2022

John added a comment to T9033: Decide whether to switch to opensearch as graylog seems to not be supporting versions passed 7.10.

@Paladox Is this something for us to do with switching servers?

Jul 25 2022, 21:50 · Infrastructure (SRE)

Jul 19 2022

John placed T9252: bilibili.com CSP whitelist up for grabs.

-> EM Review to take into account both SRE and T&S review

Jul 19 2022, 16:27 · Trust & Safety, CSP Review, MediaWiki (SRE)

Jul 3 2022

Dmehus awarded T9268: [Existing] Server Resource Request for db112 a Haypence token.
Jul 3 2022, 01:57 · Infrastructure (SRE)

Jul 2 2022

John updated the task description for T8801: Formalise and agree on Infrastructure SLOs.
Jul 2 2022, 23:03 · Goal-2022-Jul-Dec, Goal-2022-Jan-Jun, Infrastructure (SRE)
John added a comment to T8801: Formalise and agree on Infrastructure SLOs.
  • Mail
    • SLO: Availability of Mail servers to be at least 99.5%.
    • SLO: Errors for sending mail is below 1%.
    • SLO: Latency of message delivery is below 30 seconds.
  • MariaDB
    • SLO: Availability of MariaDB is at least 99.5%.
    • SLO: Error rates for access are below 5%.
  • LDAP
    • SLO: Availability of LDAP to be at least 99.5%.
Jul 2 2022, 22:24 · Goal-2022-Jul-Dec, Goal-2022-Jan-Jun, Infrastructure (SRE)
John added a comment to T8801: Formalise and agree on Infrastructure SLOs.

Proposal for some below are:

Jul 2 2022, 19:30 · Goal-2022-Jul-Dec, Goal-2022-Jan-Jun, Infrastructure (SRE)
John moved T8801: Formalise and agree on Infrastructure SLOs from Backlog to Infrastructure on the Goal-2022-Jul-Dec board.
Jul 2 2022, 17:31 · Goal-2022-Jul-Dec, Goal-2022-Jan-Jun, Infrastructure (SRE)
John moved T8802: Formalise and agree on MediaWiki SLOs from Backlog to MediaWiki on the Goal-2022-Jul-Dec board.
Jul 2 2022, 17:31 · Universal Omega, Goal-2022-Jul-Dec, MediaWiki (SRE), Goal-2022-Jan-Jun
John added a project to T8802: Formalise and agree on MediaWiki SLOs: Goal-2022-Jul-Dec.
Jul 2 2022, 17:31 · Universal Omega, Goal-2022-Jul-Dec, MediaWiki (SRE), Goal-2022-Jan-Jun
John added a project to T8801: Formalise and agree on Infrastructure SLOs: Goal-2022-Jul-Dec.
Jul 2 2022, 17:31 · Goal-2022-Jul-Dec, Goal-2022-Jan-Jun, Infrastructure (SRE)
John archived Goal-2022-Jan-Jun.
Jul 2 2022, 17:31
John added a comment to T9481: Phabricator loads slowly.

It’s very likely related, though I’ll note I’ve never experienced this problem

Jul 2 2022, 12:19 · Infrastructure (SRE), Phabricator
John closed T9268: [Existing] Server Resource Request for db112 as Declined.
Jul 2 2022, 12:18 · Infrastructure (SRE)

Jun 25 2022

John added a comment to T9425: Investigate recent outages.

Because there's no MTRs showing the problem or any other evidence of this issue, closing as declined as it is untraceable.

Jun 25 2022, 13:57 · Infrastructure (SRE)
John closed T9309: Increase Roundcube session timeout as Resolved.
Jun 25 2022, 13:51 · Mail, Infrastructure (SRE)
John moved T9430: [Existing] Server Resource Request for matomo101 from Incoming to Short Term on the Infrastructure (SRE) board.
Jun 25 2022, 13:49 · Infrastructure (SRE)