Page MenuHomeMiraheze

John (John Lewis)
Engineering Manager, Infrastructure, Site Reliability EngineeringAdministrator

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Apr 17 2016, 19:20 (254 w, 4 d)
Roles
Administrator
Availability
Available
IRC Nickname
JohnLewis
GitHub User
JohnFLewis
Miraheze User
John [ Global Accounts ]

Hi I'm John. I'm the Co-Founder of Miraheze, Engineering Manager for the Infrastructure team and a Steward.

If you don't know who to go to about something, then the answer is probably me or I'll know (hint, it's probably me!).

And if it's about any of the following, you're better of coming to me because these are my areas of expertise:
DNS
Mail

Recent Activity

Wed, Mar 3

John assigned T6922: Approved Revisions extension stopped working to Paladox.

Another issue caused by deployers merging without reviewing the changes being deployed. I've ask @Paladox to run the SQL.

Wed, Mar 3, 15:52 · MediaWiki (SRE), Extensions, Database, Universal Omega

Tue, Mar 2

John closed T6909: Mail seems very lagged as Resolved.
Tue, Mar 2, 20:14 · Infrastructure (SRE), Mail
John claimed T6909: Mail seems very lagged.
Tue, Mar 2, 16:30 · Infrastructure (SRE), Mail

Mon, Mar 1

John changed the visibility for T6917: Internal server error if user object includes mail but not postfixUser objectClass.
Mon, Mar 1, 23:20 · Mail, Infrastructure (SRE)
John closed T6917: Internal server error if user object includes mail but not postfixUser objectClass as Invalid.

Not an issue

Mon, Mar 1, 23:19 · Mail, Infrastructure (SRE)
John added a comment to T6917: Internal server error if user object includes mail but not postfixUser objectClass.

mail and postfixUser are two distinct things

Mon, Mar 1, 22:37 · Mail, Infrastructure (SRE)

Sun, Feb 28

John closed T6909: Mail seems very lagged as Resolved.

Paladox has blocked the IP which is causing most of the traffic

Sun, Feb 28, 22:15 · Infrastructure (SRE), Mail
John assigned T6909: Mail seems very lagged to Paladox.

Update on the logging I asked you to look into a few days ago?

Sun, Feb 28, 20:29 · Infrastructure (SRE), Mail

Thu, Feb 25

John reassigned T5044: Setup centralised logging for services from Southparkfan to Paladox.

Quite a few actions are blocked on you.

Thu, Feb 25, 20:24 · Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
John closed T6874: Sending to /dev/null'd email should be ignored rather than bounced as Resolved.
Thu, Feb 25, 20:14 · Infrastructure (SRE), Mail
John closed T4019: Encrypt Redis traffic as Declined.

Redis is no longer being used for caching

Thu, Feb 25, 16:38 · Infrastructure (SRE), Goal-2019-Jul-Dec, Goal-2020-Jan-Jun
John closed T4019: Encrypt Redis traffic, a subtask of T4016: Encrypt all traffic inside Miraheze Cluster, as Declined.
Thu, Feb 25, 16:38 · Infrastructure (SRE), Goal-2019-Jul-Dec, Goal-2020-Jan-Jun
John added a project to T6740: Implement local nameserver cache daemons on servers: Performance.
Thu, Feb 25, 16:33 · Performance, Infrastructure (SRE)
John closed T6740: Implement local nameserver cache daemons on servers as Resolved.

Non-reliable numbers here from small scaling testing but:

Thu, Feb 25, 16:23 · Performance, Infrastructure (SRE)
John committed rPUPCc2fb09a6323b: add monitoring + resolvconf (authored by John).
add monitoring + resolvconf
Thu, Feb 25, 16:00
John committed rPUPC423f68422acf: use hiera for pdns check (authored by John).
use hiera for pdns check
Thu, Feb 25, 15:32
John committed rPUPC8a6acf024ec4: use hiera for pdns check (authored by John).
use hiera for pdns check
Thu, Feb 25, 15:31
John committed rPUPCe3d0eb8f7378: add service stanza for pdns (authored by John).
add service stanza for pdns
Thu, Feb 25, 15:14
John committed rPUPC83b5c2803a1f: add pdns to all servers (authored by John).
add pdns to all servers
Thu, Feb 25, 15:10
John reassigned T6873: Log locally in addition to remotely from Southparkfan to Paladox.

Discussion happened and support was given, outcome?

Thu, Feb 25, 11:05 · Infrastructure (SRE)

Wed, Feb 24

John moved T6874: Sending to /dev/null'd email should be ignored rather than bounced from Incoming to Short Term on the Infrastructure (SRE) board.
Wed, Feb 24, 20:20 · Infrastructure (SRE), Mail
John moved T6868: Graylog: Increase max results for elasticsearch from Incoming to Short Term on the Infrastructure (SRE) board.
Wed, Feb 24, 20:20 · Infrastructure (SRE)
John added a comment to T6788: Enhancements to RequestWiki workflow for both requestors and creators.

https://github.com/miraheze/CreateWiki/commit/c9dd807fffa119e47558ce820ec2ef876a9a26f2 And deployment will be whenever someone chooses to deploy it onto production

Wed, Feb 24, 15:49 · MediaWiki (SRE), CreateWiki, Universal Omega
R4356th awarded T6788: Enhancements to RequestWiki workflow for both requestors and creators a Like token.
Wed, Feb 24, 15:37 · MediaWiki (SRE), CreateWiki, Universal Omega
John closed T6857: RequestWikiQueue should handle Unknown Request as Resolved.
Wed, Feb 24, 12:43 · MediaWiki (SRE), Universal Omega, CreateWiki
John closed T6788: Enhancements to RequestWiki workflow for both requestors and creators as Resolved.
Wed, Feb 24, 12:43 · MediaWiki (SRE), CreateWiki, Universal Omega

Sun, Feb 21

John added a comment to T6881: Let each wiki control their robots.txt via MediaWiki:Robots.txt.

Technically the location of the file means little and moving it to be in MirahezeMagic would seem out of scope - as it's a root-level file. Keeping it within puppet seems easiest as theres no gain from moving it to mw-config.

Sun, Feb 21, 20:46 · MediaWiki (SRE), revi, MediaWiki

Fri, Feb 19

John merged T6872: Process messages instantly into T6858: Messages take a while to be sent to graylog.
Fri, Feb 19, 23:17 · Infrastructure (SRE)
John merged task T6872: Process messages instantly into T6858: Messages take a while to be sent to graylog.
Fri, Feb 19, 23:17 · Infrastructure (SRE)

Thu, Feb 18

John closed T6864: Request email forward for packages-bots as Resolved.

Same password as bots-noreply

Thu, Feb 18, 17:38 · Infrastructure (SRE), Mail
John created P386 Resources Table.
Thu, Feb 18, 16:07 · Cloud Infrastructure, Infrastructure (SRE)
John moved T6858: Messages take a while to be sent to graylog from Incoming to Short Term on the Infrastructure (SRE) board.
Thu, Feb 18, 15:03 · Infrastructure (SRE)
John moved T6862: Use memcache for the cache from Incoming to Short Term on the Infrastructure (SRE) board.
Thu, Feb 18, 15:03 · Infrastructure (SRE)
John edited projects for T6862: Use memcache for the cache, added: Infrastructure (SRE); removed MediaWiki (SRE).
Thu, Feb 18, 15:03 · Infrastructure (SRE)

Wed, Feb 17

John added a comment to T6861: Switch to kask for sessions.

which means persistency and tls will be supported.

Wed, Feb 17, 18:31 · MediaWiki (SRE)

Mon, Feb 15

Dmehus awarded T6852: New Server Resource Request for bots1 a Like token.
Mon, Feb 15, 18:22 · Infrastructure (SRE)
John closed T6852: New Server Resource Request for bots1 as Declined.

Per our discussion, Zppix's access will remain revoked and in this case any new access request needs approval anyway of Miraheze SRE.

The bots team are granting automatic approval to all current members as of this comment in the Miraheze Infra team to full root access to our cluster.

Per policy, any member of Infra can approve access for the rest.

I hope this reassured you. The relay token got deleted by @Zppix but I've made a new one and myself @Void will deploy that later so full access will be restored by end of day.

I think this is a sensible proposal and prevents unnecessary spending by Miraheze Limited for another server.

Mon, Feb 15, 18:20 · Infrastructure (SRE)
John changed the status of T6852: New Server Resource Request for bots1 from Open to Stalled.

The relay is restored. Further discussions will occur today

Mon, Feb 15, 11:10 · Infrastructure (SRE)
Dmehus awarded T6850: Bug exists with some Private wikis' Special:Interwiki table visible while logged out, but not for other Private wikis a Like token.
Mon, Feb 15, 05:36 · Universal Omega, MediaWiki (SRE)

Sun, Feb 14

John removed a member for Infrastructure (SRE): NDKilla.
Sun, Feb 14, 22:14
John revoked Site Reliability Engineers from recipient: NDKilla.
Sun, Feb 14, 21:57
John defrocked NDKilla.
Sun, Feb 14, 21:56
John removed a member for acl*security: NDKilla.
Sun, Feb 14, 21:56
John committed rPUPCb6e3b27dbd06: Remove NDKilla from ops (authored by John).
Remove NDKilla from ops
Sun, Feb 14, 21:50
John added a comment to T6850: Bug exists with some Private wikis' Special:Interwiki table visible while logged out, but not for other Private wikis.

I guess Special;Interwiki it needs to be available to non-registered users so search engines can crawl it

My understanding is the reason this was added was to allow Interwiki Administrators to handle IW on private wikis.

Sun, Feb 14, 17:54 · Universal Omega, MediaWiki (SRE)
John closed T6850: Bug exists with some Private wikis' Special:Interwiki table visible while logged out, but not for other Private wikis as Declined.

This isn't a bug - it's expected behaviour by https://github.com/miraheze/MirahezeMagic/blob/master/includes/MirahezeMagicHooks.php#L211

Sun, Feb 14, 17:43 · Universal Omega, MediaWiki (SRE)

Thu, Feb 11

John changed the edit policy for All Things Miraheze.
Thu, Feb 11, 21:47
John changed the Can Create Blogs policy for application Phame from Custom Policy to Administrators.
Thu, Feb 11, 21:46
John committed rPUPC2948dab1b7ba: give sre-mediawiki basic monitoring permissions globally with icinga (authored by John).
give sre-mediawiki basic monitoring permissions globally with icinga
Thu, Feb 11, 19:50
John added a comment to T4019: Encrypt Redis traffic.

We can install Redis v6, however to use TLS, we need PHP-Redis v5 >, which is not available in Buster, only in Bullseye (11). MediaWiki also does not have support for this, but it's a small modification to a regex in RedisConnectionPool which we can commit locally.

Thu, Feb 11, 16:32 · Infrastructure (SRE), Goal-2019-Jul-Dec, Goal-2020-Jan-Jun
John moved T6839: Upgrade puppet to puppet 7 from Short Term to Goals on the Infrastructure (SRE) board.

Is a major software upgrade of critical services really ‘short term’?

Thu, Feb 11, 11:21 · Puppet, Infrastructure (SRE)

Wed, Feb 10

John committed rPUPC4406ef12d3ed: increase AI memory to 1500M (authored by John).
increase AI memory to 1500M
Wed, Feb 10, 22:00
John added a comment to T4019: Encrypt Redis traffic.

This is possible natively with Redis 6

Wed, Feb 10, 17:55 · Infrastructure (SRE), Goal-2019-Jul-Dec, Goal-2020-Jan-Jun
John added a comment to T5044: Setup centralised logging for services.

mon1 marked as done, Icinga logs need to be local for IRC bots however I set up icinga logs to go to graylog separately under T6798

Wed, Feb 10, 12:21 · Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
John updated the task description for T5044: Setup centralised logging for services.
Wed, Feb 10, 12:20 · Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
John added a comment to T5044: Setup centralised logging for services.

@Paladox are you able to give a look over the ones that SPF has marked for you to review please?

Wed, Feb 10, 12:18 · Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Mon, Feb 8

John added a comment to T6765: Cache frequently accessed files on MediaWiki servers.

If restarting the service is required to pick up changes, databases.json and *wiki.json can't be cached

Mon, Feb 8, 18:13 · MediaWiki (SRE), Performance, MediaWiki
John changed the visibility for T6808: Unable to upload files in r2.miraheze.org.
Mon, Feb 8, 18:04 · Infrastructure (SRE)
John added a project to T6808: Unable to upload files in r2.miraheze.org: Infrastructure (SRE).
Mon, Feb 8, 18:03 · Infrastructure (SRE)

Sun, Feb 7

John closed T6798: Connect Graylog and Icinga as Resolved.
Sun, Feb 7, 19:32 · Goal-2021-Jan-Jun, Monitoring, Infrastructure (SRE)
John assigned T6828: MySQL crashed on db11 2021-02-06 ~20:46 to Paladox.

Can you do the incident report as the primary responder?

Sun, Feb 7, 19:02 · Database, Infrastructure (SRE), Production Error, MediaWiki (SRE), MediaWiki
John moved T6828: MySQL crashed on db11 2021-02-06 ~20:46 from Incoming to Short Term on the Infrastructure (SRE) board.
Sun, Feb 7, 19:01 · Database, Infrastructure (SRE), Production Error, MediaWiki (SRE), MediaWiki
John moved T6830: Add icinga/prometheus monitoring for multi-instance from Incoming to Short Term on the Infrastructure (SRE) board.
Sun, Feb 7, 19:01 · Infrastructure (SRE), Database

Sat, Feb 6

John claimed T6798: Connect Graylog and Icinga.

Icinga monitoring alerts/events are now logged in Graylog

Sat, Feb 6, 22:07 · Goal-2021-Jan-Jun, Monitoring, Infrastructure (SRE)
John committed rPUPCf173ab5e2b12: enable icinga->graylog TCP GELF port (authored by John).
enable icinga->graylog TCP GELF port
Sat, Feb 6, 21:56

Fri, Feb 5

John moved T6802: puppet: convert all custom services to be installed using systemd::service from Incoming to Short Term on the Infrastructure (SRE) board.
Fri, Feb 5, 17:49 · Infrastructure (SRE)

Thu, Feb 4

John committed rPUPCb0660f1c8f67: add vmbr1 for vm->cloud connectivity (authored by John).
add vmbr1 for vm->cloud connectivity
Thu, Feb 4, 21:20
John committed rPUPC3a44441c65c1: change netmask to /8 for private (authored by John).
change netmask to /8 for private
Thu, Feb 4, 21:11
John committed rPUPC66037c06804f: add cloud5 config for private interface (authored by John).
add cloud5 config for private interface
Thu, Feb 4, 20:53
John committed rPUPC0d473eadb823: rm .save (authored by John).
rm .save
Thu, Feb 4, 20:48
John committed rPUPC1a49becc86ae: allow private interface to be defined + define it for cloud4 (authored by John).
allow private interface to be defined + define it for cloud4
Thu, Feb 4, 20:46
John committed rDNS82aeae77126f: very bland 10.0.0.0/8 setup (authored by John).
very bland 10.0.0.0/8 setup
Thu, Feb 4, 19:53
John committed rDNS8462e85ff797: very bland 10.0.0.0/8 setup (authored by John).
very bland 10.0.0.0/8 setup
Thu, Feb 4, 19:53

Wed, Feb 3

John closed T6811: [Existing] Server Resource Request for gluster1 as Resolved.

Approved

Wed, Feb 3, 09:45 · Infrastructure (SRE)
John closed T6811: [Existing] Server Resource Request for gluster1, a subtask of T6812: New Image uploads not saving thumbnails, as Resolved.
Wed, Feb 3, 09:45 · Universal Omega, Infrastructure (SRE)

Feb 2 2021

John closed T6810: Clarification for MatomoAnalytics data update as Invalid.
Feb 2 2021, 18:04 · MediaWiki (SRE), revi, MatomoAnalytics
John closed T6810: Clarification for MatomoAnalytics data update as Invalid.

Midnight UTC for us, for other users of the extension, it would be whatever they configure it to be.

Feb 2 2021, 18:03 · MediaWiki (SRE), revi, MatomoAnalytics

Jan 31 2021

John committed rPUPC5c519800814f: grafana: spit admin into two sre ldap groups (authored by John).
grafana: spit admin into two sre ldap groups
Jan 31 2021, 23:51
John closed T6799: New Server Resource Request for Database Backups, a subtask of T6071: Set up replicas for all database clusters, as Resolved.
Jan 31 2021, 22:46 · Infrastructure (SRE), Database
John closed T6799: New Server Resource Request for Database Backups as Resolved.

Endorsement by the DSRE is enough for me.

Jan 31 2021, 22:46 · Infrastructure (SRE)
John committed rPUPCe6e9b611e424: restructure icinga group roles (authored by John).
restructure icinga group roles
Jan 31 2021, 22:37
John committed rPUPC39678a62e837: Merge branch 'master' of github.com:/miraheze/puppet (authored by John).
Merge branch 'master' of github.com:/miraheze/puppet
Jan 31 2021, 22:37
Reception123 awarded T6798: Connect Graylog and Icinga a Mountain of Wealth token.
Jan 31 2021, 20:45 · Goal-2021-Jan-Jun, Monitoring, Infrastructure (SRE)
John moved T6798: Connect Graylog and Icinga from Backlog to Site Reliability Engineering on the Goal-2021-Jan-Jun board.
Jan 31 2021, 20:38 · Goal-2021-Jan-Jun, Monitoring, Infrastructure (SRE)
John moved T6798: Connect Graylog and Icinga from Incoming to Goals on the Infrastructure (SRE) board.
Jan 31 2021, 20:38 · Goal-2021-Jan-Jun, Monitoring, Infrastructure (SRE)
John triaged T6798: Connect Graylog and Icinga as Normal priority.
Jan 31 2021, 20:38 · Goal-2021-Jan-Jun, Monitoring, Infrastructure (SRE)
John added a project to T6794: Rename "projectchaos.miraheze.org" to "cnvn.miraheze.org": MediaWiki (SRE).
Jan 31 2021, 17:48 · MediaWiki (SRE), Database

Jan 30 2021

John closed T6789: Change $wgUploadNavigationUrl and $wgUploadMissingFileUrl for PetraWiki language wikis as Invalid.

Done, though in future please make these requests at the Stewards' Noticeboard

Jan 30 2021, 16:34 · MediaWiki, MediaWiki (SRE)
John moved T4292: Add puppetdb prometheus exporter and export metrics to grafana from Incoming to Short Term on the Infrastructure (SRE) board.
Jan 30 2021, 12:21 · Infrastructure (SRE), Monitoring
John moved T4425: Fix all mysql tables that are using latin rather then utf8mb4 from Incoming to Long Term on the Infrastructure (SRE) board.
Jan 30 2021, 12:21 · Infrastructure (SRE)
John moved T4191: Redesign compression of content inside NGINX and Varnish from Incoming to Long Term on the Infrastructure (SRE) board.
Jan 30 2021, 12:20 · Infrastructure (SRE), Varnish
John moved T4017: Reconfigure TLS settings inside MariaDB from Incoming to Long Term on the Infrastructure (SRE) board.
Jan 30 2021, 12:20 · Infrastructure (SRE), Goal-2019-Jul-Dec, Goal-2020-Jan-Jun
John moved T4016: Encrypt all traffic inside Miraheze Cluster from Incoming to Long Term on the Infrastructure (SRE) board.
Jan 30 2021, 12:20 · Infrastructure (SRE), Goal-2019-Jul-Dec, Goal-2020-Jan-Jun
John moved T4019: Encrypt Redis traffic from Incoming to Long Term on the Infrastructure (SRE) board.
Jan 30 2021, 12:20 · Infrastructure (SRE), Goal-2019-Jul-Dec, Goal-2020-Jan-Jun
John moved T4432: Display landing page in their own language based on country/region (from browser) from Incoming to Short Term on the Infrastructure (SRE) board.
Jan 30 2021, 12:20 · Infrastructure (SRE)
John moved T4601: Track mediawiki-static storage space in Icinga from Incoming to Short Term on the Infrastructure (SRE) board.
Jan 30 2021, 12:20 · Infrastructure (SRE), Monitoring
John moved T5044: Setup centralised logging for services from Incoming to Goals on the Infrastructure (SRE) board.
Jan 30 2021, 12:20 · Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
John moved T5055: Setup geoip update service from Goals to Short Term on the Infrastructure (SRE) board.
Jan 30 2021, 12:19 · Infrastructure (SRE)
John moved T5055: Setup geoip update service from Incoming to Goals on the Infrastructure (SRE) board.
Jan 30 2021, 12:18 · Infrastructure (SRE)
John moved T5397: Create a logbot for server actions from Incoming to Short Term on the Infrastructure (SRE) board.
Jan 30 2021, 12:18 · Infrastructure (SRE)