Page MenuHomeMiraheze
Feed Advanced Search

Mar 15 2021

John triaged T6973: Monitor Physical Disk Health as Normal priority.
Mar 15 2021, 16:01 · Monitoring, Cloud Infrastructure, Infrastructure (SRE)
John moved T5397: Create a logbot for server actions from Long Term to Short Term on the Infrastructure (SRE) board.
Mar 15 2021, 15:43 · Infrastructure (SRE)
John moved T4302: Deploy Apache Traffic Server from Goals to Long Term on the Infrastructure (SRE) board.
Mar 15 2021, 15:43 · Goal-2021-Jul-Dec, Infrastructure (SRE)
John moved T4302: Deploy Apache Traffic Server from Incoming to Goals on the Infrastructure (SRE) board.
Mar 15 2021, 15:42 · Goal-2021-Jul-Dec, Infrastructure (SRE)

Mar 14 2021

Dmehus awarded T6967: dbbackup* and bacul2 down due to out of credits for ramnode cloud a Like token.
Mar 14 2021, 19:15 · Infrastructure (SRE)
John claimed T6967: dbbackup* and bacul2 down due to out of credits for ramnode cloud.
Mar 14 2021, 19:08 · Infrastructure (SRE)
John closed T6967: dbbackup* and bacul2 down due to out of credits for ramnode cloud as Resolved.
Mar 14 2021, 19:08 · Infrastructure (SRE)
John added a comment to T6961: Update removePII database columns.

Has the Board been made aware of this yet?

Mar 14 2021, 18:36 · Universal Omega, MediaWiki (SRE), Configuration
John updated the post content for Blog Post: 2021: New Year, New Teams, New Focus, Same Principles.
Mar 14 2021, 16:32 · Infrastructure (SRE), MediaWiki (SRE)
John published Blog Post: 2021: New Year, New Teams, New Focus, Same Principles.
Mar 14 2021, 15:52 · Infrastructure (SRE), MediaWiki (SRE)
John added a comment to T6963: loginwiki not found in matomo.

For reference this generates around 9 error messages a second, which is substantial - especially when trying to find error messages so this should be fixed permanently soon.

Mar 14 2021, 14:19 · MediaWiki (SRE)
John triaged T6963: loginwiki not found in matomo as Normal priority.
Mar 14 2021, 14:07 · MediaWiki (SRE)

Mar 13 2021

Dmehus awarded T6952: cp12 OOM'd March 12 2021 16:36 a Like token.
Mar 13 2021, 20:22 · Infrastructure (SRE)
John committed rPUPC1b7d9bd772c8: T6952: Varnish cleanup after failure.
T6952: Varnish cleanup after failure
Mar 13 2021, 16:12
John closed T6952: cp12 OOM'd March 12 2021 16:36 as Resolved.
Mar 13 2021, 16:11 · Infrastructure (SRE)

Mar 11 2021

John closed T6860: runJobs: json_decode() error (4): Syntax error as Declined.

Jobs don't fail as this is always ran after jobs are completed.

Mar 11 2021, 18:34 · MediaWiki, MediaWiki (SRE)

Mar 10 2021

John closed T6941: SSL renewals causing private keys to change causing issues (as private key is not updated automatically) as Invalid.
Mar 10 2021, 19:54 · Upstream, Infrastructure (SRE)
John triaged T6890: Restricting anonymous visitors from accessing special pages or toolbox as Normal priority.
Mar 10 2021, 19:37 · MediaWiki (SRE), MediaWiki
John triaged T6896: Can't log into my wiki: invalid session as Normal priority.
Mar 10 2021, 19:37 · MediaWiki, MediaWiki (SRE)
John added a project to T6932: Change database name for my wiki: MediaWiki (SRE).
Mar 10 2021, 19:36 · MediaWiki (SRE)
John added a project to T6929: Issue in T6928 still unresolved: MediaWiki (SRE).
Mar 10 2021, 19:36 · MediaWiki (SRE)
John closed T5105: Investigate and Implement basic Machine Learning concepts for automatic wiki creation as Resolved.

https://github.com/miraheze/CreateWiki/pull/200 makes this task resolved, only setting a configuration in LS is required now to enable this.

Mar 10 2021, 16:40 · MediaWiki (SRE), Goal-2021-Jan-Jun, Universal Omega, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun, CreateWiki
John added a comment to T6071: Set up replicas for all database clusters.

@Southparkfan Can we have an update on this please?

Mar 10 2021, 16:05 · Infrastructure (SRE), Database
John added a project to T6941: SSL renewals causing private keys to change causing issues (as private key is not updated automatically): Upstream.

https://github.com/certbot/certbot/issues/8710 has a noticeable impact on us, which requires our intervention so downstream ticket remains open

Mar 10 2021, 10:25 · Upstream, Infrastructure (SRE)
John placed T6373: Split Special:ManageWiki/settings into subpages in order to significantly boost preformance up for grabs.
Mar 10 2021, 00:47 · MediaWiki (SRE), Universal Omega, ManageWiki

Mar 8 2021

John added a comment to T6935: Let Wiki Creators Manually Respond to Requests.
In T6935#136946, @R4356th wrote:

Okay but could you please make it so that Wiki Creators are at least allowed to input their own comments?

Mar 8 2021, 08:57 · MediaWiki (SRE), Universal Omega, CreateWiki
John added a comment to T4164: Differentiate sitenotices for different cases.
In T4164#136920, @R4356th wrote:

But Stewards always close such requests as not done. See this (archived) request, for example.

Mar 8 2021, 08:51 · Amanda Catherine, Bukkit, MediaWiki (SRE), Configuration
John closed T6935: Let Wiki Creators Manually Respond to Requests as Declined.

Canned Responses are either used or they’re not within the extension, adding an other option defeats the purpose of why they were added. The extension doesn’t force canned responses to be used therefore this is an issue for configuration rather than the extension.

Mar 8 2021, 08:49 · MediaWiki (SRE), Universal Omega, CreateWiki

Mar 7 2021

Dmehus awarded T6006: Migrate to a Kafka Job Queue a Like token.
Mar 7 2021, 18:56 · MediaWiki (SRE), MediaWiki
John closed T6006: Migrate to a Kafka Job Queue as Declined.

Considering our resources and facts about the current and new proposed system, I'm not seeing enough of a gain to justify allocating a very significant portion of our remaining resources to creating an entire service cluster which would require at minimum 8GB of memory and several cores, just to run jobs. We moved sessions from Redis to Memcached which provided a good performance boost, keeping in mind JobRunner still uses Redis that could be one step. Alternatively, we could also drop dedicated runners, take our main cluster up to 8 and running jobs on all of them. If we allocate 1 runner per server, that would give us 8 instances, which is double what we have right now.

Mar 7 2021, 18:49 · MediaWiki (SRE), MediaWiki
John added a comment to T6006: Migrate to a Kafka Job Queue.

T5976#136818

Mar 7 2021, 17:57 · MediaWiki (SRE), MediaWiki
John closed T5976: Add the EventStreams service as Declined.

Going to mark as declined for now - a quick look into this suggests we'd need around 16GB to be able to process just our RC traffic alone, ignoring jobs as well. The actual gain in deploying this service as well is very small, so I don't feel like I can justify allocating the remaining RAM on a single server to a Kafka deployment.

Mar 7 2021, 17:56 · Infrastructure (SRE)
John claimed T5976: Add the EventStreams service.

Requires Kafka -> claiming as a investigatory exercise

Mar 7 2021, 17:24 · Infrastructure (SRE)
John closed T5055: Setup geoip update service as Resolved.
Mar 7 2021, 17:19 · Infrastructure (SRE)
John committed rPUPC5182fb31e780: Merge branch 'master' of github.com:/miraheze/puppet.
Merge branch 'master' of github.com:/miraheze/puppet
Mar 7 2021, 17:17
John committed rPUPCb66bc7e502ac: fetch geoip from private.
fetch geoip from private
Mar 7 2021, 17:17
John committed rPUPCa00a6dcf08ba: Merge branch 'master' of github.com:/miraheze/puppet.
Merge branch 'master' of github.com:/miraheze/puppet
Mar 7 2021, 17:17
John committed rPUPC111cdc48d8ce: add a script to run geoip updates and save to private git.
add a script to run geoip updates and save to private git
Mar 7 2021, 17:11
John reassigned T6564: Upgrade to Debian 10.7 from John to Paladox.
Mar 7 2021, 16:04 · Infrastructure (SRE)
John closed T6564: Upgrade to Debian 10.7 as Resolved.

Resolved in the sense that all that is practical is done.

Mar 7 2021, 16:03 · Infrastructure (SRE)
John assigned T6802: puppet: convert all custom services to be installed using systemd::service to Paladox.

Has shown some interest in doing this.

Mar 7 2021, 15:59 · Infrastructure (SRE)
John closed T4016: Encrypt all traffic inside Miraheze Cluster as Invalid.

Tracking tasks are bad - as this task depends on sub tasks being doing rather than something actually being done.

Mar 7 2021, 15:58 · Infrastructure (SRE), Goal-2019-Jul-Dec, Goal-2020-Jan-Jun
John closed T6771: Request to allow archive.org embedding for tuscriaturas wiki as Declined.
Mar 7 2021, 15:54 · Infrastructure (SRE)
John closed T5713: Create automated Icinga check for validity of all TLS certificates on system as Declined.

We now regularly update CA certs on each puppet run - putting the burden of management responsibility on the CAs rather than us. Monitoring all CAs on the system would add approximately another 150 SSL checks which seems disproportionate especially for something we neither maintain nor can control.

Mar 7 2021, 15:42 · Infrastructure (SRE), Monitoring
John closed T5712: Investigate why the AAA Certificate Services root is mandatory as Resolved.

We're now using the ca-certificates and capath approach with web configuration. Chains are now created and regularly updated by the CA themselves, rather than us manually adding and maintaining them. CAs also maintain their own trust chains.

Mar 7 2021, 15:41 · Infrastructure (SRE)
John closed T5714: Redesign implementation of adding own TLS certificates and CAs as Resolved.
Mar 7 2021, 15:28 · Infrastructure (SRE)
John added a comment to T4164: Differentiate sitenotices for different cases.
In T4164#136761, @R4356th wrote:

@John, if I am understanding your and @Universal_Omega's concerns correctly, differentiating the sitenotice for wikis manually closed as opposed to automatically (i.e. by a script) would allow people to easily understand if they can adopt that wiki. This should decrease the number of RfAs that cannot be undertaken and reduce Stewards' workload.

Mar 7 2021, 15:26 · Amanda Catherine, Bukkit, MediaWiki (SRE), Configuration
John committed rPUPCae869fdbec1d: Merge branch 'master' of github.com:/miraheze/puppet.
Merge branch 'master' of github.com:/miraheze/puppet
Mar 7 2021, 14:14
John committed rPUPC4348538f687c: fix path, why is this a param?.
fix path, why is this a param?
Mar 7 2021, 14:14
John committed rPUPC8cadf85c508a: local conflict.
local conflict
Mar 7 2021, 14:09
John committed rPUPCbcf2ea9fc913: add ca back to hiera.pp.
add ca back to hiera.pp
Mar 7 2021, 14:08
John committed rPUPC61728e31e552: ssl: update ca regularly and move to capath config over cafile.
ssl: update ca regularly and move to capath config over cafile
Mar 7 2021, 14:02
John committed rPUPC27d89413afa1: ssl: update ca regularly and move to capath config over cafile.
ssl: update ca regularly and move to capath config over cafile
Mar 7 2021, 12:49
John added a comment to T4164: Differentiate sitenotices for different cases.

Private wikis vs public wikis is now Done.

What remains to be done is manual closure vs script which is still blocked on community consensus.

Whether it's blocked on community concensus or not, about whether manually closed wikis can be adopted. I personally believe it should still be differentiated and to say that it was manually closed rather then it was closed after time

Mar 7 2021, 10:03 · Amanda Catherine, Bukkit, MediaWiki (SRE), Configuration

Mar 6 2021

John changed the status of T4425: Fix all mysql tables that are using latin rather than binary from Stalled to Open.
Mar 6 2021, 19:53 · Database, Infrastructure (SRE)
John changed the status of T5976: Add the EventStreams service from Stalled to Open.
Mar 6 2021, 19:52 · Infrastructure (SRE)
John closed T4432: Display landing page in their own language based on country/region (from browser) as Resolved.
Mar 6 2021, 19:51 · Infrastructure (SRE)
John committed rPUPC75c319b60b37: detect language on landing page.
detect language on landing page
Mar 6 2021, 19:51
John assigned T6830: Add icinga/prometheus monitoring for multi-instance to Paladox.
Mar 6 2021, 18:21 · Monitoring, Infrastructure (SRE), Database
John assigned T6868: Graylog: Increase max results for elasticsearch to Paladox.

Discussed yesterday and paladox found a way to do this, assigning to him.

Mar 6 2021, 18:19 · Infrastructure (SRE)
Redmin awarded T6835: Wikidiscover is listing wikis that are archived, deleted or no longer accesible a Like token.
Mar 6 2021, 06:29 · MediaWiki (SRE), WikiDiscover, Universal Omega

Mar 5 2021

Dmehus awarded T6898: Transfer miraheze.tk/gl/ml/etc a Like token.
Mar 5 2021, 22:21 · Site Reliability Engineering
John closed T6898: Transfer miraheze.tk/gl/ml/etc as Resolved.

As the domains aren't in active use, there's no real requirement for us have control over them - therefore if Zppix is willing to keep renewing them, that is absolutely fine. If he isn't, and a transfer isn't possible, registration lapsing would be the only next steps.

Mar 5 2021, 18:22 · Site Reliability Engineering
John closed T6835: Wikidiscover is listing wikis that are archived, deleted or no longer accesible as Resolved.

Deleted wikis can now be excluded by removing 'deleted' from state list (all includes deleted)

Mar 5 2021, 17:03 · MediaWiki (SRE), WikiDiscover, Universal Omega
John closed T6907: WikiDiscover API does not honor siteprop parameter as Resolved.
Mar 5 2021, 16:40 · MediaWiki (SRE), Universal Omega, WikiDiscover
John closed T6899: Inaccurate log for ManageWiki/namespaces as Resolved.
Mar 5 2021, 15:46 · MediaWiki (SRE), ManageWiki, Universal Omega

Mar 3 2021

John assigned T6922: Approved Revisions extension stopped working to Paladox.

Another issue caused by deployers merging without reviewing the changes being deployed. I've ask @Paladox to run the SQL.

Mar 3 2021, 15:52 · MediaWiki (SRE), Extensions, Database, Universal Omega

Mar 2 2021

John closed T6909: Mail seems very lagged as Resolved.
Mar 2 2021, 20:14 · Infrastructure (SRE), Mail
John claimed T6909: Mail seems very lagged.
Mar 2 2021, 16:30 · Infrastructure (SRE), Mail

Mar 1 2021

John changed the visibility for T6917: Internal server error if user object includes mail but not postfixUser objectClass.
Mar 1 2021, 23:20 · Mail, Infrastructure (SRE)
John closed T6917: Internal server error if user object includes mail but not postfixUser objectClass as Invalid.

Not an issue

Mar 1 2021, 23:19 · Mail, Infrastructure (SRE)
John added a comment to T6917: Internal server error if user object includes mail but not postfixUser objectClass.

mail and postfixUser are two distinct things

Mar 1 2021, 22:37 · Mail, Infrastructure (SRE)

Feb 28 2021

John closed T6909: Mail seems very lagged as Resolved.

Paladox has blocked the IP which is causing most of the traffic

Feb 28 2021, 22:15 · Infrastructure (SRE), Mail
John assigned T6909: Mail seems very lagged to Paladox.

Update on the logging I asked you to look into a few days ago?

Feb 28 2021, 20:29 · Infrastructure (SRE), Mail

Feb 25 2021

John reassigned T5044: Setup centralised logging for services from Southparkfan to Paladox.

Quite a few actions are blocked on you.

Feb 25 2021, 20:24 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
John closed T6874: Sending to /dev/null'd email should be ignored rather than bounced as Resolved.
Feb 25 2021, 20:14 · Infrastructure (SRE), Mail
John closed T4019: Encrypt Redis traffic as Declined.

Redis is no longer being used for caching

Feb 25 2021, 16:38 · Infrastructure (SRE), Goal-2019-Jul-Dec, Goal-2020-Jan-Jun
John closed T4019: Encrypt Redis traffic, a subtask of T4016: Encrypt all traffic inside Miraheze Cluster, as Declined.
Feb 25 2021, 16:38 · Infrastructure (SRE), Goal-2019-Jul-Dec, Goal-2020-Jan-Jun
John added a project to T6740: Implement local nameserver cache daemons on servers: Performance.
Feb 25 2021, 16:33 · Performance, Infrastructure (SRE)
John closed T6740: Implement local nameserver cache daemons on servers as Resolved.

Non-reliable numbers here from small scaling testing but:

Feb 25 2021, 16:23 · Performance, Infrastructure (SRE)
John committed rPUPCc2fb09a6323b: add monitoring + resolvconf.
add monitoring + resolvconf
Feb 25 2021, 16:00
John committed rPUPC423f68422acf: use hiera for pdns check.
use hiera for pdns check
Feb 25 2021, 15:32
John committed rPUPC8a6acf024ec4: use hiera for pdns check.
use hiera for pdns check
Feb 25 2021, 15:31
John committed rPUPCe3d0eb8f7378: add service stanza for pdns.
add service stanza for pdns
Feb 25 2021, 15:14
John committed rPUPC83b5c2803a1f: add pdns to all servers.
add pdns to all servers
Feb 25 2021, 15:10
John reassigned T6873: Log locally in addition to remotely from Southparkfan to Paladox.

Discussion happened and support was given, outcome?

Feb 25 2021, 11:05 · Infrastructure (SRE)

Feb 24 2021

John moved T6874: Sending to /dev/null'd email should be ignored rather than bounced from Incoming to Short Term on the Infrastructure (SRE) board.
Feb 24 2021, 20:20 · Infrastructure (SRE), Mail
John moved T6868: Graylog: Increase max results for elasticsearch from Incoming to Short Term on the Infrastructure (SRE) board.
Feb 24 2021, 20:20 · Infrastructure (SRE)
John added a comment to T6788: Enhancements to RequestWiki workflow for both requestors and creators.

https://github.com/miraheze/CreateWiki/commit/c9dd807fffa119e47558ce820ec2ef876a9a26f2 And deployment will be whenever someone chooses to deploy it onto production

Feb 24 2021, 15:49 · MediaWiki (SRE), Universal Omega, CreateWiki
Redmin awarded T6788: Enhancements to RequestWiki workflow for both requestors and creators a Like token.
Feb 24 2021, 15:37 · MediaWiki (SRE), Universal Omega, CreateWiki
John closed T6857: RequestWikiQueue should handle Unknown Request as Resolved.
Feb 24 2021, 12:43 · MediaWiki (SRE), Universal Omega, CreateWiki
John closed T6788: Enhancements to RequestWiki workflow for both requestors and creators as Resolved.
Feb 24 2021, 12:43 · MediaWiki (SRE), Universal Omega, CreateWiki

Feb 21 2021

John added a comment to T6881: Let each wiki control their robots.txt via MediaWiki:Robots.txt.

Technically the location of the file means little and moving it to be in MirahezeMagic would seem out of scope - as it's a root-level file. Keeping it within puppet seems easiest as theres no gain from moving it to mw-config.

Feb 21 2021, 20:46 · Universal Omega, MediaWiki (SRE), revi, MediaWiki

Feb 19 2021

John merged T6872: Process messages instantly into T6858: Messages take a while to be sent to graylog.
Feb 19 2021, 23:17 · Infrastructure (SRE)
John merged task T6872: Process messages instantly into T6858: Messages take a while to be sent to graylog.
Feb 19 2021, 23:17 · Infrastructure (SRE)

Feb 18 2021

John closed T6864: Request email forward for packages-bots as Resolved.

Same password as bots-noreply

Feb 18 2021, 17:38 · Infrastructure (SRE), Mail
John created P386 Resources Table.
Feb 18 2021, 16:07 · Cloud Infrastructure, Infrastructure (SRE)
John moved T6858: Messages take a while to be sent to graylog from Incoming to Short Term on the Infrastructure (SRE) board.
Feb 18 2021, 15:03 · Infrastructure (SRE)
John moved T6862: Use memcache for the cache from Incoming to Short Term on the Infrastructure (SRE) board.
Feb 18 2021, 15:03 · Infrastructure (SRE)
John edited projects for T6862: Use memcache for the cache, added: Infrastructure (SRE); removed MediaWiki (SRE).
Feb 18 2021, 15:03 · Infrastructure (SRE)