Page MenuHomeMiraheze

Goal-2020-Jul-DecGoal
ArchivedPublic

Members (4)

Details

Description

Miraheze's Goals for the second half of 2020 (July 1st ~ December 31st).

Tasks added to this project must be at least one of the following:

  • a long-term project that can realistically be finished before December 31st,
  • a development project that will have a positive impact on Miraheze communities,
  • an objective for Site Reliability Engineers (infrastructure, introducing a new service, major work etc.).

MediaWiki and development tasks can be added freely by anyone assuming someone has in the past hinted at the possibility they would be willing to work on it or have a noticeable impact on communities.

Site Reliability Engineering tasks should only be added by members of the Site Reliability Engineer team as they have full control over their priorities and workflow, not subject to community benefit.

< Goal-2020-Jan-Jun | Goal-2021-Jan-Jun >

Recent Activity

Jun 25 2022

Paladox closed T5044: Setup centralised logging for services as Resolved.

Resolved

Jun 25 2022, 15:54 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Paladox updated the task description for T5044: Setup centralised logging for services.
Jun 25 2022, 15:54 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
John added a comment to T5044: Setup centralised logging for services.

@Paladox less than a week until end of goal period - do we have an update on this?

Jun 25 2022, 13:02 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

May 9 2022

Unknown Object (User) moved T5044: Setup centralised logging for services from Backlog to Central Logging on the Monitoring board.
May 9 2022, 19:26 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Unknown Object (User) added a project to T5044: Setup centralised logging for services: Monitoring.
May 9 2022, 19:26 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Apr 16 2022

Paladox updated the task description for T5044: Setup centralised logging for services.
Apr 16 2022, 23:18 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Feb 21 2022

Paladox claimed T5044: Setup centralised logging for services.
Feb 21 2022, 15:09 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Paladox updated the task description for T5044: Setup centralised logging for services.
Feb 21 2022, 14:46 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Jan 1 2022

John added a comment to T5044: Setup centralised logging for services.

This task has taken a back foot, over other work which has higher priority currently such as T8469 T8350

Jan 1 2022, 10:45 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Unknown Object (User) moved T5044: Setup centralised logging for services from Backlog to Infrastructure on the Goal-2022-Jan-Jun board.
Jan 1 2022, 03:22 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Unknown Object (User) added a project to T5044: Setup centralised logging for services: Goal-2022-Jan-Jun.
Jan 1 2022, 03:15 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Dec 5 2021

John added a comment to T5044: Setup centralised logging for services.

I am going to start progress on this task, firstly by cleaning up how we define all of this in puppet. I'll introduce simply logging stanzas that we can define over and over again for each log file, that handles all of the syslog-ng logic + logrotate configuration for the new system.

Dec 5 2021, 21:06 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Nov 7 2021

Unknown Object (User) merged T7135: Ingest PHP-FPM slowlogs into Graylog into T5044: Setup centralised logging for services.
Nov 7 2021, 00:59 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Oct 20 2021

John added a comment to T5044: Setup centralised logging for services.

New server list for checking the above plan against:

Oct 20 2021, 12:45 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
John added a comment to T5044: Setup centralised logging for services.

Plan for resolving this task:

  • All services will have their logs ingested into Graylog, this isn't negotiable.
  • Where logs are ingested, we will maintain 24-48 hours of *local* logs on the server. This will be supported by log rotation.
Oct 20 2021, 12:33 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Oct 15 2021

John closed T5877: Revise MariaDB backup strategy as Resolved.

This is now resolved.

Oct 15 2021, 17:11 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

Oct 13 2021

John added a comment to T5877: Revise MariaDB backup strategy.

db13:

  • Time taken: 2 hours and 20 minutes
  • Size: 33G
Oct 13 2021, 19:35 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec
John added a comment to T5877: Revise MariaDB backup strategy.

https://github.com/miraheze/puppet/compare/6d6dcbc15b0e...139bf730eb26 automates this for daily, so we should have a live accessible copy for a 24 hour RPO - and bacula will store backups for a longer period of time (TBD).

Oct 13 2021, 13:27 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec
John added a comment to T5877: Revise MariaDB backup strategy.

The backup ran for 14 hours before I killed it as it caused T8163.

Oct 13 2021, 10:04 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

Oct 12 2021

John added a comment to T5877: Revise MariaDB backup strategy.

Currently doing the above command but over an NFS mount to dbbackup1 which is in the US. This will take significantly longer - that is the main thing I am interested in right now.

Oct 12 2021, 20:14 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec
John added a comment to T5877: Revise MariaDB backup strategy.
mydumper -G -E -R -m -v 3 -t 2 -c -x "^(?!([0-9a-z]+wiki.(objectcache|querycache|querycachetwo|recentchanges|searchindex)))" -L "/home/johnflewis/$(date +"%Y%m%d%H%M%S").log" --trx-consistency-only

On db12:

  • Time taken: 103 minutes (1 hour and 43 minutes)
  • Size: 30G
Oct 12 2021, 18:53 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

Oct 11 2021

John added a comment to T5877: Revise MariaDB backup strategy.

Trying to optimise the dump by reducing amount of data carried over (because not everything in MediaWiki is irreplaceable!)

Oct 11 2021, 22:12 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec
John claimed T5044: Setup centralised logging for services.
Oct 11 2021, 18:08 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
John added a comment to T5044: Setup centralised logging for services.

T7740 is likely to be influenced by work done on this task.

Oct 11 2021, 18:06 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Oct 9 2021

John claimed T5877: Revise MariaDB backup strategy.
Oct 9 2021, 20:26 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

Sep 28 2021

John placed T5877: Revise MariaDB backup strategy up for grabs.

De-assigned per lack of progress.

Sep 28 2021, 11:25 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

Sep 21 2021

John added a comment to T5877: Revise MariaDB backup strategy.

@Southparkfan Any updates on this task? If there isn't an update provided in a week, I'll reassign the task to ensure it gets completed.

Sep 21 2021, 20:14 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

Aug 10 2021

Paladox added a comment to T5044: Setup centralised logging for services.
In T5044#156437, @John wrote:

@Paladox has raised concerns with centralised-only logging. We should explore these concerns before pushing for things like nginx access logs as these are critical for debugging some traffic influx/DoS attacks.

Aug 10 2021, 16:57 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Reception123 added a comment to T5044: Setup centralised logging for services.

I agree with that. At least for some logs it's definitely useful to have logs stored locally in case something goes wrong and the logs don't get transmitted to graylog.

Aug 10 2021, 14:14 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
John updated subscribers of T5044: Setup centralised logging for services.

@Paladox has raised concerns with centralised-only logging. We should explore these concerns before pushing for things like nginx access logs as these are critical for debugging some traffic influx/DoS attacks.

Aug 10 2021, 12:20 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
John added a comment to T5877: Revise MariaDB backup strategy.

Updates since last one on June 1st?

Aug 10 2021, 11:24 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec

Jul 31 2021

Unknown Object (User) updated subscribers of T5044: Setup centralised logging for services.
Jul 31 2021, 00:25 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Unknown Object (User) updated the task description for T5044: Setup centralised logging for services.
Jul 31 2021, 00:25 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun

Jul 28 2021

Excelsis added a comment to T5412: Review changes made to a wiki via Special:ManageWiki before submitting them.

This has now been deployed.

Jul 28 2021, 19:18 · Goal-2021-Jul-Dec, Universal Omega, MediaWiki (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, ManageWiki
Unknown Object (User) closed T5412: Review changes made to a wiki via Special:ManageWiki before submitting them as Resolved.

This has now been deployed.

Jul 28 2021, 17:45 · Goal-2021-Jul-Dec, Universal Omega, MediaWiki (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, ManageWiki

Jul 26 2021

Unknown Object (User) added a comment to T5412: Review changes made to a wiki via Special:ManageWiki before submitting them.

Currently blocked on community consensus.

Jul 26 2021, 20:40 · Goal-2021-Jul-Dec, Universal Omega, MediaWiki (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, ManageWiki
Unknown Object (User) added a comment to T5412: Review changes made to a wiki via Special:ManageWiki before submitting them.

https://github.com/miraheze/ManageWiki/pull/290

Jul 26 2021, 17:41 · Goal-2021-Jul-Dec, Universal Omega, MediaWiki (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, ManageWiki
Unknown Object (User) moved T5412: Review changes made to a wiki via Special:ManageWiki before submitting them from Long Term to Goals on the Universal Omega board.
Jul 26 2021, 06:38 · Goal-2021-Jul-Dec, Universal Omega, MediaWiki (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, ManageWiki
Unknown Object (User) moved T5412: Review changes made to a wiki via Special:ManageWiki before submitting them from Long Term to Goals on the MediaWiki (SRE) board.
Jul 26 2021, 06:38 · Goal-2021-Jul-Dec, Universal Omega, MediaWiki (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, ManageWiki
Unknown Object (User) moved T5412: Review changes made to a wiki via Special:ManageWiki before submitting them from Backlog to Miraheze Extensions on the Goal-2021-Jul-Dec board.
Jul 26 2021, 06:38 · Goal-2021-Jul-Dec, Universal Omega, MediaWiki (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, ManageWiki
Unknown Object (User) added a project to T5412: Review changes made to a wiki via Special:ManageWiki before submitting them: Goal-2021-Jul-Dec.

I guess this wasn't moved over to the next goal period, so doing that.

Jul 26 2021, 06:38 · Goal-2021-Jul-Dec, Universal Omega, MediaWiki (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, ManageWiki
Unknown Object (User) claimed T5412: Review changes made to a wiki via Special:ManageWiki before submitting them.

I drafted a bit of JS for this, using the oojs dialogs. This should be fairly good to do, with a "review" button, next to the save button, so it does not annoy users if they don't want to review them.

Jul 26 2021, 02:51 · Goal-2021-Jul-Dec, Universal Omega, MediaWiki (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, ManageWiki

Jul 3 2021

Unknown Object (User) moved T5877: Revise MariaDB backup strategy from Backlog to Infrastructure on the Goal-2021-Jul-Dec board.
Jul 3 2021, 18:45 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec
Unknown Object (User) added a project to T5877: Revise MariaDB backup strategy: Goal-2021-Jul-Dec.

Moving over to new goal period. Feel free to remove if it isn't wanted to be moved over.

Jul 3 2021, 18:45 · Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec
Unknown Object (User) moved T5044: Setup centralised logging for services from Backlog to Infrastructure on the Goal-2021-Jul-Dec board.
Jul 3 2021, 18:44 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Unknown Object (User) added a project to T5044: Setup centralised logging for services: Goal-2021-Jul-Dec.

Moving over to new goal period. Feel free to remove if it isn't wanted to be moved over.

Jul 3 2021, 18:43 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Unknown Object (User) added a project to T5412: Review changes made to a wiki via Special:ManageWiki before submitting them: Universal Omega.
Jul 3 2021, 18:27 · Goal-2021-Jul-Dec, Universal Omega, MediaWiki (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, ManageWiki
Unknown Object (User) moved T5412: Review changes made to a wiki via Special:ManageWiki before submitting them from Goals to Long Term on the MediaWiki (SRE) board.
Jul 3 2021, 18:26 · Goal-2021-Jul-Dec, Universal Omega, MediaWiki (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, ManageWiki

Jun 14 2021

Void added a comment to T5044: Setup centralised logging for services.

I could look into taking this over from @Paladox. Is there anything not on this task that I should be aware of if I do?

Jun 14 2021, 19:57 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun
Paladox placed T5044: Setup centralised logging for services up for grabs.
Jun 14 2021, 18:59 · Monitoring, Goal-2022-Jan-Jun, Goal-2021-Jul-Dec, Infrastructure (SRE), Goal-2021-Jan-Jun, Goal-2020-Jul-Dec, Goal-2020-Jan-Jun