Page MenuHomeMiraheze
Feed Advanced Search

Yesterday

Southparkfan reassigned T5433: Evaluate regular SRE meetings from Southparkfan to Reception123.

What's the status on meetings in the MediaWiki team?

Fri, Feb 26, 22:12 · Site Reliability Engineering
Southparkfan placed T6830: Add icinga/prometheus monitoring for multi-instance up for grabs.

(no time currently)

Fri, Feb 26, 22:02 · Infrastructure (SRE), Database
Southparkfan moved T6900: Create draft of Data Processing Inventory from Radar to Management on the Site Reliability Engineering board.
Fri, Feb 26, 21:59 · Site Reliability Engineering
Southparkfan triaged T6900: Create draft of Data Processing Inventory as Low priority.
Fri, Feb 26, 21:58 · Site Reliability Engineering
Southparkfan lowered the priority of T4017: Reconfigure TLS settings inside MariaDB from Normal to Low.
Fri, Feb 26, 21:47 · Infrastructure (SRE), Goal-2019-Jul-Dec, Goal-2020-Jan-Jun

Thu, Feb 25

Southparkfan added a comment to T6765: Cache frequently accessed files on MediaWiki servers.
In T6765#134610, @John wrote:

If restarting the service is required to pick up changes, databases.json and *wiki.json can't be cached

Thu, Feb 25, 17:21 · MediaWiki (SRE), Performance, MediaWiki

Wed, Feb 24

Southparkfan edited P387 syslog-ng log to local file.
Wed, Feb 24, 22:06
Southparkfan edited P387 syslog-ng log to local file.
Wed, Feb 24, 22:05
Southparkfan created P387 syslog-ng log to local file.
Wed, Feb 24, 22:04

Wed, Feb 17

Southparkfan added a comment to T6858: Messages take a while to be sent to graylog.

Do you know what the bottleneck is? Is it MediaWiki -> syslog-ng or syslog-ng -> graylog?

Wed, Feb 17, 22:31 · Infrastructure (SRE)

Sun, Feb 14

Southparkfan added a comment to T6849: Grafana bug CVE-2019-15043 can still be exploited despite being out of vulnerable range.

Wikimedia has been contacted. Waiting on more information from the researcher.

Sun, Feb 14, 13:41 · Upstream, Infrastructure (SRE), Security
Southparkfan lowered the priority of T6849: Grafana bug CVE-2019-15043 can still be exploited despite being out of vulnerable range from Unbreak Now! to High.
  • The API endpoint is open
  • Our Grafana version is on a patched version
  • Only impact on A, not C/I, Grafana is not critical
Sun, Feb 14, 12:54 · Upstream, Infrastructure (SRE), Security
Southparkfan added a comment to T6849: Grafana bug CVE-2019-15043 can still be exploited despite being out of vulnerable range.
southparkfan@test3:~$ python3 cve.py --url 'https://grafana.miraheze.org'
[-] Testing https://grafana.miraheze.org...
[-] Status: 200
[-] Checking for version...
[-] Grafana version appears to be: 7.4.1
[!] Version seems to indicate it's probably not vulnerable.
[-] Checking if snapshot api requires authentiation...
[+] Snapshot endpoint doesn't seem to require authentication! Host may be vulnerable.
Sun, Feb 14, 12:51 · Upstream, Infrastructure (SRE), Security

Sat, Feb 13

Dmehus awarded T6796: Revise inactivity policy a Like token.
Sat, Feb 13, 15:20 · Site Reliability Engineering

Fri, Feb 12

Southparkfan committed rPUPCa44bf35019ec: mon2: use graylog for nginx (authored by Southparkfan).
mon2: use graylog for nginx
Fri, Feb 12, 22:28
Southparkfan committed rPUPC3bce55cd0c64: Remove systemd::syslog calls for irc/monitoring (authored by Southparkfan).
Remove systemd::syslog calls for irc/monitoring
Fri, Feb 12, 22:22
Southparkfan committed rPUPCc7adf3038c13: Deploy syslog-ng to mon2 (authored by Southparkfan).
Deploy syslog-ng to mon2
Fri, Feb 12, 22:18

Tue, Feb 9

Dmehus awarded T6832: npm should not run as root on MediaWiki servers a Like token.
Tue, Feb 9, 23:32 · Infrastructure (SRE), Security

Mon, Feb 8

Southparkfan updated subscribers of T6832: npm should not run as root on MediaWiki servers.

cc @John as Engineering Manager

Mon, Feb 8, 20:39 · Infrastructure (SRE), Security
Southparkfan created T6832: npm should not run as root on MediaWiki servers.
Mon, Feb 8, 20:38 · Infrastructure (SRE), Security
Southparkfan added a comment to T6765: Cache frequently accessed files on MediaWiki servers.

@Paladox: see P381. Ideally, you turn the vmtouch (daemon) into a systemd service (a type of unit) that will run this script on every puppet run, after which you can restart the systemd service (to pick up new files).

Mon, Feb 8, 13:21 · MediaWiki (SRE), Performance, MediaWiki
Southparkfan created P381 Generate list of files for vmtouch (T6765).
Mon, Feb 8, 13:18
Southparkfan added a comment to T4166: Animated Feet at Miraheze.

We have more storage now, could this be imported over the coming weeks?

Mon, Feb 8, 00:48 · MediaWiki (SRE), MediaWiki

Sun, Feb 7

Southparkfan triaged T6830: Add icinga/prometheus monitoring for multi-instance as Normal priority.
Sun, Feb 7, 16:05 · Infrastructure (SRE), Database
Southparkfan committed rPUPC5931301ca191: Fix firewall rules for connections from db backup servers (authored by Southparkfan).
Fix firewall rules for connections from db backup servers
Sun, Feb 7, 15:52
Southparkfan committed rPUPC5f9fcd731650: Run apt-get update when apt pins the mariadb package (authored by Southparkfan).
Run apt-get update when apt pins the mariadb package
Sun, Feb 7, 02:50
Southparkfan committed rPUPC620505d2beda: dbbackup: fix data type (authored by Southparkfan).
dbbackup: fix data type
Sun, Feb 7, 02:47
Southparkfan committed rPUPCad26d58666fd: Refactor database backup role, add dbbackup2 (authored by Southparkfan).
Refactor database backup role, add dbbackup2
Sun, Feb 7, 02:37
Southparkfan changed the edit policy for P220 puppet install script.
Sun, Feb 7, 00:59
Southparkfan committed rDNSb4be19048530: Change dbbackup1 IPv6 and add dbbackup2 (authored by Southparkfan).
Change dbbackup1 IPv6 and add dbbackup2
Sun, Feb 7, 00:47
Southparkfan committed rPUPCc2af52778cc5: dbbackup1: remove replication role (authored by Southparkfan).
dbbackup1: remove replication role
Sun, Feb 7, 00:18

Tue, Feb 2

Southparkfan committed rDNSf5ee1a9f404d: Revert "Depool cp6" (authored by Southparkfan).
Revert "Depool cp6"
Tue, Feb 2, 12:16
Southparkfan added a reverting change for rDNSa4ed2b21ae43: Depool cp6: rDNSf5ee1a9f404d: Revert "Depool cp6".
Tue, Feb 2, 12:16
Southparkfan committed rDNSa4ed2b21ae43: Depool cp6 (authored by Southparkfan).
Depool cp6
Tue, Feb 2, 01:10
Southparkfan committed rDNS13297943c134: Revert "experiment: depool cp7" (authored by Southparkfan).
Revert "experiment: depool cp7"
Tue, Feb 2, 01:10
Southparkfan added a reverting change for rDNSce78772f0b88: experiment: depool cp7: rDNS13297943c134: Revert "experiment: depool cp7".
Tue, Feb 2, 01:10
Southparkfan committed rDNSce78772f0b88: experiment: depool cp7 (authored by Southparkfan).
experiment: depool cp7
Tue, Feb 2, 00:44

Mon, Feb 1

Southparkfan committed rDNSe39a985b8096: Add glustermigrtemp1 (authored by Southparkfan).
Add glustermigrtemp1
Mon, Feb 1, 23:41
Southparkfan committed rPUPC52d9abaac27a: add glustermigrtemp1 (authored by Southparkfan).
add glustermigrtemp1
Mon, Feb 1, 23:39
Southparkfan updated the task description for T6787: Migrate cloud1 and cloud2 virtual machines to new servers.
Mon, Feb 1, 23:10 · Infrastructure (SRE)
Southparkfan committed rPUPC07652479e75b: Load roles on new cache proxies and jobrunners (authored by Southparkfan).
Load roles on new cache proxies and jobrunners
Mon, Feb 1, 22:45
Southparkfan committed rPUPCf5abb4ab51d8: Add hieradata for jobrunner[34] (authored by Southparkfan).
Add hieradata for jobrunner[34]
Mon, Feb 1, 22:18
Southparkfan updated the task description for T6787: Migrate cloud1 and cloud2 virtual machines to new servers.
Mon, Feb 1, 22:09 · Infrastructure (SRE)
Southparkfan closed T6791: Replace cp9 with new VPS, a subtask of T6770: Renew or decommission cloud1, cloud2 and replace cp9, as Resolved.
Mon, Feb 1, 20:28 · Infrastructure (SRE)
Southparkfan closed T6791: Replace cp9 with new VPS as Resolved.

Done.

Mon, Feb 1, 20:28 · Infrastructure (SRE)
Southparkfan committed rDNS2630a1d4fdee: Remove cp9 (authored by Southparkfan).
Remove cp9
Mon, Feb 1, 20:23
Southparkfan committed rPUPC4e68cce031a1: Decom cp9 (authored by Southparkfan).
Decom cp9
Mon, Feb 1, 20:17
Southparkfan committed R9:b24deb7e8fb6: Remove cp9 (authored by Southparkfan).
Remove cp9
Mon, Feb 1, 20:16
Dmehus awarded T5541: Describe the position of Site Reliability Engineering and how people are located within the structure a Pterodactyl token.
Mon, Feb 1, 03:03 · Site Reliability Engineering
Southparkfan triaged T6802: puppet: convert all custom services to be installed using systemd::service as Low priority.
Mon, Feb 1, 00:46 · Infrastructure (SRE)
Southparkfan committed rPUPC6db5835e44bd: Merge branch 'master' of github.com:miraheze/puppet (authored by Southparkfan).
Merge branch 'master' of github.com:miraheze/puppet
Mon, Feb 1, 00:39
Southparkfan committed rPUPC0ae5f8b56d95: Varnish: log all 5xx requests on-disk to allow troubleshooting (authored by Southparkfan).
Varnish: log all 5xx requests on-disk to allow troubleshooting
Mon, Feb 1, 00:39

Sun, Jan 31

Southparkfan raised the priority of T5433: Evaluate regular SRE meetings from Low to Normal.
Sun, Jan 31, 23:56 · Site Reliability Engineering
Southparkfan triaged T6800: Create SLOs/SLIs for services as Low priority.
Sun, Jan 31, 23:54 · Site Reliability Engineering
Southparkfan updated the task description for T6799: New Server Resource Request for Database Backups.
Sun, Jan 31, 22:31 · Infrastructure (SRE)
Southparkfan added a comment to T4017: Reconfigure TLS settings inside MariaDB.

db13

UserSource of connectionsCurrent TLS specsSupports TLS 1.3 w/ AES-{128,256}-GCM?
grafana@%Grafana app (monX)??
icinga@%Icinga monitoring agents/scripts (monX)??
icinga2@%Icinga master config (monX)??
icingaweb2@%Icinga web interface? (monX)??
mediawiki@%MediaWiki app (mwX/testX)(hack in DatabaseMysqli.php on test2): TLS 1.2 ECDHE-RSA-AES128-GCM-SHA256No, TLS 1.3 is PHP 7.4+?
phabricator@%Phabricator app (phabX)?No, TLS 1.3 is PHP 7.4+?
piwik@%Matomo app (monX)?No, TLS 1.3 is PHP 7.4+?
replica@%Database replication (dbbackupX)?MariaDB master: (openssl s_client -starttls mysql) TLS_AES_128_GCM_SHA256 supported, TLS_AES_256_GCM_SHA384 supported. Replica connections unknwon.
root@localhostRoot access (mysql client on server)No TLS (but that's expected)Is supported (via local mysql client)
wikiadmin@%MediaWiki maintenance & jobrunner (jobrunnerX)sql.php (jobrunner1): TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256No, TLS 1.3 is PHP 7.4+?
roundcubemail@{2001:41d0:800:1056::9,51.89.160.134}Roundcube webmail (mailX)No TLS, has TLS supportNo, TLS 1.3 is PHP 7.4+?
Sun, Jan 31, 22:29 · Infrastructure (SRE), Goal-2019-Jul-Dec, Goal-2020-Jan-Jun
Southparkfan added a parent task for T6799: New Server Resource Request for Database Backups: T6071: Set up replicas for all database clusters.
Sun, Jan 31, 21:01 · Infrastructure (SRE)
Southparkfan added a subtask for T6071: Set up replicas for all database clusters: T6799: New Server Resource Request for Database Backups.
Sun, Jan 31, 21:01 · Infrastructure (SRE), Database
Southparkfan created T6799: New Server Resource Request for Database Backups.
Sun, Jan 31, 21:01 · Infrastructure (SRE)
Southparkfan added a comment to T6791: Replace cp9 with new VPS.

New server is in production, but cp9 needs to be decom'd.

Sun, Jan 31, 20:57 · Infrastructure (SRE)
Southparkfan added a subtask for T5877: Revise MariaDB backup strategy: T6071: Set up replicas for all database clusters.
Sun, Jan 31, 20:49 · Infrastructure (SRE), Goal-2021-Jan-Jun, Database, Goal-2020-Jul-Dec
Southparkfan added a parent task for T6071: Set up replicas for all database clusters: T5877: Revise MariaDB backup strategy.
Sun, Jan 31, 20:49 · Infrastructure (SRE), Database
Southparkfan moved T6071: Set up replicas for all database clusters from Incoming to Short Term on the Infrastructure (SRE) board.
Sun, Jan 31, 20:49 · Infrastructure (SRE), Database
Southparkfan edited projects for T6071: Set up replicas for all database clusters, added: Infrastructure (SRE); removed Site Reliability Engineering.
Sun, Jan 31, 20:49 · Infrastructure (SRE), Database
Southparkfan reopened T6071: Set up replicas for all database clusters as "Open".

Pending a new database backup server..

Sun, Jan 31, 20:48 · Infrastructure (SRE), Database
Southparkfan triaged T6796: Revise inactivity policy as Normal priority.
Sun, Jan 31, 18:47 · Site Reliability Engineering
Southparkfan moved T6791: Replace cp9 with new VPS from Incoming to Short Term on the Infrastructure (SRE) board.
Sun, Jan 31, 17:05 · Infrastructure (SRE)
Southparkfan committed rPUPCb8918641edc9: Add host header for blackbox exporter (authored by Southparkfan).
Add host header for blackbox exporter
Sun, Jan 31, 02:15
Southparkfan committed rPUPC3cd3f2835ea6: Merge branch 'master' of github.com:miraheze/puppet (authored by Southparkfan).
Merge branch 'master' of github.com:miraheze/puppet
Sun, Jan 31, 02:15
Southparkfan committed rPUPC5888720bff92: blackbox: monitor all cache proxies (authored by Southparkfan).
blackbox: monitor all cache proxies
Sun, Jan 31, 02:15
Southparkfan committed rPUPCd645d1924179: Fix blackbox exporter config (authored by Southparkfan).
Fix blackbox exporter config
Sun, Jan 31, 01:48
Southparkfan committed rPUPCd28e8ca32753: Enable blackbox exporter (authored by Southparkfan).
Enable blackbox exporter
Sun, Jan 31, 01:42
Southparkfan committed rPUPC5da4b3b461c0: Merge branch 'master' of github.com:miraheze/puppet (authored by Southparkfan).
Merge branch 'master' of github.com:miraheze/puppet
Sun, Jan 31, 01:42
Southparkfan committed rDNSebd3cb2daa42: Replace cp9 with cp12 (authored by Southparkfan).
Replace cp9 with cp12
Sun, Jan 31, 00:20

Sat, Jan 30

Southparkfan committed R9:306bb591a7a9: Merge branch 'master' of github.com:miraheze/mw-config (authored by Southparkfan).
Merge branch 'master' of github.com:miraheze/mw-config
Sat, Jan 30, 23:15
Southparkfan committed R9:e15ff015f05b: Add cp12 (authored by Southparkfan).
Add cp12
Sat, Jan 30, 23:15
Southparkfan committed rPUPC84246c42663f: nginx: set real IP for cp12 requests (authored by Southparkfan).
nginx: set real IP for cp12 requests
Sat, Jan 30, 23:09
Southparkfan committed rPUPC97bdb969ef30: Add cp12 hieradata (authored by Southparkfan).
Add cp12 hieradata
Sat, Jan 30, 22:49
Southparkfan committed rPUPC0db49958dfd0: Merge branch 'master' of github.com:miraheze/puppet (authored by Southparkfan).
Merge branch 'master' of github.com:miraheze/puppet
Sat, Jan 30, 22:41
Southparkfan committed rPUPC132cf45393f2: Add cp12 to site.pp (authored by Southparkfan).
Add cp12 to site.pp
Sat, Jan 30, 22:41
Southparkfan committed rDNS9e2f75385ba4: add cp12 (authored by Southparkfan).
add cp12
Sat, Jan 30, 22:30
Southparkfan triaged T6791: Replace cp9 with new VPS as High priority.
Sat, Jan 30, 22:17 · Infrastructure (SRE)
Southparkfan renamed T6770: Renew or decommission cloud1, cloud2 and replace cp9 from Renew or decommission cloud1 and cloud2 to Renew or decommission cloud1, cloud2 and replace cp9.
Sat, Jan 30, 22:17 · Infrastructure (SRE)
Southparkfan updated the task description for T6787: Migrate cloud1 and cloud2 virtual machines to new servers.
Sat, Jan 30, 22:16 · Infrastructure (SRE)
Southparkfan updated the task description for T6787: Migrate cloud1 and cloud2 virtual machines to new servers.
Sat, Jan 30, 22:15 · Infrastructure (SRE)
Southparkfan added a comment to T6770: Renew or decommission cloud1, cloud2 and replace cp9.

With the OK from John, Paladox & Reception: ordering immediately instead of waiting till Feb 1.

Sat, Jan 30, 21:22 · Infrastructure (SRE)
Southparkfan added a comment to T6787: Migrate cloud1 and cloud2 virtual machines to new servers.

Per-server caveats noted at https://etherpad.wikimedia.org/p/Migration_to_new_infrastructure. Copy-pasted here since etherpad's data may be truncated at any time by Wikimedia.

Sat, Jan 30, 16:30 · Infrastructure (SRE)
Southparkfan renamed T6787: Migrate cloud1 and cloud2 virtual machines to new servers from Migrate clooud1 and cloud2 virtual machines to new servers to Migrate cloud1 and cloud2 virtual machines to new servers.
Sat, Jan 30, 00:38 · Infrastructure (SRE)
Southparkfan triaged T6787: Migrate cloud1 and cloud2 virtual machines to new servers as High priority.
Sat, Jan 30, 00:27 · Infrastructure (SRE)

Fri, Jan 29

Reception123 awarded T6770: Renew or decommission cloud1, cloud2 and replace cp9 a Like token.
Fri, Jan 29, 06:35 · Infrastructure (SRE)

Thu, Jan 28

Southparkfan reassigned T6730: Custom Access-Control-Allow-Origin from Southparkfan to Universal_Omega.

https://stackoverflow.com/a/56457665: "Access-Control-Allow-Origin: * is totally safe to add to any resource, unless that resource contains private data protected by something other than standard credentials. Standard credentials are cookies, HTTP basic auth, and TLS client certificates."

Thu, Jan 28, 23:34 · Infrastructure (SRE), Universal Omega, Puppet
Southparkfan added a comment to T6770: Renew or decommission cloud1, cloud2 and replace cp9.

For the record: cloud4/5 will be OVH Advance-2 64GB 2x4TB servers with 12-month commitment.

Thu, Jan 28, 23:23 · Infrastructure (SRE)
John awarded T6770: Renew or decommission cloud1, cloud2 and replace cp9 a Like token.
Thu, Jan 28, 23:20 · Infrastructure (SRE)
Southparkfan updated subscribers of T6770: Renew or decommission cloud1, cloud2 and replace cp9.

Approvals Policy: "30.00% or greater: sign-off from budget holder, one Senior Site Reliability Engineer (if applicable), and: at least two, or 50%3of Site Reliability Engineers (rounded up to above), whichever is higher"

Thu, Jan 28, 22:57 · Infrastructure (SRE)
Southparkfan added a comment to T6770: Renew or decommission cloud1, cloud2 and replace cp9.

Current infra costs (incl VAT)
cloud1 (OVH CA): $73.74/mo = ~£53.70/mo
cloud2 (OVH CA): $73.74/mo = ~£53.70/mo
cloud3 (OVH CA): $113.99/mo = ~£83.02/mo
cp9 (OVH CA): $6.00/mo= ~£4.37/mo
bacula2 (RN): $18/mo = ~£13.11/mo
dbbackup1 (RN): $24/mo = ~£17.48/mo
ns1 (RN): $16.20/yr = $1.35/mo = ~£0.99/mo
Total: $309.47/mo = ~£225.39/mo

Thu, Jan 28, 22:50 · Infrastructure (SRE)

Jan 24 2021

Southparkfan triaged T6770: Renew or decommission cloud1, cloud2 and replace cp9 as High priority.
Jan 24 2021, 23:22 · Infrastructure (SRE)
Southparkfan added a comment to T6756: Prevent Steward access to staffwiki per board motion.
In T6756#133156, @John wrote:

Per https://www.mediawiki.org/wiki/Manual:Hooks/UserGetRights#Usage, the https://www.mediawiki.org/wiki/Manual:Hooks/UserGetRightsRemove hook is the better choice. We are interested in a MirahezeMagic hook that:

  1. checks if the user has read rights, either by global or local rights; and
  2. verifies the user by looking at an immutable key (immutable unless you have shell access), so the centralauth user ID is good enough (whereas a username isn't, since a steward can rename a user); and
  3. revokes the read right if the user did not pass the check from 2).

A much easier step 2 would be to check for a local user group assigned to the user, if that's not met, remove 'read'.

Users with the userrights-interwiki right can overrule this behavior, though. Unfortunately, rights are not variables that are immutable to people without shell access.

Jan 24 2021, 22:41 · MediaWiki
Southparkfan added a comment to T6756: Prevent Steward access to staffwiki per board motion.

@RhinosF1 do you feel comfortable implementing this?

Jan 24 2021, 20:50 · MediaWiki
Southparkfan added a comment to T6756: Prevent Steward access to staffwiki per board motion.

Per https://www.mediawiki.org/wiki/Manual:Hooks/UserGetRights#Usage, the https://www.mediawiki.org/wiki/Manual:Hooks/UserGetRightsRemove hook is the better choice. We are interested in a MirahezeMagic hook that:

  1. checks if the user has read rights, either by global or local rights; and
  2. verifies the user by looking at an immutable key (immutable unless you have shell access), so the centralauth user ID is good enough (whereas a username isn't, since a steward can rename a user); and
  3. revokes the read right if the user did not pass the check from 2).
Jan 24 2021, 20:48 · MediaWiki
Southparkfan added a member for acl*security_reviewers: R4356th.
Jan 24 2021, 17:49