Page MenuHomeMiraheze

Southparkfan (Ferran Tufan)
Director of Site Reliability EngineeringAdministrator

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Apr 17 2016, 19:18 (241 w, 5 d)
Roles
Administrator
Availability
Available
IRC Nickname
SPF|Cloud
GitHub User
Southparkfan
Miraheze User
Southparkfan [ Global Accounts ]

Hi! I am Southparkfan; co-founder and system administrator for Miraheze. I am responsible for the smooth operation of Miraheze's servers, which includes applying configuration changes, conducting maintenance and incident investigations, performance tuning, monitoring the servers and other miscellaneous tasks.

You can usually find me on IRC in the #miraheze channel on chat.freenode.net.

Recent Activity

Sat, Nov 28

Southparkfan placed T6431: cp9 extremely latent for a number of users up for grabs.

I find no abnormal metrics regarding cp9, nor do I have any other evidence saying cp9 is constrained by the available resources. Running mtr towards @Dmehus' IP shows some terrible latencies at multiple hops. Since @John has lots of expertise with networking, I'm interested if he could narrow this down to OVH network issues.

Sat, Nov 28, 21:29 · Site Reliability Engineering

Sun, Nov 22

Southparkfan committed rPUPC3480179080cd: jobrunner/jobchron: redirect stdout/stderr to syslog (authored by Southparkfan).
jobrunner/jobchron: redirect stdout/stderr to syslog
Sun, Nov 22, 22:58
Southparkfan added a comment to T5044: Setup centralised logging for services.

With docs now: https://meta.miraheze.org/wiki/Tech:Graylog

Sun, Nov 22, 22:14 · Goal-2020-Jul-Dec, Goal-2020-Jan-Jun, Site Reliability Engineering

Sat, Nov 21

Southparkfan removed a project from T4017: Reconfigure TLS settings inside MariaDB: Goal-2020-Jul-Dec.
Sat, Nov 21, 19:22 · Goal-2019-Jul-Dec, Goal-2020-Jan-Jun, Site Reliability Engineering
Southparkfan removed a project from T5714: Redesign implementation of adding own TLS certificates and CAs: Goal-2020-Jul-Dec.
Sat, Nov 21, 19:21 · Site Reliability Engineering
Southparkfan removed a project from T4005: Execute external commands on MediaWiki servers inside sandboxes: Goal-2020-Jul-Dec.
Sat, Nov 21, 19:21 · Security, Site Reliability Engineering, MediaWiki
Southparkfan removed a project from T4016: Encrypt all traffic inside Miraheze Cluster: Goal-2020-Jul-Dec.
Sat, Nov 21, 19:21 · Goal-2019-Jul-Dec, Goal-2020-Jan-Jun, Site Reliability Engineering
Southparkfan removed a project from T4019: Encrypt Redis traffic: Goal-2020-Jul-Dec.
Sat, Nov 21, 19:20 · Goal-2019-Jul-Dec, Goal-2020-Jan-Jun, Site Reliability Engineering
Southparkfan removed a project from T5537: Replace SaltStack with Cumin: Goal-2020-Jul-Dec.
Sat, Nov 21, 19:20 · Site Reliability Engineering
Southparkfan removed a project from T5624: Create responsible disclosure policy: Goal-2020-Jul-Dec.
Sat, Nov 21, 19:20 · Security, Site Reliability Engineering

Sun, Nov 15

Southparkfan added a comment to T6438: [Access Request] Universal Omega for mw-admin.

I support this request.

Sun, Nov 15, 19:35 · Universal Omega, Site Reliability Engineering

Sun, Nov 8

Southparkfan reassigned T6360: Naleksuh as security reviewer from Southparkfan to Universal_Omega.

Are there any contributions to MediaWiki core or extensions that can serve as evidence they are familiar with the MediaWiki architecture?

Sun, Nov 8, 16:10 · Universal Omega, Site Reliability Engineering

Oct 24 2020

Southparkfan reassigned T6312: SamanthaNguyen as security reviewer from Southparkfan to SamanthaNguyen.

Hi @SamanthaNguyen, welcome! Good to see interest for this position.

Oct 24 2020, 18:41 · Site Reliability Engineering
Southparkfan lowered the priority of T6095: Lack of PK on RottenLinks tables causes huge replication lag from High to Normal.

I was deploying the ALTERs to the wikis while RottenLinks was disabled (the ALTERs cannot be done live using pt-osc, unfortunately), but I forgot to re-enable RottenLinks. For now, I have enabled RottenLinks. The extension only needs to be disabled again (and the update scripts killed) if you are deploying the schema changes on databases.

Oct 24 2020, 18:14 · RottenLinks, Database, Site Reliability Engineering

Oct 18 2020

Southparkfan reassigned T5863: Extension:Score disabled due to multiple security issues from Southparkfan to RhinosF1.

Let's keep the extension disabled, following Tim's advice at https://phabricator.wikimedia.org/T257066#6364537. However, a note must be in place.

Oct 18 2020, 15:45 · Universal Omega, Extensions, Security

Oct 13 2020

Southparkfan lowered the priority of T5624: Create responsible disclosure policy from Normal to Low.
Oct 13 2020, 21:42 · Security, Site Reliability Engineering
Southparkfan lowered the priority of T5713: Create automated Icinga check for validity of all TLS certificates on system from Normal to Low.
Oct 13 2020, 21:41 · Monitoring, Site Reliability Engineering
Southparkfan added a comment to T6095: Lack of PK on RottenLinks tables causes huge replication lag.

@John it doesn't look like larger VARCHARs are possible. Even if you don't use the rl_id field, it does seem to be working fine at first glance, without fundamentally changing the schema (the current, old fields can stay). What do you think?

Oct 13 2020, 21:40 · RottenLinks, Database, Site Reliability Engineering

Oct 10 2020

Southparkfan added a comment to T6095: Lack of PK on RottenLinks tables causes huge replication lag.
02:56:34 <+SPF|Cloud> JohnLewis: db7 doesn't seem to accept a varchar(8192) for rl_externallink
02:56:38 <+SPF|Cloud> ERROR 1071 (42000): Specified key was too long; max key length is 3072 bytes
03:01:39 <+SPF|Cloud> looking at https://mariadb.com/kb/en/innodb-system-variables/#innodb_large_prefix and https://mariadb.com/kb/en/innodb-large_prefix-deprecated-resulting-key-length/, relying on such huge varchars for index keys seems deprecated
03:09:42 <+SPF|Cloud> I don't think having the varchars as primary key is a good idea, given the deprecation comments. What do you think?
Oct 10 2020, 01:54 · RottenLinks, Database, Site Reliability Engineering
Southparkfan reassigned T6283: Exempt IABot from the Varnish rate limit from Southparkfan to Cyberpower678.

@Cyberpower678 sending 45 requests per second is still a lot. Remember this is a multi-tenant environment with few spare resources. However, if you can give me a combination of source IP and User-Agent to exempt for the rate limit temporarily, I can see how much issues are actually caused in production by the bot. I'm curious.

Oct 10 2020, 01:08 · Site Reliability Engineering, Varnish

Oct 9 2020

Southparkfan reopened T6095: Lack of PK on RottenLinks tables causes huge replication lag as "Open".

While the change was committed, the change has not been applied to existing tables.

Oct 9 2020, 21:47 · RottenLinks, Database, Site Reliability Engineering

Oct 8 2020

Southparkfan committed rPUPC66ade59bd633: Merge branch 'master' of github.com:miraheze/puppet (authored by Southparkfan).
Merge branch 'master' of github.com:miraheze/puppet
Oct 8 2020, 23:15
Southparkfan committed rPUPCfaa835b6d55b: add hostname in base::syslog + remove custom nginx sources (authored by Southparkfan).
add hostname in base::syslog + remove custom nginx sources
Oct 8 2020, 23:15
Southparkfan committed rPUPC98a45f9cc933: base::syslog: add local udp source (authored by Southparkfan).
base::syslog: add local udp source
Oct 8 2020, 23:15
Southparkfan added a comment to T6283: Exempt IABot from the Varnish rate limit.

@Cyberpower678 I can confirm on cp9 (cache proxy) traffic from IABot is being rate limited. However, sending over 40 requests per second (on average) to api.php (and thus requests that bypass the cache, and have to be served by the MediaWiki cluster) is in most cases not acceptable at all. We only send about 70 requests per second to the MediaWiki cluster.

Oct 8 2020, 20:52 · Site Reliability Engineering, Varnish

Oct 4 2020

Southparkfan claimed T5603: Comment function does not work.
Oct 4 2020, 18:18 · Universal Omega, Production Error, Extensions
Southparkfan closed T5603: Comment function does not work as Resolved.

@J-Josyu thank you for the HAR file. That was exactly what I needed.

Oct 4 2020, 18:17 · Universal Omega, Production Error, Extensions

Oct 3 2020

Southparkfan committed rPUPC0bc0e8aff1a6: Introduce base support for syslog-ng (authored by Southparkfan).
Introduce base support for syslog-ng
Oct 3 2020, 15:43
Southparkfan committed rPUPC50d7f5c2f376: Merge branch 'master' of github.com:miraheze/puppet (authored by Southparkfan).
Merge branch 'master' of github.com:miraheze/puppet
Oct 3 2020, 15:43
Southparkfan committed rPUPC750c8ccb0cff: Graylog: fix broken elasticsearch setup (authored by Southparkfan).
Graylog: fix broken elasticsearch setup
Oct 3 2020, 12:08
Southparkfan added a comment to T5603: Comment function does not work.

Sure, I'm bad, but which one isn't sincere? Of course, the bad thing is the "volunteers" who don't solve the problem.If you call yourself a volunteer, do what you do.

I see multiple volunteers were involved in fixing a part of this issue and trying to reproduce your other issues. Your attitude does not help solving this task. Please read http://meta.miraheze.org/wiki/Code_of_Conduct and stay civil. Thanks.

Oct 3 2020, 11:39 · Universal Omega, Production Error, Extensions
Southparkfan committed rPUPCd20de653afa0: Add graylog module, update java module, update graylog.pp (authored by Southparkfan).
Add graylog module, update java module, update graylog.pp
Oct 3 2020, 03:09
Southparkfan committed rPUPC726dcf802be6: Commit more fixes for graylog? (authored by Southparkfan).
Commit more fixes for graylog?
Oct 3 2020, 02:51
Southparkfan committed rPUPC35f6390e8815: Fix graylog file (authored by Southparkfan).
Fix graylog file
Oct 3 2020, 02:36
Southparkfan committed rPUPCa53dc5a5515f: add base config for graylog (authored by Southparkfan).
add base config for graylog
Oct 3 2020, 02:29
Southparkfan committed rPUPC1fd943a0cabf: Increase fallback rate limit from 5 to 12 requests per 2s (authored by Southparkfan).
Increase fallback rate limit from 5 to 12 requests per 2s
Oct 3 2020, 01:15
Southparkfan closed T6146: Varnish/Nginx? returning 429 due to DDoS/SQLi mitigations when rendering Math Images in some cases as Resolved.

Permanent fixes have been implemented for this issue, closing the task. If you still have issues, please reopen this task.

Oct 3 2020, 00:25 · Site Reliability Engineering, Extensions, Universal Omega
Southparkfan committed rPUPC45320469f4fa: Raise rate limit for Parsoid (authored by Southparkfan).
Raise rate limit for Parsoid
Oct 3 2020, 00:21
Southparkfan committed rDNSd82899f3f218: Revert "Depool cp7" (authored by Southparkfan).
Revert "Depool cp7"
Oct 3 2020, 00:09
Southparkfan added a reverting change for rDNS8ddc95087976: Depool cp7: rDNSd82899f3f218: Revert "Depool cp7".
Oct 3 2020, 00:09
Southparkfan committed rPUPC27086ae44f33: Move rate limit at cache proxies from NGINX to Varnish (authored by Southparkfan).
Move rate limit at cache proxies from NGINX to Varnish
Oct 3 2020, 00:00

Oct 1 2020

Southparkfan committed rPUPCb2b825694638: Deploy new wildcard cert for grafana.mh.o + remove grafana-new.mh.o (authored by Southparkfan).
Deploy new wildcard cert for grafana.mh.o + remove grafana-new.mh.o
Oct 1 2020, 20:10
Southparkfan committed rPUPC10003dc3daf7: Deploy new TLS wildcard cert on mediawiki services (authored by Southparkfan).
Deploy new TLS wildcard cert on mediawiki services
Oct 1 2020, 20:05
Southparkfan committed rPUPC95948bc09c06: Do not use wildcard certificate for authenticating to ldap/mariadb (authored by Southparkfan).
Do not use wildcard certificate for authenticating to ldap/mariadb
Oct 1 2020, 20:01
Southparkfan committed rPUPC4ea382c9e719: Include ssl::wildcard in dovecot and postfix manifests (authored by Southparkfan).
Include ssl::wildcard in dovecot and postfix manifests
Oct 1 2020, 19:27
Southparkfan committed rPUPCcb1505528afc: Use 2020 wildcard certificate for mediawiki, cache proxies, postfix and dovecot (authored by Southparkfan).
Use 2020 wildcard certificate for mediawiki, cache proxies, postfix and dovecot
Oct 1 2020, 19:22
Southparkfan committed rPUPC198740fff6e1: Deploy new wildcard certificate for icinga2, roundcube, matomo (authored by Southparkfan).
Deploy new wildcard certificate for icinga2, roundcube, matomo
Oct 1 2020, 19:11
Southparkfan added a comment to T6234: Renew Miraheze Wildcard SSL certificate.
  • phabricator
  • cache proxies (varnish)
  • mediawiki nginx
  • ldap server
  • icinga nginx
  • mariadb <- careful!
  • mw services
    • parsoid
    • mathoid
    • restbase
  • dovecot
  • postfix
  • matomo nginx
  • roundcubemail nginx
  • grafana nginx
  • gluster
    • gluster clients: mediawiki
  • postgresql
Oct 1 2020, 19:08 · SSL, Site Reliability Engineering
Southparkfan added a comment to T6234: Renew Miraheze Wildcard SSL certificate.

Certificate purchased. Deployment going on.

Oct 1 2020, 19:04 · SSL, Site Reliability Engineering
Southparkfan committed rPUPC70e1de3f9841: Deploy new wildcard certificate for phabricator (authored by Southparkfan).
Deploy new wildcard certificate for phabricator
Oct 1 2020, 19:02

Sep 20 2020

Southparkfan added a comment to T6146: Varnish/Nginx? returning 429 due to DDoS/SQLi mitigations when rendering Math Images in some cases.

Waiting for response from @Paladox regarding nginx changes.

Sep 20 2020, 13:01 · Site Reliability Engineering, Extensions, Universal Omega
Southparkfan closed T6056: 18/19-08-2020 cp* failures as Resolved.

Seems resolved.

Sep 20 2020, 13:00 · Amanda Catherine, Site Reliability Engineering
Southparkfan added a comment to T5877: Revise MariaDB backup strategy.

Contacted Owen for a data processing agreement for the free infra offers.

Sep 20 2020, 13:00 · Site Reliability Engineering, Database, Goal-2020-Jul-Dec

Sep 16 2020

Nomalias awarded T6146: Varnish/Nginx? returning 429 due to DDoS/SQLi mitigations when rendering Math Images in some cases a Love token.
Sep 16 2020, 02:53 · Site Reliability Engineering, Extensions, Universal Omega

Aug 24 2020

Southparkfan triaged T6095: Lack of PK on RottenLinks tables causes huge replication lag as High priority.
Aug 24 2020, 22:55 · RottenLinks, Database, Site Reliability Engineering
Southparkfan triaged T6094: gluster servers running out of space as High priority.
Aug 24 2020, 22:50 · Site Reliability Engineering
Southparkfan added a project to T6093: Catch dns.resolver.NoAnswer properly inside reverse DNS check: Site Reliability Engineering.
Aug 24 2020, 22:49 · Site Reliability Engineering, Monitoring
Southparkfan triaged T6093: Catch dns.resolver.NoAnswer properly inside reverse DNS check as Normal priority.
Aug 24 2020, 22:49 · Site Reliability Engineering, Monitoring
Southparkfan added a comment to T6064: Request to be security reviewer.

@Universal_Omega seems fine to me. Welcome. Do you have MFA enabled on your Phabricator? If so, we'll add you to acl*security_reviewers.

Aug 24 2020, 22:07 · Site Reliability Engineering
Southparkfan set the icon for acl*security_reviewers to Policy.
Aug 24 2020, 22:04
Southparkfan set the image for acl*security_reviewers to F1243031: profile.
Aug 24 2020, 22:03
Southparkfan created acl*security_reviewers.
Aug 24 2020, 22:03

Aug 20 2020

Southparkfan lowered the priority of T6056: 18/19-08-2020 cp* failures from High to Normal.

Rate limiting implemented on cache proxies, monitoring now.

Aug 20 2020, 20:58 · Amanda Catherine, Site Reliability Engineering
Southparkfan claimed T6056: 18/19-08-2020 cp* failures.
Aug 20 2020, 14:02 · Amanda Catherine, Site Reliability Engineering
Southparkfan lowered the priority of T6056: 18/19-08-2020 cp* failures from Unbreak Now! to High.
Aug 20 2020, 14:01 · Amanda Catherine, Site Reliability Engineering
Southparkfan created T6071: Set up replicas for all database clusters.
Aug 20 2020, 14:01 · Site Reliability Engineering, Database

Aug 19 2020

Southparkfan added a comment to T6064: Request to be security reviewer.

Good to see interest in this job! I would recommend doing an actual security review of a pending extension, using the checklist from https://www.mediawiki.org/wiki/Security_checklist_for_developers. Note the good (e.g. CSRF tokens used, code being compliant with MediaWiki's standards and thus making the review much easier) and bad points (e.g. a lot of htmlspecialchars usage within echo statements, instead of using the Html functions) of the extension's code, security risks you find (e.g. accessing external resources, thus introducing DoS possibilities, passing user input directly into shell commands). Especially for a somewhat larger extension, that gives me an idea about your review capabilities. If I think you are capable of doing security reviews for real, you may be grant permission to approve extensions without being in a SRE position or similar.

Aug 19 2020, 21:35 · Site Reliability Engineering
Southparkfan added a member for Site Reliability Engineering: RhinosF1.
Aug 19 2020, 21:27

Aug 13 2020

Southparkfan shifted T6013: Active SQL injection attack against some wikis from the Restricted Space space to the S1 Public space.
Aug 13 2020, 20:04 · MediaWiki, Site Reliability Engineering, Security
Southparkfan closed T6013: Active SQL injection attack against some wikis as Resolved.
Aug 13 2020, 20:03 · MediaWiki, Site Reliability Engineering, Security

Aug 7 2020

Southparkfan closed T6022: Custom domain name request as Resolved.

https://wiki.kali-team.cn/ works now.

Aug 7 2020, 00:59 · SSL
Southparkfan closed T6015: Subdomain request for LeFrenchMelee wiki as Invalid.

@Exile I can't reproduce this, the wiki works fine with me with that URL. Please let us know if you still experience issues.

Aug 7 2020, 00:43 · SSL
Southparkfan closed T6005: Changing URL for TLAwiki as Resolved.

tl.awiki.org works now.

Aug 7 2020, 00:42 · SSL
Southparkfan added a comment to T6019: Improve how we handle ToU actions.

@Zppix and this is a security-sensitive task because...?
(and what are the concrete actions here?)

Aug 7 2020, 00:27 · Site Reliability Engineering, acl*security
Southparkfan moved T5624: Create responsible disclosure policy from Backlog to Site Reliability Engineering on the Goal-2020-Jul-Dec board.
Aug 7 2020, 00:26 · Security, Site Reliability Engineering
Southparkfan added a project to T5624: Create responsible disclosure policy: Goal-2020-Jul-Dec.
Aug 7 2020, 00:26 · Security, Site Reliability Engineering
Southparkfan changed the visibility for T6012: Update to Debian Buster 10.5.
Aug 7 2020, 00:26 · Site Reliability Engineering, Security
Southparkfan assigned T6012: Update to Debian Buster 10.5 to Paladox.

The only relevant CVE here is the SecureBoot one, which doesn't matter at Miraheze (due to our configuration). Making the task public now. I don't see other fixes important for us either, don't see the need for quickly upgrading (which requires reboots of cloud servers as well).

Aug 7 2020, 00:25 · Site Reliability Engineering, Security
Southparkfan added a comment to T6013: Active SQL injection attack against some wikis.

Not observing now, monitoring phase for now.,

Aug 7 2020, 00:11 · MediaWiki, Site Reliability Engineering, Security

Aug 6 2020

Southparkfan lowered the priority of T6013: Active SQL injection attack against some wikis from Unbreak Now! to High.
Aug 6 2020, 21:17 · MediaWiki, Site Reliability Engineering, Security

Jul 30 2020

Southparkfan added a comment to T5788: Install the Aether, Discourse and Nimbus skins.

@Southparkfan Actually, Nimbus is blocked by Upstream per T5450#108244 (was already reviewed by Sam Wilson)

Jul 30 2020, 22:04 · Universal Omega, Configuration, Extensions
Southparkfan claimed T5788: Install the Aether, Discourse and Nimbus skins.

Nimbus approved, Aether not sure about one thing (P340). Discourse is declined since it's not maintained and I have some doubts about certain strategies used for the skin templating.

Jul 30 2020, 21:43 · Universal Omega, Configuration, Extensions
Southparkfan moved T5862: Adding the TwitterTag extension to the Hololive Fan Wiki from Review Needed to Reviewed Accepted on the Extensions board.
Jul 30 2020, 21:29 · Configuration, Extensions
Southparkfan added a comment to T5862: Adding the TwitterTag extension to the Hololive Fan Wiki.

Approved.

Jul 30 2020, 21:29 · Configuration, Extensions
Southparkfan moved T5815: Extension:JsCalendar from Review Needed to Reviewed Accepted on the Extensions board.
Jul 30 2020, 21:27 · Universal Omega, Configuration, Extensions
Southparkfan added a comment to T5815: Extension:JsCalendar.

Looks fine to me as well. Approved.

Jul 30 2020, 21:27 · Universal Omega, Configuration, Extensions
Southparkfan assigned T5750: File extensions + skin install to Paladox.

@Paladox, what was the reason this extension was declined as 'hard/impossible to install'?

Jul 30 2020, 21:10 · Extensions, Configuration
Southparkfan claimed T5590: Extension Re-review Request: EmbedVideo.

Give me a few days to think about it.

Jul 30 2020, 21:03 · Universal Omega, Extensions
Southparkfan claimed T5477: Extension:RatePage.

One potential finding: https://phabricator.miraheze.org/P339

Jul 30 2020, 20:59 · Universal Omega, Extensions
Southparkfan added a comment to T5024: TinyMCE alongside or instead of VisualEditor.

The bundled version of TinyMCE has various vulnerabilities: https://snyk.io/test/npm/tinymce/4.6.4

Jul 30 2020, 20:32 · Universal Omega, Extensions

Jul 29 2020

Southparkfan updated the task description for T5988: Review all custom domains with unusual rDNS entries.
Jul 29 2020, 21:17 · SSL
Southparkfan assigned T5988: Review all custom domains with unusual rDNS entries to RhinosF1.
Jul 29 2020, 21:17 · SSL
Southparkfan created T5988: Review all custom domains with unusual rDNS entries.
Jul 29 2020, 21:12 · SSL
Southparkfan committed rPUPC7b684cd7dbcc: Change output of check_reverse_dns for IPs without PTR record (authored by Southparkfan).
Change output of check_reverse_dns for IPs without PTR record
Jul 29 2020, 20:57
Southparkfan committed rPUPCb48e7e321f54: Replace DNS check with customised script (authored by Southparkfan).
Replace DNS check with customised script
Jul 29 2020, 20:38

Jul 15 2020

Southparkfan committed rPUPCa448260b48ba: Fix spelling mistake (authored by Southparkfan).
Fix spelling mistake
Jul 15 2020, 14:49
Southparkfan committed rPUPC90776b62dd8d: Add NTP check (authored by Southparkfan).
Add NTP check
Jul 15 2020, 14:47

Jul 14 2020

Southparkfan committed rDNS0f75e23ddcbc: Switch cp8 to cp9 (authored by Southparkfan).
Switch cp8 to cp9
Jul 14 2020, 23:20
Southparkfan committed R9:7fba7e4ae220: Add cp9 (authored by Southparkfan).
Add cp9
Jul 14 2020, 23:16
Southparkfan edited P220 puppet install script.
Jul 14 2020, 22:10