Running dump from db11 to dbbackup1:/srv/backups/db11. @Paladox and I are around to monitor.

New performance test (using sshfs setup, 4 mydumper threads):

• Uncompressed: 290 seconds
• Compressed: 210 seconds

For reference: mydumper is superior to mysqldump due to its better performance (using multiple threads) and the flexibility (PCRE based table inclusion/exclusion) in conjunction with transaction consistency and (almost) no locking (no read-only time required during backups). However, mydumper does not support TLS in connections, so dumping must happen at the database master.

In T7087#140328, @RhinosF1 wrote:

In T7087#140325, @Southparkfan wrote:

We have the blackbox exporter for this. Can we help you by monitoring specific URLs?

As mentioned in the task, /healthcheck is the biggest one because it has an effect on uptime if that gets too high.

/healthcheck = Meta's Main Page. We're already monitoring that.

I would recommend we do one that loads quite a few resources (eg. Images, javascript etc)

The blackbox exporter does not monitor subsequent requests, such as resources (images?) used on an article. We can monitor that though, but you'll need to provide specific URLs. :)

We have the blackbox exporter for this. Can we help you by monitoring specific URLs?

In T7073#140060, @John wrote:

Is there a use case for this that the ES data source wouldn’t fulfil? Is this the approach MediaWiki (SRE) wish to take? If so this would fall under the MW team to implement as part of their task as without a use case for Infra, what’s the point in implementing something unused?

There are more use cases than MediaWiki only. For example, I would like to monitor SSH authentication attempts and access logs of non-MediaWiki services, which is a task for us, not for the MediaWiki team. The proof of concept above was tailored for MediaWiki logs, because said logs have a higher priority.

Proof of concept:
/etc/prometheus-es-exporter/mediawiki.cfg:

[query_log_mediawiki]
QueryIntervalSecs = 900
QueryIndices = <graylog_deflector>
QueryJson = {
    "size": 0,
    "track_total_hits": true,
        "query": {
                "bool": {
                        "must": [
                                {
                                        "match": {
                                                "application_name": "mediawiki"
                                        }
                                }
                        ],
                        "filter": [
                                {
                                        "range": {
                                                "timestamp": { "gte": "now-15m", "lte": "now" }
                                        }
                                }
                        ]
                }
        },
        "aggs": {
                "mediawiki-channels": {
                        "terms": {
                                "field": "mediawiki_channel"
                        }
                }
        }
    }

The rebuild should finish in 10-30 minutes. If the errors are gone after the rebuild, you can close this task. If not, assistance will be needed.

Already did that for 'en', but without luck. Started the rebuild in a screen now.

More testing is required to determine the final backup sizes.

In T5877#139273, @Southparkfan wrote:

A maintenance window is required for dumping from masters directly. Not because impact is guaranteed, but because dumping may cause database locks for multiple seconds, hence increasing save time or knocking wikis offline.

In T7067#139966, @Reception123 wrote:

Just noting that it's been decided to discontinue SRE duty due to the new team system and it didn't seem to be functioning anymore. The dashboard and links we've compiled have still been kept though as they're useful.

In order to do proper backend verification in the certificate (CN), we have tested using ENFORCE. However, the Host header from the client (e.g. allthetropes.org) is used for the CN check at the backend. Therefore, the allthetropes.org certificate would still be mandatory at the backend, even though I prefer to remove all certificates (including our wildcard one) but a single domain (such as ats-internal.miraheze.wiki) from the MediaWiki servers.

The future of these servers depends on the outcome of testing regarding T5877#139273.

In T7033#139663, @Paladox wrote:

@Southparkfan should we make this task public viewable?

I recommend rebooting the dbbackup servers. They may or may not be affected by CVE-2021-3450, but as long as these servers are rebooted gracefully, we can survive without them for a few minutes.

I cannot find the minion in /etc/salt/roster.

Servers that haven't been rebooted, except for db1[1-3] / cloud[3-5] / mon2 / ns[12]:

• dbbackup1
• dbbackup2
• mem1
• mem2

Fixed.

RamNode is short on capacity, so we can't resize bacula yet. I hope we can resize the server next week.

+$5/mo is approved by me, only requires John's approval as the EM of Infrastructure.

A maintenance window is required for dumping from masters directly. Not because impact is guaranteed, but because dumping may cause database locks for multiple seconds, hence increasing save time or knocking wikis offline.

Spoke with @Paladox regarding ATS. Installing and testing ATS on test3 is not ideal, since that server is used for MediaWiki tests. Installing a new server as a testing cache proxy, granted that this cache proxy may not receive the 'allow 80/443 tcp' rules yet due to security reasons (we have agreed on a security review beforehand), has my support.

In T7003#138411, @RhinosF1 wrote:

Sounds Upstream.

Agreed. There are multiple issues here:

• Unclosed literals (with a single quote) are not a problem in column names, the check is too strict here
• If the bug above won't be fixed, then the extension lacks basic input validation upon creating a table

In T6985#138392, @Lakelimbo wrote:

HOWEVER
I still have to mention an actual bug: I can't see one of my tables for some reason, but it works normally. This happened just after the recent outage.

Just commenting because I don't know if it's worth a task (considering the table is working as intended, just its the page not working)

Opened T7003

Exception is generated at https://github.com/wikimedia/mediawiki-extensions-Cargo/blob/a619875ea82539bfbf525b8813036876e5cf39b4/includes/CargoUtils.php#L408
$string is string(11) "King's_Rock"
King's_Rock is a column in the cargo__Moves table:

stdClass Object
(
    [Field] => King's_Rock
    [Type] => tinyint(1)
    [Null] => YES
    [Key] => MUL
    [Default] =>
    [Extra] =>
)

Firejail helps to secure external binaries, but apparently firejail itself doesn't have a good track record either: https://github.com/netblue30/firejail/issues/3046

I've finally found the ticket, pasting my IRC comment here:
22:37:46 <+SPF|Cloud> @SRE, I can't recall who was talking about it (and where I read it), but I saw some messages regarding automating the addition of a new certificate (for https). have you considered https://wikitech.wikimedia.org/wiki/Acme-chief?

Considering the recent outages and the mention of MariaDB

I doubt it.

Perhaps, it may be possible to directly dump from the masters, with very little interruption: https://stackoverflow.com/q/56715657.
In that case, we can use the RamNode VMs to store the logical dumps (mydumper to stdout | ssh - local file). The disadvantage is that we won't have a live replica at all times (if a master crashes for good, the data between <most recent backup> and <crash> will be lost), but it's much cheaper: I/O limit is not much of an issue and since data is not replicated, there is more space for storing logical dumps.

Collecting real data: T6981
Blackbox testing: https://github.com/prometheus/blackbox_exporter (currently being tested)

I am a huge fan of data driven decision making!

Available exporters: https://github.com/jcollie/openldap_exporter https://github.com/tomcz/openldap_exporter
WMF (for dashboard examples): https://grafana.wikimedia.org/d/DnxQ26qmk/ldap?orgId=1 / https://phabricator.wikimedia.org/T181511

@Paladox I have disabled c4 replication on dbbackup1, but the lag is not decreasing. It looks like dbbackup1 still doesn't have enough room to replicate a full database cluster. Do you see room for improvements?

In T6979#138313, @John wrote:

https://grafana.com/docs/grafana/latest/datasources/elasticsearch/ data could alternatively be collected directly via ES potentially

I told you on IRC yesterday, but it's good practice to document this on Phabricator: my suggestion is to use https://github.com/braedon/prometheus-es-exporter, a tool that runs on the graylog hosts. The tool takes an elasticsearch query, performs the search and returns the result in prometheus format. Prometheus collects the metrics, after which we can use the metrics in Grafana dashboards.

Following a course regarding the basics of GDPR at the moment.

Analysing the queries from a binlog (c4):
mysqlbinlog mysql-bin.001818 | grep '::' > ~/analyse-queries-T6984.txt
(any query that has '::' in it usually includes the PHP caller as SQL comments)

Investigating load on dbbackup1.

Can't we generate the list dynamically? ie by running my 'script' periodically on a server?

Downtimed dbbackup[12] Current Load.

I am not sure if load alerts are useful for these servers. These servers are not production facing; as long as the replication processes are running and the replication lag is not greater than X seconds, I don't mind what the load average is.

Varnish was stopped at 16:36:45, gdnsd depooled cp12 at 16:37:02 on ns1 and at 16:37:04 on ns2. That's an excellent MTTR. I am not able to find out why the service ran out of memory, but since this quite a rare event, I'm tempted to skip the investigation process.

For tracking purposes: https://github.com/wikimedia/mediawiki/blob/6978577273048ca9a32baaa285d4270d6e860e6a/includes/user/UserNameUtils.php#L181 returns false for 'SHEIKH', but https://github.com/wikimedia/mediawiki/blob/6978577273048ca9a32baaa285d4270d6e860e6a/includes/user/UserNameUtils.php#L183 returns true, so I presume SHEIKH ended up being in the reserved names list.

c2 is done and replicates fine. c3 has not been cloned yet. c4 had its replica process stopped for too long and thus the binary position it's on doesn't exist on the master anymore: a reclone is needed.

In T5044#136907, @Paladox wrote:

We switched off syslog-ng logging on the cloud servers. Not sure if we want to switch it back on @John @Southparkfan ?

Yes, let's see if we can receive proxmox logs without further tweaking.

What's the status on meetings in the MediaWiki team?

(no time currently)

In T6765#134610, @John wrote:

If restarting the service is required to pick up changes, databases.json and *wiki.json can't be cached

Do you know what the bottleneck is? Is it MediaWiki -> syslog-ng or syslog-ng -> graylog?

Wikimedia has been contacted. Waiting on more information from the researcher.

• The API endpoint is open
• Our Grafana version is on a patched version
• Only impact on A, not C/I, Grafana is not critical
  • However, taking into account that Grafana resides on a system where critical systems (Icinga) are hosted..