In T6922#136520, @John wrote:

Another issue caused by deployers merging without reviewing the changes being deployed. I've ask @Paladox to run the SQL.

Someone needs to run sql/patch-approval_date.sql

Looking

I'm still seeing lag. Just had a 6 hour delay.

The broken update was rolled back.

Cargo was updated with https://github.com/wikimedia/mediawiki-extensions-Cargo/compare/8fc32040cdef3012f6c0666a85d45af4b6afe2ad...34bc7aed58def2a481918a8a6ea60497926070fe included on Monday.

Go away Herald

Tagging Site Reliability Engineering because CSP needed

Thanks for letting me know @Dmehus

@Reception123

Not sure why Parsoid is failing and haven't checked that but I don't see it actually being set in ManageWiki

Please clear your cookies. Session tokens don't persist very well over a wiki rename.

We'll run that for you soon then.

If by indexed, you mean by search engines, that is updated often but they're not always the quickest at picking it up.

You can either purge the page or we can run a script

In T6898#136060, @Zppix wrote:

In T6898#136055, @RhinosF1 wrote:

Was these the ones that were free? What provider was it?

Yes, and Freenom

By best to suggestion is to change the account email to sre@ and then we can reset the password and switch to our details.

Was these the ones that were free? What provider was it?

Will do by end of week

Please try resetting your cookies but what wiki?

Thanks!

You should be able to hide it from the sidebar but you can't block access to them completely.

Do you mean all Special pages or specific ones?

Already available from Special:ManageWiki/settings

I had a quick look and unless I'm blind it's not quite as simple as I thought but I'm looking into it.

I can look into filing a change of url notification to google but if I recall correctly I might have to ask @Southparkfan for help with that.

Thanks!

Per @Paladox and https://phabricator.wikimedia.org/T274736

Will make public once WMF confirm they have no objections too as they've been involved in a lot.

And updated with Grafana v7.4.2 (29e75ad97b)

WMF have still kept the stop gap and it's an extra layer so worth keeping as well still.

Proper fix is available with https://github.com/grafana/grafana/pull/31263

There's also the kafka JobQueue task on another get rid of redis mission

Only IR left

Sysadmins: Use https://graylog.miraheze.org/search?q=NOT+mediawiki_exception_file%3A%5C%2Fsrv%5C%2Fmediawiki%5C%2Fw%5C%2Fextensions%5C%2FCargo%5C%2F%2A+AND+mediawiki_exception_class%3AJobQueueError&rangetype=relative&streams=5f8c6fd446640840f104b0ba&relative=7200 and https://graylog.miraheze.org/search?q=mediawiki_host%3Ajobrunner%2A&rangetype=relative&streams=5f8c6fd446640840f104b0ba&relative=7200 to see errors.

Much lower rate but still some. We'll continue working on it. CC @Paladox to see if they know how to get redis working again.

In T6859#135237, @Emad4ald wrote:

It has been resolved
Thanks Team

Looks to be out of disk space

jbr3&4:

ERR Error running script (call to f_3b56824083b7c4c3509163c7e76e00904014d92e): @user_script:4: @user_script: 4: -MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.

There's a huge rise in redis errors starting 07:12

I'll look now

Per our discussion, Zppix's access will remain revoked and in this case any new access request needs approval anyway of Miraheze SRE.

Thanks. Tagging @MacFan4000 and @Voidwaker to respond to me on IRC so we can discuss long term action as soon as possible.

The situation and Zppix's future

The relay is restored. Further discussions will occur today

I have revoked Zppix's access per https://phab.MirahezeBots.org/T193 and will restore the relay

For some reason the config flag was never set. I've added it.

Sent email back to researcher on SPF's request

In T6840#134950, @R4356th wrote:

In T6840#134949, @RhinosF1 wrote:

It's CentralNotice but I'd assume no because that would break translations.

I think we could add a new setting to ManageWiki that would allow wiki bureaucrats to do what we currently do for Meta, add something like <meta name="description" content="Description defined through ManageWiki" />. This should fix this particular issue and reduce the anxiety of bureaucrats who care much about SEO. But I am not an expert on SEO and could be wrong.

It should be fairly easy to make configurable.

It's CentralNotice but I'd assume no because that would break translations.

In T6837#134870, @Reception123 wrote:

Oh, I wasn't aware there was a task.

In T6837#134868, @RhinosF1 wrote:

In T6837#134867, @Reception123 wrote:

dreamsit.com.br is pointing to cp3 instead of mw-lb, will notify.

See https://phabricator.miraheze.org/T6834

Looks like that to me now.

In T6837#134867, @Reception123 wrote:

dreamsit.com.br is pointing to cp3 instead of mw-lb, will notify.

There's a bunch of puppet failures you might wanna check

That should be Unbreak then

I think you just copy the pages it mentions to your wiki

I'm not sure this is actually an extension

I see https://github.com/miraheze/mw-config/pull/3512/files and https://phabricator.miraheze.org/T6528 but imagine this is DPL v DPL3?

In T6838#134798, @R4356th wrote:

In T6838#134794, @RhinosF1 wrote:

@Universal_Omega: Is this safe? I think you've raised performance concerns before with it.

I know there's a few different versions of DPL but can we look into whether anything in https://phabricator.wikimedia.org/T262391 is of relevance to us.

I do not believe WMF uses DPL3 or the version that @Universal_Omega maintains. But I did notice that the setting is set to false by default which is why I asked for the wiki URL instead of adding it to ManageWiki.

@Universal_Omega: Is this safe? I think you've raised performance concerns before with it.

In T6837#134789, @Reception123 wrote:

@RhinosF1 I'm not sure which 8 because I see no alerts regarding rDNS on Icinga

See above links, they all shows as warning or critical for me.

https://icinga.miraheze.org/monitoring/service/show?host=sslhost&service=data.maantietaja.org%20-%20reverse%20DNS

No longer pointing

https://icinga.miraheze.org/monitoring/service/show?host=sslhost&service=pol.wiki%20-%20LetsEncrypt

Times out

https://icinga.miraheze.org/monitoring/service/show?host=sslhost&service=wikiverte.pl%20-%20LetsEncrypt

Times out

https://icinga.miraheze.org/monitoring/service/show?host=sslhost&service=dreamsit.com.br%20-%20reverse%20DNS

Broken DNS config

https://icinga.miraheze.org/monitoring/service/show?host=sslhost&service=evenair.tk%20-%20reverse%20DNS

No longer pointing

https://icinga.miraheze.org/monitoring/service/show?host=sslhost&service=museummiddelland.nl%20-%20reverse%20DNS

No longer pointing

https://icinga.miraheze.org/monitoring/service/show?host=sslhost&service=wiki.finnsoftware.net%20-%20reverse%20DNS

No longer pointing

https://icinga.miraheze.org/monitoring/service/show?host=sslhost&service=wiki.insideearth.info%20-%20reverse%20DNS

Proxying us through CloudFlare

https://icinga.miraheze.org/monitoring/service/show?host=sslhost&service=wiki.yapsavun.com%20-%20reverse%20DNS

Proxying us through CloudFlare

https://icinga.miraheze.org/monitoring/service/show?host=sslhost&service=wiki.yumeka.team%20-%20reverse%20DNS

Registration Lapsed or held

https://icinga.miraheze.org/monitoring/service/show?host=sslhost&service=wikipuk.cl%20-%20reverse%20DNS

Not resolving

In T6837#134782, @Reception123 wrote:

@RhinosF1 The link works for me, and I see no problems except a test3 one on icinga

Which wiki, there's 8. An rDNS alert doesn't mean wiki down as well, it can mean anything from bad config to they're proxying our traffic through something first.

I believe this is deliberate and set somewhere in our config, although if I recall correct that might have been just meta, but yes I agree it needs phasing out and replacing with a proper description for everyone.

Looking at your records, you seem to have your nameserver as mw-lb.miraheze.org and 2 others.

In T6834#134692, @Pfyh wrote:

@RhinosF1 much obliged!

In T6834#134658, @Pfyh wrote:

I didn't get it, how do I do it? Which ip do I use?

Site is pointing at cp6 which has been decommissioned. Please use a CNAME to mw-lb.miraheze.org or convert to our DNS.

In T6834#134652, @Pfyh wrote:

THE WEBSITE IS POINTING TO 177.220.174.57

Sorry didn't even see the policy

In T6826#134469, @RhinosF1 wrote:

There's also https://meta.miraheze.org/wiki/Stewards%27_noticeboard#Nonciclopedia_issues

Still can reproduce from https://nonciclopedia.org/wiki/Speciale:TutteLePagine, multiple untranslated namespaces.