Page MenuHomeMiraheze
Feed Advanced Search

Sat, May 15

Southparkfan added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

Scheduler has been changed on cloud5 and VMs. We will let this run for a week, after that we can do a week-by-week comparison (https://meta.miraheze.org/m/5r6 is an example).

Sat, May 15, 19:49 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance

Thu, May 13

Southparkfan added a comment to T6471: section.execute-GitInfo->getHeadCommitDate is very slow.

GitInfo::getHeadCommitDate tries to fetch the timestamp of the last commit for each extension. This can be very slow:

southparkfan@mw8:/srv/mediawiki/w/extensions/VisualEditor$ time git show -s --format=format:%ct HEAD
1620421442
Thu, May 13, 19:59 · MediaWiki (SRE), MediaWiki, Performance
Southparkfan added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

Paladox changed the scheduler on cloud5. Let's wait for a day to see the impact on I/O performance (regular operations).

Thu, May 13, 19:10 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance

Mon, May 10

Reception123 updated subscribers of T6471: section.execute-GitInfo->getHeadCommitDate is very slow.

@Southparkfan

Mon, May 10, 09:06 · MediaWiki (SRE), MediaWiki, Performance

Fri, May 7

MarkusRost added a comment to T6471: section.execute-GitInfo->getHeadCommitDate is very slow.
In T6471#144193, @Lev wrote:

I've investigated the problem: the reason for timeouts when using Wiki-Bot is because it asks for the whole list of extensions. Miraheze wikis respond to that specific request (https://meta.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=general|extensions with |extensions) too slowly, slower than most MediaWiki sites, and that is why the bot has problems. Asking simply for general is fine. If I had to guess, it's probably something to do with ManageWiki. (Leaving this here for people searching this in the future.)

Fri, May 7, 19:16 · MediaWiki (SRE), MediaWiki, Performance
RhinosF1 added a comment to T6471: section.execute-GitInfo->getHeadCommitDate is very slow.

Updated title/description to actually state the issue

Fri, May 7, 19:02 · MediaWiki (SRE), MediaWiki, Performance
RhinosF1 renamed T6471: section.execute-GitInfo->getHeadCommitDate is very slow from Wiki-Bot to section.execute-GitInfo->getHeadCommitDate is very slow.
Fri, May 7, 19:02 · MediaWiki (SRE), MediaWiki, Performance
RhinosF1 updated the title for P421 curl -H 'X-Miraheze-Debug: test3.miraheze.org' 'https://meta.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=general|extensions&forceprofile=1' | grep '1 - main()' -A 450 | less from curl -H 'X-Miraheze-Debug: test3.miraheze.org' 'https://meta.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=general|extensions&forceprofile=1' | grep '1 - main()' -A 450 | less > to curl -H 'X-Miraheze-Debug: test3.miraheze.org' 'https://meta.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=general|extensions&forceprofile=1' | grep '1 - main()' -A 450 | less.
Fri, May 7, 18:13 · MediaWiki (SRE), Performance

Wed, May 5

Paladox added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

Ah ignore the above, having looked around I don't see how we can do it then if you cannot use ionice within the script && also using it outside it too.

Wed, May 5, 12:41 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance

Tue, May 4

Paladox added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

Ohh should we run with —low

Tue, May 4, 22:18 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Paladox added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

I even ran it with ionice the command and the load on the disk was still high.

Tue, May 4, 22:13 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Southparkfan added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

On https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592149 it says ionice -c 3 /usr/share/mdadm/checkarray --cron --all

Tue, May 4, 16:50 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance

Mon, May 3

Paladox added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

The script runs with --idle which uses ionice, so it's already using it?

Mon, May 3, 18:19 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Paladox added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

On https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592149 it says ionice -c 3 /usr/share/mdadm/checkarray --cron --all

Mon, May 3, 18:11 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Paladox added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

I found this https://github.com/kopchik/scripts/blob/master/checkarray#L70 when searching up.

Mon, May 3, 17:34 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Southparkfan moved T7230: High I/O on cloud nodes affecting GlusterFS from Incoming to Short Term on the Infrastructure (SRE) board.
Mon, May 3, 17:28 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Southparkfan assigned T7230: High I/O on cloud nodes affecting GlusterFS to Paladox.
Mon, May 3, 17:28 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance

Sun, May 2

Southparkfan added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

@RhinosF1 The task is for the Infrastructure team now, but JohnLewis couldn't have known that in the first place.

Sun, May 2, 22:45 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Southparkfan added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

Scheduler for sda and sdb: [mq-deadline] none

Sun, May 2, 22:32 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
John added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.
In T7230#143535, @John wrote:

Load times for mw*, so this is MediaWiki infrastructure. Please tag tasks correctly on future.

It's as far as we know to now caused by gluster.

Sun, May 2, 22:29 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Southparkfan added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

Facts:

Sun, May 2, 22:24 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Southparkfan renamed T7230: High I/O on cloud nodes affecting GlusterFS from Load times high enough to cause depool to High I/O on cloud nodes affecting GlusterFS.
Sun, May 2, 21:58 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Southparkfan added projects to T7230: High I/O on cloud nodes affecting GlusterFS: Cloud Infrastructure, Infrastructure (SRE).
Sun, May 2, 21:57 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
Southparkfan lowered the priority of T7230: High I/O on cloud nodes affecting GlusterFS from Unbreak Now! to High.

@Paladox and I are investigating the possibility of the /etc/cron.d/mdadm check being the cause of high I/O.

Sun, May 2, 21:56 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
RhinosF1 added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.
In T7230#143535, @John wrote:

Load times for mw*, so this is MediaWiki infrastructure. Please tag tasks correctly on future.

Sun, May 2, 21:07 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
RhinosF1 added a comment to T7230: High I/O on cloud nodes affecting GlusterFS.

https://grafana.miraheze.org/d/xtkCtBkiz/prometheus-blackbox-exporter-test-ferran-tufan?viewPanel=138&orgId=1&from=now-2d&to=now

Sun, May 2, 21:06 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
John edited projects for T7230: High I/O on cloud nodes affecting GlusterFS, added: MediaWiki (SRE); removed Site Reliability Engineering.

Load times for mw*, so this is MediaWiki infrastructure. Please tag tasks correctly on future.

Sun, May 2, 21:05 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance
RhinosF1 added a project to T7230: High I/O on cloud nodes affecting GlusterFS: Performance.
Sun, May 2, 21:03 · Infrastructure (SRE), Cloud Infrastructure, MediaWiki (SRE), Performance

Mar 19 2021

Paladox added a comment to T6765: Cache frequently accessed files on MediaWiki servers.

Can't we generate the list dynamically? ie by running my 'script' periodically on a server?

Mar 19 2021, 15:29 · MediaWiki (SRE), Performance, MediaWiki

Mar 18 2021

RhinosF1 added a comment to T6765: Cache frequently accessed files on MediaWiki servers.

Can't we generate the list dynamically? ie by running my 'script' periodically on a server?

Mar 18 2021, 13:34 · MediaWiki (SRE), Performance, MediaWiki
John added a comment to T6765: Cache frequently accessed files on MediaWiki servers.

Can we note explicitly in the documentation that it can be done at any point then and the deployment doesn't need to wait until SRE are ready to merge a puppet PR?

Mar 18 2021, 13:29 · MediaWiki (SRE), Performance, MediaWiki
Southparkfan added a comment to T6765: Cache frequently accessed files on MediaWiki servers.

Can't we generate the list dynamically? ie by running my 'script' periodically on a server?

Mar 18 2021, 13:24 · MediaWiki (SRE), Performance, MediaWiki
RhinosF1 added a comment to T6765: Cache frequently accessed files on MediaWiki servers.

Can we note explicitly in the documentation that it can be done at any point then and the deployment doesn't need to wait until SRE are ready to merge a puppet PR?

Mar 18 2021, 13:20 · MediaWiki (SRE), Performance, MediaWiki
Paladox added a comment to T6765: Cache frequently accessed files on MediaWiki servers.

No, you can continue deploying extensions. As far as I’m aware this won’t stop you nor impede your work. You just open a pull and then someone who can merge and merge it. It’s not a requirement to get done first before deploying a extension.

Mar 18 2021, 12:47 · MediaWiki (SRE), Performance, MediaWiki
RhinosF1 added a comment to T6765: Cache frequently accessed files on MediaWiki servers.

Do we seriously want to stop MW engineers deploying new extensions. This is a huge change to @Universal_Omega's work

Mar 18 2021, 06:52 · MediaWiki (SRE), Performance, MediaWiki
Paladox closed T6765: Cache frequently accessed files on MediaWiki servers as Resolved.

I've deployed this tonight and also emailed tech@.

Mar 18 2021, 01:21 · MediaWiki (SRE), Performance, MediaWiki

Mar 11 2021

Reception123 moved T6765: Cache frequently accessed files on MediaWiki servers from Backlog to Long Term on the MediaWiki (SRE) board.
Mar 11 2021, 07:05 · MediaWiki (SRE), Performance, MediaWiki

Feb 25 2021

Southparkfan added a comment to T6765: Cache frequently accessed files on MediaWiki servers.
In T6765#134610, @John wrote:

If restarting the service is required to pick up changes, databases.json and *wiki.json can't be cached

Feb 25 2021, 17:21 · MediaWiki (SRE), Performance, MediaWiki
John added a project to T6740: Implement local nameserver cache daemons on servers: Performance.
Feb 25 2021, 16:33 · Performance, Infrastructure (SRE)

Feb 8 2021

John added a comment to T6765: Cache frequently accessed files on MediaWiki servers.

If restarting the service is required to pick up changes, databases.json and *wiki.json can't be cached

Feb 8 2021, 18:13 · MediaWiki (SRE), Performance, MediaWiki
Southparkfan added a comment to T6765: Cache frequently accessed files on MediaWiki servers.

@Paladox: see P381. Ideally, you turn the vmtouch (daemon) into a systemd service (a type of unit) that will run this script on every puppet run, after which you can restart the systemd service (to pick up new files).

Feb 8 2021, 13:21 · MediaWiki (SRE), Performance, MediaWiki

Jan 30 2021

Reception123 edited projects for T6765: Cache frequently accessed files on MediaWiki servers, added: MediaWiki (SRE); removed Site Reliability Engineering.
Jan 30 2021, 12:12 · MediaWiki (SRE), Performance, MediaWiki

Jan 23 2021

Southparkfan assigned T6765: Cache frequently accessed files on MediaWiki servers to Paladox.
Jan 23 2021, 23:39 · MediaWiki (SRE), Performance, MediaWiki
Southparkfan added a comment to T6765: Cache frequently accessed files on MediaWiki servers.
southparkfan@mw4:~$ sort fatrace.txt | uniq -c | sort -nr | head -50
  51899 /srv/mediawiki/w/cache/l10n/l10n_cache-en.cdb
   4187 /srv/mediawiki/w/cache/databases.json
   3517 /srv/mediawiki/w/cache/l10n/l10n_cache-zh-cn.cdb
   3391 /etc/hosts
   3248 /etc/ssl/certs/Sectigo.crt
   2578 /srv/mediawiki/w/cache/l10n/l10n_cache-it.cdb
   1411 /home/southparkfan/fatrace.txt
    869 /srv/mediawiki/w/cache/l10n/l10n_cache-zh-hant.cdb
    775 /usr/share/ca-certificates/mozilla/DigiCert_High_Assurance_EV_Root_CA.crt
    748 /srv/mediawiki/w/cache/deleted.json
    705 /srv/mediawiki/w/cache/l10n/l10n_cache-ja.cdb
    603 /usr/share/zoneinfo/UCT
    527 /srv/mediawiki/w/cache/l10n/l10n_cache-zh.cdb
    419 /srv/mediawiki/w/cache/l10n/l10n_cache-es.cdb
    380 /srv/mediawiki/w/cache/l10n/l10n_cache-nl.cdb
    303 /srv/mediawiki/w/extensions/Echo/extension.json
    298 /srv/mediawiki/w/cache/l10n/l10n_cache-zh-tw.cdb
    292 /srv/mediawiki/w/cache/l10n/l10n_cache-ko.cdb
    291 /srv/mediawiki/w/extensions/AbuseFilter/extension.json
    290 /srv/mediawiki/w/extensions/CentralNotice/extension.json
    273 /srv/mediawiki/w/skins/Vector/skin.json
    271 /srv/mediawiki/w/skins/MonoBook/skin.json
    269 /srv/mediawiki/w/extensions/CheckUser/extension.json
    266 /srv/mediawiki/w/extensions/CreateWiki/extension.json
    265 /srv/mediawiki/w/skins/Modern/skin.json
    261 /srv/mediawiki/w/skins/CologneBlue/skin.json
    258 /srv/mediawiki/w/extensions/ConfirmEdit/extension.json
    257 /srv/mediawiki/w/skins/Timeless/skin.json
    257 /srv/mediawiki/w/extensions/AntiSpoof/extension.json
    255 /srv/mediawiki/w/extensions/GlobalPreferences/extension.json
    254 /srv/mediawiki/w/extensions/WikiEditor/extension.json
    254 /srv/mediawiki/w/extensions/CentralAuth/extension.json
    253 /srv/mediawiki/w/extensions/MobileFrontend/extension.json
    253 /srv/mediawiki/w/extensions/DataDump/extension.json
    253 /srv/mediawiki/w/extensions/ConfirmEdit/ReCaptchaNoCaptcha/extension.json
    251 /srv/mediawiki/w/extensions/BetaFeatures/extension.json
    248 /srv/mediawiki/w/extensions/OAuth/extension.json
    246 /srv/mediawiki/w/extensions/VisualEditor/extension.json
    244 /srv/mediawiki/w/extensions/Scribunto/extension.json
    244 /srv/mediawiki/w/extensions/GlobalCssJs/extension.json
    243 /srv/mediawiki/w/extensions/Interwiki/extension.json
    242 /srv/mediawiki/w/extensions/GlobalBlocking/extension.json
    242 /srv/mediawiki/w/extensions/EventStreamConfig/extension.json
    241 /srv/mediawiki/w/extensions/IncidentReporting/extension.json
    241 /srv/mediawiki/w/extensions/EventLogging/extension.json
    238 /srv/mediawiki/w/extensions/LocalisationUpdate/extension.json
    238 /srv/mediawiki/w/extensions/CookieWarning/extension.json
    236 /srv/mediawiki/w/extensions/ManageWiki/extension.json
    235 /srv/mediawiki/w/extensions/OATHAuth/extension.json
    235 /srv/mediawiki/w/extensions/Nuke/extension.json
Jan 23 2021, 23:38 · MediaWiki (SRE), Performance, MediaWiki
Southparkfan triaged T6765: Cache frequently accessed files on MediaWiki servers as Normal priority.
Jan 23 2021, 23:15 · MediaWiki (SRE), Performance, MediaWiki

Nov 11 2020

SCVSlalom added a project to T6402: "Wikimedia\Rdbms\DBQueryError": Performance.
Nov 11 2020, 23:37 · Upstream, Database, Production Error, MediaWiki

Sep 17 2020

BlackWidowMovie0000 closed T6192: Timeless Content Bar as Resolved.
Sep 17 2020, 19:05 · Universal Omega, Extensions, Upstream
BlackWidowMovie0000 reopened T6192: Timeless Content Bar as "Open".
Sep 17 2020, 18:55 · Universal Omega, Extensions, Upstream

Jun 7 2020

OnKoydenKovuldum closed T5706: Spam as Resolved.

WikiLirik sözcüleri eleştirisel pozitif-negatif düşünce karşılaştırma
PyschoLyricWiki Attack Groups
Ben ötesi psikoloji

Jun 7 2020, 03:40 · Trash

Apr 9 2020

John created Performance.
Apr 9 2020, 00:23