Seems to be echo notifications constantly triggering. Queue is growing rapidly and out of control.
Description
Description
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Resolved | Unknown Object (User) | T9301 JobQueue spiralling out of control (Late May 2022) | ||
| Resolved | Unknown Object (User) | T9302 Have a way to view job queue data | ||
| Declined | Unknown Object (User) | T9300 Handle spikes of jobs better | ||
| Resolved | Unknown Object (User) | T9299 Create an open dblist |
Event Timeline
Comment Actions
This is definitely a high priority issue, but I feel that unless we can resolve something that is unbreak now within 24 hours or or less, we should use High. Lowering to High on that basis, but @Reception123 or another MWE can feel free to raise to UBN if preferred
Comment Actions
@Dmehus: No it's not UBN now but it keeps alerting and the job queue & exceptions alerts are supposed to show user impact and by definition when they alert be unbreak now.
Comment Actions
/srv/mediawiki/w/maintenance/runJobs.php --wiki=polandballwiki --type=EchoNotificationJob --maxtime=60 --memory-limit=192M --result=json PHP Warning: Invalid argument supplied for foreach() from /srv/mediawiki/w/extensions/CommentStreams/includes/CommentStreamsStore.php(715) #0 /srv/mediawiki/w/extensions/CommentStreams/includes/CommentStreamsStore.php(715): MWExceptionHandler::handleError(integer, string, string, integer, array) #1 /srv/mediawiki/w/extensions/CommentStreams/includes/EchoInterface.php(157): MediaWiki\Extension\CommentStreams\CommentStreamsStore->getWatchers(integer) #2 /srv/mediawiki/w/extensions/Echo/includes/controller/NotificationController.php(449): MediaWiki\Extension\CommentStreams\EchoInterface::locateUsersWatchingComment(EchoEvent) #3 /srv/mediawiki/w/extensions/Echo/includes/controller/NotificationController.php(466): EchoNotificationController::evaluateUserCallable(EchoEvent, string) #4 /srv/mediawiki/w/extensions/Echo/includes/controller/NotificationController.php(116): EchoNotificationController::getUsersToNotifyForEvent(EchoEvent) #5 /srv/mediawiki/w/extensions/Echo/includes/jobs/NotificationJob.php(13): EchoNotificationController::notify(EchoEvent, boolean) #6 /srv/mediawiki/w/includes/jobqueue/JobRunner.php(373): EchoNotificationJob->run() #7 /srv/mediawiki/w/includes/jobqueue/JobRunner.php(334): JobRunner->doExecuteJob(EchoNotificationJob) #8 /srv/mediawiki/w/includes/jobqueue/JobRunner.php(247): JobRunner->executeJob(EchoNotificationJob) #9 /srv/mediawiki/w/maintenance/runJobs.php(97): JobRunner->run(array) #10 /srv/mediawiki/w/maintenance/doMaintenance.php(108): RunJobs->execute() #11 /srv/mediawiki/w/maintenance/runJobs.php(134): require_once(string) #12 {main}
Potentially related.
Comment Actions
It eventually resolves it self. Keeping in mind that if a large wiki edits one template, it can generate thousands of refreshLinks (or similar) jobs, eventually resolving, but this isn't necessarily high priority with virtually no user-sided impact immediately noticeable.
Comment Actions
We are under 300 unclaimed jobs right now, so it does recover itself eventually. It's expectant to be high sometimes per what I said above.