Page MenuHomeMiraheze

Jobs not running on wikis
Closed, ResolvedPublic

Description

A user reported some pages they added to categories didn't show up in the categories. I tried fixing it in all ways on their wiki but unsuccessfully. I tried testing this on publictestwiki.com and get the same problem (https://publictestwiki.com/wiki/Category:Testingissue https://publictestwiki.com/wiki/Testing_category_issue). ATT confirms that the same is happening to them.
Could this be a caching issue? I will investigate. Could it be the jobrunner (like yesterday)?

Event Timeline

Reception123 lowered the priority of this task from Unbreak Now! to High.Aug 9 2016, 04:53

Restarted the jobs (runJobs.php) fixed it, but the jobs should be running by themselves, will leave this task open, as we can't have someone manually restart every day.

Reception123 renamed this task from Adding pages to categories doesn't work to Jobs not running on wikis.Aug 9 2016, 04:54

@Reception123 that sounds like an issue with the job queue. The jobrunner may not be functioning correctly - try "sudo service jobrunner restart" (log it), and let me know if jobs run again.

As I said I already ran runJobs.php to fix the jobs (which worked). Will run sudo service jobrunner restart on mw1 now.

Jobs aren't working again and restarting the jobrunner didn't change anything. As the last time it worked, I ran runJobs.php which again fixed the issues (for now).

Reception123 raised the priority of this task from High to Unbreak Now!.Aug 9 2016, 10:51
Reception123 assigned this task to NDKilla.

Was a redis problem. Jobrunner is running last time I checked (2 minutes ago). If there are issues again feel free to open this task.

This is really, really irritating. It's probably related (but not completely caused by) the fact that Redis has OOM'd 5 times in the last 6 days.

Several times today I deleted redis keys (*:diff:* and *:resourceloader:*) to lower memory usage so that redis wouldn't OOM, but memory usage keeps creeping up. And ignoring that, the jobrunner (redisJobrunnerService) isn't handling jobs.. I'll have to look at redis and jobrunner logs, check the redis password in jobrunner (after it was rotated), and possibly move jobrunner to mw2.

NDKilla lowered the priority of this task from Unbreak Now! to High.Aug 11 2016, 02:22

Re-enabled 10 minute-ly job cron on mw1 (still marked as job runner but i killed the service and it doesn't auto-restart)

DO NOT RESTART JOBRUNNER, THE ICINGA ALERT IS INDEFINITELY AWK'D

Problem resulted in an cache "error" in ResourceLoader which should be fixed. Will leave the issue open until this is confirmed.

Jobrunner has been started again. If there are no issues in the following 24 hours, it means this was the issue and this task will be closed.

Still doesn't work. I think we should switch back to the cron job.

12:41:26 PM <+JohnLewis> previous all high priority issues were assigned to SPF, now they're assigned to PuppyKun xD
12:43:11 PM <+PuppyKun> rip
New messages
12:43:47 PM <+PuppyKun> cachet is still mine
12:43:53 PM <+PuppyKun> im not really sure i can do anything about jobs
12:43:58 PM — +PuppyKun casually unassigns self

@Southparkfan can you double check our config and stuff? John posted logs of service startup and it appears the jobrunner is actually 'working' (starting successfully and running) it just can't find any jobs..........

John claimed this task.

And resolved!

The RedisJobChron service wasn't running. This is the service which seems to manage the jobqueue for the jobrunner therefore no jobs were being queued for the jobrunner to ran (it's sort of a management service for this slave runner).

This has been started and puppetised and now the jobqueue is running on its own as expected.

I would like to request that this issue is being kept open and updated. I too am affected by this issue and would like at the very least to have a workaround for this, like for example a cronjob that runs a manual update every X muinutes.

As I said, I too am affected by this known issue (see also https://www.mediawiki.org/wiki/Manual:Errors_and_symptoms#Category_pages.2C_Special:Whatlinkshere_and_file_usage_aren.27t_being_updated ) and I'd really like for this to work, as my wiki's layout and navigation depends on the categories working as they should.

Next time please include the wiki in your issue report.

Jobrunner and jobchron (jobchron being the usual cause) are both running on mw1. showJobs.php indicates that your wiki barely has anymore jobs than allthetropes (the largest wiki).

I'll manually run jobs now but in the future please be sure you give it enough time.

I like to believe that 7+ hours would be enough time for the issue to resolve itself, because that's how long I have waited before reporting it here. (Waited over 5 after initially reporting it on IRC)

As for adding my wiki in the report, I will remember to do that next time, I didn't want to give anyone reason to say that I was spamming my own wiki. Something which has happened to me elsewhere in the past and wanted to avoid that.