Jobrunner

From Wikitech
Jump to: navigation, search

jobrunner is a service that continuously processes items off the MediaWiki job queue.

Overview

The job queue provides a means of deferring work that is too expensive to perform in the context of a web request. On our production environment, it does this by using Redis to enable shared access to a queue. Web MediaWiki instances enqueue operations for asynchronous execution, and a special class of app servers called job runners dequeue and execute them. The master process on the job runners is a service implemented in PHP called jobrunner.

The dispatcher configuration option specifies how a batch of jobs will be run. By default this uses the runJobs.php maintenance script. For Wikimedia specifically, the dispatcher is configured to instead make an HTTP request to an RPC endpoint on the localhost (docroot:/rpc/RunJobs.php). This allows it to optimally use HHVM (command-line invocation would have a higher startup time and no persistent compilation cache).

Configuration

When the jobrunner service starts, it reads configuration values from /etc/jobrunner/jobrunner.conf. This file is generated by Puppet from puppet:/modules/mediawiki/manifests/jobrunner.pp and puppet:/modules/mediawiki/templates/jobrunner/jobrunner.conf.erb. For an overview of configuration options, see the jobrunner.sample.json file.

Deployment

Jobrunner is deployed using Scap3 (as of T129148). The deployment directory on tin (the deployment host) is: /srv/deployment/jobrunner/jobrunner.

  1. Browse to the deployment directory and get the local repo in the state you want to deploy (e.g. git pull).
  2. Once ready, first run scap deploy-log in one terminal (or screen) to start watching the logs.
  3. In another terminal (or screen), run scap deploy -v "log message here" to start the deployment.
  4. Follow the instructions as deployment reaches each group of servers. Scap will automatically restart services on active jobrunner servers (e.g. those in the primary DC).

Restart

To manually restart the jobrunner and jobchron services without a deployment:

tin$ cd /srv/deployment/jobrunner/jobrunner
tin$ scap deploy -v --service-restart "restarting jobrunners in active dc"

This can be limited to specific hosts as follows:

tin$ cd /srv/deployment/jobrunner/jobrunner
tin$ scap deploy -v --service-restart --limit-hosts [hostname or range] "log message"

Debugging

You can push a null job to the jobqueue via eval.php:

> JobQueueGroup::singleton()->push( new NullJob( Title::newMainPage(), array() ) );

Logging and metrics

The jobrunner service logs to /var/log/mediawiki/jobrunner.log.

Metrics are reported to Graphite via statsd. See the jobrunner/ metric hierarchy on https://graphite.wikimedia.org/.

See also