Process Control

Process-control is the name of the service used to wrap our cron jobs. With process control we can do the following:

Schedule jobs using cron syntax, while isolating users from cron write access.
Configuration is done entirely through YAML files, for auditability and declarativeness.
Enforce running jobs as a service user.
Prevents overlapping runners by saving a lock file per job.
Timeout
Captures stdout and stderr, and log to a file per run.

The source code of process-control live in Gerrit but is also mirrored in GitHub:

Cron jobs

Each cron job is fully configured in a YAML file. This job.example.yaml file shows the supported configuration syntax. You can also see the already configured jobs in frpm1002 under the following path:

frpm1002:~/localsettings/process-control/civi1002

Development

The best way to create a new job is to find an existing job that runs a similar command and copy its YAML file. Update this file with your task-specific requirements.

If the job runs a CiviCRM command, you can build it using the API Explorer v4 tool via CiviCRM Support > Developer on localhost. Chaining API commands will allow you to handle some fairly complex queries, but you might need to open a second API Explorer to develop the chain call and then paste it in. For more complex cases, you can also define a CiviCRM API Action that includes all your logic which you then call in your job.

Whenever possible, test the command line locally.

Once you are confident that the command works and the YAML file is correct, obtain a virtual "+2" approval from a teammate. You can request this in the #fredgymcfredgeface IRC channel or via a Phabricator task before proceeding with the deployment.

Useful Tips:

Piped Commands: If your YAML configuration requires a piped command, wrap the entire string using sh -c.
```
command: sh -c "your | piped | command"
```
Line Length: Try to keep each line of code under 80 characters to comply with standard linting rules.

Deployment

Requirements
SSH Access: Ensure you have SSH access to frpm1002.

Approval: Your YAML file should have a virtual "+2" approval before deployment

To deploy you have to SSH into frpm1002 and follow the standard instructions outlined on the deployments page:Fundraising/Cluster/Deployments#Process-Control Jobs. The same workflow applies when introducing a new YAML file.

Deployment notes:

Linting Check: During deployment you may encounter YAML lint errors. While you should fix them if possible, they are not blockers. Many existing jobs naturally trigger line-length lint violations due to long commands.
Commit Log: Remember to provide a descriptive Git commit message and include the relevant Phabricator task ID (if applicable).
Log to SAL (Optional): Once the deployment is complete, navigate to the #wikimedia-fundraising IRC channel and type: !log process-control config revision changed This will log your change in Fundraising/SAL.

Post-Deployment Execution

1. Automated Scheduling

Once deployed to the process-control services, jobs run automatically according to their defined schedule. They will also automatically appear in Grafana.

2. Manual Execution

If you need to trigger a job manually after deployment, SSH into the appropriate process-control server (e.g., civi1002) and navigate to the directory:

cd /srv/process-control/civi1002/

Execute the job using the run-job utility. You can view full usage details by running run-job -h. An example command:

run-job --job name_of_the_yaml_file_job

Monitoring

Grafana - Process Control Jobs Dashboard: Use the dedicated Grafana dashboard to monitor general process control job health and scheduling. This dashboard verifies that your job is configured and executing correctly.
SearchKit: Use CiviCRM SearchKit to validate the data mutations caused by the job. This is highly recommended for any job that modifies existing database records.
Execution Logs: Every cron job run generates an independent output log, regardless of its execution method.
- If you trigger a job from civi1002:~/srv/process-control/civi1002/job_name.yaml the corresponding execution log will be generated under: /var/log/process-control/job_name/job_name-<YYYYMMDD>-<HHMMSS>.log (e.g., /var/log/process-control/job_name/job_name-20260716-104342.log)