Pixel/Runbook

This is the runbook for deploying and debugging Pixel problems.

Meta

• Issue tracker (Phabricator): web_team_visual_regression_framework
• Documentation: Pixel

Update to new version of Pixel

Pixel is auto updating on the production server to the latest tag. If you make a new release, make sure to add a tag and push it to git.

// TODO Monte please share the correct way to update the changelog!

git tag -a vX.Y.Z -m "Release version X.Y.Z"
git push origin vX.Y.Z

Revert to a old version of Pixel

If the current tag version of Pixel is broken and you cannot easily fix it, you need to revert to the latest working version.

1. Disable the auto update on the server.
2. Login to the server ssh production.pixel.eqiad1.wikimedia.cloud
3. Switch to the pixel user sudo su - pixel
4. Disable the auto update nano /home/pixel/pixel.sh and comment out the line git checkout "tags/$latest_tag"
5. Go to the repository: cd /home/pixel/pixel
6. Checkout the last working tag: git checkout "tags/vX.Y.Z"

When there's a fix, make sure to enable the auto update again.

Prio 1, 2 or 3 job is failing

You receive an alert email from Grafana with the subject "Pixel: Prio X job are failing qte". This means that the Pixel job failed running on https://pixel.wmcloud.org/.

1. Checkout the failure reported in Grafana. Go to https://grafana.wikimedia.org/d/lC3anj1Iz/pixel and check which jobs are failing.
2. Login to the server: ssh production.pixel.eqiad1.wikimedia.cloud