Pixel
Pixel is a tool that the Wikimedia Foundation uses to detect UI regressions. It is currently hosted on https://pixel.wmcloud.org/. For problems checkout Pixel/Runbook.
Maintainers
The code is maintained by the Wikimedia Foundation QTE Team. It was originally created by the Web Team.
Current setup
The current production version runs on production.pixel.eqiad1.wikimedia.cloud and can be accessed through https://pixel.wmcloud.org/.
How it works
The machine is hosted on Cloud VPS. You can ssh into it using
ssh production.pixel.eqiad1.wikimedia.cloud
High level overview
- The git repository is checked out inside /home/pixel/pixel
- A cron job runs a bash script at /home/pixel/pixel.sh. This executes the Pixel runAll command which creates the index.html and report pages. The priority is defined based on the hour of the day. Priority 1 jobs run hourly. Priority 2 jobs run twice a day, Priority 3 jobs run once a day.
Code
~/.bashrc
Used by cronjob to get correct Node.js version.
export NONINTERACTIVE=true
export MW_SERVER=https://en.wikipedia.org
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" # This loads nvm bash_completion
/home/pixel/pixel
The code is a clone of https://github.com/wikimedia/pixel.
/home/pixel/pixel.sh
The script generates the reports and index page.
#!/bin/bash
# At the moment the script run as root (as on the old pixel server). The reason
# is that the outcome of the docker containers are stored as root and then the
# pixel.js script cannot access those files.
#
# The script is called from the pixel users crontab.
whoami
export PATH=/usr/bin/:/home/pixel/.nvm/versions/node/v18.17.0/bin/:$PATH
source /home/pixel/.bashrc &&
cd /home/pixel/pixel || exit 1
export PIXEL_REPORT_DIRECTORY=/mnt/pixel-data/reports
export NONINTERACTIVE=true
export NVM_DIR="/home/pixel/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
nvm use || exit 1
priority=$1
if [ -z "$priority" ]; then
echo "Error: Priority argument is missing"
exit 1
fi
# Make sure we pick up the latest tag
current_tag=$(git describe --tags --exact-match 2>/dev/null || echo "")
git fetch --tags
latest_tag=$(git tag --sort=-creatordate | head -n 1)
if [ "$latest_tag" != "$current_tag" ]; then
echo "New tag: $latest_tag. Switch to that tag."
git checkout "tags/$latest_tag"
else
echo "Already on the latest tag: $current_tag"
fi
echo -e "\n\n\n"
echo -e "\e[32m===================================================\e[0m"
echo "Starting Pixel 'runAll --priority $priority'"
echo -e "\e[32m===================================================\e[0m"
echo "UTC datetime: $(date -u +"%Y-%m-%d %H:%M:%S %Z")"
echo "PDT datetime: $(TZ='America/Los_Angeles' date +"%Y-%m-%d %H:%M:%S %Z")"
echo -e "\e[34m---------------------------------------------------\e[0m"
echo -e "\n\n\n"
start_time=$SECONDS
node pixel.js runAll --priority "$priority" --directory "$PIXEL_REPORT_DIRECTORY"
exit_status=$?
end_time=$SECONDS
duration=$((end_time - start_time))
hours=$((duration / 3600))
minutes=$(((duration % 3600) / 60))
seconds=$((duration % 60))
echo "Exit code $exit_status"
echo -e "\e[34m---------------------------------------------------\e[0m"
printf "Finished Pixel 'runAll --priority %s'\nDuration (%s): %02d:%02d:%02d\n" "$priority" "$priority" $hours $minutes $seconds
echo -e "\e[32m===================================================\e[0m"
echo -e "\n\n\n"
timestamp=$(date +%s)
nc -w 5 wpt-graphite.wmftest.org 2003 <<< "pixel.jobprio.${priority}.failure ${exit_status} ${timestamp}"
if [ "$exit_status" -eq 0 ]; then
nc -w 5 performance-testing-graphite.wmftest.org 2003 <<< "pixel.jobprio.${priority}.duration ${duration} ${timestamp}"
fi
REPORT_DIR="/home/pixel/pixel/report/"
CURRENT_TIME=$(date +%s)
find "$REPORT_DIR" -type f -name "report.json" | while read -r FILE; do
ID=$(jq -r '.id' "$FILE")
PASS=$(jq '.tests | map(select(.status=="pass")) | length' "$FILE")
FAIL=$(jq '.tests | map(select(.status=="fail")) | length' "$FILE")
FILE_TIME=$(stat -c %Y "$FILE")
if [ "$ID" != "MediaWiki" ]; then
echo "Sending $ID pass[$PASS] fail[$FAIL] for time ${FILE_TIME}"
nc -w 2 performance-testing-graphite.wmftest.org 2003 <<< "pixel.job.${ID}.pass ${PASS} ${FILE_TIME}"
nc -w 2 performance-testing-graphite.wmftest.org 2003 <<< "pixel.job.${ID}.fail ${FAIL} ${FILE_TIME}"
fi
done
su crontab -l
pixel.sh is passed the priority:
PATH=/usr/bin/:/home/pixel/.nvm/versions/node/v20.11.1/bin/:$PATH
0 0,2,4,6,8,14,16,18,20,22 * * * sudo su -c "/home/pixel/pixel.sh 1 >> /var/log/pixel/pixel.log 2>&1"
0 10 * * * sudo su -c "/home/pixel/pixel.sh 2 >> /var/log/pixel/pixel.log 2>&1"
0 12 * * * sudo su -c "/home/pixel/pixel.sh 3 >> /var/log/pixel/pixel.log 2>&1"
45 4 * * * sudo su -c "cd /home/pixel/pixel && ./optimize-pngs.sh >> /var/log/pixel/pixel-optimize-png.log 2>&1"
10 5 * * * sudo su -c "cd /home/pixel/pixel && ./rebuild.sh >> /var/log/pixel/pixel-rebuild.log 2>&1"
35 5 * * * /home/pixel/archive.sh >> /var/log/pixel/pixel-archive.log 2>&1
45 23 * * * sudo su -c "find /mnt/pixel-data/reports/* -mtime +60 -exec rm {} \; >/dev/null 2>&1"
Script UTC PDT Minute Hour p1 00:00 AM 05:00 PM 0 0 p1 02:00 AM 07:00 PM 0 2 p1 04:00 AM 09:00 PM 0 4 optimize-pngs 04:45 AM 09:45 PM 45 4 rebuild 05:10 AM 10:10 PM 10 5 archive 05:35 AM 10:35 PM 35 5 p1 06:00 AM 11:00 PM 0 6 p1 08:00 AM 01:00 AM 0 8 p2 10:00 AM 03:00 AM 0 10 p3 12:00 PM 05:00 AM 0 12 p1 02:00 PM 07:00 AM 0 14 p1 04:00 PM 09:00 AM 0 16 p1 06:00 PM 11:00 AM 0 18 p1 08:00 PM 01:00 PM 0 20 p1 10:00 PM 03:00 PM 0 22 find 11:45 PM 04:45 PM 45 23
Dashboard
Metrics from the server is pushed to https://grafana.wikimedia.org/d/lC3anj1Iz/pixel? .
Setup
The server is setup using these instructions and runs a tagged version of pixel.
There is also a beta server beta.pixel.eqiad1.wikimedia.cloud that runs the same setup except that it runs the latest commit of pixel. You can access it through https://pixel-beta.wmcloud.org/.
User
Pixel runs as the user pixel. You can change to the pixel user using sudo su - pixel. Pixel runs are triggered through the crontab for the pixel user. In the users home directory, the git repository is cloned and then a tag is checked out and the tests runs from that directory. When you log in to the server you will get information about the setup.
Update to a new pixel
Documented in the Runbook.
Volumes
There are two volumes attached to the server:
- /mnt/docker - Docker uses the volume for all Docker files.
- /mnt/pixel-data - All result files from pixel.
Logs
Pixel logs to /var/log/pixel/pixel.log and the cleanup script logs to /var/log/pixel/pixel-clean.log
Workarounds
There are a couple of hacks in the current setup that we should fix:
- When you run Pixels docker containers, the output (files) are owned by root and the pixel script (started by the user pixel) do not have write access to the files. To fix that for now, the pixel user runs the script as su. That issue is tracked in T365419.
- The report directory (where pixel writes the reports) is a symlink to the volume that has been setup for pixel data. But the /report/ is committed to the pixel repo (and all files within the folder is ignored). One way to fix that can be to gitignore the full folder. Another way would be to make the output folder configurable in pixel.js, however at the moment there's mix between using the current start scripts folder (__dirname in NodeJS) and setup the report directory using docker-compose. This is tracked in T365422.