Pixel

From Wikitech

Pixel is a tool that the Wikimedia Foundation uses to detect UI regressions. It is currently hosted on https://pixel.wmcloud.org/

Maintainers

The code is maintained by the Wikimedia Foundation Web Team (see the current setup). Long term the plan is for the QTE Team to maintain this (see upcoming setup).

Current setup

The current production version runs on pixel.reading-web-staging.eqiad1.wikimedia.cloud and can be accessed through https://pixel.wmcloud.org/.

How it works

The machine is hosted on Cloud VPS. You can ssh into it using 

ssh pixel.reading-web-staging.eqiad1.wikimedia.cloud

High level overview

  • The git repository is checked out inside /srv/pixel
  • Node.js is provided by the nvm package manager which is installed for the sudo user.
  • A cron job runs a bash script at /srv/pixel.sh. This executes the Pixel runAll command which creates the index.html and report pages. The priority is defined based on the hour of the day. Priority 1 jobs run hourly. Priority 2 jobs run twice a day, Priority 3 jobs run once a day.

Disk space

The current VPS has limited disk space. When disk space issues occur you can use Ncdu to identify large files. Usually the cause is too many folders in the /var/www/html/archive folder or large log files. You should be able to safely delete any of these files without repercussions. Note, generally for the /archive folder it's preferable to retain at least the last 2 deploy branches and ideally the last 4 (1 month) to be able to refer back to uncaught regressions.

Code

~/.bashrc

Used by cronjob to get correct Node.js version.

export NONINTERACTIVE=true
export MW_SERVER=https://en.wikipedia.org
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

/srv/pixel/

The code is a clone of https://github.com/wikimedia/pixel.

/srv/pixel.sh

The script generates the reports and index page.

#!/bin/bash
source ~/.bashrc  && 
cd /srv/pixel &&  git pull origin main &&
nvm use

hr=$( date +%I )
hr2=$( date +%H )

if [ "$hr2" = "00" ]; then
   node pixel.js runAll --priority "3" --directory /var/www/html
elif [ "$hr" = "06" ]; then
   node pixel.js runAll --priority "2"  --directory /var/www/html
else
   node pixel.js runAll --priority "1" --directory /var/www/html
fi

/srv/clean.sh

Manages disk space and writes reports to archive folder since we continue to run out of disk space on this VM.

#!/bin/bash
source ~/.bashrc
  
node /srv/archive.js
systemctl restart docker
/srv/pixel/pixel.js clean

#find "/var/www/html/archive/origin/wmf/*" -type d -ctime +30

su crontab -l

PATH=/usr/bin/:/root/.nvm/versions/node/v18.17.0/bin/:$PATH

27 * * * * BASH_ENV=/etc/profile /srv/pixel.sh >> /srv/pixel.log 2>&1

0  8 * * * /srv/clean.sh

/srv/archive.js

Populates the archive folder.

Code: https://gist.github.com/jdlrobson/ccb9ec74eaa59cd05f5ff000cb084999

/var/www/html/index.html

Generated by the pixel.js runAll command.

/var/www/html/reports

A symlink to the most recent reports (/srv/pixel/reports.)

/var/www/html/archive

The reports folder is copied here using /srv/archive.js

Upcoming setup

The new setup is hosted by the QTE Team. The server that runs pixel is production.pixel.eqiad1.wikimedia.cloud, hosted at Cloud VPS and you can access it through https://pixel-production.wmcloud.org. The server is setup using these instructions and runs a tagged version of pixel.

There is also a beta server beta.pixel.eqiad1.wikimedia.cloud that runs the same setup except that it runs the latest commit of pixel. You can access it through https://pixel-beta.wmcloud.org/.

Metrics from the server will be pushed to https://grafana.wikimedia.org/d/lC3anj1Iz/pixel? (at the moment we only collect disk information).

User

Pixel runs as the user pixel. You can change to the pixel user using sudo su - pixel. Pixel runs are triggered through the crontab for the pixel user. In the users home directory, the git repository is cloned and then a tag is checked out and the tests runs from that directory.

Update to a new pixel

You need to run a tagged version of pixel on the production server. To update the version, change to the pixel user and checkout the tag.

cd ~/pixel/
git pull origin
git checkout tags/vX.Y.X

Volumes

There are two volumes attached to the server:

  • /mnt/docker - Docker uses the volume for all Docker files.
  • /mnt/pixel-data - All result files from pixel.

Logs

Pixel logs to /var/log/pixel/pixel.log and the cleanup script logs to /var/log/pixel/pixel-clean.log

Workarounds

There are a couple of hacks in the current setup that we should fix:

  • When you run Pixels docker containers, the output (files) are owned by root and the pixel script (started by the user pixel) do not have write access to the files. To fix that for now, the pixel user runs the script as su.
  • The report directory (where pixel writes the reports) is a symlink to the volume that has been setup for pixel data. But the /report/ is committed to the pixel repo (and all files within the folder is ignored). One way to fix that can be to gitignore the full folder. Another way would be to make the output folder configurable in pixel.js, however at the moment there's mix between using the current start scripts folder (__dirname in NodeJS) and setup the report directory using docker-compose.
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Pixel&oldid=2172086"