Pixel/Setup server

Install a fresh pixel server

Here's an example on how the server at https://horizon.wikimedia.org/ is setup.

Install Docker

We need to have Docker installed to be able to run pixel.

sudo apt update
sudo apt install apt-transport-https ca-certificates curl gnupg -y
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker.gpg] https://download.docker.com/linux/debian bookworm stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install docker-ce containerd.io docker-compose-plugin -y

Create a user for all pixel related work

We want to avoid using root, it's better to have specific user that we can switch to.

sudo adduser pixel
sudo usermod -aG sudo pixel
sudo echo "pixel ALL=(ALL:ALL) NOPASSWD:ALL" | sudo tee "/etc/sudoers.d/pixel"

Make sure the user can run Docker without sudo.

sudo su - pixel
sudo usermod -aG docker $USER

Disable login for the pixel user. Open the configuration: sudo nano /etc/ssh/sshd_config and add:

// Add the following line at the end, it need to be a tab between DenyUusers and pixel
DenyUsers	pixel

Restart the SSH process so pixel is locked out: sudo systemctl restart sshd

Setup two volumes to separate the data

We use two volumes, one for pixel data and one for Docker. Create the volumes at https://horizon.wikimedia.org. I created 45 gb for data and 35 for docker and attached them to the server.

Login to your server and check using lsblk. When I do it I can see the disks like this:

lsblk
> sdb       8:16   0   45G  0 disk 
> sdc       8:32   0   35G  0 disk

Next step is to partion the disks

sudo fdisk /dev/sdb

You will get multiple questions. Start by choosing n, then p and then choose the default on the rest. As a last step choose w to write. Do the same thing for the other disk sudo fdisk /dev/sdc.

Run lsblk and you will see something like this:

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
|-sda14   8:14   0    3M  0 part 
`-sdb1    8:17   0   45G  0 part 
sdc       8:32   0   35G  0 disk 
`-sdc1    8:33   0   35G  0 part

Next step is to format the disks.

sudo mkfs.ext4 /dev/sdb1
sudo mkfs.ext4 /dev/sdc1

And then mount them. Create a mount point and mount them:

sudo mkdir /mnt/pixel-data
sudo mount /dev/sdb1 /mnt/pixel-data
sudo mkdir /mnt/docker
sudo mount /dev/sdc1 /mnt/docker

Now they are mounted. Next step is to make sure they are mounted after reboot. Here it is important to use the disks UUID so they will be mounted correct after a reboot. Get the UUDI for each disk:

sudo blkid /dev/sdc1
sudo blkid /dev/sdb1

Then add the volumes to fstab: sudo nano /etc/fstab And two new lines. Make sure to use your UUID and mount points:

UUID=ee39933b-5369-454e-aeee-c2cae69b3eec /mnt/pixel-data ext4 defaults,nofail,discard,noatime 0 2
UUID=df60bbbd-fada-45ae-8e1f-3c714c694e96 /mnt/docker ext4 defaults,nofail,discard,noatime 0 2

You can verify that everything in fstab is correct by running: sudo findmnt --verify --verbose

If it's correct, reload the fstab: sudo systemctl daemon-reload

Now the volumes should be there after a reboot. As a last step, make pixel the owners of the directories:

sudo chown pixel:pixel /mnt/pixel-data
sudo chown pixel:pixel /mnt/docker

And now as a last step, reboot and check if the mounts exists after the reboot: sudo reboot

Configure Docker to use the new volume

We want to make sure that Docker put its data on the new docker volume.

First stop the Docker daemon: sudo service docker stop

Then create a daemon.json file: sudo nano /etc/docker/daemon.json . And add the following:

{
  "data-root": "/mnt/docker"
}

Move the old data and restart docker:

sudo mv /var/lib/docker/ /mnt/docker
sudo service docker start

Install NodeJS

The current version of Pixel runs using NodeJS so you need to install it:

sudo su - pixel
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
nvm install 20

Install crontab

All jobs are executed from the crontab: sudo apt-get install cron -y

Setup apache2

Apache is used to serve the result: sudo apt-get install apache2 -y

Then configure it to serve the pixel reports: sudo nano /etc/apache2/sites-enabled/000-default.conf

At the top of the file, change the document root to the following:

DocumentRoot /mnt/pixel-data

And within the same virtual host block, add a redirect to the pixel index file:

<Directory /mnt/pixel-data>
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>
Redirect 302 /index.html /reports/index.html

Restart apache: sudo systemctl restart apache2

Setup up logs

Setup up log rotation. First create a new home for our logs:

sudo mkdir /var/log/pixel

sudo chown -R pixel:pixel /var/log/pixel

Then setup the rotation: sudo nano /etc/logrotate.d/pixel

And add the following:

/var/log/pixel/pixel.log {
    size 100M
    rotate 5
    compress
    missingok
    notifempty
    create 0644 pixel pixel
}

And then do the same for the cleanup log: sudo nano /etc/logrotate.d/pixel-clean And add:

/var/log/pixel/pixel-clean.log {
    size 100M
    rotate 5
    compress
    missingok
    notifempty
    create 0644 pixel pixel
}

Setup collectd

Send disk usage to Graphite (we should change this to Prometheus) using collectd.

Install: sudo apt-get install collectd collectd-utils

Disable plugins and then configure the df plugin. This is done a little differently depending on which disks and what collectd version you use.

Open the configuration file: nano /etc/collectd/collectd.conf

Remove all LoadPlugin and add:

LoadPlugin df
LoadPlugin write_graphite

<Plugin df>
   Device "/dev/sdc1"
   MountPoint "/mnt/pixel-data"
   ValuesPercentage true
 </Plugin>

<Plugin df>
   Device "/dev/sdb1"
   MountPoint "/mnt/docker"
   ValuesPercentage true
 </Plugin>

<Plugin df>
   Device "/dev/sda1"
   MountPoint "/"
   ValuesPercentage true
 </Plugin>

And follow the Graphite plugin setup in Performance/Synthetic testing/Bare metal#Install collectd

Then restart the service: sudo service collectd restart