Nova Resource:Videocuttool/Alert Runbook

Alert Runbook for VideoCutTool

The following document contains runbook for debugging the VM alerts triggered for VideoCutTool, and some other developer related purpose documentation.

Low Disk Space Left

You might see an alert similar to the one attached. Follow these steps to resolve it.

1. Tunnel to the VM instance

Identify the VM for which the alert is triggered. Currently we have 2 environments for VideoCutTool

VideoCutTool
Beta-VideoCutTool

From the alert description or title we can easily figure out the source of the alert. Tunnel into the VM instance using the following command

For VideoCutTool

ssh -J <username>@primary.bastion.wmflabs.org <username>@videocuttool-bookworm-new.videocuttool.eqiad1.wikimedia.cloud

For Beta-VideoCutTool

ssh -J <username>@primary.bastion.wmflabs.org <username>@beta-videocuttool-bookworm.videocuttool.eqiad1.wikimedia.cloud

2. Identify the directory

The first step for resolving low disk space error is to identify which directory (within the VM) has no space left.

Run the following command to check the current status

df -h

Now based on the file check for the resolution.

Docker local space (/var/lib/docker)

If the directory mounted on /var/lib/docker. It means that docker cache got increased.

Run the following command to delete local volumes, cache and images

docker system prune -af

To delete volumes specifially run

docker system prune --volumes -af

Now observe the current status via df -h command. The memory should gets free and the alert should get resolve now.

Force deletion docker directory

In case the above mentioned steps, doesn't clear the space, then we have to manually delete the docker directory's contents.

Go to /docker directory (within VM)

cd /var/lib/docker

Clear the contents inside the docker directory

rm -rf *

Restart the docker daemon. This automatically creates the sub directories (which got deleted due to the above command)

systemctl restart docker

VideoCutTool video data (/app/VideoCutTool/data)

If space is increased for /app/VideoCutTool/data directory, then it means, we have reached the limit for video downloading within the VM.
Go to the directory

 cd /app/VideoCutTool/data

Go to the video's directory

 cd /video-data

Delete the videos

 rm -rf *

We do plan to create scripts for the above resolution, so as to reduce human involvement with this, and minimize errors.

Note: If you get permission issue, please try with sudo su