Nova Resource:Video/Help

This project is used for re-encoding video from external sources into free formats such as webm and ogg.

Instances

Encoding instances

Encoding instances (encoding01, encoding02, etc) are of g4.cores16.ram16.disk20 flavour. Each instance have port 80 open to serve transcoded files with nginx.

Misc. instances

video-redis

A redis server for data (task scheduling, etc) exchange between encoding instances and a web frontend. The used databases are:

• 0: default database
• 1: video2commons celery result backend
• 2: video2commons celery broker
• 3: video2commons misc

Please document any database that will be continuously in use.

The redis service is password-protected for security. Please refer to /etc/redis/redis.conf for the password.

videodev

Lightweight development server. Serves as both an encoding instance and a redis server, with similar configurations. Due to the lightweight-ness, please use it for testing only.

Hosted services

video2commons

This is a brief summary of the /video2commons documentation page.

This tool transfers video and audio from external sites to Commons. Web frontend is on Nova resource:Tools, and backend is on Nova resource:Video. Code is hosted on GitHub.

Usage

Please use /srv and not your home directory unless in special cases. Encoding03 is preferred due to large disk apace available.

Downloading videos

wget

wget is used when you have a direct link to a video:

$ wget http://www.example.com/example.mp4

This will download example.mp4 from http://www.example.com/example.mp4

Refer to $ man wget for manual.

youtube-dl

youtube-dl is used in more difficult cases:

$ youtube-dl https://www.youtube.com/watch?v=abcdefg

This will download https://www.youtube.com/watch?v=abcdefg

Refer to https://github.com/rg3/youtube-dl/blob/master/README.md for manual.

To upgrade, use $ sudo pip install -U youtube-dl

Transcoding

Currently only ffmpeg and ffmpeg2theora is instinstalled for transcoding purposes. Commons Help:Converting_video has some instruction for how to do so. Codec default quality is usually quite low, and it's better to refer to FFmpeg Codecs Documentation for more advanced configurations.

Uploading

Direct upload is complicated as videos are usually large and require chunked uploads or server side uploads. Downloading to your computer then uploading manually may be easier but more time consuming, for files < 2GB). Larger files require server side uploads. Ways to download files include:

• sftp: simply sftp into the instance and get the file.
• http: every encoding instance has port 80 open, served by nginx. To serve your workspace:

Create a configuration file in /etc/nginx/sites-available/ with format (replace abcd with a better name):

# abcd.wmflabs.org configuration
#
server {
	listen 80;
	listen [::]:80;

	root /srv/path/to/your/workspace;

	server_name abcd.wmflabs.org;

	location / {
		try_files $uri $uri/ =404;
	}
}

Create a symlink in /etc/nginx/sites-enabled/ and reload nginx:

$ sudo ln -s /etc/nginx/sites-available/abcd /etc/nginx/sites-enabled/abcd
$ sudo service nginx reload

Ask one of the project admins to create a proxy for you, and your workspace will be available.

Setup

New instance

Encoding instances require lots of memory, disk space and CPU. Ask a project admin for them. Project admins are recommended check with Andrew before creating encoding instances; some smaller misc. instances may be spared.

Setup storage

Most of the storage on encoding instances are in an cinder volume. To setup storage, attach a Cinder volume and apply puppet role profile::labs::cindermount::srv and wait for puppet run (or sudo wmcs-prepare-cinder-volume ; sudo puppet agent -tv)

Install software

Basic software:

$ sudo apt install build-essential python3-dev
$ sudo puppet module install puppet-cron

For youtube-dl (we do not use python-pip package as it is too outdated):

$ sudo easy_install pip
$ sudo -H pip install youtube-dl

For transcoding softwares:

$ sudo apt-get install ffmpeg ffmpeg2theora

It is recommended to have all gstreamer plugins installed too, to help cover various formats.

$ sudo apt-get install gstreamer0.10-plugins-good gstreamer0.10-plugins-ugly gstreamer0.10-plugins-bad

And nginx:

$ sudo apt-get install nginx

Maintenance

Rebooting

Unless in emergency, please don't reboot instances, as other's transcoding jobs might be interrupted. Check Nagf for host load to be sure.

Before actually rebooting, make sure services are off:

• video2commons: /video2commons#Switching off

Do a $ top and make sure nothing is interrupted, then $ sudo reboot.

Upgrading ffmpeg

From an encoding instance:

source /mnt/nfs/labstore-secondary-project/gentoo-prefix/startprefix
emaint --auto sync
exit