Jump to content

Nova Resource:Video/Help/video2commons

From Wikitech
Toolforge tools
Website toolforge:video2commons
Description Transfers video and audio from external sites to Commons
Keywords video, transfer
Author(s) Zhuyifei1999talk
Maintainer(s) Zhuyifei1999 (View all)
Source code GitHub
License GNU General Public License 3.0 or later
Issues GitHub

This tool transfers video and audio from external sites to Commons. Web frontend is on Nova resource:Tools, and backend is on Nova resource:Video. Code is hosted on GitHub.

Web frontend

Webfrontend is on Toolforge under tool video2commons.


Ensure you're in your tool account; if not, $ become video2commons

Clone repo

Somehow, clone video2commons to the tool's home dir. Some methods can do this:

  • Delete everything and clone it (be careful): $ rm -rf * && git clone https://github.com/toolforge/video2commons.git .
  • Or, clone to a tmp dir and copy git configuration back: $ git clone https://github.com/toolforge/video2commons.git tmp && mv tmp/.git . && rm -rf tmp && git reset --hard

Setup environment

Inside a shell of the tool's k8s container (launch with $ webservice --backend=kubernetes python2 shell): Setup virtualenv:

$ virtualenv ~/www/python/venv
$ echo source ~/www/python/venv/bin/activate >> .bash_profile
$ source ~/www/python/venv/bin/activate
$ pip install -U pip

Install necessary components:
* <syntaxhighlight lang="bash" inline>$ pip install -r requirements.txt
  • Anything else?

Create a config.json similar to config.json.example, and fill the necessary configs. Make sure the file is chmodded 600 before writing it!

Start the webservice

Start the frontend (outside of the container):

$ webservice --backend=kubernetes python2 start


Run $ git pull. No local commits should exist.

  • If any python code is changed, uwsgi must be restarted: $ webservice --backend=kubernetes python2 restart (and a downtime under a minute might be caused)
  • Otherwise, nothing must be done.


Backends are celery workers on video cluster encoding instances.


Ensure Nova Resource:Video/Help#Setup is done.

Use puppet to configure the rest:

  1. Get the puppet configuration: $ wget https://github.com/toolforge/video2commons/raw/master/puppet/backend.pp
  2. Ensure the first few lines of the file is correct. And, just in case, make sure nothing in the code is evil.
  3. Apply them: $ sudo puppet apply backend.pp --debug

Switching off

Run once (and only once. DO NOT run a second time unless something is already terribly broken!):

sudo celery -A video2commons.backend.worker --workdir=/srv/v2c multi stop --pidfile="/var/run/v2ccelery/%N.pid" 2

This will put celery to gracefully shutdown (i.e. it will stop handling new jobs, stop all idle workers, and wait for each worker to finish its job). Running a second time WILL KILL all the workers.