Python is one of the most used programming languages on Toolforge.
Deprecating Python 2
Many legacy tools and code examples use Python 2.x as a runtime. Use of Python 3.x is encouraged for new code as Python 2.7 stopped being maintained in 2020. Toolforge will provide some amount of support for Python 2.x through 2022 because Debian will be supporting Python 2.7 in Debian 10 (buster). This support will only extend to critical security patches however.
A python virtual environment (venv) allows the maintainers of a tool to install Python packages using
pip without affecting any other tools running in Toolforge. A venv is similar to the use of composer to install PHP libraries or bundler to install ruby libraries that are local to directory rather than system wide.
A virtual environment must be created in the same execution environment that the venv will be used from. This means:
- if the tool uses a Kubernetes backend (recommended), the venv should be boostrapped inside a container.
- if the tool uses a Grid Engine backend, the venv should be bootstrapped directly on a Toolforge bastion server like
Kubernetes python webservices
Kubernetes python jobs
Follow these instructions if you are using the Toolforge Jobs framework.
You will need to run a one-time manual job to bootstrap a python venv from inside a job container. This is similar to the instructions about using
webservice shell when bootstrapping kubernetes webservices.
Create a script similar to this to create the venv:
#!/bin/bash # use bash strict mode set -euo pipefail # delete the venv, if it already exists rm -rf pyvenv # create the venv python3 -m venv pyvenv # activate it source pyvenv/bin/activate # upgrade pip inside the venv and add support for the wheel package format pip install -U pip wheel # Change the following section depending on what your tool needs! # install some concrete packages # pip install requests # pip install pyyaml # or, install all packages from src/requirements.txt # pip install -r src/requirements.txt
Run the bootstrapping script in the desired python container:
tools.mytool@tools-sgebastion-11:~$ ls bootstrap_venv.sh bootstrap_venv.sh tools.mytool@tools-sgebastion-11:~$ chmod ug+x bootstrap_venv.sh tools.mytool@tools-sgebastion-11:~$ toolforge-jobs run bootstrap-venv --command "cd $PWD && ./bootstrap_venv.sh" --image python3.11 --wait tools.mytool@tools-sgebastion-11:~$ ls pyvenv pyvenv
Once the venv had been created, you can run your python job as many times as needed as a manual, continuous, or periodic job:
tools.mytool@tools-sgebastion-11:~$ cat src/mytool.py import requests r = requests.get('https://www.wikidata.org/wiki/Q1') print(r.status_code) tools.mytool@tools-sgebastion-11:~$ toolforge-jobs run mytool --command "pyvenv/bin/python src/mytool.py" --image python3.11 tools.mytool@tools-sgebastion-11:~$ cat mytool.out 200
Grid Engine backend (deprecated)
|Expand to view venv instructions for the deprecated Grid Engine backend.|
|The following content has been placed in a collapsed box for improved usability.|
This is retained as a reference for existing tools that use the Toolforge Grid Engine backend. Do not create new tools using these instructions.
Grid Engine python webservice
Grid Engine python jobs
WMCS is deprecating grid engine. Please migrate your python jobs to Toolforge Jobs framework.
Follow these instructions if you are using the Grid Engine backend to run your jobs. Do not create new jobs using these instructions.
You can create your first virtual environment using:
$ python3 -mvenv my_venv
This will install package manager, some basic tools, commands and prerequisites, everything into your new little unit. Once you created one, let's use it and play with it:
$ source my_venv/bin/activate (my_venv) $ pip install my_dream_package==7.0.3 ...
Once you are happy with it, you can always leave using:
(my_venv) $ deactivate
This way you can create as many separated Python environments as you wish.
You can reach it again from the inside the same way any time you want. This is handy when you want to update it for example. But for scheduled tasks, you would have to create a batch file with multiple commands to reach it, use it and leave it.
Use venv with scheduled tasks
Since it is saved in a folder in your Toolforge space, you can always use it from the outside just like any other folder. Well, you can not alter it this way, but for scheduled tasks you usually don't need to:
$ jsub -N my_task -once -quiet my_venv/bin/python3 my_script
Use your venv everywhere
You can also use your virtual environment everywhere by default. You can activate it using
$ echo "source my_venv/bin/activate" >> .profile
|The above content has been placed in a collapsed box for improved usability.|
- Help:Toolforge/Running Pywikibot scripts
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation: