Help:Toolforge/Python

From Wikitech
Jump to navigation Jump to search

Python is one of the most used programming languages on Toolforge.

For guidance on writing tools in Python, see e. g. Help:Toolforge/My first Flask OAuth tool and Help:Toolforge/My first Django OAuth tool.

Deprecating Python 2

Many legacy tools and code examples use Python 2.x as a runtime. Use of Python 3.x is encouraged for new code as Python 2.7 stopped being maintained in 2020. Toolforge will provide some amount of support for Python 2.x through 2022 because Debian will be supporting Python 2.7 in Debian 10 (buster). This support will only extend to critical security patches however.

Virtual environments

If you don't want to create one tool for each Python task you need specific environment for, that's when virtual environment comes handy. The advantages of virtual environments don't end there. Apart from packages you can also maintain several separated git/svn repositories, several command line profiles, etc. You basically create this small virtual unit in your folder system, set up its environment to your needs and then reach its contents when you need it.

The virtual environment must be created in the same execution environment that the venv will be used from. This means:

  • if the tool uses a Kubernetes backend (recommended), the venv should be boostrapped inside a container.
  • if the tool uses a Grid Engine backend, the venv should be bootstrapped directly on a Toolforge bastion server like dev.toolforge.org or login.toolforge.org.

Read below for more concrete information.

For Kubernetes backend

This is for tools that use the Toolforge Kubernetes backend (recommended).

Kubernetes python webservices

See Toolforge webservices with Python.

Kubernetes python jobs

Follow these instructions if you are using the Toolforge Jobs framework.

You need to bootstrap your python venv from inside a job itself (similar to what happens in kubernetes webservices).

Create a script similar to this:

bootstrap_venv.sh
#!/bin/bash

# use bash strict mode
set -euo pipefail

# delete the venv, if it already exists
rm -rf pyvenv

# create the venv
python3 -m venv pyvenv

# activate it
source pyvenv/bin/activate

# upgrade pip inside the venv and add support for the wheel package format
pip install -U pip wheel

# Change the following section depending on what your tool needs!

# install some concrete packages
# pip install requests
# pip install pyyaml

# or, install all packages from src/requirements.txt
# pip install -r src/requirements.txt

Then run it in the desired python container, selecting the python version you prefer, example:

tools.mytool@tools-sgebastion-11:~$ ls bootstrap_venv.sh
bootstrap_venv.sh
tools.mytool@tools-sgebastion-11:~$ chmod ug+x bootstrap_venv.sh
tools.mytool@tools-sgebastion-11:~$ toolforge-jobs run bootstrap-venv --command "cd $PWD && ./bootstrap_venv.sh" --image python3.9 --wait
tools.mytool@tools-sgebastion-11:~$ ls pyvenv
pyvenv

Now you can run your python tool using this venv, example:

tools.mytool@tools-sgebastion-11:~$ cat src/mytool.py
import requests
r = requests.get('https://www.wikidata.org/wiki/Q1')
print(r.status_code)
tools.mytool@tools-sgebastion-11:~$ toolforge-jobs run mytool --command "pyvenv/bin/python src/mytool.py" --image python3.9
tools.mytool@tools-sgebastion-11:~$ cat mytool.out
200

For Grid Engine backend

See also

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia Movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Receive mail announcements about critical changes
Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
Track work tasks and report bugs
Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself
Learn about major near-term plans
Read the News wiki page
Read news and stories about Wikimedia Cloud Services
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)