Several Python runtimes are available for use inside Toolforge:
python3: Python 3.5.3 (3.7.3 on kubernetes)
python: Python 2.7.13 (2.7.9 on kubernetes; deprecated)
Deprecating Python 2
Many legacy tools and code examples use Python 2.x as a runtime. Use of Python 3.x is encouraged for new code as Python 2.7 will not be maintained past 2020. Toolforge will provide some amount of support for Python 2.x through 2022 because Debian will be supporting Python 2.7 in Debian 10 (buster). This support will only extend to critical security patches however.
If you don't want to create one tool for each Python task you need specific environment for, that's when virtual environment comes handy. The advantages of virtual environments don't end there. Apart from packages you can also maintain several separated git/svn repositories, several command line profiles, etc. You basically create this small virtual unit in your folder system, set up its environment to your needs and then reach its contents when you need it.
You can create your first virtual environment using:
$ python3 -mvenv my_venv
|The python3 command above must be run from the same execution environment that the venv will be used from. For Grid Engine jobs, run the command on a Toolforge bastion like login.toolforge.org. For a Kubernetes job, run the command from inside a |
This will install package manager, some basic tools, commands and prerequisites, everything into your new little unit. Once you created one, let's use it and play with it:
$ source my_venv/bin/activate (my_venv) $ pip install my_dream_package==7.0.3 ...
Once you are happy with it, you can always leave using:
(my_venv) $ deactivate
This way you can create as many separated Python environments as you wish.
You can reach it again from the inside the same way any time you want. This is handy when you want to update it for example. But for scheduled tasks, you would have to create a batch file with multiple commands to reach it, use it and leave it.
Use venv with scheduled tasks
Since it is saved in a folder in your Toolforge space, you can always use it from the outside just like any other folder. Well, you can not alter it this way, but for scheduled tasks you usually don't need to:
$ jsub -N my_task -once -quiet my_venv/bin/python3 my_script
Use your venv everywhere
You can also use your virtual environment everywhere by default. You can activate it using
$ echo "source my_venv/bin/activate" >> .profile