Python/Development Tools

From Wikitech

Introduction

Overview

In this guide, we will learn how to use pyenv, tox, poetry, and direnv. These tools help create isolated environments for different Python versions, manage dependencies, automate testing, and simplify environment configuration.

There are several other similar tools in the Python ecosystem, especially for managing virtual environments and dependencies (pip, conda, Flit, Hatch...) If you already have a preferred way of doing things, this guide might still be useful for discovering some new tools.

Prerequisites

  • Basic knowledge of Python
  • A Unix-based system (Linux or macOS)

Pyenv

Overview

Pyenv is a tool that allows you to install and switch between multiple Python versions. At a high level, pyenv intercepts Python commands using shim executables injected into your PATH, determines which Python version has been specified by your application, and passes your commands along to the correct Python installation. It...

  • Lets you change the global Python version on a per-user basis.
  • Provides support for per-project Python versions.
  • Allows you to override the Python version with an environment variable.
  • Searches for commands from multiple versions of Python at a time. This can be useful for testing across Python versions with tox.

Installation

To install pyenv, follow the instructions from the official repository: https://github.com/pyenv/pyenv#installation

Usage

After installing pyenv, you can install different Python versions using the pyenv install command. For example, to install Python 3.11, run: 

$ pyenv install 3.11

This will install the latest known version. At the time of writing, this is Python 3.11.3. Running pyenv install -l gives the list of all available versions.

Once you have a few different versions installed, you can easily switch between them. E.g. to switch to the latest 3.10 release: 

$ pyenv global 3.10.9

This will set Python 3.10.9 as your default version globally, i.e. for your user account. This assumes that you have previously installed this version. You can list all installed versions with the pyenv versions command.

To set the Python version for a specific project, run this command from the project’s root directory: 

$ pyenv local <python version>

This will switch Python to the desired version and create a .python-versionfile inside the root directory.

Pyenv-virtualenv

Overview

Pyenv does not manage virtual environments, but there is a handy plugin for that: pyenv-virtualenv. This plugin provides features to manage virtualenvs and conda environments for Python on UNIX-like systems by forwarding any options to the underlying command that actually creates the virtual environment (conda, virtualenv, or python -m venv). See the output of pyenv virtualenv --help for details.

Installation

Follow the instructions from the official repository: https://github.com/pyenv/pyenv-virtualenv#installation

Usage

To create a virtual environment for the Python version used with pyenv, run pyenv virtualenv, specifying the Python version you want and the name of the virtualenv directory. For example, 

$ pyenv virtualenv 3.10.9 my-virtual-env-3.10.9

will create a virtualenv based on Python 3.10.9 under $(pyenv root)/versions in a folder called my-virtual-env-3.10.9.

For more information on how to list, delete, activate/deactivate virtualenvs, and control the behavior of pyenv-virtualenv with environment variables, you can consult the official documentation: https://github.com/pyenv/pyenv-virtualenv#usage

Poetry

Overview

Poetry is a powerful and user-friendly tool for managing dependencies and packaging Python projects. It simplifies the process of creating and maintaining projects by automating various tasks, such as installing dependencies, managing virtual environments, and building packages.

Poetry uses the pyproject.toml file to store project metadata, dependencies, and build requirements, making it a single source of truth for your project's configuration. It provides an isolated environment for each project, so you can work on multiple projects with different dependencies and Python versions.

Installation

Install Poetry following the official instructions: https://python-poetry.org/docs/#installation

If you plan to use Poetry in your CI pipelines, you can find some best practices on how to create reproducible environments here: https://python-poetry.org/docs/#ci-recommendations

Once installed, create a new Python project with Poetry: 

$ poetry new my_project
$ cd my_project

This will generate a basic project structure with a pyproject.toml file: 

my_project
├── pyproject.toml
├── README.md
├── my_project
│   └── __init__.py
└── tests
    └── __init__.py

Setting the Python version

Poetry will require you to explicitly specify what versions of Python you intend to support, and its universal locking will guarantee that your project is installable (and all dependencies claim support for) all supported Python versions.

Specify the Python version for the project in the pyproject.toml file. Under the [tool.poetry.dependencies] section, add the desired Python version constraint: 

[tool.poetry.dependencies]
python = "^3.8"

Dependency management

The add command adds required packages to your pyproject.toml and installs them. If you do not specify a version constraint, poetry will choose a suitable one based on the available package versions. 

$ poetry add requests flask

You can specify development dependencies with the -G dev option: 

$ poetry add -G dev black isort

Similarly, you can remove installed dependencies with poetry remove: 

$ poetry remove requests

For more options and other commands, see https://python-poetry.org/docs/cli/.

Virtual environments and Poetry

There are two options when it comes to using virtual environments with Poetry. By default, Poetry creates a virtual environment in {cache-dir}/virtualenvs. You can change the cache-dir value by editing the Poetry configuration. Additionally, you can use the virtualenvs.in-project configuration variable to create virtual environments within your project directory.

The second option is to use an external virtual environment: Poetry will detect and respect an existing virtual environment that has been externally activated. To take advantage of this, simply activate a virtual environment using your preferred method or tooling, before running any Poetry commands that expect to manipulate an environment.

For more information on how to activate and run commands in Poetry’s virtual environment, see https://python-poetry.org/docs/basic-usage/#activating-the-virtual-environment.

Tox

Overview

Tox is a generic virtual environment orchestrator and test command line tool you can use for:

  • Checking your package builds and installs correctly under different environments (such as different Python implementations, versions or installation dependencies),
  • Running your tests in each of the environments with the test tool of choice,
  • Acting as a frontend to continuous integration servers, greatly reducing boilerplate and merging CI and shell-based testing.

Tox can set up environments for and invoke:

  • Test runners (such as pytest),
  • Linters (e.g., flake8),
  • Formatters (for example black or isort),
  • Documentation generators (e.g., Sphinx),
  • Build and publishing tools (e.g., build with twine),
  • etc.

Installation

To install Tox, follow the instructions from the official documentation: https://tox.wiki/en/latest/installation.html

Tox needs a configuration file where you define what tools you need to run and how to provision a test environment for these. The canonical file for this is the tox.ini file. For example: 

[tox]
requires =
    tox>=4
env_list = lint, type, py{38,39,310,311}
[testenv]
description = run unit tests
deps =
pytest>=7
pytest-sugar
commands =
pytest {posargs:tests}

[testenv:lint]
description = run linters
skip_install = true
deps =
black==22.12
commands = black {posargs:.}

[testenv:type]
description = run type checks
deps =
mypy>=0.991
commands =
mypy {posargs:src tests}

It’s also possible to generate a tox.ini file automatically by running tox quickstart and then answering a few questions.

Core settings that affect all test environments or configure how tox itself is invoked are defined under the tox section. Test environments are defined under the testenv section and individual testenv:<env_name> sections, where <env_name> is the name of a specific environment.

For a basic configuration with detailed explanations, see https://tox.wiki/en/latest/user_guide.html#basic-example

Running commands with Tox

Once you have installed Tox and created a tox.ini file with your desired configuration, you can run commands against the different Python versions and environments from the command line.

To run all tox environments defined in the env_list, run tox from your project’s root directory without any flags: 

$ tox .

To run a single tox environment use the -e flag for the run sub-command: 

$ tox run -e py310

It’s also possible to run tox against a subset of environments: 

$ tox run -e format,py310

This will run the commands sequentially, in the specified order. To run two or more tox environments in parallel, use the parallel sub-command: 

$ tox parallel -e py39,py310

For more details on the tox CLI, see https://tox.wiki/en/latest/user_guide.html#cli and https://tox.wiki/en/latest/cli_interface.html

Direnv

Overview

Direnv is an extension for your shell that allows you to set environment variables on a per-directory basis, which is particularly useful when working with multiple Python projects. That said, direnv is language-agnostic and can be used in any project.

Use cases:

Load 12-factor apps environment variables Create per-project isolated development environments Load secrets for deployment

Installation

To install direnv and hook it up to your shell, follow the instructions from the official repository: https://github.com/direnv/direnv#install

Usage

In your project's root directory, create a new file named .envrc: 

$ touch .envrc

In this file, add the following lines: 

export VAR1=value1
export VAR2=value2

Run the following command to allow direnv to set environment variables for the current directory: 

$ direnv allow

These environment variables will now be available inside your project folder, including any nested directories, but not from anywhere else.

Integrate direnv with pyenv-virtualenv

Direnv can also be used in conjunction with pyenv-virtualenv to automatically activate and deactivate virtual environments when you cd in and out of project folders that use them. See this gist for instructions on how to set this up: https://gist.github.com/ZhangChen199102/da3133fc05e3b03afab405fdc3152fb3

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Python/Development_Tools&oldid=2125778"