Help:Toolforge/Web/Python

From Wikitech
Jump to navigation Jump to search

Overview

This page describes Python-specific instructions for deploying a web server on Toolforge. Python web servers on Toolforege use uWSGI which is a Web Server Gateway Interface (WSGI) server for Python web applications. uWSGI can run applicaions built with Flask, Django, and other Python web application frameworks.

Conventions

The Toolforge webservice command starts Python applications using convention rather than configuration. These conventions are expected by the Toolforge tooling:

  • Your WSGI application's entry point must be found in $HOME/www/python/src/app.py in a variable named app (example).
  • Python libraries will be loaded from a virtual environment located in $HOME/www/python/venv.
    • On the Kubernetes backend, you must use a virtual environment to install and load any libraries you depend on outside of the Python standard library.
  • Additional configuration for uWSGI can be provided in a $HOME/www/python/uwsgi.ini file.
    • Examples of configuration parameters can be found in the uWSGI manual.
    • Headers can be added using route = .* addheader:Access-Control-Allow-Origin: *
  • Logs will be written to $HOME/uwsgi.log

Starting a Python web service

To start a Python web service, use the webservice start command. For example:

Python3.7 with a default uwsgi configuration
webservice --backend=kubernetes python3.7 start
Python3.5 with a default uwsgi configuration (deprecated)
webservice --backend=kubernetes python3.5 start
Python3.4 with a default uwsgi configuration (deprecated)
webservice --backend=kubernetes python start
Python2 with a default uwsgi configuration (deprecated)
webservice --backend=kubernetes python2 start
Python2 on Grid Engine with a default uwsgi configuration
webservice --backend=gridengine uwsgi-python start
Python2 or Python3 on Grid Engine with a user supplied uwsgi configuration
webservice --backend=gridengine uwsgi-plain start

Virtual Environments and Packages

A virtual environment (venv) is a self-contained directory tree that contains a Python installation for a particular version of Python plus a number of additional packages. Using a venv allows you to install local Python packages for your tool.

The fundamental thing to remember is that a venv created directly on the bastion will only work with --backend=gridengine, and a venv created inside a webservice shell work only with --backend=kubernetes and the same Python runtime version.

Creating a virtual environment

  1. webservice --backend=kubernetes python3.7 shell (choose a different python version as appropriate for your project)
  2. mkdir -p $HOME/www/python
  3. python3 -m venv $HOME/www/python/venv
  4. source $HOME/www/python/venv/bin/activate
  5. pip install --upgrade pip wheel (This brings in newest pip, which is required for wheel support)
  6. Install the libraries you need (for example pip install -r $HOME/www/python/src/requirements.txt)
  7. exit out of webservice shell
  8. webservice --backend=kubernetes python3.7 start

Step 1 can possibly freeze with an error message Pod is not ready in time. Retrying the command again should fix it.

Steps 2-6 can be automated by using the webservice-python-bootstrap script inside the webservice shell. If you want to create a brand new virtualenv in case you're switching Python versions or have new dependencies, use webservice-python-bootstrap --fresh.

Using a uwsgi app with a default entry point that is not app.py

The default uwsgi configuration for the uwsgi webservice backend expects to find the uwsgi entry point as the variable app loaded from the $HOME/www/python/src/app.py module. If your application has another entry point, the easiest thing to do is create a $HOME/www/python/src/app.py module, import your entry point, and expose it as app. See Deploying a Django application for an example of this pattern.

Deploying a Django application

Django is a popular web framework for developing Python applications. A typical Django application will need a few changes to run with Toolforge's opinionated uWSGI configuration.

Create an app.py entry point

$HOME/www/python/src/app.py
import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<YOUR-TOOL-NAME>.settings")

app = get_wsgi_application()

Static files

To correctly locate the static files, first configure $HOME/www/python/uwsgi.ini to look for static files in your tool's $HOME/www/python/src/static directory:

$HOME/www/python/uwsgi.ini
[uwsgi]
static-map = /static=/data/project/<YOUR-TOOL-NAME>/www/python/src/static

Next configure your Django app to use this location by editing the app's settings.py file:

settings.py
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')

Finally deploy your static files into $HOME/www/python/src/static. Typically this will be done by running python manage.py collectstatic.

MySQL and utf8mb4

The version of MySQL/MariaDB currently used by ToolsDB will not work transparently with Django and the utf8mb4 character set. ROW_FORMAT=DYNAMIC must be used on all tables which will index utf8mb4 encoded character fields longer than 191 characters. This configuration, together with database server configuration to enable innodb_large_prefix, the Barracuda file format, and file per table storage, enables index key prefixes longer than 767 bytes (up to 3072 bytes) for InnoDB tables.

Django does not have a feature flag or setting for adding the needed ROW_FORMAT=DYNAMIC configuration to its database migrations. One way to work around this issue is by using a custom database engine as Volans did for his Debmonitor project. Another possible workaround is manually modifying your migration files as BryanDavis documented in a blog post.

Logs

You can find your application's log messages in $HOME/uwsgi.log.

Communication and support

We communicate and provide support through several primary channels. Please reach out with questions and to join the conversation.

Communicate with us
Connect Best for
Phabricator Workboard #Cloud-Services Task tracking and bug reporting
IRC Channel #wikimedia-cloud connect General discussion and support
Mailing List cloud@ Information about ongoing initiatives, general discussion and support
Announcement emails cloud-announce@ Information about critical changes (all messages mirrored to cloud@)
News wiki page News Information about major near-term plans
Cloud Services Blog Clouds & Unicorns Learning more details about some of our work
Wikimedia Technology Blog techblog.wikimedia.org News and stories from the Wikimedia technical movement

See also