Help:Toolforge/My first Django OAuth tool

From Wikitech
Jump to: navigation, search
Toolforge HelpFAQRulesDevelopingDatabasesJob gridKubernetesWebHow toList of ToolsGlossary

This guide will show how to set up a Django app on Toolforge. We will use the python-social-auth library to implement OAuth authentification with Wikipedia, using the Mediawiki OAuth capabilities.

If you are new to Django you should read through some of the tutorials in the Tutorials section first.

Local development and testing

First we will set the project up on your locale machine fore development.

Create your Django project

Most of the setup is not different from the many tutorials that you can find for Django.

First you need to create a Python 3 virtual environment:

$ python3 -m venv venv-my-first-django-oauth-app
$ source venv-my-first-django-oauth-app/bin/activate
$ pip install django

Then set up a new Django project:

$ mkdir my-first-django-oauth-app
$ cd my-first-django-oauth-app
$ mkdir src
$ django-admin startproject oauth_app src

Now we have the main app set up and the project folder structure will look like this:

[my-first-django-oauth-app]$ tree
.
└── src
    ├── manage.py
    └── oauth_app
        ├── __init__.py
        ├── settings.py
        ├── urls.py
        └── wsgi.py

Next we need to add an app for for our actual webpage:

$ cd src
$ django-admin startapp user_profile

The structure will then look like this:

[my-first-django-oauth-app]$ tree
.
└── src
    ├── manage.py
    ├── oauth_app
    │   ├── __init__.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    └── user_profile
        ├── admin.py
        ├── apps.py
        ├── __init__.py
        ├── migrations
        │   └── __init__.py
        ├── models.py
        ├── tests.py
        └── views.py

Then add this new app to the installed apps in settings.py:

INSTALLED_APPS = [
    ...
    'user_profile',
]

And then route the main page from the main app to our new user_profile app. This is done in the urls.py file in the main app:

from django.conf.urls import url, include
from django.contrib import admin

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'', include('user_profile.urls')),
]

Create a urls.py in your user_profile folder as well, and add the following to it:

from django.conf.urls import url
from user_profile import views

urlpatterns = [
    url(r'', views.index),
]

The index view is still missing. Create it in views.py:

from django.shortcuts import render

def index(request):
    context = {}
    return render(request, 'user_profile/index.dtl', context)

Now we only need to create the Django template in the folder templates/user_profile/index.dtl (You can also use html as file extension if you don't get syntax highlighting for dtl files).

<!DOCTYPE html>
<html>
<body>
  <h1>My first Django OAuth app</h1>
</body>
</html>

The current file structure will look something like this:

[my-first-django-oauth-app]$ tree
.
└── src
    ├── db.sqlite3
    ├── manage.py
    ├── oauth_app
    │   ├── __init__.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    └── user_profile
        ├── admin.py
        ├── apps.py
        ├── __init__.py
        ├── migrations
        │   └── __init__.py
        ├── models.py
        ├── templates
        │   └── user_profile
        │       └── index.dtl
        ├── tests.py
        ├── urls.py
        └── views.py

Now you can start Django's built in development webserver:

$ python manage.py runserver 127.0.0.1:8080

This will show you the template you just created. Note that we are running on port 8080 because port 8000 is used on some systems. In order to stay consistent we will stick with port 8080 for local development. Because the port is also part of the OAuth callback URL, we will reduce the number of consumers we need to register for development.

Adding OAuth

Starting with version 1.2 the Python package social-core has a Mediawiki backend that works for any of Wikimedia Foundations wikis, but also for any other Mediawiki installation that has the OAuth Extension enabled.

Install the package for your virtual environment:

$ pip install social-auth-app-django

Add the following in your main app's settings.py:

INSTALLED_APPS = [
    ...
    'social_django',
]

MIDDLEWARE_CLASSES = [
    ...
    'social_django.middleware.SocialAuthExceptionMiddleware',
]

TEMPLATES = [
    {
        ...
        'OPTIONS': {
            'context_processors': [
                ...
                'social_django.context_processors.backends',
                'social_django.context_processors.login_redirect',
            ],
        },
    },
]

AUTHENTICATION_BACKENDS = (
    'social_core.backends.mediawiki.MediaWiki',
    'django.contrib.auth.backends.ModelBackend',
)

Then we need to add settings for the OAuth provider. You can register your application, after reading the OAuth for developers documentation.

We will register a consumer for development at this moment, so we will use the following settings:

  • Application name: use a name that indicates that you are developing locally
  • Contact email address: Use a valid email where you can be reached.
  • Applicable project: All is fine
  • OAuth "callback" URL: http://127.0.0.1:8080/
  • Select: Allow consumer to specify a callback in requests and use "callback" URL above as a required prefix.
  • Types of grants being requested: Choose "User identity verification only, no ability to read pages or act on a user's behalf."
  • Public RSA key: You can leave this empty at the moment.

Be sure not to commit these keys in version control and keep them secret.

SOCIAL_AUTH_MEDIAWIKI_KEY = '**super secret key**'
SOCIAL_AUTH_MEDIAWIKI_SECRET = '**super secret secret**'
SOCIAL_AUTH_MEDIAWIKI_URL = 'https://meta.wikimedia.org/w/index.php'
SOCIAL_AUTH_MEDIAWIKI_CALLBACK = 'http://127.0.0.1:8080/oauth/complete/mediawiki/'

N.B. The trailing '/' at the end of the callback is important. Login will silently fail without it.

After that you need to apply the migrations in order to create the user models in the database.

$ python manage.py migrate

Next we need to up a few things for the profile view and the login. Update your user_profile apps urls.py to say something like this:

from django.conf.urls import url, include
from user_profile import views

urlpatterns = [
    url(r'profile', views.profile, name='profile'),
    url(r'accounts/login', views.login_oauth, name='login'),
    url(r'oauth/', include('social_django.urls', namespace='social')),
    url(r'', views.index),
]

In settings.py we need to add the URL pattern named "login" as the LOGIN_URL and the URL pattern "profile" for LOGIN_REDIRECT_URL:

LOGIN_URL = 'login'
LOGIN_REDIRECT_URL = 'profile'

Create two new views for the profile and the login:

from django.shortcuts import render
from django.contrib.auth.decorators import login_required


def index(request):
    context = {}
    return render(request, 'user_profile/index.dtl', context)

@login_required()
def profile(request):
    context = {}
    return render(request, 'user_profile/profile.dtl', context)

def login_oauth(request):
    context = {}
    return render(request, 'user_profile/login.dtl', context)

Then create the login.dtl file:

<!DOCTYPE html>
<html>
<body>
  <h1>Login</h1>
  <a href="{% url 'social:begin' 'mediawiki' %}">Login with Wikimedia</a>
</body>
</html>

And the profile.dtl file:

<!DOCTYPE html>
<html>
<body>
  <h1>Profile</h1>
  {{ user }}
</body>
</html>

And finally update your index.dtl to say:

<!DOCTYPE html>
<html>
<body>
  <h1>My first Django OAuth app</h1>
  <a href="{% url 'profile' %}">Profile</a>
</body>
</html>

When you now restart the server and click on the "Profile" link you will be asked to give permission to your newly created OAuth consumer. After approving the consumer you will be redirected to the user page which will print the Wikimedia user name.

Deploying to Wikimedia Toolforge

After extensive testing you might want to share your new tool with a wider audience. If the tool fulfills the criteria of Toolforge you can host it there.

You can create a new tool account on toolsadmin.

Configure project for production environment

In order to make your project work in both environments and keep your secret keys out of source control you can for example define the following variables in venv-my-first-django-oauth-app/bin/activate. That means the keys will be available only if the environment is active:

...
export django_secret="very-secret-key"
export mediawiki_key="very-secret-mediawiki-key"
export mediawiki_secret="very-secret-mediawiki-secret"
export mediawiki_callback="http://127.0.0.1:8080/oauth/complete/mediawiki/"

And after reactivating the venv you will be able to access the variables in settings.py

SECRET_KEY = os.environ.get('django_secret')
...
SOCIAL_AUTH_MEDIAWIKI_KEY = os.environ.get('mediawiki_key')
SOCIAL_AUTH_MEDIAWIKI_SECRET = os.environ.get('mediawiki_secret')
SOCIAL_AUTH_MEDIAWIKI_URL = 'https://meta.wikimedia.org/w/index.php'
SOCIAL_AUTH_MEDIAWIKI_CALLBACK = os.environ.get('mediawiki_callback')

Now you can safely add our project to git. Run this in your project directory:

$ git init

You can then also add your requirements to your project:

$ pip freeze > requirements.txt

We also need to update the allowed hosts (See: Allowed-hosts) in settings.py:

ALLOWED_HOSTS = [
    'tools.wmflabs.org',
]

Don't forget to create a .gitignore file that excludes all files that should not be under source control. For example:

*.pyc
__pycache__/
*.sqlite3

Then add all the files to git. Make sure that only source files and no temporary files, or parts of your venv are staged. After that you can make your first commit.

Use UWSGI locally

On Toolforge your Django app will run with UWSGI. In order to have a similar setup on the development machine we will create a similar setup using Nginx. First we need to create an app.ini in our src directory:

[uwsgi]
module = oauth_app.wsgi

plugins = python3

chdir = /home/YOURUSERNAME/my-first-django-oauth-app/src
home = /home/YOURUSERNAME/venv-my-first-django-oauth-app

master = true
processes = 5

uid = YOURUSERNAME
socket = /run/uwsgi/django-oauth.sock
chown-socket = YOURUSERNAME:nginx
chmod-socket = 664
vacuum = true

die-on-term = true

You should be able to start the uwsgi file on its own by using: uwsgi app.ini. If it fails this is often a permission error with the /run/uwsgi directory. The current user needs to be able to create the socket file in this directory.

For this file we will create a service in /etc/systemd/system/django-oauth.uwsgi.service:

[Unit]
Description=uWSGI instance to serve django oauth app

[Service]
ExecStartPre=-/usr/bin/bash -c 'mkdir -p /run/uwsgi; chown YOURUSERNAME:nginx /run/uwsgi'
ExecStart=/usr/bin/bash -c 'cd /home/YOURUSERNAME/my-first-django-oauth-app/src; source /home/YOURUSERNAME/venv-my-first-django-oauth-app/bin/activate; uwsgi --ini app.ini'

[Install]
WantedBy=multi-user.target

Ubuntu for example does not have /usr/bin/bash and you need to use /bin/bash. Errors like this can be seen when using sudo systemctl status django-oauth.uwsgi.service.

You can then start this service by running sudo systemctl start django-oauth.uwsgi.service. If that has worked you can continue to configure the webserver. Go to /etc/nginx/nginx.conf and add the following configuration:

http {

    ...

    server {
        listen 8080;
        server_name 127.0.0.1;

        location / {
            include uwsgi_params;
            uwsgi_pass unix:/run/uwsgi/django-oauth.sock;
        }
    }

    ...

    server { ...

After that you can either restart or start the nginx server (sudo systemctl start nginx). You can check if everything is working correctly by entering: sudo systemctl status nginx. On an SELinux enabled system you might still need to set the correct target context for the sock file. Now go to 127.0.0.1:8080 and see if your page still loads correctly.

Login and deploy

After configuring two-factor authentication you can log into Toolforge using:

$ ssh YOUR_USER_NAME@login.tools.wmflabs.org

After the successful login you need to switch to your new tool:

$ become YOUR_TOOL_NAME

You can use the following commands to control the web server:

$ webservice --backend=kubernetes python status/start/stop/restart

Create the following file structure for the web server config file:

.
└── www
    └── python
        ├──uwsgi.ini
        └──src

In the uwsgi.ini file add the following:

[uwsgi]
check-static = /data/project/YOUR_TOOL_NAME/www/python

In the python directory you can clone your git repository (The dot at the end clones into your current directory instead of creating another subfolder):

[python] $ git clone ADDRESS_OF_REPOSITORY .

You also have to create a virtual environment on the server using:

[python] $ virtualenv -p /usr/bin/python3.4 venv

Then you are ready to request you production keys (which you need to be even more carful about not committing):

  • Application name: use a name that indicates that this is production
  • Contact email address: Use a valid email where you can be reached.
  • Applicable project: All is fine
  • OAuth "callback" URL: https://tools.wmflabs.org/YOUR-TOOL-NAME/ Security Warning: Your tool name has to be part of the callback URL or the OAuth consumer will be rejected.
  • Select: Allow consumer to specify a callback in requests and use "callback" URL above as a required prefix.
  • Types of grants being requested: Choose "User identity verification only, no ability to read pages or act on a user's behalf."
  • Public RSA key: You can leave this empty at the moment.

Edit the virtual environment activate file to add your production keys to it:

export django_secret="very-secret-production-key"
export mediawiki_key="very-secret-mediawiki-production-key"
export mediawiki_secret="very-secret-mediawiki-production-secret"
export mediawiki_callback="https://tools.wmflabs.org/YOUR-TOOL-NAME/oauth/complete/mediawiki/"

Activate the environment to install the dependencies and apply the migrations, then deactivate it again:

[python] $ source venv/bin/activate
[python] $ pip install -r requirements.txt

[python] $ python src/manage.py makemigrations
[python] $ python src/manage.py migrate

[python] $ deactivate

Running the 'migrate' command will also create the database as a sqlite file. The file with our current settings is at /www/python/src/db.sqlite3. You should take care to back up this file and be confidential with the information that is stored within it.

With the current setup you also have to create a 2nd app.py in ~/www/python/src where the standard webservice will look for it. It also expects the variable to be named app:

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "oauth_app.settings")

app = get_wsgi_application()

After making these configurations you should be able to start the webserver using:

$ webservice --backend=kubernetes python start

In case there are any problems with the configuration they will appear in the tools root directory in the file: ~/uwsgi.log.

The project structure as of now will look like this:

$ tree
.
├── logs
├── replica.my.cnf
├── service.manifest
├── uwsgi.log
└── www
    └── python
        ├── requirements.txt
        ├── src
        │   ├── app.py
        │   ├── db.sqlite3
        │   ├── manage.py
        │   ├── oauth_app
        │   │   ├── __init__.py
        │   │   ├── settings.py
        │   │   ├── urls.py
        │   │   └── wsgi.py
        │   └── user_profile
        │       ├── admin.py
        │       ├── apps.py
        │       ├── __init__.py
        │       ├── migrations
        │       │   └── __init__.py
        │       ├── models.py
        │       ├── templates
        │       │   └── user_profile
        │       │       ├── index.dtl
        │       │       ├── login.dtl
        │       │       └── profile.dtl
        │       ├── tests.py
        │       ├── urls.py
        │       └── views.py
        ├── uwsgi.ini
        └── venv

You can view a working example of the OAuth app at:

https://tools.wmflabs.org/my-first-django-oauth-app/

Scale up your database

So far the tutorial only used sqlite as a database. This is fine for demo applications, but will quickly result in performance or scaling issues with any operations that require reads or writes to the database. It will also strain the Toolforge servers, which is something we also want to avoid. The solution is to use a dedicated database server. You can choose one of the solutions in this section and continue to read there.

Maria DB

First you need to install the required system packages. On Debian / Ubuntu / Fedora / ect.. the package should be called: mariadb-server. Depending on your distribution you need to start the service as:

$ service mysql start

And check if it is running:

$ service mysql status

or:

$ service mariadb start

And check if it is running:

$ service mariadb status

Links

Tutorials

Code

Further Reading

Other tools