Help:Toolforge/My first Django OAuth tool
Overview
This guide will show how to set up a Django app on Toolforge. We will use the python-social-auth library to implement OAuth authentification with Wikipedia, using the Mediawiki OAuth capabilities.
If you are new to Django you should read through some of the tutorials in the Tutorials section first.
Local development and testing
First, we will set the project up on your local machine for development.
Create your Django project
Most of the setup is not different from the many tutorials that you can find for Django.
First you need to create a Python 3 virtual environment:
$ python3 -m venv venv-my-first-django-oauth-app
$ source venv-my-first-django-oauth-app/bin/activate
$ pip install django
Then set up a new Django project:
$ mkdir my-first-django-oauth-app
$ cd my-first-django-oauth-app
$ mkdir src
$ django-admin startproject oauth_app src
Make sure you change the permissions on your settings.py file so that you don't share your app's secrets unintentionally!
$ chmod o-r src/oauth_app/settings.py
If you are checking this into source control, you should avoid checking in the settings.py file. If using git, you can use a .gitignore file at the root of your git repo with something like these contents:
*settings.py
*.pyc
__pycache__/
Now we have the main app set up and the project folder structure will look like this:
[my-first-django-oauth-app]$ tree
.
└── src
├── manage.py
└── oauth_app
├── __init__.py
├── settings.py
├── urls.py
└── wsgi.py
Next we need to add an app for for our actual webpage:
$ cd src
$ django-admin startapp user_profile
The structure will then look like this:
[my-first-django-oauth-app]$ tree
.
└── src
├── manage.py
├── oauth_app
│ ├── __init__.py
│ ├── settings.py
│ ├── urls.py
│ └── wsgi.py
└── user_profile
├── admin.py
├── apps.py
├── __init__.py
├── migrations
│ └── __init__.py
├── models.py
├── tests.py
└── views.py
Then add this new app to the installed apps in settings.py:
INSTALLED_APPS = [
...
'user_profile',
]
And then route the main page from the main app to our new user_profile app. This is done in the urls.py file in the main app:
from django.urls import path, include
from django.contrib import admin
urlpatterns = [
path('^admin/', admin.site.urls),
path('', include('user_profile.urls')),
]
Create a urls.py in your user_profile folder as well, and add the following to it:
from django.urls import path
from user_profile import views
urlpatterns = [
path('', views.index),
]
The index view is still missing. Create it in views.py:
from django.shortcuts import render
def index(request):
context = {}
return render(request, 'user_profile/index.dtl', context)
Now we only need to create the Django template in the folder templates/user_profile/index.dtl (You can also use html as file extension if you don't get syntax highlighting for dtl files).
<!DOCTYPE html>
<html>
<body>
<h1>My first Django OAuth app</h1>
</body>
</html>
The current file structure will look something like this:
[my-first-django-oauth-app]$ tree
.
└── src
├── db.sqlite3
├── manage.py
├── oauth_app
│ ├── __init__.py
│ ├── settings.py
│ ├── urls.py
│ └── wsgi.py
└── user_profile
├── admin.py
├── apps.py
├── __init__.py
├── migrations
│ └── __init__.py
├── models.py
├── templates
│ └── user_profile
│ └── index.dtl
├── tests.py
├── urls.py
└── views.py
Now you can start Django's built in development webserver:
$ python manage.py runserver 127.0.0.1:8080
This will show you the template you just created. Note that we are running on port 8080 because port 8000 is used on some systems. In order to stay consistent we will stick with port 8080 for local development. Because the port is also part of the OAuth callback URL, we will reduce the number of consumers we need to register for development.
Adding OAuth
Starting with version 1.2 the Python package social-core has a Mediawiki backend that works for any of Wikimedia Foundations wikis, but also for any other Mediawiki installation that has the OAuth Extension enabled.
Install the package for your virtual environment:
$ pip install social-auth-app-django
Add the following in your main app's settings.py:
INSTALLED_APPS = [
...
'social_django',
]
MIDDLEWARE = [
...
'social_django.middleware.SocialAuthExceptionMiddleware',
]
TEMPLATES = [
{
...
'OPTIONS': {
'context_processors': [
...
'social_django.context_processors.backends',
'social_django.context_processors.login_redirect',
],
},
},
]
AUTHENTICATION_BACKENDS = (
'social_core.backends.mediawiki.MediaWiki',
'django.contrib.auth.backends.ModelBackend',
)
# Workaround for error T353593
SOCIAL_AUTH_PROTECTED_USER_FIELDS = ['groups']
Oauth consumer registration (Wikimedia)
Then we need to add settings for the OAuth provider. You can register your application, after reading the OAuth for developers documentation.
- meta:Special:OAuthConsumerRegistration/propose is for registering OAuth to production environment
- meta.wikimedia.beta.wmflabs.org/wiki/Special:OAuthConsumerRegistration/propose is for testing environment
We will register a consumer for development at this moment, so we will use the following settings:
- Oauth version: OAuth 1.0a consumer
- Application name: use a name that indicates that you are developing locally
- Leave "This consumer is for use only by <your username>" unchecked
- Contact email address: Use a valid email where you can be reached.
- Applicable project: All is fine
- OAuth "callback" URL: http://127.0.0.1:8080/
- Select: Allow consumer to specify a callback in requests and use "callback" URL above as a required prefix.
- Types of grants being requested: Choose "User identity verification only, no ability to read pages or act on a user's behalf."
- Public RSA key: You can leave this empty at the moment.
You should get back a response something like:
Your OAuth consumer request has been received. You have been assigned a consumer token of xxxxxxxxxxxxxxxxxxxxxxxx and a secret token of xxxxxxxxxxxxxxxxxxxxxxxx. Be sure not to commit these keys in version control and keep them secret.
The consumer token is your SOCIAL_AUTH_MEDIAWIKI_KEY, and the secret token is your SOCIAL_AUTH_MEDIAWIKI_SECRET, in settings.py:
SOCIAL_AUTH_MEDIAWIKI_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxx'
SOCIAL_AUTH_MEDIAWIKI_SECRET = 'xxxxxxxxxxxxxxxxxxxxxxxx'
SOCIAL_AUTH_MEDIAWIKI_URL = 'https://meta.wikimedia.org/w/index.php'
SOCIAL_AUTH_MEDIAWIKI_CALLBACK = 'http://127.0.0.1:8080/oauth/complete/mediawiki/'
N.B. The trailing '/' at the end of the callback is important. Login will silently fail without it.
After that, you need to apply the migrations in order to create the user models in the database.
$ python manage.py migrate
Next we need to up a few things for the profile view and the login. Update your user_profile apps urls.py to say something like this:
from django.urls import path, include
from user_profile import views
urlpatterns = [
path('profile', views.profile, name='profile'),
path('accounts/login', views.login_oauth, name='login'),
path('oauth/', include('social_django.urls', namespace='social')),
path('', views.index),
]
In settings.py we need to add the URL pattern named "login" as the LOGIN_URL and the URL pattern "profile" for LOGIN_REDIRECT_URL:
LOGIN_URL = 'login'
LOGIN_REDIRECT_URL = 'profile'
Create two new views for the profile and the login:
from django.shortcuts import render
from django.contrib.auth.decorators import login_required
def index(request):
context = {}
return render(request, 'user_profile/index.dtl', context)
@login_required()
def profile(request):
context = {}
return render(request, 'user_profile/profile.dtl', context)
def login_oauth(request):
context = {}
return render(request, 'user_profile/login.dtl', context)
Then create the login.dtl file:
<!DOCTYPE html>
<html>
<body>
<h1>Login</h1>
<a href="{% url 'social:begin' 'mediawiki' %}">Login with Wikimedia</a>
</body>
</html>
And the profile.dtl file:
<!DOCTYPE html>
<html>
<body>
<h1>Profile</h1>
{{ user }}
</body>
</html>
And finally update your index.dtl to say:
<!DOCTYPE html>
<html>
<body>
<h1>My first Django OAuth app</h1>
<a href="{% url 'profile' %}">Profile</a>
</body>
</html>
When you now restart the server and click on the "Profile" link you will be asked to give permission to your newly created OAuth consumer. After approving the consumer you will be redirected to the user page which will print the Wikimedia user name.
Deploying to Wikimedia Toolforge
After extensive testing you might want to share your new tool with a wider audience. If the tool fulfills the criteria of Toolforge you can host it there.
- See Help:Toolforge/Quickstart for creating Tool account].
- You can create a new tool on the Toolforge admin console.
There small delay, 15 mins, before tool is actually created after registration.
Configure project for production environment
In order to make your project work in both environments and keep your secret keys out of source control you can for example define the following variables in venv-my-first-django-oauth-app/bin/activate. That means the keys will be available only if the environment is active:
...
export django_secret="very-secret-key"
export mediawiki_key="very-secret-mediawiki-key"
export mediawiki_secret="very-secret-mediawiki-secret"
export mediawiki_callback="http://127.0.0.1:8080/oauth/complete/mediawiki/"
And after reactivating the venv you will be able to access the variables in settings.py
SECRET_KEY = os.environ.get('django_secret')
...
SOCIAL_AUTH_MEDIAWIKI_KEY = os.environ.get('mediawiki_key')
SOCIAL_AUTH_MEDIAWIKI_SECRET = os.environ.get('mediawiki_secret')
SOCIAL_AUTH_MEDIAWIKI_URL = 'https://meta.wikimedia.org/w/index.php'
SOCIAL_AUTH_MEDIAWIKI_CALLBACK = os.environ.get('mediawiki_callback')
Now you can safely add our project to git. Run this in your project directory:
$ git init
You can then also add your requirements to your project:
$ pip freeze > requirements.txt
We also need to update the allowed hosts (See: Allowed-hosts) in settings.py:
ALLOWED_HOSTS = [
'YOUR_TOOL_NAME.toolforge.org',
]
Don't forget to create a .gitignore file that excludes all files that should not be under source control. For example:
*.pyc __pycache__/ *.sqlite3
Then add all the files to git. Make sure that only source files and no temporary files, or parts of your venv are staged. After that, you can make your first commit.
Use UWSGI locally
On Toolforge your Django app will run with UWSGI. In order to have a similar setup on the development machine we will create a similar setup using Nginx. First we need to create an app.ini in our src directory:
[uwsgi]
module = oauth_app.wsgi
plugins = python3
chdir = /home/YOURUSERNAME/my-first-django-oauth-app/src
home = /home/YOURUSERNAME/venv-my-first-django-oauth-app
master = true
processes = 5
uid = YOURUSERNAME
socket = /run/uwsgi/django-oauth.sock
chown-socket = YOURUSERNAME:nginx
chmod-socket = 664
vacuum = true
die-on-term = true
You should be able to start the uwsgi file on its own by using: uwsgi app.ini. If it fails this is often a permission error with the /run/uwsgi directory. The current user needs to be able to create the socket file in this directory.
For this file we will create a service in /etc/systemd/system/django-oauth.uwsgi.service:
[Unit]
Description=uWSGI instance to serve django oauth app
[Service]
ExecStartPre=-/usr/bin/bash -c 'mkdir -p /run/uwsgi; chown YOURUSERNAME:nginx /run/uwsgi'
ExecStart=/usr/bin/bash -c 'cd /home/YOURUSERNAME/my-first-django-oauth-app/src; source /home/YOURUSERNAME/venv-my-first-django-oauth-app/bin/activate; uwsgi --ini app.ini'
[Install]
WantedBy=multi-user.target
You can then start this service by running sudo systemctl start django-oauth.uwsgi.service. If that has worked you can continue to configure the webserver. Go to /etc/nginx/nginx.conf and add the following configuration:
http {
...
server {
listen 8080;
server_name 127.0.0.1;
location / {
include uwsgi_params;
uwsgi_pass unix:/run/uwsgi/django-oauth.sock;
}
}
...
server { ...
After that you can either restart or start the nginx server (sudo systemctl start nginx). You can check if everything is working correctly by entering: sudo systemctl status nginx. On an SELinux enabled system you might still need to set the correct target context for the sock file. Now go to 127.0.0.1:8080 and see if your page still loads correctly.
Login and deploy
After configuring two-factor authentication you can log into Toolforge using:
$ ssh YOUR_USER_NAME@login.toolforge.org
After the successful login you need to switch to your new tool:
$ become YOUR_TOOL_NAME
You can use the following commands to control the web server:
$ webservice --backend=kubernetes python3.11 status/start/stop/restart
Create the following file structure for the web server config file:
.
└── www
└── python
├──uwsgi.ini
└──src
In the uwsgi.ini file add the following:
[uwsgi]
check-static = /data/project/YOUR_TOOL_NAME/www/python/static
Any files in this static directory will be publicly accessible on the internet.
In the python directory you can clone your git repository (The dot at the end clones into your current directory instead of creating another subfolder):
[python] $ git clone ADDRESS_OF_REPOSITORY .
You also have to create a virtual environment on the server using:
[python] $ webservice --backend=kubernetes python3.11 shell
[python] $ python3 -m venv venv
Then you are ready to request your production keys (which you need to be even more careful about not committing):
- Application name: use a name that indicates that this is production
- Contact email address: Use a valid email where you can be reached.
- Applicable project: All is fine
- OAuth "callback" URL: https://YOUR-TOOL-NAME.toolforge.org/
- Select: Allow consumer to specify a callback in requests and use "callback" URL above as a required prefix.
- Types of grants being requested: Choose "User identity verification only, no ability to read pages or act on a user's behalf."
- Public RSA key: You can leave this empty at the moment.
Make the virtual environment activate file non-world-readable to ensure that nobody else will be able to access the production keys in it:
[python] $ chmod go-rwx venv/bin/activate
Then edit the file to add your production keys to it:
export django_secret="very-secret-production-key"
export mediawiki_key="very-secret-mediawiki-production-key"
export mediawiki_secret="very-secret-mediawiki-production-secret"
export mediawiki_callback="https://YOUR-TOOL-NAME.toolforge.org/oauth/complete/mediawiki/"
Activate the environment to install the dependencies and apply the migrations, then deactivate it again:
[python] $ webservice --backend=kubernetes python3.11 shell
[python] $ source venv/bin/activate
[python] $ pip install -r requirements.txt
[python] $ python3 src/manage.py makemigrations
[python] $ python3 src/manage.py migrate
[python] $ deactivate
Running the 'migrate' command will also create the database as a sqlite file. The file with our current settings is at /www/python/src/db.sqlite3. You should take care to back up this file and be confidential with the information that is stored within it.
With the current setup you also have to create a 2nd app.py in ~/www/python/src where the standard webservice will look for it. It also expects the variable to be named app:
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "oauth_app.settings")
app = get_wsgi_application()
After making these configurations you should be able to start the webserver using:
$ webservice --backend=kubernetes python3.11 start
In case there are any problems with the configuration they will appear in the tools root directory in the file: ~/uwsgi.log.
The project structure as of now will look like this:
$ tree
.
├── logs
├── replica.my.cnf
├── service.manifest
├── uwsgi.log
└── www
└── python
├── requirements.txt
├── src
│ ├── app.py
│ ├── db.sqlite3
│ ├── manage.py
│ ├── oauth_app
│ │ ├── __init__.py
│ │ ├── settings.py
│ │ ├── urls.py
│ │ └── wsgi.py
│ └── user_profile
│ ├── admin.py
│ ├── apps.py
│ ├── __init__.py
│ ├── migrations
│ │ └── __init__.py
│ ├── models.py
│ ├── templates
│ │ └── user_profile
│ │ ├── index.dtl
│ │ ├── login.dtl
│ │ └── profile.dtl
│ ├── tests.py
│ ├── urls.py
│ └── views.py
├── uwsgi.ini
└── venv
You can view a working example of the OAuth app at:
https://my-first-django-oauth-app.toolforge.org/
Using MariaDB instead of SQLite
If your tool's database stores a lot of data or if it is meant to be accessed synchronously from various processes it might be wise to use MariaDB as a database instead of SQLite. You can create a user database for this.
To configure Django to use this database, you first need to install the `mysqlclient` Python package. Installing this package on the Toolforge is a little more complicated than other packages because it uses a native library. First, get inside the virtual environment of your webservice. Once there you can define environment variables as follow, and then install the package:
$ webservice --backend=kubernetes python3.11 shell
$ cd www/python
$ source venv/bin/activate
$ export MYSQLCLIENT_CFLAGS="-I/usr/include/mariadb/"
$ export MYSQLCLIENT_LDFLAGS="-L/usr/lib/x86_64-linux-gnu/ -lmariadb"
$ pip install mysqlclient
In your Django settings.py file, you can then configure the database connection as follows:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 's12345__mytool', # the name of your user database
'HOST': 'tools.db.svc.wikimedia.cloud',
'OPTIONS': {
# somehow MariaDB will not handle unicode properly without the following two lines
'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
'charset': 'utf8mb4',
'read_default_file': os.path.expanduser("~/replica.my.cnf")
},
}
}
Scale up your database
So far the tutorial only used sqlite as a database. This is fine for demo applications, but will quickly result in performance or scaling issues with any operations that require reads or writes to the database. It will also strain the Toolforge servers, which is something we also want to avoid. You can use user databases on the tools.db.svc or create your own database server on Cloud VPS.
Howto use Wikimedia API with Oauth credentials
Howto query API:Userinfo with Oauth credientials after user has been logged in.
from social_django.models import UserSocialAuth
from requests_oauthlib import OAuth1Session
from oauthlib.oauth1 import TokenExpiredError
from oauth_app.settings import SOCIAL_AUTH_MEDIAWIKI_KEY, SOCIAL_AUTH_MEDIAWIKI_SECRET
usersocialauth = UserSocialAuth.objects.filter(provider='mediawiki').first()
if not usersocialauth:
print("UserSocialAuth user not found")
exit()
oauth_token=usersocialauth.extra_data.get('access_token').get('oauth_token')
oauth_token_secret=usersocialauth.extra_data.get('access_token').get('oauth_token_secret')
client = OAuth1Session(
SOCIAL_AUTH_MEDIAWIKI_KEY,
client_secret=SOCIAL_AUTH_MEDIAWIKI_SECRET,
resource_owner_key=oauth_token,
resource_owner_secret=oauth_token_secret
)
try:
# Print mediawiki userinfo of user logged in
url ='https://meta.wikimedia.org/w/api.php?format=json&action=query&meta=userinfo&uiprop=blockinfo%7Cgroups%7Crights%7Chasmsg'
response = client.get(url)
print(response.json())
except TokenExpiredError as e:
print("TokenExpiredError")
Howto use Pywikibot with Django Oauth credentials
- Documentation: mw:Manual:Pywikibot/OAuth/Wikimedia.
- Note Pywikibot supports only OAuth 1.0a. OAuth 2.0 is not supported.
Update user_profile/views.py
from django.contrib.auth.decorators import login_required
from social_django.models import UserSocialAuth
from oauth_app.settings import SOCIAL_AUTH_MEDIAWIKI_KEY, SOCIAL_AUTH_MEDIAWIKI_SECRET
import pywikibot
@login_required()
def profile(request):
user = request.user
try:
# Retrieve the social auth instance for the user
social_auth = user.social_auth.get(provider='mediawiki') # Or your specific provider
# Access token and secret
access_token = social_auth.extra_data['access_token'].get('oauth_token')
access_secret = social_auth.extra_data['access_token'].get('oauth_token_secret')
pywikibot.config.usernames['commons']['beta'] = user.username
authenticate = (SOCIAL_AUTH_MEDIAWIKI_KEY, SOCIAL_AUTH_MEDIAWIKI_SECRET, access_token, access_secret)
pywikibot.config.authenticate['commons.wikimedia.beta.wmflabs.org'] = authenticate
site = pywikibot.Site('beta', 'commons')
site.login()
except UserSocialAuth.DoesNotExist:
# Handle the case where the user doesn't have a social auth instance
pass
context = {'pywikibot_user':str(site.user()) }
return render(request, 'user_profile/profile.dtl', context)
And to user_profile/profile.dtl:
<!DOCTYPE html>
<html>
<body>
<h1>Profile</h1>
<ul>
<li>Django user: {{ user }}</li>
<li>Pywikibot user: {{ pywikibot_user }}</li>
</ul>
</body>
</html>
Links
- https://meta.wikimedia.org/wiki/Special:OAuthConsumerRegistration/propose
- meta.wikimedia.beta.wmflabs.org/wiki/Special:OAuthConsumerRegistration/propose is for testing environment
Tutorials
Code
Further Reading
- https://github.com/python-social-auth/social-app-django
- Pull request to social core adding Mediawiki backend
- Kubernetes Setup for Django
Other tools
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect or the bridged Telegram group
- Discuss via email after you have subscribed to the cloud@ mailing list
- Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
- Read the News wiki page
Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)