Jump to content

Help:Toolforge/My first Python tool

From Wikitech

My first Python tool

Overview

Python webservices are used by many existing tools on Toolforge. Python is a general-purpose programming language commonly used for web development, automation, data processing, and many other tasks.

This tutorial is designed to get a sample Python application deployed to Toolforge using Push-to-Deploy as quickly as possible. The application uses Flask and runs on Toolforge using the Build Service.

Then we'll use the database credentials provided by Toolforge to extract the list of available databases from the Wiki Replicas.

The guide will teach you how to:

  • Create a new tool
  • Run a Python Flask webservice on Kubernetes
  • Deploy the application using Push-to-Deploy
  • Connect to the Wiki Replicas from Python
  • Run a query and extract information

Getting started

Prerequisites

Skills

Accounts

Step-by-step guide

Step 1: Create a new tool account

  1. Follow the Toolforge quickstart guide to create a Toolforge tool and SSH into Toolforge.
    • For the examples in this tutorial, sample-python-ptd-app is used to indicate places where your unique tool name is used in another command.
  2. Make sure to create a Git repository for the tool. You can get one like this:
    1. Log into the Toolforge admin page.
    2. Select your tool.
    3. On the left side panel, under Git repositories, click create repository.
    4. Copy the URL in the Clone section.
      1. There is a private URL that we will use to clone the repository locally, starting with git: git@gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app.git
      2. There is also a public URL that Toolforge will use to build the application, starting with https: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app.git

Step 2: Create a basic Python Flask webservice

What is Flask?

Flask is a lightweight Python web framework that can be used to create web applications and APIs.

How to create a basic Python Flask webservice

Clone your tool Git repository

You will have to clone the tool repository to be able to add code to it. On your local computer, with Git installed, run:

laptop:~$ git clone git@gitlab.wikimedia.org:toolforge-repos/sample-python-ptd-app.git
laptop:~$ cd sample-python-ptd-app

That will create a folder called sample-python-ptd-app. We are going to put the code in that folder.

Add the Python dependencies

Use a file named requirements.txt to keep track of the Python dependencies required by the application.

Create requirements.txt:

laptop:~/sample-python-ptd-app$ cat > requirements.txt << EOF
Flask
gunicorn
PyMySQL
EOF

Create a "Hello World!" Python application

Create a file named app.py:

app.py
from flask import Flask

app = Flask(__name__)

@app.route("/")
def index():
    return "<p>Hello World!</p>"
Code on Toolforge must always be licensed under an Open Source Initiative (OSI) approved license. See Right to fork policy for more information on this Toolforge policy.

Create the Procfile

The Procfile defines the command that the Build Service should use to start the web application.

Create it with:

laptop:~/sample-python-ptd-app$ cat > Procfile << EOF
web: gunicorn --bind 0.0.0.0:\$PORT app:app
EOF

The web entry defines the command used to start the Flask application. Gunicorn runs the application and listens on the port provided by Toolforge through the PORT environment variable.

Create the Toolforge configuration file

Create toolforge.yaml:

laptop:~/sample-python-ptd-app$ cat > toolforge.yaml << EOF
# yaml-language-server: \$schema=https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/raw/main/openapi/tool-config-schema.json
source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml?ref_type=heads
components:
  webservice:
    build:
      repository: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app
      ref: main
    run:
      command: web
      publish: /
      port: 8000
EOF

If you don't want your webservice to be reachable from the internet, remove the publish setting. If you are deploying a process without a port, you can remove port too. See Deploy your tool for more information.

Commit your changes and push

laptop:~/sample-python-ptd-app$ git add .
laptop:~/sample-python-ptd-app$ git commit -m "First commit"
laptop:~/sample-python-ptd-app$ git push origin main

Initialize the tool configuration

Now SSH to login.toolforge.org and become your tool:

laptop:~/sample-python-ptd-app$ ssh login.toolforge.org
user@tools-sgebastion-XX:~$ become sample-python-ptd-app

Create the tool configuration using the public copy of toolforge.yaml:

tools.sample-python-ptd-app@tools-sgebastion-XX:~$ curl 'https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml?ref_type=heads' | toolforge components config create

You should see a message indicating that the configuration was updated successfully.

Push-to-Deploy is currently a beta feature of Toolforge, so the command may also display a warning indicating that you are using a beta feature.

Create the first deployment

Create a deployment:

tools.sample-python-ptd-app@tools-sgebastion-XX:~$ toolforge components deployment create

Push-to-Deploy will build the application from your Git repository and deploy the webservice.

The repository in toolforge.yaml must be publicly accessible so that Toolforge can clone and build it.

Wait for the deployment to finish

Check the deployment status:

tools.sample-python-ptd-app@tools-sgebastion-XX:~$ toolforge components deployment show

Wait for the deployment status to be ok(Succeeded).

Once the deployment has finished, navigate to:

https://sample-python-ptd-app.toolforge.org

You should see:

Hello World!

It might take a couple of minutes for the application to become reachable.

Automate deployment on push

You can follow this guide to configure and automate deployment of your tool whenever you push or merge changes to the main branch.

Connecting to the replicas

Toolforge provides database credentials to your application through environment variables. We will use these credentials to connect to the special meta_p database and retrieve the names of the available Wiki Replica databases.

First, update requirements.txt if PyMySQL is not already present:

Flask
gunicorn
PyMySQL

Now update app.py:

Sample flask code
app.py
from flask import Flask
import os
import pymysql

app = Flask(__name__)

def get_db_connection():
    return pymysql.connect(
    host="meta.web.db.svc.wikimedia.cloud",
    user=os.environ["TOOL_REPLICA_USER"],
    password=os.environ["TOOL_REPLICA_PASSWORD"],
    database="meta_p",
    cursorclass=pymysql.cursors.DictCursor,
)

def get_db_names():
    connection = get_db_connection()

    try:
        with connection.cursor() as cursor:
           cursor.execute("SELECT dbname FROM wiki;")
           return cursor.fetchall()
    finally:
        connection.close()


@app.route("/")
def index():
    databases = get_db_names()

    rows = "".join(
       f"<tr><td>{row['dbname']}</td></tr>"
       for row in databases
    )

    return f"""
       <!doctype html>
       <html lang="en">
       <head>
          <meta charset="UTF-8">
          <title>Database names</title>
       </head>
       <body>
       <p>
           Sample Python application accessing the
           <a href="https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database">
            Wiki Replicas
           </a>
           special <code>meta_p</code> database.
       </p>

       <table>
           <thead>
               <tr>
                   <th>Database name</th>
               </tr>
           </thead>
           <tbody>
               {rows}
           </tbody>
       </table>
       </body>
       </html>
   """

The application connects to meta.web.db.svc.wikimedia.cloud using the TOOL_REPLICA_USER and TOOL_REPLICA_PASSWORD environment variables provided by Toolforge.

The query:

SELECT dbname FROM wiki;

retrieves the database names from the special meta_p database.

Commit, push, and redeploy

Commit and push your changes:

laptop:~/sample-python-ptd-app$ git add .
laptop:~/sample-python-ptd-app$ git commit -m "Query meta_p database"
laptop:~/sample-python-ptd-app$ git push origin main

Then create a new deployment:

laptop:~/sample-python-ptd-app$ ssh login.toolforge.org
user@tools-sgebastion-XX:~$ become sample-python-ptd-app
tools.sample-python-ptd-app@tools-sgebastion-XX:~$ toolforge components deployment create
tools.sample-python-ptd-app@tools-sgebastion-XX:~$ toolforge components deployment show

Wait for the deployment to finish and refresh the application in your browser.

You should now see a list of Wiki Replica database names.

Configure Push-to-Deploy

Push-to-Deploy allows a CI runner to trigger a Components Service deployment after changes are pushed to your Git repository.

Below are the steps for a basic configuration. Visit Triggering a deployment from your CI runner for more details.


Configure Push-to-Deploy

To enable Push-to-Deploy, configure your Git repository's CI system to trigger a Components deployment whenever changes are pushed to the repository.

The following is a minimal example for Wikimedia GitLab. For additional options and details, see Triggering a deployment from your CI runner.

Create a deploy token

While logged in as your tool account, create a deploy token:

$ toolforge components deploy-token create

The command displays a token. Treat this token as a secret. Do not commit it to your repository.

Add the deploy token to GitLab

In your GitLab repository, go to:

Settings → CI/CD → Variables

Create a variable named:

TOOLS_DEPLOY_TOKEN

Set its value to the deploy token created in the previous step.

For security, make the variable hidden, protected, and masked where appropriate.

Add a GitLab CI configuration

Create a .gitlab-ci.yml file in the repository containing:

.gitlab-ci.yml
include:
  - project: "repos/cloud/cicd/gitlab-ci"
    file: "toolforge-cd/deploy-to-toolforge.yaml"

The included Toolforge CI configuration handles triggering the Components deployment.

This minimal configuration assumes that the GitLab project name and Toolforge tool name are the same. If they are different, or if you need to customize the deployment process, see Triggering a deployment from your CI runner.

Commit and push .gitlab-ci.yml to the repository.


When changes are pushed to the repository, GitLab CI will use the deploy token to trigger a Components deployment.

At this point, Push-to-Deploy is configured.

Test Push-to-Deploy

After configuring your CI runner, make a small change to app.py.

For example, change the title from `Database names` to `All Database Names:

app.py
<title>All Database Names</title>

Commit and push the change:

$ git add app.py
$ git commit -m "Expand database name"
$ git push

Do not manually create a deployment.

The push should trigger your CI runner, which creates a Components Service deployment automatically.

Check your repository's CI pipeline and verify that it completes successfully.

You can then inspect the deployment from Toolforge:

$ toolforge components deployment list

For the scheduled example, verify that the new schedule has been applied:

$ toolforge jobs list -o long

If the deployment was triggered automatically and the new change was applied, Push-to-Deploy is working.

Notes

You can see the code used in this example in the sample-python-ptd-app repository.

Troubleshooting

See Toolforge Build Service troubleshooting.

You can also inspect the status of your Push-to-Deploy deployments with the following:

toolforge components deployment list
toolforge components deployment show
toolforge jobs logs webservice #view logs

See also

Communication and support

Support and administration of the WMCS resources is provided by Wikimedia Foundation staff and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support

  • Chat in real time in the IRC channel #wikimedia-cloud or the bridged Telegram group.
  • Discuss via email after you have subscribed to the cloud@ mailing list.

Stay aware of critical changes and plans

Track work tasks and report bugs

Use a subproject of the Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself.

Communication and support

Support and administration of the WMCS resources is provided by Wikimedia Foundation staff and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
  • Chat in real time in the IRC channel #wikimedia-cloud connect or the bridged Telegram group
  • Discuss via email after you have subscribed to the cloud@ mailing list
Stay aware of critical changes and plans
Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read posts about Cloud Services on the Wikimedia Technical Blog. (Older posts can be read from the Cloud Services Blog.)

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/My_first_Python_tool&oldid=2464385"