Jump to content

Help:Toolforge/Auto-update a tool from GitHub/GitLab

From Wikitech
Note: Use of this deployment method has risks. If your GitHub repository is compromised by a 3rd party, your live tool could be impacted. Please think carefully before proceeding.

Overview

You can use GitHub's webhooks feature to automatically deploy any changes made on GitHub to Toolforge. With this setup, whenever a push is made to the repository on GitHub, the changes will be fetched to the Toolforge clone of the repository, without having to manually log in to the tool's account and running git pull.

Using webhooks with GitHub

Introductory-level information about using webhooks with GitHub can be found here:

PHP tool hosted in $HOME/public_html

First, set up a webhook listener that runs "git pull" whenever it's accessed from GitHub:

  1. ssh USER@login.toolforge.org
  2. become MYTOOL
  3. echo "<?php if(isset($_SERVER['HTTP_X_GITHUB_EVENT'])) { `git pull`; } ?>" > public_html/git-pull.php
    • Hint You can add more commands after git pull, e.g. to run a build script.
    • Info You can find useful information regarding running shell commands from PHP at php.net and at stackoverflow.com.

Then, set up the webhook emitter on GitHub's side:

  1. Go to https://github.com/USER/REPO/settings/hooks and click the "Add webhook" button
  2. In the "URL" field, type https://MYTOOL.toolforge.org/git-pull.php
  3. Save the hook (the remaining fields can be kept with the default values)

That's it! Now whenever you push commits to the repo on GitHub, the Toolforge clone will be automatically updated.

PHP tool hosted in $HOME/www/static

First, set up a webhook listener (in the non-static $HOME/public_html directory) that runs "git pull" whenever it's accessed from GitHub:

  1. ssh USER@login.toolforge.org
  2. become MYTOOL
  3. mkdir ~/public_html
  4. echo "<?php if(isset($_SERVER['HTTP_X_GITHUB_EVENT'])) { `git -C ../www/static pull`; } ?>" > public_html/git-pull.php
  5. webservice start

Then, set up the webhook emitter on GitHub's side:

  1. Go to https://github.com/USER/REPO/settings/hooks and click the "Add webhook" button
  2. In the "Payload URL" field, type https://MYTOOL.toolforge.org/git-pull.php
  3. Save the hook (the remaining fields can be kept with the default values)

That's it! Now whenever you push commits to the repo on GitHub, the Toolforge clone will be automatically updated.

Python tool hosted in $HOME/www/python/src

Given that we have a repository that has the code under the $HOME/www/python/src dir, we setup an endpoint to listen on the url https://MYTOOL.toolforge.org/update-server

  • if we have a Flask app we can add the following endpoint to our main file
@app.route("/update-server", methods=["POST"])
def webhook():
    if request.method == "POST":
        subprocess.check_output(["git", "pull", "origin", "main"])
        return "Updated Toolforge project successfully", 200
    else:
        return "Wrong event type", 400
  • if we have a FastAPI app then we add the following:
@app.post("/update-server")
async def webhook():
    subprocess.check_output(["git", "pull", "origin", "main"])
    return "Updated Toolforge project successfully"

In order to create the webhook we follow the same procedure with the PHP app but we put https://MYTOOL.toolforge.org/update-server as the webhook url.

Using GitHub Actions

First, you need to set your username and your private key as a secret on GitHub Actions. Be aware that this will allow any users that manage your GitHub repository to have access to other tools that your Toolforge account is a developer of.

Then, you have to modify that you would normally enter when manually pushing changes to your tool, to run in a non-interactive way instead. For example, instead of:

  • become mytool
  • cd www
  • git pull origin main

...use:

  • become mytool git -C '$HOME/www' pull origin main

This syntax relies on the feature of the become command to log in and immediately execute a command and its arguments, as described by the output of become --help: usage: become <toolname> [command [args...]].

Make sure to quote environment variables that you want to be interpreted within Toolforge rather than by the GitHub Actions runner — like $HOME in the example above.

As another example, if you need to access the shell to migrate your Django databases, then instead of:

  • toolforge webservice --backend=kubernetes python3.11 shell
  • source $HOME/www/python/venv/bin/activate
  • python manage.py migrate

...use:

  • become mytool webservice python3.11 shell -- '$HOME/www/python/venv/bin/python' '$HOME/www/python/src/manage.py' migrate

Finally, create the file .github/workflows/deploy.yml in your GitHub repository with the following contents, adjusting the lines indicated with numbered comments as appropriate: 

name: Deploy via SSH # (1) Title of the action

on:
  push:
    branches:
      - main # (2) Name of the branch where the action should run

jobs:
  deploy:
    name: Deploy to SSH Server # (3) Title of the job
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Run deployment script
        uses: appleboy/ssh-action@v1.0.3
        with:
          host: login.toolforge.org
          username: ${{ secrets.USERNAME }}
          key: ${{ secrets.KEY }}
          port: 22
          request_pty: true
          script_stop: true
          # (4) Actual command(s) to run
          script: |
            become mytool some_command args...
            become mytool some_command args...

Keep in mind that this GitHub Actions script uses the appleboy/ssh-action action, which is an external dependency and could have security breaches or stop working at any time.

GitLab webhook

In order to set up the webhook on a GitLab repository you follow the same steps on the application side as for GitHub, but for the php app you will replace HTTP_X_GITHUB_EVENT with HTTP_X_GITLAB_EVENT .

Then navigate to your GitLab project and got to Settings -> Webhooks and put the url and tick the "Push events" trigger and type the name of your main branch under the "wilcard pattern" field e.g. main, master etc. and then click the "Add webhook" button.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Stay aware of critical changes and plans
Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Auto-update_a_tool_from_GitHub/GitLab&oldid=2249678"