Help:Toolforge/Auto-update a tool from GitHub/GitLab

From Wikitech
Jump to navigation Jump to search

Overview

You can use GitHub's webhooks feature to automatically deploy any changes made on GitHub to Toolforge. With this setup, whenever a push is made to the repository on GitHub, the changes will be fetched to the Toolforge clone of the repository, without having to manually log in to the tool's account and running git pull.

Using webhooks with Github

Introductory-level information about using webhooks with GitHub can be found here:

PHP tool hosted in $HOME/public_html

First, set up a webhook listener that runs "git pull" whenever it's accessed from GitHub:

  1. ssh USER@login.toolforge.org
  2. become MYTOOL
  3. echo "<?php if(isset($_SERVER['HTTP_X_GITHUB_EVENT'])) { `git pull`; } ?>" > public_html/git-pull.php
    • Hint You can add more commands after git pull, e.g. to run a build script.
    • Info You can find useful information regarding running shell commands from PHP here and here.

Then, set up the webhook emitter on GitHub's side:

  1. Go to https://github.com/USER/REPO/settings/hooks and click the "Add webhook" button
  2. In the "Payload URL" field, type https://MYTOOL.toolforge.org/git-pull.php
  3. Save the hook (the remaining fields can be kept with the default values)

That's it! Now whenever you push commits to the repo on GitHub, the Toolforge clone will be automatically updated.

PHP tool hosted in $HOME/www/static

First, set up a webhook listener (in the non-static $HOME/public_html directory) that runs "git pull" whenever it's accessed from GitHub:

  1. ssh USER@login.toolforge.org
  2. become MYTOOL
  3. mkdir ~/public_html
  4. echo "<?php if(isset($_SERVER['HTTP_X_GITHUB_EVENT'])) { `git -C ../www/static pull`; } ?>" > public_html/git-pull.php
  5. webservice start

Then, set up the webhook emitter on GitHub's side:

  1. Go to https://github.com/USER/REPO/settings/hooks and click the "Add webhook" button
  2. In the "Payload URL" field, type https://MYTOOL.toolforge.org/git-pull.php
  3. Save the hook (the remaining fields can be kept with the default values)

That's it! Now whenever you push commits to the repo on GitHub, the Toolforge clone will be automatically updated.

Python tool hosted in $HOME/www/python/src

Given that we have a repository that has the code under the $HOME/www/python/src dir, we setup an endpoint to listen on the url https://MYTOOL.toolforge.org/update-server

  • if we have a Flask app we can add the following endpoint to our main file
@app.route("/update-server", methods=["POST"])
def webhook():
    if request.method == "POST":
        subprocess.check_output(["git", "pull", "origin", "main"])
        return "Updated Toolforge project successfully", 200
    else:
        return "Wrong event type", 400
  • if we have a FastAPI app then we add the following:
@app.post("/update-server")
async def webhook():
    subprocess.check_output(["git", "pull", "origin", "main"])
    return "Updated Toolforge project successfully"

In order to create the webhook we follow the same procedure with the PHP app but we put https://MYTOOL.toolforge.org/update-server as the webhook url.

GitLab webhook

In order to set up the webhook on a GitLab repository you follow the same steps on the application side as for GitHub, but for the php app you will replace HTTP_X_GITHUB_EVENT with HTTP_X_GITLAB_EVENT .

Then navigate to your GitLab project and got to Settings -> Webhooks and put the url and tick the "Push events" trigger and type the name of your main branch under the "wilcard pattern" field e.g. main, master etc. and then click the "Add webhook" button.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia Movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Receive mail announcements about critical changes
Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
Track work tasks and report bugs
Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself
Learn about major near-term plans
Read the News wiki page
Read news and stories about Wikimedia Cloud Services
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Auto-update_a_tool_from_GitHub/GitLab&oldid=2052335"