Jump to content

Help:Toolforge/experimentation:deploying temporary git branch with push-to-deploy

From Wikitech

This tutorial demonstrates how to temporarily deploy code from a non-default Git branch when using Toolforge Components Service and Push-to-Deploy.

This is useful when you want to experiment with a change on your Toolforge tool before merging it into the default branch.

This workflow temporarily replaces the currently deployed version of your tool. It does not create a separate preview or staging deployment. If the production version must remain available while you experiment, use a separate Toolforge tool for testing.

Before you begin

This tutorial assumes that:

  • your tool is already deployed using Components Service and Push-to-Deploy
  • your tool configuration is stored in toolforge.yaml in a Git repository
  • source_url points to that configuration
  • you are familiar with Git and the basic Push-to-Deploy workflow

If you are new to Push-to-Deploy, first complete My first Python tool or another introductory Push-to-Deploy tutorial.

This tutorial uses the sample-python-ptd-app repository as an example.

Understand the workflow

Push-to-Deploy normally deploys changes from the default branch of your repository.

There is currently no dedicated workflow for temporarily deploying an arbitrary feature branch without changing the tool configuration. Deploying an experimental branch therefore requires temporarily pointing the Components configuration at that branch and manually creating the deployment.

The workflow is:

Create test-branch
        ↓
Configure test-branch to build itself
        ↓
Temporarily point main's source_url
at test-branch/toolforge.yaml
        ↓
Load the configuration into Components
        ↓
Create a deployment
        ↓
Experiment on test-branch
        ↓
Restore the configuration to main

There are two Git references involved:

  • source_url tells Components Service where to retrieve toolforge.yaml.
  • build.ref tells Build Service which Git ref to build.

For a temporary branch deployment, both ultimately need to point to the experimental branch.

Start from the production configuration

The example tool normally deploys from main.

Its toolforge.yaml contains:

toolforge.yaml
# yaml-language-server: $schema=https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/raw/main/openapi/tool-config-schema.json
source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml
components:
  webservice:
    build:
      repository: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app
      ref: main
    run:
      command: web
      publish: /
      port: 8000

The repository also contains:

.gitlab-ci.yml
include:
  - project: "repos/cloud/cicd/gitlab-ci"
    file: "toolforge-cd/deploy-to-toolforge.yaml"

The shared CI configuration handles the normal Push-to-Deploy workflow from the default branch.

You do not need to modify .gitlab-ci.yml for the temporary branch workflow in this tutorial.

Create an experimental branch

Create a branch for the experiment:

laptop:~$ git switch -c test-branch

Make a small, visible change to the application so that you can easily identify the experimental version.

For example, change the page to display:

Hello from the test branch

Configure the experimental branch

The copy of toolforge.yaml on test-branch needs to point to the branch itself.

Change source_url from:

source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml

to:

source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/test-branch/toolforge.yaml

Also change:

ref: main

to:

ref: test-branch

The complete configuration on test-branch should now look like:

toolforge.yaml
# yaml-language-server: $schema=https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/raw/main/openapi/tool-config-schema.json
source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/test-branch/toolforge.yaml
components:
  webservice:
    build:
      repository: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app
      ref: test-branch
    run:
      command: web
      publish: /
      port: 8000

Commit and push the branch:

laptop:~$ git add .
laptop:~$ git commit -m "Test experimental branch"
laptop:~$ git push -u origin test-branch

Pushing test-branch does not by itself deploy it using the standard Push-to-Deploy configuration.

Point the production configuration to the experimental branch

Switch back to main:

laptop:~$ git switch main
$ git pull

In main's copy of toolforge.yaml, change only source_url:

source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/test-branch/toolforge.yaml

Leave build.ref in the main copy as main.

Commit and push this temporary change:

laptop:~$ git add toolforge.yaml
laptop:~$ git commit -m "Temporarily point deployment to test branch"
laptop:~$ git push

At this point, the Git repository points to the experimental configuration, but the Components configuration still needs to be updated.

Load the experimental configuration

Load main's configuration into Components Service:

myuser@tools-bastion-15:~$ curl 'https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml?ref_type=heads' | toolforge components config create

Components follows the source_url in that file and retrieves toolforge.yaml from test-branch.

Check the resulting configuration:

myuser@tools-bastion-15:~$ toolforge components config show

The stored configuration should now have both:

source_url → test-branch/toolforge.yaml
build.ref  → test-branch

The configuration relationship is:

main/toolforge.yaml
        ↓ source_url
test-branch/toolforge.yaml
        ↓ build.ref
test-branch application code

Deploy the experimental branch

Create a deployment:

myuser@tools-bastion-15:~$ toolforge components deployment create

Check the deployment:

myuser@tools-bastion-15:~$ toolforge components deployment show

You can also inspect the builds:

myuser@tools-bastion-15:~$ toolforge build list

The new build should show test-branch as its Git ref.

Visit the tool:

https://sample-python-ptd-app.toolforge.org/

You should now see the experimental change.

The experimental branch has replaced the version of the tool that was previously running. It is not running alongside the main version.

Continue experimenting

Once Components Service is configured to use test-branch, subsequent experiments are simpler.

Switch to the branch if necessary:

laptop:~$ git switch test-branch

Make another application change, commit it, and push:

laptop:~$ git add .
laptop:~$ git commit -m "Update experiment"
laptop:~$ git push

You do not need to modify main or reload the Components configuration again.

Create another deployment:

myuser@tools-bastion-15:~$ toolforge components deployment create

Components Service follows the existing source_url, retrieves the branch configuration, and Build Service builds the latest version of test-branch.

The experimentation loop is therefore:

edit test-branch
        ↓
git push
        ↓
toolforge components deployment create
        ↓
test
        ↓
make another change
        ↺

Repeat this process for as many experiments as necessary.

Restore the default branch

When you finish experimenting, restore the tool to main.

Switch to main:

laptop:~$ git switch main
laptop:~$ git pull

Change source_url back to:

source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml

The complete production configuration should again look like:

toolforge.yaml
# yaml-language-server: $schema=https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/raw/main/openapi/tool-config-schema.json
source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml
components:
  webservice:
    build:
      repository: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app
      ref: main
    run:
      command: web
      publish: /
      port: 8000

Commit and push:

laptop:~$ git add toolforge.yaml
laptop:~$ git commit -m "Restore deployment to main"
laptop:~$ git push

Reload the production configuration:

myuser@tools-bastion-15:~$ curl 'https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml?ref_type=heads' | toolforge components config create

Verify it:

myuser@tools-bastion-15:~$ toolforge components config show

Both settings should now point to main:

source_url → main/toolforge.yaml
build.ref  → main

Create the deployment:

myuser@tools-bastion-15:~$ toolforge components deployment create

Then verify it:

myuser@tools-bastion-15:~$ toolforge components deployment show
myuser@tools-bastion-15:~$ toolforge build list

Visit the tool again and confirm that the normal main version is running.

Normal Push-to-Deploy operation can now continue from main.

Merge or discard the experiment

If the experiment was successful, you can merge the application changes from test-branch into main.

Before merging, make sure that the temporary branch-specific configuration changes are not accidentally carried into the production configuration.

In particular, production should continue to use:

source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml

and:

ref: main

If the experiment is no longer needed, you can delete test-branch after confirming that the main deployment has been restored.

Understand the two references

The temporary deployment requires changing two Git references because they have different purposes.

Setting Purpose Normal deployment Temporary deployment
source_url Tells Components Service where to retrieve toolforge.yaml main/toolforge.yaml test-branch/toolforge.yaml
build.ref Tells Build Service which Git ref to build main test-branch

The complete workflow is:

main
                ↓
       create test-branch
                ↓
      make experimental change
                ↓
 configure test-branch/toolforge.yaml
                ↓
        push test-branch
                ↓
 temporarily change main/source_url
                ↓
            push main
                ↓
      load Components config
                ↓
       create deployment
                ↓
        test experiment
                ↓
        ┌───────┴────────┐
        │                │
  more experiments    finished
        │                │
 edit test-branch        ↓
        ↓          restore main
      push               ↓
        ↓          load main config
deployment create        ↓
        │          deployment create
        └────↺           ↓
                  normal deployment

Current limitations

This workflow is a workaround for deploying an experimental branch with the current Components Service and Push-to-Deploy workflow.

Keep the following limitations in mind:

  • there is no separate preview deployment; the experimental branch replaces the currently running version of the tool
  • the standard Push-to-Deploy configuration does not automatically deploy arbitrary experimental branches
  • initially switching to the experimental branch requires changing the configuration on main and manually loading it into Components Service
  • deployments while experimenting are created manually
  • returning to main requires restoring and reloading the production configuration

For experiments that must run independently of the production tool, consider creating a separate Toolforge tool for testing.

What you learned

In this tutorial, you:

  • created an experimental Git branch
  • configured Build Service to build the experimental branch
  • temporarily directed Components Service to use the branch's toolforge.yaml
  • manually deployed the experimental version
  • iterated on the experimental branch without repeatedly changing the configuration
  • restored the tool to main
  • learned the different roles of source_url and build.ref

See also

Communication and support

Support and administration of the WMCS resources is provided by Wikimedia Foundation staff and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
  • Chat in real time in the IRC channel #wikimedia-cloud connect or the bridged Telegram group
  • Discuss via email after you have subscribed to the cloud@ mailing list
Stay aware of critical changes and plans
Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read posts about Cloud Services on the Wikimedia Technical Blog. (Older posts can be read from the Cloud Services Blog.)