Help:Toolforge/experimentation:deploying temporary git branch with push-to-deploy
This tutorial demonstrates how to temporarily deploy code from a non-default Git branch when using Toolforge Components Service and Push-to-Deploy.
This is useful when you want to experiment with a change on your Toolforge tool before merging it into the default branch.
Before you begin
This tutorial assumes that:
- your tool is already deployed using Components Service and Push-to-Deploy
- your tool configuration is stored in
toolforge.yamlin a Git repository source_urlpoints to that configuration- you are familiar with Git and the basic Push-to-Deploy workflow
If you are new to Push-to-Deploy, first complete My first Python tool or another introductory Push-to-Deploy tutorial.
This tutorial uses the sample-python-ptd-app repository as an example.
Understand the workflow
Push-to-Deploy normally deploys changes from the default branch of your repository.
There is currently no dedicated workflow for temporarily deploying an arbitrary feature branch without changing the tool configuration. Deploying an experimental branch therefore requires temporarily pointing the Components configuration at that branch and manually creating the deployment.
The workflow is:
Create test-branch
↓
Configure test-branch to build itself
↓
Temporarily point main's source_url
at test-branch/toolforge.yaml
↓
Load the configuration into Components
↓
Create a deployment
↓
Experiment on test-branch
↓
Restore the configuration to main
There are two Git references involved:
source_urltells Components Service where to retrievetoolforge.yaml.build.reftells Build Service which Git ref to build.
For a temporary branch deployment, both ultimately need to point to the experimental branch.
Start from the production configuration
The example tool normally deploys from main.
Its toolforge.yaml contains:
# yaml-language-server: $schema=https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/raw/main/openapi/tool-config-schema.json
source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml
components:
webservice:
build:
repository: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app
ref: main
run:
command: web
publish: /
port: 8000
The repository also contains:
include:
- project: "repos/cloud/cicd/gitlab-ci"
file: "toolforge-cd/deploy-to-toolforge.yaml"
The shared CI configuration handles the normal Push-to-Deploy workflow from the default branch.
You do not need to modify .gitlab-ci.yml for the temporary branch workflow in this tutorial.
Create an experimental branch
Create a branch for the experiment:
laptop:~$ git switch -c test-branch
Make a small, visible change to the application so that you can easily identify the experimental version.
For example, change the page to display:
Hello from the test branch
Configure the experimental branch
The copy of toolforge.yaml on test-branch needs to point to the branch itself.
Change source_url from:
source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml
to:
source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/test-branch/toolforge.yaml
Also change:
ref: main
to:
ref: test-branch
The complete configuration on test-branch should now look like:
# yaml-language-server: $schema=https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/raw/main/openapi/tool-config-schema.json
source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/test-branch/toolforge.yaml
components:
webservice:
build:
repository: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app
ref: test-branch
run:
command: web
publish: /
port: 8000
Commit and push the branch:
laptop:~$ git add .
laptop:~$ git commit -m "Test experimental branch"
laptop:~$ git push -u origin test-branch
Pushing test-branch does not by itself deploy it using the standard Push-to-Deploy configuration.
Point the production configuration to the experimental branch
Switch back to main:
laptop:~$ git switch main
$ git pull
In main's copy of toolforge.yaml, change only source_url:
source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/test-branch/toolforge.yaml
Leave build.ref in the main copy as main.
Commit and push this temporary change:
laptop:~$ git add toolforge.yaml
laptop:~$ git commit -m "Temporarily point deployment to test branch"
laptop:~$ git push
At this point, the Git repository points to the experimental configuration, but the Components configuration still needs to be updated.
Load the experimental configuration
Load main's configuration into Components Service:
myuser@tools-bastion-15:~$ curl 'https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml?ref_type=heads' | toolforge components config create
Components follows the source_url in that file and retrieves toolforge.yaml from test-branch.
Check the resulting configuration:
myuser@tools-bastion-15:~$ toolforge components config show
The stored configuration should now have both:
source_url → test-branch/toolforge.yaml
build.ref → test-branch
The configuration relationship is:
main/toolforge.yaml
↓ source_url
test-branch/toolforge.yaml
↓ build.ref
test-branch application code
Deploy the experimental branch
Create a deployment:
myuser@tools-bastion-15:~$ toolforge components deployment create
Check the deployment:
myuser@tools-bastion-15:~$ toolforge components deployment show
You can also inspect the builds:
myuser@tools-bastion-15:~$ toolforge build list
The new build should show test-branch as its Git ref.
Visit the tool:
https://sample-python-ptd-app.toolforge.org/
You should now see the experimental change.
main version.
Continue experimenting
Once Components Service is configured to use test-branch, subsequent experiments are simpler.
Switch to the branch if necessary:
laptop:~$ git switch test-branch
Make another application change, commit it, and push:
laptop:~$ git add .
laptop:~$ git commit -m "Update experiment"
laptop:~$ git push
You do not need to modify main or reload the Components configuration again.
Create another deployment:
myuser@tools-bastion-15:~$ toolforge components deployment create
Components Service follows the existing source_url, retrieves the branch configuration, and Build Service builds the latest version of test-branch.
The experimentation loop is therefore:
edit test-branch
↓
git push
↓
toolforge components deployment create
↓
test
↓
make another change
↺
Repeat this process for as many experiments as necessary.
Restore the default branch
When you finish experimenting, restore the tool to main.
Switch to main:
laptop:~$ git switch main
laptop:~$ git pull
Change source_url back to:
source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml
The complete production configuration should again look like:
# yaml-language-server: $schema=https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/raw/main/openapi/tool-config-schema.json
source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml
components:
webservice:
build:
repository: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app
ref: main
run:
command: web
publish: /
port: 8000
Commit and push:
laptop:~$ git add toolforge.yaml
laptop:~$ git commit -m "Restore deployment to main"
laptop:~$ git push
Reload the production configuration:
myuser@tools-bastion-15:~$ curl 'https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml?ref_type=heads' | toolforge components config create
Verify it:
myuser@tools-bastion-15:~$ toolforge components config show
Both settings should now point to main:
source_url → main/toolforge.yaml
build.ref → main
Create the deployment:
myuser@tools-bastion-15:~$ toolforge components deployment create
Then verify it:
myuser@tools-bastion-15:~$ toolforge components deployment show
myuser@tools-bastion-15:~$ toolforge build list
Visit the tool again and confirm that the normal main version is running.
Normal Push-to-Deploy operation can now continue from main.
Merge or discard the experiment
If the experiment was successful, you can merge the application changes from test-branch into main.
Before merging, make sure that the temporary branch-specific configuration changes are not accidentally carried into the production configuration.
In particular, production should continue to use:
source_url: https://gitlab.wikimedia.org/toolforge-repos/sample-python-ptd-app/-/raw/main/toolforge.yaml
and:
ref: main
If the experiment is no longer needed, you can delete test-branch after confirming that the main deployment has been restored.
Understand the two references
The temporary deployment requires changing two Git references because they have different purposes.
| Setting | Purpose | Normal deployment | Temporary deployment |
|---|---|---|---|
source_url |
Tells Components Service where to retrieve toolforge.yaml |
main/toolforge.yaml |
test-branch/toolforge.yaml |
build.ref |
Tells Build Service which Git ref to build | main |
test-branch |
The complete workflow is:
main
↓
create test-branch
↓
make experimental change
↓
configure test-branch/toolforge.yaml
↓
push test-branch
↓
temporarily change main/source_url
↓
push main
↓
load Components config
↓
create deployment
↓
test experiment
↓
┌───────┴────────┐
│ │
more experiments finished
│ │
edit test-branch ↓
↓ restore main
push ↓
↓ load main config
deployment create ↓
│ deployment create
└────↺ ↓
normal deployment
Current limitations
This workflow is a workaround for deploying an experimental branch with the current Components Service and Push-to-Deploy workflow.
Keep the following limitations in mind:
- there is no separate preview deployment; the experimental branch replaces the currently running version of the tool
- the standard Push-to-Deploy configuration does not automatically deploy arbitrary experimental branches
- initially switching to the experimental branch requires changing the configuration on
mainand manually loading it into Components Service - deployments while experimenting are created manually
- returning to
mainrequires restoring and reloading the production configuration
For experiments that must run independently of the production tool, consider creating a separate Toolforge tool for testing.
What you learned
In this tutorial, you:
- created an experimental Git branch
- configured Build Service to build the experimental branch
- temporarily directed Components Service to use the branch's
toolforge.yaml - manually deployed the experimental version
- iterated on the experimental branch without repeatedly changing the configuration
- restored the tool to
main - learned the different roles of
source_urlandbuild.ref
See also
Communication and support
Support and administration of the WMCS resources is provided by Wikimedia Foundation staff and Wikimedia movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect or the bridged Telegram group
- Discuss via email after you have subscribed to the cloud@ mailing list
- Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
- Read the News wiki page
Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself
Read posts about Cloud Services on the Wikimedia Technical Blog. (Older posts can be read from the Cloud Services Blog.)
