Wikimedia Cloud Services team/EnhancementProposals/Decision record T339198 Toolforge component deployment flow details

From Wikitech

Origin task: phab:T339198

Date of the decision: 2023-06-30

People in the decision meeting (alphabetical order): No decision meeting, agreement happened in the task.

Decision taken

Option 5 was chosen, this is:

  • 0.0.X-TTTT-HHHH version scheme, where:
    • X is a consecutively increasing number
    • TTTT is a datestamp
    • HHHH is the short commit hash
  • Both image and helm chart will be shipped at the same time
  • They will be shipped automatically on every merge/push to main

Rationale

We don't care about semantic versioning, all we want is version ordinality by code history (X) and time (TTTT), and avoid version clashes (HHHH). Shipping both image and chart simplifies the process as we don't need so strict control of separated releases and automatically publishing them avoid having an extra manual step (there's already two after this, the deploy repo commit and the manual deploy).


Problem

There's an agreed high level flow for toolforge component deployment outlined here:

https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_Kubernetes_component_workflow_improvements

But at implementation time there's some details that need deciding/refining as those showed to be controversial. This is a decision request to decide on those details.


Note that the focus is on the continuous delivery of toolforge components, not of toolforge itself (that might require another discussion).


  • What artifacts do we want to ship? (image, helm chart, etc.)
  • How to version those artifacts
  • When to ship those artifacts


Some assumptions (though feel free to challenge later):

  • We want to do this on gitlab
  • We want to use gitlab ci
  • We want to store the artifacts in harbor
  • The deployment of the artifacts to toolforge (tools or toolsbeta) is handled by a different repo, assuming here that there is or will be human validation at that stage
  • We want to automate the building of the artifacts, even if the trigger to build them is not
  • We publish the artifacts at the same time in tools and toolsbeta (note that this does not mean deployment on toolforge, just publishing the artifacts on harbor)


Constraints and risks

If delivered late:

  • Some increased maintenance on our side until that is done to manually deliver the current components
  • Might become harder to adopt due to the increase of components and lack of prioritization

If not all the full flow is delivered:

  • Possible increased maintenance to build and release components
  • Mix of processes flows might make release of code error-prone (wmcs.toolforge.component.build + ./deploy.sh vs toolforge-deploy + helm chart bump)


Options considered

Partial Options

Note that these three decisions will be dependent on each other, so for a different publish flow you might want a specific versioning scheme and set of artifacts that is different than another publish flow. So take that into account when suggesting solutions.

Versioning scheme

VS Option 1

0.0.X-TTTTT-HHHHH, where:

  • X is the number of commits since the origin of the repository or the previous tag
  • TTTTT is the datestamp (YYYYmmddHHMMSS, ex. 202306151038)
  • HHHHH is the short git hash of the commit used to build

Pros:

  • Identifies the commit it was built on top of
  • Sequential code-wise (if A has more commits than B, A>B)
  • Sequential time-wise (for the same commit, a build today has priority over a build from yesterday)

Cons:

  • No semantic versioning
  • Can't be in the source code itself as it depends on the time + git hash

VS Option 2

A.B.C-TTTTT-HHHHH, where:

  • A.B.C is as semantic version, generated from the git history:
    • Extract the list of commits in historical order:
        • For each commit that has `Sem-Ver: major` bump the major version
        • For each commit that has `Sem-Ver: feature` bump the feature version
        • For any other commit, bump the bug version
  • TTTTT is the datestamp (YYYYmmddHHMMSS, ex. 202306151038)
  • HHHHH is the short git hash of the commit used to build

Pros:

  • Identifies the commit it was built on top of
  • Sequential code-wise (if A has more commits than B, A>B)
  • Sequential time-wise (for the same commit, a build today has priority over a build from yesterday)
  • Semantic versioning from git source (developer flags non-backwards compatible commits at review time)

Cons:

  • Can't be in the source code itself as it depends on the time + git hash
  • The version generation is a bit more complicated

VS Option 3

0.0.0, where:

  • 0.0.0 is a static or hardcoded string in the code

Pros:

  • Manual control of the version
  • Version comes from the code itself, no generation needed

Cons:

  • Needs manual updating
    • You might forget to bump (version does not identify code anymore)
    • You might bump to the wrong version (version does not identify code anymore, no semantic versioning, not sequential)
    • You have to review the history to decide if commits are major/feature or minor if you want to follow semantic versioning

Artifacts

Arts option 1

Helm and image are delivered separately

Pros:

  • Strong control of each artifact
  • No need to rebuild image if only chart changes (in most cases)
  • No need to update chart if only code changes (in most cases)

Cons:

  • More complex release process/flow, harder to automate
  • Need to keep track of two versions, and updating the correct one in the right place
  • Chart will not ensure it works without overriding the image version it comes with by default


Arts option 2

Helm and image are delivered at the same time

Pros:

  • Only one artifact/version to manage
  • The chart will always work by itself (as the image it comes with is built for it)
  • Easier to automate/simpler workflow

Cons:

  • Loose control of when the publish happens as both image and chart are now tied up
  • Sometimes the image rebuild will be done but not needed (as no code changes happened)
  • Sometimes the helm chart rebuild will be done but not needed (as no code chart changes happened)


When to ship

Here "when to ship" means "when to build a container image and the helm chart and push them to harbor".

Shipping Option 1

Ship on manual tag

Pros:

  • Control when the shipping happens
  • Easy to see from the git tags history when a shipping happened

Cons:

  • Might be unable to pin-point which commit was shipped with which tag (as tags are not immutable, they can be moved around, and are local to each git clone)
  • Needs a manual step
  • No peer-review on what is going to be shipped (can't review tag pushes)


Shipping Option 2

Ship on manual tag after version bump commit

Pros:

  • Control when the shipping happens
  • Easy to see from the git tags history when a shipping happened
  • Half-review of what is going to be shipped (at least the commit bump)

Cons:

  • Needs two manual steps (mr to bump + tag)
  • Might be unable to pin-point which commit was shipped with which tag (as tags are not immutable, they can be moved around, and are local to each git clone)
  • Only half-peer-review on what is going to be shipped (can't review tag pushes)

Shipping Option 3

Ship on every merge/push to main

Pros:

  • No manual steps
  • Shipping happens all the time, so hard to pin-point a single one (no tag to flag it, or special commit)
  • Review of what's going to be shipped is the same as code review before merge

Cons:

  • Shipping on every merge of a merge request, even if that version might not be deployed

Shipping Option 4

Ship on version bump manual commit.

The Git tag is a nice addition, and it helps identify things, but is not a hard requirement (or, the workflow doesn't depend on it or break because lack of git tag).

This is similar to what happens with debian packages. The tag mostly works to help backtrace in a short way which version was deployed (can be figured if no git tag anyway with a bit more work).

Pros:

  • Control when the shipping happens
  • Easy to see from the git history when a shipping happened

Cons:

  • One manual step (mr to bump)

Options

Option 1

Version option: 1 Artifacts option: 1 Shipping option: 1

Using 0.0.X-TTTTT-HHHHH, triggered on manual tag (one for artifact).

Pros:

  • Control on when publish happens
  • Version is sequential code-wise
  • Version is sequential time-wise
  • Version identifies the commit it was first built on

Cons:

  • Several manual steps, one per artifact (image/chart)
  • Lack of review on what's going to ship (can't review tags)
  • Chart might get out of sync with image
  • Tags are local to the repo and might move around, so might not identify the commit the version was generated for
  • Not following semantic versioning

Option 2

Version option: 2 Artifacts option: 2 Shipping option: 2

Using A.B.C-X-TTTTT-HHHHH, triggered by manual commit + tag (shared for both image and chart)

Pros:

  • Control on when publish happens
  • Version is sequential code-wise
  • Version is sequential time-wise
  • Version identifies the commit it was first built on
  • Version follows semantic versioning
  • Partial of review on what's going to ship (bump commit)

Cons:

  • Several manual steps (bump version commit + tag)
  • Partial of review on what's going to ship (can't review tags)
  • Tags are local to the repo and might move around, so might not identify the commit the version was generated for


Option 3

Version option: 2 Artifacts option: 2 Shipping option: 3

Using A.B.C-X-TTTTT-HHHHH, triggered by mr merge or direct push to main

Pros:

  • Version is sequential code-wise
  • Version is sequential time-wise
  • Version identifies the commit it was first built on
  • Version follows semantic versioning (non-backwards compatible commits labeled at development time)
  • Review on what's going to ship (each mr)
  • No manual steps

Cons:

  • No control of when a publish happens (well, it happens once per merge)

Option 4

Version option: 1 (always incremental, no special semantics) Artifacts option: 2 (chart and container image at the same time) Shipping option: 4 (ship on version bump, which is manual, and is encouraged to be accompanied by a git tag)

This is what Arturo thinks resembles the most the **the debian-like kung-fu way ** for packages.

Pros:

  • TBD

Cons:

  • TBD

Option 5

Version option: 1 (always incremental, no special semantics) Artifacts option: 2 (chart and container image at the same time) Shipping option: 3 (ship on every merge/push to main)

Pros:

  • TBD

Cons:

  • TBD
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Wikimedia_Cloud_Services_team/EnhancementProposals/Decision_record_T339198_Toolforge_component_deployment_flow_details&oldid=2089372"