Helm

Helm is a package manager for Kubernetes.

Helm is used to build "charts", which are packages of Kubernetes resources that are used to deploy apps to a cluster. Those charts are kept in a repository served by chartmuseum.

For deployment at Wikimedia, Helm is called via helmfile in combination with helm-diff to present the changes to be applied to the user. For instructions on how to deploy services and configuration changes with helmfile, see Kubernetes/Deployments.

Packaging

We're providing separate packages for operations/debs/helm (helm 2.x) and operations/debs/helm3; both may be installed at the same time, but note that we no longer use helm2 in production (since 2022)

The packages operations/debs/helmfile and operations/debs/helm-diff follow the same packaging workflow, so we document that here together.

Testing a new version

If you are wondering "How do I test a new helm version before reaching production", one possible strategy could be to do the following:

Start working locally, importing the new version and building the new Debian package.
Install it, and test helm with Minikube (for example, try helm lint, helm template, helm upgrade, etc.. on some charts in operations/deployment-charts).
If the results are good, proceed with importing/installing/publishing on buil2001/apt1001 and upgrade CI to use the new helm version (see below how to do it).
If CI runs fine (that involves using helm lint on all charts for example), proceed with deploying helm3 more widely (check https://debmonitor.wikimedia.org/ for more info about where to the package is installed).

Importing a new version

The packages are build using Git-buildpackage. The imported upstream tarballs should include the complete vendor directory.

Check out the version (git tag) to import
```
./debian/repack "vX.Y.Z"
```

Check the go version you are running matches the minimum version in go.mod
Check changes to environment variables and LD_FLAGS from the upstream Makefile, and add them later to debian/rules as part of the changelog CR if needed.

This drops you into a shell with the git tag checked out. Do necessary changes here and commit

go mod vendor
git add -f vendor
# git diff --name-status --cached | grep -v 'vendor/' # to make sure you only changed vendor
git commit -m "added vendor"

Exiting the shell will build a tarball to import in the parent directory of the repo:
```
gbp import-orig ../tarball.tar.xz
```

Push changes (including the tag crated by gpb) to gerrit
```
git push gerrit --all
git push gerrit --tags
```

Add a debian/changelog entry (as CR)

gbp dch # Edit debian/changelog
git commit
git review

Building a new version

Check out the git repo on the build host

Build the package

BACKPORTS=yes gbp buildpackage --git-pbuilder --git-no-pbuilder-autoconf --git-dist=bullseye -sa -uc -us

Publish a new version

# on apt1001
rsync -vaz build2001.codfw.wmnet::pbuilder-result/bullseye-amd64/*<PACKAGE VERSION>* .
sudo -i reprepro -C main include bullseye-wikimedia /path/to/<PACKAGE>.changes

# Check where the package is in reprepro
sudo -i reprepro lsbycomponent helm3
# Copy the package over to other distros (this is possible because they only contain static binaries)
sudo -i reprepro copy buster-wikimedia bullseye-wikimedia <PACKAGE>

Log the change to SAL

!log "Imported <PACKAGE> <PACKAGE VERSION> to bullseye-wikimedia and buster-wikimedia"

Update CI image

We're using helm/helmfile in operations/deployment-charts to lint and verify charts.

To update the envoy version used there, bump the changelog at dockerfiles/helm-linter/changelog:

dch -D wikimedia --force-distribution -c changelog

And add the new version to jjb/operations-misc.yaml in a second patch (example)

When this is merged and build, run CI (maybe just rebuild last at https://integration.wikimedia.org/ci/job/helm-lint/ ?) to verify the new version against our charts/deployments.

Resources