Helm is used to build "charts", which are packages of Kubernetes resources that are used to deploy apps to a cluster. Those charts are kept in a repository served by chartmuseum.
For deployment at Wikimedia, Helm is called via helmfile in combination with helm-diff to present the changes to be applied to the user. For instructions on how to deploy services and configuration changes with helmfile, see Kubernetes/Deployments.
Testing a new version
If you are wondering "How do I test a new helm version before reaching production", one possible strategy could be to do the following:
- Start working locally, importing the new version and building the new Debian package.
- Install it, and test helm with Minikube (for example, try helm lint, helm template, helm upgrade, etc.. on some charts in operations/deployment-charts).
- If the results are good, proceed with importing/installing/publishing on buil2001/apt1001 and upgrade CI to use the new helm version (see below how to do it).
- If CI runs fine (that involves using helm lint on all charts for example), proceed with deploying helm3 more widely (check https://debmonitor.wikimedia.org/ for more info about where to the package is installed).
Importing a new version
The packages are build using Git-buildpackage. The imported upstream tarballs should include the complete vendor directory.
- Check out the version (git tag) to import
- Check the go version you are running matches the minimum version in go.mod
- Check changes to environment variables and LD_FLAGS from the upstream Makefile, and add them later to debian/rules as part of the changelog CR if needed.
- This drops you into a shell with the git tag checked out. Do necessary changes here and commit
go mod vendor git add -f vendor # git diff --name-status --cached | grep -v 'vendor/' # to make sure you only changed vendor git commit -m "added vendor"
- Exiting the shell will build a tarball to import in the parent directory of the repo:
gbp import-orig ../tarball.tar.xz
- Push changes (including the tag crated by gpb) to gerrit
git push gerrit --all git push gerrit --tags
- Add a debian/changelog entry (as CR)
gbp dch # Edit debian/changelog git commit git review
Building a new version
- Check out the git repo on the build host
- Build the package
BACKPORTS=yes gbp buildpackage --git-pbuilder --git-no-pbuilder-autoconf --git-dist=bullseye -sa -uc -us
Publish a new version
# on apt1001 rsync -vaz build2001.codfw.wmnet::pbuilder-result/bullseye-amd64/*<PACKAGE VERSION>* . sudo -i reprepro -C main include bullseye-wikimedia /path/to/<PACKAGE>.changes # Check where the package is in reprepro sudo -i reprepro lsbycomponent helm3 # Copy the package over to other distros (this is possible because they only contain static binaries) sudo -i reprepro copy buster-wikimedia bullseye-wikimedia <PACKAGE>
Log the change to SAL
!log "Imported <PACKAGE> <PACKAGE VERSION> to bullseye-wikimedia and buster-wikimedia"
Update CI image
We're using helm/helmfile in operations/deployment-charts to lint and verify charts.
To update the envoy version used there, bump the changelog at dockerfiles/helm-linter/changelog:
dch -D wikimedia --force-distribution -c changelog
And add the new version to
jjb/operations-misc.yaml in a second patch (example)
When this is merged and build, run CI (maybe just rebuild last at https://integration.wikimedia.org/ci/job/helm-lint/ ?) to verify the new version against our charts/deployments.