Portal:Toolforge/Admin/Kubernetes/Components

From Wikitech
Jump to navigation Jump to search

This page contains information on how to operate, build and deploy various Toolforge Kubernetes components.

The information in here is subject to change as we move in the direction of Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_Kubernetes_component_workflow_improvements

In general, assume this is valid for both tools and toolsbeta.

Workflow

The workflow as we have it today.

Steps

This is what you normally would do:

  • code: make changes to a given code repository, push & merge as usual.
  • build: once code is ready to deploy to k8s, if the component needs it, build the container image using the cookbook.
  • pre-deploy: if the component uses a built container image, chances are that the image tag requires a refresh (avoid :latest). This is a new, separate commit, pointing to the newest built container image from previous step.
  • deploy: once ready to deploy to k8s for real, use the cookbook.
  • verify test by hand, verify, etc.
  • rollback if a rollback is needed, follow again the same steps but merging a revert commit plus (if required) an image tag bump.

Build

Use the wmcs.toolforge.k8s.component.build cookbook to build the docker container image that will later be deployed in the kubernetes cluster. 

user@laptop:~$ cookbook wmcs.toolforge.k8s.component.build -h
usage: cookbooks.wmcs.toolforge.k8s.component.build [-h] [--project PROJECT] [--task-id TASK_ID] [--no-dologmsg]
                                                    [--registry-url REGISTRY_URL]
                                                    [--docker-builder-hostname DOCKER_BUILDER_HOSTNAME] --git-url GIT_URL
                                                    [--git-name GIT_NAME] [--git-branch GIT_BRANCH]
                                                    [--docker-image-tag DOCKER_IMAGE_TAG]
                                                    [--docker-image-name DOCKER_IMAGE_NAME]

WMCS Toolforge Kubernetes - build a docker image for a custom component

Usage example:
    cookbook wmcs.toolforge.k8s.component/build \
        --git-url https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api

options:
  -h, --help            show this help message and exit
  --project PROJECT     Relevant Cloud VPS openstack project (for operations, dologmsg, etc). If this cookbook is for hardware,
                        this only affects dologmsg calls. Default is 'tools'.
  --task-id TASK_ID     Id of the task related to this operation (ex. T123456). (default: None)
  --no-dologmsg         To disable dologmsg calls (no SAL messages on IRC). (default: False)
  --registry-url REGISTRY_URL
                        docker registry URL (default: docker-registry.tools.wmflabs.org)
  --docker-builder-hostname DOCKER_BUILDER_HOSTNAME
                        docker image builder virtual machine hostname (default: tools-docker-imagebuilder-01)
  --git-url GIT_URL     git URL for the source code (default: None)
  --git-name GIT_NAME   git repository name. If not provided, it will be guessed based on the git URL (default: None)
  --git-branch GIT_BRANCH
                        git branch in the source repository (default: main)
  --docker-image-tag DOCKER_IMAGE_TAG
                        docker tag for the new image, if not provided the git hash of the latest commit will be used (default:
                        None)
  --docker-image-name DOCKER_IMAGE_NAME
                        docker image name. If not provided, it will be guessed based on the git name (default: None)

Example: 

user@laptop:~$ cookbook wmcs.toolforge.k8s.component.build --git-url https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api
[..]
user@laptop:~$ cookbook wmcs.toolforge.k8s.component.build --git-url https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer
INFO: guesses git tree name as jobs-framework-emailer
INFO: guesses docker image name as toolforge-jobs-framework-emailer
START - Cookbook wmcs.toolforge.k8s.component.build
INFO: using build node tools-docker-imagebuilder-01.tools.eqiad1.wikimedia.cloud
INFO: creating temp dir /tmp/cookbook-toolforge-k8s-component-build-ulprobicib
INFO: git cloning https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer
INFO: git checkout main on cloning /tmp/cookbook-toolforge-k8s-component-build-ulprobicib/jobs-framework-emailer
INFO: building docker image toolforge-jobs-framework-emailer:latest
INFO: cleaning up temp dir /tmp/cookbook-toolforge-k8s-component-build-ulprobicib
INFO: creating docker tag docker-registry.tools.wmflabs.org/toolforge-jobs-framework-emailer:latest
INFO: pushing to the registry docker-registry.tools.wmflabs.org/toolforge-jobs-framework-emailer:latest
[DOLOGMSG]: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-emailer:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer (084ee51)
END (PASS) - Cookbook wmcs.toolforge.k8s.component.build (exit_code=0)

Deploy

Use the wmcs.toolforge.k8s.component.deploy cookbook to deploy the custom component in the given kubernetes cluster. 

user@laptop:~$ cookbook wmcs.toolforge.k8s.component.deploy -h
usage: cookbooks.wmcs.toolforge.k8s.component.deploy [-h] [--project PROJECT] [--task-id TASK_ID] [--no-dologmsg]
                                                     [--deploy-node-hostname DEPLOY_NODE_HOSTNAME] --git-url GIT_URL
                                                     [--git-name GIT_NAME] [--git-branch GIT_BRANCH]
                                                     [--deployment-command DEPLOYMENT_COMMAND]

WMCS Toolforge Kubernetes - deploy a kubernetes custom component

Usage example:
    cookbook wmcs.toolforge.k8s.component.deploy \
        --git-url https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api

options:
  -h, --help            show this help message and exit
  --project PROJECT     Relevant Cloud VPS openstack project (for operations, dologmsg, etc). If this cookbook is for hardware,
                        this only affects dologmsg calls. Default is 'toolsbeta'.
  --task-id TASK_ID     Id of the task related to this operation (ex. T123456). (default: None)
  --no-dologmsg         To disable dologmsg calls (no SAL messages on IRC). (default: False)
  --deploy-node-hostname DEPLOY_NODE_HOSTNAME
                        k8s control node hostname (default: toolsbeta-test-k8s-control-4)
  --git-url GIT_URL     git URL for the source code (default: None)
  --git-name GIT_NAME   git repository name. If not provided, it will be guessed based on the git URL (default: None)
  --git-branch GIT_BRANCH
                        git branch in the source repository (default: main)
  --deployment-command DEPLOYMENT_COMMAND
                        command to trigger the deployment. (default: ./deploy.sh)

List

This is a list of our custom components and where to find them.

This is the only known source of truth for this information. Even if the list might drift outdated, as of this writing there is no other place in which this information is collected.

Name Requires build Repository clone URL for the cookbooks Repository browser URL Comments
volume-admission-controller yes https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller https://gerrit.wikimedia.org/r/admin/repos/cloud/toolforge/volume-admission-controller See also: redundant link1 redundant link2
ingress-admission-controller yes https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller https://gerrit.wikimedia.org/r/admin/repos/cloud/toolforge/ingress-admission-controller See also: redundant link1 redundant link2
registry-admission-controller yes https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook https://gerrit.wikimedia.org/r/admin/repos/labs/tools/registry-admission-webhook See also: redundant link1 redundant link2
nginx-ingress no https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx See also: Portal:Toolforge/Admin/Kubernetes/Networking_and_ingress#nginx-ingress
delete-crashing-pods yes https://gerrit.wikimedia.org/r/cloud/toolforge/delete-crashing-pods https://gerrit.wikimedia.org/r/admin/repos/cloud/toolforge/delete-crashing-pods See also: Portal:Toolforge/Admin/Kubernetes/Deploying#delete-crashing-pods
jobs-framework-api yes https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api https://gerrit.wikimedia.org/r/admin/repos/cloud/toolforge/jobs-framework-api See also: Portal:Toolforge/Admin/Kubernetes/Jobs_framework
jobs-framework-emailer yes https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer https://gerrit.wikimedia.org/r/admin/repos/cloud/toolforge/jobs-framework-emailer See also: Portal:Toolforge/Admin/Kubernetes/Jobs_framework
maintain-kubeusers yes https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers https://gerrit.wikimedia.org/r/admin/repos/labs/tools/maintain-kubeusers See also: Portal:Toolforge/Admin/Kubernetes/Certificates#Tool_certs
image-config no https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config See also: phabricator T311918: Cloud services enhancement proposal: Single place to configure Toolforge container images
kubernetes-metrics no https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics See also: Portal:Toolforge/Admin/Kubernetes#Monitoring
cert-manager no https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager See also Portal:Toolforge/Admin/Kubernetes/Certificates
calico no https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/ https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/ See T328539: toolforge: consider relocating core k8s components out of puppet into its own repository
api-gateway no https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/ https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/ Portal:Toolforge/Admin/APIs

Third-party components

Additional consideration for some third-party components that we use.

Name Deployment repository Current version K8s versions supported
Calico https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/ 3.21.0 1.20 - 1.22[1]
ingress-nginx https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx 1.3.0 (chart-4.2.3) 1.20 - 1.24[2]
kube-state-metrics https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics 2.2.4 (chart-4.1.0) ?
metrics-server https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics 0.6.2 (chart-3.8.3) 1.19+[3]
cadvisor https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics 0.30.2 n/a (not a Kubernetes specific tool)
Prometheus https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics n/a
cert-manager https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager 1.11.0 1.21 - 1.26[4]
reloader https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager 1.0.5 ?
Tekton Pipelines https://github.com/toolforge/buildservice/tree/main/deploy/base-tekton 0.33.2 1.20+[5]

References

  1. ↑ https://docs.tigera.io/archive/v3.21/getting-started/kubernetes/requirements
  2. ↑ https://github.com/kubernetes/ingress-nginx/tree/main#supported-versions-table
  3. ↑ https://github.com/kubernetes-sigs/metrics-server#compatibility-matrix
  4. ↑ https://cert-manager.io/docs/installation/supported-releases/
  5. ↑ https://tekton.dev/vault/pipelines-v0.33.x/install/

See also

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Portal:Toolforge/Admin/Kubernetes/Components&oldid=2061466"