Intro

Magnum relies on a host of moving parts to launch a Kubernetes cluster. As of 2025, there are two back-end implementations. For the last few years we've been using the Heat backend, but it is being deprecated by the upstream developers in favor of cluster-api backends.

We are in the process of moving from the Heat backend to the cluster-api backend. This document is about understanding and setting up Magnum to work with the magnum-cluster-api driver.

Openstack Components

When creating a cluster, Magnum creates a service account and credentials in Keystone and then delegates the actual cluster creation to the cluster-api service (via the magnum-cluster-api driver) which uses the Magnum-managed credentials. Cluster-api uses those credentials to call back into other openstack services to create and configure the components of the cluster (e.g. calling nova to create a worker node.)

Cluster-api is not an openstack project; it's a long-standing k8saas project which magnum-cluster-api wraps. Cluster-api runs on its own little utility k8s cluster.

Magnum-api

Magnum-api provides the REST api for managing kubernetes clusters and cluster templates. It is a straightforward python/uwsgi service that runs on cloudcontrol nodes and is managed by puppet just like other openstack services.

The magnum-api logs can be read via journalctl, logstash, or by looking at /var/log/magnum/magnum-api.log. There is seldom anything of interest in the logs but they may contain useful messages if there are fundamental integration issues between the different magnum components.

Typically a user will interact with magnum-api using opentofu or the openstack cli. In the cli, magnum is referred to as 'cluster coe,', for example:

 # openstack coe cluster list

Magnum-conductor

Magnum-conductor schedules and implements the various steps in cluster creation. It is installed and managed by puppet and runs on cloudcontrol nodes.

Most of the real work of creating a kubernetes cluster is delegated to cluster api, so magnum-conductor is mostly limited to maintaining and reporting cluster status.

Magnum-api and magnum-conductor are configured in /etc/magnum/magnum.conf on a cloudcontrol.

Magnum-cluster-api driver

The cluster-api driver is not packaged for Debian or part of the osbpo as of May, 2026, so we need to build it ourselves.

We maintain our own repo which clones the upstream codebase and adds the necessary bits to build a debian package. That repo contains instructions about how to build, README_WMF -- the latest instructions are on the debian/trixie-wikimedia branch. We do not currently have an established cadence for rebuilding the driver but it's probably best to update to the latest build whenever updating the rest of the openstack releases.

Octavia

Cluster-api can create an Octavia loadbalancer as a front end to the created k8s cluster. Depending on template setup, it can attach a floating-ip to that load balanacer, or leave it internal. The most common practice is to use a web proxy to point to an octavia balancer, rather than giving the balancer itself a floating IP.

Neutron

Typically we create clusters on an existing VM network (e.g. VXLAN/IPv6-dualstack).

Nova

The actual k8s worker nodes managed by magnum will appear as regular nova VMs within the project of the magnum user. They typically have names like "<cluster-name-worker-1-2>-<id>-default-worker-<id>"

Cluster-api (capi) Components

capi service cluster

Cluster api runs on a kubernetes cluster. In order to avoid chicken/egg concerns, the cluster that hosts capi is NOT managed by magnum. Instead, it runs on an easily-recreated single node k3s cluster and as described by a helm chart. This cluster can be created and configured with the puppet class 'profile::openstack::capi'

To build a new capi worker:

1. Create a new debian VM in the 'magnum' project named 'capi-worker-xxx'. One core and 2GB of ram seems sufficient.

2. Apply the 'profile::openstack::capi' class to that VM (there should already be a prefix config that does this), and allow puppet to stabilize (it will take two or three puppet runs). Puppet will create a k3s cluster and then apply a helm chart hosted on our chartmuseum repo to set up capi.

3. Apply security group rules that allow cloudcontrol access to port 644 (typically via a pre-existing service group like 'cloudcontrol-to-kubernetes')

4. Confirm that the worker is set up properly:

labtestandrew@capi-worker-1:~$ export KUBECONFIG=/etc/rancher/k3s/k3s.yaml 
labtestandrew@capi-worker-1:~$ sudo kubectl get pods --all-namespaces
NAMESPACE                           NAME                                                              READY   STATUS    RESTARTS      AGE
capi-addon-system                   cluster-api-addon-provider-756bfd798f-5gpnk                       1/1     Running   0             44h
capi-janitor-system                 cluster-api-janitor-openstack-55fb64777c-d9bgb                    1/1     Running   0             44h
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-5b959f764c-cl79q        1/1     Running   6 (41h ago)   44h
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-7556869f8-ctl5d     1/1     Running   3 (43h ago)   44h
capi-system                         capi-controller-manager-5b67d4fc7-c94md                           1/1     Running   6 (41h ago)   44h
capo-system                         capo-controller-manager-6f899b5b7b-spslb                          1/1     Running   3 (41h ago)   44h
cert-manager                        cert-manager-7d67448f59-t5vlb                                     1/1     Running   0             44h
cert-manager                        cert-manager-cainjector-666b8b6b66-mmt5k                          1/1     Running   0             44h
cert-manager                        cert-manager-webhook-78cb4cf989-6xwfm                             1/1     Running   2 (43h ago)   44h
kube-system                         coredns-5688667fd4-9hsqt                                          1/1     Running   2 (43h ago)   45h
kube-system                         local-path-provisioner-774c6665dc-g5fld                           1/1     Running   0             45h
kube-system                         metrics-server-6f4c6675d5-8fw6r                                   1/1     Running   2 (43h ago)   45h
magnum-admin                        test-cluster-capi-worker-1-2-edez5vbbrppf-autoscaler-6bff5mbbnr   1/1     Running   5 (42m ago)   47m

5. Once that worker is up and running, you need to direct magnum to use the new cluster. As created by k3s, /etc/rancher/k3s/k3s.yaml contains everything that Magnum will need to know with the exception of the host IP. Make a copy of k3s.yaml but replace 127.0.0.1 in the the 'server' line with the actual IP of the worker.

6. Add this existing k3s.yaml contents to private puppet, as modules/secret/secrets/openstack/<codfw1dev/eqiad1>/magnum/capiservicek3s.yaml -- it will then be installed on cloudcontrols.

node images

Each magnum template will refer to a base VM image. These images should be special ready-made ubuntu or debian images, prebuilt with kubernetes already installer or partially configured. As of today, the best images to be used are built and published by vexxhost, found at https://github.com/vexxhost/capo-image-elements/releases.

Installing one of these images looks roughly like this:

$ wget https://github.com/vexxhost/capo-image-elements/releases/download/2026.05-6/debian-13-v1.36.1.qcow2
$ sudo wmcs-openstack image create magnum-debian-13-v1.36.1 --public --disk-format=qcow2 --container-format=bare --property os_distro=debian --file=/home/andrew/debian-13-v1.36.1.qcow2

Troubleshooting

Any logs about failure or success will be present only on the capi worker; the most useful ones are most likely written by the capo-controller-manager. To view them, log onto the capi worker VM and note capo-controller-manager name from

labtestandrew@capi-worker-1:~$ sudo kubectl get pods --all-namespaces
...
capi-system                         capi-controller-manager-5b67d4fc7-c94md                           1/1     Running   6 (41h ago)   44h
capo-system                         capo-controller-manager-6f899b5b7b-spslb                          1/1     Running   3 (41h ago)   44h
...

Then, to see the logs:

root@capi-worker-1:~# kubectl  logs -n capo-system   capo-controller-manager-6f899b5b7b-spslb  | less

the capi-controller-manager logs may also be helpful.

Other necessary services

Image repo

We try not to build our control plain out of random upstream packages; rather, we mirror particular upstream versions in an internal repo.

The images used for creating the capi worker cluster (as well as the k3s images themselves) are stored on a VM in the cloudinfra project named 'docker-registry-XX' with the images themselves stored on a mounted cinder volume. These packages can be upgraded from various upstream repos; when version string changes the associated version numbers must also be updated in hiera with settings like 'profile::openstack::capi::cluster_api_version'

Clusterctl, orc, k3s scripts

Again, to avoid pulling random install scripts from the greater internet, we store a few scripts in cloud-vps object storage. These are in the 'magnum' project's object store.

Compatibility matrix

Version compatibility for magnum-cluster-api does not seem to be formally documented anywhere. For posterity, here is an example of a working set of components and versions provided by a dev on IRC. This isn't meant as a blueprint for how we deploy but just a jumping off 'known good' setup:


component	version	source
Cluster API (CAPI)	v1.10.5	hack/setup-capo.sh
CAPO (Cluster API Provider OpenStack)	v0.12.7	hack/setup-capo.sh
ORC (OpenStack Resource Controller)	v2.2.0	hack/setup-capo.sh
Kubernetes	v1.33.11, v1.34.7, v1.35.4	zuul.d/jobs.yaml / project.yaml
CNI – Calico	v3.31.5 (default)	magnum_cluster_api/resources.py:50
CNI – Cilium	(tested in CI gate)	zuul.d/project.yaml
OS image	Ubuntu 22.04 (capo-image-elements)	zuul.d/jobs.yaml:68
Magnum	>= 15.0.0 (2024.1 Caracal+)	pyproject.toml
OpenStack services	Barbican, Octavia (+ ovn-octavia-provider), Manila	zuul.d/jobs.yaml
Devstack node (we don't use this generally)	Ubuntu Noble (24.04)	zuul.d/jobs.yaml