Help:Managed Kubernetes

Magnum is a heavily experimental deployment at this point.

Magnum is the OpenStack Kubernetes as a Service offering. It can be used to deploy kubernetes clusters to a cloud VPS project.

To configure the openstack CLI, see Help:Using_OpenStack_APIs#OpenStack_Commandline_Interfaces.

The openstack coe subcommand used below is provided by the Magnum client (sudo apt install python3-magnum).

Deploy a cluster

openstack coe cluster create <cluster name> --cluster-template <cluster template name> --master-count 1 --node-count <worker count>

Get kube config file

After this is built get the config file to connect with kubectl using

OS_PROJECT_ID={project} openstack coe cluster config <cluster name> --dir <path to put "config" file in>

Driver Selection

As of June 2026, there are two working Magnum backends installed in cloud-vps: heat and cluster-api. The Heat driver will be used if a cluster template requests a fedora worker image (currently magnum-fedora-coreos-34); the newer cluster-api driver is used if the template requests an ubuntu or debian image (currently ubuntu-22.04-v1.34.8-for-magnum).

The Heat driver will be phased out within the next twelve months, so any new use-cases should rely on the cluster-api driver with Debian or Ubuntu worker images.

At time of writing, Heat clusters can only be deployed with a single control node. T326436 tracks getting more.

Templates

Magnum clusters are deployed using a template that defines what the cluster will look like to openstack. This includes what image will be used, networking information, control and worker node sizes, and what version of kubernetes will be deployed when using the cluster.

Create a template

Any magnum-managed cluster will be created based off a coe cluster template. Template options are documented, but be warned that different drivers implement different options and in some cases handle them differently. Additionally, templates can include the free-form 'label' option which serves to support driver-specific options which are not part of the pre-determined template schema.

Because of that uncertainty, the best strategy is to begin with an existing, working template, and iterate on that rather than creating a new aspirational template from scratch.

Example Templates, Heat driver

Here is a sample template with working network configuration for eqiad1, using the Heat driver:

openstack coe cluster template create k8s23 \
--image Fedora-CoreOS-38 \
--external-network wan-transport-eqiad \
--fixed-network VXLAN/dualstack \
--fixed-subnet vxlan-dualstack-ipv4 \
--dns-nameserver 172.20.255.1 \
--network-driver flannel \
--docker-storage-driver overlay2 \
--docker-volume-size 100 \
--master-flavor g2.cores1.ram2.disk20 \
--flavor g2.cores1.ram2.disk20 \
--coe kubernetes \
--labels kube_tag=v1.23.15-rancher1-linux-amd64,hyperkube_prefix=docker.io/rancher/,cloud_provider_enabled=true \
--floating-ip-disabled \
--public

For codfw1dev you would want different networking options:

openstack coe cluster template create core-37-k8s23-100g \
--image Fedora-CoreOS-38 \
--external-network wan-transport-codfw \
--fixed-network VXLAN/dualstack \
--fixed-subnet vxlan-dualstack-ipv4 \
--dns-nameserver 172.20.255.1 \
--network-driver flannel \
--docker-storage-driver overlay2 \
--docker-volume-size 100 \
--master-flavor g2.cores1.ram2.disk20 \
--flavor g2.cores1.ram2.disk20 \
--coe kubernetes \
--labels kube_tag=v1.23.15-rancher1-linux-amd64,hyperkube_prefix=docker.io/rancher/,cloud_provider_enabled=true \
--floating-ip-disabled \
--public

kube_tag can be found at https://hub.docker.com/r/rancher/hyperkube/tags

Example Templates, Cluster-api driver

The easiest mistake to make with a new template is a mismatched kube_tag label. It must match the k8s version embedded in the worker image and will be reflected in the worker image's name.

openstack coe cluster template create \
      --image ubuntu-22.04-v1.34.8-for-magnum \
      --external-network "wan-transport-eqiad" \
      --fixed-network "VXLAN/IPv6-dualstack" \
      --dns-nameserver 8.8.8.8 \
      --master-lb-enabled \
      --master-flavor g4.cores2.ram4.disk20 \
      --flavor g4.cores2.ram4.disk20 \
      --network-driver calico \
      --docker-storage-driver overlay2 \
      --coe kubernetes \
      --labels master_lb_floating_ip_enabled=false,kube_tag=v1.34.8 \
      capitest

Template options

There are several optional options.

--docker-volume-size: will define how large of a volume, if any, to attach to nodes, this volume is used for storing docker images. If none is specified the disk space available to the image itself (as part of its flavor definition) will be used instead.

--public: will define if the template can be seen beyond the project it was created in.

Scale a cluster

openstack coe cluster resize <cluster name> <number of workers>

Upgrade a cluster

(This has not been seen to work outside of devstack)

openstack coe cluster upgrade <cluster name> <template to upgrade to>

If this gets stuck you may need to uncordon any cordoned nodes and then run:

openstack coe cluster resize <cluster name> <current number of workers>

to reset the cluster status. Then run the upgrade again.

While updating kubectl won't work, but services will still be running and available.

Add cinder csi to cluster

Setup cinder csi: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md#using-the-manifests

example cloud.conf:

[Global]
application-credential-id = <application credential id>
application-credential-secret = <application credential secret>
domain-name = default
auth-url = https://openstack.eqiad1.wikimediacloud.org:25000/v3
tenant-id = <project id>
region = eqiad1-r

tenant-id can be found from:

openstack server show <id of a k8s cluster node>

it will be listed as project_id

base64 -w 0 cloud.conf
git clone https://github.com/kubernetes/cloud-provider-openstack.git
cd cloud-provider-openstack
vim manifests/cinder-csi-plugin/csi-secret-cinderplugin.yaml # replace base64 part with base64 output from above
kubectl create -f manifests/cinder-csi-plugin/csi-secret-cinderplugin.yaml
kubectl -f manifests/cinder-csi-plugin/ apply

Setup cinder as the default storage class: sc.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: cinder.csi.openstack.org
parameters:
  availability: nova

kubectl apply -f sc.yaml

Provisioning with OpenTofu

Tracked in Phabricator
Task T332194 Resolved

When using tofu to provision a Magnum cluster, the application credentials used must include the "Unrestricted (dangerous)" permission or things will blow up. Specifically they will blow up with a "Failed to create trustee or trust for Cluster" error when Magnum attempts to create a service account for the cluster's internal use.

Hints and tips for the cluster-api driver

This is a list of traps and surprises encountered while using the magnum-cluster-api driver. Please add yours below!

Error reporting for this driver is spotty at best. In many cases cluster creation will drag on forever rather than fail fast; in other cases it will fail but with an unhelpful error message. Better error reporting can be found using kubectl get cluster -n magnum-system {cluster id} -o json | jq .status
Typically you should set 'master_lb_floating_ip_enabled': 'false' in your template. If this is not set, cluster creation will fail if your project doesn't have available floating ip quota.
Cluster creation will fail cryptically if the kube_tag label does not match the kubernetes version installed on the requested worker image. See above for a few more details.
Cluster creation tends to hang if master_lb_enabled=false. You can explicitly set master_lb_enabled=true or leave it unset (in which case it will default to true)
Additional k8s api server certificate SANs can be managed via the api_server_cert_sans label. The value is a comma separated list of either IP addresses or DNS hostnames. In the latter case, domain names cannot end with `.`