Portal:Cloud VPS/Admin/Magnum
Magnum is the OpenStack Kubernetes as a Service offering. It can be used to easily deploy kubernetes clusters to a cloud VPS project.
Deploy a cluster
openstack coe cluster create <cluster name> --cluster-template <cluster template name> --master-count 1 --node-count <worker count>
Get kube config file
After this is built get the config file to connect with kubectl using
OS_PROJECT_ID={project} openstack coe cluster config <cluster name> --dir <path to put "config" file in>
Notes
At time of writing clusters can only be deployed with a single control node. T326436 tracks getting more.
Templates
Magnum clusters are deployed using a template that defines what the cluster will look like to openstack. This includes what image will be used, networking information, control and worker node sizes, and what version of kubernetes will be deployed when using the cluster.
Create a template
For an eqiad1 cluster the following will define the correct networking information
openstack coe cluster template create <template name> \
--image <image name> \
--external-network wan-transport-eqiad \
--fixed-network lan-flat-cloudinstances2b \
--fixed-subnet cloud-instances2-b-eqiad \
--dns-nameserver 8.8.8.8 \
--network-driver flannel \
--docker-storage-driver overlay2 \
[--docker-volume-size <size> \]
--master-flavor <flavor> \
--flavor <flavor> \
--coe kubernetes \
--labels kube_tag=<version>-rancher1-linux-amd64,hyperkube_prefix=docker.io/rancher/,cloud_provider_enabled=true \
--floating-ip-disabled \
[--public]
A generic example would look like:
openstack coe cluster template create k8s23 \
--image Fedora-CoreOS-38 \
--external-network wan-transport-eqiad \
--fixed-network lan-flat-cloudinstances2b \
--fixed-subnet cloud-instances2-b-eqiad \
--dns-nameserver 8.8.8.8 \
--network-driver flannel \
--docker-storage-driver overlay2 \
--docker-volume-size 100 \
--master-flavor g2.cores1.ram2.disk20 \
--flavor g2.cores1.ram2.disk20 \
--coe kubernetes \
--labels kube_tag=v1.23.15-rancher1-linux-amd64,hyperkube_prefix=docker.io/rancher/,cloud_provider_enabled=true \
--floating-ip-disabled \
--public
For codfw1dev you would want different networking options:
openstack coe cluster template create core-37-k8s23-100g \
--image Fedora-CoreOS-38 \
--external-network wan-transport-codfw \
--fixed-subnet cloud-instances2-b-codfw \
--fixed-network lan-flat-cloudinstances2b \
--dns-nameserver 8.8.8.8 \
--network-driver flannel \
--docker-storage-driver overlay2 \
--docker-volume-size 100 \
--master-flavor g2.cores1.ram2.disk20 \
--flavor g2.cores1.ram2.disk20 \
--coe kubernetes \
--labels kube_tag=v1.23.15-rancher1-linux-amd64,hyperkube_prefix=docker.io/rancher/,cloud_provider_enabled=true \
--floating-ip-disabled \
--public
Template options
There are several optional options.
--docker-volume-size will define how large of a volume, if any, to attach to nodes, this volume is used for storing docker images. If none is specified the disk space available to the image itself (as part of its flavor definition) will be used instead.
--public will define if the template can be seen beyond the project it was created in.
kube_tag can be found at https://hub.docker.com/r/rancher/hyperkube/tags
Scale a cluster
openstack coe cluster resize <cluster name> <number of workers>
Upgrade a cluster
(This has not been seen to work outside of devstack)
openstack coe cluster upgrade <cluster name> <template to upgrade to>
If this gets stuck you may need to uncordon any cordoned nodes and then run:
openstack coe cluster resize <cluster name> <current number of workers>
to reset the cluster status. Then run the upgrade again.
While updating kubectl won't work, but services will still be running and available.
Add cinder csi to cluster
Setup cinder csi: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md#using-the-manifests
example cloud.conf:
[Global]
application-credential-id = <application credential id>
application-credential-secret = <application credential secret>
domain-name = default
auth-url = https://openstack.eqiad1.wikimediacloud.org:25000/v3
tenant-id = <project id>
region = eqiad1-r
tenant-id can be found from:
openstack server show <id of a k8s cluster node>
it will be listed as project_id
base64 -w 0 cloud.conf
git clone https://github.com/kubernetes/cloud-provider-openstack.git
cd cloud-provider-openstack
vim manifests/cinder-csi-plugin/csi-secret-cinderplugin.yaml # replace base64 part with base64 output from above
kubectl create -f manifests/cinder-csi-plugin/csi-secret-cinderplugin.yaml
kubectl -f manifests/cinder-csi-plugin/ apply
Setup cinder as the default storage class:
sc.yaml:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: standard
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: cinder.csi.openstack.org
parameters:
availability: nova
kubectl apply -f sc.yaml
Provisioning with OpenTofu
When using tofu to provision a Magnum cluster, the application credentials used must include the "Unrestricted (dangerous)" permission or things will blow up. Specifically they will blow up with a "Failed to create trustee or trust for Cluster" error when Magnum attempts to create a service account for the cluster's internal use.