Portal:Cloud VPS/Admin/Devstack magnum/Stable xena

From Wikitech

Overview

These are magnum install instructions for installing on devstack stable/xena

Notes

You will need a user with sudo access. Often this is the stack user, but any will do.

This was tested on Ubuntu 20.04.

To install a CoreOS beyond the listed version, you'll need a newer hyperkube as cgroups updated to v2 and it does weird things that k8s >= 1.19 handles, but the last official hyperkube was 1.18. Though rancher provides an unofficial one that seems to work: 

wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20211203.3.0/x86_64/fedora-coreos-35.20211203.3.0-openstack.x86_64.qcow2.xz
unxz fedora-coreos-35.20211203.3.0-openstack.x86_64.qcow2.xz
openstack image create Fedora-CoreOS --file=fedora-coreos-35.20211203.3.0-openstack.x86_64.qcow2 --disk-format=qcow2 --container-format=bare --property os_distro='fedora-coreos' --public
openstack coe cluster template create my-template --image Fedora-CoreOS --external-network public --fixed-network private --fixed-subnet private-subnet --dns-nameserver 8.8.8.8 --network-driver flannel --docker-storage-driver overlay2 --docker-volume-size 30 --master-flavor m1.small --flavor m1.medium --coe kubernetes --labels kube_tag=v1.20.14-rancher1-linux-amd64,hyperkube_prefix=docker.io/rancher/

Additionally the following patch appears necessary. It is seen at opendev, but isn't merged yet. It would go in before building magnum with pip. 

diff --git a/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-master.sh b/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-master.sh
index 05053115..9d669e78 100644
--- a/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-master.sh
+++ b/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-master.sh
@@ -195,8 +195,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kubelet \\
     --volume /lib/modules:/lib/modules:ro \\
     --volume /run:/run \\
     --volume /dev:/dev \\
-    --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \\
-    --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \\
+    --volume /sys/fs/cgroup:/sys/fs/cgroup \\
     --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\
     --volume /var/lib/calico:/var/lib/calico \\
     --volume /var/lib/docker:/var/lib/docker \\
@@ -236,8 +235,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kube-proxy \\
     --volume /usr/lib/os-release:/etc/os-release:ro \\
     --volume /etc/ssl/certs:/etc/ssl/certs:ro \\
     --volume /run:/run \\
-    --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \\
-    --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \\
+    --volume /sys/fs/cgroup:/sys/fs/cgroup \\
     --volume /lib/modules:/lib/modules:ro \\
     --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\
     \${CONTAINER_INFRA_PREFIX:-\${HYPERKUBE_PREFIX}}hyperkube:\${KUBE_TAG} \\
diff --git a/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh b/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh
index b74e856b..ef869272 100644
--- a/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh
+++ b/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh
@@ -93,8 +93,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kubelet \\
     --volume /lib/modules:/lib/modules:ro \\
     --volume /run:/run \\
     --volume /dev:/dev \\
-    --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \\
-    --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \\
+    --volume /sys/fs/cgroup:/sys/fs/cgroup \\
     --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\
     --volume /var/lib/calico:/var/lib/calico \\
     --volume /var/lib/docker:/var/lib/docker \\
@@ -134,8 +133,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kube-proxy \\
     --volume /usr/lib/os-release:/etc/os-release:ro \\
     --volume /etc/ssl/certs:/etc/ssl/certs:ro \\
     --volume /run:/run \\
-    --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \\
-    --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \\
+    --volume /sys/fs/cgroup:/sys/fs/cgroup \\
     --volume /lib/modules:/lib/modules:ro \\
     --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\
     \${CONTAINER_INFRA_PREFIX:-\${HYPERKUBE_PREFIX}}hyperkube:\${KUBE_TAG} \\

git fetch https://review.opendev.org/openstack/magnum refs/changes/28/800428/2 
git cherry-pick fe75ca3459a5b44b835b4157149d062526953915


xena was having a strange issue where it refused to load more than one CoreOS version at a time.

dns doesn't seem to be working from pods until flannel is restarted: 

kubectl -n kube-system rollout restart daemonset kube-flannel-ds

Installation

git clone https://github.com/openstack/devstack.git --branch stable/xena
cd devstack

cat <<EOF > local.conf
[[local|localrc]]
ADMIN_PASSWORD=secret
DATABASE_PASSWORD=\$ADMIN_PASSWORD
RABBIT_PASSWORD=\$ADMIN_PASSWORD
SERVICE_PASSWORD=\$ADMIN_PASSWORD

enable_plugin barbican https://opendev.org/openstack/barbican stable/xena
enable_plugin heat https://github.com/openstack/heat stable/xena

enable_service h-eng h-api h-api-cfn h-api-cw
EOF

./stack.sh
source openrc
export OS_USERNAME='admin'
export OS_PASSWORD='secret'

cd ~
wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/34.20210518.3.0/x86_64/fedora-coreos-34.20210518.3.0-openstack.x86_64.qcow2.xz
unxz fedora-coreos-34.20210518.3.0-openstack.x86_64.qcow2.xz 
openstack image create Fedora-CoreOS --file=fedora-coreos-34.20210518.3.0-openstack.x86_64.qcow2 --disk-format=qcow2 --container-format=bare --property os_distro='fedora-coreos' --public


ssh-keygen -f admin -P ""
openstack keypair create --public-key admin.pub admin


sudo mkdir /etc/magnum
cd ~
git clone https://git.openstack.org/openstack/magnum --branch stable/xena
cd magnum
sudo pip install -e .
sudo tox -egenconfig
sudo cp etc/magnum/api-paste.ini /etc/magnum/api-paste.ini


sudo sh -c 'cat <<EOF > /etc/magnum/magnum.conf
[DEFAULT]
pybasedir = /usr/lib/python3/dist-packages/magnum
bindir = /usr/bin
state_path = /var/lib/magnum
transport_url = rabbit://magnum:secret@localhost:5672
[api]
host = <host ip>
[barbican_client]
[certificates]
cert_manager_type = barbican
[cinder]
[cinder_client]
region_name = RegionOne
api_version = 3
[cluster]
[cluster_heat]
[cluster_template]
[conductor]
[cors]
[database]
connection = mysql+pymysql://magnum:secret@localhost/magnum
[docker]
[docker_registry]
[drivers]
[glance_client]
[heat_client]
[keystone_auth]
[keystone_authtoken]
memcached_servers = localhost:11211
auth_version = v3
www_authenticate_uri = http://localhost/identity
project_domain_id = default
user_domain_id = default
password = secret
auth_url = http://localhost/identity
auth_type = password
admin_user = magnum
admin_password = secret
admin_tenant_name = service
project_name = service
project_domain_name = default
username = magnum
user_domain_name = default
region_name = RegionOne
auth_type = password
[kubernetes]
[magnum_client]
[neutron_client]
[nova_client]
[octavia_client]
[oslo_concurrency]
lock_path = /var/lock/magnum
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
driver = log
[oslo_messaging_rabbit]
[oslo_policy]
[profiler]
[quotas]
[trust]
trustee_domain_name = magnum
trustee_domain_admin_name = magnum_domain_admin
trustee_domain_admin_password = secret
trustee_keystone_interface = public
[x509]
EOF'


cd ~
git clone https://git.openstack.org/openstack/python-magnumclient --branch stable/xena
cd python-magnumclient
sudo pip install -e .


sudo vim /etc/magnum/magnum.conf # update host = <current ip>
mysql -h 127.0.0.1 -u root -psecret mysql <<EOF
CREATE DATABASE magnum;
CREATE USER 'magnum'@'%' IDENTIFIED BY 'secret';
GRANT ALL PRIVILEGES ON magnum.* TO 'magnum'@'%';
EOF


openstack user create --domain default --password secret magnum
openstack role add --project service --user magnum admin
openstack service create --name magnum --description "OpenStack Container Infrastructure Management Service" container-infra
openstack domain create --description "Owns users and projects created by magnum" magnum
openstack user create --domain magnum --password secret magnum_domain_admin
openstack role add --domain magnum --user-domain magnum --user magnum_domain_admin admin
openstack subnet set --dns-nameserver 8.8.8.8 private-subnet
# note replace <host ip> below
openstack endpoint create --region RegionOne container-infra public http://<host ip>:9511/v1
openstack endpoint create --region RegionOne keystone internal http://<host ip>/identity


sudo rabbitmqctl add_user magnum secret
sudo rabbitmqctl set_permissions -p / magnum ".*" ".*" "."
sudo magnum-db-manage upgrade
sudo magnum-api # leave this running
sudo magnum-conductor # leave this running


openstack coe cluster template create my-template --image Fedora-CoreOS --external-network public --fixed-network private --fixed-subnet private-subnet --dns-nameserver 8.8.8.8 --network-driver flannel --docker-storage-driver overlay2 --docker-volume-size 10 --master-flavor m1.small --flavor m1.small --coe kubernetes
openstack coe cluster create my-cluster --cluster-template my-template --master-count 1 --node-count 1 --keypair admin
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/Devstack_magnum/Stable_xena&oldid=2097242"