Kubernetes/Kubernetes Workshop/Step 6

Step 6: A “real” cluster

So far we have used minikube as our testcluster. In this step we will first setup a “real” kubernetes cluster.

There are a number of tools to setup up kubernetes clusters, including kubeadm and rancher. There is also a completely manual way as well (kubernetes the hard way™) and many methods that are in between. Here we will use kubespray to setup a cluster. Kubespray uses ansible to configure a set of machines into a kubernetes cluster. The machines can be bare metal or VMs, in our lab case we use VMs that are hosted at Wikimedia’s cloud service Horizon. You will need access to a project under Horizon with the ability to create 3 VMs with a total of 6 CPus and of 12 GB. We have also tested at Digitalocean with smaller machines - 1 CPU, 2GB each and that worked as well, at least for our simple tests.

Hands-on: create a cluster with kubespray

We will use kubespray largely as a black box to create a cluster, we will not go into any advanced use cases.

Log into horizon.wmflabs.org, select your project and launch 3x VMs:

name: node1, node2 and node3. These names are relevant, with other names you will get puppet errors, as kubespray changes the hostname to node1, node2 and node3.
source: debian.buster
flavor: g2.cores2.ram4.disk40
Security group default: we will need SSH access
Server groups: none

Once launched you will have to be able to ssh into each VM by its IP address, a kubespray requirement! You should do this from a 4th VM in Horizon, which is probably the most straightforward way to do, as connectivity works and anything that needs to get installed will not be done on your local machine. You can also do it from your local machine, to make it work from my laptop I had to add 172.16.*.* in the wmflabs stanza in .ssh/config on my laptop.

Host 172.16.* *.wmflabs *.wikimedia.cloud
   User <Username>
   ProxyJump bastion.wmcloud.org:22
   IdentityFile ~/.ssh/labs.key

Try each host and accept the host key.

ssh 172.16.0.199 and ssh 172.16.0.204 and 172.16.1.242 in this test case. Remember your IPs may differ and you should interpret all mentions of IPs with that in mind and substitute the IPs you got.

Clone kubespray to the VM or your local machine:

git clone https://github.com/kubernetes-sigs/kubespray.git

Add the required software:

cd kubespray; pip3 install -r requirements.txt

Add your hosts to the mycluster inventory configuration by running the inventory.py script:

cp -rfp inventory/sample inventory/mycluster
IPS=(172.16.1.242 172.16.0.204 172.16.0.199);CONFIG_FILE=inventory/mycluster/hosts.yml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

inventory/mycluster/hosts.yml:

all:
 hosts:
   node1:
     ansible_host: 172.16.1.242
     ip: 172.16.1.242
     access_ip: 172.16.1.242
   node2:
     ansible_host: 172.16.0.204
     ip: 172.16.0.204
     access_ip: 172.16.0.204
   node3:
     ansible_host: 172.16.0.199
     ip: 172.16.0.199
     access_ip: 172.16.0.199
 children:
   kube-master:
     hosts:
       node1:
       node2:
   kube-node:
     hosts:
       node1:
       node2:
       node3:
   etcd:
     hosts:
       node1:
       node2:
       node3:
   k8s-cluster:
     children:
       kube-master:
       kube-node:
   calico-rr:
     hosts: {}

Install Kubernetes - this will take a while:

ansible-playbook -i inventory/mycluster/hosts.yml -u <your user> -b cluster.yml # <your user> = your shell user name that you used before to login

Test access:

ssh 172.16.1.242
sudo su -
kubectl get nodes

kubectl get nodes
NAME    STATUS   ROLES    AGE     VERSION
node1   Ready    master   11m     v1.19.2
node2   Ready    master   11m     v1.19.2
node3   Ready    <none>   9m46s   v1.19.2

kubectl get nodes -o wide
NAME    STATUS   ROLES    AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION   CONTAINER-RUNTIME
node1   Ready    master   10m     v1.19.2   172.16.1.242   <none>        Debian GNU/Linux 10 (buster)   4.19.0-6-amd64   docker://19.3.12
node2   Ready    master   10m     v1.19.2   172.16.0.204   <none>        Debian GNU/Linux 10 (buster)   4.19.0-6-amd64   docker://19.3.12
node3   Ready    <none>   8m56s   v1.19.2   172.16.0.199   <none>        Debian GNU/Linux 10 (buster)   4.19.0-6-amd64   docker://19.3.12

Kubespray has built a kubernetes cluster with 3 nodes - 2 control plane and 1 worker node.

The 2 control plane nodes run the etcd database, the api-server, the scheduler and the controller-manager. The worker nodes run the kubelet and the kube-proxy. Note that the control plane nodes are also worker nodes in this limited configuration, i.e. kubernetes can also schedule pods to run on the control plane nodes.

Modified Kubernetes Architecture Chart

For an overview of the kubernetes architecture watch Joe Breda’s talk: https://drive.google.com/file/d/192ewPeKLzXM4EUsO3ksfiIGcvzlpec6f/view, if you are with limited time start at 13:30 forward, else watch it all.

You can verify what is running on the individual nodes by ssh’ing into them and check for the processes of etcd, apiserver, scheduler, controller-manager and kubelet, kube-proxy. Check all three nodes and see what the differences are…

Hands-on: run a simple workload

You can run a simple workload on this kubernetes cluster by ssh’ing into node1 and running the steps for our checksum generating program.

Create the yaml file on node1 and run kubectl create -f cron_mywpchksumbot.yaml as root.

Don’t forget to substitute the <userid> for the username you have used on dockerhub.

cron_mywpchksumbot.yaml:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
 name: cronpywpchksumbot
spec:
 schedule: "*/5 * * * *"
 jobTemplate:
   spec:
     template:
       spec:
         containers:
         - name: pywpchksumbot
           image: <userid>/pywpchksumbot
           imagePullPolicy: IfNotPresent
         restartPolicy: OnFailure

See if the process is posting the checksums on the IRC channel.(##demok8sws ^connect on Libera.chat)

Stop the process: kubectl delete cronjob --all

Hands-on: run a simple service workload

Now let’s run a simple service again, our baseapache image.

baseapache.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: baseapache
 labels:
   app: baseapache
spec:
 replicas: 1
 strategy:
   type: RollingUpdate
 selector:
   matchLabels:
     app: baseapache
 template:
   metadata:
     labels:
       app: baseapache
   spec:
     containers:
      - name: baseapache
        image: <userid>/baseapache:latest
        imagePullPolicy: Always

On node1 (as root)

kubectl create -f baseapache.yaml
kubectl get pods -o wide

Create the corresponding service.

baseapacheservice.yaml:

kind: Service
apiVersion: v1
metadata:
 name: baseapache
spec:
 selector:
   app: baseapache
 ports:
 - protocol: TCP
   port: 80
   targetPort: 80

kubectl create -f baseapacheservice.yaml
On node3 we can use curl to get the page
- curl 10.233.41.127
- The IP ^^^ comes from kubectl get svc on node1

SSH into the node and let’s see what we can find out about the pod:

ps -ef | grep apache2
who is the parent process for apache2?
In the parent process there is a workdir on the command line, take a look at that:

Docker uses the overlay filesystem to represent the layers in a docker image. For example when we use “FROM ubuntu” in our baseapache docker image that means that we base our image on the ubuntu image and that any changes get captured in one or more layers. You can see the layers with “docker inspect <imagename>”. Try it on node3 (or wherever the image is running)

docker image inspect <userid>/baseapache

We can use the filesystem ids in the baseapache docker image to identify the filesystem that is used by that pod.

docker image inspect <userid>/baseapache
- Get the “MergedDir” id (after /var/lib/docker/overlay2/
- Grep in /var/lib/docker/overlay2/l for the id: ls -l | grep <id> and get the mapped string as the new id
- Find the mounted filesystem by mount -l | grep <new id>
  - That is the filesystem used by the container
  - cat /var/lib/docker/overlay2/<hex string found>/merged/var/www/html/index.html
- We can use the filesystem to look at apache logs (substitute the right filesystem id
  - tail -f /var/lib/docker/overlay2/0a17a81f309e6d7e4e5afdb83bea72519d31beb2ea99d725390a60b8528c6cf4/merged/var/log/apache2/access.log
- We can edit the index.html file here, which would change the container image in a non-controlled way: not recommended, but nevertheless possible
  - Edit the index.html file
  - curl again to see the change

Hands-on: Browser access to our service

In WMCS one can define web proxies to allow external access to a service. Two configuration need to be created and adjusted:

Web proxy under Menu item DNS
Security group under Menu item Network

With "kubectl get svc" get the port mapped for our service:

root@node1:~# kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
baseapache   NodePort    10.233.3.209   <none>        80:31818/TCP   14m

Here 31818 is the port that we are looking for. A "curl http://172.16.1.242:31818" should work and give us a response from our service

Define a webproxy for that port:

Note that the name baseapache might be taken by another instance of this tutorial. You will get an Error: Duplicate Recordset message in that case

In addition the firewall for the project needs to allow the port, add a rule to the security group that you are using, might be called the Default group if you did not select anything specific when creating the VMs.

Now external access should work. A browser pointed at http://baseapache.wmcloud.org should get you an answer from the service.

You can add more web proxies for other services that might run on the kubernetes cluster by following the same sequence.

Hands-on: use kubectl locally

Often we want to use kubectl from our local workstation to create and delete deployments. To do so we need to change the config file (in ~/.kube/config) and point it to the new cluster. Our local workstation then needs access to the api-server on the cluster. Since the cluster runs in the WMCS cloud behind a bastion host the only way I have found to do that is to setup an SSH tunnel.

Make a backup copy of the local config file
Copy the config file from node1
Setup the SSH tunnel: (172.16.1.242 is the IP for node1)
- ssh -L 6443:172.16.1.242:6443 172.16.1.242
Set the server: field to https://127.0.0.1:6443 in the config file
Test via: kubectl get nodes
On our digitalocean test this can work directly, without any ssh forwarding. Copy the config file as detailed above, but use the IP address of node1 in the config file instead of 127.0.0.1 and make sure port 6443 is accessible in the associated firewall

Now let’s run our database deployment on the machine, bookdb (deployment and service), bookapp (configmap, secrets, deployment and service) and check its proper functioning.

Hands-on: use kubectl locally with Floating IPs

Horizon has Floating IPs that can be forwarded to a local IP. If we take a floating IP and forward it to one of the API servers and open the firewall for port 6443 we can access the k8s cluster directly. Just copy the .kube/config fiel from node1 and use it locally, substitute the 127.0.0.1 address with the floating IP address. This can be useful if you want to test another application's access to the cluster, for example GitLab runners.

PS: Floating IPs have to requested and take some time to approve

Hands-on: use Google’s Google Kubernetes Engine (GKE)

Let’s try running our database at Google’s GKE .

Login into the Google Cloud console: gkeuser.wikimedia@gmail.com/1Jdr5&VWsffd
Click on Kubernetes Engine/Clusters, then Create Cluster
- Defaults here are ok: name, zone and version
- You can change the default node type to an N1 machine to save some money. It will probably only be pennies as the plan is to delete the cluster right after testing…
Click connect
- In a terminal, install the gcloud CLI - cloud shell will not work for what we want to do...
  - https://cloud.google.com/sdk/docs/quickstart
  - You can do this on a VM on your machine, a VM in the cloud (Horizon works) or your local machine
- gcloud auth login with user above
- gcloud container clusters get-credentials cluster-2 --zone us-central1-c --project constant-host-290520 or similar
- kubectl get nodes should now work

First let’s see if our IRC bot works.

kubectl create -f cron_pywpchksumbot.yaml
Check the IRC channel

Now let’s run our database deployment on the machine, bookdb (deployment and service), bookapp (configmap, secrets, deployment and service) and check its proper functioning.

Btw GKE has a networking part that is integrated with kubernetes, something that we have not configured yet, so the bookdbapp gets an outside address and is accessible through the Internet.

At the end delete the cluster.