Kubernetes/Kubernetes Workshop/Set up a batch application on Kubernetes

From Wikitech

Overview

At the end of this module, you should be able to:

  • Clone and run an example python batch application, pywpchksumbot.
  • Build and run Docker images on minikube.

Step 1 - Running the pywpchksumbot.py Application Locally

To run a batch application on Kubernetes (k8s), you must first develop the application and package it into a Docker image. This tutorial uses a sample application, pwypchksumbot, which calculates a checksum on the content of the main pages of the English, German, and Greek Wikipedias and publishes the checksum to an [channel]. You can access the IRC channel #demok8sws via Libera Chat's website.

A checksum is a small-sized block of data derived from another block of digital data to detect errors that may have been introduced during the transmission or storage of data. The application's functionality is not essential for the workshop – you can use a different application or implement the pwypchksumbot application in another programming language. However, the pwypchksumbot application has been tested and serves as a good stand-in for a "real" application.

Running the Batch Application Locally

You can run the application on either a local machine (Linux OS, MacOS, or Windows OS), or a Virtual Machine (VM) For the purpose of this tutorial, you should use a Wikimedia Cloud Service (WMCS) VM of 4vCPUs, 8GB RAM, 20GB Disk running Debian buster.

Setting up a Wikimedia Foundation (WMF) Cloud Virtual Private Server (VPS)

Refer to this Wiki for guidelines on how to provision a WMF VPS.

Follow these steps to run the application locally:

  • Open a terminal window and run the following command to complete cloning the GitHub repository to your local machine:
$ git clone https://gitlab.wikimedia.org/repos/sre/pywpchksumbot
Cloning into 'pywpchksumbot'...
……………………………………………………….
Unpacking objects: 100% (6/6), done.
  • Run the Python application:
$ cd pywpchksumbot/
$ python3 pywpchksumbot/
:pywpchksumbot MODE pywpchksumbot :+iw
:pywpchksumbot!~pywpchksu@nat.cloudgw.eqiad1.wikimediacloud.org JOIN #demok8sws
:strontium.libera.chat 353 pywpchksumbot @ #demok8sws :pywpchksumbot guest mutante nikkinikk duesen bpirkle Seddon clarakosi tchin
:strontium.libera.chat 366 pywpchksumbot #demok8sws :End of /NAMES list.
  • View the result on channel #demok8sws on Libera Chat’s website.

After the application runs locally, you can package it as a Docker image and run it under Docker. Running this batch application on a third party VM (e.g. AWS EC2, GCP, Digital Ocean etc) will incur costs.

Hands-on Demo: Setting up a Simple Batch Application Using Docker and Kubernetes (minikube)

In this section, you will be exposed to the following concepts:

  • Methods of building a Docker image.
  • Kubernetes minikube.

Building a Docker Image

A Docker image is a read-only template with instructions for creating a Docker container. In this section, you will learn how to create a Docker image of the pywpchksumbot batch application.

For this tutorial, you will make use of a Dockerfile to create an image of the batch application. Take the steps listed below:

  • Install Docker by referring to this wiki.
  • Create a Dockerfile using any text editor of your choice and type in the code below as its content. Name the file Dockerfile:
FROM python
ADD pywpchksumbot.py /
CMD ["/pywpchksumbot.py"]

The first line defines the base image you will build on. For this tutorial, you will use the image named Python. Docker image registries are the default repository for downloading container images, and Docker Hub is the default registry when none is specified.

There are a few image registries, but Docker Hub is prevalent, having 3.7 million images as of July 2020. WMF has an image registry. You can utilize its images by adding the code block FROM docker-registry.wikimedia.org:xxx to your Dockerfile.

Note: You have to create the Dockerfile in the pywpchksumbot directory.

  • Build and tag your image:
$ docker build -t <your-dockerhub-username>/<given-image-name>:<tag> .
………………………………………………………
Successfully built b4c95e387a8c
  • You can inspect the images available on your system by running:
$ docker image ls
REPOSITORY          TAG       IMAGE ID       CREATED          SIZE
<image-name>        <tag>    b4c95e387a8c   19 minutes ago   920MB
  • You can also conduct an inspection on your Docker image to view the layers of your image:
$ docker image inspect <image_id>

A Docker container is a runnable instance of an image. You can create a container instance of your batch application by running: 

$ docker run <image_name>:<tag>

You will notice that the output of the above code is similar to the output you got when running the application locally.

  • View the result on channel #demok8sws on Libera Chat’s website.

You can access more Docker commands on the Docker cheat sheet.

Running the Docker Image on minikube

There are several ways of running a non-production k8s cluster on a development machine. minikube, a simple k8s distribution, is sufficient for this workshop. minikube runs k8s as a standalone docker setup or inside a virtual machine.

minikube requires a system with at least two CPUs and four Gigabytes of memory to run. Follow these steps to have minikube pull the image from the Docker hub by default:

  • Create an account on Docker Hub.
  • Then you log in to Docker as follows
  • Push the built image to Docker Hub:
$ docker push <dockerhub_username>/<image_name>:<tag>
The push refers to repository [docker.io/<dockerhub_username>/<image_name>]
……………………………………………………….
size: 2426
  • Run the image on minikube, using either of the below commands:
$ minikube start
$ kubectl run <pod_name> --image=<your_account>/<image_name>:<tag> --restart=Never
pod/<pod_name> created

The tag --restart=Never instructs kubernetes to never restart the bot automatically.

  • Wait a few seconds before running the next command to view the available pods:
$ kubectl get pods
  • Fetch the logs from the pod:
$ kubectl logs <pod_name>
  • View the result on the #demok8sws channel on Libera Chat’s website.
  • Shut down minikube when you feel you have a good grasp of this module:
$ minikube stop

Next Module

Module 2: Build a service application on Kubernetes

Previous Module

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Kubernetes/Kubernetes_Workshop/Set_up_a_batch_application_on_Kubernetes&oldid=1998097"