Help:Toolforge/Kubernetes

From Wikitech
Jump to navigation Jump to search

This page describes the Toolforge Kubernetes cluster and which programming languages it currently supports (#Available container types).

Overview

Kubernetes (often abbreviated k8s) is a platform for running containers. It is used in Toolforge to isolate Tools from each other and allow distributing Tools across a pool of servers.

You can think about container like a "micro virtual machine" whose only task is to execute a single application. It has its own (minimal) file system and limited CPU and memory resources. In Kubernetes each container is inside a pod, which is what connects the container to the tools directories, the db replicas, the internet, and with other pods.

One characteristic of containers is that, due to the small size, it can not have all packages that you can often find in other Toolforge virtual machines like the tools-login and grid engine nodes. To run your application you either need to choose a pre-built container image that has the packages you need (you can see the images available in the Container images section below) or use the Toolforge Build Service to build a custom container image that contains the packages your application needs.

Kubernetes webservices

The Toolforge webservice command has a --backend=kubernetes mode that will start, stop, and restart containers designed to run web services for various languages. See our Webservice help for more details.

Kubernetes backend has the following options:

  -m MEMORY, --mem MEMORY
                        Set higher Kubernetes memory limit
  -c CPU, --cpu CPU     Set a higher Kubernetes cpu limit
  -r REPLICAS, --replicas REPLICAS
                        Set the number of pod replicas to use


Kubernetes jobs

Every non-trivial task performed in Toolforge (like executing a script or running a bot) should be dispatched to a job scheduling backend (in this case, Kubernetes), which ensures that the job is run in a suitable place with sufficient resources.

The basic principle of running jobs is fairly straightforward:

  • You create a job from a submission server (usually login.toolforge.org)
  • Kubernetes finds a suitable execution node to run the job on, and starts it there once resources are available
  • As it runs, your job will send output and errors to files until the job completes or is aborted.

Jobs can be scheduled synchronously or asynchronously, continuously, or simply executed once.

There are two ways of running jobs on Kubernetes.

Previous to allowing jobs on Kubernetes, Toolforge offered Grid Engine as job scheduling backend.

Namespaces

Each tool has been granted control of a Kubernetes "namespace". Your tool can only create and control objects in its namespace. A tool's namespace is the same as the tool's name with "tool-" appended to the beginning (e.g. tool-admin, tool-stashbot, tool-hay, etc).

You can see monitoring data of your namespace in Grafana, enter in this page and select your namespace in the select box at the top of the page.

Quotas and Resources

On the Kubernetes cluster, all containers run with CPU and RAM limits set. Defaults are set at 0.5 CPU and 512Mi of memory per container. Users can adjust these up to the highest level allowed without any help from an administrator (the top limit is set at 3 CPU and 6Gi of memory per pod) with command line arguments to the webservice and toolforge jobs run commands (--cpu and --mem). By default, the entire tool is limited to 8 CPU and 8Gi of memory total. Both the per-pod and total limits can be increased upon request (see below), however, larger per-pod limits will almost certainly result in delays when starting the pod since they take up a large chunk of an individual worker.

The Toolforge admin team encourages you to try running your webservice with the defaults before deciding that you need more resources. We believe that most PHP and Python3 webservices will work as expected with the lower values. Java webservices will almost certainly need higher limits due to the nature of running a JVM.

The storage size limit of a container, including the image size, is 10GB. That gives you approximately 9GB of free space to use inside the /tmp directory while the container is running, when the container ends all data is deleted. That can be useful to use with some kind of file based database (SQLite, dbm, csv, etc) when working with data that is larger than the memory available. If you need a larger temporary space you can try to use an emptydir volume. For persistent storage use your tool directory (NFS mounted in /data/project) or the toolsdb (SQL server).

Namespace-wide quotas

Your entire tool account can only consume so many cluster resources. The cluster places quota limits on an entire namespace which determine how many pods can be used, how many service ports can be exposed, total memory, total CPU, and others. To view the live quotas that apply to your tool, run toolforge jobs quota (or kubectl describe resourcequotas).

Quota increases

It is possible to request a quota increase if you can demonstrate your tool's need for more resources than the default namespace quota allows. Instructions and a template link for creating a quota request can be found at Toolforge (Quota requests) in Phabricator. Please read all the instructions there before submitting your request.

Take into account that memory / RAM is a peculiar resource in Toolforge. As of this writing, Toolforge Kubernetes worker nodes have either 8GB or 16GB RAM. If the RAM allocated to a single job gets close (or beyond) these numbers, Toolforge Kubernetes may be unable to schedule the tool's workloads. This applies to both webservices and jobs.

Container images

The Toolforge Kubernetes cluster is restricted to loading Docker images published at docker-registry.tools.wmflabs.org (see Portal:Toolforge/Admin/Kubernetes#Docker Images for more information). These images are built using the Dockerfiles in the operations/docker-images/toollabs-images git repository.

Available container types

The webservice command has an optional type argument that allows you to choose which Docker container to run your Tool in.

The newest supported images for each language are the following:

  • bookworm (Debian base image)
  • jdk17
  • node18
  • perl5.36
  • php8.2
  • python3.11
  • ruby3.1
  • tcl8.6

For example to start a webservice using a php8.2 container, run:

webservice --backend=kubernetes php8.2 start

A complete list of images is available from the docker-registry tool which provides a pretty frontend for browsing the Docker registry catalog.

See Help:Toolforge/Build Service for information on an alternative system which allows creating custom container images for a Toolforge tool. That system includes support for installing additional system packages using apt which can be used to support projects which require multiple language runtimes, additional libraries, and other assets which are supported by Debian's packaging system.

PHP

PHP uses lighttpd as a webserver, and looks for files in ~/public_html/.

PHP versions & packages

The latest version of PHP is php8.2. You can view the installed PHP extensions on the phpinfo tool.

PHP Upgrade

To upgrade from PHP 5.6 to php8.2, run the following two commands:

$ webservice stop
$ webservice --backend=kubernetes php8.2 start

To switch back:

$ webservice stop
$ webservice --backend=kubernetes php5.6 start

Running Locally

You may run the container on your local computer (not on Toolforge servers) by executing a command like this:

$ docker run --name toolforge -p 8888:80 -v "${PWD}:/var/www/html:cached" -d docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web sh -c "lighty-enable-mod fastcgi-php && lighttpd -D -f /etc/lighttpd/lighttpd.conf"

Then the tool will be available at http://localhost:8888

Node.js

The Node.js container images contain a version Node.js LTS, NPM and Yarn either packaged by Debian or by Nodesource.

Troubleshooting

"failed to create new OS thread" from kubectl

If kubectl get pods or a similar command fails with the error message "runtime: failed to create new OS thread (have 12 already; errno=11)", use GOMAXPROCS=1 kubectl ... to reduce the number of resources that kubectl requests from the operating system.

The active thread quota is per-user, not per-session or per-tool, so if you have multiple shell sessions open to the same bastion server this will effect the available quota for each of your shells. To check the active running threads for your user, use $ ps -Lf --user $YOUR_SHELL_USERNAME.

Get a shell inside a running Pod

Kubectl can be used to open a shell inside a running Pod: $ kubectl exec -it $NAME_OF_POD -- bash

See Get a Shell to a Running Container at kubernetes.io/docs for more information.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Stay aware of critical changes and plans
Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

See also