Portal:Toolforge/Admin/Kubernetes/Networking and ingress

This page describes the design, topology and setup of Toolforge's networking and ingress, specifically those bits related to webservices in the new kubernetes cluster.

Network topology

The following diagram is used to better describe the network topology and the different components involved.

When an user visits a webservice hosted in Toolforge, the following happens:

This step no longer exists, but the diagram has not been updated yet.
haproxy: A pair of HAProxy instances are responsible for terminating inbound HTTPS traffic for *.toolforge.org and sending it on to the Kubernetes cluster. In addition it proxies port 6443/tcp from internal clients to the Kubernetes API server. There is a keepalived-backed service IP address with the DNS name k8s.tools.eqiad1.wikimedia.cloud, as well as a floating IP mapped to the Keepalived address for public v4 ingress.
nginx-ingress-svc: There is a nginx-ingress service of type NodePort, which means every ingress worker node listens on tcp/30000 and direct request to the nginx-ingress pod.
nginx-ingress pod: The nginx-ingress pod will use ingress objects to direct the request to the appropriate service, but ingress objects need to exists beforehand.
api-server: Ingress objects are created using the k8s API served by the api-server. They are automatically created using the webservice command, and the k8s API allows users to create and customize them too.
ingress-admission-controller: There is a custom admission controller webhook that validates ingress config objects to enforce valid configurations.
ingress object: After the webhook, the ingress objects are added to the cluster and the nginx-ingress pod can consume them.
tool svc: The ingress object contained a mapping between URL/path and a service. The request is now in this tool specific service, which knows how to finally direct the query to the actual tool pod.
tool pod: The request finally arrives at the actual tool pod.

Components

This section contains specific information about the different components involved in this setup.

There are mainly 2 different kinds of elements: those running outside kubernetes and those running inside.

outside kubernetes

Information about components running outside kubernetes.

haproxy

This setup is fairly simple, deployed by puppet using the role role::wmcs::toolforge::k8s::haproxy.

We have a DNS name k8s.tools.eqiad1.wikimedia.cloud with A/AAAA records pointing to a Keepalived VIP on one of the VMs.

The haproxy configuration involves 2 ports, the "virtual servers":

443/tcp for HTTPS traffic
6443/tcp for the main kubernetes api-server

Each "virtual server" has several backends:

in the case of the api-server, backends are the controller nodes.
in the case of the ingress, backends are ingress worker nodes.

inside kubernetes

Explanation of the different components inside kubernetes.

calico

We use calico as the network overlay inside kubernetes. There is not a lot to say here, since we mostly use the default calico configuration. We only specify the CIDR of the pod network. There is a single yaml file containing all the configuration, and deployed by puppet.

This file is modules/toolforge/templates/k8s/calico.yaml.erb in the puppet tree and /etc/kubernetes/calico.yaml in the final control nodes.

To load (or refresh) the configuration inside the cluster, use:

root@k8s-control-1:~# kubectl apply -f /etc/kubernetes/calico.yaml 
configmap/calico-config unchanged
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org unchanged
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrole.rbac.authorization.k8s.io/calico-node unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged
daemonset.apps/calico-node configured
serviceaccount/calico-node unchanged
deployment.apps/calico-kube-controllers unchanged
serviceaccount/calico-kube-controllers unchanged

Some things to take into account:

Mind that all configuration is based on a specific version of calico. By the time of this writting, the version is v3.8.

nginx-ingress

We use nginx to process the ingress configuration in the k8s cluster.

Worth mentioning that we are using the comunity-based one (https://kubernetes.github.io/ingress-nginx/) rather than the NGINX Inc. one (https://github.com/nginxinc/kubernetes-ingress).

Starting from ingress-nginx 0.46.0 we deploy it using Helm. Helm is installed on the control plane nodes and uses the current users kubeconfig file to authenticate.

To deploy the component you can follow the standard toolforge procedure Portal:Toolforge/Admin/Kubernetes#Deploy_new_version.

error handling

We have a tool called fourohfour which is set as the default backend for nginx-ingress. This tool presents an user friendly 404 page.

ingress objects

Ingress objects can be created in 2 ways:

directly using the kubernetes API
by using the webservice command.

Objects have this layout. Toolforge tool name sal is used as example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app.kubernetes.io/component: web
    app.kubernetes.io/managed-by: webservice
    name: sal
    toolforge: tool
  name: sal-subdomain
  namespace: tool-sal
spec:
  ingressClassName: toolforge
  rules:
  - host: sal.toolforge.org
    http:
      paths:
      - backend:
          service:
            name: sal
            port:
              number: 8000
        path: /
        pathType: Prefix

NOTE: the rewrite/redirect configuration is really important and is part of the behaviour users expect. See phabricator ticket T242719 for example.

This ingress object is pointing to a service which should have this layout:

kind: Service
metadata:
  labels:
    app.kubernetes.io/component: web
    app.kubernetes.io/managed-by: webservice
    name: sal
    toolforge: tool
  name: sal
  namespace: tool-sal
spec:
  ports:
  - name: http
    port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    name: sal
  type: ClusterIP

Note that both objects are namespaced to the concrete tool namespace.

ingress admission controller

This k8s API webhook checks ingress objects before they are accepted by the API itself. It enforces (or prevents) ingress configurations that may produce malfunctioning in the webservices running in kubernetes, like pointing URLs/path to tools that are not ready to handle them.

The code is written in golang, and can be found here:

How to test the setup

See logs for nginx-ingress:

root@tools-k8s-control-3:~# kubectl logs -lapp.kubernetes.io/name=ingress-nginx -n ingress-nginx
192.168.50.0 - [192.168.50.0] - - [18/Dec/2019:12:02:21 +0000] "GET /potd-feed/potd.php/commons/potd-400x300.rss HTTP/1.1" 503 2261 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" 402 0.003 [upstream-default-backend] [] 192.168.15.133:8000 2261 0.000 503 2d08390256b1629bc60552710e4b47e1
192.168.34.128 - [192.168.34.128] - - [18/Dec/2019:12:02:22 +0000] "GET /favicon.ico HTTP/1.1" 404 1093 "https://tools.wmflabs.org/wikisense/Gallery.php?&wikifam=commons.wikimedia.org&img_user_text=Hoanglong2807" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) coc_coc_browser/83.0.144 Chrome/77.0.3865.144 Safari/537.36" 576 0.529 [upstream-default-backend] [] 192.168.15.133:8000 2116 0.528 404 5fc83ada234e3a5a5da5e42a6583f992
192.168.15.128 - [192.168.15.128] - - [18/Dec/2019:12:02:23 +0000] "GET /mediawiki-feeds/ HTTP/1.1" 503 2236 "https://tools.wmflabs.org/mediawiki-feeds/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)" 487 0.003 [upstream-default-backend] [] 192.168.15.133:8000 2236 0.004 503 e0341d943aba393e30fc692151da0790
[..]

TODO: extend this. Perhaps refactor into its own wiki page?