Portal:Toolforge/Admin/Kubernetes/Networking and ingress

From Wikitech
Jump to navigation Jump to search

This page describes the design, topology and setup of Toolforge's networking and ingress, specifically those bits related to webservices in the new kubernetes cluster.

Network topology

The following diagram is used to better describe the network topology and the different components involved.

Toolforge k8s network topology.png

When an user visits a webservice hosted in Toolforge, the following happens:

  1. nginx+lua: The first proxys is what we know as dynamicproxy. The nginx+lua setup knows how to send requests to the legacy webservices grid. There is a fall-through route for the k8s cluster.
    This proxy provides SSL termination for toolforge.org. There is a DNS A record pointing to the floating IP associated with the active proxy VM.
  2. haproxy: Then, haproxy knows which actual ingress worker nodes are alive in the k8s cluster. Proxy for both the k8s api-server on tcp/6443 and the web ingress on tcp/30000.
    There is a DNS A record with name k8s.tools.eqiad1.wikimedia.cloud pointing to the IP of this VM.
  3. nginx-ingress-svc: There is a nginx-ingress service of type NodePort, which means every ingress worker node listens on tcp/30000 and direct request to the nginx-ingress pod.
  4. nginx-ingress pod: The nginx-ingress pod will use ingress objects to direct the request to the appropriate service, but ingress objects need to exists beforehand.
  5. api-server: Ingress objects are created using the k8s API served by the api-server. They are automatically created using the webservice command, and the k8s API allows users to create and customize them too.
  6. ingress-admission-controller: There is a custom admission controller webhook that validates ingress config objects to enforce valid configurations.
  7. ingress object: After the webhook, the ingress objects are added to the cluster and the nginx-ingress pod can consume them.
  8. tool svc: The ingress object contained a mapping between URL/path and a service. The request is now in this tool specific service, which knows how to finally direct the query to the actual tool pod.
  9. tool pod: The request finally arrives at the actual tool pod.


This section contains specific information about the different components involved in this setup.

There are mainly 2 different kinds of elements: those running outside kubernetes and those running inside.

outside kubernetes

Information about components running outside kubernetes.


See Toolforge dynamicproxy

This component is described in another wikitech page.


This setup is fairly simple, deployed by puppet using the role role::wmcs::toolforge::k8s::haproxy.

We have a DNS name k8s.tools.eqiad1.wikimedia.cloud with an A record pointing to the active VM. There should be a couple of VMs in a cold-standby setup (only one is actually working).

The haproxy configuration involves 2 ports, the "virtual servers":

  • 6443/tcp for the main kubernetes api-server
  • 30000/tcp for the ingress

Each "virtual server" has several backends:

  • in the case of the api-server, backends are the controller nodes.
  • in the case of the ingress, backends are ingress worker nodes.

inside kubernetes

Explanation of the different components inside kubernetes.


We use calico as the network overlay inside kubernetes. There is not a lot to say here, since we mostly use the default calico configuration. We only specify the CIDR of the pod network. There is a single yaml file containing all the configuration, and deployed by puppet.

This file is modules/toolforge/templates/k8s/calico.yaml.erb in the puppet tree and /etc/kubernetes/calico.yaml in the final control nodes.

To load (or refresh) the configuration inside the cluster, use:

root@k8s-control-1:~# kubectl apply -f /etc/kubernetes/calico.yaml 
configmap/calico-config unchanged
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org unchanged
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrole.rbac.authorization.k8s.io/calico-node unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged
daemonset.apps/calico-node configured
serviceaccount/calico-node unchanged
deployment.apps/calico-kube-controllers unchanged
serviceaccount/calico-kube-controllers unchanged

Some things to take into account:

  • Mind that all configuration is based on a specific version of calico. By the time of this writting, the version is v3.8.
  • The version of calico we use harcodes calls to iptables-legacy, so in Debian Buster we were forced to switch everything to that, rather than the default iptables-nft. See toolforge::k8s::calico_workaround for details.


We use nginx to process the ingress configuration in the k8s cluster. There is a single yaml file containing all the configuration, and deployed by puppet.

This file is modules/toolforge/files/k8s/nginx-ingress.yaml in the puppet tree and /etc/kubernetes/nginx-ingress.yaml in the final control nodes.

To load (or refresh) the configuration into the cluster, use:

root@k8s-control-01:~# kubectl apply -f /etc/kubernetes/nginx-ingress.yaml 
namespace/ingress-nginx unchanged
configmap/nginx-configuration unchanged
configmap/tcp-services unchanged
configmap/udp-services unchanged
serviceaccount/nginx-ingress unchanged
clusterrole.rbac.authorization.k8s.io/nginx-ingress unchanged
role.rbac.authorization.k8s.io/nginx-ingress unchanged
rolebinding.rbac.authorization.k8s.io/nginx-ingress unchanged
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress unchanged
deployment.apps/nginx-ingress configured
service/ingress-nginx unchanged

Some things to take into account:

  • Mind that all configuration is based on a specific version of kubernetes/ingress-nginx. By the time of this writting, the version is v0.25.1.
  • There are serveral modifications done to the configuration as presented in the docs, to fit into our environments. Changes are docummented in the yaml file.
  • One of the most important changes is that we configured the nginx-ingress pods to listen on 8080/tcp rather than the default (80/tcp).
  • The nginx-ingress deployment requires a very specific PodSecurityPolicy to be deployed beforehand (likely /etc/kubernetes/kubeadm-nginx-ingress-psp.yaml).
  • we don't know yet which scale factor do we need for the nginx-ingress deployment (i.e, how many pods to run).

Worth mentioning that we are using the comunity-based one (https://kubernetes.github.io/ingress-nginx/) rather than the NGINX Inc. one (https://github.com/nginxinc/kubernetes-ingress).

error handling

We have a tool called fourohfour which is set as the default backend for nginx-ingress. This tool presents an user friendly 404 page.

ingress objects

Ingress objects can be created in 2 ways:

  • directly using the kubernetes API
  • by using the webservice command.

Objects have this layout. Toolforge tool name fourohfour is used as example:

apiVersion: extensions/v1beta1
kind: Ingress
    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite ^(/fourohfour)$ $1/ redirect;
    nginx.ingress.kubernetes.io/rewrite-target: /fourohfour/$2
    name: fourohfour
    toolforge: tool
    tools.wmflabs.org/webservice: "true"
    tools.wmflabs.org/webservice-version: "1"
  name: fourohfour
  namespace: tool-fourohfour
  - host: toolsbeta.wmflabs.org
      - backend:
          serviceName: fourohfour
          servicePort: 8000
        path: /fourohfour(/|$)(.*)
  loadBalancer: {}

NOTE: the rewrite/redirect configuration is really important and is part of the behaviour users expect. See phabricator ticket T242719 for example.

This ingress object is pointing to a service which should have this layout:

apiVersion: v1
kind: Service
    name: fourohfour
    toolforge: tool
    tools.wmflabs.org/webservice: "true"
    tools.wmflabs.org/webservice-version: "1"
  name: fourohfour
  namespace: tool-fourohfour
  clusterIP: x.x.x.x
  - name: http
    port: 8000
    protocol: TCP
    targetPort: 8000
    name: fourohfour
  sessionAffinity: None
  type: ClusterIP
  loadBalancer: {}

Note that both objects are namespaced to the concrete tool namespace.

ingress admission controller

This k8s API webhook checks ingress objects before they are accepted by the API itself. It enforces (or prevents) ingress configurations that may produce malfunctioning in the webservices running in kubernetes, like pointing URLs/path to tools that are not ready to handle them.

The code is written in golang, and can be found here:

How to test the setup

See logs for nginx-ingress:

root@tools-k8s-control-3:~# kubectl logs -lapp.kubernetes.io/name=ingress-nginx -n ingress-nginx - [] - - [18/Dec/2019:12:02:21 +0000] "GET /potd-feed/potd.php/commons/potd-400x300.rss HTTP/1.1" 503 2261 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" 402 0.003 [upstream-default-backend] [] 2261 0.000 503 2d08390256b1629bc60552710e4b47e1 - [] - - [18/Dec/2019:12:02:22 +0000] "GET /favicon.ico HTTP/1.1" 404 1093 "https://tools.wmflabs.org/wikisense/Gallery.php?&wikifam=commons.wikimedia.org&img_user_text=Hoanglong2807" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) coc_coc_browser/83.0.144 Chrome/77.0.3865.144 Safari/537.36" 576 0.529 [upstream-default-backend] [] 2116 0.528 404 5fc83ada234e3a5a5da5e42a6583f992 - [] - - [18/Dec/2019:12:02:23 +0000] "GET /mediawiki-feeds/ HTTP/1.1" 503 2236 "https://tools.wmflabs.org/mediawiki-feeds/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)" 487 0.003 [upstream-default-backend] [] 2236 0.004 503 e0341d943aba393e30fc692151da0790

TODO: extend this. Perhaps refactor into its own wiki page?

See also