Jump to content

User:JMeybohm (WMF)/Kubernetes/Ingress

From Wikitech


Istio 1.8 latest version supported with k8s 1.16

Istio 1.11 tested with k8s 1.16 but not supported

ml currently uses istio 1.9 which is tested but not supported on k8s 1.16

https://istio.io/latest/docs/releases/supported-releases/#support-status-of-istio-releases


IstioOperator Options: https://istio.io/v1.9/docs/reference/config/istio.operator.v1alpha1/

istioctl builtin helm charts: https://github.com/istio/istio/tree/1.9.5/manifests/charts

https://phabricator.wikimedia.org/P17224

TLS Stuff / cert-manager

There is a cert-manager issuer for cfssl: https://github.com/OpenSource-THG/cfssl-issuer (does not look very maintained/active)

But it might be easier (and less dependencies) to generate a new intermediate and use that with the ca-issuer that's build in to cert-manager https://cert-manager.io/docs/configuration/ca/

Could we have namespaces create "kind: Certificate" objects, let cert-manager create the corresponding "kind: Secret" (in the same namespace) and sync (with something like https://github.com/kubeops/kubed)those secrets to the istio-system namespace (for ingressgateway to be able to pick them up)?


Figure out pilotCertProvider setting, https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/ and https://cert-manager.io/docs/usage/kube-csr/

Kubernetes Ingress

By using only the kubernetes ingress resource it is possible to create a fan out ingress and route "foo.example.com/a/*" to a service in namespace "a" while routing "foo.example.com/b/*" to a service in namespace "b". This is done by simply creating a Ingress for "foo.example.com" with a corresponding path matcher.

When the same host and path is specified in two ingress resources, the first one (created) wins.

This means it is not possible to break existing ingress objects while it is possible to "hijack" subpaths of a domain by creating another ingress (if the first one uses subpath matching rather than "/*" ofc.). If a subpath ingress is created first (e.g. "path: /one/*" ad afterwards created match all ingress ("path: /*") is not able to take over "/one/*". But this is pretty dangerous as de-deploying might lead to a different creation order and yield a different result

"path: /one/*" is interpreted as "path: /one*"

Ingress gateway as DaemonSet

May spare a network hop and increase resiliency as the gateway would run on every node then. Should be possible with overlays https://istio.io/v1.9/docs/reference/config/istio.operator.v1alpha1/#K8sObjectOverlay:?

components:
  ingressGateways:
    - name: foo
      overlays:
        - kind: Deployment
          name: foo
          patches:
            - path: kind
              value: Daemonset

Configuration

There are three ways to configure istio-ingressgateway(s) with different levels of granularity and different feature sets

Kubernetes Ingress (Simple)

https://istio.io/v1.9/docs/tasks/traffic-management/ingress/ingress-control/

  • Uses Kubernetes default Ingress objects for configuration: https://people.wikimedia.org/~jayme/k8s-docs/v1.16/docs/concepts/services-networking/ingress/
  • But supports sharing of hostnames between namespaces (due to the central nature of istio-ingressgateway)
  • L7 (HTTP(S)) only
  • TSL certificates need to be placed in the namespace of istio-ingressgateway
  • Just plain host and path prefix matching
  • No advanced features like weight, header matching
  • No "role model" or binding restrictions (e.g. allowing namespaces to use a specific hostname only, enforce default policies ...)
  • Least number of API objects involved

Kubernetes Gateway API (Medium)

https://istio.io/v1.9/docs/tasks/traffic-management/ingress/gateway-api/

  • Uses the "new standard" Kubernetes API which is deemed the successor of Ingress:
  • L7 and L4 routing
  • TSL certificates need to be placed in the namespace of istio-ingressgateway
  • Host and path (prefix and exact) matching
  • Advanced features like weight, header matching and modification,...
  • "Role model" or binding restrictions (e.g. allowing namespaces to use a specific hostname only, enforce default policies ...).
  • API still in v1alpha1 (3rd release), soon to be v1aplpha2
  • Needs to be added (CRD) to the cluster
  • May be subject to breaking changes in the future (not completely clear to me but I think we should assume)
  • Already implemented by some other popular ingress controllers (like ambassador, contour, traefik and HAProxy)

Ingress Gateways (Advanced)

https://istio.io/v1.9/docs/tasks/traffic-management/ingress/ingress-control/

  • Uses Istio specific API
  • L7 and L4 routing
  • TSL certificates can be placed anywhere
  • Host and path (prefix and exact) matching
  • No "role model" or binding restrictions (e.g. allowing namespaces to use a specific hostname only, enforce default policies ...).
  • Advanced features like weight, header matching and modification,...
  • Very advanced features like fault injection, circuit breaking, mirroring etc.
  • Needs to be added (CRD) to the cluster
  • Specific to istio
  • High number of API objects involved