User:JMeybohm (WMF)/Kubernetes/Ingress
 | Loose collection of notes, don't read! |
Istio 1.8 latest version supported with k8s 1.16
Istio 1.11 tested with k8s 1.16 but not supported
ml currently uses istio 1.9 which is tested but not supported on k8s 1.16
https://istio.io/latest/docs/releases/supported-releases/#support-status-of-istio-releases
IstioOperator Options: https://istio.io/v1.9/docs/reference/config/istio.operator.v1alpha1/
istioctl builtin helm charts: https://github.com/istio/istio/tree/1.9.5/manifests/charts
https://phabricator.wikimedia.org/P17224
TLS Stuff / cert-manager
There is a cert-manager issuer for cfssl: https://github.com/OpenSource-THG/cfssl-issuer (does not look very maintained/active)
But it might be easier (and less dependencies) to generate a new intermediate and use that with the ca-issuer that's build in to cert-manager https://cert-manager.io/docs/configuration/ca/
Could we have namespaces create "kind: Certificate" objects, let cert-manager create the corresponding "kind: Secret" (in the same namespace) and sync (with something like https://github.com/kubeops/kubed)those secrets to the istio-system namespace (for ingressgateway to be able to pick them up)?
Figure out pilotCertProvider setting, https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/ and https://cert-manager.io/docs/usage/kube-csr/
Kubernetes Ingress
By using only the kubernetes ingress resource it is possible to create a fan out ingress and route "foo.example.com/a/*" to a service in namespace "a" while routing "foo.example.com/b/*" to a service in namespace "b". This is done by simply creating a Ingress for "foo.example.com" with a corresponding path matcher.
When the same host and path is specified in two ingress resources, the first one (created) wins.
This means it is not possible to break existing ingress objects while it is possible to "hijack" subpaths of a domain by creating another ingress (if the first one uses subpath matching rather than "/*" ofc.). If a subpath ingress is created first (e.g. "path: /one/*" ad afterwards created match all ingress ("path: /*") is not able to take over "/one/*". But this is pretty dangerous as de-deploying might lead to a different creation order and yield a different result
Ingress gateway as DaemonSet
May spare a network hop and increase resiliency as the gateway would run on every node then. Should be possible with overlays https://istio.io/v1.9/docs/reference/config/istio.operator.v1alpha1/#K8sObjectOverlay:?
components:
ingressGateways:
- name: foo
overlays:
- kind: Deployment
name: foo
patches:
- path: kind
value: Daemonset
Configuration
There are three ways to configure istio-ingressgateway(s) with different levels of granularity and different feature sets
Kubernetes Ingress (Simple)
https://istio.io/v1.9/docs/tasks/traffic-management/ingress/ingress-control/
- Uses Kubernetes default Ingress objects for configuration: https://people.wikimedia.org/~jayme/k8s-docs/v1.16/docs/concepts/services-networking/ingress/
- But supports sharing of hostnames between namespaces (due to the central nature of istio-ingressgateway)
- L7 (HTTP(S)) only
- TSL certificates need to be placed in the namespace of istio-ingressgateway
- Just plain host and path prefix matching
- No advanced features like weight, header matching
- No "role model" or binding restrictions (e.g. allowing namespaces to use a specific hostname only, enforce default policies ...)
- Least number of API objects involved
Kubernetes Gateway API (Medium)
https://istio.io/v1.9/docs/tasks/traffic-management/ingress/gateway-api/
- Uses the "new standard" Kubernetes API which is deemed the successor of Ingress:
- L7 and L4 routing
- TSL certificates need to be placed in the namespace of istio-ingressgateway
- Host and path (prefix and exact) matching
- Advanced features like weight, header matching and modification,...
- "Role model" or binding restrictions (e.g. allowing namespaces to use a specific hostname only, enforce default policies ...).
- API still in v1alpha1 (3rd release), soon to be v1aplpha2
- Needs to be added (CRD) to the cluster
- May be subject to breaking changes in the future (not completely clear to me but I think we should assume)
- Already implemented by some other popular ingress controllers (like ambassador, contour, traefik and HAProxy)
Ingress Gateways (Advanced)
https://istio.io/v1.9/docs/tasks/traffic-management/ingress/ingress-control/
- Uses Istio specific API
- L7 and L4 routing
- TSL certificates can be placed anywhere
- Host and path (prefix and exact) matching
- No "role model" or binding restrictions (e.g. allowing namespaces to use a specific hostname only, enforce default policies ...).
- Advanced features like weight, header matching and modification,...
- Very advanced features like fault injection, circuit breaking, mirroring etc.
- Needs to be added (CRD) to the cluster
- Specific to istio
- High number of API objects involved