In the course of the Liberica project we need to migrate all LVS realservers/services that use our low-traffic LVS (via PyBal) to IPIP encapsulation. On Kubernetes clusters this is usually the cluster workload services as well as the kubernetes control plane API.

Preparations

To ensure IPIP encapsulated packages still fit a 1500 byte MTU, we need to reduce the maximum segment size (MSS) of outgoing packages.

For the Kubernetes Workload this is achieved by reducing the MTU of the Pod veth pairs calico manages to 1460 bytes via calico cni config.

LVS will send IPIP encapsulated packets to an (realserver) address on the physical K8s host itself
The host will have the ipip0 and ipip60 virtual interfaces on it, where the decapsulated packets arrive
The decapsulated packets are routed to the correct Pod from the host
The veth pairs connecting Pods to the default network namespace on the hosts will have MTU set at 1460
This means the PODs will never announce a TCP MSS greater than 1420

For the Kubernetes API this is not feasible since it’s not running within a Pod (there is no veth pair where we could reduce the MTU). Instead we clamp the MSS to 1400 bytes using ferm (iptables) via the profile::lvs::realserver::ipip puppet profile.

Kubernetes API (control plane)

Include profile::lvs::realserver::ipip in the k8s master role (Note: the ...::enabled in the code below is not sufficient).
Add the following hiera settings:

# Enable ipip encapsulation 
profile::lvs::realserver::ipip::enabled: true
# Allow to disable rp_filter on ipip interfaces (by setting all to disabled) and retain strict mode for all other interfaces
profile::base::enable_rp_filter:
  all_rp_filter: 0
  default_rp_filter: 1

When the change has been applied to the relevant k8s master servers, you can check that the realservers accept IPIP traffic for the kubemaster service, before proceeding.
- There is a cookbook that checks this behaviour and you can run it like this: cookbook sre.loadbalancer.check-ipip --dc codfw --query A:wikikube-staging-master-codfw kubestagemaster
You can also carry out a dry-run of the migrate-service-ipip on the kubemaster service.
- cookbook --dry-run sre.loadbalancer.migrate-service-ipip --dc codfw --alias wikikube-staging-master-codfw kubestagemaster
Now switch the service in the service::catalog to IPIP and from Weighted Round Robin to Maglev. The latter is required because Katran, the future load balancer, does not support WRR. Please don't merge it yet.

kubestagemaster: 
  lvs:
    scheduler: mh
    scheduler_flag: mh-port
    ipip_encapsulation:
      - eqiad
      - codfw

Run the sre.loadbalancer.migrate-service-ipip cookbook for your kubemaster service, without the --dry-run option.
- It will ask you to merge the above change as specific step
- It will restart Pybal for you. Please sync with Traffic SREs on #wikimedia-traffic (IRC) before proceeding.

Kubernetes Workload (applications running in containers)

Lower the MTU on calico interfaces to 1460 bytes (already done for all clusters in hieradata/common/kubernetes.yaml)
Include profile::lvs::realserver::ipip in the k8s worker role
Add the following hiera settings:

# Enable ipip encapsulation
profile::lvs::realserver::ipip::enabled: true
# but disable MSS clamping. We reduce the MTU via Calico CNI instead
profile::lvs::realserver::ipip::clamping_enabled: false
# Allow to disable rp_filter on ipip interfaces (by setting all to disabled) and retain strict mode for all other interfaces
profile::base::enable_rp_filter:
  all_rp_filter: 0
  default_rp_filter: 1

Verify the IPIP packages are be handled correctly by dry running the migrate-service-ipip on any service running on the cluster: cookbook --dry-run sre.loadbalancer.migrate-service-ipip --dc codfw --alias wikikube-staging-worker-codfw k8s-ingress-staging
Now each service in the service::catalog can switch to IPIP and from Weighted Round Robin to Maglev. The latter is required because Katran, the future load balancer, does not support WRR. Please don't merge it yet.

blabla-service:
  lvs:
    scheduler: mh
    scheduler_flag: mh-port
    ipip_encapsulation:
      - eqiad
      - codfw

Run the sre.loadbalancer.migrate-service-ipip cookbook without the --dry-run option.
- It will ask you to merge the above change as specific step.
- It will restart Pybal for you. Please sync with Traffic SREs on #wikimedia-traffic (IRC) before proceeding.