Incident documentation/2021-11-02 Cloud VPS networking
document status: in-review
After a kernel upgrade for several Cloud VPS network components (cloudnet, cloudgw servers; see T291813), we found problems with Toolforge NFS in Kubernetes. Later LDAP connections were found to be affected. Eventually it turned out to be a problem with all ingress traffic to the network edge for cloud VMs (except those with floating IPs, which were unaffected). The issue was resolved by rolling back the kernel upgrade.
Impact: For about 1 hour and 40 minutes, Toolforge services and VMs in Cloud VPS may have experienced connectivity issues.
- Incident tracking task, T294853