Kubernetes/Remove a service

All steps below assume you want to remove a service named service-foo from the clusters of the main (wikikube) group.

The ml-serve and dse-k8s groups are modelled closely on the main clusters, so many of the steps below are also applicable to these groups. Where there are specific differences they may be highlighted in the steps below.

Removal of Kubernetes Ingress or LVS is out of scope of this documentation for now.

Destroy the service deployments

Kubernetes

• Clusters
• Docker Image building
• Deployments
• Deployment Charts
• Enabling TLS
• kubectl
• kubectl/Cheat Sheet
• Service ports
• Education and learning resources
• Troubleshooting

Policies and administration

• Administration
• Add a new service
• Remove a service
• Creating a new cluster
• Infrastructure upgrade policy
• Upstream Helm charts policy
• Packages

Infrastructure

• Calico
• cert-manager
• ChartMuseum
• Deployment pipeline
• Docker-registry
• Dragonfly
• Helm packaging
• Ingress
• Istio
• Logging
• Metrics

Local development

• Minikube
• Local-charts

Related

• Kubernetes on Toolforge
• Mirror of the Kubernetes v1.16 docs
• Mediawiki On Kubernetes

[]

Destroy the service deployment in staging

On deploy1002:

cd /srv/deployment-charts/helmfile.d/services/service-foo
helmfile -e staging -i destroy

The command above will show a diff of everything that will be removed, make sure that everything looks fine and then hit Yes to proceed.

Destroy the service deployment in production

1. Ensure you have silenced alarms for your service, and removed LVS if necessary.
2. On deploy1002:

   cd /srv/deployment-charts/helmfile.d/services/service-foo
   helmfile -e codfw -i destroy
   # if that went fine
   helmfile -e eqiad -i destroy
   

The service deployment is now destroyed.

Private data cleanup

Remove the service's accounts and tokens

User accounts and tokens need to be removed from the puppet private and labs/private repos.

Edit the hieradata/common/profile/kubernetes.yaml file in each repository - specifically the profile::kubernetes::infrastructure_user key, and remove your service's stanza. Example diff

Remove your service's other secrets

Edit profile::kubernetes::deployment_server_secrets::services in the private repository (hieradata/role/common/deployment_server/kubernetes.yaml) and remove the relevant configuration. Example diff

Example CR

Remove your service's certificates

Remove your services stanzas from ./modules/secret/secrets/certificates/certificate.manifests.d/kube_services.certs.yaml on the private repository as well as the certificate directory ./modules/secret/secrets/certificates/service-foo.discovery.wmnet and commit.

Remove them from the puppet CA on the puppetmaster: puppet ca cert clean service-foo.discovery.wmnet

Remove the kubernetes service definitions

Preparing the change

Removing the namespace

In order to remove a namespace, prepare a change to the relevant values file in the the deployment-charts repo.

i.e. for the wikikube clusters this is: helmfile.d/admin_ng/values/main.yaml but namespaces for the ml-serve, dse-k8s, and aux-k8s cluster groups are managed in their own files. Here is an example commit for removing a namespace from the wikikube clusters.

Remove your deployment

git rm -rf helmfile.d/services/service-foo

Example commit

Review your changes

Commit and create a CR.

At this point, you can safely merge the changes (after somebody from Service Ops validates).

After merging, it is important to deploy your changes on all the affected environments to avoid impacting other people rolling out changes later on.

Deploy changes to helmfile.d/admin_ng

The following example shows how to deploy these changes to the wikikube clusters. If you are working with a different cluster group, substitute the relevant environment names.

ssh to deploy1002 and then run the following:

sudo run-puppet-agent
sudo -i
cd /srv/deployment-charts/helmfile.d/admin_ng/
helmfile -e staging-codfw -i apply
# if that went fine
helmfile -e staging-eqiad -i apply
helmfile -e codfw -i apply
helmfile -e eqiad -i apply

The command above should show you a diff in namespaces/quotas/etc.. related to the removal of your service. If you don't see a diff, or something seems off, ping somebody from the Service Ops team! Check that everything is ok:

sudo -i
kube_env admin staging-codfw
kubectl describe ns service-foo
Error from server (NotFound): namespaces "service-foo" not found

Remember to deploy to staging-eqiad, eqiad and codfw clusters even if you aren't ready to fully deploy your service.
Leaving undeployed things will impede further operations by other people.

Tell the deployment server to remove the kubeconfig files.

Removal

This is done by modifying the profile::kubernetes::deployment_server::services hiera key (hieradata/common/profile/kubernetes/deployment_server.yaml) as in the example below:

profile::kubernetes::deployment_server::services:
  main:
    mathoid:
      usernames:
        - name: mathoid
        - name: mathoid-deploy
...
    service-foo:
+     ensure: absent
      usernames:
        - name: service-foo
        - name: service-foo-deploy

Example CR

Once commited, merged and applied, this will remove the relevant directory and its files.

Cleanup

Create a second CR that completely removes your service's stanza. Example CR

Remove the certificates (for the services proxy)

Remove the certificates from the private puppet repo and the references in modules/secret/secrets/certificates/certificate.manifests.d/kube_services.certs.yaml and commit

git rm modules/secret/secrets/certificates/service-foo.discovery.wmnet

Remove the certificates from the puppet CA

sudo -i
puppet cert clean service-foo.discovery.wmnet

Remove the chart (if needed)

If your service was the last deployment of a chart, and it will not need to be reused, you can also remove the chart from the deployment-charts repo