Kubernetes/Remove a service

All steps below assume you want to remove a service named service-foo from the clusters of the main (wikikube) group.

The ml-serve and dse-k8s groups are modelled closely on the main clusters, so many of the steps below are also applicable to these groups. Where there are specific differences they may be highlighted in the steps below.

Removal of Kubernetes Ingress or LVS is out of scope of this documentation for now.

Example of a removed service: T365742

1. Destroy the service deployments

Staging

deploy1002:# cd /srv/deployment-charts/helmfile.d/services/service-foo
deploy1002:# helmfile -e staging -i destroy

The command above will show a diff of everything that will be removed, make sure that everything looks fine. Hit Yes to proceed.

Production

Note: Ensure you have silenced alarms for your service, and removed LVS if necessary

deploy1002:# cd /srv/deployment-charts/helmfile.d/services/service-foo
deploy1002:# helmfile -e codfw -i destroy

# All well?

deploy1002:#  helmfile -e eqiad -i destroy

The service deployment is now destroyed.

2. Private data cleanup

Puppet: remove kubeconfig files

User accounts and tokens need to be removed from the deployment server

hieradata/common/profile/kubernetes/deployment_server.yaml
- profile::kubernetes::deployment_server::services

profile::kubernetes::deployment_server::services:
  main:
    mathoid:
      usernames:
        - name: mathoid
        - name: mathoid-deploy
...
    service-foo:
+     ensure: absent
      usernames:
        - name: service-foo
        - name: service-foo-deploy

Example CR

Once committed, merged and applied, this will remove the relevant directory and its files in the deployment server.
- Create a second CR that completely removes your service's stanza.

Example CR

Puppermaster secrets

Applications may have secrets (eg tokens, db user/pass etc), those are stored here

hieradata/role/common/deployment_server/kubernetes.yaml
- profile::kubernetes::deployment_server_secrets::services
- Remove the relevant stanza

same for labs/private repos

4. Remove the kubernetes service definitions

Preparing the commit

1. Removing the namespace

Wikikube:

helmfile.d/admin_ng/values/main.yaml
- namespaces:
- example

ml-serve, dse-k8s, and aux-k8s:

cluster groups are managed in their own files.

2. Remove your deployment

git rm -rf helmfile.d/services/service-foo

example

3. Review your changes

Commit, and ask somebody from Service Ops to validate and merge.

After merging, it is important to deploy your changes on all the affected environments to avoid impacting other people rolling out changes later on.

Deploy changes to helmfile.d/admin_ng

Someone from ServiceOps should be the one running the following:

deploy1002:# sudo run-puppet-agent
deploy1002:# sudo -i 
deploy1002:# cd /srv/deployment-charts/helmfile.d/admin_ng/
deploy1002:# helmfile -e staging-codfw -i apply --context 5

# All well?

deploy1002:# helmfile -e staging-eqiad -i apply  --context 5
deploy1002:# helmfile -e codfw -i apply --context 5
deploy1002:# helmfile -e eqiad -i apply --context 5

Note: You should be seeing diffs here. If not, something is off!

All Alright?

deploy1002:# sudo -i
deploy1002:# kube_env admin staging-codfw
deploy1002:# kubectl describe ns service-foo
Error from server (NotFound): namespaces "service-foo" not found

Remember to deploy to staging-eqiad, eqiad and codfw clusters even if you aren't ready to fully deploy your service.
Leaving undeployed things will impede further operations by other people.

5. Remove the chart (if needed)

If your service was the last deployment of a chart, and it will not need to be reused, you can also remove the chart from the deployment-charts repo