Kubernetes/Remove a service
All steps below assume you want to remove a service named service-foo from the clusters of the main (wikikube) group.
The ml-serve and dse-k8s groups are modelled closely on the main clusters, so many of the steps below are also applicable to these groups. Where there are specific differences they may be highlighted in the steps below.
Removal of Kubernetes Ingress or LVS is out of scope of this documentation for now.
Example of a removed service: T365742
1. Destroy the service deployments
Staging
deploy1002:# cd /srv/deployment-charts/helmfile.d/services/service-foo
deploy1002:# helmfile -e staging -i destroy
The command above will show a diff of everything that will be removed, make sure that everything looks fine. Hit Yes to proceed.
Production
Note: Ensure you have silenced alarms for your service, and removed LVS if necessary
deploy1002:# cd /srv/deployment-charts/helmfile.d/services/service-foo
deploy1002:# helmfile -e codfw -i destroy
# All well?
deploy1002:# helmfile -e eqiad -i destroy
The service deployment is now destroyed.
2. Private data cleanup
Puppet: remove kubeconfig files
User accounts and tokens need to be removed from the deployment server
hieradata/common/profile/kubernetes/deployment_server.yaml
profile::kubernetes::deployment_server::services
profile::kubernetes::deployment_server::services:
main:
mathoid:
usernames:
- name: mathoid
- name: mathoid-deploy
...
service-foo:
+ ensure: absent
usernames:
- name: service-foo
- name: service-foo-deploy
- Once committed, merged and applied, this will remove the relevant directory and its files in the deployment server.
- Create a second CR that completely removes your service's stanza.
Puppermaster secrets
Applications may have secrets (eg tokens, db user/pass etc), those are stored here
hieradata/role/common/deployment_server/kubernetes.yaml
profile::kubernetes::deployment_server_secrets::services
- Remove the relevant stanza
- same for labs/private repos
4. Remove the kubernetes service definitions
Preparing the commit
1. Removing the namespace
Wikikube:
helmfile.d/admin_ng/values/main.yaml
namespaces:
- example
ml-serve, dse-k8s, and aux-k8s:
- cluster groups are managed in their own files.
2. Remove your deployment
git rm -rf helmfile.d/services/service-foo
3. Review your changes
Commit, and ask somebody from Service Ops (for wikikube) or IF (for aux cluster) to validate and merge.
Deploy changes to helmfile.d/admin_ng
Someone from ServiceOps (for wikikube) or IF (for aux cluster) should be the one running the following:
deploy1002:# sudo run-puppet-agent
deploy1002:# sudo -i
deploy1002:# cd /srv/deployment-charts/helmfile.d/admin_ng/
deploy1002:# helmfile -e staging-codfw -i apply --context 5
# All well?
deploy1002:# helmfile -e staging-eqiad -i apply --context 5
deploy1002:# helmfile -e codfw -i apply --context 5
deploy1002:# helmfile -e eqiad -i apply --context 5
Note: You should be seeing diffs here. If not, something is off!
All Alright?
deploy1002:# sudo -i
deploy1002:# kube_env admin staging-codfw
deploy1002:# kubectl describe ns service-foo
Error from server (NotFound): namespaces "service-foo" not found
Remember to deploy to staging-eqiad, eqiad and codfw clusters even if you aren't ready to fully deploy your service.
5. Remove the chart (if needed)
If your service was the last deployment of a chart, and it will not need to be reused, you can also remove the chart from the deployment-charts repo