Deployments on kubernetes

From Wikitech
(Redirected from Migrating from scap-helm)
Jump to navigation Jump to search

Deployments on kubernetes happen using helmfile.

Deploying with helmfile

New service

If you are deploying a new service these extra steps apply

  1. If this is a new service, create a service request task under (use the link at the bottom of the description) and follow the steps to be onboarded in the cluster. At the end of that mentioned process you will have a kubernetes namespace created in every cluster named like your service and a set of credentials available on the deployment hosts.

Code deployment/configuration changes

Note that both new code deployments as well as configuration changes are considered a deployment!

  1. Clone deployment-charts repo.
  2. Using your editor modify under the helmfile.d folder of the service you want to modify. As an example, myservice deployment on staging cluster lives under deployment-charts/helmfile.d/services/staging/myservice. Most of the changes are usually made on the values.yaml file to tune the deployment parameters.
  3. If you need to update or add a secret like a password or a certificate ask an SRE to commit it into the private puppet repo do not commit secrets in deployment-charts repo.
  4. Make a CR and after a successful review merge it.
  5. After merge, log in in a deployment server, there is a cron (1 minute) that will update the /srv/deployment-charts directory with the contents from git.
  6. Go to /srv/deployment-charts/helmfile.d/services/${CLUSTER}/${SERVICE} where CLUSTER is one of (staging,eqiad,codfw) and SERVICE is the name of your service i.e myservice.
  7. execute source .hfenv; helmfile -i apply . This will show the changes that it will be applied on the cluster and prompt you to confirm. Then it will materialize the previous diff in the cluster and also will log into SAL the change.
  8. all done!

In case there are multiple releases of your service in the same helmfile, you can use the --selector name=RELEASE_NAME option, e.g. source .hfenv; helmfile --selector name=test status.

Seeing the current status

This is done using helmfile

  1. Change directory to /srv/deployment-charts/helmfile.d/services/${CLUSTER}/${SERVICE} on a deployment server
  2. Unless you are mid un-applied changes the current values files should reflect the deployed values
  3. You can check for unapplied changes with: source .hfenv; helmfile diff
  4. You can see the status with source .hfenv; helmfile status

Rolling back changes

If you need to roll back a change because something went wrong:

  1. Revert the git commit to the deployment-charts repo
  2. Merge the revert (with review if needed)
  3. Wait one minute for the cron job to pull the change to the deployment server
  4. Change directory to /srv/deployment-charts/helmfile.d/services/${CLUSTER}/${SERVICE} where CLUSTER is one of (staging,eqiad,codfw) and SERVICE is the name of your service
  5. execute source .hfenv; helmfile diff to see what you'll be changing
  6. execute source .hfenv; helmfile apply

Rolling back in an emergency

If you can't wait the one minute, or the cron job to update from git fails etc. then it is possible to manually roll back using helm. This is discouraged over using helmfile though.

  1. Find the revision to roll back to
    1. source .hfenv; helm history <production> --tiller-namespace YOUR_SERVICE_NAMESPACE
    2. Find the revision to roll back to
    3. e.g. perhaps the penultimate one
      REVISION        UPDATED                         STATUS          CHART           DESCRIPTION     
      1               Tue Jun 18 08:39:20 2019        SUPERSEDED      termbox-0.0.2   Install complete
      2               Wed Jun 19 08:20:42 2019        SUPERSEDED      termbox-0.0.3   Upgrade complete
      3               Wed Jun 19 10:33:34 2019        SUPERSEDED      termbox-0.0.3   Upgrade complete
      4               Tue Jul  9 14:21:39 2019        SUPERSEDED      termbox-0.0.3   Upgrade complete
  2. Rollback with: source .hfenv; helm rollback <production> 3 --tiller-namespace YOUR_SERVICE_NAMESPACE

Why do I need to source that file?

That is a temporary workaround for helmfile and helm-diff not honoring --helm-home and --kubeconfig flags the moment this tools uses that flags we are going to remove it.

Advanced use cases: using kubeconfig

If you need to use kubeconfig (for a port-forward or to get logs for debugging) you can go to /srv/deployment-charts/helmfile.d/services/${CLUSTER}/${SERVICE} where CLUSTER is one of (staging,eqiad,codfw) and SERVICE is the name of your service i.e myservice. And execute source .hfenv; kubectl COMMAND, e.g. source .hfenv; kubectl logs POD_NAME -c CONTAINER_NAME for logs.

Advanced use cases: using helm

Sometimes you might need to use helm, this is completely discouraged use it only at your own risk and in emergencies.

  • go to /srv/deployment-charts/helmfile.d/services/${CLUSTER}/${SERVICE} where CLUSTER is one of (staging,eqiad,codfw) and SERVICE is the name of your service i.e myservice
  • source .hfenv; helm COMMAND --tiller-namespace YOUR_SERVICE_NAMESPACE