Jump to content

Kubernetes/Deployment Charts

From Wikitech

Deployment Charts

All of our existing Kubernetes clusters make use of a single git repository located at operations/deployment-charts:

This repository is organised as follows:

Path Description
/charts Helm charts
/helmfile.d Helmfile deployments
/_scaffold Common templates when bootstrapping a helm chart
/modules Template modules
/jsonschema JSON schemas of all CustomResourceDefinition (CRDs) objects
/custom_deploy.d Any custom deployments that cannot use helm for any reason
*.yaml, *.sh, *.rb Supporting test files and scripts

See the operations/deployment-charts/README for more detailed information.

Helm Charts

A Helm chart is a set of files that describes a packaging together of Kubernetes resources so that they can be easily deployed as a unit and updated programmatically.

Once committed to the master branch, our helm charts are automatically published to ChartMuseum.

Common Templates

We have developed a set of common templates for bootstrapping a helm chart.

Create new chart

To create a new Helm chart, follow the steps below:

Prerequisites

  • Python 3.9+
  • Sextant – our Helm chart management tool (install via pip3 install sextant)


The following are needed for validating:

  • Ruby 3.0+
  • Rake – Ruby task executor
  • Docker or another container engine

Steps

  1. Run ./create_new_service.sh.
  2. Follow the prompts to generate your new chart under charts/<your-chart-name>.
  3. Modify the default chart to meet your service’s needs.
  4. Test and validate your Chart

Using modules

Each generated chart includes:

file/directory description
package.json Lists module dependencies using major.minor versions
package.lock Locks dependencies to exact versions
templates/vendor Contains the correct module versions

Module dependencies are managed with Sextant

Enabling egress to services external to Kubernetes

Install the external-services helm template

In a new chart

The scaffolding should inject the necessary templates automatically. No need to do anything.

In an existing chart

Note: As a prerequisite, you will need to have Sextant installed.

1. In order to enable egress to services living outside of Kubernetes (such as Kafka, Presto, Kerberos, etc), add "base.external-services-networkpolicy:1.0", to your chart's package.json file.

Run the following command to update the chart dependency lockfile, as well as install the base/external-services-networkpolicy template in your chart templates/vendor directory:

# Assuming you're working on charts/superset
$ cd path/to/deployment-charts
$ sextant --modulepath modules vendor charts/superset
INFO:sextant:Copied modules/base/external-services-networkpolicy_1.0.1.tpl => charts/superset/templates/vendor/base/external-services-networkpolicy_1.0.1.tpl

2. Add the following line at the end of your chart's templates/networkpolicy.yaml file

{{ include "base.networkpolicy.egress.external-services" $ }}

3. Add the following lines in your chart's values.yaml file

# The set of external services to allow egress to
# Example:
# kafka:
# - main-codfw
# - main-eqiad
# presto:
# - analytics
#
# See https://wikitech.wikimedia.org/wiki/Kubernetes/Deployment_Charts#Enabling_egress_to_services_external_to_Kubernetes
# for the list of supported services
external_services: {}

4. Finally, bump your chart version in your chart's Chart.yaml file.

Select what services you want to enable egress to

Add the following structure to your chart or release values, by adapting the values associated with each service type:

# Note: only include the services you actually need, and not *all* of them.
# We mainly include them here to help you figure out what's available.
external_services:
  presto: [analytics]
  kerberos: [kdc]
  cas: [idp]
  druid: [analytics, public]
  redis-6379: [misc]
  kafka: [jumbo-eqiad]
  zookeeper: [flink-eqiad]
  cassandra: [ml-cache-a-eqiad]
  hadoop-worker: [analytics]
  hadoop-master: [analytics]
  mariadb: [analytics-meta]
  postgresql: [analytics]

The NetworkPolicy resource(s) allowing egress traffic to reach the specified services will be created next time you redeploy your application. Note: all exposed service types and associated instances are listed in Kubernetes. Connect to the deployment server, and use kubectl to list all services in the external-services namespace.

brouberol@deploy1002:~$ kubectl get services -n external-services -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n'
cas-idp
cas-idp-test
cassandra-analytics-query-service-storage-a-codfw
cassandra-analytics-query-service-storage-a-eqiad
cassandra-analytics-query-service-storage-b-codfw
<snip>

Testing a Chart

Our CI will run a set of tests to validate a chart. In order to do that locally you will need:

  • Python 3.9 or higher
  • python3-yaml linux package or PyYAML (via pip)
  • Ruby 3.0 or higher
  • rake, a ruby make-like utility
  • helm3
  • docker or another container engine

To ensure the validity of our templates, we use two sets of fixtures.

  • fixtures under charts/mychart/.fixtures for various test cases
  • fixtures under .fixtures providing general data (eg mariadb sections, kafka_brokers etc) for misc enviroments (staging, eqiad, etc)
    • run rake refresh_fixtures to get the latest ones from our puppet repo.

Quick tests

To quickly test against a specific fixture: 

helm template \ 
  --values .fixtures/general-eqiad.yaml \
  -f charts/mychart/.fixtures/tls_enabled.yaml \
  charts/mychart \
  --debug

CI tests

To Validate against our CI: 

rake run_locally["check_charts[lint/validate\,mychart]"]

Previewing changes

To see what changes woud be applied to deployments as a result of your change:

rake run_locally["check_deployments[diff\,myservice]"]

Upgrading Charts

TBA

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Kubernetes/Deployment_Charts&oldid=2355830"