Jump to content

Deployment pipeline/Components

From Wikitech
This page describes the previous deployment pipeline mechanism based on PipelineLib. For information about the current deployment pipeline, which uses GitLab, Blubber, and Kokkuri, see the deployment pipeline documentation on mediawiki.org.

Blubber

Blubber creates Dockerfiles from a higher level description expressed as YAML. The Deployment pipeline relies on an application's Blubberfile(s) to build images for testing and production.

PipelineLib

PipelineLib is a library of code that consumes an application's .pipeline/config.yaml. This configuration directs the pipeline through integration telling it:

  • What blubber.yaml
  • What unit tests to run
  • What integration tests to run
  • Which tests can be run in parallel
  • What container image is to be used in production

Helm

Helm uses Helm charts to define the resources needed to deploy an application to production. Helm manages things like:

  • Config files
  • Resource limits
  • What ports your application uses
  • Smoke tests for your application

In short, Helm allows the easy management of mapping deployment resources to an environment. If you need a different configuration for local development and production, chances are it'll be easy to configure with Helm.

Helmfile

Helmfile is what you will use to deploy your application in production. Helmfile uses the environments and values in the deployment charts repository to define a deployment to a specific environment. A deployment of an existing production service looks like:

$ cd /srv/deployment-charts/helmfile.d/services/blubberoid
$ helmfile -e staging -i apply

Kubernetes

Kubernetes is the execution framework that the pipeline works on. See Kubernetes

Application Metrics monitoring

Monitoring in the pipeline happens via prometheus and exposing via Grafana.

Prometheus

Applications need to either provide Prometheus native endpoints (recommended), or emit Statsd (deprecated) compatible metrics to a local prometheus-statsd-exporter instance. In the latter case, SRE provides a ready to use component as part of any Helm chart in the deployment-charts repo.

In all cases, applications are discovered automatically and scraped by Prometheus.

Grafana

Deployment pipeline applications are currently exposed under the Service folder in grafana.wikimedia.org. See Grafana.wikimedia.org#Pipeline for more