Deployment pipeline/Components
Blubber
Blubber creates Dockerfiles from a higher level description expressed as YAML. The Deployment pipeline relies on an application's Blubberfile(s) to build images for testing and production.
PipelineLib
PipelineLib is a library of code that consumes an application's .pipeline/config.yaml. This configuration directs the pipeline through integration telling it:
- What
blubber.yaml - What unit tests to run
- What integration tests to run
- Which tests can be run in parallel
- What container image is to be used in production
Helm
Helm uses Helm charts to define the resources needed to deploy an application to production. Helm manages things like:
- Config files
- Resource limits
- What ports your application uses
- Smoke tests for your application
In short, Helm allows the easy management of mapping deployment resources to an environment. If you need a different configuration for local development and production, chances are it'll be easy to configure with Helm.
Helmfile
Helmfile is what you will use to deploy your application in production. Helmfile uses the environments and values in the deployment charts repository to define a deployment to a specific environment. A deployment of an existing production service looks like:
$ cd /srv/deployment-charts/helmfile.d/services/blubberoid
$ helmfile -e staging -i apply
Kubernetes
Kubernetes is the execution framework that the pipeline works on. See Kubernetes
Application Metrics monitoring
Monitoring in the pipeline happens via prometheus and exposing via Grafana.
Prometheus
Applications need to either provide Prometheus native endpoints (recommended), or emit Statsd (deprecated) compatible metrics to a local prometheus-statsd-exporter instance. In the latter case, SRE provides a ready to use component as part of any Helm chart in the deployment-charts repo.
In all cases, applications are discovered automatically and scraped by Prometheus.
Grafana
Deployment pipeline applications are currently exposed under the Service folder in grafana.wikimedia.org. See Grafana.wikimedia.org#Pipeline for more