Overview

The WMF is moving to deploying services via Docker images to Kubernetes, and pipelinelib was developed to support that.

The WMF CI system supports a form of self-serve for the software we maintain and deploy to our production systems. The basic idea is that each repository contains all configuration necessary for CI to do the following.

Build and containerize all applications implemented in the repo.
Gate applications by executing a dependency graph of pipeline stages, which:
1. Build and run the application's test images.
2. Deploy application images to a testing cluster and perform end-to-end tests.
3. Publish application images to a public WMF Docker registry for subsequent production deployment.

This system makes some assumptions and and has some constraints:

Each repository can specify how it is built and tested, and how the Docker image needed to deploy it is built.
The details of where and the Docker image it deployed are specified to CI by sysadmins, not developer of individual services. Deployment is done using helm and its charts. The charts are provided by SRE or RelEng.
We only support building of Docker images via Blubber, another in-house tool. Docker images may only be based on our own, vetted base images, not arbitrary base images from the Internet.
We publish any Docker images we build to our own Docker registry.
We only support microservices.
For pipelinelib, we don’t care about CI/CD outside of WMF. This is not meant to be a generic solution.

The Blubber software supports Docker images with multiple variants. Typically our projects have variants for building, testing, and using in production.

The repository is expected to have a .pipeline/config.yaml file at its root. That file is read and inspected by CI via pipelinelib. If the file does not exist, CI does nothing for the project.

config.yaml specifies one or more independent pipelines. Each pipeline builds its own Docker image, or variant, which can be its own service, indepenent of any other services built from the same repository. To build (and test) the software, each pipeline specifies a directed acyclic graph or DAG, consisting of stages. Each stage is a step to build the software, test it, or otherwise move towards the goal of getting the software running in production.

Example .pipeline/config.yaml

pipelines:
  test:
    blubberfile: blubber.yaml
    stages:
      - name: test
      - name: candidate
        build: production

  rehearse:
    blubberfile: blubber.yaml
    stages:
      - name: test
      - name: candidate
        build: production
        publish:
          image: true
      - name: rehearsal
        deploy:
          chart: https://releases.wikimedia.org/charts/blubberoid-0.0.9.tgz
          image: '${candidate.imageName}'
          tag: '${candidate.imageTag}'
          test: true

  publish:
    blubberfile: blubber.yaml
    stages:
      - name: test
      - name: candidate
        build: production
        publish:
          image: true
      - name: rehearsal
        deploy:
          chart: https://releases.wikimedia.org/charts/blubberoid-0.0.9.tgz
          image: '${candidate.imageName}'
          tag: '${candidate.imageTag}'
          test: true
      - name: production
        publish:
          image:
            id: '${candidate.imageID}'
            tags: [stable]