PipelineLib/Guides/How to define a golang test pipeline

Welcome to How to define a Go test pipeline, a simple PipelineLib guide to get your Go project patchsets tested and linted.

Prerequisites

• A basic understanding of the Blubber image build tool. The Blubber tutorials can help you with that.
• An understanding of .pipeline/config.yaml basics, namely how to define a simple pipeline that will build a test image variant and run it. Some of the PipelineLib tutorials should help you learn these basics.

And of course, don't hesitate to get in touch with the WMF Release Engineering Team for help with following this guide.

Guide

The basic steps to getting your Go test/lint tasks running in CI using Blubber and PipelineLib are:

1. Define image variants in .pipeline/blubber.yaml that invoke your test and lint entry points.
2. Define a new pipeline that includes stages for building and running your image variants.
3. Tell our CI system about your new pipeline by following the How to configure CI for your project supplemental guide.

Define test and lint image variants

dev@laptop:~$ cd src/project/foo

dev@laptop:foo$ mkdir -p .pipeline/
dev@laptop:foo$ vim .pipeline/blubber.yaml

version: v4

variants:
  build:
    base: docker-registry.wikimedia.org/golang:latest
    apt: { packages: [ca-certificates, g++, gcc, git] }
    builder:
      requirements: [go.mod, go.sum]
      command:
        - sh
        - -c
        - >-
          go mod download &&
          go mod verify &&
          go get -u golang.org/x/lint/golint
    copies: [local]
    runs:
      insecurely: true
  lint:
    includes: [build]
    entrypoint: [golint, ./...]
  test:
    includes: [build]
    entrypoint: [go, test, ./...]

This .pipeline/blubber.yaml gives you two runnable image variants, one that invokes golint ./... and another that invokes go test ./.... In addition, both variants use a base build variant that installs commonly needed development packages from our APT repository and downloads/verifies your project's Go module dependencies.

Define a simple test pipeline

Now that you have two variants for linting and testing, define a test pipeline that builds and runs them.

dev@laptop:foo$ vim .pipeline/config.yaml

pipelines:
  test:
    blubberfile: blubber.yaml
    stages:
      - name: lint
      - name: test

(Optionally) Perform lint and test in parallel

Depending on your testing requirements, you may wish to run these two (or more) tasks in parallel. Configuration of execution is how you accomplish that.

dev@laptop:foo$ vim .pipeline/config.yaml

pipelines:
  test:
    blubberfile: blubber.yaml
    stages:
      - name: build
        build: build
        run: false
      - name: lint
      - name: test
    execution:
      - [build, lint]
      - [build, test]

This configuration essentially means: There are three distinct stages in two branches of execution. Execute build first, then branch out and execute lint and test independently in parallel. As a visual graph, it ends up looking something like:

     [set-up]
         ↓
       build
     ⇙       ⇘
lint           test
     ⇘       ⇙
    [tear-down]

Next up

Now that you have a test pipeline defined for your project, head on over to How to configure CI for your project to tell our CI how to run it for each patchset submitted to Gerrit.