PipelineLib/Reference

From Wikitech
Jump to navigation Jump to search

Overview

Reference documentation for PipelineLib configuration and interfaces. For the time being, this article is manually maintained according to the canonical code-generated documentation for available configuration and stage-action exports at [1].

Configuration

The .pipeline/config.yaml file specifies a dict/mapping with the following structure and values.

pipelines: {...}
Required. A dict/mapping of arbitrary pipeline names to each pipeline's distinct configuration.
Example
pipelines:
  foo:
    # foo pipeline definition ...
  bar:
    # ...

Pipelines

Each pipeline is again a dict/mapping, with the following structure:

blubberfile: file
Optional. The name of the Blubber configuration file used to build container image variants for the pipeline. The name is relative to the project's .pipeline/ directory. If not given, the default foo/blubber.yaml is used, where foo is the name of the pipeline.
directory: directory
Optional. The directory, relative to the root of the project, where the application source code used by this pipeline is stored.
fetch: {}
Optional. Git clone/fetch options for the patchset.
shallow: true|false
Optional. Whether to perform a shallow clone/fetch.
Default: true
depth: n
Optional. History depth of the fetch operation if shallow.
Default: 20
submodules: true|false
Optional. Whether to fetch/update submodules. Note this is performed recursively using the same shallow/depth options.
Default: true
stages: [...]
Required. List of stage definitions. (See Stages below for supported stage configuration.)
execution: [...]
Optional. The pipeline's stage execution graph represented as a list of stage adjacency lists. The default execution graph is simply all the defined stages performed serially.
Example
pipelines:
  foo:
    blubberfile: my/blubber.yaml
    directory: src/
    stages:
      - name: built
        # stage actions ...
      - name: functional
      - name: analyzed
      - name: staged
      - name: accepted
    execution:
      # by default, the stage execution graph would be:
      #
      #   built → analyzed → functional → staged → accepted
      #
      # but this defines a branching (parallelized) execution of:
      #
      #            analyzed
      #         ⇗            ⇘
      #   built                staged → accepted
      #         ⇘            ⇗
      #           functional
      #
      - [built, analyzed, staged]  # static follows built, staged follows analyzed
      - [built, functional, staged]  # functional follows built, staged follows functional
      - [staged, accepted]  # accepted follows staging

Stages

Each stage is a dict/mapping that provides a distinct stage name and a number of actions.

name: stageName
Required. Name of the stage. Each stage name must be unique within the pipeline.

Actions

A stage may include one or more of the actions listed below. Should a stage contain more than one action, they will be executed in the same order they are listed here.

Actions may result in dynamic output (image IDs, Helm release names, etc.) bound to predefined variable names. See each action's output variables reference and examples for details.

Setup

A special built-in action that clones the project repo and binds a number of initial output variables for use in subsequent pipeline stages. This action cannot be specified or altered by user configuration.

Output variables
${setup.project}
ZUUL_PROJECT parameter value if getting a patchset from Zuul.
Jenkins JOB_NAME value otherwise.
${setup.projectShortName}
The string after the last forward slash in the ZUUL_PROJECT parameter if getting a patchset from Zuul.
The string after the last forward slash in the Jenkins JOB_NAME value otherwise.
${setup.timestamp}
Timestamp at the start of pipeline execution. Used in image tags, etc.
${setup.imageLabels}
Default set of image labels: jenkins.job, jenkins.build, ci.project, ci.pipeline
${setup.commit}
Git commit SHA of the checked out patchset.
${setup.branch}
Git branch of the checked out patchset.
${setup.remote}
Git remote URL of the checked out patchset.
(Additional job parameters and checkout variables)
In addition to the above, all job parameters and checkout variables are included as well.
For example, if the job is configured with a parameter called "FOO", the parameter value will be available as ${setup.FOO}.

Build

build: variant|{}
Blubber variant name and build context options.
Specifying build: "foo" expands to build: { variant: "foo", context: "." }.
variant
Blubber variant to build
Default: {$.stage} (same as the stage name)
context
Build context directory or URL supported by docker build
Default: "."
excludes
Patterns of files/directories to exclude from the build context. Providing this will result in any local .dockerignore file being overwritten prior to the build and only has an effect when the context is a local directory. Patterns must be valid Docker ignore rules.
Output variables
${[stage].imageID}
ID (SHA256) of built image variant.

Run

run: true|{}
Optional. Run a container using a previously built image variant.
image: id
Optional. A valid image ID (SHA256).
Default: ${.imageID} ID of image built in this stage using a build action.
arguments: []
Optional. List of run-time arguments passed to the variant's entry point.
env: {}
Optional. Environment variables to inject into the container at run time.
credentials: []
Optional. List of Jenkins credentials to inject into the container as environment variables at run time.
- id: string
The id of the credentials in Jenkins.
name: string
The name for the environment variable.
Output variables
${[stage].imageID}
ID (SHA256) of built image variant.
Example
pipelines:
  test:
    stages:
      - name: functional
        build: test
        run: true
Example
pipelines:
  test:
    stages:
      - name: built
        build: test
      - name: functional
        run:
          image: '${built.imageID}'
          arguments: [unittests]
          env:
            VAR1_NAME: ${VALUE}
            VAR2_NAME: 'some value'
          credentials:
            - id: SONAR_API_KEY
              name: SONAR_API_KEY

Publish

publish: {}
Optional. Publish artifacts built by previous stages or this stage's previous actions.
image: true|{}
Optional. Publish an image built by previous stages or this stage's previous actions.
id: imageID
Optional. ID of built image variant to publish.
Default: ${.imageID} Image built by a build: action in this stage.
name: imageName
Optional. Name used for the published image.
Default: ${setup.project} Sanitized project name.
tag: imageTag
Optional. Primary tag added to the published image.
Default: ${setup.timestamp}-${.stage} Setup timestamp followed by this stage's name.
tags: []
Optional. Additional tags added to the published image.
Output variables
${[stage].imageName}
Short name under which the image was published.
${[stage].imageFullName}
Fully qualified name (registry/repository/imageName) under which the image was published.
${[stage].imageTag}
Primary tag under which the image was published.
${[stage].publishedImage}
Fully qualified name and tag (${.imageFullName}:${.imageTag}).
Example
pipelines:
  integration:
    stages:
      - name: built
        build: production
        publish:
          image: true
Example
pipelines:
  integration:
    stages:
      - name: built
        build: production
      - name: published
        publish:
          image:
            id: '${built.imageID}'
            name: my-project-image
            tag: production
            tags: [stable]

Promote

promote: true|[]
Optional. Promote artifacts built by previous stages or this stage's previous actions.
- chart: chartName
Optional. Name of the deployment-chart to promote to.
Default: ${setup.project} Sanitized project name.
environments: []
Optional. List of the environments to promote to.
Default: [] An empty list, indicating all environments.
version: imageTag
Optional. A published image tag.
Default: ${.imageTag} Primary tag of this stage's published image.
Example
pipelines:
  integration:
    stages:
      - name: built
        build: production
        publish:
          image: true
        promote: true
Example
pipelines:
  integration:
    stages:
      - name: built
        build: production
      - name: published
        publish:
          image:
            id: '${built.imageID}'
            name: my-project-image
            tag: production
            tags: [stable]
        promote:
          - chart: my-chart-name
            environments: ['staging']
            version: production

Deploy

deploy: {}
Optional. Deploy an image built by previous stages or this stage's actions to the staging cluster.
chart: chartURL
Required. URL/path to the Helm chart used for deployment.
image: imageRef
Optional. Reference to deployable image.
Default: ${.publishedImage} Reference to an image published by a publish action in this stage.
test: true|false
Optional. Whether helm test should be run following deployment.
Default: true Runs helm test following deployment by default.
overrides: {}
Optional. Additional Helm values to pass to helm install.
Example
pipelines:
  acceptance:
    stages:
      - name: staged
        build: production
        publish:
          image: true
        deploy:
          chart: https://releases.wikimedia.org/charts/my-thing-0.1.0.tar.gz
Example
pipelines:
  acceptance:
    stages:
      - name: candidate
        build: production
        publish:
          image: true
      - name: staged
        deploy:
          chart: https://releases.wikimedia.org/charts/my-project-0.1.0.tar.gz
          image: '${candidate.publishedImage}'
          test: true
      - name: accepted
        build: acceptance-tester
        run:
          arguments: ['https://${staged.releaseName}:8080/']

Export

exports: {}
Optional. Binds new output values from the current stage to variable names that can be used in subsequent stage definitions.
Example
pipelines:
  acceptance:
    stages:
      - name: staged
        build: production
        publish:
          image: true
        deploy:
          chart: https://releases.wikimedia.org/charts/my-project-0.1.0.tar.gz
        exports:
          url: 'https://${.releaseName}:8080/'
      - name: accepted
        build: acceptance-tester
        run:
          arguments: ['${staged.url}']