Blubber/User Guide

For example use see the Blubber: Getting Started page.

Blubber produces Dockerfiles that can be used to build Docker images. Blubber is written and maintained by the Release Engineering team as part of a toolset for delivering software to testers, developers, testing in CI, and into Wikimedia's production environment. Blubber is part of Wikimedia's Deployment pipeline.

Blubber reads a higher-level specification written in YAML and produces a Dockerfile on stdout. Docker tooling can use the Blubber produced Dockerfile to build a container image. The image can then be used to run an application container. The image is also suitable for use in a helm chart for deployment to a Kubernetes cluster, for development purposes, or in production. The image can be shared with anyone who wants to run the application on their own, perhaps to test, debug, or develop it.

Blubber runs on its own. Blubber is a statically compiled Go binary. It is not directly connected to WMF infrastructure. Blubber does not run Docker itself.

Blubberfiles

Blubber specification files use the YAML format. The top level is a map of key/value pairs. The toplevel keys that Blubber knows about are:

Special Config

These are special configuration keys that may only appear once at the toplevel of a Blubberfile.

version (string)

the version of the Blubber specification file syntax; use v4 or Blubber will be very upset indeed.

variants (map)

value should be a map, whose keys name the variants that can be built; see below section about variants for more details

Common Config

These are common configuration keys that may be included as a variant, or they may appear at the toplevel of a Blubberfile.

base (string)

the base image upon which the new Docker image will be built; a list of available images can be found by querying the Wikimedia Docker Registry

runs (map)

settings for things run in the container, value should be a dict, which can have the following keys:

environment (map)

a map of environment variables to set, variable is the key, value is the...well...value

insecurely (boolean)

should the application in the container be run as a user that can't write anything to the filesystem, including caches, or as a user that can? Production variants should have this set to `false`, but other variants may set it to `true` in some circumstances, for example when enabling caching for ESLint.

apt (map)

settings for apt, has two subkeys

source (sequence)

additional APT sources to add before installing packages. subkeys:

url (string)

the URL of the repository

distribution (string)

see this page for more information about our use of the distribution and component fields.

component (sequence)

packages (sequence)

additional Debian packages to install with the apt command; the value should be a list of package names

Example

apt:
  sources:
    - url: http://apt.wikimedia.org/wikimedia
      distribution: buster-wikimedia
      components: [thirdparty/confluent]
  packages: [ ca-certificates, confluent-kafka-2.11, curl ]

lives (map)

sets WORKDIR within the generated Dockerfile

in (string)

the desired working directory

Variant Config

Variant configuration keys are typically only seen under a particular variant rather than at the toplevel of a Blubberfile. Some of these are general, and some are build configurations.

General

entrypoint (sequence)

command to run as entrypoint of container image as a list

includes (sequence)

inherit all keys from the variants specified by this sequence. The keys will be combined with this variant's keys subject to specific key merge rules

copies (sequence)

copy files from the local build context or from one variant to another (producing a multi-stage build). Note that prior to v4 configuration, copying of local build-context files was implied by the omission of copies but with v4 the configuration must always be explicit. Omitting the field will result in no COPY instructions whatsoever, which may be helpful in building very minimal utility images. subkeys:

from (string)

specify local to copy build-context files that match the source pattern, or another variant name to copy files from the variant's filesystem that match the source pattern.

source (string)

glob pattern that specifies which files to copy

destination (string)

destination within this variant's image for copied files

Builders

These keys correspond to predefined configurations for well-known building tools (node, python and php) and custom builder commands.

Builders come in two flavors: standalone keys (older approach) and grouped builders (more flexible and recommended). Grouped builders defines an order in which the builders should be executed and also allows to define multiple custom builder commands. Both styles can be combined in a single blubber file, but different variants must use different styles; note this also applies when including variants or top-level keys. For more details see the README

Standalone builders

node (map)

configuration related to node/npm environment has the following subkeys

requirements (sequence)

list of files that are needed for npm install to work; e.g., [package.json, package-shrinkwrap.json]

env (string)

"production" or another environment. Sets the environment variable NODE_ENV. Will pass npm install --production and run npm dedupe if set to production.

use-npm-ci (boolean)

Instructs variant to do an npm ci, instead of npm install

python (map)

configuration related to installation of pip dependencies. subkeys:

version (string)

version of python to use; e.g., python3

requirements (sequence)

files to copy to the container before running pip; e.g., [requirements.txt]

poetry (map)

configuration related to installation of pip dependencies using Poetry. subkeys:

version (string)

simplified Python package version specification describing which version of Poetry to install

devel (boolean)

boolean flag indicating whether or not to install development dependencies in the Poetry managed virtual environment

builder (map)

run an arbitrary build command. subkeys:

command (sequence)

command to run as a list, e.g. [make, build]

requirements (sequence)

files to copy to the container for builder to work, either from the local build context or another variant

Note that there are two possible formats for requirements.

The first is a simple shorthand notation that means copy a list of source files from the local build context to a destination of the same relative path in the image. The second is a longhand form that gives more control over the source context (local or another variant), source and destination paths.

Example (shorthand)

builder:
  command: ["some", "build", "command"]
  requirements: [config.json, Makefile, src/] # copy files/directories to the same paths in the image

Example (longhand/advanced)

builder:
  command: ["some", "build", "command"]
  requirements:
    - from: local
      source: config.production.json
      destination: config.json
    - Makefile # note that longhand/shorthand can be mixed
    - src/
    - from: other-variant
      source: /srv/some/previous/build/product
      destination: dist/product

Grouped builders

Grouped builders always appear together under the key builders.

builders (sequence)

run a set of build commands in order. Any of the predefined standalone builder keys (node, python and php) can be specified here, but only one of each type can appear. Additionally, any number of custom keys can appear; their definition and subkeys are the same as the standalone builder key.

--

In addition to the keys listed as Variant Config, any of the properties specified in Common Config may specified under the Variant Config.

Config Merge Rules

When a Variant configuration overrides the Common configuration the configurations are merged. The way in which configuration is merged depends on whether the type of the configuration is a compound type; e.g., a map or sequence, or a scalar type; e.g., a string or integer.

In general, configuration that is a compound type is appended, whereas configuration that is of a scalar type is overridden.

For example in this Blubberfile:

version: v4
base: scratch
apt: { packages: [cowsay] }
variants:
  test:
    base: nodejs
    apt: { packages: [libcaca] }

The base scalar will be overwritten, whereas the apt[packages] sequence will be appended so that both cowsay and libcaca will be installed in an image produced from the Dockerfile resulting from the test Blubberfile variant.

Variants

Blubber can build several variants of an image from the same specification file. The variants are named and described under the variants top level item. Typically, there are variants for development versus production: the development variant might have more debugging tools, for example, which are not included in the production variant, which should have no extra software installed to minimize risk of security problems, or other problems.

A variant is built using the top level items, combined with the items for the variant. So if the top level apt installed some packages, and the variant's apt some other packages, both sets of packages get installed in that variant.

Shorthand copies

To ease the transition from v3 to later configuration formats, and to generally provide a more succinct syntax for the most common file-copying patterns, copies configuration can be provided in either a shorthand and longhand form. The basic rules are:

copies: [local]

Expands to configuration that will copy all local build-context files to the variant image's filesystem:

copies:
 - from: local
   source: .
   destination: .

And

copies: [othervariant]

Expands to configuration that will copy all shared library and project files from the specified variant's filesystem to that of the current one.

copies:
 - from: othervariant
   source: /opt/lib
   destination: /opt/lib
 - from: othervariant
   source: /srv/app # or whatever lives.in is
   destination: /srv/app # same