Blubber/User Guide
This page is currently a draft. More information and discussion about changes to this draft on the talk page. This is the beautiful and perfect and absolutely correct and error-free documentation for Blubber. If reality disagrees, reality is unambiguously wrong and needs to be replaced. (Not really. Please update and fix this wiki page instead.) |
For example use see the Blubber: Getting Started page.
Blubber produces Dockerfiles that can be used to build Docker images. Blubber is written and maintained by the Release Engineering team as part of a toolset for delivering software to testers, developers, testing in CI, and into Wikimedia's production environment. Blubber is part of Wikimedia's Deployment pipeline.
Blubber reads a higher-level specification written in YAML and produces a Dockerfile on stdout. Docker tooling can use the Blubber produced Dockerfile to build a container image. The image can then be used to run an application container. The image is also suitable for use in a helm chart for deployment to a Kubernetes cluster, for development purposes, or in production. The image can be shared with anyone who wants to run the application on their own, perhaps to test, debug, or develop it.
Blubber runs on its own. Blubber is a statically compiled Go binary. It is not directly connected to WMF infrastructure. Blubber does not run Docker itself.
Blubberfiles
Blubber specification files use the YAML format. The top level is a map of key/value pairs. The toplevel keys that Blubber knows about are:
Special Config
These are special configuration keys that may only appear once at the toplevel of a Blubberfile.
version
(string)- the version of the Blubber specification file syntax; use
v4
or Blubber will be very upset indeed. variants
(map)- value should be a map, whose keys name the variants that can be built; see below section about variants for more details
Common Config
These are common configuration keys that may be included as a variant, or they may appear at the toplevel of a Blubberfile.
base
(string)- the base image upon which the new Docker image will be built; a list of available images can be found by querying the Wikimedia Docker Registry
runs
(map)- settings for things run in the container, value should be a dict, which can have the following keys:
environment
(map)- a map of environment variables to set, variable is the key, value is the...well...value
insecurely
(boolean)- should the application in the container be run as a user that can't write anything to the filesystem, including caches, or as a user that can? Production variants should have this set to `false`, but other variants may set it to `true` in some circumstances, for example when enabling caching for ESLint.
apt
(map)- settings for apt, has two subkeys
source
(sequence)- additional APT sources to add before installing packages. subkeys:
url
(string)- the URL of the repository
distribution
(string)- see this page for more information about our use of the distribution and component fields.
component
(sequence)
packages
(sequence)- additional Debian packages to install with the
apt
command; the value should be a list of package names- Example
apt: sources: - url: http://apt.wikimedia.org/wikimedia distribution: buster-wikimedia components: [thirdparty/confluent] packages: [ ca-certificates, confluent-kafka-2.11, curl ]
lives
(map)- sets WORKDIR within the generated Dockerfile
in
(string)- the desired working directory
Variant Config
Variant configuration keys are typically only seen under a particular variant rather than at the toplevel of a Blubberfile. Some of these are general, and some are build configurations.
General
-
entrypoint
(sequence) - command to run as entrypoint of container image as a list
-
includes
(sequence) - inherit all keys from the variants specified by this sequence. The keys will be combined with this variant's keys subject to specific key merge rules
-
copies
(sequence) - copy files from the
local
build context or from one variant to another (producing a multi-stage build). Note that prior to v4 configuration, copying of local build-context files was implied by the omission ofcopies
but with v4 the configuration must always be explicit. Omitting the field will result in noCOPY
instructions whatsoever, which may be helpful in building very minimal utility images. subkeys:from
(string)- specify
local
to copy build-context files that match thesource
pattern, or another variant name to copy files from the variant's filesystem that match thesource
pattern. source
(string)- glob pattern that specifies which files to copy
destination
(string)- destination within this variant's image for copied files
Builders
These keys correspond to predefined configurations for well-known building tools (node, python and php) and custom builder commands.
Builders come in two flavors: standalone keys (older approach) and grouped builders (more flexible and recommended). Grouped builders defines an order in which the builders should be executed and also allows to define multiple custom builder commands. Both styles can be combined in a single blubber file, but different variants must use different styles; note this also applies when including variants or top-level keys. For more details see the README
Standalone builders
-
node
(map) - configuration related to node/npm environment has the following subkeys
requirements
(sequence)- list of files that are needed for
npm install
to work; e.g.,[package.json, package-shrinkwrap.json]
env
(string)- "production" or another environment. Sets the environment variable
NODE_ENV
. Will passnpm install --production
and runnpm dedupe
if set toproduction
. use-npm-ci
(boolean)- Instructs variant to do an
npm ci
, instead ofnpm install
-
python
(map) - configuration related to installation of pip dependencies. subkeys:
version
(string)- version of python to use; e.g.,
python3
requirements
(sequence)- files to copy to the container before running pip; e.g.,
[requirements.txt]
poetry
(map)- configuration related to installation of pip dependencies using Poetry. subkeys:
version
(string)- simplified Python package version specification describing which version of Poetry to install
devel
(boolean)- boolean flag indicating whether or not to install development dependencies in the Poetry managed virtual environment
-
builder
(map) - run an arbitrary build command. subkeys:
command
(sequence)- command to run as a list, e.g.
[make, build]
requirements
(sequence)- files to copy to the container for builder to work, either from the local build context or another variant
- Note that there are two possible formats for
requirements
. - The first is a simple shorthand notation that means copy a list of source files from the local build context to a destination of the same relative path in the image. The second is a longhand form that gives more control over the source context (local or another variant), source and destination paths.
- Example (shorthand)
builder: command: ["some", "build", "command"] requirements: [config.json, Makefile, src/] # copy files/directories to the same paths in the image
- Example (longhand/advanced)
builder: command: ["some", "build", "command"] requirements: - from: local source: config.production.json destination: config.json - Makefile # note that longhand/shorthand can be mixed - src/ - from: other-variant source: /srv/some/previous/build/product destination: dist/product
Grouped builders
Grouped builders always appear together under the key builders.
-
builders
(sequence) - run a set of build commands in order. Any of the predefined standalone builder keys (node, python and php) can be specified here, but only one of each type can appear. Additionally, any number of custom keys can appear; their definition and subkeys are the same as the standalone builder key.
--
In addition to the keys listed as Variant Config, any of the properties specified in Common Config may specified under the Variant Config.
Config Merge Rules
When a Variant configuration overrides the Common configuration the configurations are merged. The way in which configuration is merged depends on whether the type of the configuration is a compound type; e.g., a map or sequence, or a scalar type; e.g., a string or integer.
In general, configuration that is a compound type is appended, whereas configuration that is of a scalar type is overridden.
For example in this Blubberfile:
version: v4
base: scratch
apt: { packages: [cowsay] }
variants:
test:
base: nodejs
apt: { packages: [libcaca] }
The base
scalar will be overwritten, whereas the apt[packages]
sequence will be appended so that both cowsay
and libcaca
will be installed in an image produced from the Dockerfile resulting from the test
Blubberfile variant.
Variants
Blubber can build several variants of an image from the same
specification file. The variants are named and described under the
variants
top level item. Typically, there are variants for
development versus production: the development variant might have more
debugging tools, for example, which are not included in the production
variant, which should have no extra software installed to minimize
risk of security problems, or other problems.
A variant is built using the top level items, combined with the items
for the variant. So if the top level apt
installed some packages,
and the variant's apt
some other packages, both sets of packages get
installed in that variant.
Shorthand copies
To ease the transition from v3
to later configuration formats, and to generally provide a more succinct syntax for the most common file-copying patterns, copies
configuration can be provided in either a shorthand and longhand form. The basic rules are:
copies: [local]
Expands to configuration that will copy all local build-context files to the variant image's filesystem:
copies:
- from: local
source: .
destination: .
And
copies: [othervariant]
Expands to configuration that will copy all shared library and project files from the specified variant's filesystem to that of the current one.
copies:
- from: othervariant
source: /opt/lib
destination: /opt/lib
- from: othervariant
source: /srv/app # or whatever lives.in is
destination: /srv/app # same