Blubber: Getting Started

Blubber is an abstraction for container build configurations. It provides a handful of declarative constructs that give developers control over build configurations without sacrificing security and maintainability.

The tutorials below are intended to provide an overview of the basic parts of Blubber. The hope is that these examples will allow developers to become productive with Blubber as quickly as possible.

Hello World

The standard example for any software is one that can print "Hello World!" to the console.

Here is a Blubberfile that will print "Hello World!" to the console and exit.

version: v3
base: docker-registry.wikimedia.org/wikimedia-stretch

variants:
  hello:
    entrypoint: [echo, "Hello, world!"]

I have the file above saved as hello-blubber.yaml. To generate a Dockerfile from hello-blubber.yaml I pass the blubber command line application two arguments: (1) the path to hello-blubber.yaml file and (2) the name of the variant (from the keys listed under variants) from which I would like to generate a Dockerfile. In this case hello-blubber.yaml only has one variant named hello.

developer@laptop:~/blubber-tutorial$ blubber hello-blubber.yaml hello

This command produces the following Dockerfile on stdout (the output you see may vary depending on your version of blubber):

FROM docker-registry.wikimedia.org/wikimedia-stretch
USER "root"
ENV HOME="/root"
RUN groupadd -o -g "65533" -r "somebody" && useradd -o -m -d "/home/somebody" -r -g "somebody" -u "65533" "somebody" && mkdir -p "/srv/app" && chown "65533":"65533" "/srv/app" && mkdir -p "/opt/lib" && chown "65533":"65533" "/opt/lib"
RUN groupadd -o -g "900" -r "runuser" && useradd -o -m -d "/home/runuser" -r -g "runuser" -u "900" "runuser"
USER "somebody"
ENV HOME="/home/somebody"
WORKDIR "/srv/app"
COPY --chown=65533:65533 [".", "."]
USER "runuser"
ENV HOME="/home/runuser"
ENTRYPOINT ["echo", "Hello, world!"]
LABEL blubber.variant="hello" blubber.version="0.4.0+60add2d"

Blubber's only purpose is to create opinionated Dockerfiles. To generate a Dockerfile from a Blubberfile you pass the blubber command line program the path to a Blubberfile and a variant name.

Variants can be named anything, although it is common to have (at minimum) a test variant that creates a Dockerfile for an image that runs a test entrypoint, and a production variant that creates a Dockerfile to build an image that can be run in production. To create a Docker Image from this Dockerfile, you can use unix pipes to pipe the output of Blubber to the input of the docker build command.

developer@laptop:~/blubber-tutorial$ blubber hello-blubber.yaml hello | docker build --tag blubber-tutorial-hello-world --file - .

The command above creates a local Docker Image tagged with the name blubber-tutorial-hello-world. Using - as the file tells docker build to use stdin as the Dockerfile. . at the end of the command tells docker build to use the current directory as the context for any copy commands.

To run this image issue the following command:

developer@laptop:~/blubber-tutorial$ docker run --rm --interactive --tty blubber-tutorial-hello-world:latest
Hello, World!

Hello World, Redux

The previous example was extremely minimal as a means of giving a general introduction using Blubber to build Docker images that can be easily run on the command line. This example is meant to be ever so slightly more complicated as a means of introducing new Blubber concepts.

A slightly more complicated Hello, World! Blubberfile might look like:

version: v3
base: docker-registry.wikimedia.org/wikimedia-stretch
apt:
  packages:
    - figlet

runs:
  environment: { HELLO_WORLD: "Hello, world!" }

variants:
  hello:
    entrypoint: [sh, -c, 'figlet $HELLO_WORLD']

Above we've installed the figlet package from apt and we've also introduced an environment variable to hold our output.

The above Blubberfile produces the following Dockerfile on stdout (the output you see may vary depending on your version of blubber):

FROM docker-registry.wikimedia.org/wikimedia-stretch
USER "root"
ENV HOME="/root"
ENV DEBIAN_FRONTEND="noninteractive"
RUN apt-get update && apt-get install -y "figlet" && rm -rf /var/lib/apt/lists/*
RUN groupadd -o -g "65533" -r "somebody" && useradd -o -m -d "/home/somebody" -r -g "somebody" -u "65533" "somebody" && mkdir -p "/srv/app" && chown "65533":"65533" "/srv/app" && mkdir -p "/opt/lib" && chown "65533":"65533" "/opt/lib"
RUN groupadd -o -g "900" -r "runuser" && useradd -o -m -d "/home/runuser" -r -g "runuser" -u "900" "runuser"
USER "somebody"
ENV HOME="/home/somebody"
WORKDIR "/srv/app"
ENV HELLO_WORLD="Hello, world!"
COPY --chown=65533:65533 [".", "."]
USER "runuser"
ENV HOME="/home/runuser"
ENTRYPOINT ["sh", "-c", "figlet $HELLO_WORLD"]
LABEL blubber.variant="hello" blubber.version="0.4.0+60add2d"

You'll notice that the apt: {packages: [figlet]} declaration in the Blubberfile installed the figlet program as the root user on the commandline. You can specify packages to install from Debian at the top-level of a Blubberfile and under each individual variant.

Also notice that we've set an environment variable for our output message. You can set environment variables at the top level of a Blubberfile and under each individual variant.

Using the same command as above to generate a Docker image produces a slightly more interesting result when running the image in a container on the commandline:

developer@laptop:~/blubber-tutorial$ docker run --rm --interactive --tty blubber-tutorial-hello-world-redux:latest
 _   _      _ _        __        __         _     _ _
| | | | ___| | | ___   \ \      / /__  _ __| | __| | |
| |_| |/ _ \ | |/ _ \   \ \ /\ / / _ \| '__| |/ _` | |
|  _  |  __/ | | (_) |   \ V  V / (_) | |  | | (_| |_|
|_| |_|\___|_|_|\___/     \_/\_/ \___/|_|  |_|\__,_(_)

Hello Node

Many of the microservices used in Wikimedia production are written in Node.js. Many of the microservices used in Wikimedia production use the suffix -oid; which, according to Wiktionary is a suffix which modifies the root to imply that something is "Of similar form to, but not the same as."; e.g., humanoid, meaning similar-to, but not the same as a human.

This example will be similar to our previous examples (in that it will still output, "Hello, World!"), but it will be written in Node.js.

This example uses the HelloWorldOid repository.

To clone this repository and follow along:

dev@laptop:~$ git clone https://gerrit.wikimedia.org/r/blubber-doc/example/helloworldoid blubber-doc/example/helloworldoid
dev@laptop:~$ cd blubber-doc/example/helloworldoid

The directory structure of the application is fairly trivial (omitting any git-related files):

dev@laptop:helloworldoid$ tree -a --dirsfirst
.
├── lib
│   ├── helloworld.js
│   └── server.js
├── .pipeline
│   └── blubber.yaml
├── test
│   └── test.js
├── .dockerignore
├── index.js
├── LICENSE
├── package.json
└── README.md

3 directories, 9 files

There are a few files that are noteworthy and not typical in basic Node.js sample applications:

. .dockerignore - The .dockerignore ( on https://docs.docker.com) file is used to keep files necessary for development, but unnecessary for production out of a container image. In this case I've removed the README.md and the .git directory in an attempt to keep the Docker image created by Blubber small.
. .pipeline directory - this directory is where the [#Use_in_the_Continuous_Delivery_Pipeline Wikimedia Continuous Delivery Pipeline] expects to find the blubber.yaml file; so that's where it is!

Another thing to notice is that the Blubberfile is a bit longer than in previous examples, and has more variants:

version: v3
base: docker-registry.wikimedia.org/nodejs-slim
runs:
  environment:
    HELLO_WORLD: Hi, I’d like to add you to my professional network on LinkedIn.

variants:
  build:
    base: docker-registry.wikimedia.org/nodejs-devel
    node: {requirements: [package.json]}
  test:
    includes: [build]
    entrypoint: [npm, test]
  prep:
    includes: [build]
    node: { env: production }
  production:
    copies: prep
    entrypoint: [node, index.js]

Notice that a variant can specify a different base image than the global base image at the top of the file—in this case the build variant is using the docker-registry.wikimedia.org/nodejs-devel image so that it can use the npm tool already present in that image.

Also noteworthy is that each variant can specify a different entrypoint. In the case of the test variant we'd like to create a Docker image that runs the application's tests, but in the production variant we want to run the actual application.

The includes keyword is specified in a few variants: test and prep. Variants can inherit from one another. That is, rather than specify that we want to use all the same options as the build variant for test variant we can include that variant. This way the test variant uses the nodejs-devel base image and will run npm install after copying package.json into the image (since we've used the node keyword and specified that we need the package.json file as a requirement).

The build variant here is really only used as a set of options common to both the test and prep variants. Notice that the production variant is not including any other variant and so uses the nodejs-slim image.

The production variant copies the prep variant. The copies keyword will generate a [https://docs.docker.com/develop/develop-images/multistage-build/ multistage Dockerfile]. Docker images can easily become very large. One method to slim-down a Docker image is to copy any generated artifacts into an image rather than install the tools to generate artifacts inside the image. A multistage Dockerfile will first create an intermediary image (in this case based on the prep variant) that can be based on a different base image or include different packages that will be used to create artifacts. The artifacts generated by the intermediary Docker image can then be copied to a slimmer final Docker image (in this case based on the production variant) to allow for a smaller final image.

The prep variant varies from the build variant insofar as it uses the --production flag when calling npm install (as indicated by the node: {env:production} in the Blubberfile. In this way we do not include development dependencies like mocha in the final production image.

We can build an image based on the test variant to run the tests in much same way we've built previous images:

dev@laptop:helloworldoid$ blubber .pipeline/blubber.yaml test | docker build -t blubber-tutorial-helloworldoid-test -f - .

Then we can run the resulting image in a container to show the test results:

dev@laptop:helloworldoid$ docker run --rm -it blubber-tutorial-helloworldoid-test

> helloworldoid@0.0.1 test /srv/app
> mocha



  helloWorld
    ✓ obvs should be a string
    ✓ obvs should start with "hello"
    ✓ obvs should contain the word "world"


  3 passing (19ms)

Then we can run build and run the production image in a container, exposing port 8001:

dev@laptop:helloworldoid$ blubber .pipeline/blubber.yaml production | docker build -t blubber-tutorial-helloworldoid -f - .
dev@laptop:helloworldoid$ docker run -p8001:8001 --rm -it blubber-tutorial-helloworldoid

And now we should be able to access localhost:8001 using http to reveal the fruits of our labor:

developer@laptop:~$ curl localhost:8001
 __________________________________________________________________________________________________________________________
/  ('-. .-.   ('-.                                             (`\ .-') /`             _  .-')            _ .-') _  ,---.  \
| ( OO )  / _(  OO)                                             `.( OO ),'            ( \( -O )          ( (  OO) ) |   |  |
| ,--. ,--.(,------.,--.      ,--.      .-'),-----.          ,--./  .--.   .-'),-----. ,------.  ,--.     \     .'_ |   |  |
| |  | |  | |  .---'|  |.-')  |  |.-') ( OO'  .-.  '         |      |  |  ( OO'  .-.  '|   /`. ' |  |.-') ,`'--..._)|   |  |
| |   .|  | |  |    |  | OO ) |  | OO )/   |  | |  |         |  |   |  |, /   |  | |  ||  /  | | |  | OO )|  |  \  '|   |  |
| |       |(|  '--. |  |`-' | |  |`-' |\_) |  |\|  |         |  |.'.|  |_)\_) |  |\|  ||  |_.' | |  |`-' ||  |   ' ||  .'  |
| |  .-.  | |  .--'(|  '---.'(|  '---.'  \ |  | |  |         |         |    \ |  | |  ||  .  '.'(|  '---.'|  |   / :`--'   |
| |  | |  | |  `---.|      |  |      |    `'  '-'  '.-.      |   ,'.   |     `'  '-'  '|  |\  \  |      | |  '--'  /.--.   |
| `--' `--' `------'`------'  `------'      `-----' ',/      '--'   '--'       `-----' `--' '--' `------' `-------' '--'   |
\ Hi, I’d like to add you to my professional network on LinkedIn.                                                          /
 --------------------------------------------------------------------------------------------------------------------------
        \   ^__^
         \  (oO)\_______
            (__)\       )\/\
             U  ||--WWW |
                ||     ||~

Use in the Continuous Delivery Pipeline

In the Continuous Delievery Pipeline Jenkins will handle building Docker images from Blubberfile specifications. This is configured by convention: Jenkins expects to find a Blubberfile in .pipeline/blubber.yaml at the root of your repository. In that Blubberfile it expects to find, at minimum, two variants (1) test and (2)production.

When a patchset is proposed in Gerrit for your repository, Jenkins will build a Docker image based on your test variant and run a container based on that image. Jenkins will report back to Gerrit based on the whether the entrypoint of the that container exited cleanly (success) or not (failure).

When a patchset is merged to your repository on Gerrit, Jenkins will, again, build and execute your test variant. If that succeeds Jenkins builds a Docker image based on your production variant will be built and pushed into the Wikimedia Docker Registry.

When you push a tag to your repository on Gerrit, Jenkins will, once again, build and execute your test variant. If that succeeds, Jenkins builds a Docker image based on you production Blubber variant, tags it with the tag you pushed and pushes that to the Wikimedia Docker Registry.