Jump to content

Deployment pipeline/Tutorial

From Wikitech
This page describes the previous deployment pipeline mechanism based on PipelineLib. For information about the current deployment pipeline, which uses GitLab, Blubber, and Kokkuri, see the deployment pipeline documentation on mediawiki.org.

There are many moving pieces in the deployment pipeline. To better understand how those pieces fit together, let's work through a concrete example.

In the walkthrough below we will show how to make an update to an application already setup for use with the Deployment Pipeline.

This example uses the HelloWorldOid repository. HelloWorldOid uses FIGlet and cowsay to say "Hello World!" to folks over HTTP.

Goal

Deploy a simple update for HelloWorldOid.

Steps

Update helloworldoid

The process of deployment starts with an idea for a change. I feel like HelloWorldOid could use some pop. Some pizzazz. That'll get the VC money to start rolling in, for sure! To that end, I propose we add a date to the output of HelloWorldOid — this way we'll know:

  1. The world is a welcoming place
  2. The world is a welcoming place full of cows
  3. The time at which the world was so welcoming and cow filled

I made the following patchset and submitted a change to gerrit:

From: Tyler Cipriani <tcipriani@wikimedia.org>
Date: Thu, 25 Jul 2019 10:20:03 -0600
Subject: [PATCH] Add Timestamp to output

To help better preserve the memory of the beautiful moment when the world was greeted and all was well.                                               
---
 lib/server.js | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/server.js b/lib/server.js
index 99fecd9..5549430 100644
--- a/lib/server.js
+++ b/lib/server.js
@@ -38,12 +38,14 @@ function serve(message, port = 8001) {
+        var timestamp_text = '\n\nTimestamp: ' + new Date().toISOString();                                                                           
+                                                                      
             res.end(cowsay.say({
                 text : text,
                 r : true,
-            }));
+            }) + timestamp_text);
--
2.20.1

After pushing this diff up to Gerrit (because blubber-doc/example/helloworldoid is setup in the integration/config repo to use the service-pipeline) CI builds and runs the image described by the test Blubber variant, and reports the result in Gerrit:

PipelineBot Test Reply

Since all tests pass, I'll vote CR+2 on the patchset and the deployment pipeline will:

  1. Build my production variant
  2. Deploy my production image to an integration environment
  3. Run any integration tests defined by helm test
  4. If my integration tests pass:
  • My production image is pushed to docker-registry.wikimedia.org
  • The pipeline comments on my patch with a pointer to the new image and tags it made:

PipelineBot Gating Reply

Update the deployment chart

Using the information from PipelineBot, I can update the HelloWorldOid chart in the operations/deployment-charts repo used in production. I'll make a patchset to bump the image version in all production environments to use the new image.

From 0c5eda0f9cd19d5a14526335123a31963d090f91 Mon Sep 17 00:00:00 2001
From: Tyler Cipriani <tcipriani@wikimedia.org>
Date: Fri, 26 Jul 2019 15:09:38 -0600
Subject: [PATCH] helloworldoid: bump production image

Change-Id: I0d2e93d983b204534b5e7ba3273ff29c0a026991
---
 helmfile.d/services/codfw/helloworldoid/values.yaml   | 2 +-
 helmfile.d/services/eqiad/helloworldoid/values.yaml   | 2 +-
 helmfile.d/services/staging/helloworldoid/values.yaml | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/helmfile.d/services/codfw/helloworldoid/values.yaml b/helmfile.d/services/codfw/helloworldoid/values.yaml
index bdfc6ed..6bb005f 100644
--- a/helmfile.d/services/codfw/helloworldoid/values.yaml
+++ b/helmfile.d/services/codfw/helloworldoid/values.yaml
@@ -30,7 +30,7 @@ main_app:
-  version: 2019-07-23-155043-production
+  version: 2019-07-25-125909-production
diff --git a/helmfile.d/services/eqiad/helloworldoid/values.yaml b/helmfile.d/services/eqiad/helloworldoid/values.yaml
index bdfc6ed..6bb005f 100644
--- a/helmfile.d/services/eqiad/helloworldoid/values.yaml
+++ b/helmfile.d/services/eqiad/helloworldoid/values.yaml
@@ -30,7 +30,7 @@ main_app:
-  version: 2019-07-23-155043-production
+  version: 2019-07-25-125909-production
diff --git a/helmfile.d/services/staging/helloworldoid/values.yaml b/helmfile.d/services/staging/helloworldoid/values.yaml
index eda1542..5e532a4 100644
--- a/helmfile.d/services/staging/helloworldoid/values.yaml
+++ b/helmfile.d/services/staging/helloworldoid/values.yaml
@@ -30,7 +30,7 @@ main_app:
-  version: 2019-07-23-155043-production
+  version: 2019-07-25-125909-production
-- 
2.20.1

After merging this simple change, I'm ready to deploy!

Run a deployment

Deployment is accomplished via helmfile and is a few short commands:

WARNING: The following commands are outdated, see Kubernetes/Deployments#Deploying_with_helmfile instead for more current information.

you@your-laptop:~ $ ssh deployment.eqiad.wmnet
you@deploy1001:~ $ cd /srv/deployment-charts/helmfile.d/services/staging/helloworldoid
you@deploy1001:helloworldoid $ source .hfenv 
you@deploy1001:helloworldoid $ helmfile diff
you@deploy1001:helloworldoid $ helmfile apply
you@deploy1001:~ $ cd /srv/deployment-charts/helmfile.d/services/codfw/helloworldoid
you@deploy1001:helloworldoid $ source .hfenv 
you@deploy1001:helloworldoid $ helmfile diff
you@deploy1001:helloworldoid $ helmfile apply
you@deploy1001:~ $ cd /srv/deployment-charts/helmfile.d/services/eqiad/helloworldoid
you@deploy1001:helloworldoid $ source .hfenv 
you@deploy1001:helloworldoid $ helmfile diff
you@deploy1001:helloworldoid $ helmfile apply

And that's it! Our application is now deployed to all production environments using the values.yaml from operations/deployment-charts that points to the image that the pipeline created with our change.

Results

The results speak for themselves: we have a much more satisfying and remarkable user experience with so little effort:

__________________________________________________________________________________________________________________________
/  ('-. .-.   ('-.                                             (`\ .-') /`             _  .-')            _ .-') _  ,---.  \
| ( OO )  / _(  OO)                                             `.( OO ),'            ( \( -O )          ( (  OO) ) |   |  |
| ,--. ,--.(,------.,--.      ,--.      .-'),-----.          ,--./  .--.   .-'),-----. ,------.  ,--.     \     .'_ |   |  |
| |  | |  | |  .---'|  |.-')  |  |.-') ( OO'  .-.  '         |      |  |  ( OO'  .-.  '|   /`. ' |  |.-') ,`'--..._)|   |  |
| |   .|  | |  |    |  | OO ) |  | OO )/   |  | |  |         |  |   |  |, /   |  | |  ||  /  | | |  | OO )|  |  \  '|   |  |
| |       |(|  '--. |  |`-' | |  |`-' |\_) |  |\|  |         |  |.'.|  |_)\_) |  |\|  ||  |_.' | |  |`-' ||  |   ' ||  .'  |
| |  .-.  | |  .--'(|  '---.'(|  '---.'  \ |  | |  |         |         |    \ |  | |  ||  .  '.'(|  '---.'|  |   / :`--'   |
| |  | |  | |  `---.|      |  |      |    `'  '-'  '.-.      |   ,'.   |     `'  '-'  '|  |\  \  |      | |  '--'  /.--.   |
| `--' `--' `------'`------'  `------'      `-----' ',/      '--'   '--'       `-----' `--' '--' `------' `-------' '--'   |
\ Hi, I’d like to add you to my professional network on LinkedIn.                                                          /
 --------------------------------------------------------------------------------------------------------------------------
        \               ___
         \           .-'   `'.
                    /         \
                    |         ;
                    |         |           ___.--,
           _.._     |0) ~ (0) |    _.---'`__.-( (_.
    __.--'`_.. '.__.\    '--. \_.-' ,.--'`     `""`
   ( ,.--'`   ',__ /./;   ;, '.__.'`    __
   _`) )  .---.__.' / |   |\   \__..--""  """--.,_
  `---' .'.''-._.-'`_./  /\ '.  \ _.-~~~````~~~-._`-.__.'
        | |  .' _.-' |  |  \  \  '.               `~---`
         \ \/ .'     \  \   '. '-._)
          \/ /        \  \    `=.__`~-.
     jgs  / /\         `) )    / / `"".`\
    , _.-'.'\ \        / /    ( (     / /
     `--~`   ) )    .-'.'      '.'.  | (
            (/`    ( (`          ) )  '-;
             `      '-;         (-'

Timestamp: 2019-07-26T21:24:01.353Z

also see: Kubernetes/Deployments#Deploying_with_helmfile