Scap/Scap3 migration guide

This page is a guide for migrating Trebuchet deployed services to Scap3.

If you are migrating a Node.js service, check out the Services Migration Guide.

This guide does not represent every consideration for moving a deployment to Scap3, but attempts to highlight important details that are part of the migration from Trebuchet to Scap3.

Migration checklist

Decide on canary deployment hosts and deployment checks
Create Puppet patches
Create scap.cfg patches for your deployed repo
Schedule a migration window on the Deployments page and make sure you have the support you need from both ops and releng.

Canary hosts and checks

Use of canary hosts is highly recommended

Scap3 supports using canary hosts. This means you can deploy to a subset of nodes before deploying a service everywhere. While this is not *required* it is highly recommended that you use canary nodes, particularly for the first deployment.

By default, Scap3 will assume that a deployment has been successful if no errors were encountered during deployment; however, it also supports more advanced checks including running arbitrary commands at any stage of deployment. For more information about what types of checks are available refer to the more detailed documentation on Service Checks.

Puppet Patches

There are a few files all deployments will need to edit:

hieradata/common/role/deployment.yaml
hieradata/role/common/deployment_server.yaml
The puppet file that defines all service targets.

`hieradata/common/role/deployment.yaml`

First, you'll need to remove your deployment definition from hieradata/common/role/deployment.yaml. This is a dictionary that is used by Trebuchet and stored in salt -- it is not needed by Scap3 as Scap3 does not use salt.

`hieradata/role/common/deployment_server.yaml`

Next, you'll need to add your deployment to the scap::sources dictionary inside hieradata/role/common/deployment_server.yaml. This definition will be used on the deployment host to clone your repository under /srv/deployment/[repo/name]. Here is an example of what to add to the sources dictionary to clone a repo from gerrit (at https://gerrit.wikimedia.org/r/an/example/repo) to tin (at tin:/srv/deployment/example/repo:

example/repo:
  repository: an/example/repo

Alternatively, if your repo on gerrit has the same name as the directory structure on tin (e.g., https://gerrit.wikimedia.org/r/real/repo → tin:/srv/deployment/real/repo) the yaml inside server.yaml should look like:

real/repo: {}

`scap::target`

Finally, you'll need to add a scap::target definition to the puppet code that executes on your target machines.

Things to consider in your scap::target definition:

What user should own my remote files to be deployed?
Should Scap3 handle service restarts as part of deployments?
Do I want Scap3 to handle any other checks that require sudoer permissions?

The default answers to those questions are: The deploy-service user does deploys, Scap3 restarts a service as part of deployment, Scap3 performs no other checks that require sudoer priveliges. If the default answers suffice, the following scap::target definition should be sufficient (using the example/repo example):

scap::target { 'example/repo':
    service_name => 'example-service',
    deploy_user  => 'deploy-service',
}

New User

Sometimes, it is inappropriate to use the deploy-service user. In that instance, you'll need to setup a new user. scap::target can do that for you with the manage_user option.

scap::target { 'example/repo':
    service_name => 'example-service',
    deploy_user  => 'deploy-my-service',
    manage_user  => true,
}

scap/target will do all user, group, and ssh-key management. You will need to add an ssh-key for that user under modules/secret/secrets/keyholder. Any non-word character (i.e., \W) will be replaced by _ in the key name.

For example, for the user deploy-my-service, scap::target will expect to find a public key under modules/secret/secrets/keyholder/deploy_my_service.pub.

Noteworthy, that any new key added to Keyholder MUST be password protected.

Other considerations

Scap3 replaces the directory into which code is deployed with each deployment. This means any puppet generated and/or non-version-controlled files will be replaced on every deploy!

Scap3 Configuration

Scap3 uses a configuration file that lives in the root of your repository at ./scap/scap.cfg. A basic scap.cfg that supports:

Submodules
Service restart
Canary nodes
Post-service-restart port check

is shown here:

[global]
git_repo: <repo/name> # (e.g. example/repo)
git_deploy_dir: /srv/deployment
git_repo_user: deploy-service
ssh_user: deploy-service
server_groups: canary, default
canary_dsh_targets: target-canary
dsh_targets: targets
git_submodules: True
service_name: <service-name>
service_port: <service-port>
lock_file: /tmp/scap.<service-name>.lock

[wmnet]
git_server: deployment.eqiad.wmnet

[deployment-prep.eqiad.wmflabs]
git_server: deployment-tin.deployment-prep.eqiad.wmflabs
server_groups: default
dsh_targets: betacluster-targets

The dsh_targets and canary_dsh_targets point to files under the ./scap directory that contain a list of targets. You can use comments beginning with # to help maintain target lists:

$ cat target-canary 
scb1001.eqiad.wmnet

$ cat targets
# eqiad
# scb1001 is a canary
scb1002.eqiad.wmnet
# codfw
scb2001.codfw.wmnet
scb2002.codfw.wmnet

First Deployment

After the puppet patches and scap.cfg and other configuration files are written, the first deployment can be tricky and should be done in a specific order.

Merge the scap/scap.cfg and other patches within the repo
Fetch those patches down to /srv/deployment/[repo] on tin or the current active deployment server
Stop puppet on the target hosts using cumin
Merge the puppet patches
Run puppet on tin (this should effectively be a no-op)
On tin, as a regular user, run:
```
$ cd /srv/deployment/[repo]
$ scap deploy --init
```
This should create /srv/deployment/[repo]/.git/DEPLOY_HEAD that will act as the configuration for scap running on the remote hosts
Run puppet on each of the target hosts. This should ensure that /srv/deployment/[repo] is owned by the deploy_user defined in scap::target definition in puppet.

Now you should be able to run a deployment from the deployment server:

$ cd /srv/deployment/[repo]
$ scap deploy -v "message for [[SAL]]"

The important steps to keep in mind are that scap deploy --init must be run on the deployment server before puppet runs on any of the target machines and that puppet must run on the target machines before the first deployment.