Services/FirstDeployment

From Wikitech
Jump to: navigation, search

Repositories

We require that all services are hosted on our Gerrit servers. It does not have to be your primary development technique or tool, even though you are strongly encouraged to do so.

Because Node.js services use npm dependencies which can be binary, these need to be pre-built. Therefore, two repositories are needed; one for the source code of your service, and the other, so-called deploy repository. Both should be available as WM's Gerrit repositories with the paths mediawiki/services/your-service-name and mediawiki/services/your-service-name/deploy, respectively. When requesting them ask for the former to be a clone of the service template (or of your own service repository) and the latter to be empty.

It is important to note that the deploy repository is only to be updated directly before (re-)deploying the service, and not on each patch merge entering the master branch of the regular repository. In other words, the deploy repository mirrors the code deployed in production at all times.

The remainder of this guide assumes these two repositories have been created and that you have cloned them using your Gerrit account, i.e. not anonymously, with the following outline:

~/code/
  |- your-service
  -- deploy

This guide refers to these two repositories as the source repository and the deploy repository, respectively.

Source Repo Configuration

The service template includes an automation script which updates the deploy repository, but it needs to be configured properly in order to work.

package.json

The first part of the configuration involves keeping your source repository's package.json updated. Look for its deploy stanza. Depending on the exact machine on which your service will be deployed, you may need to set target to either ubuntu or debian (most likely and default value if missing).

If you want to specify a version of Node.JS, different from the official distribution package, set the value of the node stanza to the desired version, following nvm versions naming. To explicitly force official distribution package, "system" version can be used.

The important thing is keeping the dependencies field up to date at all times. There you should list all of the extra packages that are needed in order to build the npm module dependencies. The _all field denotes packages which should be installed regardless of the target distribution, but you can add other, distribution-specific package lists, e.g.:

"deploy": {
  "target": "ubuntu",
  "node": "system",
  "dependencies": {
    "ubuntu": ["pkg1", "pkg2"],
    "debian": ["pkgA", "pkgB"],
    "_all": ["pkgOne", "pkgTwo"]
  }
}

In this example, with the current configuration, packages pkg1, pkg2, pkgOne and pkgTwo are going to be installed before building the dependencies. If, instead, the target is changed to debian, then pkgA, pkgB, pkgOne and pkgTwo are selected.

As a rule of thumb, whenever you need to install extra packages into your development environment for satisfying node module dependencies, add them to deploy.dependencies to ensure the successful build and update of the deploy repository.

Local Git

The script needs to know where to find your local copy of the deploy repository. To that end, when in your source repository, run:

$ git config deploy.dir /absolute/path/to/deploy/repo

Using the aforementioned local outline, you would type:

$ git config deploy.dir /home/YOU/code/deploy

Deploy Repo Set-up

If you haven't yet done so, initialise the deploy repository:

$ cd ~/code/deploy
$ git review -s
$ touch README.md
$ git add README.md
$ git commit -m "Initial commit"
$ git push -u origin master  # or git review -R if this fails
# go to Gerrit and +2 your change, if needed and then:
$ git pull

Next, you need prepare the deploy repository for usage with Scap3. Create the scap directory inside your deploy repository and fill the contents of scap/scap.cfg with:

[global]
git_repo: <service-name>/deploy
git_deploy_dir: /srv/deployment
git_repo_user: deploy-service
ssh_user: deploy-service
server_groups: canary, default
canary_dsh_targets: target-canary
dsh_targets: targets
git_submodules: True
service_name: <service-name>
service_port: <service-port>
lock_file: /tmp/scap.<service-name>.lock

[wmnet]
git_server: tin.eqiad.wmnet

[deployment-prep.eqiad.wmflabs]
git_server: deployment-tin.deployment-prep.eqiad.wmflabs
server_groups: default
dsh_targets: betacluster-targets

This represents the basic configuration needed by Scap3 to deploy the service. We still need to tell Scap3 on which nodes to deploy and which checks to perform after the deployment on each of the nodes. First, the list of nodes. One file needs to be created: scap/target-canary, where you need to put the FQDN of the node that will act as the canary deployment node, i.e. the node that will first receive the new code, For example, if your target nodes are in the SCB cluster, this file should look like this:

$ cat target-canary 
scb1002.codfw.wmnet

The complete list of all targets should be ops-controlled and derives from puppet and conftool. Ask ops to set it up before you do your first production deploy. In the same vein, you need to create the scap/betacluster-targets file which will contain the FQDNs of the targets in BetaCluster.

Finally, enable the automatic checker script to check the service after each deployment by placing the following in scap/checks.yaml:

checks:
  depool:
    type: command
    stage: promote
    command: depool-<service-name>
  endpoints:
    type: nrpe
    stage: restart_service
    command: check_endpoints_<service-name>
  repool:
    type: command
    stage: restart_service
    command: pool-<service-name>

Commit your changes, send them to Gerrit for review and merge them.

The deployment process includes a script that builds the deployment repository using Docker containers, so make sure you have the latest version installed. Additionally, you need to add your user to the `docker` group after installation so that you don't need to use `sudo` when running the build script:

$ sudo usermod -a -G docker <your-user>

You need to log out of all of the terminals in order for the change to take effect.

New Service Request

There are various prerequisites that need to be taken care of on the operational side before your service can see the day of light in production: machine allocation, IPs, LVS, etc. In order to express the intent of deployment, you need to complete a new service request, by filing a task against the service-deployment-requests project in Phabricator. Be prepared to give the following information:

  • name: the name of the service to be deployed
  • description: a paragraph explaining clearly what the service does and why it is needed
  • timeline: the desired deployment timeline; note that you should allow a minimum of at least two to three weeks cadence
  • point person: the person responsible for the service; this is the person that will get called when there are problems with the service when running in production
  • technologies: additional information about the service itself, including, but not limited to, the language used for development and any frameworks used
  • request flow diagram: a link to a request flow diagram that explains the interaction between your service and any other parts of the operational stack inside the production cluster, such as requests made to MediaWiki, RESTBase, etc.

For some example tickets see Task T105538, Task T117560, Task T128463.

Role and Profile Creation

While you are waiting for the service request to be completed, do not fear: you still have useful things to do. You may start by creating your service's Puppet role and profile in the operations/puppet repository. First, add your service's deploy repository to the list of repositories deployed in production by appending the following block to hieradata/common/role/deployment.yaml (note the extra spaces at the beginning of each line):

  <service-name>/deploy:
    upstream: https://gerrit.wikimedia.org/r/mediawiki/services/<service-name>/deploy
    checkout_submodules: true

Next, create modules/profile/manifests/<service-name>.pp and put the following content in it:

# == Class: profile::<service-name>
#
# Describe the service here ...
#
# === Parameters
#
# [*param_name1*]
#   Description of param_name1
#
# [*param_name2*]
#   Description of param_name2
#
class profile::<service-name>(
    $param_name1 = hiera('profile::<service-name>::param_name1'),
    $param_name2 = hiera('profile::<service-name>::param_name2,
) {

    service::node { '<service-name>':
        port            => <service-port>,
        config          => {
            param_name1 => $param_name1,
            param_name2 => $param_name2,
        },
        healthcheck_url => '',
        has_spec        => true,
        deployment      => 'scap3',
    }

}

Note that only configuration specific to your service should be listed here and not the whole configuration file, i.e. only the configuration parameters that your service code accesses via app.conf.*. Instead of in-lining it directly in the module, you can also store the configuration in form of an ERB YAML template in modules/<service-name>/templates/config.yaml.erb. Then, simply use it directly for the config parameter for the service::node resource like so:

        config          => template('<service-name>/config.yaml.erb'),

You will also need a role for your service. Put the following code fragment into manifests/role/<service-name>.pp:

# Role class for <service-name>
class role::<service-name> {

    system::role { 'role::<service-name>':
        description => 'short description',
    }

    include ::profile::<service-name>
}

and add the values for the hiera lookups in place inside hieradata/role/common/<service-name>.yaml as follows:

profile::<service-name>::param_name1: "some-value"
profile::<service-name>::param_name2: "some-other-value"

You can now submit the patch for review. Don't forget to mention the service request bug in your commit message.

Access Rights

As the service owner and maintainer, you need to be able to log onto the nodes where your service is running. Once the exact list of target nodes is known, you need to file an access request ticket with the following information:

  • Ttile: Access Request for <list-of-maintainers> for <service-name>
  • Description: <list-of-maintainers> needs access to <list-of-nodes> for operating <service-name>. We need to be able to read the logs at /srv/log/<service-name> and be able to start/stop/restart it. The task asking for the service's deployment is {<service-request-task-number>}

This request implies sudo rights on the target nodes, so you will need the approval from your manager on the task.