User:Mobrovac/Service Deployment

Current Process

For Node.js services two repositories are created, called the source and deploy repositories. The former is a normal repository where developers regularly commit new changes. The latter is a special-purpose repository containing the source repository as a submodule and all of the node module dependencies needed for the service to run correctly in production. In terms of Gerrit, developers have +2 on the source repository, while deployers have +2 on the deploy repository. For example outlines of both types of repos, look at available service repositories in Gerrit, where name is the source repository, and name/deploy is its deploy repository.

Code Deployment

The current process is shared amongst most of our Node.js services and comprises the following steps:

1. Developers commit changes in the source repository
2. At a certain, predefined point in time (number of commits, importance of changes, number of days from last deploy) the deployer starts the deployment process
3. The source repository submodule of the deploy repository is updated and node module dependencies are rebuilt manually
4. git commit && git review (for the deploy repo)
5. C:+2, V:+2 in Gerrit
6. Log onto tin, go to the needed deploy directory (cd /srv/deployment/service-name/deploy)
7. Do the deploy dance:
   • git deploy start
   • git pull && git submodule update --init
   • git deploy sync
8. Log onto one of the production nodes and restart the service
9. Check the service
   • process is up
   • bound to the needed port
   • optionally, inspect the logs to verify the start up happened correctly
10. Verify the service works as expected, usually involves issuing curl calls and checking the response
11. Log onto the rest of the service's hosts and restart the service
12. Repeat steps 6 - 11 for deployment-prep and any other Labs projects the service resides on

Note for step 3: Because building these dependencies may introduce compatibilities with the production environment (different node.js or npm versions, different distro/OS, etc), the current practice for services based on service-runner (most notable exceptions here are Paroid, CXServer, OCG) is to spin up a Docker container based on the target production system (most services - Trusty, RESTBase - Jessie) and build the dependencies there to ensure environment/version match.

Config Deployment

With the exception of Parsoid, the configuration files for our services reside in the ops/puppet repository. Thus, the process is:

1. Sumbit a patch for ops/puppet
2. Wait for opsens to +2 it
3. Puppet runs on the target hosts, applies the new config and restarts the service
4. Check the service as listed previously

Note On Deployment-prep

Due to the usage of the specialised deploy repository, services cannot be tested in deployment-prep prior to production deployment. While there is value in this for the services' clients, service developers cannot test new features properly. Instead, developers need to set up their own instances in a separate Labs project, check out the source repository there and automatically update it (and rebuild the dependencies) upon each commit. As an example, the MobileApps service uses an update script called by cron every 3 minutes.