Maps/Services deployment

From Wikitech
Jump to navigation Jump to search

Deploying maps services

Developer setup for updating deployment repo

Check out the basics for deployment.

We build the deploy repo in Docker in order to ensure that the Node modules have the correct binaries. We build the deploy repo using Docker for Mac or a Linux machine. Before you start the first build of the deploy repo run through the instructions.

Kartotherian is developed on Gerrit, and the old Github repository is abandoned and is no longer used by WMF infrastructure.

Publish libraries to NPM before building for deployment

In order to be able to package kartotherian and tilerator, the process to bump the version and publish to npm is crucial, therefore you should have permission to publish packages in the @wikimedia npm organization. The version bump occurs using manually package by package or using the lerna dependency manager.

Building for deployment

Kartotherian and Tilerator are deployed according to the standard process for deploying Wikimedia Node.js services, with the important difference that deployments are built from purpose-specific "package" repos rather than directly from service code repos. This is to facilitate the bundling of additional maps-specific dependencies.

Kartotherian: https://gerrit.wikimedia.org/r/#/admin/projects/maps/kartotherian/package
Tilerator: https://gerrit.wikimedia.org/r/#/admin/projects/maps/tilerator/package

Update and build the deployment repo

Sync the code and deploy repos with current master:

cd ~/path/to/kartotherian/package
git status
git reset --hard origin/master
git checkout .
git clean -fd
git checkout master
git pull
git status
git --no-pager log --decorate -n 1
rm -rf node_modules/
npm i service-runner

cd ../deploy
git status
git reset --hard origin/master
git checkout .
git clean -fd
git checkout master
git pull
git --no-pager log --decorate -n 1
git submodule update --init
git branch
git status
cd ../package

If using Docker for Mac start the Docker daemon by clicking on the Whale icon in the menu bar. (Should work automatically on Linux.) Run the tests in Docker and build the new commit for the deploy repo:

./server.js build --deploy-repo --force

And push to Gerrit:

cd ../deploy
git review

Deploy to Beta Cluster

The steps are similar to #Deploy to Production but using different machines, of course.

To deploy:

ssh deployment-deploy01.deployment-prep.eqiad.wmflabs

(instead of instead of ssh deployment.)

To verify something on the box you can ssh into deployment-maps07.deployment-prep.eqiad1.wikimedia.cloud.

Use #wikimedia-releng (instead of #wikimedia-operations) to see if there are issues. You may want to log manually in this channel using !log until phab:T156079 is resolved. This logs to the Releng team's Server admin log

Deploy to Production

Scan through recent chat in #wikimedia-operations channel on IRC to make sure there's nothing blocking the deploy.

Optional: Look at deployment logs:

ssh deployment  #deployment.eqiad.wmnet, currently points to deploy1002
cd /srv/deployment/kartotherian/deploy/
scap deploy-log

In another terminal start the actual deployment:

ssh deployment
cd /srv/deployment/kartotherian/deploy/
git pull && git submodule update
git log -n 1
scap deploy "`git log --pretty=format:'%s' -n 1`"

The scap deploy command above takes a reason string argument. If this string contains phab tasks, those tasks will get comments about the deployment happening (start + finish). So, let's say in the deployment we have fixes for tasks T123 and T234 you could write instead of the last command:

scap deploy "`git log --pretty=format:'%s' -n 1` (T123 T234)"

Do the same for Tilerator

Deploying services

Refer to Services/Deployment for general instructions.

In most cases, Kartotherian and Tilerator should be deployed together, to ensure that all dependencies (in particular, styles) are in sync.