Portal:Toolforge/Admin/Striker/Build

This page contains historical information. This scap3 build and deploy process was replaced with PipelineLib and Docker. phab:T306469

The Striker build and deploy process involves updating the labs-striker-deploy git repository and then pushing the repository out to servers running Striker via scap3.

Deploy directory structure

/srv/deployment/striker/deploy/
 ├── _build/                    # Generated by make_wheels.sh, only on Cloud VPS staging server
 |    └── venv/                 # virtualenv used for creating wheels & top level 
 ├── ddl/                       # Manually created SQL patches
 ├── make_wheels.sh             # Build script for preparing labs-striker-wheels patches
 ├── public_html/               # Apache/nginx docroot
 |    └── static/               # labs-striker-staticfiles submodule
 ├── requirements.txt           # `pip --freeze` generated requirements
 ├── scap/
 |    ├── checks/
 |    |    └──  virtualenv.sh   # Check script to rebuild venv on target hosts
 |    ├── checks.yaml           # Post-deploy checks for Scap3
 |    ├── dsh/                  # Scap3 target host sets
 |    ├── log/                  # Generated by scap3, only on deploy master
 |    └── scap.cfg              # Scap3 config file
 ├── striker/                   # striker submodule
 |    └── requirements.txt      # Product level requirements
 └── wheels/                    # labs-striker-wheels submodule

Submodules

Most of the interesting code changes will happen in submodules of labs-striker-deploy:

labs-striker: The Striker Django application
labs-striker-staticfiles: Django managed static files (css, js, images)
labs-striker-wheels: Python binary packages in wheel format

labs-striker

There is nothing special to do to prepare this repo for deployment. Merge commits using Gerrit.

Changes made here however will trigger changes to other submodules. At the point a release is being made:

labs-striker-staticfiles must be updated if any changes were made to requirements.txt or if files were modified in the static directory.
labs-striker-wheels must be updated if any changes were made to requirements.txt.

labs-striker-staticfiles

Previously there was a build-step to update and commit static files. That is now handled automatically during scap deployment. There should be no need to explicitly rebuild these on Vagrant.

labs-striker-wheels

This repo is the collection of Python wheels to be installed in the virtualenv that scap3 manages for each deploy target. We do not allow direct interaction with pypi from inside the production network, so instead we create a collection of wheels outside of prod and save them in gerrit for use inside the prod network. This is not pretty, but it works.

:# Clone the striker/deploy.git repo
$ git clone ssh://gerrit/labs/striker/deploy striker-deploy
$ cd striker-deploy
:# Update the striker submodule checkout to the HEAD of master (or whatever you are prepping for deploy)
$ cd striker
$ git fetch
$ git reset --hard origin/master
:# Remove old wheels locally
$ cd ../wheels
$ git checkout master
$ rm *.whl
:# Re-build wheels cache from scratch
$ cd ..
$ rm -rf _build
$ ./make_wheels.sh
:# Turn the new wheels into a patch for gerrit review
$ cd wheels
$ git add .
$ git commit
$ git review

Testing in Cloud VPS project

TODO: explain how I typically stage pending changes in the striker Cloud VPS project.

$ ssh striker-deploy04.striker.eqiad.wmflabs
$ cd /srv/deployment/striker/deploy/striker
$ git fetch "https://gerrit.wikimedia.org/r/labs/striker" refs/changes/74/521774/1 && git cherry-pick FETCH_HEAD
$ scap deploy

Manual tests

create new account
link to phab profile
add ssh key
request toolforge membership
approve toolforge membership
create tool
create git repo
create phab project
edit toolinfo
diff toolinfo
revert toolinfo
suppress toolinfo
add 2nd toolinfo
delete 2nd toolinfo
toolinfo export via /tools/toolinfo/v1/toolinfo.json

Deploy

This section describes the process of getting the live service modified with new changes.

download the source code git repository: https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker
download the deployment git repository: https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker/deploy
develop: make your code changes, tests, etc (out of the scope of this section)
git review the source code (gerrit) and get it merged
update the submodule commit in the deployment repo pointing to the commit you want to deploy
- https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker is a submodule in the deploy repo
- Occasionally this will not fetch no matter what you do, so I found that running git submodule update --recursive --init captured the latest code, allowing one to cd into the striker subdirectory and run git checkout <desired commit> which seems to be how this is often updated.
- Add, commit and submit your changes for review on the deploy repo. This is the repo you are deploying!
ssh jump to deployment server, for example deploy1001.eqiad.wmnet and cd to /srv/deployment/striker/deploy
git fetch & merge/rebase & git submodule update --recursive
- The submodule won't update unless you changed it on the deploy repository in the last step above
scap deploy -v "A good explanation of the reason for deploying goes here"