User:Thcipriani/Scap3 Stagingdev

Need a way to dev scap3 mediawiki deploys. We can't do it on beta without being disruptive. *Maybe* we can do it on staging. It has to happen somewhere.

Staging Setup

Create Hiera pages for 3 hosts. The deployment server (Hiera:Staging/host/thcipriani-tin) should have the content:

---
"classes":
    - role::deployment::server
deployment_server: thcipriani-tin.staging.eqiad.wmflabs
"scap::deployment_server": thcipriani-tin.staging.eqiad.wmflabs
"scap::wmflabs_master": thcipriani-tin.staging.eqiad.wmflabs
scap::dsh::scap_proxies:
    - thcipriani-proxy.staging.eqiad.wmflabs

The proxy and appserver (Hiera:Staging/host/thcipriani-proxy and Hiera:Staging/host/thcipriani-mediawiki) should have this content:

---
classes:
    - role::beta::mediawiki
    - role::mediawiki::appserver
deployment_server: thcipriani-tin.staging.eqiad.wmflabs
"scap::deployment_server": thcipriani-tin.staging.eqiad.wmflabs
"scap::wmflabs_master": thcipriani-tin.staging.eqiad.wmflabs
"scap::dsh::scap_proxies":
    - thcipriani-proxy.staging.eqiad.wmflabs

Example hiera pages:

Create the 3 hosts:
1. thcipriani-tin -- Will act as your deployment-host (tin)
2. thcipriani-proxy -- Will act as your deployment-proxy (mwXXXX)
3. thcipriani-mediawiki -- Will act as your deployment-target (mwXXXX)

Finally, to use scap to deploy MediaWiki, we need to ensure that the dsh files exist, and keyholder is armed on thcipriani-tin.staging.eqiad.wmflabs

$ sudo touch /etc/dsh/group/mediawiki-{installation,api-canaries,appserver-canaries}
$ sudo sh -c 'printf "%s\n%s\n" "thcipriani-mediawiki.staging.eqiad.wmflabs" "thcipriani-proxy.staging.eqiad.wmflabs" > /etc/dsh/group/mediawiki-installation'
$ sudo /usr/local/sbin/keyholder arm && sudo service keyholder-proxy restart

You should be able to run a mediawiki deployment with just that

Developing Scap on Staging

Now that we have a few working MediaWiki appservers, we can test deployments to them using scap; however, to test a development version of scap, we must add a version that shadows the version installed on these dev boxes via apt. These instructions create a working scap clone on thcipriani-tin that can be deployed to thcipriani-proxy and thcipriani-mediawiki via scap.

deploy scap via scap

a.k.a scapception

Clone scap on thcipriani-tin and add to your $PATH:

$ sudo mkdir -p /srv/deployment/scap && sudo chown -R $(whoami):wikidev /srv/deployment/scap
$ git clone https://phabricator.wikimedia.org/diffusion/MSCA/scap /srv/deployment/scap/scap

Add a scap.cfg to your scap repo to deploy scap via scap:

/srv/deployment/scap/scap/scap/scap.cfg (yeah, it's ridiculous):

[global]
git_repo: scap/scap
dsh_targets: mediawiki-installation

Create the scap directory on the scap targets: thcipriani-mediawiki and thcipriani-proxy:

$ sudo mkdir -p /srv/deployment/scap && sudo chown -R mwdeploy:wikidev /srv/deployment/scap

Create a virtualenv and install scap and dependencies:

$ sudo apt-get install python-pip python-dev
$ sudo pip install virtualenv
$ cd /srv/deployment/scap/scap
$ virtualenv .venv
$ . .venv/bin/activate
$ pip install -r requirements.txt
$ python setup.py install
$ echo '. /srv/deployment/scap/scap/.venv/bin/activate' >> "$HOME/.bashrc"

Now you can deploy scap to thcipriani-mediawiki and thcipriani-proxy by logging into thcipriani-tin and running scap deploy from /srv/deployment/scap/scap. You can then use scap in /srv/mediawiki-staging to test out changes to MediaWiki deploys.

Meta Notes

Had to add this this to Hiera:Staging to ensure that we could crate deployment servers.

mediawiki_memcached_servers:
    - 127.0.0.1:11211:1
"mediawiki::redis_servers::eqiad":
    shard01:
        host: 127.0.0.1
        port: 6379
"scap::deployment_server": scaptest.staging.eqiad.wmflabs
"scap::wmflabs_master": scaptest.staging.eqiad.wmflabs
"role::mediawiki::webserver::pool": one-pool-to-rule-them-all