Portal:Toolforge/Admin/emergency guides/single tool webservice
During the june 2015 NFS outage, we did an emergency deployment of the demw wiki loves earth jury tool on toolsbeta. This is a step-for-step guide what we did at that point, although some intermediate steps might be missing.
1) Find a suitable host.
- We chose a host on Toolforge because it had on-line instances that could be deleted safely afterwards (spinning up new instances took an hour or more).
- We chose a host with a suitable Security Group setup -- with port 8080 open to 10.0.0.0/8.
- We chose a host with a running web service setup so we could have an environment akin to the existing Toolforge one
Best host: toolsbeta-webgrid-04
2) Disable NFS and restart host (human intervention by Yuvi)
3) create basic structure in /data/project:
/data/project/jury /data/project/jury/public_html (containing salvaged code) /data/project/jury/replica.my.cnf
4) hack around the safeguards in lighttpd-starter
- lighttpd-starter checks whether the server is allowed to start. It also tries to write in places where we don't have permission to write
--- /usr/local/bin/lighttpd-starter 2015-04-05 06:13:33.878601295 +0000 +++ lighttpd-starter 2015-07-03 21:45:21.945025980 +0000 @@ -23,14 +23,10 @@ tool="$1" port="$2" -prefix=$(/bin/cat /etc/wmflabs-project) +prefix=tools #$(/bin/cat /etc/wmflabs-project) user="$prefix.$tool" -home=$(getent passwd $user | cut -d : -f 6 | sed -e's/\/$//') -if [ "$(getent group $user | cut -d : -f 1)" !="$user" ]; then - echo "$0: $1does not appear to be a tool" >&2 - exit 1 -fi -spool="/var/run/lighttpd" +home=/data/project/jury #$(getent passwd $user | cut -d :-f 6 | sed -e 's/\/$//') +spool="/data/project/jury/lighttpd" runbase="$spool/$tool" if ["$home" = "" -o ! -d "$home/public_html" ]; then
5) start the webserver in a screen
sudo su tools.jury script /dev/null screen ./lighttpd-starter jury 8080
6) Test if the webservice is running (ssh tunnel)
7) Disable webservice on Toolforge (requires root login)
- stop webservice, make sure it stays stopped (no service.manifest, but that's hard without NFS)
8) set up proxy to redirect to temporary host
- ssh to primary webproxy
- hset prefix:<toolname> '.*' 'http://hostname:port'
9) test on tools.wmflabs.org