Portal:Toolforge/Admin/emergency guides/single tool webservice

During the june 2015 NFS outage, we did an emergency deployment of the demw wiki loves earth jury tool on toolsbeta. This is a step-for-step guide what we did at that point, although some intermediate steps might be missing.

1) Find a suitable host.

• We chose a host on Toolforge because it had on-line instances that could be deleted safely afterwards (spinning up new instances took an hour or more).
• We chose a host with a suitable Security Group setup -- with port 8080 open to 10.0.0.0/8.
• We chose a host with a running web service setup so we could have an environment akin to the existing Toolforge one

Best host: toolsbeta-webgrid-04

2) Disable NFS and restart host (human intervention by Yuvi)

3) create basic structure in /data/project:

     /data/project/jury
     /data/project/jury/public_html (containing salvaged code)
     /data/project/jury/replica.my.cnf

4) hack around the safeguards in lighttpd-starter

• lighttpd-starter checks whether the server is allowed to start. It also tries to write in places where we don't have permission to write

--- /usr/local/bin/lighttpd-starter     2015-04-05 06:13:33.878601295 +0000
+++ lighttpd-starter   2015-07-03 21:45:21.945025980 +0000
@@ -23,14 +23,10 @@
 
 tool="$1"
 port="$2"
-prefix=$(/bin/cat /etc/wmflabs-project)
+prefix=tools #$(/bin/cat /etc/wmflabs-project)
 user="$prefix.$tool"
-home=$(getent passwd $user | cut -d : -f 6 | sed -e's/\/$//')
-if [ "$(getent group $user | cut -d : -f 1)" !="$user" ]; then
-  echo "$0: $1does not appear to be a tool" >&2
-  exit 1
-fi
-spool="/var/run/lighttpd"
+home=/data/project/jury #$(getent passwd $user | cut -d :-f 6 | sed -e 's/\/$//')
+spool="/data/project/jury/lighttpd"
 runbase="$spool/$tool"
 
 if ["$home" = "" -o ! -d "$home/public_html" ]; then

5) start the webserver in a screen

sudo su tools.jury
script /dev/null
screen
./lighttpd-starter jury 8080

6) Test if the webservice is running (ssh tunnel)

7) Disable webservice on Toolforge (requires root login)

• stop webservice, make sure it stays stopped (no service.manifest, but that's hard without NFS)

8) set up proxy to redirect to temporary host

• ssh to primary webproxy
• redis-cli
• hset prefix:<toolname> '.*' 'http://hostname:port'

9) test on tools.wmflabs.org