User:Elukey/Ops/MediaWikiHostRepool

Background

https://phabricator.wikimedia.org/T122005

Target hosts

mw1228.eqiad.wmnet

Prerequisites

Software up to date on the host

ssh mw1228.eqiad.wmnet
sudo apt-get update && sudo apt-get upgrade

Puppet is running correctly

sudo service puppet status
* agent is not running
sudo puppet agent --enable

Accept Salt key

ssh neodymium.eqiad.wmnet
salt-key -a mw1228.eqiad.wmnet

Force puppet run

ssh mw1228.eqiad.wmnet
sudo puppet agent -tv

Add the host back to the DSH list

The opposite of https://gerrit.wikimedia.org/r/#/c/260251/

Get the code review approved, +2 and then merge it in puppet. Either wait for the next puppet run or just force a puppet agent -tv on mw1228

Run scap pull on the host (was sync-common)

Background Wikimedia binaries#scap pull

ssh mw1228.eqiad.wmnet
scap pull

Final check before repooling

sudo service apache2 restart
curl -i http://localhost/

Add the host to conftool data

File a code review for conftool-data/nodes/eqiad.yaml adding the mw1228 host to api_appserver.

Review, +2 and the puppet-merge on Palladium. Wait for puppet to run.

Repool the host

ssh palladium.eqiad.wmnet
confctl --find --action set/pooled=yes mw1228.eqiad.wmnet

Final metric check

https://grafana.wikimedia.org/dashboard/db/server-board => mw1228

ssh mw1228.eqiad.wmnet
sudo tail -n 100 /var/log/apache2/other_vhosts_access.log | awk '{print $4" "$6}'  | sort | uniq -c | sort -n -k 1

You should see proxied/200s. Please also check for a while the access logs with tail -f to ensure that the host is running correctly.