Search servers relocation - 12 august 2010

Batch one: searchidx1, search1 - search12

Batch two: search13 - search20

Before moving servers

 $wgEnableLucenePrefixSearch = true;

This will re-route ajax suggestions from search boxes back to the database.

Before turning off and relocating search1-12 do the following:

on lvs3, searchpool1 should be "search14, search18, search19, search20", searchpool2 should be "search16", searchpool3 should be "search17". Make sure to backup the old configuration.
In /home/wikipedia/common/php/wmf-config/lucene.php replace

  '10.0.3.11' # search 11

with

  '10.0.3.13' # search 13

bring up search1-12 and make sure lsearchd is started ("sudo /etc/init.d/lsearchd start")
bring up searchidx1 - lsearchd should *not* be running here - I'll start it later
revert the changes from previous section, i.e. revert changes on lvs3 and on /home/wikipedia/common/php/wmf-config/lucene.php. This will direct queries back to search1-12.
Try some test queries on en.wp, de.wp and commons. Allow up to 15 minutes for search to start working (it might take some time for linux to cache the appropriate pages, will see lots of I/O in ganglia stats). If it still doesn't work after 15 minutes, and no CPU is used on any of search1-12, revert the changes and wait for tomorrow to do the second batch.
if everything is fine, search13-20 are ready for relocation and can be switched off