Obsolete:Grosley-Aluminium-Migration
(Redirected from Grosley-Aluminium-Migration)
This page contains historical information. It may be outdated or unreliable.
as of Tue Nov 1 16:19:22 EDT 2011
Building Aluminium
1. base+puppet install
2. then a bunch of stuff that hasn't been puppetized
svn co http://svn.wikimedia.org/svnroot/wikimedia/branches/deployment/fundraising-civicrm/d620c34 /srv/org.wikimedia.civicrm
svn co http://svn.wikimedia.org/svnroot/wikimedia/trunk/fundraising-civicrm /srv/org.wikimedia.civicrm-dev
svn co http://svn.wikimedia.org/svnroot/wikimedia/branches/deployment/fundraising-civicrm/d620c34 /srv/org.wikimedia.civicrm-stage
svn co http://svn.wikimedia.org/svnroot/wikimedia/branches/deployment/fundraising-misc /opt/fundraising-misc
mkdir /srv/org.wikimedia.fundraising
puppet --test
svn co http://svn.wikimedia.org/svnroot/wikimedia/vendors/drush/current /opt/drush
chmod a+x /opt/drush/drush
ln -s /opt/drush/drush /usr/local/bin/drush
svn co http://svn.wikimedia.org/svnroot/wikimedia/vendors/django/current/ /usr/local/src/django_src
cd /usr/local/src/django_src
python setup.py install
3. copy over faulkner's django reporting stuff [document, package, put in svn . . . something]
4. some form of python/django config [document!]
fetch httpagentparser-0.8.2.tar.gz
cd /usr/locals/src; tar xzf httpagentparser-0.8.2.tar.gz; cd httpagentparser-0.8.2; python setup.py install
Cutover Prep
1) finish config cleanup [jgreen]
- stuff that's installed from svn--most is under /srv
- /etc/php5/apache2/php.ini - puppetize from grosley copy **DONE**
- /usr/sbin/gmond missing on Aluminium. puppetized and check ganglia. **DONE**
- dpkg --get-selections double-check **DONE**
2) mail. puppet config dealt with outbound. make sure any inbound mail works. [jgreen] **DONE**
3) jenkins. copy over jobs from grosley to aluminium. strategy for making one active [jgreen, arichards]
- consider keeping jobs in svn? http://jenkins-ci.org/content/keeping-your-configuration-and-data-subversion
- jenkins cron jobs:
0 12 * * * /root/jenkins.tarring.sh
*/5 * * * * /usr/local/scripts/jenkins_watcher
4) dump storage3 faulkner database and take offline [jgreen]
- disable Faulkner's automatic analytic scripts by disabling cronjob [rfaulk] **DONE**
- copy user-mysql permissions from storage3 to db1008
* dumped to storage3:/archive/backups/20111103-storage3-mysql-mysql.sql.gz
- flush/lock and dump faulkner database to storage3:/archive/backups/20111103-storage3-mysql-faulkner.sql.gz **DONE**
- stop mysql on storage3 **DONE**
- tar/gz storage3 database to storage3:/archive/backups/20111103-storage3-mysql.sqldata.tgz **DONE**
5) prep db1008 to be master for faulkner db, otrs-free, but still middle-master for civi/fundraising
- configure db1008 to skip slaving of faulkner, otrs db's from db9 and restart **DONE**
- drop otrs db from db1008 **DONE**
- restore faulkner db to db1008 **DONE**
6) reconfigure storage3 as slave of db1008
- apt-get upgrade and reboot storage3 (punted on dist-upgrade) **DONE**
- update storage3 to the -facebook mysql version, configure like db1025 **DONE**
- use db1025 to re-image storage3 **DONE**
7) move faulkner's scripts and analytics to aluminium+db1008/db1025 (dns switch, cron jobs, etc) [jgreen, rfaulk]
Cutover Event
1) stop everything on grosley/aluminium that uses db9
- apache **DONE**
- jenkins **DONE**
2) stop slave and remove slave settings on db1008 so it becomes fundraising master
- STOP SLAVE; **DONE**
- CHANGE MASTER TO MASTER_HOST=''; **DONE**
3) discover that with the puppet switch, we picked up skip-name-resolve and must redo mysql auth IP-based **DONE**
4) hit mysql bad behavior around privileges tables, attempt fetch from db9 + upgrade and fail, rebuild privileges table from scratch **DONE**
GRANT LOCK TABLES,SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES ON `civicrm`.* TO 'civicrm'@'blah' IDENTIFIED BY 'bleh';
ERROR 1133 (42000): Can't find any matching row in the user table
5) discover there was stuff on payments* that uses fundraising db's on db9, reconfigure **DONE**
- /srv/www/org/wikimedia/payments/.LocalSettingsNoBackup.php
6) discover there was stuff on prod that uses fundraising db's on db9, reconfigure **DONE**
- /home/wikipedia/common/wmf-config/contribution-tracking-setup.php
7) flip various DNS to point to aluminium
- web hosts **DONE**
- donate.wikimedia.org mx record [jgreen] **DONE**
8) reconfigure everything on grosley/aluminium that points to db9/db10 to use instead db1008/db1025 [jgreen/arthur]
- /srv/org.wikimedia.civicrm/sites/default/settings.php **DONE**
- /srv/org.wikimedia.civicrm/sites/default/civicrm.settings.php **DONE*
- /srv/org.wikimedia.civicrm-dev/sites/default/settings.php **DONE**
- /srv/org.wikimedia.civicrm-dev/sites/default/civicrm.settings.php **DONE**
- /opt/fundraising-misc/public_reporting/update.php **DONE**
- /opt/fundraising-misc/queue_handling/paypal/IPN/IPNListener_Recurring.php **DONE**
- /opt/fundraising-misc/queue_handling/paypal/IPN/IPNListener_Standalone.php **DONE**
9) reconfigure/modify faulkner's analytics stuff to use new master/slave db's [faulkner]
- /srv/org.wikimedia.fundraising-analytics/Fundraiser_Tools/config/settings.py
- /srv/org.wikimedia.fundraising-analytics/Fundraiser_Tools/scripts/settings.py (settings file for shell scripts)
- /srv/org.wikimedia.fundraising-analytics/Fundraiser_Tools/web_reporting/settings.py (no references to dbs but a config file not stored in svn)
- /srv/org.wikimedia.fundraising-analytics/classes/DataLoader.py (references to storage3 appear to be isolated in code comments)
- creation of /srv/org.wikimedia.fundraising-analytics/mplconfigdir with full access rights -> python matplotlib module dependency (part of env setup)
- /srv/org.wikimedia.fundraising-analytics/Fundraiser_Tools/web_reporting/static/images/ full access rights -> data image files are written here (part of env setup)
- /srv/org.wikimedia.fundraising-analytics/fundraiser-analysis -- checkout http://svn.wikimedia.org/svnroot/wikimedia/branches/deployment/fundraiser-analysis/ here
10) restart services on aluminium with new settings **DONE**
11) test test test fix test test fix test fix test
Post-Cutover Cleanup
1) get storage3 my.cnf into puppet and reenable puppet on storage3
2) clean up puppetization of my.cnf for db1008 **DONE**
3) fix the config mess for /opt/fundraising-misc/public_reporting/update.php
4) Check out stomp library from svn into /opt/stomp (svn co http://svn.wikimedia.org/svnroot/wikimedia/vendors/stomp_php/pre-1.0.0 /opt/stomp)
5) puppetize grosley:/etc/php5/cli/php.ini
6) figure out who still used storage3, get their privileges back into db1025/storage3
7) restore stored procedures to mysql.civicrm **DONE**
http://svn.wikimedia.org/viewvc/wikimedia/branches/deployment/fundraising-civicrm/d620c34/sites/all/bin/public_reporting/triggers.sql?revision=161&view=co
8) get faulkner's js libraries and configs into puppet, svn, whatever