Obsolete:Yaseo migration plan

Obsolete

Planning to start around 18:00 UTC Friday, 29 September 2006.

Now with Tim-ish goodness:

1. Take dryas out of rotation
2. Stop replication on dryas
3. Save master position from dryas
   • 18:22 done. Exec_master_log_pos: 731100395
4. Make a mysqldump of (ko|ja|ms|th)wiki from dryas
   • 18:23 started. should take about 45 minutes
5. rsync uploads over while that's going
   • 18:36 started
   • 23:?? finished
6. Transfer dump to pmtpa
   • 19:08 started
7. Read yaseo dump into pmtpa non-english slaves
   • 21:18 started [adler, lomaria, thistle, ixia, db1]
   • 23:50ish finished. adler disk space is low due to extra binlogs, which can be trimmed
8. Switch pmtpa non-english masters
   1. Switch non-enwiki wikis to read-only
      • 00:36
   2. Set samuel to global read-only
      • 00:38
   3. Switch master setting in wikis
      • 00:40
   4. Stop replication
      • 00:42
   5. RESET MASTER on adler to start a new binlog
   6. CHANGE MASTER TO on all slaves including samuel
      • 00:44
   7. START SLAVE on all slaves, check that they are connecting
      • 00:46
   8. Set adler to read/write
   9. Set wikis to read/write
      • 00:47
   10. Edit my.cnf on adler and samuel so that samuel starts in read-only mode and adler doesn't
       • 00:51
9. Read yaseo dump into the new pmtpa non-english slave
   • 00:53 started
   • 01:01 had to switch masters also on holbach, which got forgotten (thought it was offline)
   • 03:13 started reading the jawiki import again; had filled disk with binlogs and needed to free space and resync things
   • 05:38 done.
10. Lock yaseo wikis to read-only
11. rsync uploads again
    • 05:52
12. Generate delta from henbane with mysqlbinlog
13. Transfer delta dump to pmtpa
    • 06:02
14. Read delta dump into pmtpa non-english master
15. Fiddle with configuration files so pmtpa knows about former yaseo wikis
    • 06:26
16. Switch squid configurations to send them at pmtpa
    • 06:30ish

And fiddled with external storage clusters 6 and 7, 06:35

old notes

It may be advisable to run external-storage compression on jawiki first to avoid bloating the database; about half of the 68GBs are jawiki's text table.

• Safer not to, though, to make sure there won't be conflicts or need to set up a separate cluster just for the ja transfer
• There's ~90gb of free innodb space on adler and samuel, so the full data should import ok. It will bloat slaves a little but they'll survive.

jawiki currently has a blob table listed as cluster3 which contains about 2gb of older compressed data. Ensure that this won't conflict with the move back to pmtpa.

• Don't bother transferring it; this data is still sitting in cluster3, ready to be picked right back up.

Uploads also have to be transferred in here somewhere.

You don't need to have the yaseo wikis locked for the entire duration of the transfer. Here's what I did when I moved them in the other direction: I made a dump from a stopped slave, recorded the master position. Transferred the backup, read it into the destination MySQL server. Only then do you lock the wikis. Then you use mysqlbinlog to generate an SQL file updating from the time you made the snapshot to the time you locked the wikis. It will be relatively small. You transfer that, read it in to the destination, then do the configuration switch. It should save hours of read-only time. -- Tim 04:50, 29 September 2006 (PDT)