Obsolete:Labs Eqiad Migration

From Wikitech

Tools to use during migration (well, one tool at least) are documented here: Labs_Eqiad_Migration_Tools


  1. Set up 2nd openstack infra in labs Yes Done
    1. get eqiad labs servers properly organized and named Yes Done
    2. Make sure to bond ports for the network node, or use 10Gb Yes Done
    3. Prepare new puppet classes for Havana Yes Done
  2. Ensure OpenStackManager works properly with two zones Yes Done
  3. Try the upstream OpenStack modules (Nope!)
  4. Investigate neutron (Nope!)
  5. Set up eqiad, using havana nova-network (Andrew)
    1. Set up keystone Yes Done
    2. Set up glance, migrate images from pmtpa Yes Done
    3. Set up nova services Yes Done
      1. All services running and communicating Yes Done
    4. Update eqiad labs network config Yes Done
      1. Move to a larger fixed IP range (currently limited to Yes Done
      2. Rearrange interfaces on labnet1001 so that we're using 10g everywhere Yes Done
        1. Reassign eth4.1102 to eth1.1102 Yes Done
        2. Switch eth0 to 10g Yes Done
      3. Figure out why labnet1001 has virbr0/192.168.122 (removed) Yes Done
      4. Instances cannot access virt0/virt1000/other public wmf boxes Yes Done
      5. DNS Yes Done
    5. Set up /public/keys shared volume in eqiad nfs Yes Done
    6. Create new eqiad bastion cluster
    7. Set up wikitech mediawiki mirror on virt1000
    8. Rewrite the instance wikistatus code to work with havana Yes Done
  6. Test Yes Done
  7. Migrate shared storage
  8. Make region-specific puppet changes so instances can move from tampa to eqiad Yes Done
  9. Update OpenStackManager on wikitech to be aware of both datacenters Yes Done
    1. Test keystone/redis with havana Yes Done
    2. Deploy keystone/redis on virt1001 Yes Done
    3. Update virt1001 keystone with tampa services, test Yes Done
    4. Point wikitech DNS to virt1001 -- or -- deploy multi-region changes on virt0 Yes Done (the latter)
  10. Set up new service group schema to support nfs on eqiad Yes Done
  11. Move some test instances from a project (begin week of 2014-01-27) (mostly Andrew): Yes Done
    1. Creating test instances in pmtpa
    2. Creating instances with the same name in eqiad
    3. Shut down the instances in eqiad and pmtpa
    4. Transfer the images from pmtpa to eqiad
    5. Start the instances in eqiad and ensure a puppet run makes the system work properly
    6. Delete the instances in pmtpa
  12. Get deployment-prep and tools to start moving (begin week of 2014-02-03)
  13. Start moving projects, scheduling each one


  • 2012-02-21: Internal test migrations. Requires use of wikitech and virt1000.
  • 2012-02-24: Tools, Deployment-prep migration begins (carefully). Wikitech exposes access to both datacenters.
  • 2012-03-03: Public migration period begins. New instance creation and IP allocation disabled in pmtpa.
  • 2012-03-18: Deadline for self-migration. After this, labs staff starts to shutdown and transfer remaining instances, where they will remain in SHUTOFF state until a specific request is made.

Relevant RT tickets

Relevant Bugzilla tickets

All open Labs related bugs in Bugzilla

Meeting notes and other pages

project migration example (gluster)

  1. Eqiad already knows about all the projects in pmtpa. So nothing needs to be done regarding project creation, security groups, or service groups.
  2. Migrate shared storage (home) by rsyncing from labstore2:
   root@labstore2:/tmp# mkdir -p /tmp/testlabs/home
   root@labstore2:/tmp# mount -t glusterfs localhost:/testlabs-home /tmp/testlabs/home
   root@labstore2:/tmp# rsync -S -av /tmp/testlabs/home labstore1001.eqiad.wmnet:/srv/project/testlabs/
   root@labstore2:/tmp# umount /tmp/testlabs/home
   root@labstore2:/tmp# rmdir /tmp/testlabs/home
  1. Migrate shared storage (project)
   root@labstore2:/tmp# mkdir -p /tmp/testlabs/project
   root@labstore2:/tmp# mount -t glusterfs localhost:/testlabs-project /tmp/testlabs/project
   root@labstore2:/tmp# rsync -S -av /tmp/testlabs/project labstore1001.eqiad.wmnet:/srv/project/testlabs/
   root@labstore2:/tmp# umount /tmp/testlabs/project
   root@labstore2:/tmp# rmdir /tmp/testlabs/project
  1. Alternately, with tar (from NFS):
   root@labstore1001:~# ssh labstore4.pmtpa.wmnet tar cf - -C /srv deployment-prep | tar xf - -C /srv/project