Obsolete:Labs Eqiad Migration/Howto
The Wikimedia Foundation is shutting down its Tampa datacenter ('pmtpa') and, with it, the Labs hosts that live there. A new Labs cluster is running in our new datacenter ('eqiad'). All labs projects and data need to be moved to eqiad in order to continue working.
Note: This page is about migrating individual labs projects. For documentation about moving your ToolLabs tool, visit the Tools migration page.
Ops: Tips and docs about how to migrate are on the poorly-named migration tools page.
Preparing to migrate
Slim down: Remove any big log files or unnecessary files from your instances or shared storage. Delete instances that are no longer in use. If your entire project is no longer in use, better yet! File a bug and the Labs ops will remove it.
Clean up puppet: If you are planning to move an instance to the new data center intact, it will need to have a functioning, up-to-date puppet install. For normal instances, run
$ sudo puppetd -tv
and make sure you see something like "Applying configuration version '1393501630'" in green. For instances that use self-hosted puppet, update your repo and then run puppet:
$ cd /var/lib/git/operations/puppet
$ sudo GIT_SSH=/var/lib/git/ssh git pull --rebase
[sudo] password for username: <enter your labs password>
$ sudo puppetd -tv
If you're unable to update your repo due to conflicts you'll need to stash your changes or resolve the conflicts by hand.
In either case, if you're unable to get puppet running on your instance then please ask for help in #wikimedia-cloud or open a bugzilla ticket.
Assisted migration of instances
Transferring an instance from pmtpa to eqiad requires the assistance of a Labs Op. If you want your instances transferred, please open a bugzilla ticket. Specify the name of your project and the names of the instances that need moving. Note that instance migration requires an instance to be shut down for several minutes. If you have a specific window that is preferred for your downtime please mention that in the ticket as well.
The instance migration process will leave your old pmtpa instance in a 'shutdown' state and produce a new eqiad instance with a different name and IP address but the same contents, security groups, puppet classes, etc.
Any floating IPs or web proxies will need to be reassigned to point to the new eqiad instances.
Note: Instances that use self-hosted puppet rather than upstream puppet may not survive assisted migration. For these instances it is highly recommended that you rebuild them by hand in eqiad.
Self-migration of instances
Many projects will simply build new instances in eqiad rather than transferring the old ones. If your instances are fully puppetized this should be a fairly simple process; you can move files between old and new instances using scp.
The contents of your shared /data/project or /home directories will not be immediately available in eqiad. A Labs Op can copy these files from pmtpa to eqiad -- keep in mind, however, that this transfer may take many hours (or, in the case of large projects, days), and shared data will be read-only in pmtpa once the transfer begins.
If you're feeling patient, please open a bugzilla ticket requesting transfer of your shared storage. Please specify whether you require a copy of /home, /data/project, or both.
Instances in eqiad have a fairly fast data connection to pmtpa instances. If you have instances running in both datacenters, you may be able to copy files between them using scp, like this:
new-host$ scp instancename.pmtpa.wmflabs:/data/project/myfile /data/project/myfile
If your project's security groups don't give you permission to do this, you can always copy from old-pmtpa-host to your laptop, then copy back out to new-eqiad-host.
Similar things can be accomplished using rsync; in general, though, you will see better performance if you make a big .tgz of all your files and copy them over in one go.
MediaWiki server
For MediaWiki database content, copying just files from one db host to another _might_ work, but only if everything else is 100% identical. Dump and import is always safer:
old-host$ mysqldump --all-databases -u root -p > /tmp/mybackup.sql old-host$ gzip /tmp/mybackup.sql
transfer and gunzip the file, then to restore DB testwiki:
new-host$ mysql -u root testwiki < /tmp/mybackup.sql
If you restore the DB to a new single-node MW instance (role mediawiki-install::labs), /srv/mediawiki/orig/adminpass will be incorrect.
Also back up and restore the images subdirectory. E.g. (create the directories first, or use alternate paths):
old-host$ sudo tar -pcjf /data/project/$(hostname)/$(hostname)_mediawiki_images.tar.bz2 /srv/mediawiki/images new-host$ scp $(hostname).pmtpa.wmflabs:/data/project/$(hostname)/$(hostname)_mediawiki_images.tar.bz2 /data/project/$(hostname)/ new-host$ cd /data/project/$(hostname) new-host$ sudo tar pxjf $(hostname)_mediawiki_images.tar.bz2 new-host$ sudo rm -r /srv/mediawiki/images new-host$ sudo mv srv/mediawiki/images /srv/mediawiki/
Web access in eqiad
If you move an instance with a public IP to eqiad, you're encouraged to create a proxy for the new instance rather than using a public IP. Any existing proxies will also need to be recreated in eqiad. Note that the old http://INSTANCENAME.instance-proxy.wmflabs.org that we've been using at pmtpa is going to be shut down in favor of this dynamic proxy system.
If that doesn't work (because you need access to other ports besides http and https) then open a bug requesting a floating IP for your project in eqiad. You will need to assign this new IP to your new instance, remove the old DNS entry from the old IP and set it for the new one. All this can be done via the Manage Addresses page.
Migration in progress?
If you have intentions to migrate your project but are still working on it or expect to work on it in the next week or two, please make a note with your name and plans on the migration progress page.
Feeling lazy?
If you do nothing at all (possibly because you never read this page) and the Labs ops don't hear from you, your project will not be deleted. It will, however, be mothballed at the end of the migration window. A mothballed project will have its instances left in a shutdown state and its shared storage stashed in a tarball. You will probably need assistance to revive your project or rescue your data.
Finished migrating?
Once your project is fully moved to eqiad, please move your project name from 'pending' to 'finished' on the migration progress page. Be warned that once a project has been moved to 'finished' all remaining instances and storage in pmtpa will be deleted without further notice.
Useless project?
If your project is no longer used for anything and you would like it to be deleted, move your project name to the 'Discarded' section on the migration progress page. Your instances and storage will be wiped to make room for future projects.
Troubleshooting
Security Groups
If you can't access your eqiad instances, look at the Manage Security Groups page.
Many of the standard firewall rules (aka 'Security Groups') for existing projects are tailored for pmtpa access. For instance, a rule like "22 22 tcp 10.4.0.0/21" allows ssh access for all pmtpa instances, but will not permit access from eqiad. The IP range for eqiad labs is 10.68.0.0/20. To allow access to all labs instances, use something like '10.0.0.0/8' or, to allow access to everyone everywhere, '0.0.0.0/0'
Accessing Eqiad Instances
During the transitional period, there will be bastions in both pmtpa and eqiad. The pmtpa bastions have the same names as always, like bastion.wmflabs.org, bastion2.wmflabs.org, etc. The eqiad bastions are named bastion-eqiad.wmflabs, bastion2-eqiad.wmflabs.org, etc. Fingerprints for all of the bastions are here.
If you log into a tampa bastion and want to access an eqiad instance (or the reverse) you will need to specify a fully-qualified name. For example:
$ ssh -A bastion.wmflabs.org bastion1:~$ ssh mynewinstance.eqiad.wmflabs
Of course if you are going to scp between datacenters, you'll need to forward your key further:
$ ssh -A bastion.wmflabs.org bastion1:$ ssh -A mynewinstance.eqiad.wmflabs mynewinstance:$ scp myoldinstance.pmtpa.wmflabs:myfavoritefile.zip .
Post migration, the old pmtpa bastion names will be redirected to point at the eqiad bastions. When that happens you'll see a man-in-the-middle warning and will need to reset your host keys.