Jump to content

Phabricator/Disaster Recovery

From Wikitech

Disaster recovery plan for phabricator.wikimedia.org

High level overview

  • Switch Phabricator production to phab2002 (in codfw)
    • verify that git-ssh is working on phab2002
    • shut down phd on phab1004
    • manually rsync the git data from phab1004 to phab2002 (this is already rsync'd periodically, just need a one-time refresh)
    • verify that phd works on phab2002
    • switch `phabricator_active_server` to `phab2002` in hiera, `role/common/phabricator/main.yaml`
    • test phabricator's web interface by locally overriding dns records to point to phab2002
  • update dns / redirect traffic to phab2002
    • Update SFP records for the phabricator hostname (example)
  • Make phab1004 a warm standby for phab2002
    • verifying that everything is installed correctly
    • manually rsync the git repositories.
    • make sure the rsync cron job is set up correctly to sync from phab2002

See Also