Jump to content

Fundraising/techops/procedures/services-analytics datacenter swap

From Wikitech
  1. sync all the minio data
  2. stop all update jobs
  3. sync minio data final time
    1. Verify access tokens in eqiad minio setup
  4. put analytics host/services into maint mode
  5. switch analytics mariadb origin server
  6. point all the codfw bits at eqiad
    1. update hieradata/common.yaml
      • analytics_db_writer: 'frdb1007.frack.eqiad.wmnet'
      • analytics_active_server: 'fran1002.frack.eqiad.wmnet'
      • le_certbot_host: analytics: 'fran1002.frack.eqiad.wmnet'
    2. update hieradata/role/analytics.yaml
      • frdeploy_local_localsettings: repo: 'fran1002.frack.eqiad.wmnet:/var/lib/git/localsettings.git'
    3. fundraising_code_update on fran1002
  7. apply db permissions to both dbs for maximum middle-origin potential
    1. `make_grants | sudo mysql` on both hosts
  8. restart various services
  9. Update public DNS for fran sites
  10. migrate jupyter
    • hosts.yaml from fran2001 to fran1002
    • copy user data from fran2001:/home to fran1002
    • copy /srv/jupyter from fran2001
  11. clobber frban1002:/srv/banner_logs/2026 with frban1002:/srv/banner_logs-frban2002.frack.codfw.wmnet/2026
  12. migrate process-control jobs to fran1002
  13. update metabase trino connection with new host
  14. test reads/ writes with trino
  15. check for dupes
  16. test metabase

... Profit

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Fundraising/techops/procedures/services-analytics_datacenter_swap&oldid=2413593"