Nova Resource:Cloudinfra/SAL

From Wikitech

2024-04-23

  • 07:13 taavi: taavi@cloudinfra-cloudvps-puppetserver-1:~$ sudo systemctl restart puppetserver

2024-04-18

  • 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloud-puppetmaster-05
  • 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance cloud-puppetmaster-05

2024-04-04

  • 14:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-acme-chief-01
  • 14:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-acme-chief-01
  • 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-internal-puppetmaster-02
  • 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-internal-puppetmaster-02

2024-04-03

  • 11:30 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance mx-out03
  • 11:29 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance mx-out03
  • 11:28 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance mx-out03
  • 11:28 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance mx-out03
  • 11:28 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance mx-out04
  • 11:28 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance mx-out04
  • 11:28 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance mx-out03
  • 11:28 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance mx-out03
  • 11:27 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance mx-out04
  • 11:27 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance mx-out04

2024-03-27

  • 12:23 taavi: hard reboot cloudinfra-cloudvps-puppetserver-1

2024-03-25

  • 18:34 andrewbogott: moved mx records to point to service names (mx-out-a.wmcloud.org and mx-out-b.wmcloud.org) and assoviated those service names with new mx servers mx-out05 and mx-out06

2024-02-21

  • 14:08 taavi: update cloudinfra-idp-1 to java 17 to fix CAS class loading errors on idp.wmcloud.org T357749

2023-12-01

  • 09:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_user_from_project (exit_code=0) for user 'jbond' (T352508)
  • 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_user_from_project for user 'jbond' (T352508)

2023-09-29

  • 09:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 08:34 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 08:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 08:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 08:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 08:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 08:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 08:28 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 08:25 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 08:25 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=97)
  • 08:25 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 08:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 08:08 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 07:41 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 07:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

2023-07-04

  • 12:32 taavi: configure DKIM signing for wmflabs.org and wmcloud.org on mx-out servers T208281

2023-06-19

  • 08:39 arturo: update delegation for codfw1dev.wmcloud.org. to point to ns0/ns1 T307357

2023-06-12

2023-06-08

  • 08:05 taavi: [codfw1dev] start mariadb.service on cloudinfra-db-01

2023-06-01

  • 13:27 dhinus: cloud-cumin-04 successfully restored from backup (T332734)
  • 13:09 dhinus: shutting down cloud-cumin-04 and restoring from backup, to test backups are working (T332734)

2023-05-10

  • 17:16 wm-bot2: Increased quotas by 8 cores, 16384 ram (T336423) - cookbook ran by taavi@runko

2023-04-17

  • 15:05 taavi: add firewall rules allowing public (authenticated) access to the enc api - T317478

2023-02-16

  • 15:49 arturo: aborrero@cloud-cumin-03:~$ sudo keyholder arm (password in pw)

2023-02-14

  • 08:09 taavi: arm keyholder on enc-2 T329589
  • 00:18 bd808: enc-2.cloudinfra.eqiad1.wikimedia.cloud: `shutdown -r now` (T329589)
  • 00:13 bd808: enc-2.cloudinfra.eqiad1.wikimedia.cloud: `service puppet-enc-git-worker restart` (T329589)

2023-02-13

  • 23:59 bd808: enc-1.cloudinfra.eqiad1.wikimedia.cloud: `service uwsgi-puppet-enc restart` (T329589)

2023-01-03

2022-12-21

  • 13:58 taavi: re-enable puppet on cloud-cumin-03

2022-10-14

  • 14:07 wm-bot2: Increased quotas by 1 instances - cookbook ran by dcaro@vulcanus

2022-07-20

  • 07:03 dcaro: manually running fsck through costole on puppetmaster-03 do to disk errors (T313380)
  • 06:54 dcaro: rebooting puppetmaster-03 do to disk errors

2022-02-16

  • 14:44 taavi: delete old cloudinfra-db instances (cloudinfra-db01 cloudinfra-db02)
  • 12:10 taavi: enable gtid (master_use_gtid=current_pos) on cloudinfra-db04

2022-02-13

  • 17:14 taavi: encapi db switchover: cloudinfra-db01 -> cloudinfra-db03
  • 11:38 taavi: generate new profile::mariadb::grants::cloudinfra::repl_pass, previous one seems to have been lost in the 2020-06-04 labs/private incident
  • 10:48 taavi: shutdown and delete ntp-01 and ntp-02 running stretch, replaced by -03 and -04

2022-02-11

  • 15:41 taavi: switch floating ip 185.15.56.3 from ntp-01 to ntp-03
  • 15:18 taavi: switch floating ip 185.15.56.27 from ntp-02 to ntp-04

2022-02-09

  • 18:33 taavi: deleted cloud-cumin-01 T255980
  • 18:31 taavi: deleted cloud-cumin-02 T255980

2022-01-06

  • 13:18 taavi: shut down cloud-cumin-02 T255980

2021-11-11

  • 10:50 arturo: add user `srv-networktests` as project user (T294955)

2021-11-04

  • 16:42 majavah: provisioning cloud-cumin-03 as bullseye

2021-10-06

  • 09:47 dcaro: upgraded cloudmetrics to grafana 7.5 (T292614)

2021-07-19

  • 22:49 bstorm: fixed rebase conflicts in labs-private
  • 22:43 bstorm: disabling puppet on entire project before meddling with labs-private rebase

2021-05-05

  • 14:33 arturo: delete also old VMs mx-out01/02
  • 14:33 arturo: delete zones 'mx-out01.wmflabs.org' and 'mx-out02.wmflabs.org'

2021-05-04

  • 15:58 arturo: shutoff mx-out01 and mx-out02 (migrated to mx-out03/mx-out04)
  • 15:56 arturo: relocate floating IPs 185.15.56.18 and .19 from mx-out01/mx-out02 to mx-out03/mx-out04
  • 15:33 arturo: created VMs mx-out03/mx-out04 as debian buster
  • 15:31 arturo: bump instance quota from 12 to 14
  • 15:28 arturo: created anti-affinity server group 'mx'

2021-03-01

  • 10:37 dcaro: rebooting cloudinfra-acme-chief-01 to ensure hostname stability (T276041)

2021-02-26

  • 20:46 andrewbogott: rebooting all hosts

2021-01-08

2021-01-07

  • 09:49 dcaro: Added recordset for mx-out02.wmflabs.org (T271322)
  • 09:46 dcaro: Added recordset for mx-out01.wmflabs.org (T271322)

2021-01-05

2020-11-09

  • 10:23 arturo: added jmm (moritz) as user & projectadmin

2020-11-05

  • 16:28 dcaro: add myself as user and projectadmin

2020-10-21

  • 21:17 andrewbogott: deleting broken cloud-puppetmaster-04
  • 21:14 andrewbogott: switching secondary puppetmaster from cloud-puppetmaster-04 to cloud-puppetmaster-05

2020-10-06

  • 11:31 arturo: cleanup local changes in ops/puppet git repo in cloud-puppetmaster-03

2020-10-05

  • 21:58 bstorm: setting "mtail::from_component: true" on both mx-out servers to make puppet work again

2020-09-02

  • 15:01 arturo: linvehacking ended
  • 14:20 arturo: live-hacking cloud-puppetmaster-03

2020-08-28

  • 19:12 bstorm: moving aside weird old mitaka-jessie sources.list file on cloudinfra-db02

2020-08-04

  • 18:24 bd808: Made DNS entry lists.wmcloud.org A 185.15.56.49 (Tried CNAME to mailman.wmcloud.org first but Horizon didn't like that) (T259444)

2020-07-31

  • 20:08 bd808: Added nskaggs as projectadmin

2020-07-20

  • 21:33 bd808: Created CNAME record for www.wmcloud.org pointing to wmcloud.org (T258415)
  • 21:25 bd808: Created A record for wmcloud.org pointing to proxy-proxy well known IP (T258415)

2020-04-15

  • 20:10 jeh: update default security group to allow prometheus01.metricsinfra.eqiad.wmflabs TCP 9100 T250206

2020-03-30

  • 16:55 arturo: dropping `_psl.wmcloud.org` record (T168677)

2020-03-04

  • 22:33 Krenair: Shutoff cloudinfra-internal-puppetmaster01, replaced with -02 per T241719

2020-02-21

  • 15:46 jeh: cloud-puppetmaster-03 cleanup puppet agent certs that were missed by wmfsink
  • 00:18 andrewbogott: temporarily shutting down cloud-puppetmaster-01 and -02 as part of debugging the new puppetmasters

2020-02-20

  • 22:45 Krenair: Swapped 185.15.56.64 floating IP (backing puppetmaster.cloudinfra.wmflabs.org) over to cloud-puppetmaster-03 from cloud-puppetmaster-01

2020-01-28

  • 10:03 arturo: delegated `codfw1dev.wmcloud.org` to designate @ codfw1dev ns0.openstack.codfw1dev.wikimediacloud.org (T242976 and T243766)
  • 09:53 arturo: the DNS zone wmcloud.org now belongs to this project (T242976)

2020-01-02

  • 23:34 mutante: cloud-puppetmaster-01 puppet cert clean puppetmaster-1001.devtools.eqiad.wmflabs

2019-11-29

  • 10:29 arturo: re-arm keyholder in cloud-cumin-02 (password in pwstore)
  • 10:13 arturo: re-arm keyholder in cloud-cumin-01

2019-11-09

  • 18:30 Krenair: Drop internal-puppetmaster old cherry-pick of https://gerrit.wikimedia.org/r/c/operations/puppet/+/545567 which was preventing rebase and blocking cloud-puppetmaster machines from getting puppet commits themselves - new version of that change will have appeared there in the rebase

2019-10-23

  • 09:55 arturo: manually restart mariadb in cloudinfra-db02 to fix replication
  • 09:41 arturo: the cloudinfra-db01 VM was rebooted bc the hypervisor rebooted
  • 09:41 arturo: manually start mariadb in cloudinfra-db01

2019-10-17

  • 16:29 jeh: cleanup old fullstackd puppet certs
  • 12:52 arturo: add phamhi as user and projectadmin

2019-09-09

  • 18:23 Krenair: Started mariadb service on cloudinfra-db02 - was not running for some reason

2019-04-08

  • 14:17 andrewbogott: moved cloudinfra-internal-puppetmaster01 to cloudvirt1026

2019-04-02

  • 20:07 andrewbogott: moving cloudinfra-db02 to cloudvirt1017

2019-03-26

  • 17:43 andrewbogott: removing "profile::ldap::client::labs::restricted_to: ops" from project puppet to allow volunteer admin access
  • 17:21 andrewbogott: adding Krenair as projectadmin as per T218448

2019-01-07

  • 21:08 gtirloni: upgraded kernel and rebooted mx-out{01,02}
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Cloudinfra/SAL&oldid=2171112"