Nova Resource:Cloudinfra/SAL

2023-06-01

• 13:27 dhinus: cloud-cumin-04 successfully restored from backup (T332734)
• 13:09 dhinus: shutting down cloud-cumin-04 and restoring from backup, to test backups are working (T332734)

2023-05-10

• 17:16 wm-bot2: Increased quotas by 8 cores, 16384 ram (T336423) - cookbook ran by taavi@runko

2023-04-17

• 15:05 taavi: add firewall rules allowing public (authenticated) access to the enc api - T317478

2023-02-16

• 15:49 arturo: aborrero@cloud-cumin-03:~$ sudo keyholder arm (password in pw)

2023-02-14

• 08:09 taavi: arm keyholder on enc-2 T329589
• 00:18 bd808: enc-2.cloudinfra.eqiad1.wikimedia.cloud: `shutdown -r now` (T329589)
• 00:13 bd808: enc-2.cloudinfra.eqiad1.wikimedia.cloud: `service puppet-enc-git-worker restart` (T329589)

2023-02-13

• 23:59 bd808: enc-1.cloudinfra.eqiad1.wikimedia.cloud: `service uwsgi-puppet-enc restart` (T329589)

2023-01-03

• 15:17 taavi: enc: create git update sql tables for https://gerrit.wikimedia.org/r/c/operations/puppet/+/871298

2022-12-21

• 13:58 taavi: re-enable puppet on cloud-cumin-03

2022-10-14

• 14:07 wm-bot2: Increased quotas by 1 instances - cookbook ran by dcaro@vulcanus

2022-07-20

• 07:03 dcaro: manually running fsck through costole on puppetmaster-03 do to disk errors (T313380)
• 06:54 dcaro: rebooting puppetmaster-03 do to disk errors

2022-02-16

• 14:44 taavi: delete old cloudinfra-db instances (cloudinfra-db01 cloudinfra-db02)
• 12:10 taavi: enable gtid (master_use_gtid=current_pos) on cloudinfra-db04

2022-02-13

• 17:14 taavi: encapi db switchover: cloudinfra-db01 -> cloudinfra-db03
• 11:38 taavi: generate new profile::mariadb::grants::cloudinfra::repl_pass, previous one seems to have been lost in the 2020-06-04 labs/private incident
• 10:48 taavi: shutdown and delete ntp-01 and ntp-02 running stretch, replaced by -03 and -04

2022-02-11

• 15:41 taavi: switch floating ip 185.15.56.3 from ntp-01 to ntp-03
• 15:18 taavi: switch floating ip 185.15.56.27 from ntp-02 to ntp-04

2022-02-09

• 18:33 taavi: deleted cloud-cumin-01 T255980
• 18:31 taavi: deleted cloud-cumin-02 T255980

2022-01-06

• 13:18 taavi: shut down cloud-cumin-02 T255980

2021-11-11

• 10:50 arturo: add user `srv-networktests` as project user (T294955)

2021-11-04

• 16:42 majavah: provisioning cloud-cumin-03 as bullseye

2021-10-06

• 09:47 dcaro: upgraded cloudmetrics to grafana 7.5 (T292614)

2021-07-19

• 22:49 bstorm: fixed rebase conflicts in labs-private
• 22:43 bstorm: disabling puppet on entire project before meddling with labs-private rebase

2021-05-05

• 14:33 arturo: delete also old VMs mx-out01/02
• 14:33 arturo: delete zones 'mx-out01.wmflabs.org' and 'mx-out02.wmflabs.org'

2021-05-04

• 15:58 arturo: shutoff mx-out01 and mx-out02 (migrated to mx-out03/mx-out04)
• 15:56 arturo: relocate floating IPs 185.15.56.18 and .19 from mx-out01/mx-out02 to mx-out03/mx-out04
• 15:33 arturo: created VMs mx-out03/mx-out04 as debian buster
• 15:31 arturo: bump instance quota from 12 to 14
• 15:28 arturo: created anti-affinity server group 'mx'

2021-03-01

• 10:37 dcaro: rebooting cloudinfra-acme-chief-01 to ensure hostname stability (T276041)

2021-02-26

• 20:46 andrewbogott: rebooting all hosts

2021-01-08

• 09:04 dcaro: manually testing patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/655019 to the puppetmaster to test (T271509)

2021-01-07

• 09:49 dcaro: Added recordset for mx-out02.wmflabs.org (T271322)
• 09:46 dcaro: Added recordset for mx-out01.wmflabs.org (T271322)

2021-01-05

• 14:11 dcaro: finished ssl tests for enc, cleaned up cloud-puppetmaster-03 (T268877)
• 13:07 dcaro: adding custom nginx config for labspuppetbackend on cloud-puppetmaster-03 to test ssl (T268877)
• 12:41 arturo: live-hacking cloudinfra-internal-puppetmaster02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/654415 (T260834)
• 12:31 arturo: refresh acme-chief config for mx certs https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/949f1b4e81f3a1c6d4f4825292343f1ee17c48a1%5E%21/ (T260834)
• 12:21 arturo: resolve git merge conflicts and rebase cloudinfra-internal-puppetmaster-02 /var/lib/git/labs/private
• 12:12 arturo: created puppet prefix `mx-out` and added hiera to use internal puppetmaster (T260834)

2020-11-09

• 10:23 arturo: added jmm (moritz) as user & projectadmin

2020-11-05

• 16:28 dcaro: add myself as user and projectadmin

2020-10-21

• 21:17 andrewbogott: deleting broken cloud-puppetmaster-04
• 21:14 andrewbogott: switching secondary puppetmaster from cloud-puppetmaster-04 to cloud-puppetmaster-05

2020-10-06

• 11:31 arturo: cleanup local changes in ops/puppet git repo in cloud-puppetmaster-03

2020-10-05

• 21:58 bstorm: setting "mtail::from_component: true" on both mx-out servers to make puppet work again

2020-09-02

• 15:01 arturo: linvehacking ended
• 14:20 arturo: live-hacking cloud-puppetmaster-03

2020-08-28

• 19:12 bstorm: moving aside weird old mitaka-jessie sources.list file on cloudinfra-db02

2020-08-04

• 18:24 bd808: Made DNS entry lists.wmcloud.org A 185.15.56.49 (Tried CNAME to mailman.wmcloud.org first but Horizon didn't like that) (T259444)

2020-07-31

• 20:08 bd808: Added nskaggs as projectadmin

2020-07-20

• 21:33 bd808: Created CNAME record for www.wmcloud.org pointing to wmcloud.org (T258415)
• 21:25 bd808: Created A record for wmcloud.org pointing to proxy-proxy well known IP (T258415)

2020-04-15

• 20:10 jeh: update default security group to allow prometheus01.metricsinfra.eqiad.wmflabs TCP 9100 T250206

2020-03-30

• 16:55 arturo: dropping `_psl.wmcloud.org` record (T168677)

2020-03-04

• 22:33 Krenair: Shutoff cloudinfra-internal-puppetmaster01, replaced with -02 per T241719

2020-02-21

• 15:46 jeh: cloud-puppetmaster-03 cleanup puppet agent certs that were missed by wmfsink
• 00:18 andrewbogott: temporarily shutting down cloud-puppetmaster-01 and -02 as part of debugging the new puppetmasters

2020-02-20

• 22:45 Krenair: Swapped 185.15.56.64 floating IP (backing puppetmaster.cloudinfra.wmflabs.org) over to cloud-puppetmaster-03 from cloud-puppetmaster-01

2020-01-28

• 10:03 arturo: delegated `codfw1dev.wmcloud.org` to designate @ codfw1dev ns0.openstack.codfw1dev.wikimediacloud.org (T242976 and T243766)
• 09:53 arturo: the DNS zone wmcloud.org now belongs to this project (T242976)

2020-01-02

• 23:34 mutante: cloud-puppetmaster-01 puppet cert clean puppetmaster-1001.devtools.eqiad.wmflabs

2019-11-29

• 10:29 arturo: re-arm keyholder in cloud-cumin-02 (password in pwstore)
• 10:13 arturo: re-arm keyholder in cloud-cumin-01

2019-11-09

• 18:30 Krenair: Drop internal-puppetmaster old cherry-pick of https://gerrit.wikimedia.org/r/c/operations/puppet/+/545567 which was preventing rebase and blocking cloud-puppetmaster machines from getting puppet commits themselves - new version of that change will have appeared there in the rebase

2019-10-23

• 09:55 arturo: manually restart mariadb in cloudinfra-db02 to fix replication
• 09:41 arturo: the cloudinfra-db01 VM was rebooted bc the hypervisor rebooted
• 09:41 arturo: manually start mariadb in cloudinfra-db01

2019-10-17

• 16:29 jeh: cleanup old fullstackd puppet certs
• 12:52 arturo: add phamhi as user and projectadmin

2019-09-09

• 18:23 Krenair: Started mariadb service on cloudinfra-db02 - was not running for some reason

2019-04-08

• 14:17 andrewbogott: moved cloudinfra-internal-puppetmaster01 to cloudvirt1026

2019-04-02

• 20:07 andrewbogott: moving cloudinfra-db02 to cloudvirt1017

2019-03-26

• 17:43 andrewbogott: removing "profile::ldap::client::labs::restricted_to: ops" from project puppet to allow volunteer admin access
• 17:21 andrewbogott: adding Krenair as projectadmin as per T218448

2019-01-07

• 21:08 gtirloni: upgraded kernel and rebooted mx-out{01,02}