Nova Resource:Clouddb-services/SAL

From Wikitech
Jump to navigation Jump to search

2021-09-02

  • 18:52 bstorm: removed strange old duplicate cron for osmupdater T285668

2021-08-31

  • 20:19 bstorm: attempting to resync OSMDB back to Feb 21st 2019 T285668

2021-08-30

  • 22:07 bstorm: restarting osmdb on clouddb1003 to try to capture enough connections T285668
  • 21:53 bstorm: disable puppet and osm updater script on clouddb1003 T285668

2021-05-26

  • 16:53 bstorm: restarting postgresql since T220164 was closed. Hoping all connections don't get used up again.

2021-05-25

  • 14:35 dcaro: taking down clouddb1002 replica for reboot of cloudvirt1020 (T275893)

2021-05-04

  • 22:57 bstorm: manually added cnames for toolsdb, osmdb and wikilabelsdb in db.svc.wikimedia.cloud zone T278252

2021-04-05

  • 09:56 arturo: make jhernandez (IRC joakino) projectadmin (T278975)

2021-03-10

  • 12:44 arturo: briefly stopping VM clouddb-wikireplicas-proxy-2 to migrate hypervisor
  • 10:57 arturo: briefly stopped VM clouddb-wikireplicas-proxy-1 to disable VMX cpu flag

2021-03-02

  • 23:29 bstorm: bringing toolsdb back up 😟

2021-02-26

  • 23:20 bstorm: rebooting clouddb-wikilabels-02 for patches
  • 22:55 bstorm: rebooting clouddb-wikireplicas-proxy-1 and clouddb-wikireplicas-proxy-2 before (hopefully) many people are using them

2021-01-29

  • 18:21 bstorm: deleting clouddb-toolsdb-03 as it isn't used

2020-12-23

  • 19:20 bstorm: created clouddb-wikireplicas-proxy-1 and clouddb-wikireplicas-proxy-2 as well as the 16 neutron ports for wikireplicas proxying

2020-12-17

  • 02:14 bstorm: toolsdb is back and so is the replica T266587
  • 01:10 bstorm: the sync is done and we have a good copy of the toolsdb data, proceeding with the upgrades and stuff to that hypervisor while configuring replication to work again T266587

2020-12-16

  • 18:34 bstorm: restarted sync from toolsdb to its replica server after cleanup to prevent disk filling T266587
  • 17:31 bstorm: sync started from toolsdb to its replica server T266587
  • 17:29 bstorm: stopped mariadb on the replica T266587
  • 17:28 bstorm: shutdown toolsdb T266587
  • 17:24 bstorm: settings toolsdb to readonly to prepare for shutdown T266587
  • 17:06 bstorm: switching the secondary config back to clouddb1002 in order to minimize concerns about affecting ceph performance T266587

2020-11-15

  • 19:45 bstorm: restarting the import to clouddb-toolsdb-03 with --max-allowed-packet=1G to rule out that as a problem entirely T266587
  • 19:36 bstorm: set max_allowed_package to 64MB on clouddb-toolsdb-03 T266587

2020-11-11

  • 08:20 bstorm: loading dump on the replica T266587
  • 06:06 bstorm: dump completed, transferring to replica to start things up again T266587

2020-11-10

  • 16:24 bstorm: set toolsdb to read-only T266587

2020-10-29

  • 23:49 bstorm: launching clouddb-toolsdb-03 (trying to make a better naming convention) to replace failed replica T266587

2020-10-27

  • 18:04 bstorm: set clouddb1002 to read-only since it was not supposed to be writeable anyway T266587

2020-10-20

  • 18:36 bstorm: brought up mariadb and replication on clouddb1002 T263677
  • 17:14 bstorm: shutting down clouddb1003 T263677
  • 17:13 bstorm: stopping postgresql on clouddb1003 T263677
  • 17:08 bstorm: poweroff clouddb1002 T263677
  • 17:08 bstorm: stopping mariadb on clouddb1002 T263677
  • 17:07 bstorm: shut down replication on clouddb1002 (now with task) T263677
  • 17:05 bstorm: shut down replication on clouddb1002

2020-09-29

  • 23:48 bstorm: restarting postgresql on clouddb1003 to clear the number of connections

2020-09-08

  • 18:21 bstorm: copied the profile::mariadb::section_ports key into prefix puppet to fix puppet after the refactor for wmfmariadbpy

2020-07-16

  • 17:56 bstorm: Significantly lifted traffic shaping limits on clouddb1001/toolsdb to improve network performance T257884

2020-06-09

  • 22:55 bstorm_: Reset the passwords for T254931

2020-06-04

  • 17:41 bd808: Restarting mariadb after hand applying {{gerrit|602433}} (T253738)

2020-06-01

  • 18:51 bstorm_: restarting mariadb on clouddb1002 T253738

2020-05-29

  • 23:42 bstorm_: stopped puppet and restarting mariadb on clouddb1002 after filtering out a table T253738

2020-03-27

  • 12:26 arturo: add myself as project admin

2020-03-20

  • 13:43 jeh: upgrade puppetmaster to v5 T241719

2020-03-18

  • 17:49 bstorm_: updated the tools-prometheus security group with the current IP addresses for the prometheus servers

2019-10-25

2019-10-24

  • 18:34 bstorm_: downgraded clouddb1001 to 10.1.39 T236420 T236384
  • 18:19 bstorm_: stopped puppet on clouddb1001 and removed unattended-upgrades for now T236384
  • 16:03 bstorm_: stopped puppet on clouddb1002 and removed unattended-upgrades for now T236384
  • 13:27 phamhi: add phamhi as user and projectadmin
  • 12:34 arturo: start both clouddb1001 and clouddb1004 (T227540)
  • 11:15 arturo: poweroff VM clouddb1001, hypervisor will be powered off (T227540)
  • 11:13 arturo: poweroff VM clouddb1004 (T227540)
  • 11:09 phamhi: stopped postgresl in clouddb1004 (T227540)
  • 11:04 arturo: stopped mariadb in clouddb1001 (T227540)

2019-09-30

  • 17:15 bstorm_: restart wikilabels postgresql
  • 17:15 bstorm_: restart osmdb on clouddb1003

2019-09-16

  • 19:41 jeh: add openstack security group rules for Tools prometheus scrapes to mariadb exporter
  • 19:28 jeh: add openstack security group rules for labmon prometheus scraps to mariadb exporter

2019-05-24

  • 23:38 bstorm_: T224062 clouddb1002 is now free of postgresql services and the volume is reclaimed for toolsdb use
  • 23:06 bstorm_: T224318 cloudb-wikilabels-02 is now a replica of clouddb-wikilabels-01!
  • 21:00 bstorm_: T224062 Moved wikilabels postgres db to clouddb-wikilabels-01

2019-05-22

  • 00:14 bstorm_: T224062 wikilabels postgres is now replicating to clouddb-wikilabels-01

2019-04-05

  • 00:35 bstorm_: added the rsyncd ports to security groups for the osmdb cluster

2019-04-04

  • 23:58 bstorm_: pg_basebackup finished on clouddb1004 and streaming replication is flowing
  • 18:46 bstorm_: T219652 pg_basebackup started on clouddb1004
  • 18:35 bstorm_: T219652 postgresql on clouddb1003 had to be restarted to get it to accept WAL connections
  • 18:00 bstorm_: T219652 clouddb1003 is now the OSMdb primary

2019-03-28

  • 23:17 bstorm_: clouddb1003 is now a full osmdb replica T193264
  • 19:32 bstorm_: pg_basebackup started on clouddb1003 for osmdb T193264

2019-02-28

  • 02:09 bstorm_: clouddb1002 is now the replica for clouddb1001/toolsdb and is catching up slowly

2019-02-18

  • 19:01 bstorm_: labsdb1004 is now a replica of clouddb1001, which is toolsdb now.
  • 18:26 arturo: (jaime T193264) setting clouddb1001 in read_write mode
  • 18:12 arturo: T193264 pointing tools.db.svc.eqiad.wmflabs to clouddb1001
  • 17:55 arturo: (jaime T193264) set clouddb1001 in read_only=1
  • 16:11 andrewbogott: deleting clouddb-utils-01 VM and associated puppet prefix; we aren't going to run maintain_dbusers here after all
  • 15:51 andrewbogott: removing dns and public IP from clouddb1001

2019-02-17

  • 21:21 bstorm_: The slave of labsdb1005.eqiad.wmnet is now clouddb1001.clouddb-services.eqiad.wmflabs
  • 19:16 arturo: T193264 delete VM clouddb-services-01
  • 18:54 arturo: T193264 create VM clouddb-services-01 for PoC of running maintain-dbusers from here

2019-02-16

  • 13:59 arturo: T193264 switched clouddb1001/1004 to the new project local puppetmaster
  • 13:54 arturo: T193264 create 'clouddb10' puppet prefix to store puppet/hiera config for database servers in this project
  • 13:47 arturo: T193264 create 'clouddb-services-puppetmaster' puppet prefix to store puppet/hiera config for this project puppetmaster
  • 13:43 arturo: T193264 create 'clouddb-services-puppetmaster-01' instance
  • 13:33 arturo: add myself as user and projectadmin

2019-02-15

  • 23:42 bd808: Added BryanDavis (self), Arturo Borrero Gonzalez, Marostegui, and Jcrespo as admins in project
  • 22:49 bstorm_: created mariadb security group and lvs for a new database T193264