Nova Resource:Clouddb-services/SAL

From Wikitech

2024-01-09

  • 12:01 taavi: delete old neutron ports used for old wiki replica VIPs T346947

2024-01-08

  • 12:57 taavi: shut off old wiki replica proxies in cloud vps T346947

2023-11-29

  • 13:28 taavi: restarting haproxy on clouddb-wikireplicas-proxy to kill long-lived mariadb connections

2023-06-05

  • 18:36 taavi: add btullis to the project

2023-04-14

  • 09:21 dhinus: shutdown old ToolsDB VMs (clouddb1001, clouddb1002) T301949

2023-04-04

  • 08:35 taavi: revoke puppet certs for now-deleted clouddb1003/4 (osmdb replicas) T323159

2023-03-28

  • 20:19 andrewbogott: deleted old osmdb servers clouddb1003 and clouddb1004. Both have been shut down for some time.

2023-03-06

  • 15:38 dhinus: mariadb is up again on clouddb1001 T329970
  • 15:33 dhinus: stopping mariadb in clouddb1001 for a few minutes T329970

2023-02-21

  • 17:18 andrewbogott: shutting down postgres on clouddb1004/1003, then shutting down the vms

2023-02-20

  • 20:19 taavi: manually start prometheus-mysql-exporter on clouddb1001
  • 17:11 dhinus: mariadb on clouddb1001 is running again
  • 17:07 dhinus: stopping mariadb on clouddb1001 for about 5 minutes (T329970)
  • 11:23 dhinus: mariadb on clouddb1001 is running again
  • 11:16 dhinus: stopping mariadb on clouddb1001 for about 2 minutes (T329970)

2023-02-17

  • 16:13 dhinus: drop unused database s51203__baglama2_p T323502
  • 16:12 dhinus: drop unused database s52467__new_hashtags T329461

2023-01-30

  • 11:02 dhinus: 'SET GLOBAL read_only = 0;' after restarting mariadb in clouddb1001 (T328273)
  • 10:53 dhinus: restarting mariadb in clouddb1001 to apply minor version upgrade (T328273)

2023-01-27

  • 15:52 taavi: repaired wmf-mariadb101 installation on clouddb1001
  • 10:35 dhinus: clouddb1001 unlock tables; (T301949)
  • 10:33 dhinus: clouddb1001 flush tables with read lock; (T301949)

2023-01-25

  • 14:37 dhinus: clouddb1001 unlock tables; (T301949)
  • 14:34 dhinus: clouddb1001 flush tables with read lock; (T301949)

2023-01-24

  • 16:01 dhinus: clouddb1001 unlock tables; (T301949)
  • 15:46 dhinus: clouddb1001 flush tables with read lock; (T301949)

2023-01-05

  • 09:12 dhinus: disabled puppet checks on clouddb100[3-4] (touch /.no-puppet-checks) T323159

2023-01-04

  • 14:12 dhinus: deleted CNAME wikilabels.db.svc.eqiad.wmflabs (T307389)

2022-12-09

  • 17:20 wm-bot2: Increased quotas by 4000 gigabytes (T301949) - cookbook ran by fran@wmf3169
  • 13:50 dhinus: clouddb100 set innodb_file_format=Barracuda and innodb_large_prefix=1 (those values were lost after a reboot)

2022-12-07

  • 22:51 dhinus: remove read_only mode on clouddb1001 (SET GLOBAL read_only = 0;)
  • 18:05 wm-bot2: Increased quotas by 100 gigabytes (T301949) - cookbook ran by fran@wmf3169

2022-11-11

  • 17:37 dhinus: mariadb restarted on clouddb1002 and again in sync with primary (took a bit to catch up) T301949
  • 16:14 dhinus: rsyncing a single database from clouddb1002 to tools-db-2 (as a test for T301949)
  • 16:12 dhinus: stopping mariadb for a few minutes on replica server clouddb1002 (T301949)

2022-11-08

  • 08:38 taavi: delete clouddb-wikilabels-01,02 T307389

2022-11-03

  • 14:39 dhinus: depooling dbproxy1019 'confctl select "service=wikireplicas-b,name=dbproxy1019" set/pooled=no' T313445
  • 13:58 dhinus: shutting down dbproxy1019 T313445

2022-11-02

  • 09:37 taavi: shut down the wikilabels servers T307389

2022-07-02

  • 05:56 taavi: toolsdb: add s54518__mw to list of replication ignored databases, data mismatch between primary and replica

2022-06-28

  • 16:30 taavi: stopped mariadb on clouddb1002, starting rsync clouddb1002->tools-db-1 on a root screen session on tools-db-1 T301949

2022-05-17

  • 14:12 taavi: enable gtid on toolsdb T301993

2022-02-17

  • 12:03 taavi: add myself as a project member so I don't need to ssh in as root@

2021-09-02

  • 18:52 bstorm: removed strange old duplicate cron for osmupdater T285668

2021-08-31

  • 20:19 bstorm: attempting to resync OSMDB back to Feb 21st 2019 T285668

2021-08-30

  • 22:07 bstorm: restarting osmdb on clouddb1003 to try to capture enough connections T285668
  • 21:53 bstorm: disable puppet and osm updater script on clouddb1003 T285668

2021-05-26

  • 16:53 bstorm: restarting postgresql since T220164 was closed. Hoping all connections don't get used up again.

2021-05-25

  • 14:35 dcaro: taking down clouddb1002 replica for reboot of cloudvirt1020 (T275893)

2021-05-04

  • 22:57 bstorm: manually added cnames for toolsdb, osmdb and wikilabelsdb in db.svc.wikimedia.cloud zone T278252

2021-04-05

  • 09:56 arturo: make jhernandez (IRC joakino) projectadmin (T278975)

2021-03-10

  • 12:44 arturo: briefly stopping VM clouddb-wikireplicas-proxy-2 to migrate hypervisor
  • 10:57 arturo: briefly stopped VM clouddb-wikireplicas-proxy-1 to disable VMX cpu flag

2021-03-02

  • 23:29 bstorm: bringing toolsdb back up 😟

2021-02-26

  • 23:20 bstorm: rebooting clouddb-wikilabels-02 for patches
  • 22:55 bstorm: rebooting clouddb-wikireplicas-proxy-1 and clouddb-wikireplicas-proxy-2 before (hopefully) many people are using them

2021-01-29

  • 18:21 bstorm: deleting clouddb-toolsdb-03 as it isn't used

2020-12-23

  • 19:20 bstorm: created clouddb-wikireplicas-proxy-1 and clouddb-wikireplicas-proxy-2 as well as the 16 neutron ports for wikireplicas proxying

2020-12-17

  • 02:14 bstorm: toolsdb is back and so is the replica T266587
  • 01:10 bstorm: the sync is done and we have a good copy of the toolsdb data, proceeding with the upgrades and stuff to that hypervisor while configuring replication to work again T266587

2020-12-16

  • 18:34 bstorm: restarted sync from toolsdb to its replica server after cleanup to prevent disk filling T266587
  • 17:31 bstorm: sync started from toolsdb to its replica server T266587
  • 17:29 bstorm: stopped mariadb on the replica T266587
  • 17:28 bstorm: shutdown toolsdb T266587
  • 17:24 bstorm: settings toolsdb to readonly to prepare for shutdown T266587
  • 17:06 bstorm: switching the secondary config back to clouddb1002 in order to minimize concerns about affecting ceph performance T266587

2020-11-15

  • 19:45 bstorm: restarting the import to clouddb-toolsdb-03 with --max-allowed-packet=1G to rule out that as a problem entirely T266587
  • 19:36 bstorm: set max_allowed_package to 64MB on clouddb-toolsdb-03 T266587

2020-11-11

  • 08:20 bstorm: loading dump on the replica T266587
  • 06:06 bstorm: dump completed, transferring to replica to start things up again T266587

2020-11-10

  • 16:24 bstorm: set toolsdb to read-only T266587

2020-10-29

  • 23:49 bstorm: launching clouddb-toolsdb-03 (trying to make a better naming convention) to replace failed replica T266587

2020-10-27

  • 18:04 bstorm: set clouddb1002 to read-only since it was not supposed to be writeable anyway T266587

2020-10-20

  • 18:36 bstorm: brought up mariadb and replication on clouddb1002 T263677
  • 17:14 bstorm: shutting down clouddb1003 T263677
  • 17:13 bstorm: stopping postgresql on clouddb1003 T263677
  • 17:08 bstorm: poweroff clouddb1002 T263677
  • 17:08 bstorm: stopping mariadb on clouddb1002 T263677
  • 17:07 bstorm: shut down replication on clouddb1002 (now with task) T263677
  • 17:05 bstorm: shut down replication on clouddb1002

2020-09-29

  • 23:48 bstorm: restarting postgresql on clouddb1003 to clear the number of connections

2020-09-08

  • 18:21 bstorm: copied the profile::mariadb::section_ports key into prefix puppet to fix puppet after the refactor for wmfmariadbpy

2020-07-16

  • 17:56 bstorm: Significantly lifted traffic shaping limits on clouddb1001/toolsdb to improve network performance T257884

2020-06-09

  • 22:55 bstorm_: Reset the passwords for T254931

2020-06-04

  • 17:41 bd808: Restarting mariadb after hand applying {{gerrit|602433}} (T253738)

2020-06-01

  • 18:51 bstorm_: restarting mariadb on clouddb1002 T253738

2020-05-29

  • 23:42 bstorm_: stopped puppet and restarting mariadb on clouddb1002 after filtering out a table T253738

2020-03-27

  • 12:26 arturo: add myself as project admin

2020-03-20

  • 13:43 jeh: upgrade puppetmaster to v5 T241719

2020-03-18

  • 17:49 bstorm_: updated the tools-prometheus security group with the current IP addresses for the prometheus servers

2019-10-25

2019-10-24

  • 18:34 bstorm_: downgraded clouddb1001 to 10.1.39 T236420 T236384
  • 18:19 bstorm_: stopped puppet on clouddb1001 and removed unattended-upgrades for now T236384
  • 16:03 bstorm_: stopped puppet on clouddb1002 and removed unattended-upgrades for now T236384
  • 13:27 phamhi: add phamhi as user and projectadmin
  • 12:34 arturo: start both clouddb1001 and clouddb1004 (T227540)
  • 11:15 arturo: poweroff VM clouddb1001, hypervisor will be powered off (T227540)
  • 11:13 arturo: poweroff VM clouddb1004 (T227540)
  • 11:09 phamhi: stopped postgresl in clouddb1004 (T227540)
  • 11:04 arturo: stopped mariadb in clouddb1001 (T227540)

2019-09-30

  • 17:15 bstorm_: restart wikilabels postgresql
  • 17:15 bstorm_: restart osmdb on clouddb1003

2019-09-16

  • 19:41 jeh: add openstack security group rules for Tools prometheus scrapes to mariadb exporter
  • 19:28 jeh: add openstack security group rules for labmon prometheus scraps to mariadb exporter

2019-05-24

  • 23:38 bstorm_: T224062 clouddb1002 is now free of postgresql services and the volume is reclaimed for toolsdb use
  • 23:06 bstorm_: T224318 cloudb-wikilabels-02 is now a replica of clouddb-wikilabels-01!
  • 21:00 bstorm_: T224062 Moved wikilabels postgres db to clouddb-wikilabels-01

2019-05-22

  • 00:14 bstorm_: T224062 wikilabels postgres is now replicating to clouddb-wikilabels-01

2019-04-05

  • 00:35 bstorm_: added the rsyncd ports to security groups for the osmdb cluster

2019-04-04

  • 23:58 bstorm_: pg_basebackup finished on clouddb1004 and streaming replication is flowing
  • 18:46 bstorm_: T219652 pg_basebackup started on clouddb1004
  • 18:35 bstorm_: T219652 postgresql on clouddb1003 had to be restarted to get it to accept WAL connections
  • 18:00 bstorm_: T219652 clouddb1003 is now the OSMdb primary

2019-03-28

  • 23:17 bstorm_: clouddb1003 is now a full osmdb replica T193264
  • 19:32 bstorm_: pg_basebackup started on clouddb1003 for osmdb T193264

2019-02-28

  • 02:09 bstorm_: clouddb1002 is now the replica for clouddb1001/toolsdb and is catching up slowly

2019-02-18

  • 19:01 bstorm_: labsdb1004 is now a replica of clouddb1001, which is toolsdb now.
  • 18:26 arturo: (jaime T193264) setting clouddb1001 in read_write mode
  • 18:12 arturo: T193264 pointing tools.db.svc.eqiad.wmflabs to clouddb1001
  • 17:55 arturo: (jaime T193264) set clouddb1001 in read_only=1
  • 16:11 andrewbogott: deleting clouddb-utils-01 VM and associated puppet prefix; we aren't going to run maintain_dbusers here after all
  • 15:51 andrewbogott: removing dns and public IP from clouddb1001

2019-02-17

  • 21:21 bstorm_: The slave of labsdb1005.eqiad.wmnet is now clouddb1001.clouddb-services.eqiad.wmflabs
  • 19:16 arturo: T193264 delete VM clouddb-services-01
  • 18:54 arturo: T193264 create VM clouddb-services-01 for PoC of running maintain-dbusers from here

2019-02-16

  • 13:59 arturo: T193264 switched clouddb1001/1004 to the new project local puppetmaster
  • 13:54 arturo: T193264 create 'clouddb10' puppet prefix to store puppet/hiera config for database servers in this project
  • 13:47 arturo: T193264 create 'clouddb-services-puppetmaster' puppet prefix to store puppet/hiera config for this project puppetmaster
  • 13:43 arturo: T193264 create 'clouddb-services-puppetmaster-01' instance
  • 13:33 arturo: add myself as user and projectadmin

2019-02-15

  • 23:42 bd808: Added BryanDavis (self), Arturo Borrero Gonzalez, Marostegui, and Jcrespo as admins in project
  • 22:49 bstorm_: created mariadb security group and lvs for a new database T193264