Nova Resource:Library-upgrader/SAL

2024-04-15

• 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance upgrader-06
• 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance upgrader-06

2024-04-01

• 16:57 taavi: stop upgrader cron on upgrader-06 to prepare for move to libup-runner08 in T361488

2024-02-05

• 20:02 taavi: upgrade docker on upgrader-06 to fix building the worker image
• 17:21 taavi: add James_F, Amir1, Reedy and myself to labs-libraryupgrader Gerrit group T345930

2024-02-03

• 11:26 taavi: cleanup /srv/data/cache on upgrader-06.library-upgrader, freed up 75% of that 60G disk
• 11:25 taavi: raise trove quota to 30G T356560

2024-01-30

• 22:21 bd808: Restarted libup-db02 instance via Horizon to try to fix "Instance 7d785002-371c-4a72-973f-629a6a4f3084 is not currently available for an action to be performed (instance status was ACTIVE)."

2023-01-26

• 06:35 legoktm: libup@upgrader-06:/srv/libraryupgrader$ pip install -U wikimediaci-utils hotfix

2022-10-06

• 01:44 legoktm: deleted stuff from upgrader-06 because it ran out of disk space oops

2022-10-05

• 04:51 legoktm: rebooting libup-db02, unsure what's wrong
• 04:47 legoktm: rebooting libup-web02 after I fixed the full root partition :/

2021-11-08

• 05:21 legoktm: deleting libup-db01, replaced by trove

2021-11-06

• 22:03 legoktm: deleting libup-web01 instance
• 22:03 legoktm: disabled libup-web unit on `upgrader-06` instance
• 22:00 legoktm: https://libraryupgrader2.wmcloud.org/ now points to libup-web02, which is fully containerized and auto-deployed via the deployment pipeline
• 21:35 legoktm: launching libup-web02 (bullseye) to replace libup-web01
• 21:26 legoktm: shutting down libup-db01, will delete in a few days
• 20:58 legoktm: launching libup-db02 via trove to replace libup-db01

2021-10-25

• 17:00 legoktm: temporarily stopped libup-celery service to test new monitoring/alerting

2021-08-25

• 08:26 legoktm: restarted failed libup-push systemd unit and cleared queue to avoid pushing really old patches

2021-04-15

• 09:44 arturo: bumping quota: 16->20GB RAM, 8->10 CPUs (T280103)

2021-04-09

• 08:12 legoktm: importing mysql database on libup-db01
• 07:34 legoktm: creating libup-db01 instance
• 07:32 legoktm: deleting libup-diff01 experiment and upgrader-07 (puppet tests) instances

2021-03-05

• 05:01 legoktm: sudo systemctl enable libup-push
• 04:11 legoktm: added SSH key to ssh-agent

2020-10-26

• 23:51 legoktm: added key to ssh-agent and started libup-celery

2020-09-03

• 21:50 legoktm: added libraryupgrader2.wmcloud.org DNS proxy and remove wmflabs.org one for automatic redirect (T261995)

2020-06-23

• 02:52 legoktm: created libup-diff01 instance
• 02:41 legoktm: deleted upgrader-05 instace, which has been shutoff for a while now

2020-03-24

• 06:33 legoktm: building fresh docker image

2020-01-10

• 09:34 legoktm: shutdown upgrader-05 instance
• 07:52 legoktm: switched over libraryupgrader2.wmflabs.org to point to upgrader06
• 06:59 legoktm: upgrader06: adduser libup && adduser libup docker
• 06:59 legoktm: upgrader06: enabling buster-wikimedia/thirdparty/kubeadm-k8s apt repo and installing docker-ce
• 06:53 legoktm: disabling all services on upgrader-05
• 06:33 legoktm: creating upgrader-06 instance, large flavor this time

2019-12-28

• 23:49 legoktm: deleting upgrader-04 instance (T236559)

2019-12-20

• 01:14 legoktm: moved a lot of old logs into /srv/data/archive/

2019-12-14

• 22:56 legoktm: dropped concurrency down to 1
• 22:52 legoktm: restarted libup-celery now that gerrit-replica has been restarted
• 20:06 legoktm: systemctl stop libup-celery # because of T240763
• 11:47 legoktm: temporarily bump concurrency to 2 to clear backlog

2019-10-13

• 22:45 legoktm: rebooting upgrader-05 because of DNS failures when run in docker

2019-07-31

• 04:16 legoktm: added ssh key to libraryupgrader Gerrit account that lives on upgrader-05

2019-06-21

• 04:49 legoktm: set up new upgrader-05 stretch instance (using docker from upstream docker's repo :((), using run.py as a stress test

2019-06-11

• 15:03 legoktm: -04 has a full disk, and is broked. oops.

2018-11-14

• 15:09 andrewbogott: migrating project to eqiad1-r