Nova Resource:Library-upgrader/SAL

From Wikitech

2024-04-01

  • 16:57 taavi: stop upgrader cron on upgrader-06 to prepare for move to libup-runner08 in T361488

2024-02-05

  • 20:02 taavi: upgrade docker on upgrader-06 to fix building the worker image
  • 17:21 taavi: add James_F, Amir1, Reedy and myself to labs-libraryupgrader Gerrit group T345930

2024-02-03

  • 11:26 taavi: cleanup /srv/data/cache on upgrader-06.library-upgrader, freed up 75% of that 60G disk
  • 11:25 taavi: raise trove quota to 30G T356560

2024-01-30

  • 22:21 bd808: Restarted libup-db02 instance via Horizon to try to fix "Instance 7d785002-371c-4a72-973f-629a6a4f3084 is not currently available for an action to be performed (instance status was ACTIVE)."

2023-01-26

  • 06:35 legoktm: libup@upgrader-06:/srv/libraryupgrader$ pip install -U wikimediaci-utils hotfix

2022-10-06

  • 01:44 legoktm: deleted stuff from upgrader-06 because it ran out of disk space oops

2022-10-05

  • 04:51 legoktm: rebooting libup-db02, unsure what's wrong
  • 04:47 legoktm: rebooting libup-web02 after I fixed the full root partition :/

2021-11-08

  • 05:21 legoktm: deleting libup-db01, replaced by trove

2021-11-06

  • 22:03 legoktm: deleting libup-web01 instance
  • 22:03 legoktm: disabled libup-web unit on `upgrader-06` instance
  • 22:00 legoktm: https://libraryupgrader2.wmcloud.org/ now points to libup-web02, which is fully containerized and auto-deployed via the deployment pipeline
  • 21:35 legoktm: launching libup-web02 (bullseye) to replace libup-web01
  • 21:26 legoktm: shutting down libup-db01, will delete in a few days
  • 20:58 legoktm: launching libup-db02 via trove to replace libup-db01

2021-10-25

  • 17:00 legoktm: temporarily stopped libup-celery service to test new monitoring/alerting

2021-08-25

  • 08:26 legoktm: restarted failed libup-push systemd unit and cleared queue to avoid pushing really old patches

2021-04-15

  • 09:44 arturo: bumping quota: 16->20GB RAM, 8->10 CPUs (T280103)

2021-04-09

  • 08:12 legoktm: importing mysql database on libup-db01
  • 07:34 legoktm: creating libup-db01 instance
  • 07:32 legoktm: deleting libup-diff01 experiment and upgrader-07 (puppet tests) instances

2021-03-05

  • 05:01 legoktm: sudo systemctl enable libup-push
  • 04:11 legoktm: added SSH key to ssh-agent

2020-10-26

  • 23:51 legoktm: added key to ssh-agent and started libup-celery

2020-09-03

  • 21:50 legoktm: added libraryupgrader2.wmcloud.org DNS proxy and remove wmflabs.org one for automatic redirect (T261995)

2020-06-23

  • 02:52 legoktm: created libup-diff01 instance
  • 02:41 legoktm: deleted upgrader-05 instace, which has been shutoff for a while now

2020-03-24

  • 06:33 legoktm: building fresh docker image

2020-01-10

  • 09:34 legoktm: shutdown upgrader-05 instance
  • 07:52 legoktm: switched over libraryupgrader2.wmflabs.org to point to upgrader06
  • 06:59 legoktm: upgrader06: adduser libup && adduser libup docker
  • 06:59 legoktm: upgrader06: enabling buster-wikimedia/thirdparty/kubeadm-k8s apt repo and installing docker-ce
  • 06:53 legoktm: disabling all services on upgrader-05
  • 06:33 legoktm: creating upgrader-06 instance, large flavor this time

2019-12-28

  • 23:49 legoktm: deleting upgrader-04 instance (T236559)

2019-12-20

  • 01:14 legoktm: moved a lot of old logs into /srv/data/archive/

2019-12-14

  • 22:56 legoktm: dropped concurrency down to 1
  • 22:52 legoktm: restarted libup-celery now that gerrit-replica has been restarted
  • 20:06 legoktm: systemctl stop libup-celery # because of T240763
  • 11:47 legoktm: temporarily bump concurrency to 2 to clear backlog

2019-10-13

  • 22:45 legoktm: rebooting upgrader-05 because of DNS failures when run in docker

2019-07-31

  • 04:16 legoktm: added ssh key to libraryupgrader Gerrit account that lives on upgrader-05

2019-06-21

  • 04:49 legoktm: set up new upgrader-05 stretch instance (using docker from upstream docker's repo :((), using run.py as a stress test

2019-06-11

  • 15:03 legoktm: -04 has a full disk, and is broked. oops.

2018-11-14

  • 15:09 andrewbogott: migrating project to eqiad1-r