21:50 framawiki: shutdown potentially unused servers: quarry-nfs-1, quarry-puppet-master-02, quarry-dev-03 (this last one is started when there is need to test patchs)
2023-05-19
09:59 wm-bot2: added user isaacj to the project as reader (T337019) - cookbook ran by arturo@endurance
2023-04-18
19:24 Rook: remove db entries ending in semicolon 91d66e5
2023-02-28
12:17 Rook: Don't crash if multiple columns share a name a2c6d3fT170464
11:48 Rook: More dropdown options for page length 09fb982T126540
12:04 dcaro: rebooting quarry-worker-04.quarry.eqiad1.wikimedia.cloud due to stuck nfs (T304681)
2022-03-21
17:30 Rook: updating home page link to profile T85175
2022-02-20
19:49 andrewbogott: moving nfs service from quarry-nfs-1 (bullseye) to quarry-nfs-2 (buster), testing to see if T302154 is a kernal or nfs-version issue
19:23 taavi: hard rebooted quarry-nfs-1 again T302154
20:45 zhuyifei1999_: set read-only again on new database due to a bug in worker code
20:42 zhuyifei1999_: unset read-only on new database T202588
20:41 framawiki: switched quarry.wmflabs.org proxy to new quarry-web-01.quarry.eqiad.wmflabs T202588
20:27 zhuyifei1999_: backed up old db to /data/project/dump-2018-09-12.sql and restoring to new server T202588
20:03 zhuyifei1999_: set quarry-main-01 mariadb read-only T202588
20:02 zhuyifei1999_: stopped celery-quarry-worker on quarry-runner-0{1,2} T202588
19:45 zhuyifei1999_: created new quarry database and user in quarry-db-01.quarry.eqiad.wmflabs T202588
2018-09-11
17:22 zhuyifei1999_: doing another backup of main db: `sudo mysqldump quarry | sudo tee /data/project/dump-$(date '+%Y-%m-%d').sql > /dev/null` T202588
17:14 zhuyifei1999_: disabling puppet on quarry-main-01, quarry-runner-0{1,2} T202588
2018-09-07
21:06 zhuyifei1999_: reverted hotpatch, deployed till 3375dc3
20:47 zhuyifei1999_: hotpatch /etc/uwsgi/apps-enabled/quarry-web.ini processes 8 -> 1 for some gdb-ing
19:56 framawiki: deployed 501695f to quarry-main-01 (T202588)
18:11 framawiki: deployed 769cace to quarry-main-01 (T202588)
2018-08-24
21:21 framawiki: deployed 4814d58 (T124625) to quarry-main-01
2018-05-31
15:52 zhuyifei1999_: live-patch `/srv/quarry/quarry/web/connections.py` on `quarry-main-01` and restart uwsgi
2018-05-07
22:40 framawiki: deployed 24038e3 to quarry-main-01
18:23 framawiki: deployed af2f7e6 to quarry-main-01
18:04 zhuyifei1999_: Deployed f4e86f1 and restarted everything
2018-04-24
21:36 framawiki: removing old /srv/venv on quarry-main-01 T192731
2018-04-23
22:50 zhuyifei1999_: Does quarry only have an effective concurrency limit of 3, despite having a few dozen celery worker processes?!
22:46 zhuyifei1999_: behaving abnormally. https://quarry.wmflabs.org/query/26629 has been queued for 16 mins... (hopefully) have some time to investigate
2018-04-18
23:21 framawiki: deployed 02049d9 to quarry-main-01
19:35 zhuyifei1999_: deployed c6cd55e to quarry-main-01
17:36 zhuyifei1999_: deployed 8eeeff8 to quarry-main-01
2018-04-17
23:00 zhuyifei1999_: forgot to restart uwsgi on last deployment. restarted it now
00:34 zhuyifei1999_: Deploy b5fd6b0 on quarry-main-01
2018-03-24
03:21 zhuyifei1999_: revert back to d9cc1c8 again on quarry-runner0{1,2} T188564T190608
2018-03-16
00:29 zhuyifei1999_: deploying fc109c2 to both runners T188564
2018-03-15
19:27 zhuyifei1999_: switch back to d9cc1c8 on both hosts
16:29 zhuyifei1999_: quarry-runner-02 is on d9cc1c8
16:21 zhuyifei1999_: installed python-dbg on quarry-runner-02 because it's so good
16:18 zhuyifei1999_: depool quarry-runner-01
15:56 zhuyifei1999_: deploying d653400 to quarry-runner-0{1,2} T188564
2018-03-01
18:41 zhuyifei1999_: deploying d5e2845 to quarry-runner-01 & 02
00:37 zhuyifei1999_: `UPDATE query join query_revision on query.latest_rev_id = query_revision.id join query_run on latest_run_id = query_run.id SET status=1 where (status = 2 or status = 1) and query_run.timestamp <= DATE_ADD(NOW(), INTERVAL -1 HOUR);` 251 rows affected (1.81 sec) T139162T172086T188564
2018-02-28
22:57 zhuyifei1999_: killed two IO-intensive query saves
2018-02-09
01:06 bd808: Removed TestingAccount2 at user request (T186289)
01:06 bd808: Removed Yuvipanda at user request (T186289)
2018-01-02
10:52 zhuyifei1999_: deploying d9cc1c8 to quarry-runner-01 & 02 T172143
2017-12-13
19:19 zhuyifei1999_: Deployed 62676f2 to quarry-main-01 and restarted uwsgi
2017-12-10
05:52 zhuyifei1999_: deployed e835a46 to quarry-main-01 and restarted uwsgi T165169
05:49 zhuyifei1999_: quarry-main-01: `ALTER IGNORE TABLE star ADD UNIQUE INDEX star_user_query_index (user_id, query_id);` Records: 728 Duplicates: 17 Warnings: 0 T165169
2017-12-05
18:56 zhuyifei1999_: quarry-main-01: `MariaDB [quarry]> UPDATE user SET username = '-revi' WHERE username = 'Hym411';` T182064
2017-10-02
05:15 zhuyifei1999_: Deployed 644b293 to quarry-main-01 and restarted uwsgi
2017-09-26
04:08 zhuyifei1999_: Restarting service 'uwsgi-quarry-web' on quarry-main-01, 'celery-quarry-worker' on quarry-runner-01 & quarry-runner-02 T176694
03:59 zhuyifei1999_: Switching REPLICA_HOST from 'enwiki.labsdb' to 'enwiki.analytics.db.svc.eqiad.wmflabs' T176694 (Executing `sudo -- sudo -u quarry sed -i 's/enwiki.labsdb/enwiki.analytics.db.svc.eqiad.wmflabs/' /srv/quarry/quarry/config.yaml` on all hosts)
2017-09-10
16:22 zhuyifei1999_: Deployed a6173a2 on quarry-main-01 T175466
2017-09-08
18:06 zhuyifei1999_: Deployed d9e8a4a to quarry-main-01 T175285
2017-09-06
00:05 zhuyifei1999_: backup quarry main database to /data/project/dump-2017-09-05.sql because I fear it die :(. Executing `sudo mysqldump quarry | sudo tee /data/project/dump-2017-09-05.sql > /dev/null`
2017-08-11
11:12 zhuyifei1999_: deployed 2834160 on quarry-main-01
2017-08-01
12:51 zhuyifei1999_: Deployed ba54a61 on quarry-main-01 T164390
2017-07-31
16:31 zhuyifei1999_: Repeated for quarry-main-01, but restarted uwsgi T146483
16:30 zhuyifei1999_: Repeated for quarry-runner-02
16:28 zhuyifei1999_: Restarted celery-quarry-worker on quarry-runner-01 T146483
20:10 zhuyifei1999_: Yuvi gave me access after I asked about T76126
2017-06-30
22:34 bd808: Added BryanDavis (self) as project admin
18:09 bd808: Ran service uwsgi-quarry-web restart on quarry-main-01. People seeing intermittent 502s
2017-06-26
18:33 milimetric: Restarted celery workers on quarry-runner-01 and quarry-runner-02 (systemctl restart celery-quarry-worker.service)
17:20 madhuvishy: Add milimetric as project admin
2017-01-19
12:07 yuvipanda: run chown -R 998:998 quarry/ on labstore1004
2016-10-05
19:48 mutante: quarry-runner-01 has a problem starting exim4
19:47 mutante: merged gerrit 308313 - should definitely be no-op, but noticed that puppet is disabled on quarry-main-01
2016-10-04
19:33 valhallasw`cloud: removed myself as admin
2016-09-23
20:41 yuvipanda: add halfak as projectadmin
2016-05-08
21:35 Krenair: restarted quarry-runner-01 to attempt to get the queue working again - new queries are going through but some old ones (from the last few hours) seem stuck
2016-04-02
14:06 valhallasw`cloud: systemctl restart celery-quarry-worker.service hangs. Will now reboot quarry-runner-02.
14:02 valhallasw`cloud: killed 100% CPU using process on quarry-runner-02 (ptrace suggested some sort of idle loop). Let's see if that has any effect.
July 4
15:45 YuviPanda: deploying to latest master and hoping!
April 30
21:57 andrewbogott: moving quarry-main-01 to labvirt1003
20:53 andrewbogott: moving quarry-runner-01 to labvirt1004
19:41 andrewbogott: cold-migrating quarry-runner-test to labvirt1003
December 8
19:36 YuviPanda: deploying to get in valhallasw`cloud’s patches
September 30
15:44 andrewbogott: enabled puppet on quarry-runner-test, updated, installed a bunch of maria stuff, rebooted
September 16
21:53 YuviPanda: appllying db, web and redis roles to quarry-runner-test, will act as db and web host until labs issues clear up
August 21
19:47 YuviPanda: upgraded all text and varchar columns to utf8
August 17
02:03 YuviPanda: increased mysql connection limit manually to 1024, re-running all old query-runs to produce output in new sqlite format
August 14
20:21 YuviPanda: upgrade to MariaDB 10.1, because fuck offline ALTER TABLEs
