Nova Resource:Ores/SAL
2018-01-17
- 21:44 halfak: created ores-misc-01 as Debian Stretch instance
- 21:42 halfak: deleted ores-misc-01
- 21:30 halfak: deploying ores-wmflabs-deploy:96d7f12
- 21:27 halfak: staging ores-wmflabs-deploy:96d7f12
2018-01-16
- 23:36 halfak: created ores-worker-05/06 as Debian Stretch
- 23:18 halfak: deleting ores-worker-08/09/10
- 23:14 halfak: shutting down ores-worker-08/09/10
- 22:33 halfak: creating ores-worker-01/02/03/04 as stretch instances.
- 22:27 halfak: deleting ores-worker-05/06/07
- 22:25 halfak: shutting down ores-worker-05/06/07
- 22:23 halfak: deleting ores-web-03/05
- 22:15 halfak: pooling ores-web-01/02
- 22:06 halfak: created ores-web-02 as a debian stretch instance.
- 22:01 halfak: deleted ores-redis-01 (seemingly unused)
2018-01-11
- 20:06 halfak: created ores-web-01 as Stretch instance
2018-01-05
- 18:26 halfak: deleting ores-staging-02 to recreate as a stretch instance.
2017-12-02
- 16:54 halfak: deploying ores-wmflabs-deploy:8cebc43
- 16:50 halfak: staging ores-wmflabs-deploy:8cebc43
2017-11-27
- 21:03 halfak: deployed ores-wmflabs-deploy:02aebc5
- 20:59 halfak: staged ores-wmflabs-deploy:02aebc5
2017-10-07
- 17:16 halfak: deploying ores-wmflabs-deploy:3e854cf
- 17:10 halfak: staging ores-wmflabs-deploy:3e854cf
2017-10-05
- 19:13 halfak: deployed ores-wmflabs-deploy:5b61eec (rollback)
- 18:56 halfak: deployed ores-wmflabs-deploy:b7c5337
- 18:55 halfak: staged ores-wmflabs-deploy:b7c5337
2017-07-25
- 16:48 halfak: deploying ores-wmflabs-deploy:68326ba
- 16:12 halfak: staging ores-wmflabs-deploy:68326ba
2017-07-17
- 15:42 halfak: deleting ores-web-04 (intermittent puppet issues and wasn't pooled anyway)
2017-07-06
- 00:07 bd808: bd808's root key is rejected by ores-compute-01.ores.eqiad.wmflabs. Puppet busted there?
2017-06-28
- 17:55 halfak: deleting mediawiki-ores and hashing-vector instances from ores-staging (unused)
2017-06-22
- 00:03 awight: labs ores was down 17:13-17:22 due to maintenance.
2017-06-16
- 21:37 halfak: deploying ores-wmflabs-deploy:e8f6932
- 21:34 halfak: staging ores-wmflabs-deploy:e8f6932
- 21:06 halfak: deploying ores-wmflabs-deploy:c952f57 (reverting due to issues)
- 20:52 halfak: deploying ores-wmflabs-deploy:7529113
- 20:26 halfak: staging ores-wmflabs-deploy:7529113
2017-05-20
- 14:47 halfak: staging ores-wmflabs-deploy:9df7b5a
- 14:44 halfak: staging ores-wmflabs-deploy:ed1991c
2017-04-25
- 21:33 halfak: deployed ores-wmflabs-deploy:dc934e8
- 21:30 halfak: staged ores-wmflabs-deploy:dc934e8
2017-04-06
- 16:00 halfak: deploying ores-wmflabs-deploy:7061f0a
- 15:56 halfak: staging ores-wmflabs-deploy:7061f0a
2017-03-28
- 15:58 halfak: deployed ores-wmflabs-deploy:18beebf (T160638)
- 15:57 halfak: deployed ores-wmflabs-deploy:18beebf
- 15:48 halfak: staged ores-wmflabs-deploy:18beebf
2017-03-26
- 07:39 Amir1: restart ores-web-03
2017-03-15
- 19:26 halfak: deployed ores-wmflabs-deploy:0990b1d
- 19:24 halfak: staged ores-wmflabs-deploy:0990b1d
- 19:03 halfak: deployed ores-wmflabs-deploy:080418d
- 17:36 halfak: deployed ores-wmflabs-deploy:080418d to staging
2017-03-06
- 20:26 halfak: restarted precached and uwsgi-ores on ores-web-03
2017-02-23
- 18:55 halfak: restarted uwsgi-ores and precached on ores-web-03 (memory issues)
2017-02-21
- 03:40 halfak: restarted uwsgi and precached on ores-web-03
2017-02-19
- 15:54 Amir1: (except redis ones)
- 15:54 Amir1: rebooting all instances
2017-02-11
- 18:12 halfak: deployed ores-wmflabs-deploy:f58d3fd
- 18:09 halfak: staged ores-wmflabs-deploy:f58d3fd
2017-02-01
- 23:39 halfak: deployed ores-wmflabs-deploy:ce8861d
2017-01-25
- 21:20 halfak: deployed ores-wmflabs-deploy:0f90516
2017-01-18
- 19:47 halfak: restarted precached and uwsgi-ores on ores-web-03. Memory usage issues.
2017-01-15
- 03:03 Amir1: restarting ores-web-03
2017-01-04
- 12:51 halfak: ran 'sudo service uwsgi-ores restart' on ores-web-03
- 11:41 Amir1: stopping precaching in ores in labs to reduce load
- 11:40 Amir1: reboot ores-lb-02
2016-11-22
- 16:20 mutante: ores-web-03 has an issue (Socket timeout, CRIT)
2016-11-07
- 20:45 halfak: restarted uwsgi-ores and precached on ores-web-03
2016-11-03
- 17:36 halfak: manually ran "sudo mkdir /srv/log; sudo mkdir /srv/log/ores; sudo service celery-ores-worker restart" on ores-worker-(05|06|07|08|09|10)
- 17:23 halfak: deployed ores-wmflabs-deploy:ffb5dbb
- 17:16 halfak: deployed ores-wmflabs-deploy:ffb5dbb to staging (fixes logging bug)
- 17:04 halfak: deployed ores-wmflabs-deploy:74699db to staging
2016-09-25
- 16:03 Amir1: deploying c2ba205 (T146581)
- 15:59 Amir1: c2ba205 going to staging
- 15:26 halfak: restarted precached and uwsgi-ores on ores-web-03
2016-09-23
- 20:49 halfak: restarted ores-uwsgi and precached services on ores-web-03
2016-09-13
- 03:03 halfak: deployed ores-wmflabs-deploy:022fb7d
2016-09-12
- 13:47 halfak: restarted precached service on ores-web-03
- 04:31 halfak: manually killed since "python" process consuming 80% of memory and restarted uwsgi-ores service on ores-web-03
- 04:29 halfak: restarted uwsgi-ores service on ores-web-03
2016-09-02
- 20:25 halfak: deployed ores-wmflabs-deploy:751860f
2016-08-29
- 14:03 chasemp: reboot 94886e74-5be4-4669-a1c1-840ce7c65de9 ores-compute-01
2016-08-24
- 22:00 halfak: restarted uwsgi-ores on ores-web-05
2016-08-19
- 21:10 halfak: deployed ores-wmflabs-deploy:f0fc59b
2016-08-12
- 18:25 halfak: ran FLUSHALL on ores-redis-02:6380
- 18:24 halfak: deployed ores-wmflabs-deploy:b015348
2016-08-09
- 22:08 halfak: restarted precached service on ores-web-03
- 22:08 halfak: deployed ores-wmflabs-config:1b1c56d
2016-08-08
- 20:44 Amir1: depooling ores-web-04
2016-08-05
- 17:44 halfak: repooled ores-web-04 in hiera and ran puppet on ores-lb-02
- 16:15 halfak: creates ores-web-04 debian-8.5-jessie
- 16:11 halfak: terminated ores-web-04 (re-creating instance with medium image size)
2016-08-04
- 18:02 halfak: rebooting ores-web-05 to address OOM
- 17:56 halfak: reduced web workers per core from 28 to 16
2016-07-29
- 00:20 halfak: deployed ores-wmflabs-deploy:770d131
2016-07-13
- 18:20 Amir1: ladsgroup@ores-lb-02:~$ sudo service nginx restart
- 17:21 Amir1: deploying to web again
- 17:14 Amir1: ladsgroup@ores-web-05:~$ sudo puppet agent -tv
- 16:32 halfak: terminated ores-web-05
- 15:58 halfak: repooled ores-web-04
- 15:20 halfak: deployed ores-wmflabs-deploy:e638f1b
2016-07-12
- 11:25 Amir1: deploying aad92ac to web and worker nodes
- 11:18 Amir1: aad92ac goes to staging
2016-07-08
- 20:20 halfak: deployed ores-wmflabs-deploy:3f116cd
2016-06-10
- 20:25 halfak: deleted 99-redis.yaml and restarted celery-ores-worker on all worker nodes
- 07:36 Amir1: deploying 4efc5b7 into prod
2016-06-03
- 23:57 Amir1: removing role role::labs::ores::flower from ores-web-03
2016-05-27
- 13:31 Amir1: adding role::deployment::server to ores project
2016-05-26
- 10:05 Amir1: rebooting ores-web-04
- 10:02 Amir1: deploying 4dc6050 into web and worker nodes, updating to revscoring 1.2.6
2016-05-25
- 19:33 Amir1: running puppet agent manually in ores-web-03
- 19:29 Amir1: deploying 7992fd1 into web and worker nodes
2016-05-23
- 22:38 Amir1: precaching brought back online
- 22:33 Amir1: stopping precaching in ores-web-03 manually. Testing something
2016-05-18
- 16:48 Amir1: running puppet agent on ores-lb-02 manually
2016-05-16
- 13:00 Amir1: deploying e51c4f3 to celery and web nodes
2016-05-09
- 07:52 Amir1: manually restarted nginx load balancer (ores-lb-02) It wasn't needed since puppet did it already.
2016-04-30
- 18:05 Amir1: deployed d4f63a3 to web and worker nodes
- 17:49 Amir1: deploy d4f63a3 to the staging
- 17:33 Amir1: deploying 30ba552 to the staging
- 14:54 Amir1: running puppet agent manually in ores-web-03
- 14:52 Amir1: added precaching role to ores-web-03
2016-04-27
- 20:48 halfak: deployed ores-wikimedia-config:6453fe5
2016-04-23
- 15:25 Amir1: sudden disruption due to an unexpected deploy
2016-04-07
- 14:32 halfak: deployed ores-wikimedia-config:34ba0fa
2016-03-30
- 16:13 halfak: deployed ores-wikimedia-config:58905c5
2016-03-23
- 17:39 halfak: deployed ores-wikimedia-config:39b622e
- 01:24 halfak: deployed ores-wikimedia-config:8af4377
2016-03-19
- 20:04 halfak: ores-web-01/02 and ores-worker-01/02/03/04 deleted. ores-web-03/04/05 and ores-worker-05/06/07/08/09/10 started and configured as replacements.
- 16:43 halfak: Manually ran `sudo apt-get install aspell-ar aspell-pl` across web and worker nodes
- 15:48 halfak: Ran puppet and restarted uwsgi on web-01 with 28 forks rather than 32
- 15:32 halfak: workers_per_core 32 --> 28
2016-03-18
- 01:17 mutante: while doing that i noticed that ores-web-02 has an "Cannot allocate memory" problem
- 01:17 mutante: but no class names have changed and confirmed no-op on every single instance
- 01:17 mutante: merged https://gerrit.wikimedia.org/r/#/c/270102/ - role classes have been moved to modules/role and split into one file per class
- 01:07 mutante: testing puppet on all instances before merging change that moves role classes to module
2016-03-05
- 17:58 halfak: Shut down the uwsgi service and started up the uwsgi-ores-web service on web-01 and -02.
2016-03-04
- 22:54 andrewbogott: migrating ores-worker-02 to labvirt1010
2016-02-15
- 16:26 halfak: restarted workers on ores-worker -02 and -03
- 04:01 halfak: Ammending last message -- Restarted the uwsgi-ores-web service on *ores-web-01*. The service on ores-web-02 was left alone.
- 03:59 halfak: Restarted the uwsgi-ores-web service on ores-web-02
2016-02-07
- 19:21 halfak: Deployed ores-wikimedia-config:0be5afc
2015-12-30
- 20:23 halfak: Deployed with wb-vandalism:d940cea
- 20:22 YuviPanda: create ores-cache-01
2015-12-21
- 20:28 YuviPanda: depool ores-web-01 from lb
2015-12-16
- 14:24 YuviPanda: updated quota for instances to arbitrary number (40)
2015-12-08
- 17:34 halfak: Deployed 1ad37c5 with ores:b745570, revscoring:, editquality:b41b7c1, wikiclass:bbfa9ce, and wb-vandalism:1075596
2015-11-12
- 20:50 halfak: Deployed 334b4b2 with ores:b745570, revscoring:, editquality:b41b7c1, wikiclass:bbfa9ce, and wb-vandalism:a09d026
2015-10-28
- 21:05 halfak: Deployed ores-wikimedia-config:c27c8f7, ores==0.5.4, wb-vandalism==0.1.5, revscoring==0.6.7
2015-09-12
- 02:30 halfak: Deployed ores-wikimedia-config:c852404, ores==0.5.0 and revscoring==0.6.1
2015-09-11
- 14:51 halfak: removed ores-web-02 from ores-lb-02 pool
- 03:31 YuviPanda: restart redis server on ores-redis-02 to apply tcp-keepalive
2015-09-10
- 18:28 YuviPanda: disabled restart crontab on ores-worker-01 under halfak's user, have set it to keep insane levels of logs to debug
2015-09-08
- 16:35 halfak: deployed ores-wikimedia-config:ca10888, ores==0.4.0 and revscoring==0.5.0
- 16:34 valhallasw`cloud: should work now