Nova Resource:Ores/SAL

2017-06-22

• 00:03 awight: labs ores was down 17:13-17:22 due to maintenance.

2017-06-16

• 21:37 halfak: deploying ores-wmflabs-deploy:e8f6932
• 21:34 halfak: staging ores-wmflabs-deploy:e8f6932
• 21:06 halfak: deploying ores-wmflabs-deploy:c952f57 (reverting due to issues)
• 20:52 halfak: deploying ores-wmflabs-deploy:7529113
• 20:26 halfak: staging ores-wmflabs-deploy:7529113

2017-05-20

• 14:47 halfak: staging ores-wmflabs-deploy:9df7b5a
• 14:44 halfak: staging ores-wmflabs-deploy:ed1991c

2017-04-25

• 21:33 halfak: deployed ores-wmflabs-deploy:dc934e8
• 21:30 halfak: staged ores-wmflabs-deploy:dc934e8

2017-04-06

• 16:00 halfak: deploying ores-wmflabs-deploy:7061f0a
• 15:56 halfak: staging ores-wmflabs-deploy:7061f0a

2017-03-28

• 15:58 halfak: deployed ores-wmflabs-deploy:18beebf (T160638)
• 15:57 halfak: deployed ores-wmflabs-deploy:18beebf
• 15:48 halfak: staged ores-wmflabs-deploy:18beebf

2017-03-26

• 07:39 Amir1: restart ores-web-03

2017-03-15

• 19:26 halfak: deployed ores-wmflabs-deploy:0990b1d
• 19:24 halfak: staged ores-wmflabs-deploy:0990b1d
• 19:03 halfak: deployed ores-wmflabs-deploy:080418d
• 17:36 halfak: deployed ores-wmflabs-deploy:080418d to staging

2017-03-06

• 20:26 halfak: restarted precached and uwsgi-ores on ores-web-03

2017-02-23

• 18:55 halfak: restarted uwsgi-ores and precached on ores-web-03 (memory issues)

2017-02-21

• 03:40 halfak: restarted uwsgi and precached on ores-web-03

2017-02-19

• 15:54 Amir1: (except redis ones)
• 15:54 Amir1: rebooting all instances

2017-02-11

• 18:12 halfak: deployed ores-wmflabs-deploy:f58d3fd
• 18:09 halfak: staged ores-wmflabs-deploy:f58d3fd

2017-02-01

• 23:39 halfak: deployed ores-wmflabs-deploy:ce8861d

2017-01-25

• 21:20 halfak: deployed ores-wmflabs-deploy:0f90516

2017-01-18

• 19:47 halfak: restarted precached and uwsgi-ores on ores-web-03. Memory usage issues.

2017-01-15

• 03:03 Amir1: restarting ores-web-03

2017-01-04

• 12:51 halfak: ran 'sudo service uwsgi-ores restart' on ores-web-03
• 11:41 Amir1: stopping precaching in ores in labs to reduce load
• 11:40 Amir1: reboot ores-lb-02

2016-11-22

• 16:20 mutante: ores-web-03 has an issue (Socket timeout, CRIT)

2016-11-07

• 20:45 halfak: restarted uwsgi-ores and precached on ores-web-03

2016-11-03

• 17:36 halfak: manually ran "sudo mkdir /srv/log; sudo mkdir /srv/log/ores; sudo service celery-ores-worker restart" on ores-worker-(05|06|07|08|09|10)
• 17:23 halfak: deployed ores-wmflabs-deploy:ffb5dbb
• 17:16 halfak: deployed ores-wmflabs-deploy:ffb5dbb to staging (fixes logging bug)
• 17:04 halfak: deployed ores-wmflabs-deploy:74699db to staging

2016-09-25

• 16:03 Amir1: deploying c2ba205 (T146581)
• 15:59 Amir1: c2ba205 going to staging
• 15:26 halfak: restarted precached and uwsgi-ores on ores-web-03

2016-09-23

• 20:49 halfak: restarted ores-uwsgi and precached services on ores-web-03

2016-09-13

• 03:03 halfak: deployed ores-wmflabs-deploy:022fb7d

2016-09-12

• 13:47 halfak: restarted precached service on ores-web-03
• 04:31 halfak: manually killed since "python" process consuming 80% of memory and restarted uwsgi-ores service on ores-web-03
• 04:29 halfak: restarted uwsgi-ores service on ores-web-03

2016-09-02

• 20:25 halfak: deployed ores-wmflabs-deploy:751860f

2016-08-29

• 14:03 chasemp: reboot 94886e74-5be4-4669-a1c1-840ce7c65de9 ores-compute-01

2016-08-24

• 22:00 halfak: restarted uwsgi-ores on ores-web-05

2016-08-19

• 21:10 halfak: deployed ores-wmflabs-deploy:f0fc59b

2016-08-12

• 18:25 halfak: ran FLUSHALL on ores-redis-02:6380
• 18:24 halfak: deployed ores-wmflabs-deploy:b015348

2016-08-09

• 22:08 halfak: restarted precached service on ores-web-03
• 22:08 halfak: deployed ores-wmflabs-config:1b1c56d

2016-08-08

• 20:44 Amir1: depooling ores-web-04

2016-08-05

• 17:44 halfak: repooled ores-web-04 in hiera and ran puppet on ores-lb-02
• 16:15 halfak: creates ores-web-04 debian-8.5-jessie
• 16:11 halfak: terminated ores-web-04 (re-creating instance with medium image size)

2016-08-04

• 18:02 halfak: rebooting ores-web-05 to address OOM
• 17:56 halfak: reduced web workers per core from 28 to 16

2016-07-29

• 00:20 halfak: deployed ores-wmflabs-deploy:770d131

2016-07-13

• 18:20 Amir1: ladsgroup@ores-lb-02:~$ sudo service nginx restart
• 17:21 Amir1: deploying to web again
• 17:14 Amir1: ladsgroup@ores-web-05:~$ sudo puppet agent -tv
• 16:32 halfak: terminated ores-web-05
• 15:58 halfak: repooled ores-web-04
• 15:20 halfak: deployed ores-wmflabs-deploy:e638f1b

2016-07-12

• 11:25 Amir1: deploying aad92ac to web and worker nodes
• 11:18 Amir1: aad92ac goes to staging

2016-07-08

• 20:20 halfak: deployed ores-wmflabs-deploy:3f116cd

2016-06-10

• 20:25 halfak: deleted 99-redis.yaml and restarted celery-ores-worker on all worker nodes
• 07:36 Amir1: deploying 4efc5b7 into prod

2016-06-03

• 23:57 Amir1: removing role role::labs::ores::flower from ores-web-03

2016-05-27

• 13:31 Amir1: adding role::deployment::server to ores project

2016-05-26

• 10:05 Amir1: rebooting ores-web-04
• 10:02 Amir1: deploying 4dc6050 into web and worker nodes, updating to revscoring 1.2.6

2016-05-25

• 19:33 Amir1: running puppet agent manually in ores-web-03
• 19:29 Amir1: deploying 7992fd1 into web and worker nodes

2016-05-23

• 22:38 Amir1: precaching brought back online
• 22:33 Amir1: stopping precaching in ores-web-03 manually. Testing something

2016-05-18

• 16:48 Amir1: running puppet agent on ores-lb-02 manually

2016-05-16

• 13:00 Amir1: deploying e51c4f3 to celery and web nodes

2016-05-09

• 07:52 Amir1: manually restarted nginx load balancer (ores-lb-02) It wasn't needed since puppet did it already.

2016-04-30

• 18:05 Amir1: deployed d4f63a3 to web and worker nodes
• 17:49 Amir1: deploy d4f63a3 to the staging
• 17:33 Amir1: deploying 30ba552 to the staging
• 14:54 Amir1: running puppet agent manually in ores-web-03
• 14:52 Amir1: added precaching role to ores-web-03

2016-04-27

• 20:48 halfak: deployed ores-wikimedia-config:6453fe5

2016-04-23

• 15:25 Amir1: sudden disruption due to an unexpected deploy

2016-04-07

• 14:32 halfak: deployed ores-wikimedia-config:34ba0fa

2016-03-30

• 16:13 halfak: deployed ores-wikimedia-config:58905c5

2016-03-23

• 17:39 halfak: deployed ores-wikimedia-config:39b622e
• 01:24 halfak: deployed ores-wikimedia-config:8af4377

2016-03-19

• 20:04 halfak: ores-web-01/02 and ores-worker-01/02/03/04 deleted. ores-web-03/04/05 and ores-worker-05/06/07/08/09/10 started and configured as replacements.
• 16:43 halfak: Manually ran `sudo apt-get install aspell-ar aspell-pl` across web and worker nodes
• 15:48 halfak: Ran puppet and restarted uwsgi on web-01 with 28 forks rather than 32
• 15:32 halfak: workers_per_core 32 --> 28

2016-03-18

• 01:17 mutante: while doing that i noticed that ores-web-02 has an "Cannot allocate memory" problem
• 01:17 mutante: but no class names have changed and confirmed no-op on every single instance
• 01:17 mutante: merged https://gerrit.wikimedia.org/r/#/c/270102/ - role classes have been moved to modules/role and split into one file per class
• 01:07 mutante: testing puppet on all instances before merging change that moves role classes to module

2016-03-05

• 17:58 halfak: Shut down the uwsgi service and started up the uwsgi-ores-web service on web-01 and -02.

2016-03-04

• 22:54 andrewbogott: migrating ores-worker-02 to labvirt1010

2016-02-15

• 16:26 halfak: restarted workers on ores-worker -02 and -03
• 04:01 halfak: Ammending last message -- Restarted the uwsgi-ores-web service on *ores-web-01*. The service on ores-web-02 was left alone.
• 03:59 halfak: Restarted the uwsgi-ores-web service on ores-web-02

2016-02-07

• 19:21 halfak: Deployed ores-wikimedia-config:0be5afc

2015-12-30

• 20:23 halfak: Deployed with wb-vandalism:d940cea
• 20:22 YuviPanda: create ores-cache-01

2015-12-21

• 20:28 YuviPanda: depool ores-web-01 from lb

2015-12-16

• 14:24 YuviPanda: updated quota for instances to arbitrary number (40)

2015-12-08

• 17:34 halfak: Deployed 1ad37c5 with ores:b745570, revscoring:, editquality:b41b7c1, wikiclass:bbfa9ce, and wb-vandalism:1075596

2015-11-12

• 20:50 halfak: Deployed 334b4b2 with ores:b745570, revscoring:, editquality:b41b7c1, wikiclass:bbfa9ce, and wb-vandalism:a09d026

2015-10-28

• 21:05 halfak: Deployed ores-wikimedia-config:c27c8f7, ores==0.5.4, wb-vandalism==0.1.5, revscoring==0.6.7

2015-09-12

• 02:30 halfak: Deployed ores-wikimedia-config:c852404, ores==0.5.0 and revscoring==0.6.1

2015-09-11

• 14:51 halfak: removed ores-web-02 from ores-lb-02 pool
• 03:31 YuviPanda: restart redis server on ores-redis-02 to apply tcp-keepalive

2015-09-10

• 18:28 YuviPanda: disabled restart crontab on ores-worker-01 under halfak's user, have set it to keep insane levels of logs to debug

2015-09-08

• 16:35 halfak: deployed ores-wikimedia-config:ca10888, ores==0.4.0 and revscoring==0.5.0
• 16:34 valhallasw`cloud: should work now