Nova Resource:Ores/SAL

From Wikitech
Jump to: navigation, search

2017-07-25

  • 16:48 halfak: deploying ores-wmflabs-deploy:68326ba
  • 16:12 halfak: staging ores-wmflabs-deploy:68326ba

2017-07-17

  • 15:42 halfak: deleting ores-web-04 (intermittent puppet issues and wasn't pooled anyway)

2017-07-06

  • 00:07 bd808: bd808's root key is rejected by ores-compute-01.ores.eqiad.wmflabs. Puppet busted there?

2017-06-28

  • 17:55 halfak: deleting mediawiki-ores and hashing-vector instances from ores-staging (unused)

2017-06-22

  • 00:03 awight: labs ores was down 17:13-17:22 due to maintenance.

2017-06-16

  • 21:37 halfak: deploying ores-wmflabs-deploy:e8f6932
  • 21:34 halfak: staging ores-wmflabs-deploy:e8f6932
  • 21:06 halfak: deploying ores-wmflabs-deploy:c952f57 (reverting due to issues)
  • 20:52 halfak: deploying ores-wmflabs-deploy:7529113
  • 20:26 halfak: staging ores-wmflabs-deploy:7529113

2017-05-20

  • 14:47 halfak: staging ores-wmflabs-deploy:9df7b5a
  • 14:44 halfak: staging ores-wmflabs-deploy:ed1991c

2017-04-25

  • 21:33 halfak: deployed ores-wmflabs-deploy:dc934e8
  • 21:30 halfak: staged ores-wmflabs-deploy:dc934e8

2017-04-06

  • 16:00 halfak: deploying ores-wmflabs-deploy:7061f0a
  • 15:56 halfak: staging ores-wmflabs-deploy:7061f0a

2017-03-28

  • 15:58 halfak: deployed ores-wmflabs-deploy:18beebf (T160638)
  • 15:57 halfak: deployed ores-wmflabs-deploy:18beebf
  • 15:48 halfak: staged ores-wmflabs-deploy:18beebf

2017-03-26

  • 07:39 Amir1: restart ores-web-03

2017-03-15

  • 19:26 halfak: deployed ores-wmflabs-deploy:0990b1d
  • 19:24 halfak: staged ores-wmflabs-deploy:0990b1d
  • 19:03 halfak: deployed ores-wmflabs-deploy:080418d
  • 17:36 halfak: deployed ores-wmflabs-deploy:080418d to staging

2017-03-06

  • 20:26 halfak: restarted precached and uwsgi-ores on ores-web-03

2017-02-23

  • 18:55 halfak: restarted uwsgi-ores and precached on ores-web-03 (memory issues)

2017-02-21

  • 03:40 halfak: restarted uwsgi and precached on ores-web-03

2017-02-19

  • 15:54 Amir1: (except redis ones)
  • 15:54 Amir1: rebooting all instances

2017-02-11

  • 18:12 halfak: deployed ores-wmflabs-deploy:f58d3fd
  • 18:09 halfak: staged ores-wmflabs-deploy:f58d3fd

2017-02-01

  • 23:39 halfak: deployed ores-wmflabs-deploy:ce8861d

2017-01-25

  • 21:20 halfak: deployed ores-wmflabs-deploy:0f90516

2017-01-18

  • 19:47 halfak: restarted precached and uwsgi-ores on ores-web-03. Memory usage issues.

2017-01-15

  • 03:03 Amir1: restarting ores-web-03

2017-01-04

  • 12:51 halfak: ran 'sudo service uwsgi-ores restart' on ores-web-03
  • 11:41 Amir1: stopping precaching in ores in labs to reduce load
  • 11:40 Amir1: reboot ores-lb-02

2016-11-22

  • 16:20 mutante: ores-web-03 has an issue (Socket timeout, CRIT)

2016-11-07

  • 20:45 halfak: restarted uwsgi-ores and precached on ores-web-03

2016-11-03

  • 17:36 halfak: manually ran "sudo mkdir /srv/log; sudo mkdir /srv/log/ores; sudo service celery-ores-worker restart" on ores-worker-(05|06|07|08|09|10)
  • 17:23 halfak: deployed ores-wmflabs-deploy:ffb5dbb
  • 17:16 halfak: deployed ores-wmflabs-deploy:ffb5dbb to staging (fixes logging bug)
  • 17:04 halfak: deployed ores-wmflabs-deploy:74699db to staging

2016-09-25

  • 16:03 Amir1: deploying c2ba205 (T146581)
  • 15:59 Amir1: c2ba205 going to staging
  • 15:26 halfak: restarted precached and uwsgi-ores on ores-web-03

2016-09-23

  • 20:49 halfak: restarted ores-uwsgi and precached services on ores-web-03

2016-09-13

  • 03:03 halfak: deployed ores-wmflabs-deploy:022fb7d

2016-09-12

  • 13:47 halfak: restarted precached service on ores-web-03
  • 04:31 halfak: manually killed since "python" process consuming 80% of memory and restarted uwsgi-ores service on ores-web-03
  • 04:29 halfak: restarted uwsgi-ores service on ores-web-03

2016-09-02

  • 20:25 halfak: deployed ores-wmflabs-deploy:751860f

2016-08-29

  • 14:03 chasemp: reboot 94886e74-5be4-4669-a1c1-840ce7c65de9 ores-compute-01

2016-08-24

  • 22:00 halfak: restarted uwsgi-ores on ores-web-05

2016-08-19

  • 21:10 halfak: deployed ores-wmflabs-deploy:f0fc59b

2016-08-12

  • 18:25 halfak: ran FLUSHALL on ores-redis-02:6380
  • 18:24 halfak: deployed ores-wmflabs-deploy:b015348

2016-08-09

  • 22:08 halfak: restarted precached service on ores-web-03
  • 22:08 halfak: deployed ores-wmflabs-config:1b1c56d

2016-08-08

  • 20:44 Amir1: depooling ores-web-04

2016-08-05

  • 17:44 halfak: repooled ores-web-04 in hiera and ran puppet on ores-lb-02
  • 16:15 halfak: creates ores-web-04 debian-8.5-jessie
  • 16:11 halfak: terminated ores-web-04 (re-creating instance with medium image size)

2016-08-04

  • 18:02 halfak: rebooting ores-web-05 to address OOM
  • 17:56 halfak: reduced web workers per core from 28 to 16

2016-07-29

  • 00:20 halfak: deployed ores-wmflabs-deploy:770d131

2016-07-13

  • 18:20 Amir1: ladsgroup@ores-lb-02:~$ sudo service nginx restart
  • 17:21 Amir1: deploying to web again
  • 17:14 Amir1: ladsgroup@ores-web-05:~$ sudo puppet agent -tv
  • 16:32 halfak: terminated ores-web-05
  • 15:58 halfak: repooled ores-web-04
  • 15:20 halfak: deployed ores-wmflabs-deploy:e638f1b

2016-07-12

  • 11:25 Amir1: deploying aad92ac to web and worker nodes
  • 11:18 Amir1: aad92ac goes to staging

2016-07-08

  • 20:20 halfak: deployed ores-wmflabs-deploy:3f116cd

2016-06-10

  • 20:25 halfak: deleted 99-redis.yaml and restarted celery-ores-worker on all worker nodes
  • 07:36 Amir1: deploying 4efc5b7 into prod

2016-06-03

  • 23:57 Amir1: removing role role::labs::ores::flower from ores-web-03

2016-05-27

  • 13:31 Amir1: adding role::deployment::server to ores project

2016-05-26

  • 10:05 Amir1: rebooting ores-web-04
  • 10:02 Amir1: deploying 4dc6050 into web and worker nodes, updating to revscoring 1.2.6

2016-05-25

  • 19:33 Amir1: running puppet agent manually in ores-web-03
  • 19:29 Amir1: deploying 7992fd1 into web and worker nodes

2016-05-23

  • 22:38 Amir1: precaching brought back online
  • 22:33 Amir1: stopping precaching in ores-web-03 manually. Testing something

2016-05-18

  • 16:48 Amir1: running puppet agent on ores-lb-02 manually

2016-05-16

  • 13:00 Amir1: deploying e51c4f3 to celery and web nodes

2016-05-09

  • 07:52 Amir1: manually restarted nginx load balancer (ores-lb-02) It wasn't needed since puppet did it already.

2016-04-30

  • 18:05 Amir1: deployed d4f63a3 to web and worker nodes
  • 17:49 Amir1: deploy d4f63a3 to the staging
  • 17:33 Amir1: deploying 30ba552 to the staging
  • 14:54 Amir1: running puppet agent manually in ores-web-03
  • 14:52 Amir1: added precaching role to ores-web-03

2016-04-27

  • 20:48 halfak: deployed ores-wikimedia-config:6453fe5

2016-04-23

  • 15:25 Amir1: sudden disruption due to an unexpected deploy

2016-04-07

  • 14:32 halfak: deployed ores-wikimedia-config:34ba0fa

2016-03-30

  • 16:13 halfak: deployed ores-wikimedia-config:58905c5

2016-03-23

  • 17:39 halfak: deployed ores-wikimedia-config:39b622e
  • 01:24 halfak: deployed ores-wikimedia-config:8af4377

2016-03-19

  • 20:04 halfak: ores-web-01/02 and ores-worker-01/02/03/04 deleted. ores-web-03/04/05 and ores-worker-05/06/07/08/09/10 started and configured as replacements.
  • 16:43 halfak: Manually ran `sudo apt-get install aspell-ar aspell-pl` across web and worker nodes
  • 15:48 halfak: Ran puppet and restarted uwsgi on web-01 with 28 forks rather than 32
  • 15:32 halfak: workers_per_core 32 --> 28

2016-03-18

  • 01:17 mutante: while doing that i noticed that ores-web-02 has an "Cannot allocate memory" problem
  • 01:17 mutante: but no class names have changed and confirmed no-op on every single instance
  • 01:17 mutante: merged https://gerrit.wikimedia.org/r/#/c/270102/ - role classes have been moved to modules/role and split into one file per class
  • 01:07 mutante: testing puppet on all instances before merging change that moves role classes to module

2016-03-05

  • 17:58 halfak: Shut down the uwsgi service and started up the uwsgi-ores-web service on web-01 and -02.

2016-03-04

  • 22:54 andrewbogott: migrating ores-worker-02 to labvirt1010

2016-02-15

  • 16:26 halfak: restarted workers on ores-worker -02 and -03
  • 04:01 halfak: Ammending last message -- Restarted the uwsgi-ores-web service on *ores-web-01*. The service on ores-web-02 was left alone.
  • 03:59 halfak: Restarted the uwsgi-ores-web service on ores-web-02

2016-02-07

  • 19:21 halfak: Deployed ores-wikimedia-config:0be5afc

2015-12-30

  • 20:23 halfak: Deployed with wb-vandalism:d940cea
  • 20:22 YuviPanda: create ores-cache-01

2015-12-21

  • 20:28 YuviPanda: depool ores-web-01 from lb

2015-12-16

  • 14:24 YuviPanda: updated quota for instances to arbitrary number (40)

2015-12-08

  • 17:34 halfak: Deployed 1ad37c5 with ores:b745570, revscoring:, editquality:b41b7c1, wikiclass:bbfa9ce, and wb-vandalism:1075596

2015-11-12

  • 20:50 halfak: Deployed 334b4b2 with ores:b745570, revscoring:, editquality:b41b7c1, wikiclass:bbfa9ce, and wb-vandalism:a09d026

2015-10-28

  • 21:05 halfak: Deployed ores-wikimedia-config:c27c8f7, ores==0.5.4, wb-vandalism==0.1.5, revscoring==0.6.7

2015-09-12

  • 02:30 halfak: Deployed ores-wikimedia-config:c852404, ores==0.5.0 and revscoring==0.6.1

2015-09-11

  • 14:51 halfak: removed ores-web-02 from ores-lb-02 pool
  • 03:31 YuviPanda: restart redis server on ores-redis-02 to apply tcp-keepalive

2015-09-10

  • 18:28 YuviPanda: disabled restart crontab on ores-worker-01 under halfak's user, have set it to keep insane levels of logs to debug

2015-09-08

  • 16:35 halfak: deployed ores-wikimedia-config:ca10888, ores==0.4.0 and revscoring==0.5.0
  • 16:34 valhallasw`cloud: should work now


=============