Nova Resource:Video/SAL

From Wikitech
Jump to navigation Jump to search

2019-05-16

  • 21:06 zhuyifei1999_: repool v2ccelery on encoding02, drain encoding01

2019-04-24

  • 20:50 zhuyifei1999_: upgraded youtube_dl on the two alive hosts. restarting v2ccelery on encoding02 and trying to stop that on encoding01 (still have a task running)

2019-04-07

  • 14:37 andrewbogott: restarting encoding01

2019-01-31

  • 12:06 arturo: VM instances encoding02,encoding03, were stopped briefly due to issue in hypervisor (T215012)

2018-12-06

  • 12:42 gtirloni: restarted gfg01

2018-11-20

  • 21:41 zhuyifei1999_: building a gentoo prefix on /data/project/gentoo-prefix so v2c can use multi-thread encoding with libvpx-vp9 1.7

2018-11-16

  • 17:07 zhuyifei1999_: re-pooling encoding01 & encoding02
  • 16:00 andrewbogott: moving video-redis to eqiad1-r
  • 15:43 zhuyifei1999_: depooling all 3 encoding*
  • 04:43 zhuyifei1999_: start depool encoding02. encoding03 is pooled in reboot in eqiad-1 migration

2018-04-15

  • 19:30 zhuyifei1999_: deleted the files (for server side upload) and unmounted the bind mount on encoding01

2018-04-13

  • 20:01 zhuyifei1999_: bind mount /srv/zhuyifei1999 /srv/v2c/ssu on encoding01 so I can copy the videos to a publicly-accessible (v2c.wmflabs.org) instance local storage rather than the slooowww NFS T191572

2018-04-11

  • 20:42 zhuyifei1999_: repool v2ccelery on encoding03
  • 16:38 zhuyifei1999_: depool v2ccelery on encoding01 as well for 'snapshot' generation T191572
  • 01:26 zhuyifei1999_: undo that. load went down (2.03 -> 2.01). probably not worth it when it's running in a hypervisor T191572
  • 00:52 zhuyifei1999_: set cpu affinity (gfg01$ taskset -p -c 0 29926; taskset -p -c 1 30181) on the main threads of two ffmpeg processes in an attempt to speed it up T191572

2018-04-10

  • 19:37 zhuyifei1999_: depool v2ccelery on encoding03

2018-03-29

  • 17:18 zhuyifei1999_: upgraded youtube_dl in frontend from 2017.9.2 to 2018.3.26.1

2018-03-28

  • 21:07 zhuyifei1999_: systemd didn't restart them. Maybe because exitcode is 0? Manually started
  • 18:18 zhuyifei1999_: depooling encoding0[1-3]. systemd will restart them
  • 18:15 zhuyifei1999_: upgrading youtube_dl from 2017.8.27 to 2018.3.26.1 on encoding0[1-3]

2018-02-15

  • 19:09 zhuyifei1999_: schedule reboot on encoding02 1 day later with `sudo shutdown -r +1440`, scheduled for Fri 2018-02-16 19:07:47 UTC
  • 19:04 zhuyifei1999_: reboot encoding03
  • 19:03 zhuyifei1999_: depool encoding02, encoding01 is pooled by reboot
  • 18:59 madhuvishy: Rebooting encoding-01

2017-09-05

  • 03:38 zhuyifei1999_: upgrading youtube_dl from 2017.8.6 to 2017.9.2 on frontend and restarting webservic

2017-08-26

  • 22:02 zhuyifei1999_: restart v2ccelery on encoding03, depool encoding01, repool encoding02
  • 21:56 zhuyifei1999_: upgrading youtube_dl on encoding{01..03} from 2017.8.6 to 2017.8.27

2017-08-14

  • 01:13 zhuyifei1999_: deployed 0a234d6..3c4ce0c frontend

2017-08-06

  • 03:41 zhuyifei1999_: upgrading youtube_dl on encoding0{1..3} & frontend, from version 2017.6.18

2017-06-26

  • 23:56 zhuyifei1999_: oops, depooled encoding02 instead
  • 23:54 zhuyifei1999_: depool encoding03
  • 23:50 zhuyifei1999_: Doing that again, should be finally working
  • 23:34 zhuyifei1999_: Doing that again :(
  • 23:24 zhuyifei1999_: Fixing tmpfiles configuration on encoding0{1..3}, and rebooting; expecing to see workers back alive after reboot
  • 23:16 zhuyifei1999_: failed to start because systemd-tmpfiles-setup failed. investigating
  • 23:00 zhuyifei1999_: starting v2c workers at encoding02 & 03, somehow they didn't start automatically after reboot

2017-06-21

  • 00:04 zhuyifei1999_: depool v2c worker on encoding01,02; repool encoding01,03
  • 00:03 zhuyifei1999_: `sudo -H pip install -U youtube-dl` on all encoding hosts

2017-06-11

  • 14:25 zhuyifei1999_: Restart v2c frontend
  • 14:24 zhuyifei1999_: Repool v2c on encoding0{1,2}
  • 14:23 zhuyifei1999_: Depool v2c on encoding0{2,3}
  • 14:15 zhuyifei1999_: Updating code on encoding0{1..3} and frontend to 1e60ecc

2017-03-27

  • 13:49 zhuyifei1999_: Disable autoconfirmed + 50 edits requirement in v2c for GSoC demonstration

2017-02-17

  • 11:03 zhuyifei1999_: git pulling v2c frontend to 6f329f3 and restarting webservice

2017-02-16

  • 11:59 zhuyifei1999_: depooled v2c worker at encoding01, depooled & repooled encoding02, repooled encoding03
  • 11:57 zhuyifei1999_: `pip install -U youtube-dl` on all v2c worker instances, version 2017.1.10 -> 2017.2.16

2017-02-07

  • 20:04 zhuyifei1999_: git pulling v2c frontend to 74a4ad8 and restarting webservice
  • 19:31 zhuyifei1999_: git pulling v2c frontend to a651871 and restarting webservice
  • 17:15 zhuyifei1999_: git pulling v2c frontend to bc77044 and restarting webservice
  • 16:54 zhuyifei1999_: git pulling v2c frontend to 2e55341 and restarting webservice

2017-02-05

  • 10:56 zhuyifei1999_: Git pulling v2c frontend to 9c86a9f and restarting webservice
  • 10:52 zhuyifei1999_: `$ pip install -U youtube-dl` from 2017.1.10 to 2017.2.4.1 on v2c frontend

2017-01-29

  • 08:53 zhuyifei1999_: git pulling frontend to master (1cad581) and restarting webservice

2017-01-25

  • 10:21 zhuyifei1999_: `$ sudo apt update && sudo apt dist-upgrade` on all hosts
  • 07:48 zhuyifei1999_: repooling encoding02

2017-01-24

  • 21:33 zhuyifei1999_: depooling encoding02 & 03, repooling encoding01
  • 21:30 zhuyifei1999_: git pulling v2c code to 8cf788e across all encoding hosts
  • 18:27 zhuyifei1999_: repooling encoding02 & encoding03, depooling encoding01 T155455
  • 18:23 zhuyifei1999_: killed the rest of tasks on encoding02 & encoding03, they are gonna fail anyways
  • 16:17 zhuyifei1999_: encoding03 free disk space < 10G, renicing PID 9136 (50c81be) & 3499 (a32b3fa) & 3611 (0e73d27) to niceness of 1, so these near-complete tasks can finish early and clear some disk space
  • 15:32 zhuyifei1999_: git pulling v2c code to b2d6e84 across all encoding hosts T155455
  • 14:58 zhuyifei1999_: killing the rest of the tasks on encoding01 because they aren't going to succeed anyways; depooling encoding03; reloading encoding01 T155455
  • 14:19 zhuyifei1999_: depooling encoding01 & 02 and waiting for tasks to finish before restarting (reload the code) T155455
  • 14:15 zhuyifei1999_: git pulling v2c code to 31479a4 across all encoding hosts T155455
  • 14:10 zhuyifei1999_: pip installing pywikibot at 7ac0142 across all encoding hosts T155455

2017-01-23

  • 12:10 zhuyifei1999_: starting v2c worker at encoding03 due to flooding

2017-01-20

  • 12:01 zhuyifei1999_: restarting all currently running v2c workers (celery{1,2}@encoding0{1,2}) and web frontend due to NFS outage, and we aren't very sure which component is erroring (T155803)

2017-01-15

  • 09:28 zhuyifei1999_: pooled encoding02

2017-01-12

  • 09:12 zhuyifei1999_: depooling encoding02 + pooling encoding01 + restarting frontend, to upgrade youtube_dl

2017-01-11

  • 05:46 zhuyifei1999_: re-enabling v2c frontend & pooling worker at encoding02; transcode queue is < 500 now, and mostly containing relatively small files (T153488)

2016-12-29

  • 08:57 zhuyifei1999_: depooled encoding02 per request from operations

2016-12-28

  • 10:58 zhuyifei1999_: pooling encoding02, depooling encoding01&03 to update its code

2016-12-27

  • 19:20 zhuyifei1999_: depooling encoding02 since the flood is under control
  • 15:56 zhuyifei1999_: starting the two other workers because of flood, trying to restore the system to a "ready for new tasks" state

2016-12-22

  • 12:10 zhuyifei1999_: depooling v2c workers at encoding02 and encoding03 in order to make transcoding slower and have production videoscalers catch up (T153488)

2016-12-11

  • 15:36 zhuyifei1999_: depooling encoding02, less than 5G of disk space left

2016-07-13

  • 15:02 andrewbogott: deleting/recreating encoding01

2016-07-11

  • 21:22 andrewbogott: deleting instances encoding02 and encoding03, soon to recreate

2016-03-22

  • 14:53 andrewbogott: rebooting encoding3 because it's mostly broken

2015-07-31

  • 19:39 andrewbogott: migrating encoding01 to a new virt host.

November 19

  • 04:57 Ryan_Lane: created video project for development and testing of an external video extension for MediaWiki