20:29 andrewbogott: rebooting all VMs to switch NFS mounts to the new project-local nfs server
2021-12-07
18:16 chicocvenancio: deploying latest change to retry downloads without local cache
2021-11-17
19:56 chicocvenancio: manually create /srv/v2c/apicache-py3 to solve T295115
19:03 chicocvenancio: update video2commons to latest version, fix service unit file
2021-10-28
17:49 chicocvenancio: move encoding05 and encoding06 to python3 T294520
17:25 chicocvenancio: move enconding04 to python3 T294520
2020-06-15
08:51 matanya: rebooting encoding* nodes due to cpu overload
2020-01-04
16:09 bstorm_: moving VMs encoding04 and encoding05 from cloudvirt1024 to cloudvirt1003 due to hardware errors T241884
2019-10-26
21:42 matanya: buster migration done zhuyifei1999 is awesome
21:41 matanya: deleting gfg01 as part of buster migration
21:32 matanya: deleting encoding03 as part of buster migration
20:13 zhuyifei1999_: copying data over from gfg01:/srv/zhuyifei1999 to gfg:/srv/zhuyifei1999
19:41 zhuyifei1999_: replicated data from video-redis to video-redis-buster and then disabled replication
19:31 matanya: deleting encoding02 and videodev as part of buster migration
19:23 matanya: deleting encoding01 as part of buster migration
2019-05-16
21:06 zhuyifei1999_: repool v2ccelery on encoding02, drain encoding01
2019-04-24
20:50 zhuyifei1999_: upgraded youtube_dl on the two alive hosts. restarting v2ccelery on encoding02 and trying to stop that on encoding01 (still have a task running)
2019-04-07
14:37 andrewbogott: restarting encoding01
2019-01-31
12:06 arturo: VM instances encoding02,encoding03, were stopped briefly due to issue in hypervisor (T215012)
2018-12-06
12:42 gtirloni: restarted gfg01
2018-11-20
21:41 zhuyifei1999_: building a gentoo prefix on /data/project/gentoo-prefix so v2c can use multi-thread encoding with libvpx-vp9 1.7
16:00 andrewbogott: moving video-redis to eqiad1-r
15:43 zhuyifei1999_: depooling all 3 encoding*
04:43 zhuyifei1999_: start depool encoding02. encoding03 is pooled in reboot in eqiad-1 migration
2018-04-15
19:30 zhuyifei1999_: deleted the files (for server side upload) and unmounted the bind mount on encoding01
2018-04-13
20:01 zhuyifei1999_: bind mount /srv/zhuyifei1999 /srv/v2c/ssu on encoding01 so I can copy the videos to a publicly-accessible (v2c.wmflabs.org) instance local storage rather than the slooowww NFS T191572
2018-04-11
20:42 zhuyifei1999_: repool v2ccelery on encoding03
16:38 zhuyifei1999_: depool v2ccelery on encoding01 as well for 'snapshot' generation T191572
01:26 zhuyifei1999_: undo that. load went down (2.03 -> 2.01). probably not worth it when it's running in a hypervisor T191572
00:52 zhuyifei1999_: set cpu affinity (gfg01$ taskset -p -c 0 29926; taskset -p -c 1 30181) on the main threads of two ffmpeg processes in an attempt to speed it up T191572
2018-04-10
19:37 zhuyifei1999_: depool v2ccelery on encoding03
2018-03-29
17:18 zhuyifei1999_: upgraded youtube_dl in frontend from 2017.9.2 to 2018.3.26.1
2018-03-28
21:07 zhuyifei1999_: systemd didn't restart them. Maybe because exitcode is 0? Manually started
18:18 zhuyifei1999_: depooling encoding0[1-3]. systemd will restart them
18:15 zhuyifei1999_: upgrading youtube_dl from 2017.8.27 to 2018.3.26.1 on encoding0[1-3]
2018-02-15
19:09 zhuyifei1999_: schedule reboot on encoding02 1 day later with `sudo shutdown -r +1440`, scheduled for Fri 2018-02-16 19:07:47 UTC
19:04 zhuyifei1999_: reboot encoding03
19:03 zhuyifei1999_: depool encoding02, encoding01 is pooled by reboot
18:59 madhuvishy: Rebooting encoding-01
2017-09-05
03:38 zhuyifei1999_: upgrading youtube_dl from 2017.8.6 to 2017.9.2 on frontend and restarting webservic
2017-08-26
22:02 zhuyifei1999_: restart v2ccelery on encoding03, depool encoding01, repool encoding02
21:56 zhuyifei1999_: upgrading youtube_dl on encoding{01..03} from 2017.8.6 to 2017.8.27
18:23 zhuyifei1999_: killed the rest of tasks on encoding02 & encoding03, they are gonna fail anyways
16:17 zhuyifei1999_: encoding03 free disk space < 10G, renicing PID 9136 (50c81be) & 3499 (a32b3fa) & 3611 (0e73d27) to niceness of 1, so these near-complete tasks can finish early and clear some disk space
15:32 zhuyifei1999_: git pulling v2c code to b2d6e84 across all encoding hosts T155455
14:58 zhuyifei1999_: killing the rest of the tasks on encoding01 because they aren't going to succeed anyways; depooling encoding03; reloading encoding01 T155455
14:19 zhuyifei1999_: depooling encoding01 & 02 and waiting for tasks to finish before restarting (reload the code) T155455
14:15 zhuyifei1999_: git pulling v2c code to 31479a4 across all encoding hosts T155455
14:10 zhuyifei1999_: pip installing pywikibot at 7ac0142 across all encoding hosts T155455
2017-01-23
12:10 zhuyifei1999_: starting v2c worker at encoding03 due to flooding
2017-01-20
12:01 zhuyifei1999_: restarting all currently running v2c workers (celery{1,2}@encoding0{1,2}) and web frontend due to NFS outage, and we aren't very sure which component is erroring (T155803)
05:46 zhuyifei1999_: re-enabling v2c frontend & pooling worker at encoding02; transcode queue is < 500 now, and mostly containing relatively small files (T153488)
2016-12-29
08:57 zhuyifei1999_: depooled encoding02 per request from operations
2016-12-28
10:58 zhuyifei1999_: pooling encoding02, depooling encoding01&03 to update its code
2016-12-27
19:20 zhuyifei1999_: depooling encoding02 since the flood is under control
15:56 zhuyifei1999_: starting the two other workers because of flood, trying to restore the system to a "ready for new tasks" state
2016-12-22
12:10 zhuyifei1999_: depooling v2c workers at encoding02 and encoding03 in order to make transcoding slower and have production videoscalers catch up (T153488)
2016-12-11
15:36 zhuyifei1999_: depooling encoding02, less than 5G of disk space left
