Nova Resource:Cvn/SAL

From Wikitech
Jump to: navigation, search



  • 18:33 Krinkle: moved CVNBot1 to cvn-app8 and upgraded to v1.22.0-alpha
  • 17:54 Krinkle: CVNBot12 is now running v1.22.0-alpha on cvn-app8 (Debian 9, Mono 4)


  • 23:29 Krinkle: Disabling /etc/cron.hourly/cvn-collect-log on all servers




  • 21:00 Krinkle: Ran sudo apt-get clean on cvn-apache8 to free up some space.
  • 20:47 Krinkle: Soft-rebooting cvn-apache8


  • 18:06 Krinkle: Restarted servers to have them re-join #cvn-broadcast following the random K-like from freenode staff


  • 18:26 Krinkle: CVNBot12 won't restart. Has been in a once-a-minute restart loop for the past 24 hours (log: "IRC: Closing Link: (Connection timed out)")
  • 18:19 Krinkle: CVNBot12 is online, but not in any channel. Process is running. Restarting..



  • 19:25 Krinkle: Changed e-mail address of Clogger cvnwiki account from my personal gmail to


  • 03:49 Krinkle: Fix cvndb-pull.cron permissions on cvn-apache8 (caused api to not get updated every 5 minutes)


  • 18:54 Krinkle: Deleted cvn-app5 instance


  • 01:54 Krinkle: [] Retiring cvn-app5 (removing public IP, shutting down instance). To be deleted in a few days.
  • 01:49 Krinkle: Restored CVNBot20 database from cvn-app5 (retired) to cvn-app7 (got corrupted in transfer somehow)
  • 01:43 Krinkle: CVNBot20 database backup 2 days ago (Nov 16) also corrupt.
  • 01:40 Krinkle: CVNBot20 database was corrupted for unknown reasons. Replaced with a copy of CVNBot14's database.
  • 00:15 Krinkle: 'sudo chmod 660 */CVNBot.ini' in /srv/cvn/services/cvnbot on cvn-app6 and cvn-app7
  • 00:09 Krinkle: 'sudo chmod 775 */CVNBot.exe */Lists.sqlite' in /srv/cvn/services/cvnbot on cvn-app6 and cvn-app7


  • 23:06 rxy_: rxy@cvn-app7 "sudo chmod g+w -R /srv/cvn/git/stillalive" for write permission of group members
  • 23:05 rxy_: rxy@cvn-app6 "sudo chmod g+w -R /srv/cvn/git/stillalive" for write permission of group members
  • 22:56 rxy_: rxy@cvn-app6 "sudo chmod g+w /srv/cvn/services/cvnbot" for write permission of group members
  • 22:56 rxy_: rxy@cvn-app6 "sudo chgrp project-cvn /srv/cvn/services/cvnbot" for write permission of group members
  • 22:49 rxy_: rxy@cvn-app7 "sudo chmod g+w /srv/cvn/services/cvnbot" for write permission of group members
  • 22:48 rxy_: rxy@cvn-app7 "sudo chgrp project-cvn /srv/cvn/services/cvnbot" for write permission of group members


  • 02:17 Krinkle: Moving CVNBot1, CVNBot4, CVNBot5, CVNBot12, and CVNBot13 from cvn-app5 to cvn-app7
  • 02:02 Krinkle: Moving Cubbie and SWBot3 from cvn-app5 to cvn-app7
  • 01:55 Krinkle: Moving CVNBot21 (#cvn-mediawiki) from cvn-app5 to new cvn-app7
  • 00:57 Krinkle: Provisioning cvn-app7


  • 06:13 Krinkle: Creating cvn-app7 (m1.large; ubuntu-14.04-trusty) - replacement for cvn-app5 (m1.xlarge; ubuntu-12.04-precise)
  • 06:00 Krinkle: Decom cvn-app4 (created 2014; ubuntu-12.04-precise) - all bots migrated to cvn-app6 (ubuntu-14.04-trusty)
  • 06:00 Krinkle: Moving CVNBot23 from cvn-app4 to cvn-app6
  • 05:55 Krinkle: Moving CVNBot9, CVNBot10, and CVNBot19 from cvn-app4 to cvn-app6
  • 05:40 Krinkle: Moving CVNBot6, CVNBot7, and CVNBot8 from cvn-app4 to cvn-app6
  • 05:17 Krinkle: Moving CVNBot3 from cvn-app4 to cvn-app6


  • 11:09 rxy: "rxy@cvn-app4:~$ sudo kill $CVNBot3pid" for restarting.
  • 11:09 rxy: "rxy@cvn-app5:~$ sudo kill $CVNBot4pid, $CVNBot14pid, $CVNBot17pid" for restarting.


  • 18:58 rxy: rebooting cvn-app4.eqiad.wmflabs


  • 00:21 rxy: "sudo kill $CVNBot16pid" for restarting.


  • 21:34 Krinkle: cvn-app6 is now up and running. 1/10 bots from cvn-app4 (CVNBot2) has been migrated to it. No other bots are migrated yet as cvn-app6 does not yet have a dedicated IP and therefore is subject to the shared freenode limit for labs.


  • 10:47 Krinkle: cvn-app6 (trusty) running stable. Migration from cvn-app4 to continue as planned.
  • 10:47 Krinkle: Restarted CVNBot14 (#cvn-commons) - IRC connection was lost


  • 02:00 Krinkle: Provision cvn-app6 based on infrastructure setup.yaml


  • 17:26 Krinkle: Creating cvn-app6 (Ubuntu 14 Trusty, Size: Large) as migration target for cvn-app4 (*deprecated* Ubuntu 12 Precise, Size: Extra large)


  • 11:24 rxy: setup CVNBot23 in cvn-app4


  • 05:44 Krinkle: Edit CVNBot19 Projects.xml configuration: Correct projectName for to 'testwikidata.wikipedia' (per legacy wmgRC2UDPPrefix override)
  • 05:14 Krinkle: rsync'ed CVNBot3 from cvn-app5 to cvn-app4
  • 05:14 Krinkle: rsync'ed CVNBot2 from cvn-app5 to cvn-app4
  • 05:14 Krinkle: rsync'ed CVNBot19 from cvn-app5 to cvn-app4


  • 14:48 rxy: "sudo rm /var/lib/puppet/state/agent_catalog_run.lock" for fix to puppet failure.



  • 19:35 rxy: Setting up Cubbie in cvn-app5.


  • 23:57 Krinkle: Restore localised messages (nl) for CVNBot12


  • 17:00 Krinkle: Unscheduled reboot of cvn-app5. Bots are having trouble writing to local disk.


  • 19:03 Krinkle: Restarted cvn-app4. Somehow it became unreachable for over an hour.
  • 18:11 Krinkle: Promoted Rxy from member to projectadmin


  • 17:13 andrewbogott: rebooted cvn-app5 and cvn-app4 (not at the same time) because I suspect them of running amok
  • 15:42 andrewbogott: migrating cvn-app4 to labvirt1001 in an attempt to redistribute load a bit


  • 12:47 andrewbogott: migrating cvn-app4 and cvn-app5 to labvirt1005 and labvirt1006, respectively. Seems to be working now.
  • 03:00 Krinkle: Archived cvn-apache5 access logs to /data/project/cvn-common/backup/cvn-apache5-accesslogs.tar.gz
  • 02:54 Krinkle: now points to cvn-apache8 (UbuntuTrusty; Apache 2.4). The old cvn-apache5 (UbuntuPrecise; Apache 2.2) will be deleted shortly after archiving web logs to NFS.


  • 21:26 Krinkle: Apply role::labs::lvm::srv via Hiera
  • 21:01 andrewbogott: cold-migrating cvn-app4 to labvirt1005


  • 14:43 andrewbogott: moved cvn-apache5 to virt1007


  • 01:51 Krinkle: Disabled role::labs::lvm::mnt, and enabled role::labs::lvm::srv instead (separate changes)
  • 01:12 Krinkle: Enable role role::labs::lvm::mnt on all cvn instances


  • 18:30 Krinkle: CVNBot1 (feeds #cvn-wp-en; migrated from rcbot) is now hosted in Wikimedia Labs, runs on cvn-app4.


  • 04:12 Krinkle: Renamed KrinkleBot2 to CVNBot2 (assigned NickServ account CVN-Bots, shard0)
  • 03:50 Krinkle: CVNBot13 (feeds #cvn-wp-da; migrated from KyluBot) is now hosted in Wikimedia Labs, runs on cvn-app5.
  • 03:50 Krinkle: CVNBot12 (feeds #cvn-wp-nl; migrated from KrinkleBot7) is now hosted in Wikimedia Labs, runs on cvn-app4.
  • 03:50 Krinkle: Installed pidentd package on cnv-app servers.


  • 00:00 Krinkle: Migrated web services from pmtpa to eqiad. cvn-apache*.pmtpa.wmflabs can be deleted.


  • 19:11 Krinkle: Migrated all applications from pmtpa to eqiad. cvn-app*.pmtpa can be shut down.
  • 17:36 Krinkle: Installing packages on new eqiad instances (cvn-app4, cvn-app5, cvn-apache5)


  • 05:34 Krinkle: Finished migration of all bots from the old and small app server (cvn-app1) to the large cvn-app2. Instance to be decomissioned.
  • 05:26 Krinkle: Created cvn-app3 as the first cvn server in wmflabs-eqiad. Don't use for now (not yet set up with proper packages), but will replace cvn-app1 and cvn-app2 from pmtpa soon.


  • 20:45 Krinkle: Setting up cvn-app2 as app server (same packages as on cvn-app1)
  • 20:42 Krinkle: Deleted cvn-apache3, re-creating as cvn-apache4 with 'web' security group
  • 17:52 Krinkle: Setting up cvn-apache3 as web server


  • 23:34 Krinkle: Apache instances keeps going brain dead (time out, no significant spike in ganglia, just gradually going OOM). Rebooted again..




  • 05:16 Krinkle: Moving CVNBot14 from to cvn-app1.wmflabs


  • 17:29 Krinkle: Rebooted cvn-app1


  • 23:23 Krinkle: Added seahorseruler to cvn access group.


  • 14:03 matanya: rebooted by krinkle due to out of memory
  • 13:25 matanya: migrating Cluekie from Toolserver to labs


  • 21:40 Krinkle: CVNBot-nl and CVNBot-es were down because it was unable to open i18n files. Files restored from git, bots automatically started by stillalive.



  • 22:11 Krinkle: cvn-app1 went OOM and kernel failed because of it (/home and /data unavailable). stillalive has auto-started the bots and we're back up.


  • 00:19 Krinkle: Installing python-twisted-words and python-mysqldb from apt on cvn-app1 (for


  • 23:40 Krinkle: Installing mysql-server and mysql-client from apt on cvn-app1 (for
  • 23:06 Krinkle: Installing subversion from apt on cvn-app1 (for pywikipedia)


  • 14:04 Krinkle: gluster had another hickup, /data became unaccessible ("Transport endpoint is not connected"). Bots down for almost 24 hours. Rebooting the instance fixed it.


  • 16:55 Krinkle: Pool configuration pushed to github:
  • 16:54 Krinkle: Add cvn-app1 to application pool.
  • 16:12 Krinkle: Depooling cvn-apache1 from application
  • 16:11 Krinkle: Installing php5-cli and mono-complete on cvn-app1
  • 08:05 Krinkle: cvn-apache2 is getting too small for all the cvn apps (wasn't supposed to run them in the first place, just apache). Created instance cvn-app1 with more cpu/memory. Moving over application pool later today.


  • 13:02 Krinkle: Installing package "mono-complete"

Archived at Special:PrefixIndex/Nova_Resource:Cvn/SAL/.