Nova Resource:Bots/SAL

From Wikitech
Jump to navigation Jump to search

2018-09-26

2016-11-02

  • 18:27 Matthew_: Manually restarted wm-bot as it didn't come back after reboot.

2016-11-01

  • 21:17 Matthew_: Restarted wm-bot as it didn't come back on its own

2016-10-17

  • 17:04 Krenair: Restarted wm-bot instance, was not responding to HTTP or SSH

2016-07-07

  • 07:11 Matthew_: Restarted wm-bot, RSS feed error again

2016-07-06

  • 00:51 Matthew_: Re-enabled huggle-pg for wm-bot
  • 00:32 Matthew_: Restarted wm-bot, encountered a 500 error when reading an RSS feed and crashed.

2016-07-04

  • 10:43 yuvipanda: migrate wm-bot instance to labvirt1011 for T139264

2016-06-27

  • 16:14 Matthew_: Restarted wm-bot to fix file permission error

2016-02-21

  • 11:54 sDrewth: (re)started wm-bot

2015-11-02

  • 19:36 addshore: addshore@wm-bot:/mnt/share/wm-bot$ sudo ./easydepl.sh
  • 19:35 addshore: cd /mnt/share/wikimedia-bot && sudo git pull (46b014c..a623eae)

2015-08-19

  • 11:55 sDrewth: cleaned up and started wmbot

2015-07-15

  • 11:41 sDrewth: retarted wm-bot throwing errors and rc hosed

June 20

  • 13:01 sDrewth: restarted wm-bot not receiving recentchanges data

May 20

  • 07:27 petan: test

May 18

  • 09:38 petan: removed all backups for year 2014 in order to save some space (freed ~ 20GB) on /data/project storage

May 16

  • 04:31 sDrewth: changed admins config from admin to root for self

May 9

  • 05:48 sDrewth: restarted wm-bot

April 3

  • 18:54 mutante: restarting grrrit-wm for config change

March 31

  • 16:51 petan: cycling wm-bot

January 24

  • 05:07 sDrewth: restarted wm-bot services, unresponsive in irc

September 5

  • 09:41 petan: upgrading wm-bot to latest version, let's hope it's gonna work :o

September 4

  • 15:42 sDrewth: add user billinghurst to wm-bot/configuration/admins, fix case
  • 15:30 sDrewth: add user billinghurst to wm-bot/configuration/admins

August 31

  • 22:09 petan: ';DELETE FROM users WHERE username LIKE '%Damianz%'; COMMIT;
  • 00:22 jeremyb: booted wm-bot

August 18

  • 22:49 Thehelpfulone: Restarted instance as wm-bot as frozen for almost 24 hrs

July 14

  • 18:11 petan: hi

February 19

  • 08:25 andrewbogott: moved bots-sql to virt12 to free up space on virt6
  • 08:25 andrewbogott:

December 30

  • 14:58 jeremyb: booted wm-bot again

December 24

  • 05:20 jeremyb: booted bots-labs and started boncers/wm-bot back up again

July 24

  • 07:47 petan: deleting all application servers

July 22

  • 09:08 wm-bot: petrb: test

July 21

  • 16:05 petan: deleting -bnr1 -gs and -login

July 20

  • 15:03 petan: restarting wm-bot just to make sure it's ok

July 18

  • 12:01 addshore: installed nano on bots-labs :>

July 2

  • 11:57 petan: deleting exec node #2
  • 11:57 petan: deleting exec node #@

June 24

  • 15:15 petan: rebooting salebot instance

June 18

  • 15:59 wm-bot: petrb: scheduled bnr3 for removal

June 16

  • 15:25 petan: removing most of instances we dont use anymore (keeping only -login -gs -*sql* -bnr*

June 14

  • 13:44 wm-bot: petrb: created bots-master to replace -gs
  • 13:11 wm-bot: petrb: removing queue for ibnr host
  • 13:10 petan: deleting bots-ibnr1

June 11

  • 09:35 wm-bot: petrb: blabla
  • 09:29 wm-bot: petrb: this is a test ignore it
  • 09:25 wm-bot: petrb: test
  • 09:15 wm-bot: petrb: test

June 3

  • 13:51 wm-bot: petrb: killing gluster daemon on bots-bnr1 it eats 6gb of ram

May 28

  • 17:55 andrewbogott: rebooting bots-gs to fix /data/project mounting problem
  • 17:31 andrewbogott: rebooting bots-login in hopes of getting /data/project back on line

May 25

  • 08:10 wm-bot: petrb: installed subversion on login wheeeeeeeee

May 23

  • 12:33 wm-bot: petrb: rebooted -login just to fix gluster
  • 09:12 wm-bot: petrb: bots-4 is OOM one local bot seems to eat too many resources

May 22

  • 15:04 wm-bot: petrb: blah

May 18

  • 04:32 jeremyb: [wm-bot] booted wmib && bouncer @ bots-labs. (wasn't relaying my test edits)

May 8

  • 16:15 labs-logs-bottie: addshore: installing nano on -login

May 3

  • 15:20 labs-logs-bottie: petrb: created new instance bots-login that will replace bots-gs in order to make things look like in tools

April 28

  • 17:41 labs-logs-bottie: root: rebooting bots-gs to fix gluster outage

April 24

  • 18:43 labs-logs-bottie: petrb: installed fcron to -gs
  • 17:12 labs-logs-bottie: petrb: installing python-mysqldb everywhere
  • 10:29 labs-logs-bottie: addshore: added svn up to cron for both pywikipedia rewrite and trunk (on -gs)
  • 10:29 labs-logs-bottie: addshore: update svn of pywikipedia rewrite and trunk
  • 07:27 petan: syncing list with tools

April 19

  • 23:21 labs-logs-bottie: addshore: restarting -apache01 apache2 service

April 17

  • 09:13 labs-logs-bottie: petrb: deploying requests to /mnt/share/wmib/ on -1

April 16

  • 19:59 petanb|bnc-fu: added Jelte to bots and tools

April 15

  • 10:48 labs-logs-bottie: addshore: bots-gs installed python-twisted, sqlite, sqlit3. Updated jre
  • 09:08 labs-logs-bottie: petrb: restarted morebots
  • 08:05 labs-logs-bottie: petrb: restarted http to get new module

April 14

  • 18:16 labs-logs-bottie: root: bots-bsql01 is back, no idea what query was taking up the time
  • 17:59 labs-logs-bottie: addshore: bots-bsql01 seems to have ground to a halt waiting for some process...
  • 11:13 petan: changing apache conf and restarting

April 6

  • 09:25 labs-logs-bottie: petrb: disabled /data/project/public_html/dgideas/ideasbot/source because of problems with proprietary code

April 5

  • 13:58 labs-logs-bottie: petrb: install jobutils.deb

March 26

  • 08:47 labs-logs-bottie: petrb: deleted bots-3
  • 08:43 labs-logs-bottie: root: moving crontabs from bots-3 to bots-4
  • 08:41 labs-logs-bottie: petrb: creating -bnr4 to handle the queue backlog

March 21

  • 16:15 Coren: Added tools-login.wmflabs.org public IP for the tools-login instance and allowed incoming ssh to it.
  • 14:13 labs-logs-bottie: root: scrubbing fs on bsql01
  • 14:10 labs-logs-bottie: petrb: sudo apt-get install uuid-dev libattr1-dev zlib1g-dev libacl1-dev e2fslibs-dev libblkid-dev liblzo2-dev

March 20

  • 13:38 labs-logs-bottie: petrb: installing some basic and useful packages on master server
  • 08:17 labs-logs-bottie: root: killing swap on bsql01 and usin 10gb of that space for db files

March 19

  • 14:36 petan: recovery finished, db is back :))
  • 14:28 petan: started recovery of innodb.. will take some time
  • 14:26 labs-logs-bottie: root: restarting mysql server with new configuration
  • 14:23 petan: not that it would really worked :P
  • 14:23 petan: killing mysqld with -9 :|
  • 14:19 petan: turning off mysqld to fix ram problems and moving binary logs
  • 14:18 petan: mysql db ate 31gb of ram
  • 14:15 petan: -sql has some troubles - Damianz is responsible for sure :P
  • 12:57 labs-logs-bottie: petrb: previous message was supposed to be logged as me
  • 12:57 labs-logs-bottie: madman: deleting bots-2
  • 10:28 labs-logs-bottie: petrb: updated some packages on sql
  • 10:20 labs-logs-bottie: root: restart of mysql is needed to update configuration
  • 10:13 labs-logs-bottie: root: mysql is running fine though - just btrfsck is crashing
  • 10:13 labs-logs-bottie: root: houston filesystem /db is broken - new kernel is needed to fix it
  • 10:13 labs-logs-bottie: root: created /logs_db for binlogs
  • 10:03 labs-logs-bottie: root: running online fsck
  • 10:01 labs-logs-bottie: root: mysql: btrfs filesystem resize -10G /db

March 18

  • 21:33 petan: purged binary logs
  • 20:23 Damianz: bots-bsql01 / full, mysql dead with too many connections. Clearing root, restarting mysql
  • 13:41 petan: inserted scfc as member
  • 13:26 labs-logs-bottie: root: created /home/addshore/.forward
  • 13:20 labs-logs-bottie: petrb: everywhere iptables -A OUTPUT -p tcp --dport 25 -j REJECT
  • 10:54 labs-logs-bottie: petrb: found log files for master server in /var/spool/gridengine/qmaster yaylog removed exec from gs
  • 10:54 labs-logs-bottie: petrb: removed exec from gs

March 17

  • 17:52 labs-logs-bottie: petrb: somehow apache was installed to ibnr1 and no one logged it - next time log it and discuss before, deleting
  • 17:19 petan|wk: this will be very useful when debuggin problems with SGE
  • 17:19 petan|wk: replaced exim4 and setup local delivery so that mail now works on local system
  • 17:05 labs-logs-bottie: petrb: qdeploying postfix
  • 16:24 labs-logs-bottie: petrb: killing hang addshore processes from 1

March 16

  • 18:43 labs-logs-bottie: petrb: btw when you create these things log it please and document them
  • 18:43 labs-logs-bottie: petrb: fixed 1 security bug and 2 other bugs in scripts/mysql_backup.sh
  • 18:38 labs-logs-bottie: petrb: /data/project/ is a mess we need to clean it up
  • 18:06 labs-logs-bottie: petrb: +Jan to bots
  • 13:56 labs-logs-bottie: petrb: temporarily changed some parameters so that load get distributed better

March 15

  • 21:44 mutante: Fox_Wilson/voxelbot could you check for cronspam from python: can't open file '/data/project/voxelbot/VandalismInformation/bot.py': [Errno 2] No such file or directory .. thanks
  • 12:31 labs-logs-bottie: petrb: qdeploy python-irclib
  • 10:36 labs-logs-bottie: petrb: changed addshore's cron so that it does not affect minimal server
  • 09:54 labs-logs-bottie: petrb: created new queue for irc bots

March 13

  • 14:04 Coren: (Last log entry is a lie: wrong project)
  • 14:04 Coren: restarted webserver: relax AllowOverride options

March 12

  • 16:36 addshore: increase gid_range to 1000, bring avg loads down to 4 on all queues
  • 15:36 addhappy: reset some crazy load thresholds that were used back to more normal values
  • 15:26 addshore: increased gid_range by 100 to allow more simultaneous jobs
  • 12:09 petan: deleting -nr1
  • 12:03 petan: deleted bots-liwa
  • 09:12 petan: deleting all qw jobs of addshore from queu

March 11

  • 22:49 addshore: OG load formula to use short (1min avg) to enable faster submission after a recovery from high load
  • 21:20 addshore: OG check load every 10 seconds instead of 40 seconds, also alter load check to 15 from 10, load decay adjust from 7mins to 1min, load weigth from 1cpu to 0.7cpu 0.3mem
  • 21:00 addshore: Changed OG administrator_mail to 'addshore' to check spam
  • 19:12 labs-logs-bottie: petrb: changed email in /etc/gridengine/configuration
  • 16:22 labs-logs-bottie: petrb: disabling MTA on whole bots project to resolve spam
  • 12:16 labs-logs-bottie: petrb: giving root to beetstra on -liwa
  • 12:13 labs-logs-bottie: petrb: rebooting -liwa per request
  • 09:23 labs-logs-bottie: petrb: addshore to motd so that people know who to blame :)
  • 01:29 labs-logs-bottie: damian: pip install requests
  • 01:28 labs-logs-bottie: damian: apt-get remove python-requests -y
  • 00:20 labs-logs-bottie: damian: apt-get install python-requests python-virtualenv python-twisted

March 10

  • 23:27 labs-logs-bottie: addshore: rebooted bsql01 to solve lock issue and also to act as the previously needed reboot
  • 22:57 labs-logs-bottie: addshore: restarted bnr1
  • 12:00 labs-logs-bottie: petrb: install liblua5 to bnr2
  • 11:05 labs-logs-bottie: root: folder removed
  • 11:04 labs-logs-bottie: root: aborted reboot
  • 11:03 labs-logs-bottie: root: rebooting bots-4 to fix gluster reconnect
  • 11:00 labs-logs-bottie: root: bots removing DrTrigonBot per request on irc

March 9

  • 21:31 Damianz: rebooted bots-4 weird read only fs issues (for 15 folders) and has updates
  • 20:19 petan: created -bnr2 feel free to use, it though it's a part of pre-setup for grid... (grid of 1 box really suck)
  • 12:03 labs-logs-bottie: petrb: removed nr2 (no users were using it)
  • 08:43 Makecat: added DGideas

March 8

  • 15:34 petan: create new instance to play with grid scheduler

March 7

  • 11:22 petan: enforcing new security rules for sql and passwords storage
  • 07:00 Ryan_Lane: installed libmysqlclient-dev on bots-bnr1

March 6

  • 20:26 petan: added Platonides
  • 15:22 labs-logs-bottie: petrb: updated buffer size of sql server, was too big
  • 13:52 labs-logs-bottie: addshore: optimize bsql01 my.cnf to utilise more server resources
  • 03:51 labs-logs-bottie: addshore: legoktm user created on bsql01

March 5

  • 23:33 Ryan_Lane: rebooting bots-nr2, bots-analytics, and bots-abogott-devel
  • 19:54 Ryan_Lane: rebooting bots-apache01
  • 18:22 petan: given admin to addshore
  • 15:53 labs-logs-bottie: petrb: kill 5261, legoktm 5261 5242 81 11:39 pts/4 03:25:25 python xmlrevisionscanner.py
  • 15:32 petan: permanently removing bots-sql1

March 3

  • 22:49 labs-logs-bottie: petrb: installing mariadb on bsql01
  • 02:31 petan: by heavy I mean addshore :)))
  • 02:30 petan: creating sql4 for heavy queries

March 1

  • 11:52 Makecat: added Hercule, trusted user

February 28

  • 22:21 Ryan_Lane: added coren to project admins and sudo on all bots instances
  • 20:54 Vacation9: Added Riley to Bots

February 25

  • 02:17 Makecat: added Beta16

February 23

  • 23:07 labs-logs-bottie: danmichaelo: installed libyaml-dev on bots-3
  • 22:36 labs-logs-bottie: danmichaelo: apt-get install gettext
  • 09:56 legoktm: upgrading node, npm, nodejs-dev
  • 09:56 legoktm: ran apt-get update
  • 09:37 legoktm: installing npm, nodejs-dev on bots-3
  • 09:35 legoktm: installing nodejs on bots-3

February 14

  • 14:32 labs-logs-bottie: root: shutting down sql1 which is not being used to upgrade it to mariadb and convert fs
  • 14:24 labs-logs-bottie: addshore: fixed addbot process in TASK_UNINTERRUPTIBLE state
  • 14:21 labs-logs-bottie: petrb: addshore bot eats about 90% of ram on bnr1 0.o
  • 09:03 labs-logs-bottie: petrb: nr1 installing python-virtualenv
  • 09:03 labs-logs-bottie: petrb: nr1 only 100mb of free ram, needs fix
  • 00:07 labs-logs-bottie: danmichaelo: bots installed libxml2-dev, libxslt-dev on bots-3

February 13

  • 23:25 labs-logs-bottie: danmichaelo: installed exuberant-ctags on bots-3
  • 23:08 labs-logs-bottie: danmichaelo: sudo pip-2.7 install ipython on bots-3
  • 23:02 labs-logs-bottie: danmichaelo: apt-get install libfreetype6-dev, libpng12-dev on bots-3
  • 23:01 labs-logs-bottie: danmichaelo: --help
  • 22:19 danmichaelo: sudo apt-get install liblapack-dev on bots-3
  • 22:18 danmichaelo: sudo apt-get install libatlas-base-dev on bots-3
  • 22:11 danmichaelo: sudo apt-get install python2.7-dev on bots-3
  • 21:19 danmichaelo: sudo pip-2.7 install virtualenvwrapper on bots-3
  • 21:12 danmichaelo: sudo pip-2.7 install virtualenv on bots-3
  • 21:11 danmichaelo: sudo python2.7 get-pip.py on bots-3
  • 21:10 danmichaelo: sudo python2.7 distribute_setup.py on bots-3
  • 21:07 danmichaelo: sudo apt-get install python2.7 on bots-3
  • 21:06 danmichaelo: sudo add-apt-repository ppa:fkrull/deadsnakes on bots-3 (for python 2.7)

February 12

  • 18:30 Vacation9: added both Makecat and Geraki; trusted users
  • 12:44 labs-logs-bottie: petrb: bots-1 updating seen module in wm-bot

February 10

  • 23:44 Ryan_Lane: installed libevent-dev and python-dev on bots-bnr1
  • 23:39 Ryan_Lane: installed python-setuptools on bots-bnr1

February 9

  • 20:03 labs-logs-bottie: addshore: added ceradon to bastion and bots

February 5

  • 14:02 petan: +Darkdadaah
  • 09:08 petan: recovered bots-3

February 4

  • 11:35 labs-logs-bottie: root: reinstalling python-twisted on bots-bnr1
  • 11:05 labs-logs-bottie: root: rebooting bots-bnr1
  • 11:05 labs-logs-bottie: root: bots-bnr1 - process 14202 is stucked waiting for IO, unable to kill it, machine needs to be rebooted
  • 11:03 labs-logs-bottie: root: because of gluster failure on bots-bnr1 it's needed to kill all processes that access the fs, which is gifti 10091 f.c.m (gifti)java valhallasw 13912 ..c.. (valhallasw)bash valhallasw 14202 ..c.. (valhallasw)file killing them now
  • 10:54 labs-logs-bottie: petrb: on bnr1
  • 10:54 labs-logs-bottie: petrb: installing python-twisted and php5-curl

February 3

  • 21:28 labs-logs-bottie: addshore: rebooted bots-4

February 1

  • 23:32 rschen7754: installed python-twisted

January 28

  • 06:34 rschen7754: installed sqlite3 on bots-4
  • 06:34 rschen7754: installed sqlite on bots-4
  • 06:32 rschen7754: installed python-twisted on bots-4

January 27

  • 12:30 labs-logs-bottie: petrb: sql2 is running
  • 09:39 labs-logs-bottie: root: maintenance on bots-sql2 in 20 minutes
  • 09:33 petan: added rschen7754
  • 09:28 labs-logs-bottie: root: rm /data/project/testfile
  • 09:24 labs-logs-bottie: root: /data/project/testfile is -rw-r--r-- 1 root root 619511808 2013-01-24 22:24 testfile - wtf??
  • 09:24 labs-logs-bottie: root: sql2 upgraded system

January 25

  • 01:57 labs-logs-bottie: addshore: bots2 installing php5-curl
  • 01:56 labs-logs-bottie: addshore: bots3 installing php5-curl
  • 01:35 labs-logs-bottie: fwilson: Move VoxelBot to bots-3

January 24

  • 09:08 labs-logs-bottie: addshore: killing 5716 mono by eikes using 100% cpu

January 23

January 21

  • 23:05 JasonDC: ContinuityBot deployment server selected to be bots-nr2.

January 19

  • 17:48 addshore: addshore: bots-4 updating libfreetype6, libnspr4-0d, libnss3-1d
  • 17:47 labs-logs-bottie: addshore: updating libfreetype6, libnspr4-0d, libnss3-1d
  • 14:50 labs-logs-bottie: petrb: sql3 done
  • 13:10 labs-logs-bottie: root: sql3 will reboot approximately in 1h
  • 13:09 labs-logs-bottie: root: scheduling sql3 to reboot
  • 12:57 labs-logs-bottie: petrb: moving the data to gluster until I recreate filesystem
  • 12:46 labs-logs-bottie: petrb: disabling sql3 in order to change the device

January 18

  • 19:03 giftpflanze: russblau created /etc/cron.daily/pywikipedia on bots-4 to update /data/project/pywikipedia/{trunk,rewrite}
  • 18:24 russblau: Created shared Pywikipedia repositories at /data/project/pywikipedia/trunk and /data/project/pywikipedia/rewrite (you'll still need separate user-config files and so on for each bot)

January 16

  • 12:58 Vacation9: Now running VoxelBot on bots-4
  • 10:04 petan: inserting Vacation9 to project
  • 10:02 petan: inserting Fox Wilson to project

January 15

  • 22:32 labs-logs-bottie: petrb: the indians, of course
  • 22:32 labs-logs-bottie: petrb: changing the apache configs per hints from apache people
  • 16:50 labs-logs-bottie: petrb: inserting wikivoyage.org subdomains to wm-bot db

January 14

  • 09:50 labs-logs-bottie: petrb: updating some configs in apache to fix broken urls

January 12

  • 07:33 labs-logs-bottie: petrb: public_html is failing, changing to a+rx in a loop
  • 07:28 labs-logs-bottie: petrb: fixing public htm
  • 07:22 labs-logs-bottie: petrb: explain: d--------- 48 root root 8192 2013-01-12 07:12 public_html

January 10

  • 16:04 labs-logs-bottie: petrb: moving afc bot to bots-bnr1
  • 16:03 labs-logs-bottie: root: created 20gb swap on bots-bnr1 to avoid OOM crashes
  • 08:34 petan: testing

January 9

  • 16:10 Thehelpfulone: added Alchimista

January 8

  • 03:11 labs-logs-bottie: madman: Installed php-pear, HTTP_Request2, Log on bots-2.

January 7

  • 13:05 Beetstra: beetstra: linkwatcher is now running from bots-liwa
  • 13:05 Beetstra: beetstra: Installed perl modules POE, WWW::Mechanize, XML::Simple (forced) and POE::Component::IRC on bots-liwa
  • 11:38 Beetstra: beetstra: Starting to initialise bots-liwa for running LinkWatcher - installing perl modules

January 5

  • 17:14 labs-logs-bottie: petrb: created 2 new instances per irc log
  • 17:08 ireas: ireas: Installed qt4-qmake and libqt4-dev on bots-4.

January 4

  • 20:17 Ryan_Lane: installed exuberant-ctags on bots-4 for danmichaelo
  • 15:31 labs-logs-bottie: petrb: rebooting bots-3 which died for some reason
  • 15:29 labs-logs-bottie: petrb: restarted adminbot on -labs

December 30

  • 12:13 DrTrigon: DrTrigonBot rewrite migrated from TS to labs (with lua support)

December 29

  • 22:36 petan: bots-3 died OOM because of some python

December 28

  • 20:44 DrTrigon: installed liblua5.1-0-dev on bots-4

December 27

  • 13:54 DrTrigon: installed python-dev on bots-4
  • 10:23 DrTrigon: autoremoved libzmq1 on bots-4
  • 10:21 DrTrigon: installed python-irclib on bots-4

December 26

  • 22:19 Damianz: restarted bots-1 and bots-nr1 to switch over mounts+maybe fix auth for some people
  • 22:02 DrTrigon: removed python-numpy on bots-4 again - needs to be installed on bots-apache01 (for cgi)
  • 21:53 DrTrigon: installed python-numpy on bots-4 (for DrTrigonBot)

December 18

  • 20:09 giftpflanze: rebooted bots-4

December 14

  • 00:17 Ryan_Lane: rebooted bots-3

December 13

  • 09:23 labs-logs-bottie: root: nohup su beetstra -c perl xlinkbot.pl fails see logs
  • 09:18 labs-logs-bottie: root: started coibot as beetstra
  • 09:18 labs-logs-bottie: root: started xlinkbot as beetstra
  • 09:17 labs-logs-bottie: root: started unblockbot as beetstra
  • 09:14 labs-logs-bottie: petrb: restarted bottie
  • 09:14 labs-logs-bottie: petrb: if you are to remove stuff from motd, LOG it next time. thanks
  • 00:52 Ryan_Lane: installing python-twisted on bots-1

December 11

  • 21:44 andrewbogott: added a puppetized logbot for analytics. It runs on bots-analytics and logs to https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
  • 18:38 andrewbogott: puppetized labs-morebots via role::logbot::wikimedia-labs
  • 18:35 andrewbogott: testing
  • 18:13 andrewbogott: do you work over here?
  • 17:57 andrewbogott: taking down labs-morebots so that I can bring it back via puppet

December 10

  • 23:55 andrewbogott: I am testing this bot.

November 26

  • 04:14 jeremyb: [bots-labs] labs-morebots was gone after netsplit. booted
  • 04:14 jeremyb: [bots-1] wm-bot was gone after netsplit. booted bouncer && wmib

November 25

  • 05:56 jeremyb: bots-3 apparently beetstra's bots don't start on their own (no init script). started them all manually based on http://bots.wmflabs.org/~hydriz/minimanual.txt and root's bash history. they're running as beetstra's user
  • 05:36 jeremyb: bots-labs booted labs-morebots (init.d=adminbot)
  • 05:35 jeremyb: bots-3 rebooted from labsconsole. ganglia showed nearly 4 hrs unresponsive, couldn't connect myself, couldn't even view console log on labsconsole. (but console log was working for other instances)

November 9

  • 23:34 andrewbogott: Maybe sort of fixed the adminbot for the time being. Is this in source-control someplace?
  • 23:33 andrewbogott: This is my final log message for the day.
  • 23:32 andrewbogott: This is another test message
  • 23:28 andrewbogott: This is a test message

November 5

  • 14:19 labs-logs-bottie: petrb: sudo pip install requests
  • 13:54 petan: enabled +ExecCGI on apache for userdir's

November 2

  • 14:45 petan: booting bots-3
  • 14:44 petan: bots-3 dead OOM

November 1

  • 15:14 petan: apache server now has 4gb of ram, everything is in puppet except for public_html
  • 12:15 petan: expect web server outage
  • 12:15 petan: upgrading web server

October 30

  • 12:25 jeremyb: [bots-labs labs-morebots] booted again...
  • 12:24 jeremyb: [bots-1 wm-bot] fixed metawiki's info in ./sites && cleaned up the channels that were watching the original one. tested 2 channels and they both work!

October 29

  • 11:45 labs-logs-bottie: petrb: patch of wm-bot [critical]
  • 03:47 jeremyb: [bots-1 - wm-bot] booted wmib and that didn't fix anything, booted bouncer and then wmib again and that worked. must have been hung post netsplit

October 28

  • 12:52 legoktm: installed python-imaging on bots-3
  • 01:48 jeremyb: (timestamps are UTC of course)
  • 01:47 jeremyb: booted labs-morebots twice (it broke itself again in fairly short order after the first time) and then re!log'd the stuff it had missed in the interim from my own scrollback buffer

October 27

  • 04:03 legoktm: installed sqlite3 on bots-3
  • 04:01 legoktm: installed sqlite on bots-3

October 17

  • 15:42 petan: +huji

October 15

  • 14:28 labs-logs-bottie: petrb: rebooting nr1

October 12

  • 00:47 Ryan_Lane: added feature to labs-morebots to check users against a trust list; we need user profiles before we can realistically enable this.
  • 00:46 Ryan_Lane: updated labs-morebots to lower the cache from 30 minutes to 5 minutes

October 10

  • 11:33 labs-logs-bottie: petrb: patching wm-bot completely
  • 10:54 labs-logs-bottie: root: patched wm-bot
  • 09:44 labs-logs-bottie: petrb: rebooting nr1

October 6

  • 20:55 legoktm: Damianz installed git-core on bots-1
  • 19:39 Damianz: checkout of DrTrigonBot framework to /data/project/DrTrigonBot; svn co svn.toolserver.org/svnroot/drtrigon/ .
  • 14:51 Damianz: Added DrTrigon as a member
  • 14:05 Damianz: bots-3 restarted linkwatcher, coibot, xlinkbot & unblockbot under beetstra's user
  • 14:03 Damianz: bots-3 oom, killed beetstra's processes to get a gb of ram back

October 4

  • 22:53 Damianz: bots-sql2 is back from a long reboot

October 3

  • 16:32 giftpflanze: installed rlwrap on bots-4
  • 14:55 labs-logs-bottie: petrb: created on nr1, of course
  • 14:54 labs-logs-bottie: petrb: created public_html :P
  • 14:42 labs-logs-bottie: root: installing updates on nr1
  • 14:22 petan: adding Quentinv57
  • 10:42 legoktm: installed gcc on bots-3
  • 10:21 petan: giving rights to Mardetanha for the project

October 2

  • 23:51 legoktm: Installed python3-minimal, python-virtualenv, python-pyexiv2, git-core, git-svn, git-doc on bots-3
  • 12:39 labs-logs-bottie: petrb: rebooted wm-bot
  • 12:38 labs-logs-bottie: petrb: upgrading bot

October 1

  • 09:26 petan: waiting for someone from ops to let me create big instance...
  • 09:22 petan: creating new bots-sql2r
  • 09:21 petan: upgrading bots-sql2 to higher ram, going to take while
  • 08:37 Damianz: restarting mysql on bots-sql2 - going oom again

September 30

  • 13:13 giftpflanze: installed tclsh8.6 with tip-389-impl for full unicode support on bots-4

September 29

  • 02:30 mutante: adding new member cupco

September 28

  • 16:07 Damianz: Said script is github.com/DamianZaremba/labs-bots-vhost-builder/blob/master/update.py
  • 16:06 Damianz: Added script to bots-apache1's crontab to auto create member dirs in /data/project/public_html
  • 15:35 Hydriz: New instance bots-salebot created dedicated for Gribeco to run Salebot (an antivandal bot) for frwiki and ptwiki.
  • 15:34 Hydriz: Added Gribeco to the project. Created a public_html directory for him as well.

September 26

  • 21:16 Damianz: Ryan fixed acls, apache1 can now talk to sql again
  • 20:57 Damianz: bots-apache1 up again, issues with connections to sql at the moment. Everything else should be there, we need to puppetize this
  • 20:05 Damianz: deleting bots-apache1, stuff will be down until I install another instance
  • 09:17 petan: fixed bug in wm-bot
  • 01:03 Damianz: test
  • 00:51 mutante: added new member legoktm

September 25

  • 21:35 Damianz: TEST
  • 21:34 mutante: foobar
  • 18:17 mutante: apt-get dist-upgrade on bots-2
  • 18:16 Damianz: moved apache data from bots-nfs to /data/project/public_html so it's at least replicated
  • 18:16 Damianz: rm -rf /tmp/.s on apache1
  • 18:16 Damianz: Copied /tmp to /root/tmp on apache1
  • 18:16 Damianz: rm -rf /var/tmp/* on apache1
  • 18:16 Damianz: Killed 14654 on apache1
  • 18:16 Damianz: Copied /var/tmp to /root/var-tmp on apache1
  • 18:16 Damianz: Disabling phpmyadmin on apache1 due to new exploit
  • 18:16 Damianz: apache1 running crazy shit processes under www-data, has at least 3 exploits and 1 udp processes downloaded.

September 14

  • 10:59 labs-logs-bottie: petrb: performing a huge update of wm-bot, killing all core processes

September 13

  • 17:05 Damianz: fixed pupept hostname on bots-apache1

September 10

  • 16:30 labs-logs-bottie: petrb: let's try :D
  • 16:17 labs-logs-bottie: petrb: performing big update of wm-bot

August 28

  • 05:46 Ryan_Lane: live-hacked adminbot on bots-labs to match new DIT LDAP structure for keystone

August 21

  • 00:35 Damianz: bots-sql2 oom, rebooting

August 3

  • 19:12 andrewbogott: restarted 'Articles For Creation bot' on bots-1
  • 19:11 andrewbogott: restarted wm-bot (bouncer.exe and wmib.exe) on bots-1
  • 19:11 andrewbogott: restarted Log bot on bots-labs
  • 19:01 andrewbogott: migrated all VMs to new hardware

August 1

  • 23:30 Fastily: Installed jdk/jre 6 on bots-4

July 26

  • 00:21 labs-logs-bottie: gifti: Installed fcron on bots-4

July 24

  • 23:57 labs-logs-bottie: gifti: Installed tcl8.5, tclcurl, tcllib on bots-4

July 22

  • 09:58 Hydriz: Restarted morebots on bots-2, seems to have been down for a really long time.

July 2

  • 14:10 Damianz: chmod /mnt/public_html/damian/api.php to 000 on bots-apache1 - think the report/review sync for cbng is broken and looping, testing if this fixed the bw/spam issues.
  • 08:48 Hydriz: Rebooting bots-sql3 as we humans are born evil (aka leap second bug/high CPU)

June 20

  • 08:00 labs-logs-bottie: petrb: patching bot

June 18

  • 14:19 labs-logs-bottie: petrb: done
  • 14:18 labs-logs-bottie: petrb: patching bot

June 17

  • 17:09 labs-logs-bottie: petrb: patching bouncer
  • 10:45 labs-logs-bottie: petrb: installing new io cache for wmbot

June 16

  • 17:48 labs-logs-bottie: petrb: patching bot
  • 16:58 labs-logs-bottie: petrb: patching bot
  • 16:35 labs-logs-bottie: petrb: fixing RC of wmib
  • 16:04 labs-logs-bottie: wmib: wm-bot
  • 16:04 labs-logs-bottie: wmib: patching bot

June 14

  • 15:49 Ryan_Lane: moved adminbot to bots-labs

June 4

  • 20:50 labs-logs-bottie: wmib: inserting wikimania to bot config and reloading it
  • 20:42 labs-logs-bottie: wmib: updating wm-bot

May 30

  • 17:08 labs-logs-bottie: jeremyb: [bots-1,bots-nfs] did some IRC log redaction surgery. booted wm-bot (wmib) a couple times. (same way as before. kill bot; do surgery; kill sleep; didn't touch restart.sh) had some weird permissions issue that ended up causing #wikimedia-tech's log to lose messages. will restore those from my personal log later (probably after midnight so I don't have to do any more bot killing)
  • 08:54 labs-logs-bottie: wmib: done
  • 08:52 labs-logs-bottie: petrb: patching wm bot

May 29

  • 20:26 labs-logs-bottie: jeremyb: [bots-1] added hostmasks for krinkle and jeremyb to admin config. killed the mono proc and then killed the sleep. (didn't kill the restart.sh)
  • 13:06 labs-logs-bottie: petrb: patching wm-bot
  • 12:21 mutante: restarting labs-morebots on bots-2

May 25

  • 14:01 petan|wk: I want to be Mr. Obvious

May 24

  • 13:27 labs-logs-bottie: petrb: upgrading wm-bot

May 23

  • 16:04 Damianz: Ran mysqladmin flush-hosts on bots-sql2 as it was blocking cbng's report interface.

May 22

  • 15:49 labs-logs-bottie: wmib: restarting wm-bot
  • 15:29 Thehelpfulone: added Sven_Manguard a few days ago to the *project* :P
  • 15:25 Thehelpfulone: gave tanvir bots access, will be running interwiki bots
  • 15:19 labs-logs-bottie: petrb: patching wm-bot
  • 12:17 mutante: running puppet on bots-4
  • 04:11 hashar: bots-apache1 has two defunct processes eating CPU: pdflushsh (pid 6382) and 10 (pid 6278)

May 21

  • 21:11 hashar: restarted labs-morebot : root@bots-2:~# service adminbot restart
  • 02:28 jeremyb: [bots-2] should find out what prod uses
  • 02:28 jeremyb: [bots-2] could use some lockfiles... either in wrapper or in python itself
  • 02:27 jeremyb: [bots-2] then investigated further (after the restart) and it turns out there were 3 adminlogbot.py procs (including the new one that had just been started). the other 2 were from May 9 and May 12. killed them all and started again from scratch
  • 02:24 jeremyb: [bots-2] labs-morebots was running but not working. $ sudo service adminbot status; * logslogbot is running; $ sudo service adminbot restart; * Restarting IRC Logging bot for WMF labs logslogbot; ...done.

May 18

  • 10:42 mutante: restarted labs-morebots on bots-2
  • 10:21 mutante: restarting labs-morebots on bots-2

May 16

  • 23:36 Damianz: Installed default-jre on bots-3 for svenmanguard's bot

May 13

  • 18:39 labs-logs-bottie: petrb: patching wm-bot

May 9

  • 12:01 Hydriz: Restarted morebots due to some connection issue on Freenode servers. log-bottie remains down due to sudo policy (If only I had sudo access...)

May 7

  • 09:33 Thehelpfulone: made a little tweak to content.html

May 4

  • 10:40 mutante: killed duplicate adminbot procs, restarted one
  • 10:34 mutante: restarted labs-morebots on bots-2

May 1

  • 11:02 Hydriz: Restarted morebots, was offline due to network-wide restart of virtual machine hosts.

April 30

  • 14:16 labs-logs-bottie: petrb: this is a message I just logged

April 29

  • 11:21 Beetstra: synchronizing databases 'coibot' and 'linkwatcher' from bots-sql3 to bots-sql2, adapting the bot-code to store/query on bots-sql2, will restart bots after transfer is complete

April 20

  • 22:55 Ryan_Lane: added Kaldari to bots

March 15

  • 20:12 petan: added DeltaQuad to bots

March 11

  • 20:24 Damianz: Added addshore as a member.

March 6

  • 15:35 labs-sexy-bottie: petrb: this is a test :o
  • 15:20 labs-sexy-bottie: petrb: ggsgsh

March 5

  • 14:12 Hydriz: Restarted as service user now...
  • 14:07 petan|wk: restart logbot

March 4

  • 12:39 Hydriz: Fixed logbot. People: Please start logbot with sudo python adminlogbot.py now...
  • 12:38 Hydriz: Running with sudo now...
  • 08:24 Damianz: Restarted morebots

March 2

  • 20:45 Damianz: Also crated /var/run/adminbot which was breaking the bot for starting - owner please fix
  • 20:44 Damianz: restarted bots-2

March 1

  • 14:25 petan|wk: fixed you, logbot

February 24

  • 22:32 Ryan_Lane: added a bots-labs instance for running bots meant for labs

February 23

  • 19:05 Ryan_Lane: deleting bots-5, it was stuck in a pending state thanks to the scheduler change
  • 16:47 petan|wk: created a bots-5 instance
  • 04:17 jeremyb: UTC: 22 08:14:20 -!- labs-morebots [~labs-more@208.80.153.192] has quit [Ping timeout: 252 seconds]
  • 04:14 Krinkle: Redesigned front-page at http://bots.wmflabs.org/
  • 04:14 jeremyb: bots-2:~$ sudo service adminbot restart

February 13

  • 16:06 petan|wk: moved nohup from beetstra's home to /mnt/share on bots-1

February 5

  • 16:18 Damianz: bots-3 lowmem is due to LinkSaver.pl + Parser.pl running under beetstra

January 27

  • 12:33 Beetstra: Moving also XLinkBot to bots-3. LiWa3 slows down reverting on XLinkBot too much.

January 25

  • 19:56 Beetstra: moved unblockbot from bots-2 to bots-3
  • 10:28 Hydriz: Fix nagios issue of bots-4 on SSL handshake by enabling 10.4.0.34 as allowed host in /etc/nagios/nrpe_local.cfg
  • 03:41 methecooldude: (bots-cb) Restarting ClueBot 3... how much RAM do you need!
  • 03:22 methecooldude: Update packages on all bots instances (excluding apache1 which was done on the 23rd)

January 24

  • 16:20 petan|wk: Adding hydriz to project

January 23

  • 16:02 methecooldude: Rebooting apache1
  • 16:01 methecooldude: Updated packages on apache1 including new kernel from apt

January 17

  • 00:01 petan: rebooting bots-4
  • 00:01 petan: installed all standart libs on bots-4 + patched kernel

January 16

  • 23:51 petan: new vm for dozen of bots of Krinkle created :o
  • 12:53 Beetstra: installed libcarp-always-perl
  • 12:52 Beetstra: installed libxml-perl
  • 12:51 Beetstra: installed liburi-perl
  • 12:51 Beetstra: installed libhtml-entities-numbered-perl

January 15

  • 17:19 Beetstra: running COIBot from bots-3
  • 17:17 Beetstra: installing libdbd-mysql-perl
  • 17:17 Beetstra: installing libdbd-mysql
  • 17:16 Beetstra: installing libpoe-component-irc-perl
  • 17:15 Beetstra: installing libpoe-perl
  • 16:51 Beetstra: upgraded installed perl packages on bots-2
  • 16:38 Coren: wp:en:CorenSearchBot now running on bots-3
  • 16:27 Beetstra: POE install failed via CPAN, JSON installed - JSON::XS failed, Net::OAuth fails
  • 16:27 Coren: installing libjson-perl on bots-3
  • 16:25 Coren: installing libnet-oauth-perl on bots-3
  • 16:05 Beetstra: installing perl module POE
  • 12:49 Beetstra: COIBot started with limited modules

January 14

  • 19:27 Beetstra: started XLinkBot on bots-2 - utf8-encoding seems fixed
  • 19:27 Beetstra: started LiWa3 on bots-2 - parsed revid parsing

January 13

  • 19:27 Beetstra: Bots writing to Wiki disabled due to utf8 problems in perl
  • 19:12 petan: created new vm for bots
  • 19:09 petan: added Beetstra to project
  • 19:07 petan: added coren to project
  • 08:42 Beetstra: Installed XLinkBot on bots-2, running properly

January 12

  • 20:18 Beetstra: Installed perl modules: POE::Component::IRC::State, WWW::Mechanize, XML::Simple, DBI
  • 20:04 Beetstra: updated CPAN, POE installed
  • 19:55 Beetstra: 2nd attempt installing perl modules - installing POE
  • 18:02 Beetstra: installation of POE on own account .. failed
  • 17:08 Beetstra: installing perl module POE (+needed modules) for beetstra
  • 16:24 Beetstra: connected to bots-2 for COIBot, LiWa3, XLinkBot
  • 09:39 methecooldude: Given access to Beetstra

January 4

  • 00:08 petan: deleted irc2
  • 00:05 methecooldude: LDAP issue on bots-irc2 > "init: nss-ldap: do_open: do_start_tls failed:stat=-1" and "init: nss_ldap: could not search LDAP server - Server is unavailable"

January 3

  • 23:30 petan: deleted irc1

January 1

  • 21:26 petan: added jeremyb to nfs

December 19

  • 20:20 Damianz: Move cluebot3 to the /mnt/share/cluebot/cluebot3 dir + added a new process group to supervisor for it on bots-cb.
  • 12:15 methecooldude: Packages update which includes a new kernel

December 14

  • 13:22 petan|wk: installed bots-sql3 db server

December 12

  • 20:05 petan: reinstalled bots-sql1 it was totaly broken :O
  • 20:02 petan: installed mysql on sql1
  • 19:59 petan: created new instance bots-sql3 for mariadb

December 10

  • 20:13 petan: fixed permission of /home/*
  • 19:53 petan: reconfigured apache shared directories are now on nfs
  • 19:23 petan: created nfs
  • 00:27 Damianz: Damianz installed php5-cli on bots-apache1
  • 00:05 petan: configured apache, back again :)

December 9

  • 23:49 petan: created apache1 again hopefully ok
  • 23:46 Damianz: Installed MediaWiki::API on bots-cb
  • 23:45 petan: killed apache1
  • 22:43 Damianz: Damianz symlinked /mnt/share/cluebot/cluebotng/{api,reviewinterface} to ~damian/public_html/cluebotng-{api,report} on bots-cb
  • 21:22 Damianz: Damianz Installed php5 php5-curl php5-cli php5-mysql on bots-cb
  • 21:11 Damianz: Damianz Added user 'cluebot' on bots-cb with the home dir /mnt/share/cluebot/
  • 21:09 petan: Damianz installed supervisor on bots-cb
  • 20:52 petan: rebooted apache1 for updates to take effect
  • 20:21 petan: new instance bots-sql2 for mysql (80gb storage)
  • 19:45 petan: deployed some more libraries for sql on bots-cb
  • 15:14 petan|working: created /mnt/share on all servers in bots cluster
  • 15:08 petan|working: sql is back up data are in /mnt/data
  • 14:44 petan|working: disabled on sql1 in order to finish data move
  • 14:26 petan|working: moved sql data files to /mnt
  • 11:00 petan|working: Petrb packaged logbot
  • 03:02 methecooldude: Updated ldconfig to reflect package changes

December 8

  • 23:55 hyperon: morebots packaged successfully
  • 23:26 Ryan_Lane: test
  • 23:18 hyperon: taking down morebot again for testing :)
  • 23:15 hyperon: installed adminbot on bots-1, i know what i did wrong
  • 23:13 hyperon: i think you like me better
  • 23:12 hyperon: test
  • 22:57 hyperon: bringing adminbot down for testing in a minute...
  • 21:04 petan: deployed bunch of libraries for cluebot
  • 08:53 petan|w: reinstall done
  • 08:36 petan|w: reinstalling bots-1
  • 08:29 petan|w: killing bots-1 to reinstall to lucid
  • 08:12 Ryan_Lane: killing the log bot for testing

December 7

  • 13:41 petan|wk: Petrb deployed libconfig-dev to bots-cb
  • 07:03 Ryan_Lane: bumping bots project to the top of the list
  • 06:50 Ryan_Lane: test
  • 03:50 Ryan_Lane: Adding categorization of the SALs
  • 03:12 hyperon: another test
  • 03:02 Ryan_Lane: made adminbot more configurable. moved config to /etc/adminbot/config.py, made identica and projects optional
  • 01:54 Ryan_Lane: enabled labs-morebots
  • 01:54 hyperon: enabled mod_userdir on bots-apache1 settings UserDir disabled; UserDir enabled hyperon petrb; Directory /home/*/public_html

December 6

  • 11:04 hyperon: test