00:15 YuviPanda: killed all db and tools-webproxy aliases in /etc/hosts for tools-webproxy, since otherwise puppet fails because ec2id thinks we’re not in labs because hostname -d is empty because we set /etc/hosts to resolve IP directly to tools-webproxy
21:11 YuviPanda: disable puppet on tools-dev to check shinken
21:00 scfc_de: qmod -cq continuous@tools-exec-09,continuous@tools-exec-11,continuous@tools-exec-13,continuous@tools-exec-14,mailq@tools-exec-09,mailq@tools-exec-11,mailq@tools-exec-13,mailq@tools-exec-14,task@tools-exec-06,task@tools-exec-09,task@tools-exec-11,task@tools-exec-13,task@tools-exec-14,task@tools-exec-15,webgrid-lighttpd@tools-webgrid-01,webgrid-lighttpd@tools-webgrid-02,webgrid-lighttpd@tools-webgrid-04 (fallout from /var being full).
20:38 YuviPanda: didn't actually stop puppet, need more patches
20:38 YuviPanda: stopping puppet on tools-dev to test shinken
22:07 StupidPanda: enabled puppet on tools-exec-07
21:47 StupidPanda: removed coredumps from tools-webgrid-04 to reclaim space
21:45 StupidPanda: removed coredump from tools-webgrid-01 to reclaim space
20:31 YuviPanda: disabling puppet on tools-exec-07 to test shinken
November 7
13:56 scfc_de: tools-submit, tools-webgrid-04: rm -f /var/log/exim4/paniclog (OOM around the time of the filesystem outage).
November 6
13:21 scfc_de: tools-dev: Gzipped /var/log/account/pacct.0 (804111872 bytes); looks like root had his own bigbrother instance running on tools-dev (multiple invocations of webservice per second).
November 5
19:15 mutante: exec nodes have p7zip-full now
10:07 YuviPanda: cleaned out pacct and atop logs on tools-login
November 4
19:50 mutante: - apt-get clean on tools-login, and gzipped some logs
November 1
12:51 scfc_de: Removed log files in /var/log/diamond older than five weeks (pdsh -f 1 -g tools sudo find /var/log/diamond -type f -mtime +35 -ls -delete).
October 30
14:37 YuviPanda: cleaned out pacct and atop logs on tools-dev
06:18 paravoid: killed a "vi" process belonging to user icelabs and running for two days saturating the I/O network bandwidth, and rm'ed a 3.5T(!) .final_mg.txt.swp
October 27
16:06 scfc_de: tools-mail: Killed -HUP old queue runners and restarted exim4; probably the source of paniclog's "re-exec of exim (/usr/sbin/exim4) with -Mc failed: No such file or directory".
15:36 scfc_de: tools-exec-07, tools-exec-14, tools-exec-15: Recreated (empty) /var/log/apache2 and /var/log/upstart.
October 26
12:35 scfc_de: tools-exec-07, tools-exec-14, tools-exec-15: Created /var/log/account.
12:33 scfc_de: tools-trusty: Went through shadowed /var and rebooted.
12:31 scfc_de: tools-exec-07, tools-exec-14, tools-exec-15: Created /var/log/exim4, started exim4 and ran queue.
October 24
20:31 andrewbogott: moved tools-exec-12, tools-shadow and tools-mail to virt1006
23:22 YuviPanda|zzz: removed stale puppet lockfile and ran puppet manually on tools-exec-07
October 11
15:31 andrewbogott: rebooting tools-master, stab in the dark
06:01 YuviPanda: restarted gridengine-master on tools-master
October 4
18:31 scfc_de: tools-mail: Deleted /usr/local/bin/collect_exim_stats_via_gmetric and root's crontab; clean-up for Ic9e0b5bb36931aacfb9128cfa5d24678c263886b
October 2
17:59 andrewbogott: added Ryan back to tools admins because that turned out to not have anything to do with the bounce messages
17:32 andrewbogott: removing ryan lane from tools admins, because his email in ldap is defunct and I get bounces every time something goes wrong in tools
September 28
14:45 andrewbogott: rebased /var/lib/git/operations/puppet on toolsbeta-puppetmaster3
September 25
14:43 YuviPanda: cleaned up ghost /var/log (from before biglogs mount) that was taking up space, /var space situation better now
September 17
21:40 andrewbogott: caused a brief auth outage while messing with codfw ldap
September 15
11:00 YuviPanda: tested CPU monitoring on tools-exec-12 by running stress, seems to work
September 13
20:52 yuvipanda: cleaned out rotated log files on tools-webproxy
September 12
21:54 jeremyb: [morebots] booted all bots, reverted to using systemwide (.deb) codebase
20:05 scfc_de: Deployed release 1.0.11 of jobutils and miscutils
August 15
16:45 legoktm: fixed grrrit-wm
16:36 legoktm: restarting grrrit-wm
August 14
22:36 scfc_de: Removed again jobs in error state due to LDAP with "for JOBID in $(qstat -u \* | sed -ne 's/^\([0-9]\+\) .*Eqw.*$/\1/p;'); do if qstat -j "$JOBID" | fgrep -q "can't get password entry for user"; then qdel "$JOBID"; fi; done"; cf. also bug #69529
August 12
03:32 scfc_de: tools-exec-08, tools-exec-wmt, tools-webgrid-02, tools-webgrid-03, tools-webgrid-04: Removed stale "apt-get update" processes to get Puppet working again
August 2
16:39 scfc_de: tools.mybot's crontab uses qsub without -M, added that as a temporary measure and will inform user later
16:36 scfc_de: Manually rerouted mails for tools.mybot@tools-submit.eqiad.wmflabs
August 1
22:41 scfc_de: Deleted all jobs in "E" state that were caused by an LDAP failure at ~ 2014-07-30 07:00Z ("can't get password entry for user [...]")
July 24
20:53 scfc_de: Set SGE "mailer" parameter again for bug #61160
14:51 scfc_de: Removed ignored file /etc/apt/preferences.d/puppet_base_2.7 on all hosts
July 21
18:39 scfc_de: Removed stale Redis entries for currentevents, misc2svg, osm4wiki, wp-signpost, wscredits and yadfa
18:38 scfc_de: Restarted webservice for stewardbots because it wasn't in Redis
18:33 scfc_de: Stopped eight (!) webservices of tools.bookmanagerv2 and started one again
July 18
14:29 scfc_de: admin: Set up .bigbrotherrc for toolhistory
13:24 scfc_de: Made tools-webgrid-04 a grid submit host
12:58 scfc_de: Made tools-webgrid-03 a grid submit host
16:00 YuviPanda: manually removed mariadb remote repo from tools-exec-12 instance, won't be added to new instances (puppet patch was merged)
01:33 YuviPanda|zzz: tools-exec-11 and tools-exec-13 have been added to the @general hostgroup
July 9
23:14 YuviPanda: applied execnode, hba and biglogs to tools-exec-11 and tools-exec-13
23:09 YuviPanda: created tools-exec-13 with precise
23:08 YuviPanda: created tools-exec-12 as trusty by accident, will keep on standby for testing
23:07 YuviPanda: created tools-exec-12
23:06 YuviPanda: created tools-exec-11
19:23 scfc_de: tools-webproxy: "iptables -A INPUT -p tcp \! --source 127/8 --dport 6379 -j REJECT" to block connections from other Tools instances to Redis again
14:12 scfc_de: tools-exec-cyberbot: Reran Puppet successfully and hotfixed the Peachy temporary file issue; will mail labs-l later
16:59 Betacommand: Coren: It may take a while though; what the catscan queries was blocking is a DDL query changing the schema and that pauses replication.
16:58 Betacommand: Coren: transactions over 30ks killed; the DB should start catching up soon.
14:37 Betacommand: replication for enwiki is halted current lag is at 9876
July 2
00:21 YuviPanda: restarted diamond on almost all nodes to stop sending nfs stats, some still need to be flushed
00:21 YuviPanda: restarted diamond on all exec nodes to stop sending nfs stats
July 1
23:09 legoktm: tools-pywikibot started the webservice, don't know why it wasn't running
21:08 scfc_de: Reset queues in error state again
17:51 YuviPanda: tools-exec-04 removed stale pid file and force puppet run
16:07 YuviPanda: applied biglogs to tools-exec-02 and rejigged things
15:54 YuviPanda: tools-exec-02 removed stale puppet pid file, forcing run
15:51 Coren: adjusted resource limits for -exec-07 to match the smaller instance size.
15:50 Coren: created logfile disk for -exec-07 by hand (smaller instance)
01:53 YuviPanda: tools-exec-10 applied biglogs, moved logs around, killed some old diamond logs
01:41 YuviPanda: tools-exec-03 restarted diamond, atop, exim4, ssh to pick up new log partition
01:40 YuviPanda: tools-exec-03 applied biglogs, moved logs around, killed some old diamond logs
22:06 YuviPanda: stale lockfile in tools-login as well, removing and forcing puppet run
22:01 YuviPanda: removed stale lockfile for puppet, forcing run
19:58 YuviPanda|food: added tools-webgrid-04 to webgrid queue, had to start portgranter manually
17:43 YuviPanda: created tools-webgrid-04, applying webnode role and running puppet
17:27 YuviPanda: created tools-webgrid-03 and added it to the queue
June 29
19:45 scfc_de: magnustools: "webservice start"
18:24 YuviPanda: rebooted tools-webgrid-02. Could not ssh, was dead
June 28
21:07 YuviPanda: removed alias for tools-webproxy and tools.wmflabs.org from /etc/hosts on tools-webproxy
June 21
20:09 scfc_de: Created tool mediawiki-mirror (yuvipanda + Nemo_bis) and chown'ed & chmod o-w /shared/mediawiki
June 20
21:01 scfc_de: tools-webgrid-tomcat: Added to submit host list with "qconf -as" for bug #66882
14:47 scfc_de: Restarted webservice for mono; cf. bug #64219
June 16
23:50 scfc_de: Shut down diamond services and removed log files on all hosts
June 15
17:12 YuviPanda: deleted tools-mongo. MongoDB pre-allocates db files, and so allocating one db to every tool fills up the disk *really* quickly, even with 0 data. Their non preallocating version is 'not meant for production', so putting on hold for now
16:48 scfc_de: tools-exec-cyberbot: No DNS entry (again)
June 13
22:59 YuviPanda: "sudo -u ineditable -s" to force creation of homedir, since the user was unable to login before. /var/log/auth.log had no record of their attempts, but now seems to work. straange
June 10
21:51 scfc_de: Restarted diamond service on all Tools hosts to actually free the disk space :-)
21:36 scfc_de: Deleted /var/log/diamond/diamond.log on all Tools hosts to free up space on /var
June 3
17:50 Betacommand: Brief network outage. source: It's not clearly determined yet; we aborted the investigation to rollback and restore service. As far as we can tell, there is something subtly wrong with the switch configuration of LACP.
June 2
20:15 YuviPanda: create instance tools-trusty-test to test nginx proxy on trusty
19:00 scfc_de: zoomviewer: Set TMPDIR to /data/project/zoomviewer/var/tmp and ./webwatcher.sh; cannot see *any* temporary files being created anywhere, though. iipsrv.fcgi however has TMPDIR set as planned.
May 27
18:49 wm-bot: petrb: temporarily hardcoding tools-exec-cyberbot to /etc/hosts so that host resolution works
10:36 scfc_de: tools-webgrid-01: removed all files of tools.zoomviewer in /tmp
10:22 scfc_de: tools-webgrid-01: /tmp was full, removed files of tools.zoomviewer older than five days
07:52 wm-bot: petrb: restarted webservice of tool admin in order to purge that huge access.log
May 25
14:27 scfc_de: tools-mail: "rm -f /var/log/exim4/paniclog" to leave only relay_domains errors
May 23
14:14 andrewbogott: rebooting tools-webproxy so that services start logging again
14:10 andrewbogott: applying role::labs::lvm::biglogs on tools-webproxy because /var/log was full and causing errors
May 22
02:45 scfc_de: tools-mail: Enabled role::labs::lvm::biglogs, moved data around & rebooted.
02:36 scfc_de: tools-mail: Removed all jsub notifications from hazard-bot from queue.
01:36 scfc_de: tools-mail: Freezing all messages to Yahoo!: "421 4.7.1 [TS03] All messages from 208.80.155.162 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html"
01:12 scfc_de: tools-mail: /var is full
May 20
18:34 YuviPanda: back to homerolled nginx 1.5 on proxy, newer versions causing too many issues
16:31 scfc_de: tools-webproxy: "iptables -A INPUT -p tcp \! --source 127/8 --dport 6379 -j REJECT" to block connections from other Tools instances to Redis
18:36 scfc_de: Pointed tools-dev.wmflabs.org at tools-dev.eqiad.wmflabs; cf. [[Bugzilla:62883]]
March 5
13:57 wm-bot: petrb: test
March 4
22:35 wm-bot: petrb: uninstalling it from -login too
22:32 wm-bot: petrb: uninstalling apache2 from tools-dev it has nothing to do there
March 3
19:20 wm-bot: petrb: shutting down almost all services on webserver-02 in order to make system useable and finish upgrade
19:17 wm-bot: petrb: upgrading all packages on webserver-02
19:15 petan: rebooting webserver-01 which is totally dead
19:07 wm-bot: petrb: restarting apache on webserver-02 it complains about OOM but the server has more than 1.5g memory free
19:03 wm-bot: petrb: switched local-svg-map-maker to webserver-02 because 01 is not accessible to me, hence I can't debug that
16:44 scfc_de: tools-webserver-03: Apache was swamped by request for /guc. "webservice start" for that, and pkill -HUP -u local-guc.
12:54 scfc_de: tools-webserver-02: Rebooted, apache2/error.log told of OOM, though more than 1G free memory.
12:50 scfc_de: tools-webserver-03: Rebooted, scripts were timing out
12:42 scfc_de: tools-webproxy: Rebooted; wasn't accessible by ssh.
March 1
03:42 Coren: disabled puppet in pmtpa tool labs\
February 28
14:46 wm-bot: petrb: extending /usr on tools-dev by 800mb
00:26 scfc_de: tools-webserver-02: Rebooted; inaccessible via ssh, http said "500 Internal Server Error"
February 27
15:28 scfc_de: chmod g-w ~fsainsbu/.forward
February 25
22:48 rdwrer: Lol, so, something happened with grrrit-wm earlier and nobody logged any of it. It was yoyoing, Yuvi killed it, then aude did something and now it's back.
February 23
20:46 scfc_de: morebots: labs HUPped to reconnect to IRC
February 21
17:32 scfc_de: tools-dev: mount -t nfs -o nfsvers=3,ro labstore1.pmtpa.wmnet:/publicdata-project /public/datasets; automount seems to have been stuck
15:24 scfc_de: tools-webserver-03: Rebooted, wasn't accessible by ssh and apparently no access to /public/datasets either
10:42 scfc_de: tools-mail: rm -f /var/log/exim4/paniclog ("User 0 set for local_delivery transport is on the never_users list", cf. [[bugzilla:61583]])
10:28 scfc_de: Reset error status of task@tools-exec-09 ("can't get password entry for user 'local-voxelbot'"); "getent passwd local-voxelbot" works on tools-exec-09, possibly a glitch
February 19
20:21 scfc_de: morebots: Set "enable_twitter=False" in confs/labs-logbot.py and restarted labs-morebots
19:14 scfc_de: tools-login: Disabled crontab and pkill -HUP -u fatemi127
February 18
11:42 scfc_de: tools-mail: Rerouted queued mail (@tools-login.pmtpa.wmflabs => @tools.wmflabs.org)
11:34 scfc_de: tools-exec-08: Rebooted due to not responding on ssh and SGE
10:39 scfc_de: tools-mail: rm -f /var/log/exim4/paniclog ("User 0 set for local_delivery transport is on the never_users list" => probably artifacts from Coren's LDAP changes)
07:02 andrewbogott: merged a lint patch to the gridengine module. Should be a noop
January 16
17:11 scfc_de: tools-exec-09: "iptables-restore /data/project/.system/iptables.conf" after reboot
January 15
13:36 scfc_de: After reboot of tools-exec-09, all continuous jobs were successfully restarted ("Rr"); task jobs (1974113, 2188472) failed ("19 : before writing exit_status")
07:38 legoktm: restarting grrit-wm, for some reason it reconnected and lost its cloak
December 23
18:30 marktraceur: restart grrrit-wm for subbu
December 21
06:50 scfc_de: tools-exec-01: Commented out duplicate MariaDB entries in /etc/apt/sources.list and re-ran apt-get update
December 19
17:22 marktraceur: deploying grrrit config change
December 17
23:19 legoktm: rebooted grrrit-wm with new config stuffs
December 14
18:13 marktraceur: restarting grrrit-wm to fix its nickname
13:17 scfc_de: tools-exec-08: Purged packages libapache2-mod-suphp and suphp-common (probably remnants from when the host was misconfigured as a webserver)
13:09 scfc_de: tools-dev, tools-login, tools-mail, tools-webserver-01, tools-webserver-02: rm /var/log/exim4/paniclog (mostly out of memory errors)
December 4
22:15 Coren: tools-exec-01 rebooted to fix the autofs issue; will return to rotation shortly.
16:33 Coren: rebooting webproxy with new kernel settings to help against the DDOS
December 1
14:05 Coren: underlying virtualization hardware rebooted; tools-master and friends coming back up.
November 25
21:03 YuviPanda: created tools-proxy-test instance to play around with the dynamicproxy
12:16 wm-bot: petrb: deswapping -login (swapoff -a && swapon -a)
November 24
07:19 paravoid: disabled crontab for user avocato on tools-login, see above
07:17 paravoid: pkill -u avocato on tools-login, multiple /home/avocato/pywikipedia/redirect.py DoSing the bastion
November 14
09:12 ori-l: Added aude to lolrrit-wm maintainers group
November 13
22:36 andrewbogott: removed 'imagescaler' class from tools-login because that class hasn't existed for a year. Which, a year ago is before that instance even existed so what the heck?
November 3
16:49 ori-l: grrrit-wm stopped receiving events. restarted it; didn't help. then restarted gerrit-to-redis, which seems to have fixed it.
November 1
16:11 wm-bot: petrb: restarted terminator daemon on -login to sort out memory issues caused by heavy mysql client by elbransco
October 23
15:19 Coren: deleted tools-tyrant and tools-exec-cyberbot (cleanup of obsoleted instances)
October 20
18:52 wm-bot: petrb: everything looks better
18:51 wm-bot: petrb: restarting apache server on tools-webproxy
18:49 wm-bot: petrb: installed links on -dev and going to investigate what is wrong with apaches, documentation, Coren, please update it
October 15
21:03 Coren: labs-login rebooted to fix the ownership/take issue with success.
October 10
09:49 addshore: tools-webserver-01is getting a 500 Internal Server Error again
September 23
06:44 YuviPanda: remove unpuppetized install of openjdk-6 packages causing problems in -dev (for bug: 54444)
06:44 YuviPanda: remove unpuppetized install of openjdk-6 packages causing problems in -dev (for bug: 54444)
05:15 legoktm: logging a log to test the log logging
05:13 legoktm: logging a log to test the log logging
September 11
09:39 wm-bot: petrb: started toolwatcher
August 24
18:00 wm-bot: petrb: freed 1600mb of ram by killing yasbot processes on -login
17:59 wm-bot: petrb: killing all python processes of yasbot on -login, this bot needs to run on grid, -login is constantly getting OOM because of this bot
August 23
12:17 wm-bot: petrb: test
12:15 wm-bot: petrb: making pv from /dev/vdb on new nodes
11:49 wm-bot: petrb: syncing packages of -login with exec nodes
11:48 petan: someone installed firefox on exec nodes, should investigate / remove
21:40 wm-bot: petrb: there is no lvm on -db which we need as hell - therefore no swap either nor storage for binary logs :( I got a feeling that mysql will die oom soonish
21:39 wm-bot: petrb: db has 5% free RAM eeeek
18:36 wm-bot: root: removed lot of ?audit? logs from exec-04 they were eating too much storage
18:23 wm-bot: petrb: temporarily disabling /tmp on exec-04 in order to set up lvm
18:23 wm-bot: petrb: exec-04 96% / usage, creating a new volume
12:33 wm-bot: petrb: installing redis on tools-mc
June 14
12:35 wm-bot: petrb: updating logsplitter to new version
June 13
21:59 wm-bot: petrb: replaced logsplitter on both apache servers with far more powerfull c++ version thus saving a lot of resources on both servers
12:43 wm-bot: petrb: tools-webserver-01 is running quite expensive python job (currently eating almost 1gb of ram) it may need to be fixed or moved to separate webserver, adding swap to prevent machine die OOM
12:22 wm-bot: petrb: killing process 31187 sort -T./enwiki/target -t of user local-enwp10 for same reason as previous one
12:21 wm-bot: petrb: killing process 31190 sort -T./enwiki/target of user local-enwp10 for same reason as previous one
12:17 wm-bot: petrb: killing process 31186 31185 69 Jun11 pts/32 1-13:14:41 /usr/bin/perl ./bin/catpagelinks.pl ./enwiki/target/main_pages_sort_by_ids.lst ./enwiki/target/pagelinks_main_sort_by_ids.lst because it seems to be a bot running on login server eating too many resources
08:45 wm-bot: petrb: updated /usr/local/bin/logsplitter on webserver-01 in order to fix !b 49383
08:45 wm-bot: petrb: updated /usr/local/bin/logsplitter on webserver-01 in order to fix become afcbot 49383
08:44 wm-bot: petrb: updated /usr/local/bin/logsplitter on webserver-01 in order to fix become afcbot 49383
08:25 wm-bot: petrb: fixing missing packages on exec nodes
June 9
20:44 wm-bot: petrb: moved logs on -login to separate storage
June 8
21:24 wm-bot: petrb: installing python-imaging-tk on grid
21:20 wm-bot: petrb: installing python-tk
21:16 wm-bot: petrb: installing python-flickrapi on grid
21:16 wm-bot: petrb: installing
16:49 wm-bot: petrb: turned off wmf style of vi on tools-dev feel free to slap me :o or do cat /etc/vim/vimrc.local >> .vimrc if you love it
15:33 wm-bot: petrb: grid is overloaded, needs to be either enlarged or jobs calmed down :o
09:55 wm-bot: petrb: backporting tcl 8.6 from debian
09:38 wm-bot: petrb: update python requests to version 1.2.3.1
June 7
15:29 Coren: Deleted no-longer-needed tools-exec-cg node (spun off to its own project)
June 5
09:52 wm-bot: petrb: on -dev
09:52 wm-bot: petrb: moving /usr to separate volume expect problems :o
09:41 wm-bot: petrb: moved /var/log to separate volume on -dev
09:31 wm-bot: petrb: houston we have problem, / on dev is 94%
09:28 wm-bot: petrb: installed openjdk7 on -dev
09:00 wm-bot: petrb: removing wd-terminator service
08:39 wm-bot: petrb: started toolwatcher
07:04 wm-bot: petrb: installing maven on -dev
June 4
14:49 wm-bot: petrb: installing sbt in order to fix b48859
13:28 wm-bot: petrb: installing csh on cluster
08:37 wm-bot: petrb: installing python-memcache on exec nodes
June 3
21:40 Coren: Rebooting -login; it's trashing. Will keep an eye on it.
14:15 wm-bot: petrb: removing popularity contest
14:11 wm-bot: petrb: removing /etc/logrotate.d/glusterlogs on all servers to fix logrotate daemon
09:43 wm-bot: petrb: syncing packages on exec nodes to avoid troubles with missing libs on some etc
June 2
08:39 wm-bot: petrb: installing ack-grep everywhere per yuvipanda and irc
June 1
20:57 wm-bot: petrb: installed this to exec nodes because it was on some and not on others cpp-4.4 cpp-4.5 cython dbus dosfstools ed emacs23 ftp gcc-4.4-base iptables iputils-tracepath ksh lsof ltrace lshw mariadb-client-5.5 nano python-dbus python-egenix-mxdatetime python-egenix-mxtools python-gevent python-greenlet strace telnet time -y
10:46 wm-bot: petrb: created new instance for experiments with sasl memcache tools-mc
May 31
19:17 petan: deleting xtools project (requested by Cyberpower678)
17:24 wm-bot: petrb: removing old kernels from -dev because / is almost full
17:17 wm-bot: petrb: installed lsof to -dev
15:55 wm-bot: petrb: installed subversion to exec nodes 4 legoktm
15:47 wm-bot: petrb: replacing mysql with maria on exec nodes
15:46 wm-bot: petrb: replacing mysql with maria on exec nodes
15:14 wm-bot: petrb: installing default-jre in order to satisfy its dependencies
15:13 wm-bot: petrb: installing /data/project/.system/deb/all/sbt.deb to -dev in order to test it
13:04 wm-bot: petrb: installing bashdb on tools and -dev
12:27 wm-bot: petrb: removing project local-jimmyxu - per request on irc
10:54 wm-bot: petrb: killing process 3060 on -login (mahdiz 3060 1964 88 May30 ? 21:32:51 /bin/nano /tmp/crontab.Ht3bSO/crontab) it takes max cpu and doesn't seem to be attached
May 30
12:24 wm-bot: petrb: deleted job 1862 from queue (error state)
20:00 wm-bot: petrb: installing p7zip-full to -dev and -login
May 27
08:46 wm-bot: petrb: changed config of mysql to use /mnt as path to save binary logs, this however requires server to be restarted
May 24
08:44 petan: setting up lvm on new exec nodes because it is more flexible and allows us to change the size of volumes on the fly
08:28 petan: created 2 more exec nodes, setting up now...
May 23
09:20 wm-bot: petrb: process 27618 on -login is constantly eating 100% of cpu, changing priority to 20
May 22
20:54 wm-bot: petrb: changing ownership of /data/project/bracketbot/ to local-bracketbot
14:28 labs-logs-bottie: petrb: installed netcat as well
14:28 labs-logs-bottie: petrb: installed telnet to -dev
14:02 Coren: tools-webserver-02 now live; / and /cluebot/ moved there
May 21
20:27 labs-logs-bottie: petrb: uploaded hosts to -dev
May 19
13:40 labs-logs-bottie: petrb: killing that nano process seems to be some hang and unattached anyway
12:59 labs-logs-bottie: petrb: changed priority of nano process to 19
12:55 labs-logs-bottie: petrb: local-hawk-eye-bot /bin/nano /tmp/crontab.d4JhUj/crontab eat too much cpu
12:50 petan: nvm previous line
12:50 labs-logs-bottie: petrb: vul alias viewuserlang
May 14
21:22 labs-logs-bottie: petrb: created a separate volume for /tmp on login so that temp files do not fragment root fs and it does not get filled up by them, it also makes it easier to track filesystem usage
13:16 Coren: reboot -dev, need to test kernel upgrade
May 10
15:08 Coren: create tools-webserver-02 for Apache 2.4 experimentation
May 9
04:12 Coren: added -exec-03 and -exec-04. Moar power!!1!
May 6
19:59 Coren: made tools-dev.wmflabs.org public
08:04 labs-logs-bottie: petrb: created a small swap on -login so that users can not bring it to OOM so easily and so that unused memory blocks can be swapined in order to use the remaining memory more effectively
08:00 labs-logs-bottie: petrb: making lvm from unused disk from /mnt on -login so that we can eventually use it somewhere if needed
May 4
17:50 labs-logs-bottie: petrb: foobar as well
17:47 labs-logs-bottie: petrb: removing project flask-stub using rmtool
15:33 labs-logs-bottie: petrb: fixing missing db user for local-stub
12:51 labs-logs-bottie: petrb: creating mysql accounts by hand for alchimista and fubar
May 2
20:49 labs-logs-bottie: petrb: uploaded motd to exec-N as well, with information which server users connected to
May 1
16:59 labs-logs-bottie: petrb: fixed invalid permissions on /home
April 27
18:54 labs-logs-bottie: petrb: installing pymysql using pip on whole grid because it is needed for greenrosseta (for some reason it is better than python-mysql package)
April 26
23:55 Coren: reboot to finish security updates
08:00 labs-logs-bottie: petrb: patching qtop
07:57 labs-logs-bottie: petrb: added tools-dev to admin host list so that qtop works and fixing the bug of qtop
07:28 labs-logs-bottie: petrb: installing GE tools to -dev so that we can develop new j|q* stuff there
April 25
19:00 Coren: Maintenance over; systems restarted and should be working.
18:18 labs-logs-bottie: petrb: we are getting in troubles with memory on tools-db there is only less than 20% free memory
18:01 Coren: Begin maintenance (login disabled)
13:21 petan: removing local-wikidatastats from ldap
