Jump to content

Server Admin Log/Archive 22: Difference between revisions

From Wikitech
Content deleted Content added
spage synchronized php-1.22wmf4/extensions/GettingStarted 'E3 latest GettingStarted' (logmsgbot)
bsitu synchronized wmf-config/InitialiseSettings.php 'Enable echo email bundling on enwiki' (logmsgbot)
Line 1: Line 1:
== May 23 ==
== May 23 ==
* 21:23 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable echo email bundling on enwiki'
* 20:31 logmsgbot: spage synchronized php-1.22wmf4/extensions/GettingStarted 'E3 latest GettingStarted'
* 20:31 logmsgbot: spage synchronized php-1.22wmf4/extensions/GettingStarted 'E3 latest GettingStarted'
* 20:29 logmsgbot: spage synchronized php-1.22wmf4/extensions/GuidedTour 'E3 latest GuidedTour'
* 20:29 logmsgbot: spage synchronized php-1.22wmf4/extensions/GuidedTour 'E3 latest GuidedTour'

Revision as of 21:22, 23 May 2013

May 23

  • 21:23 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable echo email bundling on enwiki'
  • 20:31 logmsgbot: spage synchronized php-1.22wmf4/extensions/GettingStarted 'E3 latest GettingStarted'
  • 20:29 logmsgbot: spage synchronized php-1.22wmf4/extensions/GuidedTour 'E3 latest GuidedTour'
  • 19:17 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo/modules 'syncing Echo modules for css updates'
  • 18:54 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to Master
  • 18:48 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to Master
  • 18:31 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Add new eventlogging schema: EchoPrefUpdate'
  • 12:19 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
  • 02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 23 02:25:13 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Thu May 23 02:06:32 UTC 2013

May 22

  • 23:40 logmsgbot: aaron synchronized php-1.22wmf4/maintenance/runJobs.php '2188c14239b74d202b2109933612259193f6fa41'
  • 22:44 logmsgbot: krinkle synchronized php-1.22wmf4/extensions/VisualEditor/modules/ve/ve.EventEmitter.js 'touch, attempt to fix VisualEditor cache snafu on dewiki'
  • 22:33 paravoid: removing users sara & mwang from ldap/ops
  • 19:12 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
  • 19:05 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
  • 18:51 apergos: restarted gerrit
  • 18:42 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Swithced remaining wikis to 1.22wmf4
  • 18:26 logmsgbot: kaldari synchronized php-1.22wmf4/includes/Preferences.php 'syncing preferences.php for bug 48693'
  • 18:02 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
  • 18:00 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
  • 17:46 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
  • 17:41 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
  • 07:50 apergos: restarted swift proxy on ms-fe1
  • 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 22 02:35:44 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 22 02:14:48 UTC 2013
  • 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Wed May 22 02:08:27 UTC 2013
  • 00:21 Ryan_Lane: deployed change 64875 to virt0

May 21

  • 21:11 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Regular weekly MobileFrontend deployment
  • 21:04 logmsgbot: awjrichards Started syncing Wikimedia installation... : Regular weekly MobileFrontend deployment
  • 19:27 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 19:22 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 18:31 mwalker: (previous log referring to fundraising jenkins!) now upgrading fundraising jenkins for auth plugins
  • 18:31 mwalker: restarted jenkins for rekeying of jobs dir; then updated non auth plugs
  • 18:17 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Disable AFTv5 talk page links'
  • 18:16 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Disable AFTv5 talk page links'
  • 17:49 cmjohnson1: powering down db26 for last time
  • 17:45 Jeff_Green: rebooting several fundraising hosts for kernel upgrades
  • 17:14 cmjohnson1: replacing disk10 (L11) on ms-be1009
  • 14:38 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Disable $wgCentralAuthLoginWiki for testwiki'
  • 13:59 logmsgbot: anomie synchronized wmf-config/CommonSettings.php 'Set $wgCentralAuthLoginWiki for test & test2 (SUL2 testing)'
  • 13:59 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Set $wgCentralAuthLoginWiki for test & test2 (SUL2 testing)'
  • 13:14 logmsgbot: anomie synchronized php-1.22wmf4/extensions/CentralAuth 'Update CentralAuth to master, for SUL2 testing on testwiki and test2wiki'
  • 13:13 logmsgbot: anomie synchronized php-1.22wmf3/extensions/CentralAuth 'Update CentralAuth to master, for SUL2 testing on testwiki and test2wiki'
  • 09:41 hashar: made Jeroen De Dauw a maintainer of https://packagist.org/packages/mediawiki/data-values
  • 08:10 hashar: zuul: running git gc on all Zuul git repositories: find /var/lib/zuul/git -maxdepth 4 -type d -name .git -print -exec git --git-dir={} gc \;
  • 08:02 hashar: Running git gc on Zuul git repositories (gallium: /var/lib/zuul/git/operations/puppet )
  • 07:18 LeslieCarr: allowed labs access to the new labsdb hosts
  • 02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 21 02:26:30 UTC 2013
  • 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 21 02:09:52 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue May 21 02:06:41 UTC 2013
  • 01:31 cmjohnson1: cycling titanium to check bios settings

May 20

  • 23:59 mutante: dist-upgrading celsus
  • 23:36 logmsgbot: mflaschen synchronized php-1.22wmf4/extensions/GettingStarted/ '-m Sync GettingStarted for E3 lightning deploy'
  • 22:26 mutante: rebooting Tampa parsoid servers, one by one
  • 22:20 mutante: dist-upgrading, rebooting kuo
  • 22:11 mutante: dist-upgrading, rebooting arsenic
  • 22:04 mutante: dist-upgrading, rebooting strontium
  • 21:48 mutante: dist-upgrading, rebooting palladium
  • 21:31 mutante: dist-upgrading, rebooting niobium
  • 20:58 logmsgbot: krinkle synchronized php-1.22wmf4/extensions/VisualEditor/modules/ve/ve.EventEmitter.js 'touch, fixup cache snafu'
  • 20:51 Ryan_Lane: temporarily disabling nova-api ratelimiting
  • 20:46 akosiaris: added cloudera debian packages hadoop, zookeeper, bigtop-jsvc, bitop-utils to apt.wikimedia.org
  • 20:07 Ryan_Lane: rebooting all pmtpa virt nodes
  • 20:07 Ryan_Lane: shutdown all labs instances
  • 19:12 mutante: dist-upgrading cp1(sq51)
  • 19:09 logmsgbot: csteipp synchronized php-1.22wmf4/includes/ 'Security fix'
  • 19:08 logmsgbot: csteipp synchronized php-1.22wmf3/includes 'Security fix'
  • 19:04 logmsgbot: reedy synchronized w/
  • 19:01 logmsgbot: reedy synchronized wmf-config/wgConf.php
  • 19:00 logmsgbot: reedy synchronized wmf-config/wgConfVHosts.php
  • 18:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable translate on commonswiki'
  • 18:46 Reedy: Created translate tables on commonswiki
  • 18:44 logmsgbot: krinkle synchronized php-1.22wmf4/resources/mediawiki/mediawiki.js 'touched'
  • 18:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:30 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache
  • 18:21 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache
  • 18:15 logmsgbot: reedy synchronized php-1.22wmf4/extensions/Wikibase/
  • 18:14 logmsgbot: reedy synchronized php-1.22wmf4/extensions/ProofreadPage/
  • 18:08 logmsgbot: reedy synchronized wmf-config/ 'Enable TemplateData on all wikis'
  • 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf4
  • 17:49 MaxSem: Reindexing solr
  • 17:48 mutante: upgrading/rebooting mobile varnish cache cp1041-cp1044
  • 17:35 mutante: kernel upgrade cp1043
  • 17:00 mutante: resurrecting icinga-wm and nsca
  • 16:59 mutante: ping logmsgbot
  • 16:52 mutante: rebooting neon for upgrade
  • 16:51 cmjohnson1: shutting down lanthanum to add ssd rt5074
  • 15:49 ottomata: rebooting gadolinium (source of webrequest multicast stream) to upgrade kernel
  • 14:55 ottomata: rebooting emery to upgrade to 3.2.0-43-generic kernel
  • 14:53 ottomata: upgraded all analytics* machines to linux 3.2.0-43-generic kernel
  • 02:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 20 02:29:25 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Mon May 20 02:13:42 UTC 2013
  • 02:05 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Mon May 20 02:05:04 UTC 2013

May 19

  • 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 19 02:06:55 UTC 2013
  • 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 19 02:01:54 UTC 2013
  • 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sun May 19 02:01:19 UTC 2013

May 18

  • 02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 18 02:29:09 UTC 2013
  • 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 18 02:11:28 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sat May 18 02:06:30 UTC 2013
  • 01:23 mutante: singer boot issue - affects service contacts.wm (but no others)
  • 00:13 mutante: dist-upgrading yvon
  • 00:09 mutante: dist-upgrading nitrogen

May 17

  • 23:48 mutante: dist-upgrading singer
  • 23:46 mutante: dist-upgrading nickel
  • 23:45 RobH: all tampa based apaches have had kernel upgrades
  • 23:40 mutante: dist-upgrading praseodymium
  • 23:33 RobH: upgrading all srv*.pmtpa.wmnet via dist-upgrade in salt.
  • 23:28 mutante: dist-upgrading gurvin
  • 23:13 RobH: rebooting all pmtpa mw servers
  • 23:13 mutante: dist-upgrading hydrogen, manutius
  • 23:12 RobH: still doing pmtpa mw upgrades, ignore all icinga alarms for now
  • 23:05 mutante: dist-upgrading chromium
  • 23:02 RobH: using salt to dist-upgrade all the tampa apaches... nothing could go wrong....right?
  • 22:58 akosiaris: upgrading and rebooting cp1001-1020 (3 at a time)
  • 22:55 mutante: dist-upgrading capella ( IPv6 tunnel relay)
  • 22:34 mutante: shutting down zinc
  • 22:23 mutante: !!log doesn't log
  • 22:21 Reedy: [23:18:29] <binasher> !!log running hotbackup of db71 to pre-labsdb for s1
  • 22:14 mutante: rebooting spence for upgrades
  • 22:10 logmsgbot: reedy synchronized w
  • 22:08 logmsgbot: reedy synchronized docroot
  • 22:08 mutante: rm -rf php-1.21wmf12 on tin per reedy
  • 22:03 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
  • 21:58 apergos: mw80 memtest failure dimm1, all other image scalers in pmtpa updated
  • 21:53 mutante: dist-upgrading zhen (mobile vumi)
  • 21:49 mutante: DNS update - kill storage1 and 2
  • 21:34 akosiaris: about to reboot fenari
  • 21:13 hashar: gallium Manually blackholed some web engine crawler (via ip route)
  • 19:55 mutante: Wikimedia IRC server working again
  • 19:46 Ryan_Lane: rebooted ssl1/2 ssl3001 ssl1001/1002
  • 19:42 hashar: Jenkins restarted successfully.
  • 19:40 Ryan_Lane: depooling ssl3001 ssl1/2 ssl1001/2
  • 19:40 Ryan_Lane: repooling ssl1003/4 ssl3/4
  • 19:31 Ryan_Lane: repooling ssl3002/3
  • 19:30 hashar: restarted Jenkins
  • 19:29 Ryan_Lane: rebooted ssl3/4 ssl3002/3 ssl1003/1004
  • 19:29 Ryan_Lane: depooled ssl3/4 ssl3002/3 ssl1003/1004
  • 19:19 RobH: blog back up, whew.
  • 19:16 RobH: holmium (blog server) rebooting for kernel upgrade
  • 19:12 cmjohnson1: shutting titanium down to add disks (take 2)
  • 19:04 apergos: eqiad imagescalers done
  • 19:04 mutante: rebooting gallium
  • 19:03 mutante: wikimedia irc is back
  • 19:00 hashar: gallium: stopping jenkins and zuul
  • 18:56 RobH: cerium rebooting for kernel upgrade
  • 18:56 Ryan_Lane: dist-upgrading all ssl boxes
  • 18:55 mutante: dist-upgrading ekrem (IRC)
  • 18:55 cmjohnson1: shutting down titanium for add'l disk
  • 18:50 RobH: i hate you morebots.
  • 18:50 RobH: caesium rebooting
  • 18:47 mutante: dist-upgrading calcium
  • 18:42 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Removing france as a redirect country for fundraising (7649276ddb3f20678de265c37fe75e1d5e642956)'
  • 18:31 hashar: Jenkins: created a Jenkins slave on gallium using jenkins-slave as user and /srv/ssd/jenkins-slave as working directory.
  • 18:17 RobH: carbon rebooting for kernel update, tftp in eqiad will be offline for a few
  • 17:50 apergos: updating and rebooting image scalers one by one
  • 17:46 cmjohnson1: shutting down cerium to add add'l disk
  • 17:39 apergos: rebooted snapshots 1001-4 (new kernel)
  • 17:35 mutante: bugzilla is back
  • 17:31 ^demon: rebooting formey (svn) in 2 minutes...why are you still using SVN?
  • 17:30 mutante: rebooting kaulen
  • 17:24 mutante: rebooting kaulen (Bugzilla) in 5 minutes, please save your bugs and expect 5 minutes downtime
  • 16:49 ^demon: rebooting antimony
  • 15:00 paravoid: setting Varnish send_timeout to 600 on upload, mobile, bits (eqiad), upload, bits (esams) for both frontend & backend
  • 13:31 hashar: Zuul: applying project templates 63674
  • 12:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 12:29 logmsgbot: reedy synchronized php-1.22wmf4/maintenance/checkUsernames.php
  • 05:06 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
  • 02:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 17 02:33:21 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 17 02:12:44 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Fri May 17 02:07:14 UTC 2013
  • 00:56 K4-713: Woohoo! Logs!
  • 00:51 logmsgbot: reedy synchronized php-1.22wmf3/maintenance/checkUsernames.php

May 16

  • 23:48 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/AccountAudit/ 'Updating AccountAudit to master. Adding aa_method'
  • 23:45 logmsgbot: pgehres synchronized php-1.22wmf4/extensions/AccountAudit/ 'Updating AccountAudit to master. Adding aa_method'
  • 23:43 logmsgbot: reedy synchronized database lists files:
  • 23:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:39 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
  • 22:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:29 logmsgbot: mwalker synchronized php-1.22wmf4/extensions/CentralNotice/ 'Updating CentralNotice to master'
  • 22:28 logmsgbot: reedy synchronized wmf-config/
  • 22:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:02 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
  • 22:00 logmsgbot: mflaschen Finished syncing Wikimedia installation... : E3 scap for deployment of GettingStarted and GuidedTour
  • 21:53 logmsgbot: mflaschen Started syncing Wikimedia installation... : E3 scap for deployment of GettingStarted and GuidedTour
  • 21:50 superm401: Scap failed due to wrong ssh configuration
  • 21:49 Ryan_Lane: rebooting virt1005 and virt1007
  • 21:49 logmsgbot: mflaschen Finished syncing Wikimedia installation... : E3 deployment
  • 21:48 logmsgbot: mflaschen Started syncing Wikimedia installation... : E3 deployment
  • 21:47 Ryan_Lane: rebooting virt1000
  • 21:40 superm401: Started scap for E3 deployment of GettingStarted and GuidedTour
  • 21:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 21:29 binasher: running accountaudit_login aa_method schema migrations on all wikis
  • 21:20 Ryan_Lane: rebooting virt0
  • 20:56 Ryan_Lane: running dist-upgrade on virt cluster
  • 20:51 logmsgbot: reedy synchronized wmf-config/
  • 20:40 logmsgbot: reedy synchronized php-1.22wmf4/extensions/Wikibase/
  • 20:35 paravoid: restarting a ceph-osd osd.3 for testing
  • 20:03 logmsgbot: reedy synchronized wmf-config/ 'Updating interwiki cache'
  • 19:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 19:58 logmsgbot: reedy synchronized database lists files:
  • 19:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:51 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 19:49 logmsgbot: reedy synchronized database lists files:
  • 19:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:01 logmsgbot: reedy synchronized wmf-config/
  • 18:56 logmsgbot: reedy synchronized wmf-config/
  • 18:55 logmsgbot: reedy synchronized wmf-config/session.php
  • 18:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'EP on dewikiversity'
  • 18:41 mwalker|0_0: updating payments from fdf308c8fa92ca75adbb3154b8b31a9f7237e885 to head
  • 18:34 gwicke: updated Parsoid to b0229051eb08c8
  • 18:29 mutante: magnesium (racktables): upgrading kernel and rebooting
  • 18:21 mutante: zirconium (planet): upgrading kernel and rebooting
  • 18:17 paravoid: restarting a ceph-osd osd.0 for testing
  • 18:02 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuilding l10n cache for wikidata deploy
  • 17:51 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuilding l10n cache for wikidata deploy
  • 17:27 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
  • 17:13 logmsgbot: reedy synchronized php-1.22wmf4/extensions/
  • 16:53 ottomata: rebooting stat1 for kernel upgrade
  • 16:24 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pooling db1043'
  • 16:18 gwicke: Parsoid update to current master: done
  • 16:16 gwicke: Parsoid update to current master: start
  • 15:57 Krinkle: Graceful reload of Zuul deploying Ic9194c079fa7d46
  • 15:28 Krinkle: Graceful reload of Zuul deploying I7565eff83a7a128e
  • 13:50 Krinkle: Graceful reload of Zuul deploying I0ea73ca1439c8aa6
  • 07:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/GeoData/ 'I9d5afff9c328'
  • 07:47 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/GeoData/ 'I9d5afff9c328'
  • 07:30 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
  • 06:42 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
  • 02:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 16 02:30:28 UTC 2013
  • 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May 16 02:11:55 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Thu May 16 02:06:34 UTC 2013
  • 00:37 mutante: puppetstoredconfigclean.rb ocg3.pmtpa.wmnet
  • 00:14 mutante: puppetstoredconfigclean.rb ms2.pmtpa.wmnet
  • 00:06 logmsgbot: reedy synchronized database lists files:
  • 00:05 logmsgbot: reedy synchronized database lists files:

May 15

  • 23:44 mutante: pushing wikimedia-task-appserver 2.9-1
  • 23:32 logmsgbot: reedy synchronized database lists files:
  • 23:30 logmsgbot: reedy synchronized database lists files:
  • 22:30 logmsgbot: krinkle synchronized php-1.22wmf4/resources/mediawiki/mediawiki.js 'touched'
  • 22:01 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
  • 21:54 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
  • 20:57 Ryan_Lane: OpenStackManager is reenabled on virt0. login to wikitech is working again
  • 20:56 Ryan_Lane: openstack in pmtpa now at the folsom release
  • 20:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/
  • 20:51 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/MobileFrontend/
  • 20:15 Ryan_Lane: force running puppet on all virt nodes to bring them back up
  • 20:14 Ryan_Lane: upgrading schema for keystone, then glance, then nova
  • 20:13 Ryan_Lane: backing up keystone, nova, and glance databases
  • 20:12 logmsgbot: aaron cleared profiling data
  • 20:12 logmsgbot: aaron cleared profiling data
  • 20:10 Ryan_Lane: upgrading glance, keystone and nova-scheduler on virt0
  • 20:09 Ryan_Lane: upgrading nova services on compute nodes
  • 20:07 logmsgbot: aaron synchronized php-1.22wmf4/includes/specials/SpecialActiveusers.php 'df79a82'
  • 20:05 Ryan_Lane: disabling OpenStackManager
  • 20:05 Ryan_Lane: stopping all openstack services. it won't be possible to log into wikitech during this process
  • 19:44 paravoid: running refreshFileHeaders.php on terbium
  • 19:25 logmsgbot: aaron cleared profiling data
  • 19:19 Ryan_Lane: adding temporary ubuntu-cloud archive apt changes in a non-puppetized way on pmtpa virt nodes
  • 19:18 Ryan_Lane: disabling puppet on pmtpa virt cluster
  • 19:09 logmsgbot: reedy synchronized wmf-config/ 'touch!'
  • 19:09 logmsgbot: reedy synchronized php-1.22wmf4/resources/ 'touch!'
  • 18:42 Ryan_Lane: cherry-picked change 62100 on virt0 until permanent fix for jquery.chosen minimum width is added
  • 18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commons back to 1.22wmf4
  • 18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commons back to 1.22wmf3 due to broken UploadWizard
  • 18:35 ^demon: restarting gerrit
  • 18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: aaaand wikisource to 1.22wmf4
  • 18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf4
  • 18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikibooks to 1.22wmf4
  • 18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage and wiktionary to 1.22wmf4
  • 18:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: fishbowl, special, wikimedia and private to 1.22wmf4
  • 18:01 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/
  • 17:58 RobH: authdns update, killin dataset1 entries
  • 17:56 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
  • 17:50 Ryan_Lane: deploying new project filter on virt0
  • 17:44 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
  • 16:57 RobH: authdns update, cleaning out old decom entries
  • 15:35 Jeff_Green: rebooting boron
  • 11:16 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 11:15 logmsgbot: reedy synchronized wmf-config/squid-labs.php
  • 11:14 logmsgbot: reedy synchronized wmf-config/squid.php
  • 11:13 logmsgbot: reedy synchronized wmf-config/squid-labs.php 'wmf-config/squid.php'
  • 11:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 15 02:34:37 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Wed May 15 02:14:58 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 15 02:08:15 UTC 2013
  • 00:49 mutante: graceful apache on stat1001
  • 00:31 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/63806/'
  • 00:29 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/63806/'

May 14

  • 23:57 mwalker: lies -- went from d895730067e3325fd630d09196ee81eeba24b5ee back to head on civi because it was working -- the tester was just broken
  • 23:42 mwalker: rolling back civi from deploy head to d895730067e3325fd630d09196ee81eeba24b5ee because thank you isn't working
  • 23:38 logmsgbot: aaron synchronized wmf-config/StartProfiler.php
  • 23:23 logmsgbot: maxsem synchronized php-1.22wmf3/resources/startup.js
  • 22:38 RobH: authdns-update for a bunch of decom updates
  • 22:32 mutante: DNS update - removing owa servers
  • 22:23 logmsgbot: tstarling synchronized wmf-config/db-pmtpa.php
  • 22:17 Ryan_Lane: finished upgrading wikitech
  • 22:10 Reedy: Running sync-common on srv193
  • 22:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 22:03 Ryan_Lane: upgrading mediawiki to 1.22wmf4 on wikitech
  • 22:03 mwalker: upgrading fundraising civi (d895730067e3325fd630d09196ee81eeba24b5ee --> head)
  • 21:53 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 21:30 RoanKattouw: Repooled wtp1004, added wtp1005-wtp1024 to the pool
  • 21:01 sbernardin: changing HD in ms-be5
  • 20:52 binasher: s4 now all mariadb, new master info = MASTER_LOG_FILE='db1059-bin.000012', MASTER_LOG_POS=473448023
  • 20:45 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/63596/'
  • 20:44 sbernardin: changing drives on ms-be10
  • 20:39 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/Gadgets/ 'https://gerrit.wikimedia.org/r/#/c/60954/'
  • 20:33 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
  • 20:32 mutante: apply umask 0002 for wikidev users fix on tin and terbium, class generic::wikidev-umask
  • 20:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'setting s4 to writeable on new master'
  • 20:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'setting s4 to read-only for a sec'
  • 20:22 logmsgbot: maxsem synchronized docroot/bits/mobile/W.png
  • 20:20 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
  • 19:53 RoanKattouw: Running salt-call state.highstate on wtp1005-24
  • 19:51 logmsgbot: aaron synchronized php-1.22wmf3/includes/filerepo/file/LocalFile.php 'd58dc596a25607deaecf0aa1c3be877f834da533 '
  • 19:47 RoanKattouw: Accepted salt keys for wtp1005-1024 on sockpuppet
  • 19:22 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to master
  • 19:12 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to master
  • 18:52 mutante: gracefulling apaches with dsh
  • 18:46 Ryan_Lane: running an apache-graceful-all
  • 18:44 ottomata: deployed puppet change to make nginx send webrequest logs back to main multicast firehose.  NOTE: These logs are not sent to the udp2log instance on emery.
  • 18:35 sbernardin: replacing bad drive in ms-be11
  • 18:34 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo 'syncing Echo for 1.22wmf4'
  • 18:31 logmsgbot: reedy synchronized wmf-config/
  • 18:30 logmsgbot: aaron synchronized php-1.22wmf3/includes/filerepo/file/LocalFile.php '0cf6b27aa43ad2b65725081ee4d36ccb0f8a34c6'
  • 18:28 logmsgbot: reedy synchronized wmf-config/
  • 18:27 mutante: powercycling frozen mw1173
  • 18:25 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo 'syncing Echo for 1.22wmf4'
  • 18:12 logmsgbot: reedy synchronized wmf-config/
  • 17:56 logmsgbot: reedy synchronized wmf-config/
  • 17:53 logmsgbot: reedy synchronized wmf-config/
  • 17:28 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enable upload of XML files on Incubator (for MediaWiki dumps)'
  • 17:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Namespace configuration for ur.wikipedia'
  • 17:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 17:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 16:18 logmsgbot: reedy synchronized wmf-config/
  • 15:14 Reedy: Created EducationProgram tables on dewikiversity
  • 14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'thwiki collation'
  • 13:53 cmjohnson1: running dns update
  • 13:53 hashar: Zuul: applying a patch to prevent it from fetching a change multiple time.
  • 10:51 apergos: re-enabling puppet on ssl1001 for a bit of real traffic
  • 10:42 apergos: repooled ssl1
  • 09:58 apergos: puppet temp disabled on all ssl terminators except ssl1 in pmtpa
  • 09:37 apergos: depooled ssl1 in pmtpa for testing
  • 08:51 logmsgbot: tstarling synchronized php-1.22wmf4/LocalSettings.php 'remove testwiki special case'
  • 08:51 logmsgbot: tstarling synchronized php-1.22wmf3/LocalSettings.php 'remove testwiki special case'
  • 08:34 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 08:26 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 08:11 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 07:44 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 07:34 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 07:16 logmsgbot: tstarling synchronized w/MWVersion.php
  • 07:15 logmsgbot: tstarling synchronized refresh-dblist
  • 07:14 logmsgbot: tstarling synchronized multiversion
  • 07:12 logmsgbot: tstarling synchronized docroot/noc/db.php
  • 06:40 logmsgbot: tstarling synchronized README
  • 04:34 Tim: moved scap to tin. Copied source from NFS to tin. Fixed up broken git submodule config using sed.
  • 04:03 logmsgbot: tstarling synchronized cgi-bin
  • 03:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 14 03:09:49 UTC 2013
  • 02:31 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue May 14 02:31:15 UTC 2013
  • 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 14 02:12:10 UTC 2013
  • 02:11 logmsgbot: LocalisationUpdate failed: mwversionsinuse returned empty list
  • 01:13 logmsgbot: tstarling synchronized README
  • 01:00 logmsgbot: tstarling synchronized README
  • 00:20 akosiaris: added myself to root email alias at mchenry
  • 00:17 akosiaris: installing package-upgrades on zirconium

May 13

  • 23:19 logmsgbot: catrope synchronized php-1.22wmf4/extensions/VisualEditor/ 'VE bug fix'
  • 22:46 RobH: authdns-update for wikipedia.is
  • 21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 21:11 hashar: jenkins: updating all phpcs-HEAD jobs to make them ignore i18n.php files bug 48420 / 63571
  • 21:02 logmsgbot: reedy synchronized wmf-config/
  • 20:34 logmsgbot: aaron cleared profiling data
  • 20:33 cmjohnson1: morebots back
  • 14:40 gallium: deleted an ip route blacklist
  • 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 13 02:35:49 UTC 2013
  • 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Mon May 13 02:06:03 UTC 2013

May 12

  • 14:59 Krinkle: Graceful reload of Zuul to deploy Ic22d7c6cb811fd7e3
  • 14:31 jeremyb: imported missing !log entries from channel log
  • 14:23 bblack: restarted adminbot on wikitech-static
  • 14:22 bblack: fixed typo in rt.wikimedia.org vhost hostname, was breaking RT http stuff
  • 14:19 bblack: morebots is broken, someone should fix that when they read this
  • 09:32 logmsgbot: olivneh synchronized wmf-config/throttle.php '(Bug 48301) Add throttle exception for Haifa University workshop'
  • 03:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 12 03:03:03 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 12 02:13:17 UTC 2013
  • 01:28 logmsgbot: aaron synchronized php-1.22wmf3/maintenance/copyFileBackend.php '3e103e93f01c24d7e454aaa5db07418405d273c3'
  • 01:22 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend/FileBackendMultiWrite.php '308ebddc237e76a9602b20585386fc26876f0bee'

May 11

  • 13:24 hashar: restarted Jenkins. Got hit by a memory bug again.
  • 03:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 11 03:48:00 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 11 02:16:23 UTC 2013

May 10

  • 23:20 mutante: DNS update - remove various old Tampa hosts
  • 22:49 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/ 'Small little update to maintenance script'
  • 21:46 mutante: shutdown -h blondel .. bye
  • 21:39 RobH: switch port disabled for blondel & bellin
  • 21:27 RobH: decom of bellen/blondel : setting network ports to disabled
  • 21:27 AaronSchulz: Running listing based copy-sync scripts in terbium for all originals+transcodes on all wikis for swift->ceph to fix any outdated/missing files
  • 21:05 mutante: fixing some more varnishncsa monitoring
  • 20:58 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend/FileBackendStore.php '2717d876ba2507df26ac2c5ab746ad7895af1bbf'
  • 20:31 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend
  • 20:06 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS
  • 19:57 logmsgbot: aaron synchronized php-1.22wmf3/maintenance
  • 19:56 logmsgbot: aaron synchronized php-1.22wmf3/includes '6c01414775bb9c5269b391bee7ffd1611f9ff74f & d1940caa1f4a11e7c7d8d2ff5b178da7de8e2a92'
  • 18:50 mutante: killing and restarting nagios-nrpe-server on a bunch of cp10xx boxes to fix monitoring of varnishncsa
  • 18:23 mutante: powercycling srv291
  • 18:17 RobH: authdns-update to remove blog.wikimediafoundation as it wasnt setup and pointed to decom server
  • 17:42 RobH: authdnsupdate - removing racktables2 entry
  • 17:19 hashar: jenkins: cleared all occurrences of an old plugin in build histories. bug 48025. Jenkins should be fine now. Will monitor tonight.
  • 13:20 mark: Migrated toolserver routing to cr1-esams and cr2-esams via csw2-esams access
  • 12:28 mark: Removed knsq23-knsq30 from the text squid config
  • 12:25 hashar: jenkins: cleaning up all occurrences of an old plugin in build histories. The script run in a tmux on gallium (see https://bugzilla.wikimedia.org/show_bug.cgi?id=48025#c19 )
  • 12:09 mark: Depooled knsq23-knsq30 text squid frontends
  • 11:52 mark: Moved amslvs1 and 2 from csw1-esams to csw2-esams
  • 11:52 mark: Started PyBal on amslvs1
  • 11:51 mark: Stopped PyBal on amslvs1
  • 11:49 mark: Reenabled PyBal on amslvs2
  • 11:44 hashar: restarting jenkins
  • 11:44 mark: Disabled Puppet and stopped PyBal on amslvs2
  • 11:27 mark: Moved ssl3001-3004 from csw1-esams to csw2-esams
  • 10:09 hashar: jenkins: upgrading plugins and restarting.
  • 09:43 hashar: restarted jenkins twice to get it to finally serve something :(
  • 09:27 hashar: killed -9 jenkins :(
  • 09:25 hashar: restarting jenkins
  • 09:18 hashar: gallium: dumping java heap of Jenkins using jmap -dump:format=b,file=/root/jenkins-bug48025.jmap -F 24508
  • 08:34 hashar: Jenkins hit by the evil bug again bug 48025. Taking as much trace as possible before restarting it
  • 04:46 paravoid: swift: pushing new rings, set weight 0 for ms-be5 sdb1, ms-be11 sdh1 (broken disks); balance 999.99, needs further rebalance
  • 03:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 10 03:37:41 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 10 02:15:49 UTC 2013
  • 00:15 logmsgbot: pgehres synchronized wmf-config 'Actually enabling wgCentralAuthAutoMigrate'

May 9

  • 23:34 logmsgbot: mwalker synchronized wmf-config/InitialiseSettings.php 'Enabling siteNotice div on all alpha and beta mobile sites'
  • 23:27 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/CentralAuthUser.php 'Moving one more CentralAuth log line'
  • 23:22 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/ 'Updating CentralNotice to master for some bugfixes'
  • 23:07 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/ 'Updates to CentralAuth'
  • 21:55 hashar: Jenkins / Zuul is back up (for now).
  • 21:45 hashar: restarted Jenkins to enable access logs in /var/log/jenkins/access.log 63068
  • 21:42 hashar: forced Zuul to stop. All events have been dropped :(
  • 21:31 hashar: restarting Zuul to make it query Jenkins directly without SSL + apache proxy 63069.
  • 21:29 Jeff_Green: pdns-update to cut back to aluminium, restarted pdns on nescio b/c it failed
  • 21:28 ^demon: gerrit repositories will now recursively merge
  • 21:17 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/WikimediaMessages 'Updating WikimediaMessages to master'
  • 20:48 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/GuidedTour 'Actually updating GuidedTour to master'
  • 20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/EventLogging 'Updating GuidedTour to master'
  • 20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/EventLogging 'Updating EventLogging to master'
  • 20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/ConfirmEdit 'Updating ConfirmEdit to master'
  • 20:21 hashar: restarting Jenkins.
  • 20:20 hashar: Jenkins died :/
  • 19:17 mutante: fixing file permissions in /a/e3/E3Analysis repo on stat1001
  • 19:09 mutante: restarted Apache and then puppetmaster on stafford, that revived it
  • 18:56 mutante: attempting to restart broken puppetmaster on stafford
  • 18:27 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Enabling CentralAuth debug log'
  • 18:00 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Enabling wgCentralAuthAutoMigrate on all wikis that use CentralAuth'
  • 15:56 cmjohnson1: powering down barium /relocate server to c1
  • 15:27 logmsgbot: anomie synchronized wmf-config/CommonSettings.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
  • 15:27 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
  • 15:26 logmsgbot: anomie synchronized wmf-config/InitialiseSettings-labs.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
  • 11:25 hashar: upgrading Zuul to latest master (makes report times friendlier)
  • 08:54 hashar: Jenkins apparently working fine. I have upgraded a few plugins, that seems to have fixed the Gerrit jobs.
  • 08:22 hashar: restarting Jenkins again
  • 08:21 hashar: upgrading Jenkins plugins 'promoted build' and 'M2 release'. That should fix the Gerrit jobs disappearing (ping qchris)
  • 07:39 hashar: restarting jenkins. All its http thread are locked :/
  • 07:25 hashar: Jenkins HTTP interface died :/
  • 03:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 9 03:39:55 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May 9 02:15:54 UTC 2013

May 8

  • 22:27 RobH: authdns update to migrate racktables from hooper to magnesium. both servers are already using db1001 as master, which is why hooper is slow to load racktables data
  • 22:24 logmsgbot: mholmquist synchronized php-1.22wmf3/extensions/UploadWizard/UploadWizardHooks.php 'Fixing Special:Preferences for wikis with UploadWizard'
  • 21:45 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates
  • 21:37 logmsgbot: mholmquist Started syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates
  • 21:07 Krinkle: Running small maintenance script foreachwiki to gather data about odd wgRC2UDPPrefix values (bug 28276)
  • 20:21 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/ 'CentralNotice security bugfix 48255'
  • 20:15 Jeff_Green: authdns-update to swap in grosley for dead aluminium
  • 19:41 RobH: magnesium is future racktables host, disregard errors for now
  • 19:41 cmjohnson1: accidently removed power from magnesium
  • 19:00 Reedy: Ensuring securepoll database tables exist on all wikis
  • 18:35 logmsgbot: reedy synchronized php-1.22wmf3/extensions/CentralAuth
  • 18:31 Jeff_Green: updating and rebooting grosley
  • 18:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:17 logmsgbot: reedy synchronized wmf-config/
  • 18:16 logmsgbot: reedy synchronized database lists files:
  • 18:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable EducationProgram on mkwiki'
  • 18:07 Reedy: Created EducationProgram tables on mkwiki
  • 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.22wmf3
  • 18:00 RobH: authdnsupdate for racktables2
  • 17:46 apergos: )
  • 17:46 apergos: about to graceful apaches *and api servers* (which somehow are not in the actual apache graceful all script
  • 17:42 Jeff_Green: kernel updates and reboot on pappas
  • 17:29 logmsgbot: yurik Finished syncing Wikimedia installation... : Zero extension
  • 17:18 RobH: restarting all mw jobrunners
  • 17:13 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Disabled TTMServerMessageUpdateJob jobs'
  • 17:12 logmsgbot: aaron synchronized php-1.22wmf3/includes/job/JobQueueGroup.php 'revert hack'
  • 17:11 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueGroup.php 'revert hack'
  • 17:08 logmsgbot: yurik Started syncing Wikimedia installation... : Zero extension
  • 15:01 logmsgbot: mark synchronized wmf-config/mc-eqiad.php
  • 14:22 paravoid: ceph osd out 39, primary for stuck unclean pg 3.5c6
  • 14:21 paravoid: ceph osd out 128, primary for stuck unclean pg 3.19e1
  • 13:59 logmsgbot: mark synchronized wmf-config/mc-eqiad.php
  • 13:59 mark: Removing mc1009 from the eqiad memcached pool
  • 06:14 apergos: restarted adminbot (morebot) on wikitech-static

May 7

  • 23:42 logmsgbot: catrope synchronized php-1.22wmf3/resources/startup.js 'touch'
  • 23:42 logmsgbot: catrope synchronized php-1.22wmf3/extensions/VisualEditor/modules/ve/ve.js 'touch'
  • 23:39 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:35 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/ 'Appling update to fix CentralNotice message editing bug'
  • 23:30 mutante: running addWiki.php for iegcomwiki
  • 23:26 logmsgbot: dzahn synchronized wmf-config/InitialiseSettings.php
  • 23:23 mutante: depool mw1041 in pybal
  • 23:22 mutante: removed mw1041 from dsh groups mediawiki-installation and apaches
  • 23:20 mutante: mw1041 - RT-4517, RT-5065, RT-5105 ...and broken again :p
  • 23:17 logmsgbot: mflaschen synchronized php-1.22wmf3/extensions/GettingStarted/ 'E3 Lightning deploy for GettingStarted'
  • 23:17 LeslieCarr: upgrading srv* machines
  • 22:48 Tim: on marmontel: disabled logrotate
  • 22:36 logmsgbot: aaron synchronized wmf-config/mc-pmtpa.php 'removing mc15 for hw maint'
  • 22:34 mutante: upgrading PHP_CodeSniffer on gallium
  • 22:31 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/62744/'
  • 22:10 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/62650/'
  • 22:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 22:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/
  • 21:47 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 21:09 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueGroup.php
  • 20:53 logmsgbot: aaron synchronized php-1.22wmf3/includes/job/JobQueueGroup.php 'Live hack to avoid translationNotificationJob jobs.'
  • 20:44 logmsgbot: reedy synchronized database lists files:
  • 20:30 andrewbogott: copied passwords from stafford to sockpuppet, then reset --hard on stafford to get things back in sync.
  • 20:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 20:11 logmsgbot: reedy synchronized docroot/login/
  • 20:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 19:57 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: loginwiki
  • 19:56 logmsgbot: reedy synchronized wmf-config/
  • 19:55 logmsgbot: reedy synchronized database lists files: loginwiki
  • 19:37 mutante: DNS update - switch old secure.wm
  • 19:30 mutante: apache-graceful-all, move old secure.wm redirects
  • 18:54 cmjohnson1: authdns update
  • 18:42 logmsgbot: dzahn synchronized docroot
  • 18:42 mutante: sync-docroot for login.wm
  • 18:38 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1042,1059 to s4'
  • 18:03 logmsgbot: awjrichards synchronized php-1.22wmf2/extensions/MobileFrontend/stylesheets/common/ui.css 'Cherry-pick e37e9d9ae6b8a581569c39ede38fc21827983128'
  • 17:59 logmsgbot: awjrichards synchronized php-1.22wmf3/extensions/MobileFrontend/stylesheets/common/ui.css 'Cherry-pick e37e9d9ae6b8a581569c39ede38fc21827983128'
  • 17:43 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/FundraiserLandingPage/ 'Updating FundraiserLandingPage for WMF fundraising in FR'
  • 17:38 kaldari: killed copyExistingEmailPreference.php on enwiki
  • 17:29 Reedy: Killed php-1.21wmf1 through 1.21wmf5 on fenari and mediawiki-installation hosts
  • 17:26 logmsgbot: mwalker Finished syncing Wikimedia installation... : Update CentralNotice and core for HTMLForms, iframe preview, and devices!
  • 17:19 mutante: killed hanging pdns related processes on ns1 and restarted pdns
  • 17:15 kaldari: begin running extensions/Echo/maintenance/copyExistingEmailPreference.php on enwiki db
  • 17:02 mutante: DNS update - add login.wikimedia and login.m.wikimedia
  • 16:57 logmsgbot: mwalker Started syncing Wikimedia installation... : Update CentralNotice and core for HTMLForms, iframe preview, and devices!
  • 16:37 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
  • 16:35 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
  • 16:31 RobH: professor being worked on, network is offline =(
  • 16:29 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
  • 16:19 gwicke: updated Parsoid to current master
  • 16:03 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
  • 15:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 15:46 logmsgbot: reedy synchronized docroot
  • 14:47 cmjohnson1: authdns update
  • 14:33 cmjohnson1: authdns update
  • 13:09 mark: Power cycled line card 2 in csw1-esams (which has the SSL servers) to fix esams SSL reachability issues
  • 09:15 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'New ULS config part 2'
  • 09:14 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'New ULS config part 1'
  • 09:09 logmsgbot: nikerabbit synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/ 'ULS to master'
  • 08:21 paravoid: ceph: all radosgw & mon + osd.24 upgraded to 0.56.6
  • 07:59 paravoid: partially upgrading ceph to 0.56.6
  • 07:27 Tim: on holmium: reverted firewall, too hard
  • 07:22 Tim: on holmium: added some ad hoc iptables rules intended to keep the rest of the site mostly safe from it overnight
  • 07:00 Tim: on holmium: chown'd to root all script files writable by www-data and all directories which are configured to allow PHP execution
  • 06:28 Tim: on marmontel: disabled apache logrotate, stopped apache, disabled apache init script
  • 04:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 7 04:19:39 UTC 2013
  • 02:25 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue May 7 02:25:11 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 7 02:16:06 UTC 2013
  • 01:29 logmsgbot: bsitu synchronized php-1.22wmf3/includes/User.php 'Two fixes to User::getNewMessageRevisionId() from kaldari'
  • 01:29 logmsgbot: reedy synchronized php-1.22wmf3/extensions/WikimediaMaintenance
  • 00:38 logmsgbot: reedy synchronized php-1.22wmf3/includes/User.php 'Live hack to stop fatals'
  • 00:07 logmsgbot: bsitu synchronized php-1.22wmf3/extensions/Echo 'Update Echo to master'
  • 00:07 logmsgbot: bsitu synchronized php-1.22wmf3/docs/hooks.txt 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
  • 00:06 logmsgbot: bsitu synchronized php-1.22wmf3/RELEASE-NOTES-1.22 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
  • 00:06 logmsgbot: bsitu synchronized php-1.22wmf3/includes/WikiPage.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
  • 00:03 logmsgbot: bsitu synchronized php-1.22wmf3/includes/User.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
  • 00:03 logmsgbot: bsitu synchronized php-1.22wmf3/includes/Skin.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
  • 00:02 logmsgbot: bsitu synchronized php-1.22wmf3/includes/OutputPage.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'

May 6

  • 23:25 mutante: restarting prelabsdbdb on db1054
  • 22:48 mutante: apache-graceful-all, add iegcom virtual host
  • 21:00 mutante: blog is back up
  • 21:00 mutante: holmium: bring up interface lo, comment "search" line in resolv.conf, restart varnish, ...
  • 20:46 paravoid: swift replication scripts running on ms-fe1002 under screen -- kill if they have any effect on production
  • 20:15 mutante: DNS update - sync with ns2
  • 20:04 mutante: restarting parsoid on wtp1001
  • 20:03 mutante: restarting pdns on ns0
  • 20:01 mutante: killing parsoid server.js on wtp1001
  • 19:59 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'f257ddac39194f8bfcfd145104999fdba83d33f5'
  • 19:59 mutante: DNS update - switch all blog cnames over to holmium
  • 19:30 logmsgbot: reedy synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
  • 19:25 LeslieCarr: moved gateway from csw1-esams to cr1-esams
  • 19:18 Krinkle: Reloading Zuul to deploy I259585e78ee
  • 19:18 LeslieCarr: rebooting ns2
  • 19:16 logmsgbot: reedy synchronized wmf-config/
  • 19:13 binasher: rebuilt db1016 from hotbackup of db1001; pmtpa side of m1 needs to be rebuilt as well
  • 18:42 RobH: ns2/nescio appears offline, investigating
  • 18:37 RobH: ns2 appears offline
  • 18:37 RobH: authdns-update crashed on ns2, rerunning
  • 18:33 RobH: holmium
  • 18:33 RobH: authdns-update for new server homium
  • 18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf3
  • 17:18 LeslieCarr: marmontel's interface is now in the sandbox vlan
  • 17:16 LeslieCarr: turned off marmontel's network port
  • 16:12 logmsgbot: reedy synchronized docroot
  • 13:55 hashar: gallium: installed manually libcidr0-dev libanon0-dev . Puppet change is 62434
  • 09:51 paravoid: upgrading Ganglia Web to 3.5.7
  • 08:25 hashar: Zuul upgrading to upstream version 7191ee8 + 1 wmf hack.
  • 06:45 Nemo_bis: Translate search on vanadium keeps being broken https://toolserver.org/~nemobis/tmp/SearchTranslations.log (now RT-5085)
  • 06:15 paravoid: pushing new swift rings: ms-be1 weight 50->0, ms-be2 33->75
  • 02:19 paravoid: restarting gerrit
  • 02:18 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

May 5

  • 10:56 Nemo_bis: "Search service is temporarily unavailable" since unknown time https://meta.wikimedia.org/wiki/Special:SearchTranslations?query=amen&uselang=en
  • 03:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 5 03:36:17 UTC 2013
  • 02:26 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun May 5 02:25:44 UTC 2013
  • 02:18 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 5 02:17:22 UTC 2013

May 4

  • 22:42 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
  • 21:44 Reedy: Manually force ran TorBlock/loadExitNodes.php
  • 21:43 logmsgbot: reedy synchronized php-1.22wmf3/extensions/TorBlock/
  • 16:03 logmsgbot: reedy synchronized wmf-config/
  • 16:02 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS/
  • 04:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 4 04:36:09 UTC 2013
  • 02:21 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat May 4 02:20:30 UTC 2013
  • 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 4 02:12:01 UTC 2013
  • 01:17 Krinkle: binasher restarted gerrit

May 3

  • 23:49 Krinkle: Gerrit is completely unresponsive (all http and ssh time out, serving HTTP 500). Filed details at https://bugzilla.wikimedia.org/show_bug.cgi?id=48061
  • 22:19 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'adding new pmtpa slave'
  • 21:10 cmjohnson1: db10 is decommissioned...powering down for the final time
  • 20:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileFrontend.hooks.php
  • 19:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/includes/MobileFrontend.hooks.php
  • 07:52 MaxSem: OMG logmsgbot is a terminator!
  • 06:16 logmsgbot: tstarling synchronized wmf-config/missing.php
  • 04:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 3 04:03:12 UTC 2013
  • 02:26 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri May 3 02:26:02 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 3 02:15:45 UTC 2013
  • 01:25 hashar: Zuul processed all its events, seems to be fine now. *sleep*
  • 00:33 logmsgbot: dzahn synchronized docroot
  • 00:32 mutante: sync-docroot for iegcom wiki
  • 00:25 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php '6512451451ac6617bedbfce7be17045bed794d70'
  • 00:16 mutante: DNS update - add iegcom

May 2

  • 23:20 mutante: restarting lucene on search1021, search1022, search1017, search1018 (with some waiting in between)
  • 23:18 hashar: yeah after hours and hours of fighting, Jenkins is finally working again.
  • 23:11 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php
  • 22:59 hashar: restarting jenkins… it got a few threads blocked and the main process is at 100% usage for now reason
  • 22:59 mutante: repooling search1016
  • 22:40 mutante: gallium, run puppet, graceful Apache to deploy split log files
  • 22:34 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1020 for upgrade'
  • 22:32 mutante: depooling search1016, restarting lucene, etc.. (Search#Adding_new_wikis)
  • 22:29 mutante: repooling search1015
  • 22:23 hashar: Zuul restarted. The bug about slowness is bug 48025
  • 22:15 hashar: restarting Zuul
  • 22:13 hashar: Deploying a workaround on Zuul to make it stop querying the Jenkins API when it just want to check whether a job exist. 62095
  • 22:04 mutante: depooling search1015 in pybal
  • 21:43 logmsgbot: mflaschen synchronized php-1.22wmf3/extensions/GuidedTour/ 'Sync GuidedTour to 1.22wmf3 for E3 deployment'
  • 21:42 logmsgbot: mflaschen synchronized php-1.22wmf2/extensions/GuidedTour/ 'Sync GuidedTour to 1.22wmf2 for E3 deployment'
  • 21:38 mutante: importing wikimania2014wiki into search indexers
  • 21:35 mutante: restarting search indexers on searchidx2, searchidx1001 to make sure the indexer knows about new wiki
  • 21:28 logmsgbot: mflaschen synchronized php-1.22wmf3/skins/common/shared.css 'Sync font-size change for edit section links'
  • 21:15 logmsgbot: spage synchronized php-1.22wmf2/extensions/ConfirmEdit 'update 1.22wmf2 to wmf3 version of ConfirmEdit'
  • 21:14 LeslieCarr: adding GEANT via fiberring to avoid-paths
  • 20:21 logmsgbot: reedy synchronized php-1.22wmf3/includes/
  • 20:15 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS/
  • 19:51 Jeff_Green: authdns-update to move db1025 to frack.eqiad.wmnet
  • 19:47 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
  • 19:40 ottomata: rebooting oxygen
  • 19:33 ottomata: re-enabling puppet on cp1031, it was administratively disabled. running puppet there.
  • 19:24 RobH: professor was already powered down (why?) starting it back up now
  • 19:23 ottomata: all webrequest udp2log loggers (squid and varnish) now send to gadolinium for socat unicast -> multicast relay
  • 19:22 RobH: rebooting professor
  • 19:16 Jeff_Green: moving db1025 into frack-fundraising1-c-eqiad
  • 19:14 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo 'sync Echo ext for en.wiki'
  • 19:14 ottomata: deployed squid frontend.conf.php changes to remove locke and send logs directly to gadolinium for multicast relay
  • 19:11 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo 'sync Echo ext'
  • 19:09 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo 'sync Echo ext'
  • 18:52 ottomata: removed varnishcsa-locke instance from varnish hosts: (dsh -c -g varnishncsa-all 'test -f /etc/init.d/varnishncsa-locke && service varnishncsa-locke stop && update-rc.d -f varnishncsa-locke remove && rm -v /etc/init.d/varnishncsa-locke')
  • 18:47 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
  • 18:41 ottomata: varnishncsa now sends traffic to gadolinium instead of oxygen for multicast relay
  • 18:30 Krinkle: Installed Monitoring plugin from Jenkins control panel <https://wiki.jenkins-ci.org/display/JENKINS/Monitoring>
  • 17:59 Krinkle: Jenkins restart complete. No visible improvement. Jenkins is still idling most of the time while Zuul is still halted by an unknown factor on spawning jobs.
  • 17:56 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 17:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:48 logmsgbot: reedy synchronized wmf-config/
  • 17:37 Krinkle: Jenkins keeps clogging up. Starting an emergency restart.
  • 17:00 Krinkle: Jenkins is nearing 100% CPU on gallium, what is Jenkins doing?
  • 16:59 Krinkle: Zuul is somehow having trouble kicking off Jenkins jobs (less than 1 event processed per minute). Jenkins shows that 10/10 executors are idle. Investigating...
  • 16:52 logmsgbot: nikerabbit synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/ 'ULS to master'
  • 15:13 hashar: Jenkins made Jenkins to instantly restart (was bug 47120) ) by deleting the downstream-buildview plugin.
  • 15:11 hashar: restarted Zuul
  • 14:12 hashar: Upgraded Jenkins from 1.480.3 to LTS 1.509.1. Restarted it.
  • 14:10 hashar: shutting down Zuul
  • 14:09 hashar: upgrading Jenkins (unscheduled maintenance).
  • 13:01 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/61941/'
  • 12:59 mark: Migrated amslvs3 and amslvs4 PyBal BGP peerings from csw2-esams to cr2-knams
  • 09:55 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/'
  • 09:53 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/'
  • 09:50 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/'
  • 09:40 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/GeoData/GeoData.body.php 'https://gerrit.wikimedia.org/r/#/c/61962/'
  • 06:04 apergos: powercycle snapshot1001, upgrade to precise
  • 05:46 apergos: powercycle snapshot1004, upgrade to precise
  • 05:28 apergos: snapshot1003 powercucle, upgrading to precise
  • 05:28 Tim: on professor: manually started carbon-cache.py
  • 05:15 Tim: professor is down, no response on serial console, rebooting
  • 05:11 logmsgbot: tstarling synchronized wmf-config/jobqueue-pmtpa.php
  • 05:08 logmsgbot: tstarling synchronized wmf-config/jobqueue-eqiad.php
  • 04:47 Tim: on rdb1001-1002, set requirepass and masterauth in soft state
  • 04:44 Tim: on mc1-16 and mc1001-1016, setting requirepass and masterauth to the new password in soft state
  • 04:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 2 04:39:48 UTC 2013
  • 04:04 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
  • 03:59 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
  • 03:57 logmsgbot: tstarling synchronized wmf-config/jobqueue-eqiad.php
  • 03:47 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
  • 03:40 logmsgbot: tstarling synchronized php-1.22wmf3/extensions/GettingStarted
  • 02:46 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu May 2 02:32:10 UTC 2013
  • 02:22 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May 2 02:21:30 UTC 2013
  • 02:21 Tim: on wikibugs-l: disabled bounce processing and re-enabled mail delivery to wikibugs-irc (was disabled due to excessive bounces)
  • 02:21 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 01:20 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'Poking bits to try and get the new banner controller deployed for CentralNotice'
  • 01:17 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'Poking bits to try and get the new banner controller deployed for CentralNotice'
  • 00:16 mutante: @stafford:~# puppetstoredconfigclean.rb db10.pmtpa.wmnet

May 1

  • 23:28 logmsgbot: tstarling synchronized php-1.22wmf3/extensions/GettingStarted
  • 23:23 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
  • 23:18 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
  • 22:42 mutante: powercycling frozen mw1041
  • 22:14 logmsgbot: mwalker Finished syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring
  • 22:06 logmsgbot: dzahn synchronized ./multiversion/MWMultiVersion.php
  • 22:02 logmsgbot: dzahn synchronized docroot
  • 21:44 logmsgbot: mwalker Started syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring
  • 21:32 logmsgbot: dzahn gracefulled all apaches
  • 21:25 ottomata: syncing new redirects.conf to apachces to redirect wikipediazero.org
  • 21:14 Jeff_Green: stopping mysql on db78 for cloning
  • 20:50 ottomata: adding new wikipediazero.org zone alias, running authdns-update on dobson
  • 19:45 logmsgbot: reedy synchronized php-1.22wmf3/cache/l10n/
  • 19:15 mutante: DNS update - remove project2, anthony and yongle :p
  • 19:05 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild l10n cache for wikidata
  • 18:36 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 1 18:36:02 UTC 2013
  • 18:32 mutante: DNS update - add wikimania2014 entries
  • 18:31 RobH: disabled owa1-3 switchports, starting remote wipe
  • 18:21 RobH: owa1/2/3 offlinging and wiping
  • 18:20 logmsgbot: reedy synchronized php-1.22wmf3/extensions/ 'Update extensions for wikibase deploy'
  • 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wiktionary to 1.22wmf3
  • 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource, wikivoyage and wikiversity to 1.22wmf3
  • 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf3
  • 18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special and wikimedia to 1.22wmf3
  • 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, private and fishbowl to 1.22wmf3
  • 17:58 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo/Hooks.php 'syncing Hooks in Echo'
  • 17:33 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo/Hooks.php 'syncing Hooks in Echo'
  • 17:20 LeslieCarr: puppet working on bits caches again after gerrit change 61793
  • 17:01 RobH: torrus compiling, manutilus apache offline
  • 16:53 RobH: torrus deadlocked, fixin
  • 15:54 mark: Converted link between cr2-knams:xe-1/0/0 and csw2-esams:xe-0/1/1 to LACP aggregated link cr2-knams:ae1 / csw2-esams:ae2
  • 14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 14:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 10:30 logmsgbot: reedy synchronized php-1.22wmf2/extensions/ParserFunctions/ 'Update to master to fix common fatals'
  • 10:08 mark: Removed old Toolserver IP range 91.198.174.192/27 on primary and backup router interfaces
  • 03:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 1 03:49:55 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 1 02:27:30 UTC 2013
  • 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed May 1 02:16:40 UTC 2013
  • 02:09 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Turned autoResync back on again for multiwrite backend'
  • 00:46 logmsgbot: tstarling synchronized wmf-config/PrivateSettings.php

April 30

  • 23:52 Krinkle: Gracefully restarted Zuul on gallium
  • 23:52 Krinkle: Ryan_Lane restarted the gerrit service
  • 23:44 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Syncing CommonSettings for Echo'
  • 23:34 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Syncing InitialiseSettings for Echo'
  • 23:28 logmsgbot: kaldari synchronized /php-1.22wmf3/extensions/Echo 'Updating Echo for some minor bugfixes'
  • 23:18 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/ 'Revert MobileFrontend'
  • 23:15 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'Revert MobileFrontend'
  • 23:12 logmsgbot: tstarling synchronized wmf-config/missing.php 'rewrite (I17989fc4)'
  • 23:01 Krinkle: Graceful restart of Zuul on gallium
  • 22:56 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Enabling Extension:AccountAudit on all wikis and wmgEchoHelpPage'
  • 22:09 Jeff_Green: rolling back db1025 change b/c "mysqld: unknown option '--federated'"
  • 22:06 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'grrr'
  • 22:00 logmsgbot: maxsem synchronized wmf-config
  • 21:58 Jeff_Green: restarting mysqld on db1025 to enable federated engine for testing
  • 21:56 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'Weekly mobile deployment'
  • 21:40 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/ 'Weekly mobile deployment'
  • 20:04 logmsgbot: kaldari synchronized echowikis.dblist 'syncing echowikis.dblist for cron jobs'
  • 19:59 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on Echo for English Wikipedia'
  • 19:49 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 19:19 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 19:03 cmjohnson1: removing db10 certs from puppet
  • 17:48 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Syncing CommonSettings for Echo deployment'
  • 17:45 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Syncing InitialiseSettings for Echo deployment'
  • 13:57 cmjohnson1: jenkins seems to not be working -- restarting
  • 11:27 Nikerabbit: refresh-translatable-pages.php finished for mediawikiwiki and metawiki
  • 04:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 30 04:06:11 UTC 2013
  • 02:34 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue Apr 30 02:33:50 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 30 02:15:03 UTC 2013
  • 02:04 paravoid: cleaning up neon's /var/log (100% full), restarting icinga...
  • 00:59 logmsgbot: catrope synchronized php-1.22wmf3/extensions/VisualEditor/ 'Update VE'
  • 00:44 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Touching InitialiseSettings'
  • 00:32 logmsgbot: pgehres Finished syncing Wikimedia installation... : Deploying Extension:AccountAudit and an Echo thing
  • 00:00 logmsgbot: pgehres Started syncing Wikimedia installation... : Deploying Extension:AccountAudit and an Echo thing

April 29

  • 23:20 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'updated s5 pmtpa topology'
  • 23:09 binasher: s5 pmtpa master changed to db73 - MASTER_LOG_FILE='db73-bin.000090', MASTER_LOG_POS=978211000
  • 23:08 AaronSchulz: finished creating AccountAudit tables
  • 23:06 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Re-enable WikiLove, bug fix is deployed'
  • 23:04 logmsgbot: catrope Finished syncing Wikimedia installation... :
  • 22:40 logmsgbot: catrope Started syncing Wikimedia installation... :
  • 22:28 anomie: (Including backport of gerrit change 61494 in the scap)
  • 22:27 RoanKattouw: Running scap
  • 22:23 RoanKattouw: Running scap again
  • 22:19 logmsgbot: catrope Started syncing Wikimedia installation... :
  • 22:13 hashar: gallium: restarted apache2 to kill off http connections established by mod_proxy to the jenkins backend.
  • 22:03 hashar: jenkins web server is no more responding on port 8080. Might have to restart it.
  • 22:00 hashar: gallium / jenkins : blackholed a web drawer using ip route add blackhole <IP>/32
  • 21:54 hashar: Jenkins is in trouble
  • 21:52 logmsgbot: reedy synchronized php-1.22wmf3/includes/DefaultSettings.php 'Fix wgVersion'
  • 21:31 notpeter: upgrading db45 to precise
  • 21:24 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enabling VisualEditor on viwiki and test2wiki'
  • 20:31 AaronSchulz: Doing another original/math/timeline sync/copy run
  • 20:28 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Enabled multiwrites for ceph/swift'
  • 19:19 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo/maintenance/setEmailOptionTemp.php
  • 19:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf3
  • 19:06 logmsgbot: reedy Finished syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it
  • 18:46 logmsgbot: reedy Started syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it
  • 18:37 logmsgbot: reedy synchronized php-1.22wmf2/.git/modules/extensions/WikiLove
  • 18:21 logmsgbot: reedy synchronized php-1.22wmf3/extensions/
  • 18:06 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache
  • 18:03 notpeter: deploying new frontend squid conf
  • 17:51 LeslieCarr: restarting icinga-wm
  • 17:35 cmjohnson1: removing asw-c1 for replacement
  • 17:30 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache
  • 17:09 mutante: fix NRPE checks on ms-be2, running as "weird user id" bug
  • 17:05 mutante: removing reinstalled ms-be2 from ssh_known_hosts with ssh-keygen and then making it world-readable again manually so deployers don't get errors :p
  • 17:05 logmsgbot: csteipp synchronized php-1.22wmf2/includes/
  • 17:05 logmsgbot: reedy synchronized live-1.5
  • 17:02 mutante: attempt to fix NRPE checks on ms-be9, restart as correct user
  • 16:49 logmsgbot: reedy synchronized docroot
  • 16:44 logmsgbot: reedy synchronized php-1.22wmf3/ 'initial sync of 1.22wmf3 files'
  • 03:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 29 03:25:35 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Mon Apr 29 02:12:33 UTC 2013

April 28

  • 13:14 hashar: restarting Zuul, it has a finished build stuck in the `postmerge` pipeline.
  • 02:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 28 02:48:51 UTC 2013
  • 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun Apr 28 02:12:10 UTC 2013

April 27

  • 03:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 27 03:06:47 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat Apr 27 02:13:53 UTC 2013
  • 00:08 mutante: make SSL on Bugzilla also use SSLCACertificatePath and c_rehash certs on kaulen

April 26

  • 23:58 mutante: upgrading mysql packages on marmontel (blog)
  • 23:53 mutante: make SSL on the Blog use SSLCACertificatePath and c_rehash certs on marmontel
  • 20:11 hashar: killed a node process on gallium. Probably a daemon that has gone standalone when jenkins died
  • 19:19 hashar: jenkins died again sorry :( But this time I most probably found out the reason why it is slowed down: a web spider attempt to crawl all our builds!
  • 18:04 mwalker: Upgraded fundraising civicrm install from a3db312836b6ac004694f831071b1c7d00560fbf to a65fbcec0dbb5bb79a97aca220a998e56d08b067
  • 15:56 hashar: jenkins: enabled ExtraSettings.php loading 61022. That enables a bunch of MediaWiki settings for all mediawiki related jobs, namely displaying errors, moving the temp dir to tmpfs, showing exceptions details and enabling development warnings.
  • 13:49 mark: Rebooting hooft
  • 13:48 mark: dist upgrade of hooft to precise
  • 13:48 mark: Moved udpmcast from hooft to nescio
  • 13:16 ^demon: gerrit replication freaked out, restarting
  • 06:38 hashar: Restarted Jenkins. It was stuck in some while loop doing tons of stat() on unexisting file :( Would be up in roughly 40 minutes :(
  • 03:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 26 03:45:44 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri Apr 26 02:12:38 UTC 2013
  • 00:57 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'temporarily disabling WikiLove per bug 47457'
  • 00:53 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor/ 'Update VisualEditor to master'
  • 00:33 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Clean up VisualEditor config'
  • 00:33 Tim: on fenari: fixed permissions on /etc/ssh/ssh_known_hosts
  • 00:33 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Remove wmgVisualEditorHide'

April 25

  • 23:09 logmsgbot: mwalker synchronized docroot/bits/favicon/ 'Pushing new favicons for incubator, wiktionary, wiktionary.en, wikibooks, wikinews, and wikispecies'
  • 22:32 logmsgbot: mflaschen Finished syncing Wikimedia installation... :
  • 22:22 ^demon: new ssl cert (gerrit.wikimedia.org) live on gerrit.
  • 20:55 notpeter: reimaging db55
  • 20:45 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 20:17 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 20:07 awight: payments cluster updated from 7f70ccd580d to fdf308c8f
  • 19:12 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'updating for new schema number'
  • 18:52 logmsgbot: reedy synchronized docroot
  • 18:19 LeslieCarr: cleaning up old home directories on fenari
  • 17:00 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Removing oldskin notice'
  • 16:50 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VisualEditor on 14 more Wikipedias'
  • 16:38 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor/ 'Update VisualEditor to master'
  • 16:20 hashar: Jenkins is busy reparsing all the build history for some reason. That slow down the continuous integration box by an order of magnitude :/ Should take roughly 40 minutes (no idea when it started)
  • 16:09 hashar: Zuul git operations are slow again :/
  • 14:57 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php 'rm debug hack'
  • 14:56 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch debug log back off, its not phps fault'
  • 14:53 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch debug log on'
  • 14:52 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php 'Debugging'
  • 11:56 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I2f0990fa087a'
  • 11:56 logmsgbot: krinkle synchronized docroot
  • 03:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 25 03:39:44 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu Apr 25 02:14:20 UTC 2013
  • 00:45 mutante: upgrading packages on manutius
  • 00:30 mutante: DNS update - removing internproxy, removing tesla (minus mgmt)

April 24

  • 23:04 logmsgbot: dzahn gracefulled all apaches
  • 22:39 logmsgbot: dzahn gracefulled all apaches
  • 22:39 LeslieCarr: upgrading mw's
  • 22:38 LeslieCarr: replaced smokeping targets for sdtpa c3 and sdtpa d3
  • 22:26 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'New location for icons - Ib5554c54729'
  • 22:26 mutante: sync-apache, add coffee
  • 22:24 logmsgbot: krinkle synchronized docroot/bits/apple-touch/ 'New location for icons - Ib5554c54729'
  • 21:47 mutante: DNS update - remove survey, remove wikitech-old, remove argon
  • 21:47 logmsgbot: demon synchronized php-1.22wmf2/extensions/Wikibase 'b4b5fb4046175ca29834a4669c253c97ac5ccd07'
  • 21:38 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
  • 21:34 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch log back on'
  • 21:33 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php
  • 21:26 mutante: DNS update - readding voip
  • 21:17 cmjohnson1: removing sq33 from dsh node groups and pybal cfg
  • 20:32 RobH: racktables on hooper upgraded to recent stable
  • 20:26 RobH: racktables flap again, working on it
  • 20:16 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch
  • 20:04 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch 'Deploying I48480cb6'
  • 20:03 logmsgbot: demon synchronized php-1.22wmf2/includes/api/ApiQuerySearch.php 'Deploying Ib7abe8d5'
  • 19:57 logmsgbot: aaron synchronized php-1.22wmf2/includes/filerepo/file/LocalFile.php 'deployed 0e42bdadd9349fe5b6119ff65322e6c52d813a6e'
  • 19:17 mutante: deploying Bugzilla wikimedia extension changes for improved wikilinking
  • 19:16 RobH: racktables going offline a moment for upgrade
  • 18:45 logmsgbot: reedy synchronized wmf-config/ 'wikidata stuffs'
  • 18:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non enwiki to 1.22wmf2
  • 18:09 mutante: DNS update - removing unused remnants
  • 18:09 sbernardin: replacing bad disk on db59
  • 17:31 sbernardin: taking down ms-be2 for hw replacement
  • 15:09 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/skins/MinervaTemplate.php 'bug 47597'
  • 15:05 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/skins/MinervaTemplate.php 'bug 47597'
  • 09:05 hashar: restarted Zuul. Upgraded to e403579 (latest master)
  • 08:58 hashar: upgrading Zuul
  • 06:04 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
  • 03:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 24 03:42:12 UTC 2013
  • 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 24 02:31:43 UTC 2013
  • 02:19 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 24 02:18:59 UTC 2013
  • 00:03 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Disable mobile sitenotice everywhere but testwiki'
  • 00:03 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disable mobile sitenotice everywhere but testwiki'

April 23

  • 23:38 mutante: cleaning up all unused Apache sites on singer, gzip, move to /root for backup
  • 23:33 mutante: disabling old techblog apache site config on singer
  • 23:32 logmsgbot: maxsem synchronized wmf-config/mobile.php
  • 23:15 logmsgbot: mwalker synchronized docroot/bits/favicon/ 'Pushing wikidata, wikimania, wikiquote, and wikiversity favicons from Isarra'
  • 23:12 logmsgbot: dzahn gracefulled all apaches
  • 23:07 logmsgbot: maxsem synchronized wmf-config 'Mobile banners again, hopefully fixed'
  • 23:05 mutante: sync-apache, add wikepdia.org and .com typo domains
  • 22:28 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Score
  • 22:21 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score
  • 22:12 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php 'debugging'
  • 22:05 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php 'debugging'
  • 22:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'rv debugging'
  • 22:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Score'
  • 21:59 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php 'debugging'
  • 21:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable score debug log'
  • 21:40 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/60512/'
  • 21:23 logmsgbot: maxsem synchronized wmf-config
  • 21:17 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Score
  • 21:15 logmsgbot: maxsem synchronized wmf-config 'Mobile banners everywhere - https://gerrit.wikimedia.org/r/#/c/60362/'
  • 21:14 logmsgbot: aaron synchronized php-1.22wmf2/includes/db/Database.php 'temp logging'
  • 21:13 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/ 'Weekly mobile deployment'
  • 21:12 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/ZeroRatedMobileAccess/ 'Weekly mobile deployment'
  • 21:06 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/ZeroRatedMobileAccess/ 'Weekly mobile deployment'
  • 21:05 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'Weekly mobile deployment'
  • 20:56 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Wikibase
  • 19:36 logmsgbot: anomie synchronized php-1.22wmf2/extensions/Cite 'Update to master for bug 47291'
  • 18:44 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Re-enable AFTv5 on enwiki'
  • 17:49 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 17:49 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 16:34 logmsgbot: reedy synchronized php-1.22wmf2/extensions/TorBlock
  • 16:34 logmsgbot: reedy synchronized php-1.22wmf1/extensions/TorBlock
  • 12:34 paravoid: reprepro python-statsd 2.0.1-1~wmf1 & voluptuous 0.6.1-1~wmf1 for precise-wikimedia
  • 09:57 pp-pdf1: restarted all services
  • 09:56 pp-pdf1: - alow numprocs to be configured via initenv.sh
  • 09:16 pp-pdf1: restarted all services
  • 09:12 pp-pdf1: restarted all services
  • 09:10 pp-pdf1: restarted all services
  • 08:46 logmsgbot: maxsem synchronized docroot/ 'https://gerrit.wikimedia.org/r/#/c/60375/'
  • 08:41 logmsgbot: maxsem synchronized wmf-config 'Beta changes, no-op for production'
  • 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 23 02:25:25 UTC 2013
  • 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 23 02:10:32 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 23 02:06:54 UTC 2013
  • 00:02 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'bookkeeping for pmtpa slaves (this time for real)'
  • 00:01 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'bookkeeping for pmtpa slaves'

April 22

  • 23:51 notpeter: reimaging db44
  • 22:57 binasher: mysql query logs / ishmael migrated from db9 to db1001
  • 21:41 LeslieCarr: upgrading even more mw servers in tampa
  • 20:12 Reedy: Running namespaceDupes on kowikiversity
  • 19:57 Reedy: Running namespaceDupes on zhwikibooks
  • 19:55 Reedy: Running namespaceDupes on uzwiki
  • 19:55 logmsgbot: reedy synchronized wmf-config/
  • 19:50 logmsgbot: reedy synchronized images/
  • 19:50 logmsgbot: reedy synchronized docroot
  • 19:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Score extension on the cluster'
  • 18:49 logmsgbot: catrope synchronized echowikis.dblist
  • 18:44 LeslieCarr: sq33 is going to be decomissioned… long live…. nothing else in tampa
  • 18:39 mutante: changing #mediawiki to #wikimedia-dev in wikibugs startup script (just in case, should not even be needed by puppetized ircecho)
  • 18:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:26 LeslieCarr: still doing package upgrades on mw* machines
  • 18:20 LeslieCarr: rebooting nonresponsive sq33
  • 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf2
  • 17:37 LeslieCarr: dist-upgrading some mw machines, should be low/no impact
  • 16:51 mutante: restarting wikibugs on mchenry
  • 16:43 logmsgbot: mark synchronized wmf-config/InitialiseSettings.php
  • 16:37 mutante: added abaso to wmf-deployment group on gerrit (new mediawiki deployer)
  • 16:11 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'New caching everywhere'
  • 16:02 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Wikibase
  • 15:57 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php '$wgMFVaryResources on enwiki, https://gerrit.wikimedia.org/r/#/c/60286/'
  • 14:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 02:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 22 02:14:58 UTC 2013
  • 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 22 02:08:56 UTC 2013
  • 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Mon Apr 22 02:05:30 UTC 2013
  • 00:13 Reedy: Running namespaceDupes.php --fix via foreachwiki in screen session on terbium

April 21

  • 02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 21 02:37:21 UTC 2013
  • 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 21 02:16:56 UTC 2013
  • 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun Apr 21 02:10:46 UTC 2013

April 20

  • 16:54 ottomata: resetarting apache2 on stafford in attempt to fix puppetmaster
  • 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 20 02:35:55 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 20 02:14:31 UTC 2013
  • 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat Apr 20 02:09:07 UTC 2013
  • 00:09 logmsgbot: aaron synchronized php-1.22wmf1/includes/job/JobQueueRedis.php 'deployed c9e702e7f73f41f7bcd2deb045085689d5cb4229'
  • 00:03 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueRedis.php 'deployed d2e4fdfed03148fdd1a564925ce351bec6f9b935'

April 19

  • 23:07 Coren: brief reboot of labstore3 -> test to see that everything starts up as expected.
  • 20:39 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Setup redis job queue debug log'
  • 18:36 Jeff_Green: increased php memory limit for fundraising civicrm instance
  • 17:55 LeslieCarr: restarting etherpad on hooper
  • 17:15 binasher: running enwiki.abuse_filter_log migration - drop afl_user index, added user_timestamp
  • 16:10 LeslieCarr: flushed celsus varnish cache
  • 06:31 Tim: deploying apache conf change for www.wikidata.org redirect (I7bb872fd)
  • 03:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 19 03:32:40 UTC 2013
  • 02:27 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 19 02:27:17 UTC 2013
  • 02:26 logmsgbot: aaron cleared profiling data
  • 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri Apr 19 02:16:42 UTC 2013
  • 01:54 logmsgbot: aaron cleared profiling data
  • 01:54 logmsgbot: aaron synchronized wmf-config/StartProfiler.php 'Removed cli profiling for now.'
  • 00:29 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor 'Update VisualEditor to master'
  • 00:29 logmsgbot: catrope synchronized php-1.22wmf1/extensions/VisualEditor 'Update VisualEditor to master'

April 18

  • 23:40 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor 'Update VisualEditor to master'
  • 23:40 logmsgbot: catrope synchronized php-1.22wmf1/extensions/VisualEditor 'Update VisualEditor to master'
  • 23:38 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Deploy 5c5455c81bc8dee0ae2a0410655da639e2f84e87 ($wgVisualEditorParsoidProblemReportURL)'
  • 23:25 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/processEchoEmailBatch.php 'More echo db fix'
  • 23:25 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/includes/DbEchoBackend.php 'More echo db fix'
  • 22:51 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Enable mobile sitenotice on testwiki'
  • 22:51 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable mobile sitenotice on testwiki'
  • 22:48 notpeter: reimaging db35
  • 22:46 logmsgbot: mwalker synchronized docroot/bits/favicon/wikisource.ico 'New and shiney ws.o favicon by Isarra'
  • 22:45 logmsgbot: mwalker synchronized docroot/bits/favicon/mediawiki.ico 'New and shiney mw.o favicon by Isarra'
  • 22:35 logmsgbot: reedy synchronized php-1.22wmf2/extensions/EducationProgram
  • 22:35 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'adjusting weights on pmtpa slaves'
  • 22:26 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
  • 22:15 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
  • 22:09 notpeter: correction: that was db33 out, not db31
  • 22:03 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'db31 and db35 out, db72 and db73 in'
  • 21:56 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/includes/EchoDbFactory.php 'Db cache fix'
  • 21:41 logmsgbot: reedy synchronized php-1.22wmf2/extensions/EducationProgram
  • 21:39 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
  • 21:38 mutante: merging changes on bugzilla to now support RT in "See also" field
  • 20:45 logmsgbot: mflaschen synchronized php-1.22wmf2/extensions/GettingStarted/ 'Sync GettingStarted 1.22wmf2 for E3 deploy'
  • 20:43 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/ 'Sync GettingStarted for E3 deploy'
  • 20:36 logmsgbot: reedy synchronized wmf-config/
  • 20:15 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 19:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgMaxImageArea and wgMaxAnimatedGifArea to 50MP'
  • 19:52 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 19:14 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Echo email config + db cluster update to testwiki'
  • 19:13 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Echo email config + db cluster update to testwiki'
  • 18:57 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/jobs 'deployed 1e3eaed6fbe7fd91c15399cc17eac8602b34c9cd'
  • 18:28 logmsgbot: reedy Finished syncing Wikimedia installation... : rebuild l10n cache for wikidata deploy
  • 17:57 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild l10n cache for wikidata deploy
  • 17:50 RobH: authdns0update for git.w.o
  • 17:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.22wmf2
  • 17:06 logmsgbot: reedy synchronized php-1.22wmf2/extensions/
  • 16:58 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Wikibase '5d496b4ea2c8a68b9e1280a282530f7ce5232b68'
  • 16:47 logmsgbot: ariel synchronized wmf-config/throttle.php
  • 16:10 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 16:10 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 15:39 cmjohnson1: authdns update
  • 15:10 mark: Reversed replication of volume fr_archive, now nas1001-a:fr_archive -> nas1-a:fr_archive
  • 14:26 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
  • 14:24 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
  • 14:14 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
  • 14:06 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
  • 14:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
  • 13:24 logmsgbot: maxsem synchronized wmf-config/mobile.php
  • 13:03 mark: Rebooting cp1041
  • 08:48 logmsgbot: maxsem synchronized wmf-config/mobile.php
  • 07:57 logmsgbot: olivneh synchronized php-1.22wmf2/extensions/ConfirmEdit 'Updating extension/ConfirmEdit (Bug 46132 / Change I8ee3fd136)'
  • 07:57 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/ConfirmEdit 'Updating extension/ConfirmEdit (Bug 46132 / Change I8ee3fd136)'
  • 07:43 hashar: jenkins raised max # of executors from 6 to 8
  • 07:07 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/59767/'
  • 05:59 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 05:21 logmsgbot: tstarling synchronized php-1.22wmf1/includes/specials/SpecialSearch.php
  • 05:15 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php
  • 05:12 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/MWSearch/MWSearch_body.php
  • 05:09 logmsgbot: tstarling synchronized php-1.22wmf1/includes/specials/SpecialSearch.php
  • 03:57 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/MWSearch
  • 03:55 logmsgbot: tstarling synchronized php-1.22wmf2/includes/HttpFunctions.php
  • 03:48 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/MWSearch
  • 03:46 logmsgbot: tstarling synchronized php-1.22wmf1/includes/HttpFunctions.php
  • 03:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 18 03:26:48 UTC 2013
  • 02:24 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 18 02:23:54 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu Apr 18 02:14:26 UTC 2013
  • 01:03 mutante: re-enabling Tomasz' RT account
  • 00:42 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'setting db1004 to full weight'
  • 00:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1004 at a warmup weight'
  • 00:09 logmsgbot: demon synchronized docroot

April 17

  • 23:39 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1004'
  • 23:21 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/ContributionReporting 'Finished update of CN and CR to latest'
  • 23:20 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice/
  • 23:14 hashar: If jenkins/zuul goes wild overnight: connect to gallium, killoff puppet in crontab then /etc/init.d/zuul stop and drop me an email. Will take of it whenever I wake up.
  • 23:12 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/ContributionReporting/
  • 23:12 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice 'Beginning update of CentralNotice and ContributionTracking to latest'
  • 22:59 hashar: restarting Zuul
  • 22:20 hashar: Jenkins repointing all jobs to use the new Zuul repo ( /srv/ssd/zuul/git ) 59744
  • 22:06 hashar: jenkins restarted. Restarting Zuul.
  • 21:20 hashar: stopped Zuul. Will wait a bit till jenkins start up
  • 21:19 hashar: restarted Jenkins
  • 21:19 hashar: restarted Zuul
  • 21:16 hashar: finished migrating zuul git repositories ( rsync -av /var/lib/zuul/git /srv/ssd/zuul )
  • 21:09 hashar: Zuul ETA: it is copying MobileFrontend extension right now so half way done
  • 21:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionaries to 1.22wmf2
  • 21:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.22wmf2
  • 21:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikiversity to 1.22wmf2
  • 21:00 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki back to 1.22wmf1
  • 20:59 hashar: stopping Jenkins that somehow restarted
  • 20:59 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php 'moar'
  • 20:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:55 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews.dblist and special wikis to 1.22wmf2
  • 20:51 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php 'debugging'
  • 20:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikimania and wikimedia to 1.22wmf2
  • 20:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Some wikis to create some noise
  • 20:42 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php 'debugging'
  • 20:30 hashar: Switching zuul dir to /srv/ssd/zuul ( 58898 and 58899 )
  • 20:18 hashar: stopping Zuul gracefully
  • 20:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Revert that due to large number of warnings...
  • 20:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: fishbowl, private and special to 1.22wmf2
  • 19:57 hashar: gallium slowness is partly due to the ssd install. The git repositories are on the slow raid where as the workspaces are on the ssd. That means `git clone` can't do hardlink anymore :(
  • 19:43 binasher: enwiki, dewiki, wikidata are fully on mariadb 5.5.30
  • 19:33 hashar: jenkins: reducing number of executors from 10 to 6.
  • 19:32 hashar: jenkins gallium has a huge queue of jobs. investigating.
  • 19:31 hashar: jenkins / gallium has a huge queue of jobs. Investigating.
  • 19:19 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 's1/s5 writeable from pmtpa'
  • 19:17 logmsgbot: asher synchronized wmf-config/db-eqiad.php 's1/5 writeable'
  • 19:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php 's1/5 read-only'
  • 19:12 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'preparing for master swaps'
  • 19:03 logmsgbot: aaron synchronized php-1.22wmf2/maintenance/doMaintenance.php 'deployed 1437b57bd475e5ecb06800f9f64b14ccffd0dedf'
  • 19:00 logmsgbot: aaron synchronized php-1.22wmf2/maintenance/doMaintenance.php 'deployed 1306792ad24be5b8d17355c58da655eb88c2fcc6'
  • 18:27 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1050'
  • 18:18 LeslieCarr: installing new linecard in cr2-eqiad , .1% chance of issues
  • 18:16 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1050'
  • 17:50 logmsgbot: aaron synchronized php-1.22wmf2/includes/User.php 'deployed 6a66776fd7afdf883ad6f82270453fc76833508b'
  • 16:20 ^demon: running puppet on manganese (will restart gerrit)
  • 13:05 apergos: restarted icinga on neon (the usual: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe)
  • 13:03 hashar: jenkins: refreshing job mediawiki-core-phpcs-HEAD (points to a wrong git repo path)
  • 09:50 mark: Inserted libvpx 1.1.0-1+wmf1 packages into the precise-wikimedia APT repository
  • 06:57 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 17 06:57:02 UTC 2013
  • 06:51 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 17 06:51:28 UTC 2013
  • 06:48 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 17 06:47:37 UTC 2013
  • 03:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 17 03:13:56 UTC 2013
  • 02:20 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 17 02:19:57 UTC 2013
  • 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 17 02:10:22 UTC 2013
  • 00:38 mutante: blog: enabled new eventlogging by ori, removed old logging class, converted plugins/WMBlog from svn to git and sync with gerrit
  • 00:31 Tim: on hume: changed UID for l10nupdate to be 10002, for consistency with puppet admins.pp

April 16

  • 23:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'raising db1058 weight'
  • 23:57 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1005, adding db1058 to s5'
  • 23:55 logmsgbot: aaron cleared profiling data
  • 23:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:48 logmsgbot: reedy synchronized wmf-config/abusefilter.php
  • 23:46 logmsgbot: aaron synchronized wmf-config/StartProfiler.php 'Enabled 1:1 profiling for cli scripts and put "cli" in the profile ID'
  • 23:22 logmsgbot: awjrichards synchronized php-1.22wmf2/extensions/MobileFrontend/
  • 23:22 logmsgbot: awjrichards synchronized php-1.22wmf1/extensions/MobileFrontend/
  • 23:00 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/ContributionReporting/ 'Contribution reporting now pulls 2013 data'
  • 22:59 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/ContributionReporting/
  • 22:51 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 16 22:51:08 UTC 2013
  • 22:37 Tim: running l10nupdate manually on fenari to test change 58911
  • 22:03 logmsgbot: anomie synchronized php-1.22wmf2/extensions/WikimediaMaintenance 'Update WikimediaMaintenance to master for l10nupdate fix'
  • 22:02 logmsgbot: anomie synchronized php-1.22wmf1/extensions/WikimediaMaintenance 'Update WikimediaMaintenance to master for l10nupdate fix'
  • 21:49 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/59449'
  • 21:46 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/ZeroRatedMobileAccess/ 'https://gerrit.wikimedia.org/r/#/c/59525/'
  • 21:43 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 21:13 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 19:54 logmsgbot: pgehres synchronized php-1.22wmf2/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
  • 19:53 logmsgbot: pgehres synchronized php-1.22wmf1/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
  • 19:38 logmsgbot: pgehres synchronized php-1.22wmf2/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
  • 19:33 logmsgbot: pgehres synchronized php-1.22wmf1/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
  • 19:18 binasher: upgrading db1005 to precise + mariadb
  • 19:07 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1005 from s5 for upgrade'
  • 19:02 binasher: streaming s5 hotbackup from db45 to db1058
  • 18:49 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1058 from s1'
  • 18:34 logmsgbot: aaron synchronized php-1.22wmf2/includes/cache/HTMLCacheUpdate.php 'deployed 344ddf85b6d7b79c40a1b6b9ec7e0a1e05c0705b'
  • 18:26 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 18:26 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 18:17 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turning on footer contact link for en.wiki'
  • 18:13 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable email bundling for test2wiki and mediawikiwiki'
  • 18:13 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Enable email bundling for test2wiki and mediawikiwiki'
  • 16:42 logmsgbot: aaron synchronized php-1.22wmf2/includes/WikiPage.php 'deployed 38d60cdafe036625a6ec8fa3a9e8a94ef450c85c'
  • 16:41 logmsgbot: aaron synchronized php-1.22wmf2/includes/db/Database.php 'deployed fa819f8e3579855f6d95552655de98f92ec1789c'
  • 16:40 logmsgbot: aaron synchronized php-1.22wmf2/includes/cache/BacklinkCache.php 'deployed 1a94124385df4c452c68fadaa74bc2714a204761'
  • 16:37 logmsgbot: aaron synchronized php-1.22wmf2/includes/OutputPage.php 'deployed 997ab92458946d10fed96ff4ba41c74eeaee0d20'
  • 05:04 Tim: moving most cron jobs from hume to terbium
  • 03:23 legoktm: test
  • 02:35 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 16 02:34:59 UTC 2013
  • 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 16 02:16:58 UTC 2013

April 15

  • 23:57 logmsgbot: csteipp synchronized php-1.22wmf2/includes/ 'User gerrit versions of security fixes'
  • 23:18 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice
  • 23:13 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice/ 'CentralNotice banner= fix'
  • 22:31 logmsgbot: dzahn gracefulled all apaches
  • 22:16 andrewbogott: Moved labs-morebots to tools project.
  • 21:43 LeslieCarr: doing a few more random host upgrades of misc hosts
  • 21:32 LeslieCarr: upgrading mw1070-mw1079
  • 21:24 LeslieCarr: upgrading mw1060-mw1069
  • 21:22 RobH: linode wikitech is now shut down \o/
  • 21:13 logmsgbot: dzahn gracefulled all apaches
  • 20:32 mutante: restarting pdns on ns1
  • 20:30 mutante: DNS update - add wikpedia.org
  • 19:37 logmsgbot: ariel gracefulled all apaches
  • 18:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf2
  • 15:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Build localisation cache for 1.22wmf2
  • 15:19 logmsgbot: reedy Started syncing Wikimedia installation... : Build localisation cache for 1.22wmf2
  • 14:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.22wmf2, scap to come
  • 14:56 logmsgbot: reedy synchronized docroot
  • 14:55 logmsgbot: reedy synchronized w
  • 14:54 logmsgbot: reedy synchronized wmf-config/
  • 14:49 logmsgbot: reedy synchronized php-1.22wmf2/ 'Initial sync of 1.22wmf2'
  • 12:24 mark: Depooled amssq47-amssq62 frontend squids
  • 12:04 mark: Depooled knsq16-knsq22 frontend squids
  • 09:20 hashar: jenkins died :(
  • 08:22 hashar: updating Jenkins jobs to fetch ZUUL_COMMIT instead of ZUUL_BRANCH. That would solve a race condition between jobs ( bug 46723 / 58865 ).
  • 05:50 logmsgbot: olivneh synchronized wmf-config/throttle.php 'Adding throttling exception for itwiki GLAM event which starts in four hours (Bug 47221, change Id9ae600ac)'
  • 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 15 02:03:51 UTC 2013

April 14

  • 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 14 02:04:17 UTC 2013

April 13

  • 23:54 Coren: rebooting labstore3. autofs is being silly
  • 15:11 Coren: Fixed boot problem on labstore3; back to operating.
  • 04:54 Coren: rebooted labstore3 again; kernel and lvm2 upgrade
  • 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 13 02:12:27 UTC 2013
  • 01:10 Coren: rebooted labstore3 again -> we hates lockd. We hates it!
  • 00:41 Coren: rebooting labstore3 -> silly lockd module

April 12

  • 23:31 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'Contact Wikipedia' footer link on test/test2 (Iab1f5f527)'
  • 23:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Add 'Contact Wikipedia' footer link on test/test2 (Iab1f5f527)'
  • 23:14 Coren: rebooted labstore3 -> hung MegaCLI processes
  • 22:41 notpeter: temp depooling search1015
  • 20:53 mutante: install package upgrades on sanger
  • 19:57 Ryan_Lane: finished opendj upgrade on virt0
  • 19:23 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Disable mwsearch log'
  • 18:54 mutante: restarting parsoid on wtp1004
  • 18:25 logmsgbot: aaron synchronized wmf-config/jobqueue-pmtpa.php 'Set pmtpa queue config to match eqiad'
  • 18:17 logmsgbot: demon synchronized php-1.22wmf1/extensions/MWSearch/MWSearch_body.php 'Moar logging'
  • 18:16 logmsgbot: demon synchronized php-1.21wmf12/extensions/MWSearch/MWSearch_body.php 'Moar logging'
  • 18:00 Ryan_Lane: upgrade opendj on virt0
  • 16:00 LeslieCarr: adding another link to asw-c-eqiad's uplink
  • 15:05 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Debug group for mwsearch'
  • 15:02 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Debug group for mwsearch'
  • 15:01 hashar: Changing Jenkins workspace directories from ${ITEM_ROOTDIR}/workspace to /srv/ssd/jenkins/workspace/${ITEM_FULLNAME} . My change has not been taken in account early on :(
  • 14:54 hashar: jenkins is back up. Starting Zuul
  • 14:44 cmjohnson1: analytics1007 has never worked..but working on it
  • 14:21 hashar: restarting jenkins
  • 14:03 hashar: gallium: created primary partition on /dev/sdb . formatting using: mkfs.xfs /dev/sdb1
  • 13:36 hashar: gallium can't find the SSD, shut downing, will attempt another slot
  • 12:59 hashar: gracefully shut-downing Jenkins and Zuul for scheduled maintenance. Will shutdown server gallium just after.
  • 09:39 hashar: jenkins: updated mediawiki-core-whitespaces job to use ZUUL_COMMIT as a refspec specifier (for bug  46723 )
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 12 02:15:30 UTC 2013
  • 00:08 mutante: DNS update - point old bugzilla3 entry over to actual bugzilla server

April 11

  • 23:35 LeslieCarr: upgrading mw1052 to mw1060
  • 23:30 logmsgbot: csteipp synchronized php-1.22wmf1/extensions/RSS/
  • 23:30 LeslieCarr: upgrading mw1051
  • 23:24 logmsgbot: csteipp synchronized php-1.22wmf1/includes/
  • 23:22 mutante: shutting down xenon
  • 23:20 LeslieCarr: upgrading mw1050
  • 23:11 LeslieCarr: upgrading 1040 to 1049
  • 23:04 mutante: install package upgrades on neon
  • 23:00 LeslieCarr: upgrading mw1030 - 1039
  • 22:51 K4-713: synchronized payments to 7f70ccd580d64
  • 22:47 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralAuth 'Peter's log line change to CentralAuth'
  • 22:41 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Enable translate on CentralNotice'
  • 22:39 logmsgbot: mwalker synchronized docroot/bits/favicon/wikipedia.ico 'Bi resolution wiki favicon'
  • 22:38 LeslieCarr: upgrading mw1120 to mw1129
  • 22:37 mutante: removing spence from dsh group mediawiki-installation
  • 22:33 logmsgbot: mwalker synchronized php-1.21wmf12/extensions/CentralNotice
  • 22:26 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice 'Updating CentralNotice to the latest and greatest (from 68f0262be838320ecc9c87eb4485c28b0699bc1f)'
  • 22:22 LeslieCarr: upgrading mw1110 to mw1119
  • 21:41 logmsgbot: kaldari synchronized php-1.22wmf1/extensions/Echo 'syncing Echo for bugfix'
  • 21:35 ottomata: emery back up and upgraded to precise (thanks mutante!)
  • 21:32 ^demon: fenari: php-1.21wmf12 and php-1.22wmf1 both rebased to remove silly merge commits.
  • 21:14 mutante: emery - revoking old puppet cert, resiging, initial puppet run
  • 21:04 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'Remove jquery.clickTracking from AFT dependencies'
  • 21:01 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/GettingStarted/SpecialGettingStarted.php
  • 20:53 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/ 'Update GettingStarted extension for E3 deployment'
  • 20:34 logmsgbot: mflaschen synchronized wmf-config 'Config change for GettingStarted'
  • 20:32 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 20:11 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 19:58 mutante: emery - upgrading from lucid to precise
  • 19:46 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'syncing InitialiseSettings to turn off VectorSectionEditLinks on testwiki'
  • 19:41 Jeff_Green: running authdns-update to propagate frdata cname
  • 19:31 AaronSchulz: Refreshed the job queue aggregator hash table
  • 19:31 mutante: running puppet on brewster, testing lucid image on emery, rebooting again
  • 19:17 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turn Echo back on for testwiki'
  • 19:16 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Add checkDelay to Echo email bundle jobs'
  • 19:07 AaronSchulz: All jobs finished copying into redis
  • 19:07 sbernardin: taking down ms-be9 for hw replacement
  • 19:02 LeslieCarr: uninstalling nagios from spence
  • 18:51 mutante: PXE booting emery into installer
  • 18:48 ottomata: emery going down for upgrade to precise
  • 18:21 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turn Echo off on testwiki'
  • 17:35 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved remaining jobs to redis'
  • 17:24 sbernardin: taking down db11 for decommissioning
  • 17:19 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 11 17:19:14 UTC 2013
  • 17:15 cmjohnson1: europium power disconnected during routine data center work
  • 13:14 logmsgbot: reedy synchronized wmf-config/ 'Disable ClickTracking :D'
  • 13:08 mark: Inserted python-extras packages into the precise-wikimedia APT repository (backport by hashar)
  • 10:10 hashar: Jenkins is back up
  • 09:23 hashar: restarting Jenkins to get the git verbose mode :(
  • 08:44 hashar: jenkins: stop git cloning mw/core over and over by reusing existing clone 58502. That should help with I/O starvation on gallium.
  • 08:40 Nikerabbit: translation memory database rebuild finished
  • 08:17 Nikerabbit: emptying Translate solr instance and starting rebuilds
  • 08:15 logmsgbot: nikerabbit synchronized php-1.22wmf1/extensions/Translate/ 'to master'
  • 07:11 logmsgbot: hashar synchronized docroot/noc/conf/highlight.php 'Revert highlight.php recent modifications bug 47112 58664'
  • 07:02 hashar: Put a live hack on fenari for bug 47112
  • 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 11 02:16:35 UTC 2013
  • 01:10 Ryan_Lane: disabling puppet on labstore3/4 - making sure gluster won't die without them and we're going to rebuild them

April 10

  • 23:31 LeslieCarr: deactivated XO ipv6 transit
  • 22:25 Ryan_Lane: added live hack to OpenStackManager on wikitech, ensuring that DynamicSidebar support is only enabled for logged-in users
  • 22:24 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'moved refreshLinks to redis.'
  • 22:20 ^demon|sick: restarting gerrit, was hung
  • 22:15 Ryan_Lane: enable DynamicSidebar support in OpenStackManager
  • 22:14 Ryan_Lane: temporarily disabling DynamicSidebar support in OpenStackManager
  • 21:59 notpeter: gracefulling apache on stafford
  • 21:36 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved htmlCacheUpdate jobs to redis'
  • 21:19 ottomata: added python-statsd to apt repo for hashar
  • 21:18 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php
  • 21:07 ottomata: added python-voluptuous to apt repo (for hashar)
  • 20:46 ottomata: added python-jsonschema to wikimedia apt repo: ori-l :)
  • 20:00 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved async upload jobs to redis'
  • 19:13 notpeter: running puppetd --enable on all nodes via salt
  • 19:09 mwalker: eqiad frack is now using clustered memcache and has site local configuration for DB, ActiveMQ, MemCacheD shtuff
  • 18:24 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Enabled use of redis for null test jobs'
  • 18:06 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Switched all remaning wikis to 1.22wmf1
  • 17:45 RobH: authdns-update to rename db1012 to stat1002
  • 17:29 logmsgbot: aaron synchronized php-1.22wmf1/maintenance/copyJobQueue.php 'deployed 54f74111901ddebdc69f26847e3904140b3723d5'
  • 17:24 LeslieCarr: restart unresponsive cp1037
  • 17:14 RobH: authdns-update
  • 15:48 mark: Ran dist-upgrade (for varnish upgrade) on cp1021-1036
  • 15:40 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Nostalgia ext'
  • 15:40 logmsgbot: demon synchronized wmf-config/extension-list 'Nostalgia ext'
  • 15:38 logmsgbot: demon synchronized php-1.22wmf1/extensions/Nostalgia
  • 14:33 logmsgbot: maxsem synchronized wmf-config/mobile.php
  • 14:19 logmsgbot: maxsem synchronized wmf-config 'Enable $wgMFVaryResources on test'
  • 14:12 mark: Inserted varnish 3.0.3plus-rc1-wm10 packages into the APT repository
  • 12:23 apergos: removed outright puppet.log on neon, was > 7gb and no room left on device
  • 07:53 logmsgbot: krinkle synchronized php-1.22wmf1/resources/startup.js 'Ia54dd738b3ce0995fa'
  • 05:57 Tim: on stafford: swapdeath repeat narrowly averted via killall ruby && /etc/init.d/apache2 stop
  • 05:34 Tim: on brewster: proxy down due to full root partition, again. Will fix it properly this time.
  • 05:28 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Add 'Zero' & 'Zero talk' namespaces to metawiki (If6b3ce5e4)'
  • 04:53 Tim: stafford went into swapdeath. I killed all ruby processes a couple of times, and then eventually stopped apache2 for a while, while it recovered
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Apr 10 02:28:06 UTC 2013
  • 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 10 02:17:19 UTC 2013
  • 00:21 mutante: removing xenon and db1012 from icinga configs, running puppetstoredconfigclean.rb on them, restarting icinga
  • 00:07 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'https://gerrit.wikimedia.org/r/#/c/58436/'
  • 00:06 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'https://gerrit.wikimedia.org/r/#/c/58436/'

April 9

  • 23:29 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/Nostalgia
  • 23:21 notpeter: running upgrades on db56 and rebooting
  • 23:19 notpeter: running upgrades on db64 and rebooting
  • 22:55 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend/
  • 22:53 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/
  • 22:36 logmsgbot: reedy synchronized wmf-config/
  • 22:34 LeslieCarr: clearing a few varnish caches - fyi if db load is seen to increase
  • 22:17 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 21:45 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 21:45 hashar: Zuul is slowly dequeuing a large number of jobs. There is currently rougly 30 minutes lag between change submission and actually processing of the job. No idea why but investigating.
  • 19:56 Ryan_Lane: installed Nuke on wikitech
  • 18:11 K4-713: updated payments anti-fraud rules
  • 13:07 hashar: jenkins plugin upgrade is tracked in bug 47040. Luckily git plugin has been fixed just in time ;-D Just need to migrate the config history.
  • 12:37 hashar: jenkins plugins seems to have been upgraded overnight :(
  • 10:26 mark: Deployed experimental varnish build (3.0.3plus~rc1-wm10) on dysprosium and cp1022
  • 08:54 hashar: Jenkins tests for mediawiki/core are broken (bug 47031) due to a faulty change that has been merged in master branch. Root cause is the gating job not testing the proper change bug  46723 :/
  • 08:43 Nikerabbit: Got rid of local changes and undeployed commit related to noc
  • 08:41 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayam and WebFonts'
  • 05:42 Tim: on mw1149: enabled apache access log
  • 02:31 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 9 02:30:59 UTC 2013
  • 02:19 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 9 02:18:53 UTC 2013
  • 00:26 ^demon: jenkins is back. java gc took awhile.
  • 00:19 ^demon: jenkins is hung, looking into it.
  • 00:16 awight: payments cluster updated from 22943dc28 to d3954768e4dd

April 8

  • 23:16 logmsgbot: catrope synchronized php-1.22wmf1/extensions/WikiEditor 'Deploying 17a90bc74e87341199a50ebe5348576e05be028a'
  • 23:15 logmsgbot: catrope synchronized php-1.21wmf12/extensions/WikiEditor 'Deploying 17a90bc74e87341199a50ebe5348576e05be028a'
  • 22:55 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Changing from emery to vanadium'
  • 22:13 logmsgbot: demon synchronized php-1.21wmf12/includes/AutoLoader.php 'Backported Poolcounter convenience class'
  • 22:13 logmsgbot: demon synchronized php-1.21wmf12/includes/PoolCounter.php 'Backported Poolcounter convenience class'
  • 21:59 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-pmtpa.php 'Enable poolcounter for search'
  • 21:59 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-eqiad.php 'Enable poolcounter for search'
  • 21:42 logmsgbot: demon synchronized php-1.21wmf12/extensions/MWSearch/MWSearch_body.php 'Deploying Poolcounter updates'
  • 21:17 notpeter: temp stopping slave on db55
  • 20:53 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Bumped file journal ttl to a year'
  • 20:18 AaronSchulz: Moved ceph sync scripts to terbium and started a second pass
  • 20:15 logmsgbot: aaron synchronized php-1.22wmf1/extensions/FlaggedRevs 'deployed c61688baef682b7d97068fefdcc399256786a387'
  • 19:08 ^demon: gerrit back
  • 19:07 ^demon: gerrit restarting, config change to fix idle timeouts.
  • 19:00 ottomata: removing some unused filters from emery's udp2log instance
  • 18:34 Jeff_Green: deployed 1.22 on payments
  • 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf1
  • 17:09 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1058 to s1 at full weight'
  • 17:05 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1058 to s1 at a low warmup rate'
  • 11:57 ^demon: gerrit back, deployed 2.6-rc0-154-gfcdb34b which contains a temporary fix for the stream-events timeout issues. See bug 46917 for info.
  • 11:53 ^demon: bringing gerrit down for urgent update
  • 10:31 hashar: gerrit: created operations/debs/libvpx for j^ . Initialized pristine-tar and upstream branches using empty commits and force push.
  • 07:48 hashar: restarting Zuul for demo purposes :-)
  • 04:00 Tim: on brewster: root partition was full, removed squid access.log and store.log and started squid
  • 03:55 Tim: on all apaches: upgrading libpoppler
  • 02:25 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 8 02:25:07 UTC 2013
  • 02:18 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 8 02:18:18 UTC 2013

April 7

  • 22:46 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 22:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:31 logmsgbot: reedy synchronized wmf-config/ 'Kill readerfeedback config, it aint coming back'
  • 22:26 logmsgbot: reedy synchronized wmf-config/
  • 22:20 logmsgbot: reedy synchronized php-1.22wmf1/extensions/CentralAuth
  • 11:35 ^demon: restarting gerrit, stream-events seized up again.
  • 07:26 hashar: gallium: restarted Zuul just to be sure. Gerrit on manganese is stuck again and need to be restarted bug 46917
  • 02:17 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Apr 7 02:17:09 UTC 2013
  • 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 7 02:11:21 UTC 2013

April 6

  • 12:27 ^demon: also restarted zuul
  • 12:19 ^demon: restarting gerrit again, stream-events halted like friday.
  • 02:23 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Apr 6 02:22:48 UTC 2013
  • 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 6 02:11:56 UTC 2013
  • 01:55 binasher: iwlinks iwl_prefix_from_title migration completed for all projects
  • 00:52 K4-713: adjusted anti-fraud rules on payments cluster

April 5

  • 23:11 binasher: running iwlinks index migration on s4 (commons)
  • 21:27 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1021 to full weight, pulling db1023'
  • 21:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1021 at a low weight'
  • 20:05 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1021 for upgrade'
  • 19:53 binasher: streaming backup of db1056 to db1050 (converting s1 snapshot slave to file_per_table)
  • 18:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 's1: moving watchlist to db1049, pulling db1043, adding db1056'
  • 18:50 Jeff_Green: increased jvm heap size on silicon (payments activemq)
  • 18:42 notpeter: restarting gerrit
  • 18:22 logmsgbot: reedy synchronized php-1.21wmf12 'Killing active users in 1.21wmf12'
  • 09:39 hashar: Ariel kindly restarted gerrit on manganese which unbroken the stream-events process in Zuul. It again receives events.
  • 09:33 hashar: restarted Zuul
  • 09:23 Nemo_bis: varnish esams purging still broken, e.g. https://upload.wikimedia.org/wikipedia/commons/f/fe/Wikipedia-logo-v2-km.png
  • 08:58 hashar: for Jenkins / Gerrit issue see bug 46917
  • 08:48 hashar: Jenkins no more receiving ANY events from Gerrit. Effect: no jobs are triggered.
  • 02:23 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Apr 5 02:23:11 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 5 02:12:34 UTC 2013
  • 00:49 logmsgbot: csteipp synchronized php-1.21wmf12/includes/
  • 00:46 logmsgbot: csteipp synchronized php-1.22wmf1/includes/
  • 00:30 Tim: on fenari: testing a new version of mw-update-l10n which doesn't use stdout of mergeMessagesFileList.php
  • 00:20 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf1.php
  • 00:19 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to master
  • 00:07 Tim: all 1.22wmf1 wikis briefly showed error messages due to display_errors being enabled when mergeMessageFileList.php generated ExtensionMessages-1.22wmf1.php. Fixed temporarily.
  • 00:04 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf1.php
  • 00:00 logmsgbot: mflaschen synchronized wmf-config/extension-list 'Remove E3Experiments and LastModified'

April 4

  • 23:51 logmsgbot: demon synchronized wmf-config/extension-list 'Remove e3 since it was deleted'
  • 23:49 logmsgbot: mflaschen synchronized wmf-config/CommonSettings.php
  • 23:46 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
  • 23:46 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to master
  • 23:45 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043'
  • 23:44 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
  • 23:25 binasher: shutting down mysql on db1043, preparing to reboot
  • 23:24 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043'
  • 23:17 mutante: wikinews.com now redirects to wikinews.org
  • 23:11 mutante: DNS update - adding wikinews.com
  • 23:08 logmsgbot: dzahn gracefulled all apaches
  • 22:52 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Re-enable Echo on testwiki'
  • 22:49 mutante: running updateSpecialPages on cswikinews per request
  • 22:14 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Disable Echo on testwiki'
  • 22:10 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Bump Echo EventLogging schema for new notification types.'
  • 22:02 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GuidedTour/ 'GuidedTour 1.22wmf1 deployment for E3'
  • 22:02 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/ 'GettingStarted 1.22wmf1 deployment for E3'
  • 22:01 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/ 'GuidedTour 1.21wmf12 deployment for E3'
  • 22:00 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GettingStarted/ 'GettingStarted deployment for E3'
  • 20:21 LeslieCarr: stopped cp3001 varnish
  • 20:18 LeslieCarr: restarting varnish on cp3001
  • 20:15 LeslieCarr: restarted udpmcast on hooft
  • 20:13 mutante: puppetstoredconfigclean.rb xenon.eqiad.wmnet - Killing xenon ...
  • 19:34 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/57525/'
  • 19:29 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/57525/'
  • 19:20 binasher: enabled 1:100 full query logging on db1042/1043, disabled puppet [planning to revert in 4-8 hours]
  • 19:09 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/57528'
  • 19:01 binasher: resuming iwlinks migrations on s3
  • 17:31 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 17:31 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 17:27 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Updating ArticleFeedbackv5 frwiki config'
  • 16:16 paravoid: upgrading image & video scalers for libav
  • 15:55 ottomata: now generating webstatscollector dumps on gadolinium. snapshot1 syncs dumps from there over to dataset2 for dumps.wikimedia.org
  • 15:38 K4-713: adjusted anti-fraud rules on the payments cluster
  • 13:26 ^demon: running jgit gc on all repos
  • 05:43 Ryan_Lane: clearing squid logs and restarting squid on brewster
  • 04:16 Ryan_Lane: opendj upgrade on virt1000 complete
  • 03:56 Ryan_Lane: upgrading opendj on virt1000
  • 02:21 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Thu Apr 4 02:21:01 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 4 02:13:31 UTC 2013

April 3

  • 23:19 mwalker: Updating CentralNotice schema on testwiki & metawiki with patch-centralnotice-2_3.sql
  • 22:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 21:45 binasher: started iwlinks index migrations on all wikis [gerrit 43389]
  • 21:32 cmjohnson1: Running DNS update
  • 20:34 ottomata: udpated webstatscollector package in apt repo to 0.1-3
  • 18:54 mutante: DNS update - also adding softwarewikipedia.org
  • 18:49 RoanKattouw: Removed caesium and xenon from /home/wikipedia/common/docroot/noc/pybal/eqiad/parsoid
  • 18:43 mutante: restarting pdns on ns1
  • 18:42 mutante: DNS update - adding softwarewikipedia.com for RT-4355
  • 18:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikibooks to 1.22wmf1
  • 18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary to 1.22wmf1
  • 18:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikisource to 1.22wmf1
  • 18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikivoyage to 1.22wmf1
  • 18:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimania and wikimedia wikis to 1.22wmf1
  • 18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private, closed and fishbowl to 1.22wmf1
  • 17:14 logmsgbot: aaron synchronized php-1.22wmf1/includes/objectcache/SqlBagOStuff.php 'deployed b61053ca4b554f6bd18fb6408967839cdb5ccde2'
  • 17:12 logmsgbot: aaron synchronized php-1.21wmf12/includes/objectcache/SqlBagOStuff.php 'deployed 61587acc64cb62400ff7978271c54e8bd8b1f02d'
  • 16:54 logmsgbot: reedy synchronized php-1.21wmf12/includes/Collation.php
  • 16:34 cmjohnson1: authdns update
  • 14:22 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'setting weight on db1028 to 400'
  • 13:15 mark: Stopped and started MySQL on pc1003 after finding mysql deadlocked
  • 13:10 logmsgbot: reedy synchronized wmf-config/
  • 12:20 mark: Rebooting cp3003 for RAID reconfiguration
  • 10:43 mark: Deactivated AS6908 peering on cr2-knams
  • 06:09 ^demon: gerrit: finished doing jgit gc on mediawiki/core. Repo size 3G -> 323M. Fresh clone time to <5m to localhost, <1.5m to other hosts inside wmf. I rock. Backup's in /home/demon/core.git in case something goes wrong. Bed time.
  • 03:05 LeslieCarr: the final upload varnish is on 2gig instead of 1gig ! win!
  • 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 3 02:31:44 UTC 2013
  • 02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Apr 3 02:19:30 UTC 2013
  • 01:05 LeslieCarr: applying a big change to varnish interface groups - risk higher than normal
  • 00:05 binasher: removed sda3/varnish.persist on cp1021, restarted varnish
  • 00:00 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/PostEdit

April 2

  • 23:46 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php '(Ia2665c4fe) Enable PostEdit on bn, br, ca, cs, et, ka and zh wikis'
  • 23:46 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php '(Ibc2633f1c) : 10000 => 5000.'
  • 23:25 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad'
  • 23:11 logmsgbot: anomie synchronized php-1.22wmf1/includes/api/ApiQueryRecentChanges.php 'Fix for API list=recentchanges rccontinue'
  • 23:10 logmsgbot: anomie synchronized php-1.21wmf12/includes/api/ApiQueryRecentChanges.php 'Fix for API list=recentchanges rccontinue'
  • 23:01 logmsgbot: maxsem synchronized php-1.21wmf12/resources/Resources.php 'https://gerrit.wikimedia.org/r/#/c/57212/'
  • 22:49 logmsgbot: maxsem synchronized php-1.21wmf12/resources/Resources.php 'https://gerrit.wikimedia.org/r/57208'
  • 22:39 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/56696/'
  • 22:34 awight: payments cluster updated from 12bc36c238 to 0ab980b8b
  • 22:24 logmsgbot: maxsem synchronized php-1.21wmf12/includes/resourceloader/ResourceLoaderStartUpModule.php
  • 22:20 LeslieCarr: authdns-update for cp1039 and cp1040 move
  • 22:14 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'touch'
  • 22:11 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend
  • 22:08 logmsgbot: maxsem Finished syncing Wikimedia installation... :
  • 21:42 logmsgbot: maxsem Started syncing Wikimedia installation... :
  • 21:38 logmsgbot: reedy synchronized wmf-config/
  • 21:25 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 20:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1028'
  • 20:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 20:41 logmsgbot: maxsem synchronized php-1.21wmf12/includes/OutputPage.php 'https://gerrit.wikimedia.org/r/#/c/49071/'
  • 20:34 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/57078/'
  • 20:11 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'moving s1 watchlist to db1052, db1043 to full weight'
  • 20:09 Ryan_Lane: added marc to the ops ldap group
  • 20:01 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1001'
  • 19:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 19:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:56 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'adding db1028 back'
  • 18:54 notpeter: actually starting innopack from db65 to db1054 for pre-labsdb db (sanitarium)
  • 18:48 notpeter: starting innopack from db31 to db1054 for pre-labsdb db (sanitarium)
  • 18:39 binasher: completed image img_media_mime migration on all projects
  • 18:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 17:42 LeslieCarr: restarted varnishncsa-locke on cp1023
  • 17:41 LeslieCarr: restarted varnishncsa-multicast_relay on cp1028
  • 17:34 cmjohnson1: correction db1028
  • 17:34 cmjohnson1: removing disk 0 from db1048 to replace
  • 17:28 Jeff_Green: disable fundraising banner log rotation on locke
  • 17:21 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'adding db1001 back to production removing db1028'
  • 17:04 cmjohnson1: setting weight to 100 on db1001
  • 15:42 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Message for ancient skin users'
  • 15:41 logmsgbot: anomie synchronized php-1.21wmf12/cache/l10n/l10n_cache-en.cdb
  • 15:34 logmsgbot: anomie synchronized php-1.21wmf12/extensions/WikimediaMessages/WikimediaTemporaryMessages.i18n.php
  • 15:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 2 15:12:45 UTC 2013
  • 15:09 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 2 15:08:37 UTC 2013
  • 14:54 logmsgbot: demon synchronized php-1.21wmf12/extensions/WikimediaMessages/WikimediaTemporaryMessages.i18n.php '8th time is the charm'
  • 14:51 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'I hate l10nupdate'
  • 14:50 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Notice for users of disabled skins'
  • 14:24 paravoid: deploying squid config, diverting all of upload to swift
  • 04:09 binasher: increased 'wikiadmin'@'10.64.%' max connections to 200 on s1
  • 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 2 02:17:33 UTC 2013
  • 02:10 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 2 02:10:24 UTC 2013
  • 01:15 Ryan_Lane: removing labstore1 and labstore2 entries from projectstorage.wmnet rr dns entry in preparation for shrinking volumes
  • 00:44 logmsgbot: reedy synchronized wmf-config/
  • 00:43 binasher: now running the image img_media_mime migration on commons (the big one)
  • 00:41 mutante: restarting lucene on all pool4 servers (one by one)
  • 00:36 mutante: creating search index for transitionteamwiki
  • 00:30 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 00:14 logmsgbot: reedy synchronized php-1.22wmf1/cache/interwiki.cdb 'Updating 1.22wmf1 interwiki cache'
  • 00:10 logmsgbot: reedy synchronized php-1.21wmf12/cache/interwiki.cdb 'Updating 1.21wmf12 interwiki cache'
  • 00:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:

April 1

  • 23:55 LeslieCarr: search1024 is our guinea pig of using sfp-t's and is connected to asw2-a5-eqiad ge 0/0/18
  • 23:48 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:40 mutante: creating swift containers for private wiki
  • 23:39 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
  • 23:37 mutante: dumpInterwiki.php to update interwiki cache
  • 23:36 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic to pmtpa temporarily'
  • 23:33 mutante: adding transitionteamwiki to private.dblist
  • 23:25 mutante: running addWiki.php for transitionteamwiki - Access denied for user ...
  • 23:21 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 23:14 cmjohnson1: swapping disk1 (slot1) db1001
  • 23:06 mutante: creating upload dir on upload7 for transitionteamwiki
  • 22:57 binasher: beginning image.img_media_mime scehma migrations, starting with s1
  • 22:52 logmsgbot: aaron synchronized php-1.22wmf1/includes/job 'deployed ec8f42daf19f386d9af2eece6dfc5f8fa6fc42f3'
  • 22:26 logmsgbot: aaron synchronized php-1.22wmf1/maintenance 'deployed c7832c8956d79892be93452e5f4d6b52df1d3bf0'
  • 22:26 logmsgbot: aaron synchronized php-1.22wmf1/includes/job 'deployed c7832c8956d79892be93452e5f4d6b52df1d3bf0'
  • 22:26 mutante: recreating transitionteam wiki docroot from skel-1.5 and syncing docroot
  • 22:26 logmsgbot: root synchronized docroot
  • 21:09 LeslieCarr: rebooting lvs1001
  • 21:04 logmsgbot: demon synchronized php-1.22wmf1/cache/l10n 'Manually syncing 1.22wmf1 l10ncache'
  • 20:56 LeslieCarr: dist-upgrading lvs1001
  • 20:48 LeslieCarr: failing over lvs from lvs1001 to lvs1004
  • 20:44 LeslieCarr: CORRECTION restarting sshd on lvs1001
  • 20:44 LeslieCarr: restarting lvs1001
  • 20:21 logmsgbot: demon synchronized php-1.21wmf12/cache/l10n 'Manually syncing 1.21wmf12 l10ncache'
  • 20:20 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 1 20:20:29 UTC 2013
  • 20:00 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 19:59:44 UTC 2013
  • 19:54 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 19:54:21 UTC 2013
  • 19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki
  • 19:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki and testwiki to 1.22wmf1
  • 19:11 logmsgbot: reedy synchronized php-1.22wmf1/extensions/ 'Wikidata extensions to same as 1.21wmf12'
  • 19:04 logmsgbot: reedy synchronized php-1.22wmf1/includes/
  • 18:56 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf1 and rebuild l10n cache
  • 18:49 LeslieCarr: testing logbot
  • 18:20 Reedy: Ran scap-recompile to build texvc
  • 18:20 jeremyb_: re!logged from earlier today while morebots was gone (all times UTC)
  • 18:20 jeremyb_: 17:56:29 <logmsgbot> !log reedy synchronized w
  • 18:20 jeremyb_: 17:55:45 <logmsgbot> !log reedy synchronized docroot
  • 18:20 jeremyb_: 17:55:04 <logmsgbot> !log reedy synchronized php-1.22wmf1 'Initial sync of php-1.22wmf1'
  • 18:19 jeremyb_: 17:27:13 <logmsgbot> !log reedy synchronized wmf-config/ExtensionMessages-1.22wmf1.php
  • 18:19 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf1 and rebuild l10n cache
  • 18:19 jeremyb_: 17:14:31 <Jeff_Green> !log moved aside williams:/opt/otrs-home/.spamassassin/auto-whitelist and restarted spamd to test whether the bloated AWL is resulting in poor filtering
  • 18:19 jeremyb_: 14:39:19 <andrewbogott> !log restarted glusterfs-server on labstore1 and labstore2, because of read errors in public/datasets/public in bots project
  • 18:19 jeremyb_: 13:55:28 <ottomata> !log added webstatscollector-0.1-2 to apt repo
  • 18:18 jeremyb_: 11:54:15 <paravoid> !log repooling ms-fe1; staggered depool/restart/pool of ms-fe2-4
  • 18:17 Ryan_Lane: restarted morebots. wikitech-static had 100% / partition. Have also fixed that and its cause.
  • 18:16 jeremyb_: 07:45:24 <paravoid> !log depooling ms-fe1 for staging
  • 18:16 jeremyb_: 03:58:05 <logmsgbot> !log asher synchronized wmf-config/db-eqiad.php 'returning db1051'
  • 18:16 jeremyb_: 03:44:43 <logmsgbot> !log asher synchronized wmf-config/db-eqiad.php 'pulling db1051'
  • 18:16 jeremyb_: 03:22:54 <binasher> !log reinstated max_user_connections = 80 for wikiadmin on s1
  • 18:15 cmjohnson1: restarted morebots
  • 03:23 binasher: reinstated max_user_connections = 80 for wikiadmin on s1
  • 02:12 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 02:12:24 UTC 2013

March 31

  • 02:10 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Mar 31 02:10:18 UTC 2013

March 30

  • 02:14 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Mar 30 02:13:49 UTC 2013
  • 00:01 LeslieCarr: rebuilding mailman archives …. very slowly

March 29

  • 23:51 LeslieCarr: stopping mailman to remove sensitive message
  • 22:38 LeslieCarr: upgrading packages on aluminium
  • 22:25 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
  • 21:02 hashar: jenkins: raised the number of execution for deployment-bastion from 2 to 4. Will let us execute more jobs in parallel
  • 20:14 mutante: scheduling a 2-year downtime for db11 :p
  • 19:33 logmsgbot: dzahn gracefulled all apaches
  • 19:30 logmsgbot: dzahn synchronized ./docroot/transitionteam
  • 19:29 mutante: creating transitionteam docroot
  • 19:25 logmsgbot: aaron synchronized php-1.21wmf12/includes 'deployed fa5c0a6a82c9a1cb63f3cedd87e6c4aba59c994d'
  • 18:49 sbernardin: hot-swapping disk on db29
  • 18:04 ^demon: gerrit back up, now running 2.6-rc0-76-g52fb5ae
  • 18:00 mutante: DNS update - adding transitionteam wiki
  • 17:47 sbernardin: taking down db29 to replace bad hw
  • 17:45 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Re-enabled async upload for commons'
  • 17:23 ^demon: gerrit offline for an update, brb
  • 09:25 logmsgbot: olivneh synchronized wmf-config/throttle.php 'Updating throttle rules for guwiki workshop (Bug 46686)'
  • 04:16 logmsgbot: aaron synchronized php-1.21wmf12/maintenance/runJobs.php 'deployed 651884d4bdc76c29dd717dfdbbd698632223f3b5'
  • 04:15 logmsgbot: aaron synchronized php-1.21wmf12/includes/job/JobQueueGroup.php 'deployed 651884d4bdc76c29dd717dfdbbd698632223f3b5 '
  • 02:17 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Mar 29 02:17:06 UTC 2013
  • 01:24 ori-l: Stopping EventLogging daemons to allow Puppet to change 'eventlogging' user's home dir
  • 00:11 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/MoodBar 'Updating MoodBar to remove ClickTracking integration'
  • 00:03 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/GuidedTour.php 'Small bug fix to GuidedTour; removing unneeded dependency. https://gerrit.wikimedia.org/r/#/c/56546/1/GuidedTour.php'
  • 00:00 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/GuidedTour.php 'Small bug fix to GuidedTour; removing unneeded dependency. https://gerrit.wikimedia.org/r/#/c/56546/1/GuidedTour.php'

March 28

  • 23:24 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Remove E3Experiments and LastModified (2/2)'
  • 23:24 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Remove E3Experiments and LastModified (1/2)'
  • 22:35 logmsgbot: khorn synchronized php-1.21wmf12/includes/cache/MessageCache.php 'Reverting changes in language fallback behavior'
  • 22:34 logmsgbot: khorn synchronized php-1.21wmf12/languages/Language.php 'Reverting changes in language fallback behavior'
  • 22:20 logmsgbot: spage Finished syncing Wikimedia installation... : E3: Updated GettingStarted and GuidedTour extensions
  • 22:12 LeslieCarr: removed some log files from ms1004 - it was full
  • 22:10 RobH: bugzilla admin stuff: noc@w.o is now a disabled account cuz im tired of folks who arent on the alias attempting to sign up with it
  • 22:02 logmsgbot: spage Started syncing Wikimedia installation... : E3: Updated GettingStarted and GuidedTour extensions
  • 21:46 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Fix mobileMode property name (was: 'mobile')'
  • 21:36 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming
  • 20:11 Jeff_Green: add hostname samarium.wikimedia.org and pdns-update
  • 19:51 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1051'
  • 19:44 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1051'
  • 19:44 LeslieCarr: killed gmetad on neon
  • 19:36 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/ArticleFeedback/populateAFRevisions.php 'Fix fatal error triggered by populateAFRevisions.php cronjob when AFT is not loaded.'
  • 19:33 mutante: starting gmetad on neon again
  • 19:24 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo 'syncing Echo on wmf12'
  • 19:17 mutante: manually deleting db11 references from icinga configs and restarting it
  • 19:12 mutante: pkill -u snmptt on neon, restarting snmptt deamon
  • 19:10 binasher: replaced collector on professor with locally built version from I2bc117d5620b9e545a8e5a6cf3b654ee835b75d7 (testing fix for issues only under very high load)
  • 19:10 mutante: stopping gmetad and ganglia-parser on neon
  • 18:53 mutante: adding gjg (greg-g) to wmf ldap group
  • 18:23 mutante: restarting lucene on pool4 search nodes to pickup new wikivoyage indices
  • 18:12 ottomata: adding reportcard and analytics .wikimedia.org CNAMEs to stat1001
  • 18:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:56 mutante: creating search indices for new wikivoyages
  • 17:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 11:49 hashar: search19 and search20 issue with enwiki.prefix is in RT #4845
  • 11:42 hashar: search19 and search20 have enwiki.prefix marked as FAILED. (see: curl --silent http://search19.pmtpa.wmnet:8123/status |grep FAILED and curl --silent http://search20.pmtpa.wmnet:8123/status |grep FAILED).
  • 10:39 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/Translate/ 'Translate to master'
  • 04:19 paravoid: updating jsduck in apt and upgrading it on gallium
  • 03:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 02:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Thu Mar 28 02:19:24 UTC 2013
  • 00:47 K4-713: synchronized payments cluster to 12bc36c2385d8e56792c
  • 00:27 mutante: once again disabling notifications for db11
  • 00:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php

March 27

  • 23:59 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
  • 23:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:40 Ryan_Lane: finished updating wikitech-static
  • 23:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:38 Ryan_Lane: updating wikitech-static to wmf/1.21wmf12
  • 23:24 Ryan_Lane: finished upgrading wikitech
  • 23:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:23 Ryan_Lane: upgrading wikitech to wmf/1.21wmf12
  • 23:17 logmsgbot: reedy synchronized php-1.21wmf12/cache/interwiki.cdb 'Updating 1.21wmf12 interwiki cache'
  • 23:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:50 logmsgbot: maxsem synchronized php-1.21wmf11/extensions/MobileFrontend 'touch'
  • 22:47 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'touch'
  • 22:24 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
  • 22:22 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
  • 22:05 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment, 2nd attempt
  • 21:48 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment, 2nd attempt
  • 21:12 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'changing s1 slave weights for testing'
  • 20:10 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisation cache to fix he related ParserFunctions problems
  • 19:56 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisation cache to fix he related ParserFunctions problems
  • 19:43 logmsgbot: reedy synchronized php-1.21wmf12/extensions/ParserFunctions/
  • 19:38 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for WikiData deployment
  • 19:05 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for WikiData deployment
  • 18:51 cmjohnson1: pushing mw1209-1220 into service
  • 18:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:39 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
  • 18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf12
  • 18:23 mutante: installing package upgrades on neon (Icinga), incl. ganglia-monitor,gmeta
  • 18:09 mutante: bunch of mw* hosts: fix puppet runs
  • 18:08 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo 'syncing Echo for wmf12'
  • 18:05 mutante: mw1130 - kill and fix puppet runs
  • 17:41 Jeff_Green: authdns-update, pdns on ns1 croaked, restarted
  • 15:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: be_x_oldwiki to 1.21wmf12
  • 14:09 hashar: jenkins has been restarted by mistake :-( Job building will be unavailable for up to half an hour.
  • 13:58 hashar: Jenkins: disabled the old Gerrit Trigger Plugin bug 46415
  • 09:56 Nikerabbit: Testing Translate bug fixes on test.wikipedia.org
  • 05:20 Tim: on fluorine: running mw-log-cleanup once to test Id2196e7b
  • 05:02 Ryan_Lane: disabling Bayes-collector cron on sanger
  • 04:55 Ryan_Lane: purging BayesLearning files older than 70 days on mchenry
  • 02:45 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Wed Mar 27 02:44:56 UTC 2013
  • 02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Mar 27 02:19:11 UTC 2013
  • 00:55 LeslieCarr: ms-be3 was crashed with stack traces for swift-container
  • 00:54 LeslieCarr: rebooting ms-be3
  • 00:38 awight: payments cluster updated from 4862fb8eef to f295d851c
  • 00:34 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'db1052 to full weight'
  • 00:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1001, adding db1052 at a warmup weight'
  • 00:11 awight: payments cluster updated from fa2e2 to a06a03f05 wheee!

March 26

  • 23:54 awight: payments cluster rolled back from e758102 to fa2e240b6e9
  • 23:53 awight: payments cluster updated from fa2e240b6e9 to e758102
  • 23:53 Tim: on all apaches: replacing /usr/local/apache/common-local/live-1.5 with a symlink to w
  • 23:44 logmsgbot: tstarling synchronized w
  • 23:35 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/56019/'
  • 23:33 awight: payment cluster rollback, from 4862fb8 to fa2e240b
  • 23:31 mutante: restarting nrpe on professor
  • 23:29 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Workaround for buggy NavTiming implementation in IE9 (bug 46474)'
  • 23:20 awight: payments cluster updated from fa2e240b to 4862fb8
  • 23:10 mutante: db11 - disabling all notifications
  • 23:07 mutante: mw13 - killing puppet and re-running
  • 23:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:00 mutante: removing mysql packages from db1036 (broken dpkg, status "pi")
  • 22:55 mutante: re-installing ganglia-monitor on mw1135, fix dpkg
  • 22:54 logmsgbot: reedy synchronized docroot
  • 22:52 mutante: starting udpprofile (carbon-cache.py) on professor
  • 22:51 logmsgbot: reedy synchronized wmf-config/
  • 22:49 mutante: search21 - remove sun-j2sdk , replaced by oracle-jdsdk
  • 22:47 mutante: search21 - broken DPKG: iF sun-j2sdk1.6
  • 22:45 mutante: mw2 - broken DPKG: pi librsvg2-2 , pi librsvg2-bin
  • 22:39 LeslieCarr: did a service apache2 start on mw108*
  • 22:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:10 notpeter: icinga mostly trustworthy again :)
  • 22:08 andrewbogott: restarted nova-api on virt0 because it appeared to have crashed.
  • 21:31 andrewbogott: restarted memcached on virt0 in a futile attempt to address https://bugzilla.wikimedia.org/show_bug.cgi?id=46583
  • 21:30 mutante: killed stuck snmptt procs on neon to fix the others.. apparently :p
  • 21:28 mutante: stopping gmetad on neon
  • 20:52 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'syncing CommonSettings.php for Echo event tracking'
  • 20:41 mutante: more degraded RAIDs: db1001->RT-4832, db1028->4834
  • 20:35 mutante: broken disks/degraded RAID on db servers: db11->RT-4828, db29->RT-4829, db45->RT-4831
  • 20:34 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/ContributionReporting 'syncing ContributionReporting extension'
  • 20:07 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'raising db1051 to full weight'
  • 19:49 logmsgbot: khorn synchronized php-1.21wmf12/extensions/CentralNotice 'Fixing CentralNotice bug in which it was possible to initialize multiple times'
  • 19:46 logmsgbot: khorn synchronized php-1.21wmf11/extensions/CentralNotice 'Fixing CentralNotice bug in which it was possible to initialize multiple times'
  • 19:40 logmsgbot: kaldari synchronized php-1.21wmf12/resources/startup.js 'syncing php-1.21wmf12/resources/startup.js'
  • 19:33 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase/client
  • 19:27 cmjohnson1: mw1209 re-imaging
  • 19:20 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo/modules/icons 're-syncing Echo icons dir'
  • 19:19 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Include AFTv5 on frwiki'
  • 19:12 logmsgbot: mlitn synchronized wmf-config 'Enable AFTv5 on frwiki'
  • 19:05 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Collection/
  • 19:04 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 19:04 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 19:02 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo/modules/icons 'syncing Echo icons dir'
  • 18:58 mutante: starting gmetad on neon - we failed to resolve data source name ms-fe1001.eqiad.wmnet
  • 18:56 mutante: restarting ircecho on fenari
  • 18:48 mutante: stopping gmetad on neon
  • 18:44 mutante: killing all snmp related processes on neon and restarting
  • 18:43 logmsgbot: reedy synchronized php-1.21wmf12/extensions/MobileFrontend/includes/specials/SpecialMobileWatchlist.php
  • 18:41 logmsgbot: kaldari synchronized php-1.21wmf11/extensions/Echo/modules/icons 'syncing Echo icons dir'
  • 18:39 mutante: restarting snmtrapd on neon
  • 18:25 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Turning Thanks extension on on MediaWiki wiki'
  • 18:13 logmsgbot: krinkle synchronized php-1.21wmf12/resources/mediawiki/mediawiki.js 'I80af730daa815 fixing bug 46575'
  • 18:12 Krinkle: Change I34a488247 (mw.loader debug) has been reverted in wmf/1.21wmf12 and pulled on fenari
  • 17:51 cmjohnson1: removing disk from slot 0 ms-be1012
  • 17:14 Krinkle: Change I34a488247 is now merged in wmf/1.21wmf12 and pulled on fenari for test.wikipedia.org, do not sync.
  • 17:04 ottomata: added (crappy, but better than nothing) .deb for webstatscollector to apt.wikimedia.org
  • 15:24 cmjohnson1: swapping disk 6 ms-be1012
  • 14:34 logmsgbot: hashar synchronized multiversion '47564 wikiversions.cdb now vary by realm'
  • 14:10 cmjohnson1: running sync-common on mw1209
  • 10:23 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/UniversalLanguageSelector/UniversalLanguageSelector.php 'Testing fix for wikidata'
  • 09:28 pp-pdf1: updated mwlib to 0.15.4
  • 08:47 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/Translate
  • 08:46 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/UniversalLanguageSelector
  • 07:27 mark: Restarted swift-proxy on ms-fe*
  • 04:16 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
  • 02:46 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 26 02:45:52 UTC 2013
  • 02:21 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Mar 26 02:21:08 UTC 2013
  • 01:41 mutante: analytics1026 - delete 16G udp2log.log to free disk
  • 01:38 mutante: analytics1003 - gzipping jmxtrans logs to free some disk
  • 01:36 mutante: analytics1003 - gzipping kafka.log to free some disk
  • 00:39 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1051 at a low warmup weight'

March 25

  • 23:36 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'done'
  • 23:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'changing s1 weights, returning db1050'
  • 23:27 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Quick profiling'
  • 22:10 hashar: reloading Zuul
  • 21:44 logmsgbot: khorn synchronized wmf-config/InitialiseSettings.php 'Re-enabling CentralNotice'
  • 21:39 mutante: killing and restarting nagios-nrpe-server on '*'
  • 21:32 logmsgbot: khorn synchronized php-1.21wmf12/extensions/CentralNotice/ 'CentralNotice jQuery fix'
  • 21:28 logmsgbot: khorn synchronized php-1.21wmf11/extensions/CentralNotice/ 'CentralNotice jQuery fix'
  • 21:12 mutante: killing and restarting nagios-nrpe-server on gadolinium
  • 21:07 mutante: --last-lines 20 | RT ..would be nice sometimes:)
  • 20:13 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
  • 20:07 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Use ffmpeg for ogg thumbs as is done for webm'
  • 19:55 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10n cache for wikibase
  • 19:33 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10n cache for wikibase
  • 18:57 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 18:46 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
  • 18:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf12
  • 18:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable CN everywhere'
  • 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rb
  • 18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf12
  • 17:25 logmsgbot: reedy synchronized wmf-config/

March 24

  • 18:13 Ryan_Lane: restarting glusterd processes on labstore*
  • 02:31 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sun Mar 24 02:31:22 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Mar 24 02:08:00 UTC 2013

March 23

  • 02:37 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sat Mar 23 02:36:53 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Mar 23 02:13:44 UTC 2013
  • 00:17 RoanKattouw: Deployed master of Parsoid onto wtp1004; the code is broken but the box is depooled for Gabriel's benchmarking work
  • 00:15 RoanKattouw: That's me messing with wtp1004

March 22

  • 23:33 ottomata: restarting hadoop setting yarn.nodemanager.resource.memory-mb to 16G
  • 23:06 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:04 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:04 mutante: sync-wikiversions
  • 22:29 logmsgbot: aaron synchronized php-1.21wmf12/extensions/Translate 'deployed 0aab5984415a94bd214088767ff1de33ab57c4e0'
  • 22:12 mutante: killall nrpe via salt, then restart nagios-nrpe-server , runs as wrong user
  • 22:09 mutante: kill and restart nrpe on oxygen
  • 22:07 mutante: installing package upgrades on oxygen
  • 21:40 mutante: killing nagios-nrpe on gallium and testing if puppet restores it properly
  • 21:03 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Set all non-Wikipedias back to 1.21wmf12 again.
  • 20:51 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'deployed 0f4914bb4ff2d13cb124cad94dc5ded663123a18'
  • 20:46 logmsgbot: aaron synchronized php-1.21wmf12/includes/WikiPage.php 'deployed e372b635ae6d6a589f6d87f14fe4c9452d7a1b4d'
  • 20:38 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'Reverted follow-up fixes too'
  • 19:32 ottomata: removed unicast udp2logging to gadolinium; gadolinium will consume from multicast stream
  • 18:44 K4-713: synchronized payments cluster to fa2e240b6e92440b
  • 18:28 K4-713: synchronized payments cluster to df3c8dbc937c971
  • 17:33 logmsgbot: aaron synchronized php-1.21wmf12/includes/WikiPage.php 'deployed 1862a57aef08432e8b1194a54dc179649f5cae63'
  • 09:08 MaxSem: Recreating Solr index
  • 02:52 Tim: on searchidx1001: started incremental indexer, apparently it died on March 21 at 02:08 when it ran at the same time as a cron job import
  • 02:50 binasher: power cycling db1050, unresponsive on serial console
  • 02:49 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1050, crashed'
  • 02:47 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
  • 02:42 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Mar 22 02:42:32 UTC 2013
  • 02:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043, oops'
  • 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Fri Mar 22 02:30:31 UTC 2013
  • 01:28 ori-l: Finished correcting E3Experiments extension's entry point from 'Experiments.php' (which does not match the submodule name) to 'E3Experiments.php'. Updated extension-list, CommonSettings.php, and the extension itself in both wmf11 and 12.
  • 01:24 logmsgbot: olivneh synchronized wmf-config/extension-list 'Remove temporary workaround for E3 entry-point migration (2/2)'
  • 01:23 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Remove temporary workaround for E3 entry-point migration (1/2)'
  • 01:18 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/E3Experiments 'Updating E3Experiment extensions's entry point'
  • 01:18 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/E3Experiments 'Updating E3Experiment extensions's entry point'
  • 01:11 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Temporarily accepting either E3Experiments.php or Experiments.php as extension entry point to facilitate clean-up'
  • 00:55 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'NavigationTiming: log mobile mode'
  • 00:16 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.21wmf12.php 'Remove PHP errors'
  • 00:15 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'deployed cef327e945e2593fe0291880a7d1976cc5a2f248'
  • 00:09 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Thanks 'syncing Thanks ext'

March 21

  • 23:24 logmsgbot: robh Finished syncing Wikimedia installation... :
  • 23:17 logmsgbot: robh Started syncing Wikimedia installation... :
  • 23:04 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo, PageTriage to master, Add Thanks
  • 22:44 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, PageTriage to master, Add Thanks
  • 21:50 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Syncing change to CommonSettings for Thanks ext'
  • 21:49 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Syncing change to InitialiseSettings for Thanks ext'
  • 21:46 mutante: running puppet and reloading zuul on gallium. enables hashar's new shiny https://integration.wikimedia.org/zuul/
  • 21:38 RobH: authdns-update for terbium
  • 21:05 RoanKattouw: Added fenari back to /etc/dsh/groups/mediawiki-installation per https://rt.wikimedia.org/Ticket/Display.html?id=4794
  • 21:02 logmsgbot: cmjohnson Finished syncing Wikimedia installation... :
  • 20:58 binasher: pulling mw1070, may have network issues
  • 20:57 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/PageTriage/modules/ext.pageTriage.views.list 'syncing PageTriage js on wmf12'
  • 20:57 hashar: fenari disappeared from dsh group mediawiki-installation :-(
  • 20:52 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 20:51 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 20:48 awight: payments cluster updated from 18fd11a4 to df3c8dbc9
  • 20:43 logmsgbot: cmjohnson Started syncing Wikimedia installation... :
  • 20:37 LeslieCarr: restarting gmond on mw1153
  • 20:19 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 20:19 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming 'Enable NavigationTiming on mobile alpha and beta'
  • 20:19 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 20:19 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Enable NavigationTiming on mobile alpha and beta'
  • 19:53 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/GuidedTour/ 'E3 deploy'
  • 19:52 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/GettingStarted/ 'E3 deploy'
  • 19:52 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/E3Experiments/ 'E3 deploy'
  • 19:49 logmsgbot: mflaschen synchronized wmf-config/CommonSettings.php 'E3 deployment'
  • 17:39 logmsgbot: csteipp rebuilt wikiversions.cdb and synchronized wikiversions files: Update test2wiki to 1.21wmf12
  • 17:32 logmsgbot: csteipp synchronized php-1.21wmf12/includes/Title.php 'update for loadPageData()'
  • 17:24 notpeter: running puppet on all lucid nodes via salt
  • 17:23 mutante-away: purging wikispecies.net from squid
  • 17:08 notpeter: install timeout package on all lucid nodes to get puppet back on track
  • 12:49 mark: Rebooting palladium with hyperthreading disabled
  • 11:27 mark: Rebooting niobium with hyperthreading disabled
  • 09:29 hashar: created git repo operations/debs/python-statsd
  • 09:15 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'wikidata.org' to $wgCrossSiteAJAXdomains to allow CORS'
  • 09:14 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'wikidata.org' to $wgCrossSiteAJAXdomains to allow CORS'
  • 06:34 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 06:34 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 06:32 Tim: reverting all wikis to 1.21wmf11 due to move page being totally broken (bug 46397)
  • 06:32 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 06:32 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 01:10 logmsgbot: catrope synchronized php-1.21wmf12/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
  • 01:10 logmsgbot: catrope synchronized php-1.21wmf12/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
  • 01:10 logmsgbot: catrope synchronized php-1.21wmf11/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
  • 01:10 logmsgbot: catrope synchronized php-1.21wmf11/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'

March 20

  • 23:57 RobH: yay peter!
  • 23:56 notpeter: freeing up roughly 500G of ram across our cluster by calling puppet via cron :)
  • 23:52 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es100[69]'
  • 23:52 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es100[69]'
  • 23:51 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s2 servers'
  • 23:51 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s2 servers'
  • 23:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling es100[69]'
  • 23:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling es100[69]'
  • 23:36 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s1 servers'
  • 23:35 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s1 servers'
  • 23:31 RobH: authdns-update for lanthanum
  • 22:45 RobH: ignore mw1209+ they arent deployed
  • 22:04 LeslieCarr: restarting ircecho on fenari to ensure logmsgbot is on operations and tech irc channels
  • 22:03 ottomata: stopping puppet on gadolinium to test for packet loss weirdness
  • 22:01 pgehres: pgehres synchronized php-1.21wmf12/includes/GlobalFunctions.php 'Fix wfWaitForSlaves() so the parameter actually works correctly'
  • 22:01 logmsgbot: pgehres synchronized php-1.21wmf11/includes/GlobalFunctions.php 'Fix wfWaitForSlaves() so the parameter actually works correctly'
  • 21:35 LeslieCarr: restarted ircecho on fenari
  • 21:01 Krinkle: asher synchronized wmf-config/db-eqiad.php 'pulling db1049 from s1'
  • 20:53 pgehres: pgehres synchronized php-1.21wmf11/extensions/CentralAuth 'Updating CentralAuth to master'
  • 20:53 pgehres: pgehres synchronized php-1.21wmf12/extensions/CentralAuth 'Updating CentralAuth to master'
  • 20:42 Reedy: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:59 mutante: DNS update - remove zuul CNAME
  • 19:53 Reedy: wikibooks to 1.21wmf12
  • 19:51 Reedy: sync'd wmf-config/InitialiseSettings.php
  • 19:50 Reedy: wikiversity and wikiquote to 1.21wmf12
  • 19:49 Reedy: wikivoyage and wiktionary to 1.21wmf12
  • 19:47 Reedy: wikinews and wikisource to 1.21wmf12
  • 19:47 Reedy: Synchroni[sz]ed php-1.21wmf12/extensions for Wikidata deploy
  • 19:46 Reedy: special and wikimedia dblists to 1.21wmf12
  • 19:46 Reedy: wikimania, private and fishbowl dblists to 1.21wmf12
  • 19:45 Reedy: closed.dblist to 1.21wmf12
  • 19:38 RobH: sync-common on mw23
  • 19:36 mutante: restarting hung apache on mw1202
  • 19:28 LeslieCarr: sync-common on mw1085
  • 19:24 mutante: .tar.gz'ing /var/mwdocs on formey (SVN) and then manually delete it
  • 19:18 LeslieCarr: upgrading mw118* and then restarting apache2 on these boxes
  • 19:11 LeslieCarr: installing package upgrades on mw1170-1179
  • 19:10 mutante: installing package upgrades on formey
  • 18:50 mutante: apache-graceful-all, remove boards.wm redirect
  • 16:03 ottomata: added python-voluptuous package to apt repo
  • 15:12 mark: Fixed purging for all sites but eqiad (which wasn't broken)
  • 07:56 hashar: gallium: reloaded apache2 to make sure recent changes are applied.
  • 02:44 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Mar 20 02:44:03 UTC 2013
  • 02:32 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Wed Mar 20 02:32:16 UTC 2013

March 19

  • 23:52 K4-713: synchronized payments cluster to fe4fb96bea1378e
  • 23:33 mutante: DNS update - add wikiartpedia.org
  • 23:30 logmsgbot: dzahn gracefulled all apaches
  • 23:08 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Re-enable AFTv5 on dewiki/testwiki'
  • 22:38 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
  • 22:36 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
  • 22:33 ottomata: adding udp-filter_0.3.22 for precise to apt repo (lucid version already was there)
  • 22:17 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
  • 22:15 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
  • 21:52 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 21:52 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 21:37 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 21:25 mutante: installing package upgrades on kaulen
  • 21:24 mutante: installing package upgrades on sockpuppet
  • 21:13 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 20:36 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5/SpecialArticleFeedbackv5.php 'Touched AFT Special page file'
  • 20:35 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/SpecialArticleFeedbackv5.php 'Touched AFT Special page file'
  • 19:42 ottomata1: added gadolinium as a udp2log host on frontend caches
  • 19:14 logmsgbot: mlitn synchronized wmf-config/CommonSettings.php 're-include AFTv5'
  • 19:08 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 19:07 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 19:03 paravoid: adding ruby-multi-json (universe), ruby-parallel, ruby-dimensions, ruby-execjs, ruby-jsduck (main) to apt
  • 18:46 mutante: dist-upgrade on gallium
  • 18:03 logmsgbot: robh synchronized docroot
  • 18:01 logmsgbot: robh synchronized docroot
  • 17:21 mark: Ended bits.esams performance testing
  • 16:29 mark: Starting load testing with esams bits servers
  • 16:20 mark: Rebooted cp3019 with hyperthreading and DAPC disabled
  • 16:16 logmsgbot: krinkle synchronized extract2.php
  • 14:29 mark: Rebooting arsenic with hyperthreading disabled
  • 12:54 Tim: on professor: restarted collector
  • 08:46 hashar: gallium: killed stalled puppet agents. Locked on apt-get and update-java-alt
  • 08:41 hashar: gallium: restarted puppet and zuul
  • 08:38 hashar: gallium: stopped zuul and upgrading
  • 08:37 hashar: gallium: stopped puppet to deploy Zuul whenever the queue is empty
  • 08:36 hashar: zuul : giving results processing priority over enqueued jobs. Cherry picked 263fba9 from upstream. That should resolve Zuul being slow to report back to Gerrit bug 46176
  • 03:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 03:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 03:12 Tim: started incremental updater on searchidx1001 and searchidx2, apparently has not been running since March 13
  • 02:50 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Mar 19 02:50:19 UTC 2013
  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 19 02:29:02 UTC 2013
  • 01:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 00:22 Ryan_Lane: adding dns entries for labstore systems in eqiad
  • 00:08 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning servers'

March 18

  • 23:34 binasher: i <3 running "dpkg -P mysql-server-5.1 mysqlfb-client-5.1 mysqlfb-common mysqlfb-server-5.1 mysqlfb-server-core-5.1 libmysqlfbclient16 libmysqlclient16"
  • 23:11 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db10[10-11],[26-28] at low weights'
  • 22:45 binasher: converted rogue s3 tables (moodbar_feedback, trackbacks, transcache) to innodb
  • 22:28 ottomata: added python-flask-login-0.1.2-1 to apt
  • 22:23 logmsgbot: robh gracefulled all apaches
  • 22:20 LeslieCarr: intradatacenter link is flapping , switching links, this may cause some higher latency
  • 22:14 logmsgbot: robh gracefulled all apaches
  • 22:11 logmsgbot: robh gracefulled all apaches
  • 21:58 mutante: purging wikimediafoundation.org from squid
  • 21:57 logmsgbot: dzahn gracefulled all apaches
  • 21:56 logmsgbot: robh gracefulled all apaches
  • 21:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling a db from s3-7 for upgrade'
  • 21:30 paravoid: authdns-update: removing *.ts.wikimedia.org records
  • 21:29 logmsgbot: dzahn gracefulled all apaches
  • 21:25 binasher: ran "ddsh -g apaches -cM '/etc/init.d/apache2 start'"
  • 21:24 RobH: authdns-update rt4683
  • 21:22 RobH: gracefuling all apaches resulted in rendering cluster overload, they didnt restart apache, odd
  • 21:21 logmsgbot: robh synchronized docroot
  • 21:18 LeslieCarr: service apache2 start on the mw-eqiad group via dsh
  • 21:14 binasher: started apache on all imagescalers
  • 21:00 logmsgbot: robh gracefulled all apaches
  • 21:00 RobH: troubleshooting erros in apache restart script
  • 20:59 logmsgbot: robh gracefulled all apaches
  • 20:36 RobH: authdns-update rt4694
  • 20:20 RobH: authdns-update
  • 20:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 20:00 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12
  • 19:44 ^demon: jetty freaked out again, forcing a gerrit restart
  • 19:32 binasher: upgraded all coredb mariadb replicas to 5.5.30
  • 19:25 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12
  • 19:08 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043 db1009'
  • 19:04 logmsgbot: reedy synchronized docroot
  • 19:03 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'db1050 for special s1'
  • 18:56 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043, db1009 for upgrade'
  • 18:40 logmsgbot: reedy synchronized php-1.21wmf12 'Initial sync of php-1.21wmf12'
  • 18:16 awight: payments cluster updated from c19cc66 to fe4fb96b
  • 18:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 18:03 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'Removed old testing cruft'
  • 16:19 cmjohnson1: removing/replacing disk 4 ms-be1004
  • 16:01 hashar: Hard restarted Zuul which was dead locked again :-(
  • 13:23 paravoid: depooled, upgraded, restarted and pooled again all ms-fe[1-4]
  • 13:02 paravoid: depooling ms-fe1 for new swift version (patched 1.7.4)
  • 11:27 apergos: restarted both the varnishncsas on niobium, they were giant again
  • 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Mon Mar 18 02:30:03 UTC 2013

March 17

  • 02:31 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sun Mar 17 02:31:16 UTC 2013

March 16

  • 14:58 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
  • 14:48 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
  • 14:40 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
  • 13:07 hashar: Mar 16, 2013 1:06:47 PM Jenkins is fully up and running
  • 12:55 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/EducationProgram/includes/pages/EducationPage.php 'Fix fatal error in EducationProgram (Bug 46188)'
  • 12:33 hashar: restarting Jenkins on a quiet day to have it using umask 0002 (aka g+w). That needs a full restart since the umask is set by /usr/bin/daemon See 53990.
  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sat Mar 16 02:29:19 UTC 2013

March 15

  • 23:24 logmsgbot: laner synchronized wmf-config/CommonSettings.php 'Disabling aft5'
  • 22:45 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es1007,10 at lower weights'
  • 22:35 binasher: rebooted es1007,1010 for kernel upgrades
  • 22:33 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'temporarily pulling es1007, es1010'
  • 22:23 Ryan_Lane: hid domain dropdown for account creation and login on wikitech
  • 22:23 Ryan_Lane: changed wikitech configuration to require email address on account creation
  • 22:23 Ryan_Lane: deployed change 54093 to wikitech
  • 22:03 hashar: Zuul : opened a bug about this week slowness, breakage: https://bugzilla.wikimedia.org/show_bug.cgi?id=46176
  • 21:51 logmsgbot: aaron synchronized php-1.21wmf11/includes 'deployed 08295adcdc4a4130954f52fb1eb5377638cd97bc'
  • 21:36 binasher: db59 -> mariadb 5.5.30
  • 21:30 hashar: restarted Zuul with the safe version ff79197
  • 21:28 hashar: downgrading Zuul to safe version ff79197 (aka stop doing `git remote update` on ref-update events)
  • 21:25 binasher: added new build of mariadb 5.5.30 to precise-wikimedia on brewster
  • 21:20 hashar: hard restarted Zuul. It was locked down again :/
  • 21:18 mutante: installing package upgrades on gallium
  • 21:14 hashar: restarting Zuul, it is locked somehow. Definitely need to update it :(
  • 21:09 RobH: professor services seem back up
  • 20:51 RobH: professor shutting down for fan swap, short downtime (hopefully)
  • 20:47 hashar: integration.mediawiki.org is not redirecting to the new integration.wikimedia.org (apache conf is disabled). Patch in 54068
  • 20:33 mutante: puppet runs on neon broken due to mysql puppet issue - mysql-client-5.5 can not create alias mysql-client: object already exists at /var/lib/git/operations/puppet/manifests/mysql.pp:513
  • 20:31 mutante: restarting pdns on ns0
  • 20:30 mutante: DNS update - removing ipv4/ipv6/ipv6and4/results.labs.wm
  • 19:37 mutante: maerlant - re-adding 2620:0:862:1::80:2 to eth0, starting nginx
  • 19:22 sbernardin: taking down raskin for decommissioning
  • 19:13 mutante: maerlant - dpkg: error processing ganglia-monitor | backing up /var/www/ to tridge /data/other then deleting contents | removing sites-enabled/ipv6and4 | nginx can't bind to [2620:0:862:1::80:2]:443
  • 18:50 mutante: installing more package upgrades on maerlant (wikimedia-lvs-realserver, libc6, nginx...)
  • 18:46 mutante: installing package upgrades on formey
  • 18:44 mutante: fixing broken dpkg/APT on maerlant - dpkg --configure -a , re-run puppet and let it install package upgrades
  • 18:34 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
  • 18:19 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
  • 18:19 notpeter: ran puppetd --enable on all mw hosts
  • 18:15 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
  • 18:07 logmsgbot: mlitn Finished syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
  • 17:52 logmsgbot: mlitn Started syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
  • 17:36 hashar: jenkins: updated jenkins configuration to have its URL default to wikimedia.org domain instead of the legacy mediawiki.org
  • 17:22 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/ 'Update ArticleFeedbackv5 to master'
  • 17:21 hashar: gallium : reloading Zuul so it reports the status url with the wikimedia.org domain
  • 17:19 hashar: integration website going down while we deploy a new layout.
  • 17:03 hashar: gallium : restarted puppet
  • 17:03 hashar: gallium : update /srv/ to latest version of integration/docroot.git ( aaa4b0c ) and moved all content from /srv/org/mediawiki/integration to the new /srv/org/wikimedia/integration
  • 17:00 hashar: stopped puppet on gallium to fix some permissions before puppet changes are merged in and applied.
  • 05:09 Tim: restarting squid on cp1010 for squid redirector test
  • 05:06 Tim: testing new squid redirector
  • 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Fri Mar 15 02:30:02 UTC 2013
  • 02:16 logmsgbot: reedy synchronized docroot
  • 02:13 logmsgbot: dzahn gracefulled all apaches
  • 01:53 logmsgbot: reedy synchronized wmf-config/
  • 01:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 01:52 logmsgbot: reedy gracefulled all apaches
  • 01:44 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/GettingStarted/SpecialGettingStarted.php 'Fix deployed bug in check for editability'
  • 01:44 logmsgbot: reedy synchronized wmf-config/
  • 01:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 01:35 logmsgbot: reedy synchronized wmf-config/
  • 01:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 01:34 logmsgbot: reedy synchronized docroot
  • 01:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 01:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 01:21 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/GettingStarted/CategoryRoulette.php 'Fix to use single-parameter version of SRANDMEMBER'
  • 01:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 01:13 logmsgbot: dzahn gracefulled all apaches
  • 01:12 logmsgbot: reedy synchronized docroot
  • 01:10 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Missed a file in earlier sync'
  • 00:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'chapcomwiki readonly'
  • 00:47 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Additional fields for NavTiming'
  • 00:43 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deployment for Mobile and E3
  • 00:17 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deployment for Mobile and E3

March 14

  • 23:48 mutante: DNS update - add affcom in prep. to replace chapcom (bug 39482)
  • 23:47 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Updating MF copyright logo path for enwiki'
  • 23:47 logmsgbot: awjrichards synchronized php-1.21wmf11/extensions/MobileFrontend/ 'Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-03-14'
  • 23:44 mutante: installing package upgrades on sockpuppet
  • 23:30 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Updating MF copyright logo for testwiki'
  • 22:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:43 mutante: pa.us.wikimedia.org now redirects to pa-us.wikimedia.org (Wikimedia Pennsylvania) (no sub.sub domains)
  • 22:41 logmsgbot: dzahn gracefulled all apaches
  • 22:34 mutante: disabling notifications for puppet freshness on celsus
  • 22:14 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling NavigationTiming at 1:10,000 sampling factor across cluster.'
  • 22:02 logmsgbot: olivneh synchronized wmf-config 'Enable NavigationTiming on test2'
  • 21:57 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Update of extension to use EventLogging'
  • 21:51 hashar: Gerrit took roughly 5 minutes to merge the change https://gerrit.wikimedia.org/r/#/c/53412/ ( mediawiki/core ) which means Zuul does nothing during that time waiting for the merge to happen.
  • 21:48 hashar: apparently Gerrit takes ages to submit a change that in turn cause a hard lock on Zuul which is waiting for the merge to happen.
  • 21:39 hashar: Zuul is hanging out not reporting changes :/
  • 21:01 mutante: DNS update - add pa-us wiki to replace pa.us
  • 20:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:50 cmjohnson1: authdns update (labsdb1002-3 vlan)
  • 20:45 Ryan_Lane: deleting the jenkins user from ldap
  • 20:36 mutante: DNS update - add zuul as CNAME for gallium
  • 20:34 Ryan_Lane: installed WikiLove on wikitech
  • 20:30 ori-l: Added article edit event to EventLogging, which upped events/sec from ~3.5 to ~30. Expect increased load on vanadium and db1047; bits caches unaffected.
  • 20:20 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/EventLogging 'Log ['HTTP_HOST'] as webHost'
  • 20:05 logmsgbot: olivneh synchronized php-1.21wmf11/includes/WikiPage.php 'Adds CategoryAfterPageAdded / CategoryAfterPageRemoved hooks'
  • 19:32 logmsgbot: aaron synchronized php-1.21wmf11/includes/job/jobs 'deployed 6f76ede163cb114724bce7a1c1b8938e1e30606f '
  • 18:49 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Enable wgEnableAsyncUploads only on testwikis for now'
  • 18:44 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/ 'Update ArticleFeedbackv5 to master'
  • 18:41 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1009 to full weight'
  • 18:32 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1009 at a low weight for warmup'
  • 18:12 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Enabled $wgEnableAsyncUploads on all wikis'
  • 18:08 binasher: cleaned up myisam cruft (unused t���ables) on s2
  • 17:49 logmsgbot: aaron synchronized wmf-config 'Enabled redis job aggregator fully'
  • 17:48 cmjohnson1: labsdb1001 powering down
  • 17:42 logmsgbot: aaron synchronized wmf-config 'Set redis job aggregator on for testwiki.'
  • 17:27 logmsgbot: reedy synchronized php-1.21wmf11/extensions/Math/
  • 17:24 binasher: shutting down mysql on db1009, upgrading to mariadb
  • 16:59 cmjohnson1: powering down labsdb1002 and labsdb1003
  • 16:59 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1009 for mariadb migration'
  • 16:57 cmjohnson1: addin mw1041 back to dsh groups
  • 16:26 logmsgbot: mlitn synchronized wmf-config 'Prepare AFTv5 config for deployment new features'
  • 13:15 mark: Set OSPF cost to 10 on csw1-esams:ve7 to facilitate ip multipath
  • 13:15 mark: Fixed OSPF on cr2-knams:xe-0/0/0 - csw1-esams:e8/2 (earlier)
  • 12:17 hashar: gallium setting up /srv/ as a copy of integration/docroot.git
  • 09:03 apergos: restarted varnishncsa.vanadium on niobium, it was using 9gb memory or so
  • 02:25 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Thu Mar 14 02:25:19 UTC 2013
  • 02:21 hashar: jenkins: made Zuul to block changes on operations/debs/ircecho whenever pep8/pyflakes fails.
  • 01:35 mutante: running puppet on celsus to investigate puppet freshness reports from icinga - it started parsoid, but also celsus is a Wikimedia DECOMMISSIONED server (base::decommissioned).
  • 01:10 logmsgbot: reedy synchronized php-1.21wmf11/cache/l10n/
  • 01:00 logmsgbot: reedy synchronized php-1.21wmf11/extensions/Scribunto
  • 00:53 logmsgbot: reedy synchronized wmf-config/

March 13

  • 23:44 mutante: installing package upgrades on zirconium
  • 23:41 logmsgbot: demon Finished syncing Wikimedia installation... :
  • 23:38 Ryan_Lane: deploying change 53464 to OpenStackManager on wikitech
  • 23:23 RobH: no one sync apaches, we wre doing crazy magic on them
  • 23:17 logmsgbot: demon Started syncing Wikimedia installation... :
  • 23:05 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad. upgrades over.'
  • 23:00 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad. upgrades over.'
  • 22:53 logmsgbot: reedy synchronized wmf-config/
  • 22:44 logmsgbot: demon synchronized php-1.21wmf11/extensions/Scribunto/ 'Updating Scribunto to master'
  • 22:30 logmsgbot: robh gracefulled all apaches
  • 21:46 awight: payments cluster updated to 61d3f1f
  • 21:43 pp-pdf1: upgraded mwlib to 0.15.3
  • 21:38 Tim: on professor: restarted collector
  • 21:22 RobH: repooling all ssl servers (except broken ssl3004) in all sites. new unified certificate is now in use.
  • 21:15 RobH: depooling ssl servers 3 and 4 in each dc site
  • 21:14 RobH: repooling the first two ssl servers per site
  • 21:08 logmsgbot: py synchronized wmf-config/lucene-production.php 'temp moving all earch traffic to pmtpa for upgrades in eqiad'
  • 21:05 RobH: setting first 2 ssl servers per dc site to false in pybal
  • 20:21 Ryan_Lane: repooling ssl1001
  • 20:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable collection'
  • 20:16 Ryan_Lane: depooling ssl1001
  • 20:08 LeslieCarr: cp1022 unresponsive, power cycling
  • 19:44 LeslieCarr: switched preferred path between esams and eqiad
  • 19:41 LeslieCarr: deploying a possibly mobile front end traffic effecting ipv6 apache request firewall filter
  • 19:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable Collection extension'
  • 19:08 LeslieCarr: site was popedotted !
  • 18:47 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqaid'
  • 18:31 logmsgbot: py synchronized wmf-config/lucene-production.php 'temp moving all search traffic to pmtpa for upgrades in eqiad'
  • 18:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable scribunto EVERYWHERE'
  • 18:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf11
  • 17:23 pp-pdf1: downgrade to 0.14.3
  • 16:39 pp-pdf1: upgraded mwlib to 0.15.1.
  • 16:32 apergos: cleaned up borked puppet agent on ms-be12 and re-enabled/restarted it
  • 12:30 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto/common/Hooks.php 'Fix for bug 46031'
  • 12:30 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto/common/Common.php 'Fix for bug 46031'
  • 12:07 Danny_B: reenabled mail delivery to wikibugs 30 mins ago (disabled automatically for excessive bounces)
  • 11:42 apergos: restarted morebots
  • 01:10 Tim: running updateCollation.php on all uca-* wikis in 6 screens on hume
  • 00:45 logmsgbot: dzahn gracefulled all apaches
  • 00:39 Ryan_Lane: depooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
  • 00:35 Ryan_Lane: pooling ssl3 ssl4 ssl1003 ssl1004 ssl3003
  • 00:31 Ryan_Lane: depooling ssl3 ssl4 ssl1003 ssl1004 ssl3003
  • 00:30 Ryan_Lane: pooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
  • 00:16 Ryan_Lane: depooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
  • 00:12 Tim: manually creating missing /etc/apache2/wmf symlinks on servers that lack them
  • 00:07 logmsgbot: dzahn gracefulled all apaches

March 12

  • 23:39 Tim: updating PHP to php5-3.5+wmf1 for new ICU
  • 23:39 notpeter: upgrading pmtpa search nodes to lucene-search-2 2.1.7wm1
  • 23:34 logmsgbot: tstarling synchronized php-1.21wmf11/maintenance/updateCollation.php
  • 23:34 logmsgbot: tstarling synchronized php-1.21wmf10/maintenance/updateCollation.php
  • 22:33 Ryan_Lane: pooling ssl3003 ssl3 ssl4 ssl1003 ssl1004
  • 22:29 Ryan_Lane: depooling ssl3003 ssl3 ssl4 ssl1003 ssl1004
  • 22:28 Ryan_Lane: pooling ssl3001 ssl3002 ssl1 ssl2 ssl1001 and ssl1002
  • 22:23 mutante: pushing new version 2.7 of wikimedia-task-appserver to wikipedia-precise repo (to fix apache-sanity-check)
  • 22:21 Ryan_Lane: depooling ssl3001 ssl3002 ssl1 ssl2 ssl1001 and ssl1002
  • 21:23 Ryan_Lane: seems that the unified certificate is missing mediawiki.org, depooling the ssl servers that are serving it
  • 21:22 Ryan_Lane: installing the new unified ssl certificate on all ssl servers
  • 19:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:54 RobH: authdns-update for europium to gadolinium ip
  • 18:47 RobH: europium shuttin down
  • 18:43 mark: Imported ganglia 3.5.0 packages into the precise-wikimedia APT repository
  • 18:19 RobH: europium reclaimed per rt4689
  • 18:02 RobH: en-wp.com/org now redirect to en.wikipedia per rt 4669
  • 18:00 RobH: ran apache restart as myself, not root, had a bunch of fails. restarted them as root, hopefully nothing noticable to users
  • 17:58 logmsgbot: robh gracefulled all apaches
  • 17:53 Jeff_Green: live-hack trivial OTRS patch to support filtering on X-Spam-Score for RT #4713, will check in once project is copied to git
  • 17:42 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto 'Update Scribunto to master'
  • 17:29 logmsgbot: reedy synchronized php-1.21wmf10/includes/Collation.php
  • 17:28 logmsgbot: reedy synchronized php-1.21wmf11/includes/Collation.php
  • 15:42 paravoid: apache graceful to fix bz 46018 / RT 4695
  • 15:39 logmsgbot: root gracefulled all apaches
  • 15:32 hashar: Restarted Zuul process. Was stuck somehow waiting for a job that has been canceled.
  • 13:30 hashar: Jenkins restarted!
  • 12:55 hashar: gallium : upgrading Jenkins (unscheduled)
  • 08:32 hashar: jenkins: live hacked the ant build script for MediaWiki jobs. Now using the local replicated git directory instead of a handmade replication.
  • 05:03 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Math
  • 05:02 logmsgbot: tstarling synchronized php-1.21wmf11/extensions/Math
  • 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Mar 12 02:52:43 UTC 2013
  • 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 12 02:29:54 UTC 2013
  • 02:16 Ryan_Lane: deploying change 53248 and 52951 to OpenStackManager on wikitech
  • 02:10 Ryan_Lane: deployed change 53304 to OATHAuth on wikitech
  • 00:38 logmsgbot: kaldari synchronized php-1.21wmf11/extensions/UploadWizard/ 'deploying some bugfixes for UploadWizard'
  • 00:15 awight: payments cluster updated to 207a112b5b36a9e125999acc6f4098d9058ef594

March 11

  • 23:48 AaronSchulz: syncFileBackend.php Running for ceph for all wikis now
  • 23:48 AaronSchulz: copyFileBackend.php finished for ceph for all originals and transcodes (plus math/captcha)
  • 23:38 logmsgbot: aaron synchronized php-1.21wmf11/includes/clientpool/RedisConnectionPool.php 'deployed 5af8702a4b4208ac42fab2fb36da69d13f3e3b92'
  • 23:37 logmsgbot: aaron synchronized php-1.21wmf10/includes/clientpool/RedisConnectionPool.php 'deployed b9fdedc57295a4dd1da684859871aeb734bf70fe'
  • 23:10 Ryan_Lane: adminbot upgraded to 1.6.3
  • 23:09 Ryan_Lane: upgrading adminbot on wikitech-static
  • 22:31 LeslieCarr: upgrading ircecho on neon
  • 22:30 Ryan_Lane: upgrading ircecho on manganese
  • 21:20 mutante: gracefulling eqiad Apaches - wikiquote.net now redirects to wikiquote.org
  • 21:19 logmsgbot: dzahn gracefulled all apaches
  • 21:07 Ryan_Lane: deployed change 53267 to OpenStackManager on wikitech
  • 20:59 logmsgbot: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
  • 20:59 logmsgbot: reedy synchronized php-1.21wmf11/cache/interwiki.cdb 'Updating 1.21wmf11 interwiki cache'
  • 20:30 ^demon: Upgraded gerrit to 2.5.2-1636-g353e384, all's well
  • 20:18 Ryan_Lane: deploying change 52950 to OpenStackManager on wikitech
  • 19:22 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf11
  • 19:05 mutante: patching Bugzilla for security - RT-4553
  • 17:45 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:34 mwalker: Shields up! Payments deploy from a6fe052a73f1421028b44305908ec6984dbc50fa to head for Japanese localization updates.
  • 17:29 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:19 RobH: search23 shutting down for mb sap per rt4323, has been disabled in pybal
  • 17:18 RobH: search23 out of search pool1, other servers in pool online
  • 17:17 RobH: setting search23 to false for mb swap
  • 17:10 sbernardin: taking down search23 for MB replacement: https://rt.wikimedia.org/Ticket/Display.html?id=3423
  • 16:48 logmsgbot_: aaron synchronized php-1.21wmf11/includes/limit.sh 'deployed e77397fc1de324dfc6d7b515e9440bd564311dd9'
  • 15:57 ottomata: deployed new webstatscollector filter on locke. This replaces spaces in urls.
  • 11:39 apergos: ook that was wrong started apache2 on stafford (puppet uses passenger), now things look ok
  • 10:51 apergos: that would be 10.5. can't add.
  • 10:50 apergos: shot old puppet agents hanging around on stafford, also old apaches from dead puppet master, restarted puppet master, it died around 6.5 hours ago
  • 02:52 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Mon Mar 11 02:52:06 UTC 2013
  • 02:43 logmsgbot_: krinkle synchronized wmf-config/CommonSettings.php 'I7c8174bcf6ca411014a712a6416d3a74e9fecb3d'
  • 02:41 logmsgbot_: krinkle synchronized wmf-config/InitialiseSettings.php 'I7c8174bcf6ca411014a712a6416d3a74e9fecb3d'
  • 02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Mon Mar 11 02:29:27 UTC 2013

March 10

  • 22:19 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild message cache
  • 21:59 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild message cache
  • 21:10 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/CentralAuth/
  • 21:00 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php

March 9

  • 02:54 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Sat Mar 9 02:54:23 UTC 2013
  • 02:37 mutante: deploying blog WMBlog.php changes (gerrit 28835, gerrit 29823)
  • 02:33 Reedy: Created cn_notice_log on enwiki. Not sure it's actually needed, but there's dberrors appearing here and htere
  • 02:31 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Sat Mar 9 02:31:06 UTC 2013
  • 02:01 mutante: deploying WMBlog.php changes for EventLogging (gerrit 52945)
  • 01:26 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php '(Change If7ab3d016 / bug 45830) Separate slow parse log destination for private wikis'
  • 01:13 Ryan_Lane: cherry-picking in change 52950 on wikitech
  • 01:12 Ryan_Lane: scratch that
  • 01:11 Ryan_Lane: deploying OATHAuth master to wikitech
  • 00:32 mutante: deleting squid logs on brewster to prevent out-of-disk, and setting logrotate to compress
  • 00:25 Ryan_Lane: upping OpenStackManager to master on wikitech
  • 00:22 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'removing a second node per shard for streaming dumps. stopping slave'
  • 00:00 Ryan_Lane: upping OpenStackManager to master on wikitech

March 8

  • 22:28 logmsgbot_: reedy synchronized wmf-config/
  • 22:28 logmsgbot_: py synchronized wmf-config/lucene-production.php 'moving pool4 and pool5 search traffic back to eqiad'
  • 22:21 logmsgbot_: reedy synchronized php-1.21wmf10/includes/Collation.php
  • 21:47 awight: payments cluster update to a6fe052a73
  • 20:29 Ryan_Lane: enabling quorum feature for all gluster volumes
  • 19:55 RobH: authdns-update, why? because i can.
  • 19:34 Ryan_Lane: added Babel extension to wikitech
  • 19:20 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'removing a second node per shard for streaming dumps. stopping slave'
  • 15:35 ottomata: adding metrics.wikimedia.org domain, running authdns-update on dobson
  • 06:21 Tim: updated luasandbox to version 1.4 (with pauseUsageTimer)
  • 04:01 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Translate
  • 02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Fri Mar 8 02:52:54 UTC 2013
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Fri Mar 8 02:28:34 UTC 2013
  • 01:27 mutante: installing package upgrades on ekrem
  • 01:12 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php '20388a06 - Normalize VE config on testwiki'
  • 00:57 mutante: added Set(@ReferrerWhitelist, qw(rt.wikimedia.org:443 rt.wikimedia.org:80));
  • 00:55 Ryan_Lane: deploying change 52592 to wikitech
  • 00:55 mutante: killall lighttpd and restart lighttpd and apache on streber
  • 00:19 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
  • 00:06 mutante: installing openjdk/icedtea jre and other package upgrades on gallium (jenkins)
  • 00:05 LeslieCarr: upgrading cp1001

March 7

  • 23:54 logmsgbot_: kaldari Started syncing Wikimedia installation... :
  • 23:47 LeslieCarr: doing tmh upgrades
  • 23:47 mutante: bravely rebooting streber - expect short RT and Observium outage
  • 23:25 LeslieCarr: upgrading solr boxes
  • 23:14 mutante: installing kernel on streber
  • 23:11 ^demon: gerrit: disabling hooks-its plugin for the time being. It's working (yay) but it's overly spammy until we fix it next week.
  • 23:04 LeslieCarr: dist-upgrading pmtpa memcache
  • 23:04 mutante: dist-upgrading zhen
  • 22:38 mutante: rebooting yvon
  • 22:34 mutante: installing package upgrades on yvon despite (defunct apt-get procs)
  • 22:31 mutante: salt pkg.upgrade on yvon
  • 22:28 Ryan_Lane: rebooting marmontel
  • 22:25 mutante: salt pkg.upgrade on spence
  • 22:21 mutante: salt pkg.upgrade on zirconium
  • 22:18 LeslieCarr: upgrading all analytics machiens
  • 22:06 mutante: reboot formey after dist-upgrade via salt
  • 20:05 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'fa0a56a89 - parsoidcache VIP in pmtpa'
  • 19:37 logmsgbot_: bsitu synchronized wmf-config/CommonSettings.php 'Add Eventlogging configuration to Echo'
  • 19:17 logmsgbot_: pgehres synchronized php-1.21wmf11/extensions/CentralNotice/ 'Updating CEntralNotice to master, resolving bug 45846'
  • 18:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:39 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:36 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 16:07 cmjohnson1: reseating disk4 on ms-be1004
  • 15:28 mark: Reinstalled cp3003
  • 15:00 mark: Restarted gmetad on nickel
  • 02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Thu Mar 7 02:53:50 UTC 2013
  • 02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Thu Mar 7 02:29:38 UTC 2013
  • 01:53 mutante: installing package upgrades on sodium
  • 00:54 RobH: kicked icinga
  • 00:49 LeslieCarr: restarted pybal on lvs3
  • 00:40 RobH: puppet locked on ms1004, killall puppet, refired
  • 00:39 RobH: puppet halted run on cp1003, left in locked state, killed all puppet instances and refired, cleared.
  • 00:33 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/RSS
  • 00:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php

March 6

  • 23:19 LeslieCarr: reinstalling celsus as an internal host
  • 23:07 logmsgbot_: reedy synchronized php-1.21wmf10/resources/ 'touch'
  • 23:06 logmsgbot_: reedy synchronized php-1.21wmf11/resources/ 'touch'
  • 23:05 logmsgbot_: reedy synchronized wmf-config 'touch'
  • 22:54 logmsgbot_: reedy synchronized wmf-config/
  • 22:53 logmsgbot_: reedy synchronized wmf-config/
  • 22:40 mutante: starting haproxy on brewster
  • 22:39 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild message cache for Wikibase deploy
  • 22:39 notpeter: reslaving db36 and db38
  • 22:29 RobH: authdns-update
  • 22:20 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Wikibase
  • 22:20 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild message cache for Wikibase deploy
  • 22:11 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 22:09 logmsgbot_: reedy synchronized wikidataclient.dblist
  • 22:07 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:02 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 21:55 logmsgbot_: reedy cleared profiling data
  • 21:55 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Wikibase Client on all wikipedias'
  • 21:41 awight: updated payments cluster to 39a265489
  • 21:16 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/
  • 20:54 logmsgbot_: reedy synchronized wmf-config/
  • 20:38 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/DataValues/
  • 20:29 Reedy: Creating and populating site_identifiers table on all Wikipedias
  • 20:28 Reedy: Creating and populating sites table on all Wikipedias
  • 20:27 logmsgbot_: olivneh synchronized wmf-config/CommonSettings.php 'Adding redis host config. var for GettingStarted'
  • 20:12 logmsgbot_: reedy synchronized wmf-config/
  • 20:07 logmsgbot_: reedy synchronized wmf-config/
  • 20:04 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/Wikibase
  • 20:04 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/DataValues
  • 20:03 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/Diff
  • 19:57 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Wikibase
  • 19:56 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/DataValues
  • 19:43 notpeter: restarting all jobrunners
  • 19:34 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 19:29 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:28 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource to 1.21wmf11
  • 19:25 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: aawiki back to 1.21wmf10 for Aaron
  • 19:21 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 19:20 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia and wikiquote to 1.21wmf11
  • 19:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.21wmf11
  • 19:15 Reedy: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikiversity and wiktionary to 1.21wmf11
  • 19:10 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Special wikis to 1.21wmf11
  • 19:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Closed, private and fishbowl to 1.21wmf11
  • 19:03 mutante: gracefulling apaches to remove 1.17 aliases (gerrit 50609)
  • 19:03 logmsgbot_: dzahn gracefulled all apaches
  • 18:58 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:47 mutante: regenerating mailman list archives for wmfreqs
  • 18:44 mutante: renaming mailman list 'allhands' to 'wmfreqs' (RT-4640)
  • 18:36 logmsgbot_: aaron synchronized php-1.21wmf11/maintenance/nextJobDB.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
  • 18:35 logmsgbot_: aaron synchronized php-1.21wmf11/includes/job/JobQueueGroup.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
  • 18:35 logmsgbot_: aaron synchronized php-1.21wmf11/includes/DefaultSettings.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
  • 18:10 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'clear config cache'
  • 18:09 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'clear config cache'
  • 18:01 logmsgbot_: hashar synchronized wmf-config 'deploying 52340 adapt Lucene configuration file to support beta'
  • 17:59 hashar: deploying 52340 adapt Lucene configuration file to support beta
  • 07:08 jeremyb_: 02:52:46 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf11) at Wed Mar 6 02:52:46 UTC 2013
  • 07:08 jeremyb_: 02:36:35 <+logmsgbot> !log reedy synchronized wmf-config/InitialiseSettings.php
  • 07:08 jeremyb_: 02:28:21 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf10) at Wed Mar 6 02:28:21 UTC 2013
  • 07:08 jeremyb_: 01:33:50 <+logmsgbot> !log csteipp synchronized wmf-config/CommonSettings.php 'Revert https patch'
  • 07:07 jeremyb_: 01:08:27 <+logmsgbot> !log reedy synchronized wmf-config/CommonSettings.php 'Revert wgSecureLoginDefaultHTTPS'
  • 07:07 Ryan_Lane: restarted morebots
  • 01:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Set wgSecureLoginDefaultHTTPS to false for testing'
  • 00:37 binasher: added new deb of twemproxy 0.2.3 built with --enable-debug=log to precise-wikimedia
  • 00:21 logmsgbot_: csteipp synchronized wmf-config/CommonSettings.php 'Enabling HTTPS login'
  • 00:14 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings-labs.php
  • 00:09 hashar: snapshot1002: rsync: change_dir#3 "/apache/common-local" failed: No such file or directory (2)
  • 00:09 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings-labs.php

March 5

  • 23:57 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'increasing db1009 to full weight'
  • 23:54 LeslieCarr: reinstalling constable as an internal host
  • 23:49 RoanKattouw: Removed decommissioned srv226-234 from /etc/dsh/group/mediawiki-installation
  • 23:47 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php '6fd144c44f26 - Use new service IP for Parsoid in eqiad'
  • 23:46 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'repooling previously crashed mariadb pmtpa slaves'
  • 23:36 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'ombudsmenwiki logo'
  • 23:26 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'commenting out one slave per shard that's mysqldumping'
  • 23:18 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:13 notpeter: stopping slave and starting mysql dumps on dbs 71, 57, 66, 65, 55, 50, and 68
  • 23:00 logmsgbot_: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 22:40 logmsgbot_: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 22:36 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'repooling db1009 at 25% weight'
  • 22:18 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:17 logmsgbot_: reedy synchronized wmf-config/
  • 22:14 LeslieCarr: restarting pybal on lvs3
  • 22:14 LeslieCarr: restarting pybal on lvs4
  • 22:13 LeslieCarr: restarting pybal
  • 22:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 22:05 logmsgbot_: reedy synchronized wmf-config/
  • 21:57 LeslieCarr: restarting pybal on lvs1003
  • 21:55 LeslieCarr: restarting pybal on lvs1006
  • 21:44 Coren: tools rejiggered the webproxy config to be smarter about paths not leading to specific tools
  • 21:43 LeslieCarr: adding parsoidcache service to lvs1006
  • 21:42 mutante: gracefulling eqiad Apaches - add ombudsmen.wm (bug 45744)
  • 21:41 logmsgbot_: dzahn gracefulled all apaches
  • 21:35 mutante: restarting pdns on ns2
  • 21:34 mutante: DNS update - add ombudsmen (bug 45744)
  • 20:49 logmsgbot_: reedy synchronized docroot
  • 20:35 mutante: installing package upgrades on hooper
  • 20:26 logmsgbot_: reedy synchronized wmf-config/
  • 20:22 mutante: installing package upgrades on hume
  • 20:00 mutante: wikipedia.ee now redirects to et.wikipedia, wikimedia.ee now redirects to et.wikimedia
  • 19:56 mutante: gracefulling eqiad Apaches, push wiki(m|p)edia.ee redirects
  • 19:56 logmsgbot_: dzahn gracefulled all apaches
  • 19:18 K4-713: updated payments cluster to 9b063ee2e399a3b
  • 18:42 logmsgbot_: mlitn synchronized wmf-config/InitialiseSettings.php 'Disable AFTv4 (entirely) and AFTv5 (leaving only opt-in) on enwiki'
  • 18:04 cmjohnson1: restarting dhcpd3 service brewster
  • 16:55 cmjohnson1: mw1085 powering off to troubleshoot DIMM error
  • 05:35 ori-l: EventLogging db on db1047: ALTER DATABASE log CHARACTER SET utf8 COLLATE utf8_unicode_ci; (was: latin1)
  • 02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Tue Mar 5 02:48:58 UTC 2013
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Tue Mar 5 02:28:16 UTC 2013
  • 02:03 K4-713: updated payments cluster to 5361c795f4bf31980f
  • 00:46 mutante: fixing exim/mail permissions on neon to make Icinga paging work. missing +x on /var/log/exim4 dir

March 4

  • 23:12 mutante: starting dhcp3 on brewster
  • 23:11 LeslieCarr: p.s. bonding on foundry sucks
  • 23:10 LeslieCarr: bringing down virt2 network to bond it
  • 22:18 mutante: gracefulling eqiad Apaches to push indiawikipedia redirect
  • 22:17 logmsgbot_: dzahn gracefulled all apaches
  • 22:03 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'shuffling around pmtpa enwiki slaves'
  • 21:50 LeslieCarr: messing around with the network of virt2 - expect possible pmtpa labs network issues
  • 21:50 ottomata: staring puppet back up on oxygen, packet loss is better after disabling a filter
  • 21:30 cmjohnson1: authdns update
  • 21:17 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf11
  • 21:16 logmsgbot_: reedy synchronized php-1.21wmf11
  • 21:15 logmsgbot_: reedy synchronized php-1.21wmf10
  • 21:14 RobH: shutting down professor for a fan swap, should be a short window
  • 21:11 cmjohnson1: authdns update
  • 21:05 ottomata: stopping puppet on oxygen to see if we can disable filters and reduce packet loss
  • 20:55 Ryan_Lane: restarted icinga
  • 20:53 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11
  • 20:52 RobH: had to restart morebots, but it wasnt really dead, opps
  • 20:51 RobH: had to restart icinga
  • 20:51 RobH: Why arent you taking all the !log
  • 20:50 RobH: i hate you morebots
  • 20:28 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11
  • 19:55 logmsgbot_: reedy Started syncing Wikimedia installation... : Build 1.21wmf11 message cache
  • 19:41 logmsgbot_: reedy synchronized php-1.21wmf11/
  • 19:34 logmsgbot_: reedy synchronized wmf-config/
  • 19:34 logmsgbot_: reedy synchronized live-1.5/
  • 19:33 logmsgbot_: reedy synchronized docroot
  • 19:30 hashar: on gallium: stopping puppet, cleaning defunct and restarting puppet
  • 19:23 logmsgbot_: reedy synchronized wmf-config/
  • 19:20 mutante: installing package upgrades on sockpuppet
  • 19:13 sbernardin: taking down amaranth for hw replacement: https://rt.wikimedia.org/Ticket/Display.html?id=4497
  • 18:44 cmjohnson1: authdns update
  • 18:17 K4-713: updated payments cluster to 059969c3d29f685c
  • 17:44 cmjohnson1: authdns update
  • 16:56 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'depooling db1009 while investigating error'
  • 16:24 sbernardin: rebooting db70 to reset idrac: https://rt.wikimedia.org/Ticket/Display.html?id=4628
  • 16:14 sbernardin: db27 being taken down for decom per: https://rt.wikimedia.org/Ticket/Display.html?id=4623
  • 10:12 apergos: cleared some space on brewster by rmeoving old squid access/storage logs (2.2gb worth in 3 files!), its root partition was full, restarted squid
  • 10:12 apergos: reinstalled wikimedia-app-taskserver on fenari (how did it get removed?), needed for various cron jobs among other things
  • 02:04 logmsgbot_: LocalisationUpdate failed (1.21wmf10) at Mon Mar 4 02:04:10 UTC 2013

March 3

  • 02:25 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sun Mar 3 02:25:44 UTC 2013

March 2

  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sat Mar 2 02:27:35 UTC 2013
  • 01:17 mutante: powercycling emery
  • 00:35 mutante: rebooting emery for kernel upgrade
  • 00:29 mutante: installing package upgrades on emery

March 1

  • 23:59 cmjohnson1: authdns update
  • 23:56 K4-713: updated payments cluster to c8442435a9e2f5c
  • 23:29 notpeter: dropping limesurvey from db9, backup in db9:/a/backup_dumps
  • 22:53 RoanKattouw: Make that apache2-utils, that's the actual name of the package
  • 22:51 RoanKattouw: Temporarily installing apachebench on wtp1004 per Gabriel's request
  • 22:49 RoanKattouw: Depooled wtp1004 so Gabriel can use it for benchmarking
  • 22:46 mwalker: reverting payments to dced5328fd6ae4b395622abc41d2afa0ccad21c8 from 4a3ca6cc36b9d78dbb015c1bf93bbafb14eb0799 because of bug report that some people are not able to make donations due to 'required field city' being missing and then being unable to provide it
  • 22:34 cmjohnson1: authdns update
  • 22:15 cmjohnson1: auth-dns update
  • 22:12 mutante: DNS update - remove status cname
  • 21:56 cmjohnson1: taking wtp1001 down to replace w/new box
  • 21:32 mwalker: pushing 4a3ca6cc36b9d78dbb015c1bf93bbafb14eb0799 to the payments cluster (again, because the instant reviewer, me, was broken)
  • 21:23 mwalker: attempted deploy of payments cluster to new version, but then had to roll back to dced5328fd6ae4b395622abc41d2afa0ccad21c8
  • 20:02 mutante: blocked en.wikivoyage from bugmenot
  • 20:01 logmsgbot: dzahn synchronized ./docroot/wikivoyage.org/block_8708f.html
  • 19:55 logmsgbot: dzahn synchronized ./docroot/wikivoyage.org/block_ab2c3.html
  • 19:55 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'parser cache max age 30d'
  • 19:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043 to s1 watchlist service'
  • 19:31 mutante: DNS update - add CNAME for status.wp/status.wm
  • 19:28 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/51291/'
  • 19:13 mutante: install package upgrades on marmontel
  • 19:06 mutante: install package upgrades on streber
  • 19:04 mutante: create missing backup dir on streber to enable RT_Shredder plugin
  • 19:04 Ryan_Lane: maxsem synchronized wmf-config/InitialiseSettings.php 'make upload endpoint for Commons local https://gerrit.wikimedia.org/r/#/c/51612/'
  • 19:03 Ryan_Lane: LocalisationUpdate completed (1.21wmf10) at Fri Mar 1 02:31:13 UTC 2013
  • 19:03 Ryan_Lane: csteipp synchronized php-1.21wmf10/includes
  • 19:03 mutante: db9 - zipping old sql dumps of drupal and civirm to prevent full disk
  • 19:03 Ryan_Lane: dzahn synchronized ./docroot/wikivoyage.org/mywotf83102384bfcd1e52152.html
  • 19:03 Ryan_Lane: restarted glusterd service on labstore1-4
  • 19:03 Ryan_Lane: pgehres Finished syncing Wikimedia installation... : Updating i18n for DonationInterface-langonly
  • 19:02 Ryan_Lane: pgehres Started syncing Wikimedia installation... : Updating i18n for DonationInterface-langonly
  • 19:02 Ryan_Lane: binasher: shutting down mysql on db1043, converting to mariadb
  • 19:02 LeslieCarr: powering back on spence
  • 19:02 Ryan_Lane: asher synchronized wmf-config/db-eqiad.php 'pulling db1043 and shifting watchlist q's to db1050'
  • 19:02 Ryan_Lane: pgehres synchronized php-1.21wmf10/extensions/DonationInterface/ 'Updating DonatonInterface-langonly'
  • 19:02 Ryan_Lane: LeslieCarr: powering off spence
  • 19:01 Ryan_Lane: asher synchronized wmf-config/lucene.php 'sending search-pool4 traffic to pmtpa, where the index/host distribution has been rebalanced'
  • 19:01 Ryan_Lane: notpeter: upgrading mariadb boxes to 5.5.29
  • 19:01 Ryan_Lane: tstarling synchronized wmf-config/InitialiseSettings.php
  • 19:01 Ryan_Lane: Jeff_Green: stopped gmetad on neon for testing
  • 19:01 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/GettingStarted 'Update to split test (3/3)'
  • 19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/E3Experiments 'Update to split test (2/3)'
  • 19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/GuidedTour 'Update to split test (1/3)'
  • 19:00 Ryan_Lane: mutante: create missing dump directory on streber, enables "RT-shredder" plugin which comes with RT since 3.8
  • 19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/EventLogging 'Updates to JavaScript API'
  • 19:00 Ryan_Lane: changing A record for wikitech.wm.o and changing labsconsole.wm.o into a cname to wikitech
  • 18:59 Ryan_Lane: disabled editing on wikitech.wikimedia.org
  • 18:59 Ryan_Lane: binasher: resharding search-pool4 in pmtpa - restarted lsearchd on all local nodes and indexers on searchidx2
  • 18:59 mwalker: ... therefore the payments cluster is now running dced5328fd6ae4b395622abc41d2afa0ccad21c8
  • 18:59 mutante: restart nagios-nrpe,gmond,apache,salt-minion on bastion hosts
  • 18:59 Ryan_Lane: binasher: rebuilding db35 from a hotbackup of db55 (s5); testing a new build of mariadb 5.5.29
  • 18:59 Ryan_Lane: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
  • 18:59 Ryan_Lane: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
  • 18:58 Ryan_Lane: reedy synchronized php-1.21wmf10/extensions/WikimediaMaintenance
  • 18:58 Ryan_Lane: apergos: restarted icinga (not sure why it stopped, log file simply said 'caught TERM signal', in any case it complained that Error: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe )
  • 18:58 mwalker: enabled sofort forms for CH and DE on the payments cluster
  • 18:39 mutante: installing package upgrades on bast1001
  • 18:37 mutante: installing package upgrades on fenari
  • 17:40 ^demon: rescanned all trackingids for gerrit on manganese
  • 17:31 ^demon: upgrading gerrit to 2.5.2-1506-g278aa9a on manganese/formey
  • 17:07 Ryan_Lane: test

February 28

  • 12:58 apergos: restarted icinga (not sure why it stopped, log file simply said 'caught TERM signal', in any case it complained that Error: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe )
  • 06:28 apergos: stabbed searchpool4 and restarted lucene search etc gah hate hate hate
  • 03:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 03:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 03:12 logmsgbot: reedy synchronized docroot
  • 03:10 logmsgbot: reedy synchronized docroot
  • 03:09 logmsgbot: reedy synchronized docroot
  • 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Thu Feb 28 02:30:14 UTC 2013
  • 01:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 00:58 RobH: dsh restarted eqiad apaches in progress
  • 00:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 00:57 logmsgbot: robh gracefulled all apaches
  • 00:54 logmsgbot: reedy synchronized wmf-config/
  • 00:32 RobH: authdns-update to support wikinews.de
  • 00:27 logmsgbot: reedy synchronized wmf-config/
  • 00:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Lowering throttle to 5/day for new users, and'

February 27

  • 23:26 Reedy: Running sync-common on mw1111. Reporting failed opening wmf-config/mc.php
  • 22:57 mutante: rebooting zirconium (planet)
  • 22:54 mutante: installing package upgrades on zirconium
  • 22:52 logmsgbot: reedy synchronized wmf-config/
  • 22:46 mutante: rebooting bast1001
  • 22:36 mutante: installing package upgrades on bast1001
  • 22:16 logmsgbot: reedy synchronized wmf-config/
  • 21:44 RobH: rdb1001 & rdb1002 allocated, network setup, and ready for handoff, per rt4608
  • 21:36 RobH: authdns-update for rdb1001-1002
  • 21:36 RobH: authdns-update for sdb1001-1002
  • 21:31 Tim: running resetUserTokens.php on all wikis, in screen on hume, to fix user_token field corruption (bug 41586)
  • 21:26 logmsgbot: tstarling synchronized php-1.21wmf10/includes/User.php 'user_token fixes'
  • 20:05 K4-713: updated payments cluster to fcbf9211c9836b
  • 19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf10
  • 18:55 LeslieCarr: rebooting arsenic
  • 18:42 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for EducationProgram
  • 18:25 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for EducationProgram
  • 17:57 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/
  • 17:51 K4-713: updated payments cluster to eab489d0889a
  • 17:13 mutante: rebooting fenari for kernel upgrade
  • 08:33 apergos: restarted lucene search on search1015
  • 08:04 logmsgbot: reedy synchronized live-1.5/static-1.21wmf4/
  • 08:04 logmsgbot: reedy synchronized docroot/bits/static-1.21wmf4/
  • 07:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 07:05 Krinkle: done adding 1.21wmf3, 1.21wmf4, 1.21wmf5 (bug 44570)
  • 07:04 logmsgbot: krinkle Finished syncing Wikimedia installation... :
  • 06:29 apergos: snapshot1002 down for reinstallation
  • 06:27 logmsgbot: krinkle Started syncing Wikimedia installation... :
  • 06:15 Krinkle: adding 1.21wmf3, 1.21wmf4, 1.21wmf5 (bug 44570)
  • 06:15 Krinkle: added 1.21wmf1, 1.21wmf2 (bug 44570)
  • 05:35 apergos: restarted lucene search on search1016
  • 04:57 logmsgbot: krinkle Finished syncing Wikimedia installation... :
  • 04:37 logmsgbot: krinkle Started syncing Wikimedia installation... :
  • 02:57 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 27 02:57:22 UTC 2013
  • 02:31 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Wed Feb 27 02:31:06 UTC 2013
  • 01:35 Krinkle: Bringing back checkouts of MediaWiki 1.21wmf5 - 1.21wmf1 on fenari for bug 44570
  • 01:35 Jeff_Green: dist-upgrade and reboot grosley
  • 00:25 logmsgbot: tstarling cleared profiling data

February 26

  • 23:29 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057'
  • 23:27 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057'
  • 23:04 K4-713: synchronized payments cluster to 596b854787132c730
  • 22:35 mutante: installing package upgrades on fenari
  • 22:31 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 22:31 LeslieCarr: nagios is dead! spence.wikimedia.org if you want to access it directly
  • 22:16 mutante: gracefulling eqiad Apaches to push redirect for wikivoyage.eu
  • 22:15 logmsgbot: dzahn gracefulled all apaches
  • 22:15 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 20:58 RobH: stat1 puppet runs fail due to dan account error, i am working on it, but i also want to run out and get something to eat, so leaving in broken state for 10 minutes until i return to work on it
  • 20:21 mutante: DNS update - add wikiovyage.eu
  • 19:34 Jeff_Green: package updates & reboots of payments hosts
  • 19:13 paravoid: freezing all exim4 bounces to restore normal email delivery
  • 18:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 17:54 paravoid: restarting opendj on sanger
  • 17:45 AaronSchulz: Copying math files into ceph
  • 15:14 mark: suspended i-000001d7 and i-0000039e for sending out malicious emails
  • 15:06 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/GeoData/solrupdate.php
  • 15:05 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/GeoData/solrupdate.php
  • 12:26 apergos: temp blacklist live hack on sodium for exim4 to prevent some spam, see /etc/exim4/blacklist for that
  • 11:20 logmsgbot: mlitn synchronized php-1.21wmf10/extensions/ArticleFeedback/api/ApiArticleFeedback.php 'Update ArticleFeedback to master'
  • 06:53 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/
  • 06:43 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
  • 06:40 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
  • 06:28 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
  • 06:26 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
  • 06:21 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
  • 03:10 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
  • 03:06 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
  • 03:04 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
  • 03:03 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
  • 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 26 02:52:01 UTC 2013
  • 02:46 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
  • 02:44 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
  • 02:41 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
  • 02:37 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
  • 02:29 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Feb 26 02:28:06 UTC 2013
  • 02:11 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
  • 01:59 Ryan_Lane: test
  • 01:59 Ryan_Lane: test
  • 01:58 Ryan_Lane: test
  • 01:40 K4-713: updated payments cluster to ce28b210374d8
  • 00:20 mutante: mc1001,ms1004,mw1029,mw1059,mw1070,mw1157 - fixed puppet freshness
  • 00:03 mutante: amssq33,amssq44 - fix more puppet runs

February 25

  • 23:54 mutante: knsq26,knsq28 - deleting puppet lock file, fix puppet runs
  • 23:45 logmsgbot: reedy synchronized wmf-config/
  • 23:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable SearchExtraNS on commonswiki'
  • 23:34 mutante: re-enabling disabled puppet on srv253,srv235
  • 22:16 Ryan_Lane: restarting opendj/pdns on virt0
  • 22:02 Reedy: Ran namespaceDupes.php on huwikt
  • 21:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:28 logmsgbot: reedy synchronized docroot
  • 20:27 logmsgbot: reedy synchronized wmf-config/
  • 19:50 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuilding message cache to update WikimediaMessages
  • 19:40 mutante: powercycling frozen niobium
  • 19:32 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuilding message cache to update WikimediaMessages
  • 19:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf10
  • 18:54 mutante: wikivoyager.de and wikivoyager.org transferred and both redirect to wikivoyage.org
  • 18:50 mutante: deleting foundation-news-l (deprecated, replaced by WikimediaAnnounce-l)
  • 18:49 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMessages
  • 18:31 AaronSchulz: Copying all captchas into ceph
  • 18:11 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Wikibase
  • 06:38 apergos: restarted lucene search on search1016
  • 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 25 02:52:32 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Mon Feb 25 02:28:55 UTC 2013
  • 01:15 mark: Powercycled strontium
  • 01:07 paravoid: powercycling niobium (both unresponsive from the load)
  • 01:06 paravoid: powercycling arsenic

February 24

  • 06:28 apergos: restarted search1016 lucene
  • 02:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 24 02:54:30 UTC 2013
  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sun Feb 24 02:29:29 UTC 2013

February 23

  • 21:45 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb
  • 21:44 logmsgbot: reedy synchronized php-1.21wmf10/cache/interwiki.cdb
  • 21:43 Reedy: mw110 is asking for a password
  • 06:13 apergos: restarted lucene search on search1016
  • 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 23 02:52:10 UTC 2013
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sat Feb 23 02:27:17 UTC 2013
  • 00:43 RobH: lvs1003 had a ton of hung puppet processes (no output on them when tracing, so dunno whats up). killed them and kicked manual run

February 22

  • 23:59 RobH: argon is shut down now
  • 23:55 RobH: argon is going offline, killing our limesurvey install
  • 23:29 paravoid: deploying squid config for upload's new transcoded container
  • 23:26 paravoid: deploying squid
  • 22:55 Ryan_Lane: changing dns entry for wikitech-static
  • 22:41 RobH: mw110 throwing odd puppet errors, will come back to it later
  • 22:41 RobH: sq41 reinstalled, throws endless puppet errors during redeployment
  • 22:07 MaxSem: reindexing complete
  • 21:39 RobH: harmon and copper online with OS installed for datadumps misc host support
  • 21:39 MaxSem: Nuking GeoData Solr index and reindexing
  • 21:38 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/GeoData/solrupdate.php 'https://gerrit.wikimedia.org/r/#/c/50388/'
  • 21:35 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/GeoData/solrupdate.php 'https://gerrit.wikimedia.org/r/#/c/50388/'
  • 21:05 RobH: installing harmon and copper into dump generation misc use
  • 20:17 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/GuidedTour/modules/tours/gettingstarted.js 'Resume split test (3/3)'
  • 20:16 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/GettingStarted/resources/ext.gettingstarted.js 'Resume split test (2/3)'
  • 20:16 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/E3Experiments/experiments/openTask.js 'Resume split test (1/3)'
  • 20:14 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Math 'Updating Math extension to fix db caching behavior (Change-Id: I9b690e55001859c97fd40330272791d49ec6de75)'
  • 20:13 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/Math 'Updating Math extension to fix db caching behavior (Change-Id: I9b690e55001859c97fd40330272791d49ec6de75)'
  • 20:09 RobH: authdns-update
  • 19:58 RobH: sq41 boot order was fubar, fixed, reinstalling (as reboot into installer wiped mbr)
  • 19:56 RobH: wtp1002-1004 os installed
  • 19:34 Jeff_Green: start mysqldump of otrs database on db49
  • 19:18 RobH: kicked ntpd manually on mw1045
  • 19:09 RobH: sq41 reinstalling
  • 18:57 RobH: mw112-125 back into pybal api pool
  • 18:56 RobH: mw86-99 & mw111 back into pybal pool
  • 18:49 logmsgbot: robh Finished syncing Wikimedia installation... :
  • 18:37 logmsgbot: robh Started syncing Wikimedia installation... :
  • 17:53 sbernardin: taking down sq41 to replace controller card.... https://rt.wikimedia.org/Ticket/Display.html?id=4550
  • 16:24 mutante: tartupeedia.ee now redirects to et.wp portal page (bug 44893)
  • 16:12 mutante: gracefulling eqiad Apaches via dsh to push tartupeedia.ee redirect (bug 44893)
  • 16:11 logmsgbot: dzahn gracefulled all apaches
  • 16:07 mutante: mw110 - sits at partioning screen in installer
  • 15:47 hashar: gallium removed live hack in /srv/org/wikimedia/doc/puppetsource/manifests/role/labsmediawiki.pp
  • 14:52 ^demon: restarted gerrit on manganese, jetty flapping again
  • 14:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 13:14 mark: Halted ms-be3001-3003 for controller replacement
  • 13:11 mark: Halted ms-be3004 for controller replacement
  • 13:07 mark: Halted cp3009 for controller replacement
  • 12:20 mark: Halted cp3010 for controller replacement
  • 11:23 mutante: DNS update - add indiawikipedia.com
  • 07:33 apergos: restarted lucene search on search1016
  • 06:49 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 05:33 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 05:33 logmsgbot: tstarling synchronized docroot/bits/favicon/wikipedia.ico
  • 05:20 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 05:18 logmsgbot: tstarling synchronized docroot/bits
  • 02:53 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 22 02:53:05 UTC 2013
  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Fri Feb 22 02:29:09 UTC 2013
  • 01:57 binasher: migrating wikivoyage and wikidata external blob stores to innodb
  • 01:08 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed f699c5a9ad2f780dc18863592dcae4a51ba6d841'

February 21

  • 23:21 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/CodeEditor 'Syncing patch that disables background linting'
  • 23:18 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, and GuidedTour
  • 23:02 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, and GuidedTour
  • 22:35 logmsgbot: awjrichards synchronized php-1.21wmf9/extensions/MobileFrontend/
  • 22:33 logmsgbot: awjrichards synchronized php-1.21wmf10/extensions/MobileFrontend
  • 22:12 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Add photo upload schema for event logging'
  • 21:56 notpeter: running do-release-upgrade on dbs 52, 39, 51, and 35
  • 21:49 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/EventLogging
  • 21:48 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing 1 slave for s2-5 from db-secondary for upgrades and maria'
  • 21:48 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
  • 21:47 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/CodeEditor
  • 21:37 notpeter: restarting mysql on db1047 to pick up conf changes
  • 21:31 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/SyntaxHighlight_GeSHi 'Disable highlighting of very large files.'
  • 21:23 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/SyntaxHighlight_GeSHi 'Disable highlighting of very large files.'
  • 21:05 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php Enabling CodeEditor for Schema namespace on test2 and metawiki.
  • 20:02 binasher: some wikis have myisam page_props tables. converting all to innodb via osc
  • 18:39 Ryan_Lane: enabled Extension:OpenID as a provider on labsconsole
  • 11:04 hashar: Ariel saved Gerrit by restarting it!
  • 10:51 hashar: huge CPU spike on manganese (gerrit host)
  • 10:06 hashar: Jenkins: reseting git branch specifier :-/
  • 09:32 hashar: Jenkins: removing git branch specifier from all mediawiki-core jobs.
  • 05:50 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'shell cgroup'
  • 02:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 21 02:54:17 UTC 2013
  • 02:30 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'new extension1 shard (as an externalload config) - initially for AFTv5'
  • 02:30 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'new extension1 shard (as an externalload config) - initially for AFTv5'
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Thu Feb 21 02:28:20 UTC 2013
  • 00:53 RobH: issues with new mw servers 86+, took back out of pybal until i can troubleshoot
  • 00:49 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance/
  • 00:48 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMaintenance/
  • 00:34 binasher: moved wikidatawiki tables on s3 to wikidatawiki_old to keep around for a time just in case; then "drop database wikidatawiki"
  • 00:20 logmsgbot: asher synchronized wmf-config 'enabling wikidatawiki on shard s5'
  • 00:18 logmsgbot: asher synchronized s5.dblist 'adding wikidatawiki'
  • 00:17 logmsgbot: asher synchronized s3.dblist 'removing wikidatawiki'

February 20

  • 22:48 RobH: authdns-update
  • 22:35 binasher: importing wikidatawiki with schema changes into the s5 master, with sql_log_bin disabled
  • 22:24 binasher: importing wikidatawiki with schema changes into the inactive pmtpa side of the s5 shard, with replication of the import just in pmtpa. ignore repl warnings for db{35,44,45,55}
  • 21:48 RobH: authdns-update for wtp1002-1004
  • 21:45 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'Take pmtpa Solr servers out of rotation, increase load on master: https://gerrit.wikimedia.org/r/#/c/50056/'
  • 21:41 RobH: ignore db66 errors, asher is working on it
  • 21:25 binasher: running wikidata wb_items / wb_items_per_site schema migrations in parallel on the db66 slave (creating primary keys, cannot be done via online-schema-change)
  • 21:24 RobH: kicked knsq23 to clear puppet freshness alarm
  • 21:24 RobH: maerlant still offline (200 days?). dropped esams ticket for onsite troubleshooting, as mgmt is offline.
  • 21:20 logmsgbot: asher synchronized wmf-config/InitialiseSettings.php 'setting wgReadOnly for wikidatawiki durring maintenance'
  • 21:18 RobH: forced puppet run on lardner, was tired of it showing error for no runs
  • 21:13 RobH: mw112-125 now in service as api apaches in tampa
  • 20:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 20:38 RobH: mw86-mw111 tested ok, pushing into apache service in pmtpa
  • 20:32 RobH: mw86-mw111 being added to nodelists, ignore errors on them for next few minutes
  • 19:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks to 1.21wmf10
  • 19:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikisource and wikiquote to 1.21wmf10
  • 19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikinews and wiktionary to 1.21wmf10
  • 19:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, special, wikimedia, private and fishbowl to 1.21wmf10
  • 05:34 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 're-enable Scribunto profiler'
  • 04:32 Tim: updating luasandbox to version 1.2, with profiler bug fixes
  • 02:53 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Wed Feb 20 02:53:32 UTC 2013
  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 20 02:29:11 UTC 2013
  • 01:28 RobH: mw1161-1208 have wrong power setting, changing all to system.power.hotspare.enable 0
  • 01:00 notpeter: do-release-upgrade-ing on db58
  • 00:57 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing db58 from db-secondary for upgrades and maria, replacing db43'

February 19

  • 23:47 notpeter: do-release-upgrade-ing on db43
  • 23:38 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing db43 from db-secondary for upgrades and maria'
  • 23:21 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/49946/'
  • 23:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 23:03 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/
  • 22:49 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 22:41 logmsgbot: preilly synchronized php-1.21wmf9/extensions/ArticleFeedback 'fix MAX call with GREATEST'
  • 22:29 RobH: sq41 offline, rt 4550
  • 22:20 RobH: sq41 puppet freshness bad, wont accept ssh connections, serial redirection results in no output. will need to reboot it
  • 22:17 RobH: sq41 not allowing ssh, drac resetting
  • 22:15 sbernardin: taking down ms-be12 for hardware replacement per: https://rt.wikimedia.org/Ticket/Display.html?id=4546
  • 22:15 RobH: sq79/80 puppet in locked state, manually restarted puppet, removed lock file, ran manually
  • 22:11 logmsgbot: preilly synchronized php-1.21wmf9/extensions/ArticleFeedback 'fix mysql 5.5 / ansi sql incompatibility'
  • 21:54 RobH: mw48 was set to puppet agent disabled, re-enabled and fired puppet run
  • 20:04 logmsgbot: mflaschen synchronized . 'Syncing GuidedTour submodule omitted when new branch cut was first cloned.'
  • 19:59 Ryan_Lane: disabling project storage volumes for all labs projects with no instances
  • 15:02 hashar: restarting jenkins to enable some startup setting
  • 14:58 hashar: manganese CPU spike ongoing. Started roughly at 14:40 UTC
  • 13:29 mutante: wikimedia.jp.net now redirects to wikimedia.org | wikimediacommons.jp.net now redirects to commons.wm
  • 13:27 mutante: DNS update - adding wikimedia.jp.net
  • 13:25 logmsgbot: dzahn gracefulled all apaches
  • 13:20 mutante: gracefulling eqiad Apaches via dsh to push redirect change for .jp.net domains
  • 12:39 apergos: powercycling ms-be7 (unresponsive to ping or from console)
  • 11:29 mutante: installing package upgrads on neon/icinga
  • 02:49 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Feb 19 02:49:32 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 19 02:28:25 UTC 2013
  • 01:12 logmsgbot: anomie synchronized php-1.21wmf10/includes/api/ApiQuerySiteinfo.php 'Backport fix for ugly API bug'
  • 00:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 00:30 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php

February 18

  • 23:51 logmsgbot: tstarling synchronized php-1.21wmf9/extensions/Scribunto
  • 23:51 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Scribunto
  • 23:46 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 23:25 Tim: on snapshot1002 -- fixed incorrect owner on some MW files, killed broken apt-get and upgraded wikimedia-task-appserver
  • 23:24 Tim: fixed rsyncd.conf on scap proxies to allow connections from new apaches
  • 23:14 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 23:11 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 22:45 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 22:42 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 22:37 logmsgbot: tstarling synchronized php-1.21wmf9/extensions/Scribunto
  • 22:37 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Scribunto
  • 22:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 20:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.21wmf10
  • 20:37 logmsgbot: reedy synchronized php-1.21wmf10/cache/l10n/ 'Scap is a liar'
  • 20:31 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10n cache for wikidata related extensions
  • 20:15 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10n cache for wikidata related extensions
  • 19:59 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Wikibase
  • 19:59 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Diff
  • 19:58 logmsgbot: reedy synchronized php-1.21wmf10/extensions/DataValues
  • 19:58 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMessages
  • 19:44 logmsgbot: reedy synchronized php-1.21wmf10/cache/l10n/
  • 19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf10
  • 19:03 logmsgbot: reedy Finished syncing Wikimedia installation... : Build 1.21wmf10 message cache, and push it this time
  • 18:45 logmsgbot: reedy Started syncing Wikimedia installation... : Build 1.21wmf10 message cache, and push it this time
  • 17:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Build 1.21wmf10 message cache
  • 17:53 logmsgbot: reedy Started syncing Wikimedia installation... : Build 1.21wmf10 message cache
  • 17:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf10
  • 17:28 logmsgbot: reedy synchronized php-1.21wmf10 'Initial sync out'
  • 17:03 logmsgbot: reedy synchronized wmf-config/
  • 17:02 logmsgbot: reedy synchronized docroot
  • 17:01 logmsgbot: reedy synchronized live-1.5/
  • 12:33 mutante: wikiartpedia.(biz|co|info|me|mobi|net) activated - redirect to wikipedia.org (RT-4240)
  • 12:30 mutante: adding wikiartpedia domains to DNS / authdns-update
  • 12:28 logmsgbot: dzahn gracefulled all apaches
  • 12:24 mutante: gracefulling eqiad Apaches via dsh for redirect.conf change
  • 12:22 mutante: mw1045 - eth0 NO-CARRIER - check cable (RT-4545)
  • 12:13 mutante: mw1041 - down - CPU Machine Chk error on mgmt - removing from dsh groups (RT-4517)
  • 05:45 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php
  • 05:16 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php
  • 05:10 Tim: on db1043: running alter table to convert the relevant field to signed, since I figure this is less disruptive than sql_mode = 'NO_UNSIGNED_SUBTRACTION' and less likely to break again than skipping the query
  • 04:58 Tim: replication broken on db1043, possibly others, due to update of article_feedback_revisions with negative value for afr_total
  • 03:40 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 18 02:29:47 UTC 2013

February 17

  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 17 02:29:31 UTC 2013

February 16

  • 13:38 apergos: restarted lucene-search on search1015 about 20 minutes ago
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 16 02:28:26 UTC 2013
  • 01:47 logmsgbot: reedy synchronized php-1.21wmf9/
  • 01:11 logmsgbot: reedy synchronized wmf-config/filebackend.php

February 15

  • 23:56 RobH: authdns-update to correct mw110 reverse entry
  • 23:28 Ryan_Lane: *modified them in LDAP
  • 23:28 Ryan_Lane: modified project objects to indicate whether or not they are using gluster volumes for project and/or home storage
  • 23:04 RobH: disregard mw1161 flaps, im working on it
  • 20:15 Ryan_Lane: restarting opendj and pdns on virt0 and virt1000
  • 20:15 Ryan_Lane: removed opendj from nfs1/2
  • 19:07 RobH: no one sign mw86-mw125 puppet certs, im still working on them
  • 18:23 RobH: mw87-mw125 installing
  • 17:33 binasher: reimaging db1029-1031
  • 16:55 mutante: removing mw1085 from dsh groups - broken hardware (RT-4453)
  • 16:00 hashar: Enabled a linter for operations/apache-config . Any apache typo in operations/apache-config will publicly write your name on the typo wall of shame. See 49251 for implementation details and https://integration.mediawiki.org/ci/job/operations-apache-config-lint/ for some build examples.
  • 14:31 mutante: wikimediafoundation.info activated to redirect .org
  • 14:28 mutante: gracefulling Apaches in eqiad using dsh
  • 14:25 logmsgbot: dzahn gracefulled all apaches
  • 14:15 ^demon: [gerrit] updated replication plugin to 20c6fc7e (1.1-SNAPSHOT) which is back in line with upstream
  • 08:02 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Echo 'Syncing Iab077a77aa75a6ff0cf03acecca38403595dda93 to fix exception 640c60c4'
  • 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 15 02:30:05 UTC 2013
  • 01:19 paravoid: deploying squid changes for s/knams/esams/ for knsq* squids (see git log for rationale)
  • 01:09 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Enabling Schema NS on test2wiki'

February 14

  • 22:49 RobH: mw1194 online and now in service as api apache
  • 22:48 RobH: mw1200 added to api pool, online and in service
  • 22:44 RobH: mw1189-1199 (sans 1194) now online as eqiad api apaches
  • 22:37 logmsgbot: mflaschen Finished syncing Wikimedia installation... :
  • 22:29 logmsgbot: mflaschen Started syncing Wikimedia installation... :
  • 22:21 sbernardin: taking storage1 offline for decommissioning per: https://rt.wikimedia.org/Ticket/Display.html?id=4529
  • 22:18 RobH: pushing mw1161-1188 into apache service (sans 1161/1164) from false to true in pybal, otherwise all online
  • 22:16 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Updating for test2wiki'
  • 22:14 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Math
  • 22:03 Ryan_Lane: replacing gluster upstarts on all labstore nodes with a regular init script
  • 22:01 logmsgbot: reedy synchronized docroot
  • 22:00 logmsgbot: olivneh synchronized wmf-config
  • 21:49 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling PostEdit on {be|mediawiki|outreach}wikis'
  • 21:45 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
  • 21:39 sbernardin: taking down srv266 and srv268 for decommissioning ... https://rt.wikimedia.org/Ticket/Display.html?id=4534
  • 21:31 ottomata1: restarted solr jetty on solr2
  • 21:30 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update ArticleFeedbackv5, Echo, PageTriage to master
  • 21:25 ottomata: restarted solr jetty on solr3 and solr1003 for MaxSem
  • 21:03 logmsgbot: bsitu Started syncing Wikimedia installation... : Update ArticleFeedbackv5, Echo, PageTriage to master
  • 20:24 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'updating CommonSettings.php for Echo'
  • 20:15 apergos: rebooting mw1010 after some issues with OOM
  • 19:22 RobH: new servers are NOT in pybal yet, and thus not getting traffic
  • 19:22 RobH: just added mw1161-1200 into node lists for eventual service, some may throw sync script errors as I track them down
  • 19:15 Ryan_Lane: rebooting labstore3
  • 19:08 Ryan_Lane: rebooting labstore4
  • 19:01 Ryan_Lane: rebooting labstore2
  • 18:52 Ryan_Lane: rebooting labstore1
  • 15:22 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Updating wikibase sort order'
  • 15:22 logmsgbot: demon synchronized php-1.21wmf9/extensions/Wikibase/client/includes/SortUtils.php 'Updating wikibase sort order'
  • 14:52 mutante: re-enabled cp1044 in pybal - mobile varnish cache servers all on precise now
  • 14:39 hashar: Jenkins: adding in several extension to Zuul configuration 49041
  • 14:30 hashar: jenkins : updated all jobs using latest version jjb-config
  • 13:35 mark: Fixed fixing
  • 13:27 mutante: disabling cp1044 in pybal, reinstalling with precise
  • 13:22 mutante: re-enabling cp1043 in pybal
  • 13:11 mark: Upgraded varnish to latest version with updated range patch on cp1021-1028
  • 12:16 mark: Inserted varnish 3.0.3plus~rc1-wm7 into the precise-wikimedia APT repository
  • 12:14 mutante: stopping varnish backend on cp1043 and starting reinstall
  • 12:10 mutante: disabling cp1043 in pybal
  • 11:31 mutante: ms1004 - full disk, deleted large HTCPpurger.log file, fixed puppet runs
  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 14 02:29:05 UTC 2013
  • 01:17 andrewbogott: updated OpenStackManager to git head on virt0. Previous state is preserved (temporarily) in the local 'premerge' branch.
  • 01:13 paravoid: removing mysql & apache from emery(!)
  • 00:03 RobH: db29 os installed, handed off to peter

February 13

  • 23:50 RobH: db29 reinstalling
  • 23:48 paravoid: reinstalling sq48
  • 23:22 RobH: db27 and db29 will be up and down for reinstallations
  • 23:17 sbernardin: rebooting db29 per https://rt.wikimedia.org/Ticket/Display.html?id=4528
  • 23:17 sbernardin: rebooting db27 per https://rt.wikimedia.org/Ticket/Display.html?id=4528
  • 23:11 RobH: db1013 online and puppet updated, needs to be pushed into some db cluster service
  • 22:45 RobH: mw27 locked up, powercycling
  • 22:36 notpeter: powercycling all eqiad jobrunners
  • 22:13 RobH: adding mw1201-1208 to eqiad apache pool
  • 21:44 Reedy: Running sync-common on mw1203
  • 21:43 logmsgbot: reedy synchronized php-1.21wmf9 'resync to fix file permission errors'
  • 21:32 Reedy: Running sync-common on mw1204
  • 21:30 Reedy: Running sync-common on mw1202
  • 21:27 RobH: db27 rebooting for reinstallation
  • 21:27 RobH: db29 offline due to hardware issue, related ticket 4526
  • 21:19 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 21:11 logmsgbot: csteipp synchronized wmf-config
  • 21:09 logmsgbot: reedy cleared profiling data
  • 20:25 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Uncommenting enwiki from localClientDatabases'
  • 19:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Wikibase Client on enwiki'
  • 19:47 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase/
  • 19:30 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved remaning wikis to 1.21wmf9
  • 19:21 logmsgbot: aaron synchronized php-1.21wmf9/maintenance/runJobs.php 'deployed b03384f0ebd95e7f79638fb14ccd55da9c186d97'
  • 19:12 logmsgbot: py gracefulled all apaches
  • 18:59 cmjohnson1: mw1041 is borked and will be powered down and removed from network for h/w checks
  • 18:27 logmsgbot: aaron synchronized php-1.21wmf9/extensions/TranslationNotifications 'deployed d7bf60b57f472e5f75101f58505f61842848f961'
  • 16:26 Nemo_bis: Job queue weirdness and TranslationNotifications duplicates again noticed on Meta (bug 42715)
  • 13:33 logmsgbot: reedy synchronized php-1.21wmf9/resources/mediawiki
  • 09:17 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48804/'
  • 09:10 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48804/'
  • 08:52 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48797/'
  • 08:25 hashar: jenkins: changing encryption key and regenerating secrets. See bug 44592
  • 05:50 logmsgbot: tstarling synchronized multiversion/MWScript.php
  • 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Feb 13 02:52:39 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 13 02:28:06 UTC 2013
  • 00:42 notpeter: restarting varnishncsa on cp1043
  • 00:03 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend 'Touchhhhh'

February 12

  • 23:54 logmsgbot: awjrichards synchronized php-1.21wmf9/extensions/MobileFrontend/javascripts/specials/donateimage.js 'touch file'
  • 22:58 logmsgbot: maxsem Finished syncing Wikimedia installation... :
  • 22:46 notpeter: disabling innodb-flush-log-at-trx-commit=2 on db1047 w/asher's blessing. will schedule downtime to apply
  • 22:38 logmsgbot: maxsem Started syncing Wikimedia installation... :
  • 22:04 notpeter: creating metrics user on all s1-7 masters
  • 20:25 logmsgbot: reedy synchronized php-1.21wmf9/resources/mediawiki
  • 20:22 binasher: flushed mobile varnish
  • 20:21 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend
  • 20:19 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend
  • 20:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 20:05 binasher: flushed mobile varnish cache �:(
  • 19:59 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend
  • 19:57 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend
  • 19:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable WikibaseClient on enwiki'
  • 19:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable WikibaseClient on enwiki'
  • 18:49 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
  • 18:41 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
  • 18:36 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
  • 18:25 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
  • 18:20 cmjohnson1: nas1001-a swapping mainboard
  • 18:10 logmsgbot: aaron synchronized php-1.21wmf9/extensions/UploadWizard 'deployed 4548f9817d3d8025f4c19e279e9fb1f05b178087'
  • 18:09 logmsgbot: catrope synchronized php-1.21wmf9/extensions/Wikibase 'Un-rollback Wikibase'
  • 18:06 RoanKattouw: Reverted Wikibase update earlier, now reapplying
  • 17:53 paravoid: powercycling niobium, SSH & console unresponsive
  • 17:50 mark: Blocking modules=mobile.startup requests with 503 on bits varnish (earlier)
  • 17:50 mark: Forced mw1149-1150 to healthy in bits.eqiad varnish
  • 17:30 logmsgbot: catrope synchronized php-1.21wmf9/extensions/Wikibase 'Rolling back Wikibase'
  • 16:18 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 12 16:18:31 UTC 2013
  • 16:13 Jeff_Green: restarted apache on mw1152
  • 16:01 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
  • 16:00 Reedy: mw1161 and mw1165 are both missing mediawiki deployment files
  • 15:58 logmsgbot: reedy synchronized php-1.21wmf9/includes/site/
  • 15:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 12 15:54:11 UTC 2013
  • 14:25 mutante: ran puppet on professor
  • 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 12 02:52:42 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 12 02:28:53 UTC 2013
  • 01:01 ^demon: stopped puppet and gerrit on manganese and formey
  • 00:28 logmsgbot: aaron synchronized php-1.21wmf9/includes/db/Database.php 'deployed d8705542627f006a7ec9f81a9fb488fcc9a367bd'

February 11

  • 23:19 RobH: unpooled mw1161-1188 from pybal config
  • 22:03 logmsgbot: demon synchronized php-1.21wmf9/extensions/VisualEditor/modules/ve/init/mw/ve.init.mw.Platform.js 'Touching file'
  • 21:08 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Wikibase client throwing fatals, disabling on enwiki pending fix'
  • 21:04 cmjohnson1: auth-dns update
  • 21:03 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'wikidata client on enwiki'
  • 21:03 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'wikidata client on enwiki'
  • 20:30 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: huwiki and itwiki to 1.21wmf9
  • 20:27 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: hewiki and itwiki to 1.21wmf9
  • 20:22 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf9
  • 20:11 logmsgbot: demon synchronized php-1.21wmf9/includes/site 'Updating sites code to pull in I8964a03c, I42221832'
  • 18:39 RobH: disregard, pushing one to test, mw1161
  • 18:36 RobH: mw1161-mw1188 being pushed into pybal pool as general apaches
  • 16:58 logmsgbot: demon synchronized php-1.21wmf9/extensions/Wikibase/lib/includes/modules/SitesModule.php 'Touching file'
  • 16:50 logmsgbot: demon synchronized php-1.21wmf9/extensions/TorBlock 'TorBlock to master, now with less annoying logs'
  • 16:41 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Enable logging group for torblock'
  • 16:40 logmsgbot: demon synchronized php-1.21wmf9/extensions/TorBlock 'Updating TorBlock to master'
  • 16:27 cmjohnson1: commenting out srv226-234 on pybal/pmtpa
  • 14:59 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'bug 44251 hardcode $wgDBuser = "wikiuser" 47074'
  • 13:55 hashar: jenkins: regenerating all phpcs-HEAD jobs so they stop failing when no files need to be processed bug 44567 48450
  • 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Feb 11 02:52:39 UTC 2013
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 11 02:27:44 UTC 2013

February 10

  • 22:16 Tim: fixed syslog.eqiad.wmnet DNS entry, restarting rsyslogd on the eqiad apaches (RT 4500)
  • 19:20 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/48015/'
  • 02:53 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Feb 10 02:53:56 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 10 02:28:46 UTC 2013

February 9

  • 12:52 Reedy: mw1161 is missing /usr/local/apache/common-local
  • 12:51 logmsgbot: reedy synchronized php-1.21wmf8/resources/
  • 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Feb 9 02:52:45 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 9 02:28:28 UTC 2013

February 8

  • 23:48 logmsgbot: anomie synchronized php-1.21wmf8/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
  • 23:48 logmsgbot: anomie synchronized php-1.21wmf9/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
  • 23:09 RoanKattouw: and on cerium too
  • 23:08 RoanKattouw: Running puppet on titanium
  • 20:20 RobH: disregard errors for mw1161-1164
  • 20:13 RobH: mw1161-1164 offline for reallocation from job to general apache
  • 20:10 cmjohnson1: decommissioning srvs226-234
  • 19:43 RobH: mw1165-1188 os installed for apache use, initial puppet runs in progress
  • 19:33 RobH: celsus back online
  • 19:23 RobH: celsus rebooting to fix power mgmt cpu spike issue
  • 19:02 RobH: mw1161-1164 in service as eqiad jobrunners
  • 16:32 ottomata: updated vumi packages in apt to vumi_0.5.0~a+143-0~ppa3 and vumi-wikipedia_0.1~a+14-0~ppa3
  • 16:09 mark: Upgraded and rebooted cr2-eqiad (both REs)
  • 15:11 mark: Set cr2-eqiad VRRP prio to 50 to transition to backup role
  • 15:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 15:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: srwikibooks back to 1.21wmf9 to test variant fixes
  • 14:39 hashar: gallium installed php-luasandbox manually. Gerrit change is 48127.
  • 05:47 logmsgbot: tstarling synchronized php-1.21wmf9/includes/context/RequestContext.php
  • 02:54 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Feb 8 02:54:49 UTC 2013
  • 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 8 02:30:42 UTC 2013
  • 02:02 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 01:48 LeslieCarr: cr2-knams is back up on the new code
  • 01:36 LeslieCarr: cr2-knams reloading after upgrade
  • 01:34 LeslieCarr: deactivating european bgp sessions - may cause some connectivity issues as connections reset
  • 01:24 LeslieCarr: disabling notifications on icinga and nagios
  • 01:03 logmsgbot: reedy synchronized php-1.21wmf9/languages/classes/
  • 00:25 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/GettingStarted

February 7

  • 23:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling GuidedTour on additional wikis'
  • 23:25 LeslieCarr: manually cleared arp and turned managmeent interface off and on of asw-c-eqiad to solve reachability issue
  • 23:02 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy GuidedTour
  • 22:47 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy GuidedTour
  • 22:39 RobH: mw82 pooled into tampa apaches
  • 22:04 ottomata: pulled power and rebooted amaranth toolserver for multichill. Its back online. multichill will notify whomever needs to do more work.
  • 21:58 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/RelatedArticles/ 'Fixes 44761'
  • 21:58 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/RelatedArticles/ 'Fixes 44761'
  • 21:50 RobH: mw81-mw85 in pmtpa apache service
  • 21:39 cmjohnson1: performing a hard reboot on amaranth
  • 21:34 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/E3Experiments
  • 21:33 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/E3Experiments
  • 21:12 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/PostEdit
  • 21:11 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/PostEdit
  • 21:10 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/EventLogging
  • 21:09 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
  • 20:29 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Being paranoid, node lists were in flux during last sync
  • 20:23 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: All ku/sr wikis back to 1.21wmf8 - lang converter breakage
  • 20:20 Nemo_bis: !Wikipedia and !Wikimedia projects are/were down, issue being investigated
  • 20:14 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: kuwiktionary back to 1.21wmf8
  • 20:07 RobH: removing srv219-224 from rendering config in pmtpa
  • 20:06 mark: Rebooting cr1-sdtpa
  • 19:56 RobH: adding mw75-mw80 into tampa rendering pool
  • 18:42 mark: Rebooted re0.cr1-eqiad
  • 18:05 mark: Rebooting re1.cr1-eqiad
  • 16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 15:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 15:44 mark: Rebooting cr1-esams
  • 15:40 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
  • 15:39 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb 'Updating 1.21wmf9 interwiki cache'
  • 14:54 mutante: granting RT group 'toolserver admins' the 'modify ticket' permission in queue esams
  • 14:38 mutante: planet now updating hourly instead of daily
  • 14:13 mutante: planet speaks SSL now for all languages. (it does not force http->https but it can if there is a demand)
  • 12:00 Nemo_bis: amaranth dead/unreachable from Europe, ping dies at xe-5-3-1.cr2-eqiad.wikimedia.org (173.241.131.218)
  • 10:08 hashar: jenkins : regenerating jobs to make phpcs to show the sniff codes being used. 47847
  • 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Feb 7 02:52:26 UTC 2013
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 7 02:27:52 UTC 2013
  • 02:23 logmsgbot: spage synchronized php-1.21wmf9/extensions/GuidedTour/modules/externals/mediawiki.libs.guiders/mediawiki.libs.guiders.submodule 'sync GuidedTour submodule omitted from 1.21wmf9'
  • 01:57 RoanKattouw: Force-running puppet on wtp1 to figure out why it's broken
  • 01:53 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I9fd98309'
  • 00:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 00:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 00:20 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance
  • 00:00 logmsgbot: reedy synchronized langlist

February 6

  • 23:59 RobH: authdns-update to add min.wikipedia.org via addwiki automation
  • 23:56 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
  • 23:55 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb 'Updating 1.21wmf9 interwiki cache'
  • 23:50 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:43 logmsgbot: reedy synchronized langlist
  • 23:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:28 RobH: srv190-srv223 & srv225 should be offline, they are being decommissioned, can ignore nagios errors
  • 23:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:13 RobH: specifically (i can spell, maybe)
  • 23:13 RobH: removed from pybal specifially
  • 23:12 cmjohnson1: removing srv192-218 and srv225 per rt 4438
  • 23:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:05 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance/
  • 21:36 cmjohnson1: srv193 is back online
  • 21:21 cmjohnson1: srv193 power down to relocate to different rack
  • 19:47 RobH: authdns-update
  • 19:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikibooks to 1.21wmf9
  • 19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.21wmf9
  • 19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews, wikivoyage and wikiversity to 1.21wmf9
  • 19:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special and wikimedia wikis to 1.21mwf9
  • 19:11 cmjohnson1: powerdowing mw1 to reseat DIMM B1
  • 19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, private and fishbowl wikis to 1.21wmf9
  • 18:43 ^demon: running puppet on manganese, restarting gerrit
  • 18:41 RobH: authdns-update
  • 17:48 LeslieCarr: restarted morebots
  • 02:51 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 6 02:51:36 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Feb 6 02:28:05 UTC 2013
  • 02:05 logmsgbot: reedy synchronized wmf-config/
  • 00:42 RobH: mailing lists returned to normal
  • 00:41 RobH: done with data removal on mailing list server, rebuilding mbox
  • 00:37 RobH: stopping mailman to scrub an entry out of archives, will restart shortly

February 5

  • 23:29 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend/javascripts/modules/mf-photo.js 'https://gerrit.wikimedia.org/r/#/c/47670/'
  • 23:26 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/javascripts/modules/mf-photo.js 'https://gerrit.wikimedia.org/r/#/c/47670/'
  • 22:44 logmsgbot: maxsem Finished syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-02-05
  • 22:28 logmsgbot: maxsem Started syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-02-05
  • 22:08 RobH: analytics1007 coming down for troubleshooting, disregard nagios errors
  • 22:01 RobH: analytics1001 reinstalled, not puppet signed, back online
  • 22:00 RobH: analytics1001 respects my authority.
  • 21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 44688 - Please disable MoodBar extension on English Wikipedia'
  • 21:28 logmsgbot: robh synchronized wmf-config/InitialiseSettings.php
  • 21:12 RobH: analytics1001 going down for troubleshooting/reinstallation
  • 20:21 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 20:12 RobH: blog update sucessful, caching plugin update nonsuccessful, rolling it back to older plugin
  • 20:07 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 20:04 RobH: updating the blog software, here goes nothing
  • 19:17 hashar: gallium : Zuul updated!
  • 19:13 hashar: gallium : updating Zuul by cherry-picking a set of changes that let us fetch tags. ff79197..1854e32
  • 17:05 cmjohnson1: authdns update
  • 16:37 LeslieCarr: copied all equinix ashburn IPv4 peering sessions to their new /22 on cr2-eqiad
  • 16:37 Jeff_Green: dist-upgrades for payments systems
  • 16:36 Jeff_Green: restarted morebots :-P
  • 05:37 Ryan_Lane: restarting opendj on all hosts
  • 05:36 Ryan_Lane: remove virt1 replication agreement from all ldap servers
  • 02:51 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 5 02:51:10 UTC 2013
  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 5 02:29:02 UTC 2013
  • 01:07 logmsgbot: reedy synchronized php-1.21wmf9/extensions/TimedMediaHandler
  • 00:51 logmsgbot: reedy synchronized php-1.21wmf8/extensions/ZeroRatedMobileAccess
  • 00:39 logmsgbot: reedy synchronized php-1.21wmf8/extensions/ZeroRatedMobileAccess
  • 00:29 logmsgbot: reedy synchronized php-1.21wmf9/extensions/ZeroRatedMobileAccess

February 4

  • 23:49 logmsgbot: reedy synchronized multiversion/
  • 23:46 logmsgbot: reedy synchronized multiversion/
  • 23:45 logmsgbot: reedy synchronized wmf-config/
  • 23:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 21:02 hashar: gallium moved a git extension : mv /var/lib/git/rt-authen-externalauth.git /var/lib/git/operations/debs/
  • 19:44 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
  • 19:39 ottomata: manually gracefully reloaded eqiad apaches. See RT 4449
  • 19:37 logmsgbot: root gracefulled all apaches
  • 19:20 logmsgbot: reedy synchronized wmf-config/
  • 19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, mediawikiwiki and wikidatawiki to 1.21wmf9
  • 18:35 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
  • 17:56 logmsgbot: reedy Finished syncing Wikimedia installation... : Build messages for 1.21wmf9
  • 17:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:24 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
  • 17:14 logmsgbot: reedy synchronized php-1.21wmf9/extensions/GuidedTour/
  • 17:09 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
  • 17:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 17:06 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
  • 16:45 logmsgbot: reedy synchronized php-1.21wmf9/ 'Initial sync'
  • 16:22 logmsgbot: reedy synchronized wmf-config/
  • 16:06 Ryan_Lane: force starting any glusterfs volume that has a downed process
  • 15:12 Ryan_Lane: rebooting labstore1
  • 15:11 Ryan_Lane: upgrading labstore1 to precise
  • 14:33 apergos: powercycled ms-be3
  • 14:32 Ryan_Lane: make that rebooting labstore3
  • 14:32 Ryan_Lane: rebooting labstore2
  • 14:07 Ryan_Lane: upgrading labstore3 to precise
  • 11:14 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'cbd432f..82ed797 47393 Vector micro design on mlwiki, 29792 wikipedia translation on pawiki'
  • 07:37 Ryan_Lane: upgrading labstore2 to precise
  • 02:25 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Feb 4 02:25:58 UTC 2013
  • 00:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 00:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php

February 3

  • 20:58 hashar: jenkins : generating mwext-**-phpcs-HEAD jobs
  • 17:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:42 logmsgbot: reedy synchronized wmf-config/
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Feb 3 02:28:29 UTC 2013
  • 02:03 logmsgbot: reedy synchronized wmf-config/
  • 02:02 logmsgbot: reedy synchronized images/sul/foundation.png
  • 01:14 logmsgbot: reedy synchronized wmf-config/
  • 00:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 00:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php

February 2

  • 23:59 logmsgbot: reedy synchronized wmf-config/
  • 23:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:34 logmsgbot: reedy synchronized wmf-config/
  • 23:30 logmsgbot: reedy synchronized php-1.21wmf8/extensions/SearchExtraNS
  • 23:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 16:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 09:23 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/PageImages/initImageData.php 'https://gerrit.wikimedia.org/r/#/c/44556/'
  • 08:45 MaxSem: Running foreachwikiindblist wikipedia.dblist extensions/PageImages/initImageData.php --earlier-than=20121229000000
  • 03:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Feb 2 02:28:18 UTC 2013

February 1

  • 23:02 K4-713: synchronized payments cluster to b421871b43822
  • 20:00 RobH: authdns-update
  • 19:43 RobH: authdns-update
  • 14:45 cmjohnson1: authdns update "adding mw1161-1200 to eqiad mgmt and production zone files
  • 12:04 cmjohnson1: authdns update adding db1051-60 to zone files
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Feb 1 02:27:09 UTC 2013
  • 01:50 K4-713: updated payments cluster to 45f8a6c6009a4bd7

January 31

  • 23:14 andrewbogott: rebooting virt0 in a fit of optimism and/or desperation
  • 22:44 logmsgbot: spage Finished syncing Wikimedia installation... : take 2: Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
  • 22:37 logmsgbot: spage Started syncing Wikimedia installation... : take 2: Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
  • 22:26 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
  • 22:20 ottomata: deployed tab delimiter log format change on squids, varnishes and nginxes
  • 21:40 ottomata: reverted tab separartor change to squid
  • 21:37 ottomata: deloyed tab separator log format changes to squids, merged the corresponding puppet changes to varnish and nginx
  • 20:47 paravoid: deploying patched ganglia and removing .htaccess. Ganglia is public again!
  • 14:59 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
  • 14:58 notpeter: taking down db1047 for upgrade to precise
  • 13:21 hashar: gallium: installed mercurial manually (puppet change is 46931 PS2)
  • 13:19 hashar: gallium manually removed bzr: apt-get remove bzr python-keyring python-httplib2 python-launchpadlib python-zope.interface python-oauth python-bzrlib bzr python-simplejson python-configobj python-lazr.uri python-lazr.restfulclient python-wadllib
  • 13:19 hashar: gallium: manually installed "bzr"
  • 12:12 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
  • 10:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Remove strategyappswiki
  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 31 02:29:47 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Jan 31 02:28:18 UTC 2013

January 30

  • 22:40 binasher: deployed new text/upload squid confs with eqiad in proxySubnets
  • 22:38 logmsgbot: reedy synchronized wmf-config/
  • 22:23 logmsgbot: tstarling cleared profiling data
  • 22:12 logmsgbot: aaron synchronized php-1.21wmf8/includes/filerepo/file/LocalFile.php 'deployed f884911f5562fc41980664c424e7037bde6ba110'
  • 22:11 logmsgbot: aaron synchronized php-1.21wmf8/includes/filerepo/file/OldLocalFile.php 'deployed f884911f5562fc41980664c424e7037bde6ba110'
  • 21:59 LeslieCarr: killing varnishhtcpd on cp3009 for realzies this time
  • 21:58 RobH: mw23 offline, reinstalled with precise and signed into puppet, puppet throwing errors on initial run.
  • 21:52 binasher: started udplog with correct mw config on fluorine
  • 21:49 LeslieCarr: restarting varnishhtcpd on cp3010 - it appears to be locked up , netstat -l shows a huge Recv-Q for port 4827 and appears to be the previously discovered issue
  • 21:43 binasher: restarting udp2log on fluorine
  • 20:59 logmsgbot: reedy cleared profiling data
  • 20:50 RobH: authdns-update
  • 20:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:20 RobH: mw23 was stuck in installer, restarting and setting up.
  • 20:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:45 logmsgbot: reedy synchronized wmf-config/ 'Enabling wikidata client on itwiki and hewiki'
  • 19:42 ottomata: added updated udplog_1.8-5 .debs to apt.wikimedia.org. No real changes, these now include the packet-loss program in the package.
  • 19:22 logmsgbot: reedy synchronized wmf-config/
  • 19:12 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase/
  • 19:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf8
  • 18:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:48 Ryan_Lane: downgrading memcached to ubuntu version on virt0
  • 17:18 Reedy: Created wb_changes_dispatch on wikidatawiki
  • 16:55 Reedy: Created sites/site_identifier tables on hewiki and itwiki
  • 16:34 cmjohnson1: mw1072 going to down to check h/w state currently failed rt4381
  • 15:38 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 44395) Allow bureaucrats to remove the translateadmin group on wikidatawiki 46011'
  • 15:37 logmsgbot: hashar synchronized wmf-config/throttle.php '(bug 44506) raise throttle for an Israel editthon 46747'
  • 12:54 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/GeoData
  • 12:52 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/GeoData
  • 05:55 Tim: rebooted kaulen, was not responding on network or serial console
  • 03:11 mutante: wikimania.asia redirect now works, it did not before because of a bug in apache-graceful-all not actually restarting Apaches in eqiad
  • 03:07 mutante: reloading Apaches in eqiad using manual dsh (because apache-graceful-all sanity check has a bug)
  • 02:55 logmsgbot: catrope gracefulled all apaches
  • 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Wed Jan 30 02:52:03 UTC 2013
  • 02:39 logmsgbot: dzahn gracefulled all apaches
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Jan 30 02:28:22 UTC 2013
  • 00:44 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 00:43 mutante: sync InitialiseSettings.php for bug 44473
  • 00:19 binasher: restarted pdns-recursor on dobson
  • 00:13 logmsgbot: reedy synchronized wmf-config/

January 29

  • 23:54 logmsgbot: reedy synchronized wmf-config/
  • 23:21 logmsgbot: pgehres synchronized php-1.21wmf8/extensions/CentralNotice/ 'Updating CentralNotice to master, resolving namespace issues'
  • 23:17 logmsgbot: pgehres synchronized php-1.21wmf7/extensions/CentralNotice/ 'Updating CentralNotice to master, resolving namespace issues'
  • 23:00 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Enable Special:Nearby everywhere'
  • 22:51 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/46630'
  • 22:48 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/46630'
  • 22:47 Tim: rotated SAL wikitech page - test logging


2000s

2010s

2020s