Server Admin Log/Archive 25

From Wikitech

September 30

  • 23:29 logmsgbot: spage Synchronized wmf-config/InitialiseSettings.php: Create log group for Echo (duration: 00m 11s)
  • 23:27 logmsgbot: spage Synchronized php-1.25wmf1/extensions/Echo: Echo no-op (change reverted) (duration: 00m 09s)
  • 22:55 ori: re-enabling puppet on mw1019
  • 22:36 ori: disabling puppet on mw1019 to enable debug logging in apache
  • 22:09 mutante: removing linne from DNS - was already shutdown about 24 hours before
  • 21:57 K4-713: updated prod civicrm to 477a5107a0c93ceac5214
  • 21:44 ori: Spike of bitter irony from Nemo_bis on #wikimedia-operations starting 21:43 UTC
  • 21:33 logmsgbot: ori Synchronized php-1.25wmf1/languages/Language.php: I672c699c (2/2) (duration: 00m 03s)
  • 21:33 logmsgbot: ori Synchronized php-1.25wmf1/includes/specialpage/SpecialPageFactory.php: I672c699c (1/2) (duration: 00m 07s)
  • 21:23 Nemo_bis: widespread reproducible 503 errors on wikidata and elsewhere
  • 20:55 andrewbogott: powering down virt0, just to see what breaks
  • 20:48 andrewbogott: shutting down pdns on virt0
  • 20:48 andrewbogott: shutting down opendj on virt (temporary, a preview of tomorrow)
  • 18:50 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 16s)
  • 18:49 logmsgbot: reedy Synchronized database lists: (no message) (duration: 00m 15s)
  • 18:41 mutante: pc1001-1003 - can't generate tmp files for percona monitoring checks -> puppet fail
  • 18:24 mutante: killing silver from icinga and puppet
  • 18:23 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 16s)
  • 18:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf1
  • 18:05 logmsgbot: ori Synchronized wmf-config/HHVMRequestInit.php: (no message) (duration: 00m 07s)
  • 18:04 K4-713: re-enabled all queue consumers
  • 17:56 ejegg: updated civicrm from e83c999f39e6ae847d9b48e38c8c825fc10d1635 to b6c350f620c8dc1f3410de179c19cbcbdeb62270
  • 17:19 K4-713: disabled qc jobs and TY mail send for pending civi deploy
  • 15:45 hashar: Updating our Jenkins job builder fork 686265a..ee80dbc (no job changed)
  • 15:42 bblack: rebooting mexia
  • 15:33 logmsgbot: demon Synchronized docroot/bits/favicon/wikipedia.ico: Favicons are my favorite icons, especially when they're only 18% of the size of the original (duration: 00m 04s)
  • 15:16 logmsgbot: demon Synchronized php-1.25wmf1/extensions/Wikidata: (no message) (duration: 00m 11s)
  • 15:14 logmsgbot: demon Synchronized php-1.25wmf1/extensions/VisualEditor: (no message) (duration: 00m 08s)
  • 15:12 akosiaris: merging https://gerrit.wikimedia.org/r/#/c/163735/1, changing the LDAP master from sanger to ldap-mirror for inbound mail
  • 15:12 andrewbogott: running sync-common on virt1000
  • 15:12 logmsgbot: demon Synchronized visualeditor.dblist: (no message) (duration: 00m 04s)
  • 15:11 logmsgbot: demon Synchronized visualeditor-default.dblist: (no message) (duration: 00m 04s)
  • 15:06 logmsgbot: demon Synchronized wmf-config/Wikibase.php: (no message) (duration: 00m 04s)
  • 15:06 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s)
  • 14:26 _joe_: restarted apache on mw1196, lots of apc errors
  • 14:22 logmsgbot: oblivian gracefulled all apaches
  • 12:10 mark: Stopped exim daemon on mchenry
  • 09:41 godog: removed obsolete /etc/puppet/hiera from strontium and palladium, /etc/puppet/hieradata is the new location
  • 09:24 godog: reboot ms-be2001 as a test
  • 04:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 30 04:18:48 UTC 2014 (duration 18m 47s)
  • 03:17 logmsgbot: LocalisationUpdate completed (1.25wmf1) at 2014-09-30 03:17:15+00:00
  • 02:41 logmsgbot: ori Synchronized 503.html: Ia88b306ef: Make the 503 error page consistent with other 5xx error pages (duration: 00m 08s)
  • 02:34 logmsgbot: LocalisationUpdate completed (1.24wmf22) at 2014-09-30 02:34:07+00:00
  • 01:00 Krinkle: Jenkins connection seemed in order with integration-slave1007 and 8, but disconnecting and relaunching the slave agents immediately resulted in them getting jobs assigned. Cause unknown, problem resolved for now.
  • 00:58 Krinkle: integration-slave1007 and integration-slave1008 have not gotten any jobs in the past 24h. integration-slave1006 however has gotten loads of action. Investigating load balancing issue.
  • 00:24 mutante: linne - shutting down, revoking puppet cert, salt key, puppet/icinga ...
  • 00:12 logmsgbot: maxsem Synchronized w/skins-1.5: (no message) (duration: 00m 03s)
  • 00:12 MaxSem: https://gerrit.wikimedia.org/r/#/c/162520/ broke stuff, reverted
  • 00:10 logmsgbot: maxsem Synchronized live-1.5: (no message) (duration: 00m 03s)

September 29

  • 23:59 logmsgbot: maxsem Synchronized w: https://gerrit.wikimedia.org/r/#/c/162520/ (duration: 00m 03s)
  • 23:58 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/163773/ (duration: 00m 03s)
  • 23:50 logmsgbot: maxsem Synchronized php-1.25wmf1/extensions/MultimediaViewer/: second try... (duration: 00m 04s)
  • 23:42 logmsgbot: maxsem Finished scap: SWATting a bunch of stuff (duration: 18m 44s)
  • 23:32 andrewbogott: stopped apache, nova-scheduler, keystone, puppetmaster on virt0
  • 23:31 bd808: /var/lib/jenkins-slave/tmpfs 100% full on lanthanum.eqiad.wmnet
  • 23:26 andrewbogott: disabling puppet on virt0 so I can kill off services one by one...
  • 23:23 logmsgbot: maxsem Started scap: SWATting a bunch of stuff
  • 23:17 logmsgbot: maxsem Synchronized docroot/: <mutante> he killed the dolphin (duration: 00m 06s)
  • 23:13 Reedy: dist-upgraded logstash1001 and reboot
  • 22:47 Reedy: dist-upgrade logstash1002 and reboot
  • 22:36 Reedy: dist-upgrade on logstash1003 and rebooting
  • 22:34 Reedy: restarted elasticsearch on logstash1003 post java upgrades
  • 22:30 Reedy: packages upgraded on logstash1002
  • 22:28 mutante: silver - shutting down, wait with wiping it for a few days, just incase
  • 22:28 Reedy: packages upgraded on logstash1001
  • 22:24 Reedy: elasticsearch upgradeed to 1.3.2 on logstash1003
  • 22:18 andrewbogott: renaming labs-ns1 to labs-ns0 and labs-ns2 to labs-ns1
  • 22:02 Reedy: elasticsearch upgradeed to 1.3.2 on logstash1002
  • 22:01 mutante: silver - revoke puppet cert, salt-key, stopping services, disable monitoring
  • 21:58 mutante: stopping udp2log-vumi on silver - not needed anymore per Yuvipanda
  • 21:12 Reedy: elasticsearch upgradeed to 1.3.2 on logstash1001
  • 20:50 bd808: Ran sync-common on tmh1002.eqiad.wmnet for cscott's failed sync-dir there
  • 20:49 bd808: Ran sync-common on tmh1001.eqiad.wmnet for cscott's failed sync-dir there
  • 20:29 logmsgbot: cscott Synchronized wmf-config: Switch default PDF renderer to OCG (duration: 00m 15s)
  • 20:04 subbu: deployed Parsoid version deed30b2
  • 19:41 ottomata: restarted varnishkafka on cp3019 to troubleshoot drerrs
  • 19:26 Reedy: doing rolling upgrade of elasticsearch on logstash100[1-3]
  • 17:59 cscott: updated OCG to version 89d8f29a24295b05d0643abe976fea83b56575c9
  • 17:58 logmsgbot: ori Synchronized php-1.24wmf22/includes/password/Pbkdf2Password.php: I3b0a1de69: Test for string in Pbkdf2Password::crypt() (duration: 00m 05s)
  • 17:47 bblack: stopped powerdns and disabled puppet on virt1000 to prevent further cache pollution w/ bad data in public caches
  • 15:57 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT enable collection extension svwikiversity (duration: 00m 06s)
  • 15:53 hashar: Zuul jobs reregistered
  • 15:46 hashar: Zuul lost all Jenkins jobs :(
  • 15:24 logmsgbot: manybubbles Synchronized php-1.24wmf22/extensions/UploadWizard/: SWAT update UploadWizard (duration: 00m 05s)
  • 15:17 logmsgbot: manybubbles Synchronized php-1.25wmf1/extensions/Wikidata/: SWAT update wikidata to fix hhvm issues. (duration: 00m 14s)
  • 15:05 logmsgbot: manybubbles Synchronized wmf-config/wikitech.php: SWAT sync wikitech file - is a noop I believe (duration: 00m 05s)
  • 15:02 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT fix config type of flow. (duration: 00m 06s)
  • 13:32 hashar: Restarted Zuul
  • 13:27 hashar: Zuul: tweaking configuration files 162584
  • 09:31 godog: deployed new swift ring to eqiad-prod
  • 08:21 hashar: Restarting Jenkins to have a plugin installed/loaded properly
  • 03:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 29 03:25:11 UTC 2014 (duration 25m 10s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.25wmf1) at 2014-09-29 02:26:50+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf22) at 2014-09-29 02:15:34+00:00
  • 02:14 bblack: restarting squid on carbon (webproxy)

September 28

  • 23:27 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: switch db1042 load groups to db1056 (duration: 00m 06s)
  • 23:17 springle: powercycle db1042
  • 23:15 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1042, locked up (duration: 00m 07s)
  • 23:12 bblack: restarted apache on mw1123 + mw1196
  • 23:11 bblack: test
  • 23:11 bblack: restarted apache on mw1123 + mw1196
  • 20:28 ori: Puppet failures appear to be caused by apt-get timeouts
  • 10:09 _joe_: updated bash (again) across the whole cluster
  • 03:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 28 03:24:20 UTC 2014 (duration 24m 19s)
  • 02:28 logmsgbot: LocalisationUpdate completed (1.25wmf1) at 2014-09-28 02:28:14+00:00
  • 02:17 logmsgbot: LocalisationUpdate completed (1.24wmf22) at 2014-09-28 02:17:01+00:00

September 27

  • 18:35 logmsgbot: ori Synchronized php-1.25wmf1/extensions/Scribunto/engines/LuaSandbox/Engine.php: Capture traces for bug 71045 (duration: 00m 13s)
  • 18:35 logmsgbot: ori Synchronized php-1.24wmf22/extensions/Scribunto/engines/LuaSandbox/Engine.php: Capture traces for bug 71045 (duration: 00m 17s)
  • 04:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Sep 27 04:03:25 UTC 2014 (duration 3m 24s)
  • 02:46 logmsgbot: LocalisationUpdate completed (1.25wmf1) at 2014-09-27 02:46:53+00:00
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf22) at 2014-09-27 02:25:05+00:00

September 26

  • 23:26 mutante: switched noc.wikimedia.org to terbium, behind misc-web
  • 22:01 bd808: sudo apache2ctl graceful on logstash100[123] for ldap revert
  • 22:00 bd808: running puppet on logstash100[123] to revert ldap change
  • 21:56 bd808: sudo apache2ctl graceful on logstash100[123] for ldap change
  • 21:35 andrewbogott: gracefulled apache on neon
  • 21:21 mutante: graceful'ed apache on neon
  • 20:45 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Disable HHVM beta-feature on wikidatawiki (duration: 00m 06s)
  • 19:58 awight: update CRM from 25159fcfc29921b08de86f12121fb292139be09d to 3e42bac8cb7f58f5e504946f4944c69ca5553e60
  • 19:42 mutante: removing root's public_html from fenari - backup kept just in case
  • 19:15 AaronS: Deployed security patches to CentralAuth
  • 19:09 Krinkle: git-deploy: Deploying integration/slave-scripts 08147c42ea42e1a5eca1d29
  • 19:08 logmsgbot: aaron Synchronized php-1.25wmf1/extensions/CentralAuth: (no message) (duration: 00m 07s)
  • 19:06 logmsgbot: aaron Synchronized php-1.24wmf22/extensions/CentralAuth: (no message) (duration: 00m 08s)
  • 18:15 Nemo_bis: untruncated: andrewbogott> ldap is broken for gerrit, should be working elsewhere
  • 18:14 legoktm: ldap is broken
  • 18:09 K4-713: re-enabled donations queue consumer
  • 17:50 awight: CRM queue consumer disabled
  • 17:43 andrewbogott: upgraded libgnutls26 on ytterbium
  • 17:35 andrewbogott: "git reset --hard origin" to remove that terrible hotfix on palladium and strontium.
  • 17:28 awight: CRM jobs reenabled
  • 17:22 hoo: Manually ran rebuildEntityPerPage.php for Wikidata
  • 17:16 andrewbogott: hotfixing /var/lib/git/operations/puppet in hopes of fixing gerrit so I don't have to hotfix no more
  • 17:08 awight: updated crm from 06c9546f9b68f6ecbaaf510944418aa52f9ed0fb to 25159fcfc29921b08de86f12121fb292139be09d
  • 17:02 awight: disabling CRM jobs for deployment...
  • 15:29 andrewbogott: puppet is now moving all labs instances to new ldap servers: ldap-eqiad and ldap-codfw
  • 15:02 cscott: documented what I'm going to clear the OCG queues at https://wikitech.wikimedia.org/wiki/OCG#Pruning_the_queue
  • 14:36 bblack: address for ns1 switched in our local dns data - https://gerrit.wikimedia.org/r/163164
  • 13:57 hoo: Manually declared the global rename Secretary-> VlsergeyBot done after it twice timed out on pages moves on ruwiki
  • 13:39 akosiaris: moved mathoid to low-traffic lvs servers@eqiad
  • 12:48 cscott: cleared OCG caches again when I woke up to buy me more time to investigate the issue properly.
  • 08:44 awight: rollback: revision for civicrm locked to 06c9546f9b68f6ecbaaf510944418aa52f9ed0fb
  • 08:30 _joe_: updated hhvm on mw1053, kicked the jr a couple of times, working again now
  • 08:29 awight: large_donation schema migration 7000
  • 08:28 awight: skip over wmf_civicrm schema migration 7022 -- *why* did I make that unsafe
  • 08:24 awight: fundraising_code_update: revision for civicrm changed from 06c9546f9b68f6ecbaaf510944418aa52f9ed0fb to 5aca00fd4573f0fe8f385baa7238172f6ae54438
  • 08:19 awight: disabling CRM jobs during deployment
  • 08:09 cscott: cleared OCG queues and cache to quiet icinga; will try to get to the root cause tomorrow.
  • 07:41 hashar: Updated our Jenkins Job Builder fork 2d74b16..686265a
  • 07:06 logmsgbot: ori Synchronized php-1.24wmf22/extensions/WikimediaEvents: Update WikimediaEvents for 791e14cfc1d (duration: 00m 05s)
  • 06:53 logmsgbot: ori Synchronized php-1.25wmf1/extensions/WikimediaEvents: Update WikimediaEvents for 0e087daea5 (duration: 00m 07s)
  • 06:41 cscott: updated OCG to version f3a6c1cbba118d4a5e1aa019937dc50159fc823d
  • 04:43 _joe_: updating bash, USN-2363
  • 04:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 26 04:10:12 UTC 2014 (duration 10m 11s)
  • 03:09 logmsgbot: LocalisationUpdate completed (1.25wmf1) at 2014-09-26 03:09:47+00:00
  • 02:36 logmsgbot: LocalisationUpdate completed (1.24wmf22) at 2014-09-26 02:36:45+00:00
  • 00:14 awight: turning off Civi jobs before deployment

September 25

  • 23:31 logmsgbot: maxsem Synchronized php-1.25wmf1/skins/Vector/: https://gerrit.wikimedia.org/r/#/c/163021/ (duration: 00m 03s)
  • 23:15 logmsgbot: maxsem Synchronized php-1.25wmf1/extensions/CentralAuth/: https://gerrit.wikimedia.org/r/#/c/162971/ (duration: 00m 04s)
  • 23:12 logmsgbot: maxsem Synchronized php-1.25wmf1/includes/resourceloader/ResourceLoaderSiteModule.php: https://gerrit.wikimedia.org/r/#/c/163024/ (duration: 00m 03s)
  • 23:10 logmsgbot: maxsem Synchronized php-1.25wmf1/includes/api/ApiQueryAllUsers.php: https://gerrit.wikimedia.org/r/#/c/163027/ (duration: 00m 03s)
  • 23:08 logmsgbot: maxsem Synchronized php-1.24wmf22/includes/api/ApiQueryAllUsers.php: https://gerrit.wikimedia.org/r/#/c/163026/ (duration: 00m 03s)
  • 23:02 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/163048 (duration: 00m 03s)
  • 22:58 logmsgbot: ori Synchronized php-1.24wmf22/extensions/Wikidata: Update Wikidata for I0acd2096d21b (duration: 00m 11s)
  • 21:41 mutante: powercycling mw1053
  • 20:36 mutante: no !log
  • 20:36 legoktm: manually migrated "NickK" to a global account
  • 20:29 mutante: repooled mw1051
  • 19:49 bd808: Restarted logstash on logstash1001. udp2log events were not being recorded.
  • 19:30 logmsgbot: reedy Synchronized php-1.25wmf1/: (no message) (duration: 00m 46s)
  • 19:24 logmsgbot: reedy Synchronized php-1.24wmf22/resources/src/mediawiki.ui/components/buttons.less: (no message) (duration: 00m 14s)
  • 19:22 bblack: ntp work done on hosts
  • 19:18 logmsgbot: reedy Synchronized php-1.25wmf1/: (no message) (duration: 00m 55s)
  • 19:17 logmsgbot: reedy Synchronized php-1.24wmf22/extensions/CentralAuth/: (no message) (duration: 00m 14s)
  • 18:55 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 18:47 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 18:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf1
  • 18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.24wmf22
  • 17:20 logmsgbot: reedy Finished scap: testwiki to 1.25wmf1 and build l10n cache (duration: 28m 36s)
  • 16:52 logmsgbot: reedy Started scap: testwiki to 1.25wmf1 and build l10n cache
  • 16:41 Reedy: Purged php-1.24wmf9
  • 16:38 logmsgbot: reedy Purged l10n cache for 1.24wmf20
  • 15:31 bblack: testing ntpd changes on acamar, achernar, chromium, hydrogen, nescio, and baham (puppet-agent disabled)
  • 15:19 logmsgbot: mattflaschen Synchronized wmf-config/CommonSettings.php: Extend GettingStarted bucketting period end date to Sept. 28 (duration: 00m 07s)
  • 12:36 godog: update bash on elastic1014 analytics1021 elastic1013
  • 11:33 _joe_: gracefully reloaded apache on mw1139 and mw1199, apc issues
  • 11:29 logmsgbot: aude Synchronized php-1.24wmf22/extensions/Wikidata/extensions/Wikibase/lib/config/WikibaseLib.default.php: fix apc issues (duration: 00m 06s)
  • 11:03 _joe_: updated bash on elastic1007
  • 10:57 godog: upgraded bash on labsdb1003
  • 10:31 Nemo_bis: SAL is here
  • 09:22 godog: graphite temporarily down, fix incoming
  • 06:13 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1062 (duration: 00m 07s)
  • 03:58 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 25 03:58:02 UTC 2014 (duration 58m 1s)
  • 03:02 logmsgbot: LocalisationUpdate completed (1.24wmf22) at 2014-09-25 03:02:46+00:00
  • 02:32 logmsgbot: LocalisationUpdate completed (1.24wmf21) at 2014-09-25 02:32:56+00:00
  • 02:08 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Use Debian-packaged texvc on Trusty app servers (duration: 00m 04s)
  • 01:39 ori: gracefuling apaches
  • 00:55 mutante: icinga - manually deleted duplicate host labs-ns1 to fix icinga config and reloads

September 24

  • 23:21 ejegg: Updated paymentswiki from 3ac5dd1c3fade37b6f3a4879aef8ea71b3bbbf08 to 83464deed3b66da655ca5d1086852237c4793b71
  • 23:17 logmsgbot: catrope Synchronized php-1.24wmf22/extensions/VisualEditor: SWAT (duration: 00m 04s)
  • 23:14 logmsgbot: catrope Synchronized php-1.24wmf22/resources/lib/oojs-ui/: SWAT (duration: 00m 05s)
  • 23:12 greg-g: restarted jouncebot, he wasn't announcing deploy windows
  • 23:00 mutante: OCG - scheduled downtime/disabled notifications for LVS check
  • 22:44 andrewbogott: salted a bash update on labs instances, which turned out to be updated already.
  • 22:09 cscott: icinga VS HTTP IPv4 on ocg.svc.eqiad.wmnet test is most likely due to `du -s` of a 6G cache directory, not critical. timeouts can be increased to quiet it. i will look into adding a -quick parameter or some such tomorrow to make the health check faster.
  • 20:56 cscott: updated OCG to version 48acb8a2031863e35fad9960e48af60a3618def9
  • 20:43 logmsgbot: aaron Synchronized php-1.24wmf22/includes/cache/bloom: ad8a7a761d5f3bd086bbd6c88870e83c701e59e3 (duration: 00m 04s)
  • 20:00 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 15s)
  • 19:47 logmsgbot: yurik Synchronized php-1.24wmf22/extensions/ZeroBanner/: Updating to master (duration: 01m 10s)
  • 19:46 logmsgbot: yurik Synchronized php-1.24wmf21/extensions/ZeroBanner/: Updating to master (duration: 01m 07s)
  • 19:14 logmsgbot: yurik Finished scap: updating Graph, JsonConfig, ZeroBanner & ZeroPortal to master for 21 & 22 (duration: 07m 46s)
  • 19:07 logmsgbot: yurik Started scap: updating Graph, JsonConfig, ZeroBanner & ZeroPortal to master for 21 & 22
  • 18:55 logmsgbot: reedy Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 14s)
  • 18:53 logmsgbot: reedy Synchronized php-1.24wmf22/extensions/WikimediaMaintenance: (no message) (duration: 00m 14s)
  • 17:13 manybubbles: lowered throttling on Elasticsearch index transfer from one node to another speed because I hate excitement
  • 15:38 Nemo_bis: cscott> i'm working on the OCG health issue above. i'll let you know when i know what's going on. icinga-wm> PROBLEM - OCG health on ocg1002 is CRITICAL
  • 15:37 logmsgbot: demon Synchronized php-1.24wmf22/extensions/CentralAuth: (no message) (duration: 00m 05s)
  • 15:21 logmsgbot: demon Synchronized php-1.24wmf22/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php: (no message) (duration: 00m 05s)
  • 15:01 logmsgbot: demon Synchronized wmf-config/Wikibase.php: (no message) (duration: 00m 06s)
  • 14:57 Jeff_Green: restarted service ocg on ocg1001
  • 14:40 manybubbles: finished deployment - load spikes look to be gone. yay
  • 14:22 logmsgbot: manybubbles Synchronized php-1.24wmf21/extensions/CirrusSearch/: Switch implementation of Cirrus link counting jobs to hopefully lower overall load. (duration: 00m 04s)
  • 14:21 logmsgbot: manybubbles Synchronized wmf-config: More cirrus config to lower load (duration: 00m 04s)
  • 14:17 logmsgbot: manybubbles Synchronized wmf-config: Cirrus config to lower load (duration: 00m 04s)
  • 14:14 logmsgbot: manybubbles Synchronized php-1.24wmf22/extensions/CirrusSearch/: Switch implementation of Cirrus link counting jobs to hopefully lower overall load. (duration: 00m 06s)
  • 14:08 manybubbles: starting deployment to lower cirrus load spikes
  • 13:19 manybubbles: *disabled*
  • 13:17 manybubbles: disable row awareness on Cirrus's elasticsearch cluster - might help balance load better. too much load was on one row
  • 13:04 hashar: Zuul proceeding queue again
  • 13:00 hashar: Jenkins: disconnecting Gearman client from Zuul and reconnecting
  • 12:59 hashar: Zuul / Jenkins stuck
  • 09:33 hashar_: Jenkins switched mwext-UploadWizard-qunit back to Zuul cloner by applying pending change 161459
  • 09:33 hashar_: restarting zuul-merger
  • 09:32 hashar_: restarting zuul
  • 09:19 hashar_: Upgrading Zuul to f0e3688 Cherry pick https://review.openstack.org/#/c/123437/1 which fix bug 71133 Zuul cloner: fails on extension jobs against a wmf branch
  • 05:41 legoktm: ran script to back populate bug 70620 on metawiki (/home/legoktm/ca/populateBug70620.php on terbium)
  • 04:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 24 04:29:53 UTC 2014 (duration 29m 52s)
  • 03:34 logmsgbot: tstarling Finished scap: (no message) (duration: 12m 09s)
  • 03:22 logmsgbot: tstarling Started scap: (no message)
  • 03:21 logmsgbot: tstarling scap failed: RuntimeError scap requires SSH agent forwarding (duration: 00m 00s)
  • 03:12 logmsgbot: LocalisationUpdate completed (1.24wmf22) at 2014-09-24 03:12:54+00:00
  • 02:39 logmsgbot: LocalisationUpdate completed (1.24wmf21) at 2014-09-24 02:39:39+00:00
  • 02:10 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1062 (duration: 00m 06s)
  • 01:25 mutante: tridge - shutting down

September 23

  • 23:47 logmsgbot: maxsem Synchronized php-1.24wmf22/extensions/MobileFrontend/: (no message) (duration: 00m 04s)
  • 23:15 logmsgbot: maxsem Synchronized wmf-config/CommonSettings.php: fail! (duration: 00m 04s)
  • 23:12 logmsgbot: maxsem Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/162297/ (duration: 00m 03s)
  • 23:06 logmsgbot: maxsem Synchronized php-1.24wmf21/extensions/MassMessage/: https://gerrit.wikimedia.org/r/#/c/161002/ (duration: 00m 03s)
  • 22:04 logmsgbot: aaron Synchronized php-1.24wmf22/includes/jobqueue/JobRunner.php: f23f1ad35f02f6a17c9b5842aa6d8c152a273639 (duration: 00m 04s)
  • 21:54 logmsgbot: ebernhardson Finished scap: Bump flow submodule (and change an i18n message) in 1.24wmf21 and 1.24wmf22 (duration: 28m 14s)
  • 21:25 logmsgbot: ebernhardson Started scap: Bump flow submodule (and change an i18n message) in 1.24wmf21 and 1.24wmf22
  • 20:24 cscott: updated OCG to version 1cf9281ec3e01d6cbb27053de9f2423582fcc156
  • 19:38 mutante: stopped etherpad, added repairPad.js, attempted repair of pad 'WRN201409', started etherpad
  • 18:30 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 18:28 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 16s)
  • 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf22
  • 16:59 logmsgbot: aaron Synchronized wmf-config/jobqueue-eqiad.php: Removed redundant config due to new job runner (duration: 00m 05s)
  • 16:29 _joe_: manually created /srv/mediawiki bind mount on searchidx1001; moved old contents to /a/mediawiki-stale, to avoid filling the disk
  • 15:33 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT remove C and MW namspace aliases from ckbwiki (duration: 00m 07s)
  • 15:24 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT add *.beeldbank.cultureelerfgoed.nl to upload list (duration: 00m 04s)
  • 15:16 logmsgbot: manybubbles Synchronized php-1.24wmf21/extensions/CirrusSearch/: SWAT update Cirrus for better error handling (duration: 00m 04s)
  • 15:08 logmsgbot: manybubbles Synchronized php-1.24wmf22/extensions/CirrusSearch/: SWAT deploy cirrus backports (duration: 00m 05s)
  • 13:48 akosiaris: change url-downloader ip to point to the new one
  • 13:01 logmsgbot: manybubbles Synchronized wmf-config/: Throttle cirrus jobs some more. (duration: 00m 04s)
  • 12:24 logmsgbot: manybubbles Synchronized wmf-config/: Some new cirrus config (duration: 00m 07s)
  • 09:16 godog: deployed codfw-prod swift ring to palladium
  • 04:49 logmsgbot: tstarling Synchronized php-1.24wmf21/languages/Language.php: profiling (duration: 00m 05s)
  • 04:10 logmsgbot: tstarling Synchronized php-1.24wmf21/languages/Language.php: profiling (duration: 00m 05s)
  • 03:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 23 03:42:29 UTC 2014 (duration 42m 28s)
  • 03:29 logmsgbot: tstarling Synchronized wmf-config/CommonSettings.php: fix profiling (duration: 00m 07s)
  • 02:43 logmsgbot: LocalisationUpdate completed (1.24wmf22) at 2014-09-23 02:43:48+00:00
  • 02:30 logmsgbot: LocalisationUpdate completed (1.24wmf21) at 2014-09-23 02:30:38+00:00
  • 02:17 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-23 02:17:31+00:00
  • 00:26 mutante: tridge - revoking puppet cert, deleting salt key, decom ...

September 22

  • 23:49 logmsgbot: ebernhardson Synchronized php-1.24wmf22/extensions/LiquidThreads/: Bump LiquidThreads submodule in 1.24wmf22 (duration: 00m 06s)
  • 23:48 logmsgbot: ebernhardson Synchronized php-1.24wmf22/extensions/UploadWizard/: Bump UploadWizard submodule in 1.24wmf22 (duration: 00m 04s)
  • 23:46 logmsgbot: ebernhardson Synchronized php-1.24wmf21/extensions/LiquidThreads/: Bump LQT submodule in 1.24wmf21 (duration: 00m 04s)
  • 23:35 logmsgbot: ebernhardson Synchronized php-1.24wmf21/extensions/UploadWizard/: sync UploadWizard in 1.24wmf21 (duration: 00m 07s)
  • 23:32 logmsgbot: ebernhardson Synchronized php-1.24wmf21/includes/rcfeed/MachineReadableRCFeedFormatter.php: Use safe attribute accessor for RecentChange (duration: 00m 04s)
  • 23:30 logmsgbot: ebernhardson Synchronized php-1.24wmf21/extensions/UploadWizard/: Bump UploadWizard submodule in php-1.24wmf21 (duration: 00m 04s)
  • 23:30 logmsgbot: ebernhardson Synchronized php-1.24wmf21/extensions/Flow/: Bump flow submodule in php-1.24wmf21 (duration: 00m 06s)
  • 23:17 logmsgbot: ebernhardson Synchronized wmf-config/InitialiseSettings.php: Set wgUploadNavigationUrl for eowiki (duration: 00m 05s)
  • 21:04 bd808: production-logstash-eqiad healed by restarting elasticsearch on logstash1002 after OOM + split brain
  • 20:54 bd808: split brain on logstash1002 preceded by by java OOM for elasticsearch
  • 20:52 bd808: logstash1002 went split brain from rest of logstash elastic search cluster. restarting
  • 20:24 subbu: deployed Parsoid ff9476f9
  • 19:31 hashar: Jenkins is broken for extensions patches proposed against the wmf branches bug 71133
  • 18:32 Krinkle: lanthanum tmpfs filled up again, purged manually (bug 71128)
  • 17:22 ori: updated HHVM on beta cluster to HHVM to 3.3.0-20140918+wmf1
  • 17:00 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: Push Cirrus' non-content enwiki shards apart (no-op) (duration: 00m 04s)
  • 15:52 godog: reboot ms-be2001 into PXE to test a re-install
  • 15:07 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Graph extension on mediawiki.org gerrit:161908 (duration: 00m 09s)
  • 15:02 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Add securepoll-create-poll right to sysop on testwiki gerrit:161653 (duration: 00m 09s)
  • 15:01 logmsgbot: anomie Synchronized wmf-config/CommonSettings.php: SWAT: Add REL1_24 as branch in ExtensionDistributor gerrit:161666 (duration: 00m 10s)
  • 14:12 hashar: Jenkins deleted job mediawiki-core-lint , replaced by mediawiki-core-phplint
  • 12:10 apergos: shutdown of db1050 to install trusty
  • 10:04 hashar: Jenkins back and fully operational
  • 09:55 hashar: restarting jenkins
  • 09:37 hashar_: Jenkins: deleting old mediawiki extensions jobs (rm -fR /var/lib/jenkins/jobs/*testextensions-master). They are no more triggered and superseded by the *-testextension jobs.
  • 03:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 22 03:36:40 UTC 2014 (duration 36m 39s)
  • 02:41 logmsgbot: LocalisationUpdate completed (1.24wmf22) at 2014-09-22 02:41:29+00:00
  • 02:29 logmsgbot: LocalisationUpdate completed (1.24wmf21) at 2014-09-22 02:29:09+00:00
  • 02:16 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-22 02:16:20+00:00

September 21

  • 22:43 ori: ms-be1008 overloaded starting 18:00:24 UTC, syslog says "BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:1:2196]". machine became unresponsive at 21:35, coinciding with a spike of 5xxs, lasting until Coren powercycled it at 22:10.
  • 03:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 21 03:37:31 UTC 2014 (duration 37m 30s)
  • 03:16 springle: labsdb1001 mysqld restarted in gdb; crash loop with a labs user's table
  • 02:46 logmsgbot: ori Synchronized wmf-config/throttle.php: I7bb42b49a: Increase account creation throttle on enwiki for Cochrane colloquium. (duration: 00m 07s)
  • 02:41 logmsgbot: LocalisationUpdate completed (1.24wmf22) at 2014-09-21 02:41:36+00:00
  • 02:29 logmsgbot: LocalisationUpdate completed (1.24wmf21) at 2014-09-21 02:29:51+00:00
  • 02:16 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-21 02:16:56+00:00

September 20

  • 22:28 Krinkle: Reloading Zuul to deploy I0170766cfc06b8e6
  • 20:30 andrewbogott: rebooting virt1006 to make good and sure it doesn't spontaneously re-enter the compute pool
  • 20:29 andrewbogott_afk: moved all VMs off of virt1006, disabled compute service
  • 03:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Sep 20 03:46:00 UTC 2014 (duration 45m 59s)
  • 02:46 logmsgbot: LocalisationUpdate completed (1.24wmf22) at 2014-09-20 02:46:05+00:00
  • 02:33 logmsgbot: LocalisationUpdate completed (1.24wmf21) at 2014-09-20 02:33:34+00:00
  • 02:19 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-20 02:19:34+00:00

September 19

  • 22:16 RoanKattouw: Restarting Jenkins
  • 21:57 logmsgbot: spage Synchronized php-1.24wmf21/extensions/Flow/modules/new/components/flow-board.js: Flow bug 71054 backport (duration: 00m 04s)
  • 20:50 ori: restarted HHVM and cleared bytecode cache on all HHVM app servers
  • 20:47 _joe_: restarted hhvm on mw1018, cleaning the cache as well
  • 20:25 ori: Deployed Ic71064e08 (type hint fix for Wikidata) to wmf21/22.
  • 19:09 bblack: restarted hhvm on mw1021
  • 18:59 _joe_: rolling restart of hhvm servers
  • 18:22 bblack: restarting hhvm on mw1020 (again!)
  • 18:19 hashar: Jenkins: reverting job mwext-VisualEditor-qunit to previous state (i.e. without Zuul cloner)
  • 18:17 bblack: restarting hhvm on mw1020
  • 17:57 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I3e1bd5e4bb: Don't manipulate the environment to determine TZ offset (Bug: 71036) (duration: 00m 13s)
  • 17:30 bblack: turned down apache prefork procs on fenari to reduce swapping
  • 17:16 ottomata: initiating controlled shutdown of kafka broker analytics1021 to test some kafkatee weirdness, as well as a potential kafka/zookeeper bug
  • 17:07 bblack: restarting apache on fenari
  • 16:21 bblack: restarted hhvm on mw1019 + 1021
  • 14:57 hashar: Jenkins friday deploy: migrate all MediaWiki extension qunit jobs to Zuul cloner.
  • 14:37 akosiaris: initiated rsync of tridge data that is to be kept to nas1001-a
  • 13:56 springle: killing any sleeping connection on enwiki db slaves to make room
  • 13:56 mark: Stopped jobrunners on mw1001-1003
  • 12:36 springle: temporarily disable log fsync on enwiki slaves
  • 12:14 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1072 with ReadAheadNone (duration: 00m 09s)
  • 11:32 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1072. seems more susceptible to replag; find out why. (duration: 00m 10s)
  • 09:14 _joe_: restarted hhvm on mw1053, stuck to 100% cpu since last restart (activating stats)
  • 05:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 19 05:01:54 UTC 2014 (duration 1m 52s)
  • 03:45 logmsgbot: LocalisationUpdate completed (1.24wmf22) at 2014-09-19 03:45:33+00:00
  • 03:11 logmsgbot: LocalisationUpdate completed (1.24wmf21) at 2014-09-19 03:11:43+00:00
  • 02:38 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-19 02:38:25+00:00
  • 00:43 cscott: updated OCG to version ce16f7adb60d7c77409e2e11ba0e5d6cce6955d5

September 18

  • 23:55 logmsgbot: ori Started scap: Add HHVM as a beta feature
  • 23:54 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I2466f6b6e: Add HHVM to beta feature whitelist (duration: 00m 08s)
  • 23:52 logmsgbot: ori Synchronized php-1.24wmf22/extensions/WikimediaEvents: Update WikimediaEvents for cherry-picks (duration: 00m 06s)
  • 23:51 logmsgbot: ori Synchronized php-1.24wmf21/extensions/WikimediaEvents: Update WikimediaEvents for cherry-picks (duration: 00m 06s)
  • 23:25 logmsgbot: catrope Synchronized php-1.24wmf22/resources/lib/oojs-ui/: oojs-ui bugfixes (duration: 00m 06s)
  • 23:13 logmsgbot: catrope Synchronized php-1.24wmf22/extensions/VisualEditor/: SWAT (duration: 00m 08s)
  • 23:04 logmsgbot: catrope Synchronized php-1.24wmf21/extensions/UploadWizard/: SWAT (duration: 00m 08s)
  • 19:57 Jeff_Green: iridium.wm.o exim conf checked, puppet reenabled
  • 19:54 Jeff_Green: magnesium.wm.o exim conf checked, puppet reenabled
  • 19:50 Jeff_Green: sodium.wm.o exim conf checked, puppet reenabled
  • 19:48 logmsgbot: reedy Synchronized php-1.24wmf22/extensions/Flow/: (no message) (duration: 00m 16s)
  • 19:45 Jeff_Green: iodine.wm.o exim conf checked, puppet reenabled
  • 19:44 Jeff_Green: polonium.wm.o exim conf checked, puppet reenabled
  • 19:35 Jeff_Green: lead.wm.o exim conf checked, puppet reenabled
  • 19:22 logmsgbot: reedy Synchronized php-1.24wmf22: (no message) (duration: 00m 57s)
  • 19:16 Jeff_Green: disabling puppet on polonium, lead, sodium, iridium, magnesium, and iodine to monitor rollout of https://gerrit.wikimedia.org/r/155753
  • 19:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rest of group0 to 1.24wmf22
  • 19:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf21
  • 18:59 bblack: restarting apache on fenari
  • 18:49 logmsgbot: reedy Finished scap: testwiki to 1.24wmf22 and build l10n cache (duration: 30m 23s)
  • 18:44 Jeff_Green: testing exim configuration change on lead.wm.o
  • 18:18 logmsgbot: reedy Started scap: testwiki to 1.24wmf22 and build l10n cache
  • 17:49 logmsgbot: reedy Started scap: testwiki to 1.24wmf22 and build l10n cache
  • 17:08 cmjohnson1: replacing failed disk es1005
  • 17:05 logmsgbot: yurik Finished scap: (no message) (duration: 23m 26s)
  • 16:43 yurikR: yurik scaping zero - partner needs an l10n message asap
  • 16:42 logmsgbot: yurik Started scap: (no message)
  • 15:38 hashar: restarting Zuul just to be safe
  • 15:06 logmsgbot: anomie Synchronized php-1.24wmf21/resources/src/mediawiki.action/mediawiki.action.view.redirectPage.css: SWAT: mediawiki.action.view.redirectPage: Correct a CSS selector gerrit:161239 (duration: 00m 23s)
  • 15:01 logmsgbot: anomie Synchronized php-1.24wmf21/extensions/Wikidata/: SWAT: Update Wikidata to fix broken xml api output gerrit:161232 (duration: 00m 38s)
  • 11:40 apergos: forgot to log this earlier: manually started salt minion on radon, elastic1015, searchidx1001, it wasn't running there
  • 09:00 godog: updated authdns to 0c2225d
  • 08:56 springle: xtrabackup clone db1016 to db2010
  • 07:48 godog: re-enabled icinga notifications for ms-be1001
  • 07:09 bblack: removing pybal cfg "eqiad/misc_web_https" (unused now, https://gerrit.wikimedia.org/r/161183)
  • 06:53 bblack: removing pybal cfg "esams/wikimedialbsecure" (unused, points at maerlant)
  • 06:47 bblack: removing pybal symlink "$site/ipv6", also unused (old ipv6 protoproxying)
  • 06:45 bblack: removing pybal symlink "$site/text-varnish", seems to be a remnant no longer in use
  • 04:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 18 04:20:56 UTC 2014 (duration 20m 55s)
  • 03:09 logmsgbot: LocalisationUpdate completed (1.24wmf21) at 2014-09-18 03:09:44+00:00
  • 02:53 logmsgbot: yurik Synchronized wmf-config/CommonSettings.php: (no message) (duration: 01m 53s)
  • 02:52 yurikR: yurik Fixing graph ext namespace name - otherwise get screen of WMF death on graph: ns visits
  • 02:36 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-18 02:36:46+00:00
  • 00:32 logmsgbot: marktraceur Finished scap: [SWAT] Move things out of assets/ and into resources/assets/ (duration: 35m 28s)

September 17

  • 23:57 logmsgbot: marktraceur Started scap: [SWAT] Move things out of assets/ and into resources/assets/
  • 23:47 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] Enable Graph on metawiki and labswiki (duration: 00m 10s)
  • 23:42 logmsgbot: marktraceur Synchronized php-1.24wmf21/extensions/Graph/: [SWAT] Update Graph to master (duration: 00m 08s)
  • 23:41 logmsgbot: marktraceur Synchronized php-1.24wmf20/extensions/Graph/: [SWAT] Update Graph to master (duration: 00m 07s)
  • 23:35 logmsgbot: marktraceur Synchronized php-1.24wmf21/extensions/MultimediaViewer/: [SWAT] Fix reuse dropdown message weirdness (duration: 00m 07s)
  • 23:29 logmsgbot: marktraceur Synchronized php-1.24wmf20/extensions/MultimediaViewer/: [SWAT] Fix reuse dropdown message weirdness (duration: 00m 08s)
  • 23:10 logmsgbot: marktraceur Synchronized php-1.24wmf21/extensions/UploadWizard/: [SWAT] Fix EventLogging schema declarations for UploadWizard (duration: 00m 11s)
  • 21:41 mutante: fixing updates on planet feeds - file permissions
  • 21:11 manybubbles: restarting rebuilding cirrus's enwiki index now that I've found the reason it wasn't working before - the new index was putting too many shards on an already full node and overwhelming it. silly allocation algorithm! thats a bad idea!
  • 21:07 logmsgbot: yurik Synchronized php-1.24wmf21/extensions/ZeroPortal/: (no message) (duration: 01m 05s)
  • 20:19 godog: rebooting ms-be1006
  • 19:00 Krinkle: jenkins-slave tmpfs on lanthanum was filling up (> 500MB). I purged tmp dbs for old jobs. We should get these purged automatically and also increase the size as 500MB is too little.
  • 18:59 robh: disabled icinga alerts for ms-be1001, rebooting it to look at its raid bios settings for codfw deployment mirroring
  • 18:47 logmsgbot: yurik Synchronized php-1.24wmf20/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 39s)
  • 18:43 logmsgbot: yurik Synchronized php-1.24wmf21/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 35s)
  • 18:40 logmsgbot: yurik Synchronized wmf-config/: private wikis login/logout page names, zeroportal impersonator acct (duration: 01m 06s)
  • 18:23 mutante: phabricator - made aklapper an admin
  • 17:26 logmsgbot: andrew rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
  • 17:23 logmsgbot: andrew Synchronized wikiversions.json: (no message) (duration: 00m 05s)
  • 17:04 manybubbles: cirrus brownout looks just about fixed. So! My plan for periodically explicitly merging deletes has some problems.....
  • 16:42 gwicke: restarted parsoid on wtp102{2,3,4}
  • 16:31 manybubbles: just going to make this clear - the current cirrus brownout doesn't seem to be effecting my queries but we're getting hit with pool counter full events - sadness. its not caused by switching cirrus to ruwiki's primary backend - its caused by me attempting to perform index maintenance activities.
  • 16:23 akosiaris: restarted node on wtp boxes except wtp1022,wtp1023,wtp1024
  • 16:23 manybubbles: caused cirrus brownout by executing a force merge for enwiki's general index. ooops
  • 16:06 logmsgbot: manybubbles Synchronized wmf-config/: set cirrus as primary search backend for ruwiki and make permanent some settings set on the fly (duration: 00m 06s)
  • 15:57 manybubbles: manually pushed apart ruwiki and nlwiki's shards as well - might help - updated commit to reflect that
  • 15:42 manybubbles: gerrit change to lock that into place is https://gerrit.wikimedia.org/r/#/c/160974/ and I'll deploy it in my window in 15 minutes.
  • 15:41 manybubbles: manually forcing Cirrus's commonswiki's file index apart from one another in an attempt to lower the consistently high load on elastic1013
  • 15:34 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Set wgMetaNamespace for labswiki (duration: 00m 14s)
  • 14:54 springle: db1062 out of action for bug hunt https://mariadb.atlassian.net/browse/MDEV-6751
  • 14:48 logmsgbot: reedy Synchronized wmf-config/interwiki.cdb: (no message) (duration: 00m 16s)
  • 14:45 godog: restarted apache2 on magnesium, validate removal of ssl certs
  • 13:38 hashar: Zuul upgraded successfully apparently.
  • 13:33 hashar: stopping zuul for upgrade
  • 13:29 hashar: upgrading Zuul to 2.0.0.286.gb1811ab
  • 12:20 hashar: upgrading jenkins 1.565.1 -> 1.565.2
  • 09:53 akosiaris: stopped apache2 on fenari, it was leaking memory, puppet restarted it, need to kill this machine ASAP
  • 09:52 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool s1 db1061 (duration: 00m 08s)
  • 06:55 springle: xtrabackup clone db1061 to db2016
  • 06:52 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool s1 db1061 for codfw cloning (duration: 00m 07s)
  • 06:27 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool s7 db1039 (duration: 00m 08s)
  • 04:34 logmsgbot: tstarling Synchronized docroot/bits: (no message) (duration: 00m 10s)
  • 04:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 17 04:32:17 UTC 2014 (duration 32m 16s)
  • 03:17 logmsgbot: LocalisationUpdate completed (1.24wmf21) at 2014-09-17 03:17:38+00:00
  • 03:07 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool s6 db1015 (duration: 01m 41s)
  • 02:43 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-17 02:43:02+00:00
  • 02:21 springle: xtrabackup clone db1048 to db2012
  • 02:15 springle: xtrabackup clone db1046 to db2011
  • 02:00 springle: xtrabackup clone db1016 to db2010
  • 01:54 springle: xtrabackup clone db1031 to db2009
  • 01:33 springle: xtrabackup clone db1039 to db2029
  • 01:33 springle: xtrabackup clone db1015 to db2028
  • 01:29 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool s6 db1015 and s7 db1039 (duration: 00m 20s)
  • 01:15 Reedy: updateCollation on shwiki done
  • 00:59 Reedy: running `mwscript updateCollation.php --wiki=shwiki --previous-collation=uppercase` in screen on tin
  • 00:58 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: shwiki collation (duration: 00m 16s)
  • 00:53 Reedy: updateCollation on etwiki done
  • 00:52 Reedy: updateCollation on etwiktionary done
  • 00:48 Reedy: running `mwscript updateCollation.php --wiki=etwiktionary --previous-collation=uppercase` in screen on tin
  • 00:47 Reedy: etwikisource collation updated (9918 rows)
  • 00:47 Reedy: etwikiquote collation updated (706 rows)
  • 00:46 Reedy: etwikimedia collation updated (121 rows)
  • 00:46 Reedy: etwikibooks collation updated (280 rows)
  • 00:45 Reedy: running `mwscript updateCollation.php --wiki=etwiki --previous-collation=uppercase` in screen on tin
  • 00:45 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: et collations (duration: 00m 15s)
  • 00:43 Reedy: updateCollation on frwikiversity done
  • 00:42 Reedy: running `mwscript updateCollation.php --wiki=frwikiversity --previous-collation=uppercase` in screen on tin
  • 00:42 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: frwikiversity collation (duration: 00m 17s)
  • 00:40 Reedy: updateCollation on skwiki done
  • 00:26 Reedy: Running `mwscript updateCollation.php --wiki=skwiki --previous-collation=uppercase` in screen on tin
  • 00:25 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: skwiki collation (duration: 00m 15s)
  • 00:18 logmsgbot: reedy Synchronized database lists: (no message) (duration: 00m 15s)

September 16

  • 23:22 logmsgbot: maxsem Synchronized php-1.24wmf21/extensions/VisualEditor/: (no message) (duration: 00m 04s)
  • 23:21 logmsgbot: maxsem Synchronized php-1.24wmf20/extensions/VisualEditor/: (no message) (duration: 00m 04s)
  • 23:16 logmsgbot: maxsem Synchronized php-1.24wmf21/extensions/Wikidata: (no message) (duration: 00m 24s)
  • 23:15 MaxSem: Wikidata submodule in wmf21 was in the middle of rebase - reset and updating to a newer submodule commit
  • 23:12 logmsgbot: maxsem Synchronized php-1.24wmf20/extensions/Wikidata: (no message) (duration: 00m 17s)
  • 23:07 logmsgbot: maxsem Synchronized php-1.24wmf21/extensions/GettingStarted: https://gerrit.wikimedia.org/r/#/c/160084/ (duration: 00m 08s)
  • 21:36 Jeff_Green: SPF record deployed for donate.wikimedia.org
  • 21:01 logmsgbot: ejegg Synchronized php-1.24wmf20/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js: (no message) (duration: 00m 06s)
  • 19:38 csteipp: deployed patches for bugs 70469 and 70672
  • 19:17 logmsgbot: catrope Synchronized php-1.24wmf21/extensions/VisualEditor/: Revert IE hacks so Firefox will stop corrupting non-Latin characters (duration: 00m 06s)
  • 19:15 logmsgbot: catrope Synchronized php-1.24wmf20/extensions/VisualEditor/: (no message) (duration: 00m 09s)
  • 18:32 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 15s)
  • 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf21
  • 17:03 logmsgbot: bd808 Finished scap: No code change scap to test scap internal update (duration: 18m 06s)
  • 16:45 logmsgbot: bd808 Started scap: No code change scap to test scap internal update
  • 16:43 bd808|deploy: Updated scap to 663f137 (Check php syntax with parallel `php -l`)
  • 16:42 bd808|deploy: Trebuchet sync for scap reporting failure from osmium.eqiad.wmnet, mw1053.eqiad.wmnet, searchidx1001.eqiad.wmnet, fenari.wikimedia.org, and mw1110.eqiad.wmnet
  • 16:41 bd808|deploy: Trebuchet update for scap reporting failure from osmium.eqiad.wmnet, searchidx1001.eqiad.wmnet, fenari.wikimedia.org and mw1110.eqiad.wmnet
  • 16:00 _joe_: mw1018 and mw1021 in the hhvm appservers pool
  • 15:35 logmsgbot: reedy Synchronized docroot and w: Update symlinks to use /srv/mediawiki (duration: 00m 16s)
  • 15:34 hashar: Jenkins: deleting /srv/ssd/jenkins-slave/workspace/*testextensions-master on gallium and lanthanum.
  • 15:25 logmsgbot: andrew Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 03s)
  • 15:23 logmsgbot: andrew Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 19s)
  • 15:20 manybubbles: SWAT complete
  • 15:16 logmsgbot: manybubbles Synchronized php-1.24wmf20/extensions/VisualEditor/: swat update for wmf20 (duration: 00m 25s)
  • 15:13 hashar: Jenkins: mediawiki extensions phpunit jobs should pass more or less until the CI system is sent an orbit and dies out horribly. in such a case ping me / phone.
  • 15:08 logmsgbot: manybubbles Synchronized php-1.24wmf21/extensions/VisualEditor/: SWAT visual editor update wmf21 (duration: 00m 07s)
  • 14:52 ottomata: set vm.dirty_expire_centisecs to 10000 (was 30000) on analytics1021 to experiment with paging and kafka-zookeeper timeouts
  • 14:36 godog: stopped htcp-purger on ms1004 RT #8358
  • 14:32 godog: silenced ms-be1014 until torrow, pending forced reboot
  • 14:28 hashar: Jenkins: breaking continuous integration for MediaWiki repositories. Extensions are now tested with mediawiki/vendor and, mediawiki/core is checked out to the patch branch if it exist. 160656
  • 14:20 akosiaris_: restarted apache on fenari , it was leaking memory, situation back to normal, cause unknown yet
  • 14:12 akosiaris_: stopped apache on fenari . It was in swap, investigating
  • 12:35 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool s2 db1054, s3 db1027, s4 db1056, s5 db1037 (duration: 00m 10s)
  • 12:26 godog: reboot ms-be1014, xfs issues
  • 12:22 godog: temporarily chgrp wikidev /var/log/hhvm/error.log on mw1018
  • 12:21 logmsgbot: reedy Synchronized php-1.24wmf20/LocalSettings.php: Fix path to be /srv based (duration: 00m 32s)
  • 11:25 logmsgbot: reedy Synchronized docroot and w: (no message) (duration: 00m 35s)
  • 11:12 logmsgbot: reedy Purged l10n cache for 1.24wmf19
  • 11:12 logmsgbot: reedy Purged l10n cache for 1.24wmf18
  • 11:10 logmsgbot: reedy Purged l10n cache for 1.24wmf15
  • 09:21 _joe_: reimaging mw1018 and mw1021 w HAT: removing from pybal, etc.
  • 06:29 springle: xtrabackup clone db1037 to db2023
  • 05:31 springle: xtrabackup clone db1056 to db2019
  • 04:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 16 04:01:05 UTC 2014 (duration 1m 4s)
  • 03:11 springle: xtrabackup clone db1027 to db2018
  • 03:04 logmsgbot: LocalisationUpdate completed (1.24wmf21) at 2014-09-16 03:04:46+00:00
  • 02:53 springle: xtrabackup clone db1054 to db2017
  • 02:50 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool s2 db1054, s3 db1027, s4 db1056, s5 db1037 for codfw cloning (duration: 01m 12s)
  • 02:39 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1036, depool db1002 (duration: 00m 07s)
  • 02:31 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-16 02:31:16+00:00

September 15

  • 23:32 logmsgbot: maxsem Synchronized php-1.24wmf21/resources/: SWAT: https://gerrit.wikimedia.org/r/#/c/160488/1 https://gerrit.wikimedia.org/r/#/c/160543/ (duration: 00m 06s)
  • 23:26 bblack: restarting lvs1001 for HT disable + kernel upgrade
  • 23:19 logmsgbot: maxsem Synchronized php-1.24wmf21/extensions/VisualEditor/: SWAT: https://gerrit.wikimedia.org/r/#/c/160554/ (duration: 00m 07s)
  • 23:12 bblack: restarting lvs1002 for HT disable + kernel upgrade
  • 23:07 Krinkle: Running sample job on integration-slave1006 and warming up npmjs.org cache
  • 22:56 Krinkle: Running sample job on integration-slave1008 and warming up npmjs.org cache
  • 22:49 Krinkle: Running sample job on integration-slave1007 and warming up npmjs.org cache
  • 22:48 Krinkle: Pooling the newly setup Trusty-based Jenkins slaves (integration-slave1006, integration-slave1007 and integration-slave1008)
  • 22:42 bblack: dropping static routes for 2620:0:861:ed1a::[d,f,10,11] -> lvs1005 from cr[12]-eqiad (only 11 is of any consequence, misc-web-lb, and they're advertised by bgp and this is preventing failover to lvs1002)
  • 21:28 cscott: updated OCG to version 188a3c221d927bd0601ef5e1b0c0f4a9d1cdbd31
  • 20:46 subbu: deployed Parsoid version b845bff9
  • 18:49 logmsgbot: ejegg Synchronized php-1.24wmf20/extensions/CentralNotice/: Update CentralNotice to remove jquery.json dependency (duration: 00m 23s)
  • 18:46 hoo: Sync to tmh100[12] failed, according to awight
  • 18:44 logmsgbot: ejegg Synchronized php-1.24wmf21/extensions/CentralNotice/: Update CentralNotice to remove jquery.json dependency (duration: 00m 09s)
  • 18:43 manybubbles: performance tests show cirrus should handle jawiki with no problem but if load spirals out of control and I'm not around then revert https://gerrit.wikimedia.org/r/#/c/160465/
  • 18:40 hoo: Local part of the global rename of Gnumarcoo => .avgas fatally timed out on itwiki. This needs to be fixed per hand.
  • 18:40 manybubbles: Setting Cirrus to jawiki's primary search backend went well but Japan is mostly asleep. If Elasticsearch load takes a turn for the worse in four or five hours then we'll know how it went.
  • 17:14 bd808: Restarted elasticsearch on logstash1003; 2014-09-14T09:33:57Z java.lang.OutOfMemoryError
  • 17:09 _joe_: killing salt-call on all mediawiki hosts
  • 17:06 bd808: Restarted elasticsearch on logstash1001; 2014-09-15T06:12:09Z java.lang.OutOfMemoryError
  • 17:04 bblack: using salt to kill salt-minion everywhere...
  • 17:02 bd808: Restarted logstash on logstash1001. I hoped this would fix the dashboards, but it looks like the backing elasticsearch cluster is too sad for them to work at the moment.
  • 16:55 bd808: Restarted hung elasticsearch service on logstash1002
  • 16:15 manybubbles: jawiki now has cirrus as primary. we're back to where we were before the great cascading failure of two months ago
  • 16:13 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s)
  • 15:29 logmsgbot: marktraceur Synchronized php-1.24wmf21/extensions/MultimediaViewer/: [SWAT] Several backports for metrics and bugfixes in Media Viewer (duration: 00m 07s)
  • 15:27 logmsgbot: marktraceur Synchronized php-1.24wmf20/extensions/MultimediaViewer/: [SWAT] Several backports for metrics and bugfixes in Media Viewer (duration: 00m 07s)
  • 15:18 logmsgbot: marktraceur Synchronized php-1.24wmf21/extensions/GeoCrumbs/GeoCrumbs.class.php: [SWAT] Handle return value NULL of GeoCrumbs::getParserCache (duration: 00m 07s)
  • 15:17 logmsgbot: marktraceur Synchronized php-1.24wmf20/extensions/GeoCrumbs/GeoCrumbs.class.php: [SWAT] Handle return value NULL of GeoCrumbs::getParserCache (duration: 00m 07s)
  • 15:06 logmsgbot: marktraceur Synchronized wmf-config/: [SWAT] Remove 'renameuser' right from bureaucrats on CentralAuth wikis (duration: 00m 09s)
  • 14:54 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Bump wikibase memcached key for test.wikidata, test, test2 (duration: 00m 16s)
  • 14:54 hashar: Updated Jenkins Job Builder fork: e5c0c61..2d74b16
  • 14:50 logmsgbot: aude Finished scap: Put test.wikidata back on mw1.24-wmf19 extension branch (duration: 37m 27s)
  • 14:43 manybubbles: restarting the enwiki cirrus reindex process - it crashed over the weekend. why you crash and leave error message "1". "1" is not a useful error message.
  • 14:13 logmsgbot: aude Started scap: Put test.wikidata back on mw1.24-wmf19 extension branch
  • 13:03 _joe_: fenari is swapping hard, restarting apache who was eating up all the RAM
  • 09:20 logmsgbot: hashar Synchronized wmf-config/InitialiseSettings.php: *.scienceimage.csiro.au to the wgCopyUploadsDomains 159999 bug 70771 (duration: 00m 06s)
  • 09:15 hashar: Jenkins: apt-get upgrade on prod slaves (updates php5 / libc / jdk 7)
  • 03:09 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1036 (duration: 00m 09s)
  • 02:03 logmsgbot: LocalisationUpdate failed: mwversionsinuse returned empty list
  • 01:47 logmsgbot: hoo Synchronized wmf-config/liquidthreads.php: Remove global $path (duration: 00m 07s)
  • 01:47 logmsgbot: hoo Synchronized wmf-config/flaggedrevs.php: Remove global $path (duration: 00m 10s)

September 14

  • 20:37 ori_: enabling puppet on mw1053
  • 20:11 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1062, locked up (duration: 00m 09s)
  • 13:24 _joe_: stopped puppet aand the JR on mw1053
  • 12:42 hoo: Ran sync-common on mw1053 to stop "Unrecognized job type 'ChangeNotification'." exceptions
  • 11:14 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool es1005 (duration: 00m 07s)
  • 10:37 springle: restart es1005
  • 09:56 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool es1007, depool es1005 (duration: 00m 10s)
  • 02:01 logmsgbot: LocalisationUpdate failed: mwversionsinuse returned empty list
  • 00:45 ori_: fenari appears to still have twemproxy (in addition to nutcracker); decom'ing.
  • 00:29 ori_: restarting apache2 on fenari

September 13

  • 04:42 legoktm: global rename for Trevor Parscal (WMF) unstuck itself, yay
  • 04:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Sep 13 04:22:04 UTC 2014 (duration 22m 3s)
  • 03:51 legoktm: global rename for Trevor Parscal --> Trevor Parscal (WMF) looks stuck on metawiki and mswiki, in queued state for both but showJobs.php says the jobs are active and claimed
  • 03:11 logmsgbot: LocalisationUpdate completed (1.24wmf21) at 2014-09-13 03:11:40+00:00
  • 02:38 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-13 02:38:26+00:00
  • 01:45 logmsgbot: ori Synchronized php-1.24wmf21/extensions/Flow: Update flow for I4da934dfe (duration: 00m 06s)
  • 01:45 logmsgbot: ori Synchronized php-1.24wmf20/extensions/Flow: Update flow for I4da934dfe (duration: 00m 06s)
  • 01:41 logmsgbot: ori Synchronized php-1.24wmf20/extensions/Flow: Update flow for I4da934dfe (duration: 00m 08s)

September 12

  • 21:26 csteipp: deployed fixes for bugs 70620, 69008
  • 20:37 logmsgbot: mattflaschen Synchronized php-1.24wmf21/extensions/GettingStarted/: Deploy to fix GettingStarted bucketting for users with null registration date (duration: 00m 05s)
  • 20:37 logmsgbot: mattflaschen Synchronized php-1.24wmf20/extensions/GettingStarted/: Deploy to fix GettingStarted bucketting for users with null registration date (duration: 00m 07s)
  • 19:34 legoktm: running migratePass0.php across all CentralAuth wikis
  • 17:43 logmsgbot: ori updated /a/common to I4e4187285: Rename some constants to clarify their meaning and purpose
  • 14:52 manybubbles: rebuilding enwiki's Cirrus index for more performance testing. Please be faster now. k?
  • 08:37 _joe_: rolling restart of pybal finished. Adding note on Fenari
  • 08:19 _joe_: reactivated puppet on all lvs hosts, esams almost done, pending eqiad
  • 08:06 _joe_: new pybal conf applied in all of ulsfo
  • 07:39 _joe_: changing pybal config place; stopping puppet on all loadbalancers
  • 04:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 12 04:27:17 UTC 2014 (duration 27m 16s)
  • 03:15 logmsgbot: LocalisationUpdate completed (1.24wmf21) at 2014-09-12 03:15:57+00:00
  • 03:08 logmsgbot: mattflaschen Finished scap: One last CSS fix (wrapping issue for error state) for GettingStarted A/B test (duration: 24m 38s)
  • 02:43 logmsgbot: mattflaschen Started scap: One last CSS fix (wrapping issue for error state) for GettingStarted A/B test
  • 02:39 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-12 02:39:35+00:00
  • 01:33 logmsgbot: mattflaschen Synchronized php-1.24wmf21/extensions/GettingStarted/: CSS tweaks for GettingStarted A/B test (duration: 00m 07s)
  • 01:32 logmsgbot: mattflaschen Synchronized php-1.24wmf20/extensions/GettingStarted/: CSS tweaks for GettingStarted A/B test (duration: 00m 21s)
  • 01:29 logmsgbot: ori Synchronized wmf-config/wikitech.php: Ia5b81076e: Update path reference for /srv/mediawiki (duration: 00m 04s)
  • 01:28 logmsgbot: ori updated /a/common to Ia5b81076e: Update path reference for /srv/mediawiki
  • 01:19 ori: manually migrated /u/l/a/common-local to /srv/mediawiki on virt1000
  • 00:36 logmsgbot: ori Synchronized php-1.24wmf21/extensions/Wikidata: Update Wikidata to tip of master for I23b7eb54b8e (Bug: 70747) (duration: 00m 08s)
  • 00:12 logmsgbot: esanders Synchronized php-1.24wmf21/resources/lib/oojs-ui/: (no message) (duration: 00m 03s)
  • 00:12 logmsgbot: esanders Synchronized php-1.24wmf21/extensions/MultimediaViewer/: (no message) (duration: 00m 07s)
  • 00:00 logmsgbot: esanders Finished scap: SWAT deploy (duration: 28m 39s)

September 11

  • 23:31 logmsgbot: esanders Started scap: SWAT deploy
  • 23:29 logmsgbot: mattflaschen Finished scap: Deploy new GettingStarted recommendations A/B test (duration: 99m 34s)
  • 23:15 logmsgbot: esanders scap failed: LockFailedError Failed to lock /var/lock/scap: [Errno 11] Resource temporarily unavailable (duration: 00m 00s)
  • 23:00 mutante: restarting icinga-wm for config change
  • 21:49 logmsgbot: mattflaschen Started scap: Deploy new GettingStarted recommendations A/B test
  • 21:14 Krinkle: Stopping/starting zuul
  • 21:08 andrewbogott: restarting zuul on gallium
  • 20:58 andrewbogott: restarted jenkins, maybe
  • 20:56 ori: graceful'd apache on mw1053, missed it earlier
  • 20:49 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I1f3234746: Revert Scribunto: double the Lua CPU limit on the job runners (duration: 00m 05s)
  • 20:48 logmsgbot: ori updated /a/common to I1f3234746: Revert "Scribunto: double the Lua CPU limit on the job runners"
  • 20:42 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 20:15 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 13s)
  • 20:15 andrewbogott: syncing virt1000, again in hopes of moving to wmf20
  • 20:08 logmsgbot: reedy Synchronized php-1.24wmf21/extensions/Wikidata/: (no message) (duration: 00m 17s)
  • 19:58 Reedy: Running sync-common on mw1024
  • 19:52 Reedy: Running manual sync-common on mw1138
  • 19:51 logmsgbot: reedy Synchronized wmf-config/: Fix Zero settings (duration: 00m 15s)
  • 19:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf21
  • 19:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.24wmf20
  • 19:20 mutante: graceful'ed apache on mw1143
  • 19:16 Reedy: running sync-common on mw1143
  • 19:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
  • 19:02 bd808: Restarted elasticsearch on logstash1003 -- Java OOM error in logs and not recovering shards
  • 18:54 ori: graceful'd all apaches
  • 18:51 ori: graceful'd apache on mw1047, mw1151, mw1137, mw1146 and mw1076
  • 18:46 logmsgbot: ori Synchronized php-1.24wmf19/includes/WebStart.php: (no message) (duration: 00m 06s)
  • 18:45 logmsgbot: ori Synchronized php-1.24wmf19/includes/profiler/Profiler.php: (no message) (duration: 00m 07s)
  • 18:17 logmsgbot: reedy Started scap: testwiki to 1.24wmf21 and build l10n cache take 3
  • 18:16 logmsgbot: reedy scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/a/common/wmf-config/extension-list" --output="/tmp/tmp.Nd45X2RONi" --verbose' returned non-zero exit status 1 (duration: 01m 18s)
  • 18:14 logmsgbot: reedy Started scap: testwiki to 1.24wmf21 and build l10n cache
  • 18:13 logmsgbot: reedy scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/a/common/wmf-config/extension-list" --output="/tmp/tmp.IH8przTNHs" ' returned non-zero exit status 1 (duration: 04m 59s)
  • 18:08 logmsgbot: reedy Started scap: testwiki to 1.24wmf21 and build l10n cache
  • 18:02 manybubbles: raised logging on Elasticsearch cluster temporarily to get more information about merging - a process super important to keeping the index up to date in "real time"
  • 17:20 logmsgbot: ori updated /a/common to I0bda3deab: Replace remaining references to /u/l/a/common
  • 17:18 logmsgbot: ori updated /a/common to I37b0a8338: Get rid of MULTIVER_CDB_DIR_{APACHE,HOME}
  • 16:57 andrewbogott: sync-common on virt1000 -- with any luck this will upgrade us to wmf20
  • 16:56 logmsgbot: andrew rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
  • 16:53 logmsgbot: bd808 Finished scap: Preparing to move wikitech to 1.24wmf20 (second try) (duration: 24m 25s)
  • 16:46 andrewbogott: apache graceful on mw1039
  • 16:33 bd808|deploy: andrewbogott did apache graceful on mw1120 to stop wikidata APC logspam
  • 16:29 logmsgbot: bd808 Started scap: Preparing to move wikitech to 1.24wmf20 (second try)
  • 16:22 logmsgbot: andrew Finished scap: Preparing to move wikitech to 1.24wmf20 (duration: 06m 45s)
  • 16:19 bd808: Restarted logstash on logstash1001. Log empty and events not being stored in elasticsearch
  • 16:15 logmsgbot: andrew Started scap: Preparing to move wikitech to 1.24wmf20
  • 15:45 bblack: icinga config is correct now, back to normal puppet updates
  • 15:24 bblack: restarted icinga, manually removed some labsy things that were broken in config and temporarily disabled puppet :p
  • 14:44 _joe_: php upgrade finished
  • 14:23 _joe_: upgrading php across the cluster: libapache2-mod-php5 php5-cli php-pear php5 php5-common php5-curl php5-dev php5-intl php5-mysql php5-xmlrpc
  • 13:04 akosiaris: uploaded php5_5.3.10-1ubuntu3.14+wmf1 on apt.wikimedia.org
  • 10:00 _joe_: enabled puppet on mw1053
  • 09:38 _joe_: gracefulling mw1200 mw1196 and mw1186 as they have APC issues
  • 09:21 _joe_: upgrading hhvm and hhvm-luasandbox across the production cluster
  • 09:00 akosiaris: upgrading php5 to 5.3.10-1ubuntu3.14+wmf1 on mw1212
  • 08:34 _joe_: updating php-pear php5 php5-cli php5-common php5-curl php5-dev php5-intl php5-mysql php5-xmlrpc libapache2-mod-php5 on mw1018, see USN 2344-1
  • 03:41 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 11 03:41:03 UTC 2014 (duration 41m 2s)
  • 02:49 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-11 02:49:26+00:00
  • 02:36 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-09-11 02:36:37+00:00
  • 02:23 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-09-11 02:23:29+00:00
  • 00:28 mutante: graceful'ed Apaches on mw1171, mw1187
  • 00:25 logmsgbot: ori Synchronized wmf-config: Id607bf36d: Update remaining references to /u/l/a/common-local (duration: 00m 03s)
  • 00:25 logmsgbot: ori Synchronized multiversion: Id607bf36d: Update remaining references to /u/l/a/common-local (duration: 00m 04s)
  • 00:22 logmsgbot: ori Synchronized docroot and w: Id607bf36d: Update remaining references to /u/l/a/common-local (duration: 00m 04s)
  • 00:07 logmsgbot: ori updated /a/common to Id607bf36d: Update remaining references to /u/l/a/common-local

September 10

  • 23:44 mutante: graceful'ed mw1202 apache
  • 23:29 mutante: deleted labstore1003.eqiad.wmnet.org from puppet stored resource db, fixes puppet runs on hosts with ssh host key collection
  • 23:26 logmsgbot: oblivian gracefulled all apaches
  • 23:22 logmsgbot: maxsem Synchronized php-1.24wmf20/includes/specialpage/SpecialPageFactory.php: https://gerrit.wikimedia.org/r/#/c/159526/ (duration: 00m 03s)
  • 23:22 logmsgbot: maxsem Synchronized php-1.24wmf19/includes/specialpage/SpecialPageFactory.php: https://gerrit.wikimedia.org/r/#/c/159526/ (duration: 00m 03s)
  • 23:21 logmsgbot: maxsem Synchronized php-1.24wmf20/extensions/CentralAuth/: (no message) (duration: 00m 03s)
  • 23:21 logmsgbot: maxsem Synchronized php-1.24wmf19/extensions/CentralAuth/: (no message) (duration: 00m 04s)
  • 23:19 logmsgbot: maxsem Synchronized php-1.24wmf10/resources/: https://gerrit.wikimedia.org/r/#/c/159513/ (duration: 00m 05s)
  • 22:52 mutante: labstore1003 - (earlier) revoked salt and puppet key and signed new after hostname fix - same salt-minion puppet errors that happen after reinstalls
  • 19:52 Reedy: Created Echo tables on extension1 for cawikimedia
  • 19:51 RobH: puppet disabled on carbon (install server) for a livehack test of config setting
  • 18:51 yurikR: yurik CommonSettings.php - zerowiki perm changes
  • 18:51 logmsgbot: yurik Synchronized wmf-config/CommonSettings.php: (no message) (duration: 01m 05s)
  • 18:26 logmsgbot: yurik Synchronized php-1.24wmf20/extensions/ZeroBanner: (no message) (duration: 01m 09s)
  • 18:22 logmsgbot: yurik Synchronized php-1.24wmf19/extensions/ZeroBanner: (no message) (duration: 01m 11s)
  • 18:00 manybubbles: cirrus index rebuild for test2wiki went well - doing the rest of group0
  • 17:35 manybubbles: rebuilding cirrus index for test2wiki to test some performance enhancements don't break anything. test2wiki is too small to see any gain from the enhancements though.
  • 17:25 Reedy: mw1126, mw1116, mw1122, mw1146, mw1121, mw1136, mw1114, mw1068 have been gracefulled
  • 17:10 bd808: Restarted logstash on logstash1001
  • 16:03 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: nlwiki cirrus (duration: 00m 04s)
  • 15:44 logmsgbot: reedy Synchronized docroot and w: (no message) (duration: 00m 15s)
  • 15:02 logmsgbot: demon Synchronized wmf-config/wikitech.php: no-op (duration: 00m 06s)
  • 09:13 godog: rolling restart swift-proxy on ms-fe1*
  • 04:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 10 04:17:36 UTC 2014 (duration 17m 35s)
  • 03:08 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-10 03:07:59+00:00
  • 02:36 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-09-10 02:36:00+00:00
  • 02:28 ori: updated salt key for iridium and restarted salt-minion
  • 02:18 mutante: started salt-minion on iridium

September 9

  • 23:15 Krinkle: Reloading Zuul to deploy I26bc21ed2938e97e7ed6f6b
  • 23:15 logmsgbot: demon Synchronized php-1.24wmf20/extensions/CirrusSearch: Various fixes for things (duration: 00m 05s)
  • 23:00 mutante: added wikimedia.org to search in resolv.conf on terbium
  • 22:42 logmsgbot: ebernhardson Synchronized wmf-config/InitialiseSettings.php: Deploy config change I158e7c6852 (duration: 00m 04s)
  • 22:23 Krinkle: Reloading Zuul to deploy I27024680c74ca0130
  • 22:21 logmsgbot: ebernhardson Finished scap: Bump Echo and Flow versions in 1.24wmf19 (duration: 31m 25s)
  • 21:49 logmsgbot: ebernhardson Started scap: Bump Echo and Flow versions in 1.24wmf19
  • 20:42 akosiaris: service gmetad restart on nickel.wikimedia.org due to ganglia web not working
  • 20:15 cscott: updated OCG to version c9a2b4cf2502479eeabed07ab2de728695d96e46
  • 19:05 mutante: killed jgonera's screen session on stat1002 - puppet failed to deactivate otherwise
  • 18:46 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 18:32 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 15s)
  • 18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Add cawikimedia
  • 18:28 logmsgbot: reedy Synchronized multiversion/: (no message) (duration: 00m 14s)
  • 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf20
  • 16:03 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: eswiki getting cirrus (duration: 00m 04s)
  • 15:32 bblack: deploying large DNS change https://gerrit.wikimedia.org/r/#/c/158382/ - be on the lookout for any related fallout from here...
  • 15:27 marktraceur: [SCAP] Deployed fix for oojs class names at James_F's behest, sorry for lack of message.
  • 15:26 logmsgbot: marktraceur Synchronized php-1.24wmf20/extensions/MobileFrontend/less/modules/editor/VisualEditorOverlay.less: (no message) (duration: 00m 07s)
  • 15:08 logmsgbot: marktraceur Synchronized php-1.24wmf20/tests/phpunit/includes/changes/OldChangesListTest.php: [SWAT] Fix undefined argument (css classes) in OldChangesList. (duration: 00m 07s)
  • 15:06 logmsgbot: marktraceur Synchronized php-1.24wmf20/includes/changes/OldChangesList.php: [SWAT] Fix undefined argument (css classes) in OldChangesList. (duration: 00m 07s)
  • 11:29 _joe_: git.wikimedia.org works now, no action needed
  • 11:26 MatmaRex: git.wikimedia.org is down: Error: 503, Service Unavailable
  • 10:04 _joe_: also re-enabling puppet
  • 10:02 _joe_: restarting manually apache on mw1178,mw1192,mw1163,mw1130,mw1018 as they started with the wrong pidfile before my fix
  • 09:24 _joe_: disabling puppet on appservers
  • 08:55 godog: launched "iptables" on tin to check current rules and it loaded iptables modules, logging for future reference
  • 08:10 _joe_: re-enabling puppet on appservers and imagescalers, change is good
  • 08:08 _joe_: restarted apache2 on mw1018
  • 08:06 _joe_: stopping apache on mw1018 for inspection
  • 07:36 _joe_: that was on appservers
  • 07:36 _joe_: disabling puppet, releasing a potentially harmful apache change
  • 04:56 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 9 04:56:25 UTC 2014 (duration 56m 24s)
  • 03:44 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-09 03:44:07+00:00
  • 03:11 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-09-09 03:11:27+00:00
  • 02:38 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-09-09 02:38:38+00:00
  • 01:02 logmsgbot: ebernhardson Synchronized php-1.24wmf20/extensions/Flow/includes/Content/BoardContentHandler.php: Sync BoardContentHandler.php for Flow in 1.24wmf20 (duration: 00m 04s)
  • 00:22 mutante: re-enabled mw1070 in pybal
  • 00:19 logmsgbot: ebernhardson Finished scap: Repeat SWAT scap deployment due to possible sync-common failure (duration: 38m 50s)

September 8

  • 23:59 ori: restarted rsync on mw1070 to unblock scap
  • 23:40 logmsgbot: ebernhardson Started scap: Repeat SWAT scap deployment due to possible sync-common failure
  • 23:39 logmsgbot: ebernhardson Finished scap: SWAT deploy updates to Flow, Echo and Thanks (duration: 24m 00s)
  • 23:34 mutante: disabled mw1070 in pybal because it refused sync
  • 23:31 ebernhardson: scap failed to connect to mw1070. Repeated message: rsync: failed to connect to mw1070.eqiad.wmnet (10.64.16.50): Connection refused (111)
  • 23:15 logmsgbot: ebernhardson Started scap: SWAT deploy updates to Flow, Echo and Thanks
  • 23:02 logmsgbot: ebernhardson Synchronized wmf-config/InitialiseSettings.php: gerrit:159089 Enable $wgContentHandlerUseDB on mediawikiwiki, testwiki, & test2wiki (duration: 00m 05s)
  • 20:14 subbu: deployed Parsoid ce108cb5
  • 18:01 logmsgbot: demon Synchronized php-1.24wmf19/extensions/Wikidata: Updating Wikidata to f1d2110 (duration: 00m 09s)
  • 17:19 mutante: disabled notifications for puppet freshness on neon
  • 16:19 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: svwiki: Cirrus as primary (duration: 00m 04s)
  • 15:42 logmsgbot: manybubbles Synchronized php-1.24wmf20/extensions/Wikidata/: SWAT update wikidata to fix add links widget (duration: 00m 06s)
  • 15:32 logmsgbot: manybubbles Synchronized php-1.24wmf20/extensions/LiquidThreads/: SWAT update liquidthreads to fix some missing images (duration: 00m 04s)
  • 15:28 manybubbles: 15:13:53 Synchronized php-1.24wmf19/extensions/WikiLove/: SWAT fix for WikiLove (duration: 00m 04s)
  • 15:28 manybubbles: this is the missing log:
  • 15:27 manybubbles: sync logging was down so it missed some syncing I just did.
  • 15:25 logmsgbot: manybubbles Synchronized php-1.24wmf20/extensions/WikiLove/: (no message) (duration: 00m 05s)
  • 15:20 logmsgbot: manybubbles Synchronized wmf-config: SWAT another cirrus setting update (duration: 00m 04s)
  • 15:10 logmsgbot: manybubbles Synchronized wmf-config: SWAT finish updating Cirrus settings (duration: 00m 05s)
  • 15:10 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT update some cirrus settings (duration: 00m 04s)
  • 15:10 cmjohnson1: shutting down neon for memory upgrade
  • 14:41 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1072 (duration: 00m 09s)
  • 12:23 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1073, depool db1072 (duration: 00m 06s)
  • 11:07 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1073 (duration: 00m 09s)
  • 10:55 _joe_: re-enabled puppet, the change results in a no-op as expected
  • 10:42 _joe_: disabling puppet on all appservers while updating apache config.
  • 04:58 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: move enwiki api traffic to db1051/db1066 (duration: 00m 09s)
  • 03:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 8 03:36:13 UTC 2014 (duration 36m 12s)
  • 02:45 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-08 02:44:47+00:00
  • 02:33 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-09-08 02:32:37+00:00
  • 02:20 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-09-08 02:19:52+00:00

September 7

  • 23:35 Tim: upgrading liblua everywhere
  • 20:36 ori: mw1017: upgraded HHVM from 3.3-dev+20140728+wmf5 to 3.3-dev+20140728+wmf6
  • 15:12 apergos: manually changed /etc/hosts entry on analytics1004 from having "analyticas1004.eqiad.wmnet" to "analytics1004.eqiad.wmnet"
  • 06:15 godog: powercycle ms-be1005, not even responsive on console
  • 03:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 7 03:29:51 UTC 2014 (duration 29m 50s)
  • 02:43 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-07 02:42:12+00:00
  • 02:31 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-09-07 02:30:15+00:00
  • 02:18 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-09-07 02:17:44+00:00

September 6

  • 03:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Sep 6 03:42:22 UTC 2014 (duration 42m 21s)
  • 02:51 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-06 02:50:35+00:00
  • 02:38 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-09-06 02:37:41+00:00
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-09-06 02:24:35+00:00

September 5

  • 23:28 logmsgbot: kaldari Synchronized wmf-config/mobile-labs.php: enabling wikigrok on beta labs (en only) (duration: 00m 03s)
  • 23:28 logmsgbot: kaldari Synchronized wmf-config/InitialiseSettings-labs.php: enabling wikigrok on beta labs (en only) (duration: 00m 04s)
  • 23:27 logmsgbot: kaldari updated /a/common to Iec209bde0: Map config var for $wgMFEnableWikiGrok
  • 22:25 logmsgbot: kaldari Synchronized wmf-config/InitialiseSettings-labs.php: enabling wikigrok on beta labs (en only) (duration: 00m 05s)
  • 22:25 logmsgbot: kaldari updated /a/common to I6039956eb: Enable Wikigrok prototype for beta labs (enwiki only)
  • 22:24 awight: Deleted Light User and Merkle roles from the CRM
  • 20:20 RobH: coms folks still accessing blog data on holmium, powering back up
  • 20:18 bblack: restarted cp1056 bits cache and re-enabled in pybal
  • 18:34 mark: Depooled cp1056 for testing
  • 17:50 logmsgbot: ori Synchronized docroot and w: Iaa7518613: Fix spelling in symlink (duration: 00m 15s)
  • 17:45 logmsgbot: ori Synchronized docroot and w: I55a01a712: Fix relative symlinks for bits/static-master (duration: 00m 13s)
  • 13:00 Jeff_Green: lutetium dist-upgrade and reboot
  • 12:04 legoktm: running extensions/GlobalCssJs/removeOldManualUserPages.php for m:GlobalCssJs
  • 07:59 springle: dump es1007 to db1004, tokudb external storage page compression test. ok to kill in emergency
  • 06:40 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool es1007 (duration: 00m 07s)
  • 04:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 5 04:35:11 UTC 2014 (duration 35m 10s)
  • 04:27 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool es1002 (duration: 00m 06s)
  • 04:01 springle: reboot es1002, fs check
  • 03:47 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-05 03:46:28+00:00
  • 03:46 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool es1002 for upgrade (duration: 00m 07s)
  • 03:37 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1062 and db1068 (duration: 02m 06s)
  • 03:10 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-09-05 03:09:20+00:00
  • 02:56 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1062 and db1068 for upgrade (duration: 00m 56s)
  • 02:39 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-09-05 02:38:32+00:00
  • 01:01 manybubbles: applied same elasticsearch configuration to dewiki, eswiki, zhwiki, and frwiki
  • 00:18 manybubbles: configured elasticsearch to force enwiki's content shards to stay off of the same nodes. That ought to help performance.

September 4

  • 23:34 logmsgbot: catrope Synchronized php-1.24wmf20/extensions/VisualEditor/: (no message) (duration: 00m 05s)
  • 23:34 logmsgbot: catrope Synchronized php-1.24wmf20/extensions/ZeroPortal/: (no message) (duration: 00m 04s)
  • 23:34 logmsgbot: catrope Synchronized php-1.24wmf20/extensions/Flow/: (no message) (duration: 00m 05s)
  • 23:34 logmsgbot: catrope Synchronized php-1.24wmf20/includes/specials/: (no message) (duration: 00m 04s)
  • 23:31 logmsgbot: catrope Synchronized php-1.24wmf19/extensions/ZeroPortal: (no message) (duration: 00m 05s)
  • 23:30 logmsgbot: catrope Synchronized php-1.24wmf19/extensions/Flow: (no message) (duration: 00m 05s)
  • 22:52 logmsgbot: reedy Finished scap: consistency (duration: 20m 44s)
  • 22:31 logmsgbot: reedy Started scap: consistency
  • 22:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 4 22:27:45 UTC 2014 (duration 54m 38s)
  • 21:54 bd808: sync-dir failure was really on osmium, not mw1161; confusing error messages are confusing
  • 21:50 bd808: Running sync-common on mw1161 to try and reproduce error seen during sync-file
  • 21:43 logmsgbot: spage Synchronized wmf-config/InitialiseSettings.php: Enable Flow on pages, including frwiki and hewiki (duration: 00m 09s)
  • 21:40 logmsgbot: spage updated /a/common to Ib0aaa60f0: Enable Flow on several pages
  • 21:08 logmsgbot: LocalisationUpdate completed (1.24wmf20) at 2014-09-04 21:07:10+00:00
  • 20:56 MaxSem: Running cleanupPageProps.php from terbium, now for realz
  • 20:42 mutante: restarting icinga-wm, making it join #wikidata for custom output
  • 20:16 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-09-04 20:15:35+00:00
  • 19:56 Reedy: mw1088 and mw1100 rsync errors during the manual l10n update
  • 19:25 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-09-04 19:23:57+00:00
  • 18:32 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf20
  • 18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.24wmf19
  • 18:11 logmsgbot: reedy Synchronized php-1.24wmf19: (no message) (duration: 00m 55s)
  • 18:10 logmsgbot: reedy Synchronized php-1.24wmf20: (no message) (duration: 00m 35s)
  • 18:09 logmsgbot: reedy Finished scap: testwiki to 1.24wmf20 and build l10n cache (duration: 41m 33s)
  • 18:05 mutante: restarting service gitblit on antimony
  • 17:48 RobH: correction, simply surpressing alerts for the host in icinga is the better move, as the host isnt reclaimed yet, so not removing holmium from pupeptstoreddb
  • 17:46 RobH: stopping puppet on holmium and removing it from puppetstoreddb so it doesnt show in icinga once updated
  • 17:45 RobH: shutting down holmium, as blog has migrated for a month now. Not yet wiping system, please leave for me (robh)
  • 17:27 logmsgbot: reedy Started scap: testwiki to 1.24wmf20 and build l10n cache
  • 16:44 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 17s)
  • 16:36 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 15:53 logmsgbot: andrew Synchronized private/WikitechPrivateSettings.php: (no message) (duration: 00m 01s)
  • 15:40 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 15:18 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: plwiki gets Cirrus (duration: 00m 06s)
  • 14:56 bd808: ori updated scap to 773f95f (change deploy_dir to /srv/mediawiki) ~15 hours ago
  • 08:16 _joe_: running sync-common on mw1017, trying to debug the hhvm bad state
  • 06:37 godog: clear slowlog on elastic1004
  • 05:25 jeremyb: temp hack fix deployed for morebots (here and labs, not the other instances)
  • 04:47 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1035, warm up (duration: 00m 08s)
  • 04:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 4 04:31:28 UTC 2014 (duration 31m 27s)
  • 03:43 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-09-04 03:42:34+00:00
  • 03:13 logmsgbot: LocalisationUpdate completed (1.24wmf18) at 2014-09-04 03:11:58+00:00
  • 02:41 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-09-04 02:40:45+00:00
  • 01:08 mutante: production wants project name?
  • 01:02 andrewbogott: the SAL still works, but the bot fails to acknowledge. Something to do with a change on wikitech
  • 00:59 andrewbogott: testing the log
  • 00:43 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 15s)

September 3

  • 23:52 logmsgbot: reedy Synchronized php-1.24wmf15/includes/EditPage.php: (no message) (duration: 00m 14s)
  • 23:43 logmsgbot: ori Synchronized docroot and w: (no message) (duration: 00m 05s)
  • 23:28 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 23:04 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/157855/ https://gerrit.wikimedia.org/r/#/c/158265/ (duration: 00m 04s)
  • 21:09 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Disable GlobalUsage on labswiki (duration: 00m 15s)
  • 20:59 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 15s)
  • 20:47 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 15s)
  • 20:45 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 15s)
  • 20:38 logmsgbot: andrew Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s)
  • 20:38 logmsgbot: andrew Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 04s)
  • 20:37 logmsgbot: andrew Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 04s)
  • 20:16 subbu: deployed Parsoid version 78e55c6b (deploy repo sha c0761179)
  • 18:52 logmsgbot: yurik Synchronized wmf-config: enabling graph ext on zerowiki & collabwiki (duration: 01m 06s)
  • 18:51 MaxSem: Running sync-common on mw1163
  • 18:48 logmsgbot: yurik Synchronized php-1.24wmf18/extensions/Graph/: (no message) (duration: 01m 09s)
  • 18:47 logmsgbot: yurik Synchronized php-1.24wmf19/extensions/Graph/: (no message) (duration: 01m 05s)
  • 16:52 logmsgbot: andrew Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 03s)
  • 16:52 logmsgbot: andrew Synchronized private/WikitechPrivateLdapSettings.php: (no message) (duration: 00m 03s)
  • 16:51 logmsgbot: andrew Synchronized private/WikitechPrivateSettings.php: (no message) (duration: 00m 05s)
  • 16:51 logmsgbot: andrew Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 03s)
  • 16:19 logmsgbot: andrew Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 03s)
  • 16:18 logmsgbot: andrew Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 04s)
  • 16:16 logmsgbot: andrew Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 05s)
  • 15:41 _joe_: mw1020 correctly reimaged, putting it in the hhvm pool
  • 15:27 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - Update another cirrus config - this time maybe it will work (duration: 00m 05s)
  • 15:12 manybubbles: deployed throttling for Cirrus job named cirrusSearchLinksUpdate - it handles updating the index when a transcluded page changes - we'll have to check on the backlog over the next few hours/days to see if it stabilizes
  • 15:11 logmsgbot: manybubbles Synchronized php-1.24wmf19/extensions/Wikidata/: (no message) (duration: 00m 07s)
  • 15:07 manybubbles: mw1020 gets WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! during sync-dir call
  • 15:07 logmsgbot: manybubbles Synchronized wmf-config/: SWAT deploy cirrus config changes - make sure to get mw1020 (duration: 00m 04s)
  • 15:05 manybubbles: https://gerrit.wikimedia.org/r/#/c/157861/ didn't work as expected - dropped everything out of using the all field......
  • 15:03 logmsgbot: manybubbles Synchronized wmf-config/: SWAT deploy cirrus config changes (duration: 00m 06s)
  • 14:53 cmjohnson1: running sync-common on mw1178
  • 14:52 cmjohnson1: adding mw1178 back to pybal
  • 12:42 _joe_: typo: mw1020, not mw1120
  • 12:41 _joe_: mw1120: remove from pybal, schedule downtime, reimage to HAT
  • 11:23 godog: run gmond on elastic1002 manually to debug ES collector issues
  • 11:17 godog: run gmond on elastic1001 manually to debug ES collector issues
  • 07:55 _joe_: re-enabling mw1192, what we were seeing was probably load and not anything else
  • 06:56 ori: restarted memcached on virt1000 due to cache pollution from migration (different memc drivers w/different encoding)
  • 04:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 3 04:53:50 UTC 2014 (duration 53m 49s)
  • 03:51 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-09-03 03:50:17+00:00
  • 03:17 logmsgbot: LocalisationUpdate completed (1.24wmf18) at 2014-09-03 03:16:37+00:00
  • 02:43 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-09-03 02:42:24+00:00
  • 00:18 mutante: deleted PDF files older than 3d and a huge 1G one on ocg1001 in reaction to monitoring complaints
  • 00:00 logmsgbot: ori Synchronized php-1.24wmf18/extensions/WikimediaEvents: Update WikimediaEvents for cherry-picks (duration: 00m 03s)

September 2

  • 23:49 logmsgbot: ori Synchronized php-1.24wmf19/extensions/WikimediaEvents: Update WikimediaEvents for cherry-picks (duration: 00m 03s)
  • 23:32 logmsgbot: reedy Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 13s)
  • 23:22 logmsgbot: catrope Synchronized php-1.24wmf19/includes/OutputPage.php: 5094c0d9c (duration: 00m 05s)
  • 23:14 Krinkle: Running extensions/GlobalCssJs/removeOldManualUserPages.php per m:GlobalCssJs
  • 22:49 logmsgbot: reedy Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 14s)
  • 22:37 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
  • 22:26 logmsgbot: reedy Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 13s)
  • 22:10 logmsgbot: reedy Synchronized docroot and w: (no message) (duration: 00m 14s)
  • 21:57 logmsgbot: reedy Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 21s)
  • 21:55 logmsgbot: reedy Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 24s)
  • 21:51 logmsgbot: reedy Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 25s)
  • 21:45 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 23s)
  • 21:37 logmsgbot: reedy Synchronized wmf-config/db-eqiad.php: Wikitech db (duration: 00m 22s)
  • 21:34 logmsgbot: bd808 Finished scap: no-op scap to build l10n for wikitech (duration: 55m 48s)
  • 20:39 logmsgbot: bd808 Started scap: no-op scap to build l10n for wikitech
  • 20:35 logmsgbot: bd808 Synchronized wmf-config/wikitech.php: eebc99a Require before instatiate (duration: 00m 04s)
  • 20:31 logmsgbot: bd808 Synchronized private/PrivateSettings.php: Absolute path for WikitechPrivateSettings.php (duration: 00m 05s)
  • 20:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
  • 20:05 MaxSem: Running cleanupPageProps.php everywhere
  • 19:51 MaxSem: Running cleanupPageProps.php on mw.org and meta
  • 19:14 logmsgbot: reedy Synchronized wmf-config/Wikibase.php: Bump epoch (duration: 00m 14s)
  • 19:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf19, added labswiki too
  • 18:56 logmsgbot: bd808 Synchronized fishbowl.dblist: Add labswiki (wikitech) (duration: 00m 05s)
  • 18:25 logmsgbot: andrew Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 08s)
  • 17:49 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 14s)
  • 17:39 logmsgbot: andrew Synchronized multiversion/MWMultiVersion.php: (no message) (duration: 00m 04s)
  • 17:14 logmsgbot: andrew Finished scap: Deploying wikitech config (duration: 33m 03s)
  • 17:01 bd808: Fetched f711ea7 to /a/common on tin; not syncing because of in-process scap.
  • 16:41 logmsgbot: andrew Started scap: Deploying wikitech config
  • 16:21 logmsgbot: andrew scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/a/common/wmf-config/extension-list" --output="/tmp/tmp.SCRILhxGxO" ' returned non-zero exit status 1 (duration: 01m 17s)
  • 16:20 logmsgbot: andrew Started scap: Deploying wikitech config
  • 16:17 ottomata: installing newer version of webstatscollector on oxygen and gadolinium, restarting filter process on oxygen
  • 16:08 logmsgbot: andrew Synchronized /a/common/private/WikitechPrivateSettings.php: (no message) (duration: 00m 04s)
  • 16:07 logmsgbot: andrew Synchronized /a/common/private/PrivateSettings.php: (no message) (duration: 00m 03s)
  • 16:03 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: Commons gets Cirrus as primary (duration: 00m 04s)
  • 15:44 godog: bring mw1114 -> mw1131 to weight 15
  • 15:21 logmsgbot: marktraceur Synchronized wmf-config/: [SCAP] SpecialCite is now CiteThisPage (duration: 00m 07s)
  • 15:17 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SCAP] Enable the TemplateData GUI editor on Norwegian Wikipedia (duration: 00m 07s)
  • 15:14 logmsgbot: marktraceur updated /a/common to Ia1758b21e: depool db1035 for upgrade, move s3 vslow/dump to db1019
  • 15:06 logmsgbot: marktraceur Synchronized php-1.24wmf19/includes/EditPage.php: [SCAP] Revert "Toolbar: Only show on WikiText pages" (duration: 00m 07s)
  • 15:05 logmsgbot: marktraceur Synchronized php-1.24wmf18/includes/EditPage.php: [SCAP] Revert "Toolbar: Only show on WikiText pages" (duration: 00m 08s)
  • 12:36 godog: increase weight to 15 for mw1132 -> mw1148
  • 10:00 _joe_: depooling mw1192, high CPU temperatures; we may need to check fan status
  • 07:20 _joe_: powercycling mw1192, blank console, unresponsive
  • 07:02 springle: removed all-but-latest large slow logs on elastic1004 and elastic1014
  • 06:22 springle: removed txt files filling up db1047 /tmp, looked like analytics SELECT INTO OUTFILE, dated mid-August
  • 05:58 springle: dump s3 db1035 to db1069:3313
  • 05:37 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1067, warm up (duration: 00m 08s)
  • 04:59 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1067 (duration: 00m 07s)
  • 03:35 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1035 (duration: 00m 07s)
  • 03:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 2 03:15:22 UTC 2014 (duration 15m 21s)
  • 02:53 springle: restarted dbstore1002 mysqld for upgrade
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-09-02 02:25:36+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf18) at 2014-09-02 02:14:30+00:00

September 1

  • 23:00 Krinkle: Running extensions/GlobalCssJs/removeOldManualUserPages.php per m:GlobalCssJs
  • 21:50 ori: disabled gerrit account Caothu9669; spam
  • 19:12 Reedy: Deleted php-1.24wmf[6-8] from apaches via dsh
  • 19:01 logmsgbot: reedy Purged l10n cache for 1.24wmf13
  • 19:01 logmsgbot: reedy Purged l10n cache for 1.24wmf14
  • 19:00 logmsgbot: reedy Purged l10n cache for 1.24wmf15
  • 18:59 logmsgbot: reedy Purged l10n cache for 1.24wmf16
  • 18:58 logmsgbot: reedy Purged l10n cache for 1.24wmf17
  • 16:44 ottomata: removed some large slow query logs from elastic* nodes, need to look into this...
  • 12:10 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1044, take 2 (duration: 00m 06s)
  • 12:04 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1044 (duration: 00m 06s)
  • 11:31 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: pool db1044, warm up (duration: 00m 06s)
  • 09:31 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1027 (duration: 00m 07s)
  • 07:27 godog: deploy latest ring to swift eqiad-prod
  • 07:11 godog: powercycle ms-be1010 "cpu soft lockup" on console
  • 05:28 springle: xtrabackup clone db1027 to db1044
  • 05:26 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1027 while cloning (duration: 00m 07s)
  • 03:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 1 03:14:22 UTC 2014 (duration 14m 21s)
  • 02:28 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-09-01 02:27:39+00:00
  • 02:17 logmsgbot: LocalisationUpdate completed (1.24wmf18) at 2014-09-01 02:16:18+00:00

August 31

  • 19:56 hashar: Jenkins updated HHVM to (3.3-dev+20140728+wmf5) over (3.3-dev+20140728+wmf4)
  • 14:38 bblack: restarted apache on strontium
  • 14:34 bblack: restarted apache on tungsten, machine is overloaded
  • 03:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 31 03:14:41 UTC 2014 (duration 14m 40s)
  • 02:29 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-08-31 02:28:42+00:00
  • 02:18 logmsgbot: LocalisationUpdate completed (1.24wmf18) at 2014-08-31 02:17:24+00:00
  • 02:00 ori: Stopped HHVM jobrunner and disabled Puppet on mw1053 due to bug 70177.

August 30

  • 08:18 godog: restart mailman on sodium, pending https://gerrit.wikimedia.org/r/#/c/156766/
  • 08:01 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1071, warm up (duration: 00m 07s)
  • 06:33 jgage: analytics1021 back in service after election
  • 06:14 jgage: upgraded & rebooted analytics1021
  • 06:01 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: return db1037 to normal load (duration: 00m 06s)
  • 05:03 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool es1001 (duration: 00m 06s)
  • 04:06 springle: upgrade es1001 to mariadb 10
  • 03:56 springle: xtrabackup clone db1037 to db1071
  • 03:55 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: reduce db1037 load while cloning (duration: 00m 06s)
  • 03:31 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: pool db1073, depool db1071 (duration: 00m 07s)
  • 03:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 30 03:17:57 UTC 2014 (duration 17m 56s)
  • 02:33 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-08-30 02:32:06+00:00
  • 02:21 logmsgbot: LocalisationUpdate completed (1.24wmf18) at 2014-08-30 02:20:39+00:00

August 29

  • 21:15 logmsgbot: ori Synchronized wmf-config: I812c0bb6c: Scrap unused Twemproxy config files (duration: 00m 04s)
  • 21:12 logmsgbot: ori updated /a/common to I812c0bb6c: Scrap unused Twemproxy config files
  • 21:03 mutante: restarted uwsgi on tungsten
  • 20:49 mutante: tungsten extremely busy, graphite down, logging in since 5 minutes :p
  • 20:48 mutante: powercycling ms-be1006 - BUG: soft lockup - CPU#0 stuck ...
  • 19:19 mutante: installing package upgrades on iron, bast1001
  • 18:41 logmsgbot: aaron Synchronized php-1.24wmf18/maintenance/findMissingFiles.php: 994d4a556a070156fd04fb4951492f10696cc63c (duration: 00m 03s)
  • 18:30 logmsgbot: ori Synchronized php-1.24wmf19/resources/src/mediawiki.action/mediawiki.action.view.redirect.js: I19221a25a: mediawiki.action.view.redirect: Work around a IE 10+ HTML5 history API bug (duration: 00m 06s)
  • 18:30 logmsgbot: ori Synchronized php-1.24wmf18/resources/src/mediawiki.action/mediawiki.action.view.redirect.js: I19221a25a: mediawiki.action.view.redirect: Work around a IE 10+ HTML5 history API bug (duration: 00m 07s)
  • 15:36 hashar_: Jenkins: pooled a new slave 10.68.16.162 as wikidata-jenkins3 on behalf of addshore / wmde
  • 15:04 _joe_: shutting down mw1163, filled RT 8243 for repair.
  • 14:54 _joe_: re-enabled mw1130
  • 14:41 logmsgbot: aude Synchronized wmf-config/Wikibase.php: enable Wikibase badges css, follow up from last night deploy (duration: 00m 06s)
  • 14:22 _joe_: syncing mw1130
  • 14:06 _joe_: disable mw1130 from the api pool whil it gets resynced
  • 12:30 Krinkle: Running extensions/GlobalCssJs/removeOldManualUserPages.php per m:GlobalCssJs
  • 11:06 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1070 (duration: 00m 09s)
  • 08:04 hashar: Jenkins: in the jenkins-job-builder-config branch 'cloudbees' has been merged in 'master'. Unifying CI and browser tests jobs! \O/
  • 07:05 _joe_: re-enabling puppet on the jobrunner, to check if the luasandbox fix works
  • 06:33 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: return db1056 to normal load (duration: 00m 06s)
  • 04:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 29 04:13:03 UTC 2014 (duration 13m 2s)
  • 03:31 springle: xtrabackup clone db1056 to db1070
  • 03:31 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: reduce db1056 load while cloning (duration: 00m 06s)
  • 03:15 logmsgbot: LocalisationUpdate completed (1.24wmf19) at 2014-08-29 03:10:26+00:00
  • 02:48 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1070. pool db1072. (duration: 00m 07s)
  • 02:38 logmsgbot: LocalisationUpdate completed (1.24wmf18) at 2014-08-29 02:37:20+00:00
  • 01:49 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1070. pool db1072. (duration: 00m 06s)
  • 01:27 godog: repool ms-fe1002
  • 01:06 cmjohnson1: shutting down ms-fe1002 to relocate racks
  • 01:04 godog: depool ms-fe1002
  • 01:02 godog: repool ms-fe1001
  • 00:57 logmsgbot: ori Synchronized php-1.24wmf18/extensions/WikimediaEvents: Ib44fe0898: Inject 'wgPoweredByHHVM' JS config var if powered by HHVM (duration: 00m 03s)
  • 00:56 logmsgbot: ori Synchronized php-1.24wmf19/extensions/WikimediaEvents: Ib44fe0898: Inject 'wgPoweredByHHVM' JS config var if powered by HHVM (duration: 00m 04s)
  • 00:38 cmjohnson1: shutting down ms-fe1001 for rack relocation
  • 00:34 godog: depool ms-fe1001
  • 00:32 godog: repool ms-fe1004
  • 00:27 mutante: restarting gmetad on nickel
  • 00:04 cmjohnson1: shutting down ms-fe1004 to relocate racks

August 28

  • 23:58 godog: depool ms-fe1004
  • 23:51 godog: repooling ms-fe1003
  • 23:40 logmsgbot: maxsem Synchronized php-1.24wmf19/maintenance/: https://gerrit.wikimedia.org/r/#/c/156979/ (duration: 00m 04s)
  • 23:39 logmsgbot: maxsem Synchronized php-1.24wmf19/includes/: https://gerrit.wikimedia.org/r/#/c/156979/ (duration: 00m 06s)
  • 23:38 logmsgbot: maxsem Synchronized php-1.24wmf19/extensions/Flow/: https://gerrit.wikimedia.org/r/#/c/156994/ (duration: 00m 05s)
  • 23:36 logmsgbot: maxsem Synchronized php-1.24wmf19/extensions/Echo: https://gerrit.wikimedia.org/r/#/c/157008/ (duration: 00m 04s)
  • 23:35 logmsgbot: maxsem Synchronized php-1.24wmf19/extensions/Thanks/: https://gerrit.wikimedia.org/r/#/c/156898/ (duration: 00m 04s)
  • 23:34 logmsgbot: maxsem Synchronized php-1.24wmf19/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/156968/ (duration: 00m 05s)
  • 23:30 logmsgbot: maxsem Synchronized php-1.24wmf18/extensions/GlobalCssJs/: https://gerrit.wikimedia.org/r/#/c/157009/ (duration: 00m 04s)
  • 23:27 K4-713: Updated fraud filters on payments
  • 22:52 logmsgbot: aaron Synchronized php-1.24wmf18/maintenance/findMissingFiles.php: (no message) (duration: 00m 07s)
  • 22:15 mutante: restarted tools.morebots production instance - can i log now?
  • 22:13 cmjohnson1: ms-fe1003 down for relocation
  • 22:13 mutante: test
  • 21:15 robh: bast2001.wikimedia.org now online in codfw.
  • 21:15 robh: i never admin logged when install2001.wikimedia.org went online the other day, opps.
  • 21:15 ori: last sync was of Iac37a2369: resourceloader: Don't register raw modules client-side
  • 21:14 logmsgbot: ori Synchronized php-1.24wmf18/includes/resourceloader/ResourceLoaderStartUpModule.php: (no message) (duration: 00m 03s)
  • 20:57 logmsgbot: krinkle Synchronized php-1.24wmf19/includes/resourceloader/ResourceLoaderStartUpModule.php: fd5b963458c19 (duration: 00m 06s)
  • 20:33 ottomata: shutting down elastic1016
  • 20:16 ottomata: temporarily disable puppet on gadolinium
  • 19:36 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 19:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf19
  • 19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf18
  • 19:07 logmsgbot: reedy Finished scap: testwiki to 1.24wmf19 (duration: 43m 00s)
  • 18:25 godog: install build-essential and fakeroot on tin
  • 18:24 logmsgbot: reedy Started scap: testwiki to 1.24wmf19
  • 17:26 logmsgbot: aaron Synchronized rpc: 9564e93ecd4953126d91b99d7728f63401a4dc86 (duration: 00m 07s)
  • 17:13 ^d: elastic: excluded the elastic1016 node from shard allocation, shards draining so we can take it down for disk testing
  • 16:01 ottomata: restarted webstats-collector on gadolinium
  • 13:18 mark: Reactivated cr2-eqiad AS3257 transit link
  • 10:44 springle: xtrabackup clone db1051 to db1073
  • 10:18 godog: restarting mailman on sodium
  • 08:52 godog: restarted apache on mw1134
  • 08:03 godog: killed stray mailman processes on sodium (no pid file) and restarted mailman
  • 06:11 springle: xtrabackup clone db1051 to db1072
  • 06:09 springle: restarted morebots

August 26

  • 21:04 hashar: Updating our Jenkins Job Builder fork 0268581..e5c0c61 . Will let us define variables in 'default' section and override them when invoking a job template ( https://review.openstack.org/#/c/100020/ )
  • 19:58 bd808: Ran sync-common on mw1053.eqiad.wmnet to recover from failure during last scap
  • 19:48 logmsgbot: aude Finished scap: Update new messages for Wikibase (duration: 07m 16s)
  • 19:41 logmsgbot: aude Started scap: Update new messages for Wikibase
  • 19:39 logmsgbot: aude Synchronized wmf-config/Wikibase.php: add Wikibase badges css setting (duration: 00m 10s)
  • 19:25 logmsgbot: aude Synchronized wmf-config/Wikibase.php: enable new serialization format for wikidata (duration: 00m 08s)
  • 19:10 logmsgbot: reedy Synchronized php-1.24wmf18/extensions/Echo/: (no message) (duration: 00m 14s)
  • 19:05 logmsgbot: aude Synchronized wmf-config/Wikibase.php: enable otherprojects sidebar beta feature (duration: 00m 15s)
  • 18:54 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf18
  • 18:53 logmsgbot: reedy Synchronized php-1.24wmf18/extensions/MassMessage: (no message) (duration: 00m 14s)
  • 18:52 logmsgbot: reedy Synchronized php-1.24wmf17/extensions/MassMessage: (no message) (duration: 00m 16s)
  • 18:19 jgage: Failover from analytics1010-eqiad-wmnet to analytics1004-eqiad-wmnet successful
  • 17:47 logmsgbot: bd808 Synchronized private/PrivateSettings.php: Syncing file rather than symlink (duration: 00m 04s)
  • 17:36 bd808: mw1010.eqiad.wmnet was out of sync too. I suspect there is something wrong with the fanout update step in scap
  • 17:26 bd808: /usr/local/apache/common-local out of date on mw1161.eqiad.wmnet; updated via sync-common
  • 17:25 bd808: sync-* not updating terbium properly; sync-common from terbium manually got several config changes; maybe a problem with mw1161.eqiad.wmnet rsync mirror
  • 17:14 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 04s)
  • 17:11 logmsgbot: demon Synchronized wmf-config/PrivateSettings.php: adjust swift auth url for cirrus (duration: 00m 04s)
  • 17:05 cmjohnson: swapping failed disk labsdb1003 slot 1
  • 16:42 bd808: Ran sync-common on osmium to verify that it now rebuilds l10n cache by default (and it does!)
  • 16:36 legoktm: running removeOldManualUserPages.php (GlobalCssJs) for users who requested it
  • 16:29 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: Again, with feeling (duration: 00m 04s)
  • 16:26 logmsgbot: bd808 Finished scap: no-op scap to test scap code update (duration: 13m 31s)
  • 16:20 bd808|DEPLOY: Rsync sloooow to fenari "16:18:52 fenari INFO - Finished rsync common (duration: 04m 38s)"
  • 16:12 logmsgbot: bd808 Started scap: no-op scap to test scap code update
  • 16:07 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 04s)
  • 16:07 bd808|DEPLOY: Updated scap to 116027f (Make sync-common update l10n cdb files by default)
  • 15:05 logmsgbot: anomie Synchronized wmf-config: SWAT: Enable GlobalCssJs on all CentralAuth wikis minus loginwiki gerrit:154432 (duration: 00m 09s)
  • 13:32 hashar: Jenkins mediawiki-core-qunit job has been switched to Zuul cloner and pass! :-D
  • 13:29 _joe_: re-enabling puppet, change aborted as not all sites are served via hhvm on the hhvm appservers (true story). Will re-do once all configs are in their place
  • 13:12 _joe_: disabling puppet on all appservers while deploying an apache change
  • 12:48 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: db1054 to normal load (duration: 00m 06s)
  • 12:33 hashar: Jenkins reverted mediawiki-core-qunit to use Zuul cloner 156268. Gotta play with it on a new job name since it does not work out of the box as expected.
  • 12:12 hashar: Jenkins migrating mediawiki-core-qunit to use Zuul cloner 156268
  • 12:03 akosiaris: disable puppet on labsdb1006 for planet osm import
  • 11:53 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: pool db1054, warm up (duration: 00m 08s)
  • 09:04 godog: reboot ms-be1011, unresponse on network and console
  • 08:28 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1036 (duration: 00m 06s)
  • 05:41 springle: xtrabackup clone db1036 to db1054
  • 05:39 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1036 while cloning (duration: 00m 06s)
  • 05:28 springle: upgrade & restart db1054, fs check
  • 04:48 logmsgbot: demon Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 06s)
  • 04:27 springle: labsdb1002 back up
  • 04:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Aug 26 04:06:34 UTC 2014 (duration 6m 33s)
  • 03:23 ^d: restarting elasticsearch on elastic1001, elastic1003 and elastic1008. icinga may complain briefly.
  • 03:11 springle: filesystem issues on labsdb1002. stopped mysqld
  • 03:05 logmsgbot: LocalisationUpdate completed (1.24wmf18) at 2014-08-26 03:04:18+00:00
  • 02:34 logmsgbot: LocalisationUpdate completed (1.24wmf17) at 2014-08-26 02:33:00+00:00

August 25

  • 23:58 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: Add 'movefile' to 'eliminator' user group on jawiki (duration: 00m 03s)
  • 23:53 logmsgbot: maxsem Finished scap: SWAT: CentralNotice update (duration: 29m 58s)
  • 23:23 logmsgbot: maxsem Started scap: SWAT: CentralNotice update
  • 23:19 logmsgbot: maxsem Synchronized php-1.24wmf17/extensions/CentralNotice/: https://gerrit.wikimedia.org/r/#/c/156188/ (duration: 00m 05s)
  • 23:17 logmsgbot: maxsem Synchronized php-1.24wmf18/extensions/CentralNotice/: https://gerrit.wikimedia.org/r/#/c/156188/ (duration: 00m 04s)
  • 23:15 logmsgbot: maxsem Synchronized php-1.24wmf18/includes/htmlform/HTMLCheckField.php: https://gerrit.wikimedia.org/r/#/c/156015/ (duration: 00m 05s)
  • 20:06 subbu: deployed parsoid version 5b5a5ed5
  • 17:24 godog: reboot ms-be1004 to pick up kernel upgrade
  • 17:13 godog: rebooting ms-be1002 to pick up updated kernel
  • 16:54 ottomata: stopping puppet on cp3021. Testing an increase of http://kafka.queue.buffering.max.ms/ in order to avoid dropping messages during broker metadata change (e.g. leader elections)
  • 16:48 hashar: Jenkins pooled in a new slave wdjenkins-node1 that will be used to run Wikidata jenkins jobs. Work in progress with addshore. It is not running jobs yet.
  • 16:47 godog: reboot ms-be1011, xfsaild errors in dmesg
  • 16:25 hashar: Jenkins: disconnecting and reconnecting Gearman plugin from https://integration.wikimedia.org/ci/configure
  • 16:06 andrewbogott: wikitech deployment finished. Note that the OpenStackManager submodule is off of the MediaWiki branch because… the whole submodule setup there is a bit broken on account of a git bug that uses absolute paths to manage submodules.
  • 16:01 andrewbogott: deploying tiny OpenStackManager upgrade on wikitech
  • 15:58 ottomata: enabled elasticsearch shard allocation row awareness (via rest api)
  • 12:45 hashar: hard stopped/restarted Zuul (workflow config error)
  • 12:27 hashar: restarting zuul
  • 10:15 mark: setup cross-confederation BGP sessions from AS65001 (eqiad) to AS65002 (codfw)
  • 05:35 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: 156076 - Remove centralnotice-admin right assignments on 3 wikis - Basically a noop (duration: 00m 06s)
  • 03:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Aug 25 03:14:13 UTC 2014 (duration 14m 12s)
  • 02:27 logmsgbot: LocalisationUpdate completed (1.24wmf18) at 2014-08-25 02:25:58+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf17) at 2014-08-25 02:14:14+00:00

August 24

  • 23:17 ^d: slow indexing log going pretty bonanzas on elastic101[35]. Probably others too? Filling /var/log.
  • 12:02 mark: Removed IPv6 subnet 2620:0:860:2::/64 from cr2-pmtpa:irb.101
  • 03:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 24 03:17:46 UTC 2014 (duration 17m 45s)
  • 02:32 logmsgbot: LocalisationUpdate completed (1.24wmf18) at 2014-08-24 02:31:13+00:00
  • 02:19 logmsgbot: LocalisationUpdate completed (1.24wmf17) at 2014-08-24 02:18:52+00:00

August 23

  • 11:33 mark: Manually removed IPv6 addresses from fenari
  • 11:23 mark: Deactivated IPv6 router-advertisement on cr2-pmtpa
  • 11:21 mark: Manually removed IPv6 address from mchenry
  • 10:36 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1004. pool db1053. (duration: 00m 07s)
  • 03:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 23 03:06:13 UTC 2014 (duration 6m 12s)
  • 02:23 logmsgbot: LocalisationUpdate completed (1.24wmf18) at 2014-08-23 02:21:59+00:00
  • 02:18 logmsgbot: LocalisationUpdate completed (1.24wmf17) at 2014-08-23 02:17:28+00:00
  • 01:31 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1056 (duration: 00m 06s)
  • 00:29 ori: disabled puppet on osmium again to debug a leak; please don't re-enable

August 22

  • 18:10 logmsgbot: ori updated /a/common to I338d72a47: Do not define MEDIAWIKI before loading WebStart.php
  • 17:43 ottomata: moving sqstat udp2log filter from analytics1003 to analytics1026, reqstats might blip for a sec...
  • 17:41 ori: nuking /srv/deployment/rcstream on rcs1002 to verify trebuchet package provider reprovisions it
  • 15:54 springle: xtrabackup db1056 to db1053
  • 15:53 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1056 while cloning (duration: 00m 07s)
  • 15:33 ^d: elastic1008: fixed /etc/hosts to point to actual IP instead of loopback
  • 15:18 springle: upgrade & restart db1053, fs check
  • 15:08 bd808: Still no apache2.log on fluorine or in logstash. Log seems to be available on fenari.
  • 14:51 springle: switched s1 sanitarium and labsdb replication to db1069:3311 mariadb 10
  • 14:39 mark: Removed IPv6 subnets 2620:0:860:1::/64 (squid subnet) and 2620:0:860:3::/64 (sandbox subnet) from cr2-pmtpa configuration
  • 04:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 22 04:10:47 UTC 2014 (duration 10m 46s)
  • 03:19 logmsgbot: LocalisationUpdate completed (1.24wmf18) at 2014-08-22 03:18:44+00:00
  • 02:33 logmsgbot: LocalisationUpdate completed (1.24wmf17) at 2014-08-22 02:32:11+00:00
  • 00:03 logmsgbot: ori Finished scap: SWAT: d3de89777, 7abfe0d5e7, 8ec9853c32b, 476e9e90bd01 (duration: 06m 29s)

August 21

  • 23:57 logmsgbot: ori Started scap: SWAT: d3de89777, 7abfe0d5e7, 8ec9853c32b, 476e9e90bd01
  • 21:58 logmsgbot: ori Synchronized php-1.24wmf17/resources/src/mediawiki/mediawiki.js: I8d27442d1: Workaround for bug introduced by Icf6ede09b (duration: 00m 03s)
  • 21:57 manybubbles: performing elasticsearch upgrade on elastic1015
  • 21:02 logmsgbot: ori Synchronized php-1.24wmf17/resources/src/mediawiki/mediawiki.util.js: Touch resources/src/mediawiki/mediawiki.util.js (duration: 00m 06s)
  • 20:44 godog: rolling restart of swift-proxy on ms-fe1*
  • 20:11 godog: restarted swift-proxy on ms-fe1001
  • 19:55 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 13s)
  • 19:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf18
  • 19:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf17
  • 19:44 logmsgbot: reedy Finished scap: testwiki to 1.24wmf18 (duration: 34m 01s)
  • 19:31 mutante: disabled mw1178 in pybal
  • 19:27 godog: restarted memcached on ms-fe1004
  • 19:23 reedy|webirc: mw1178 returned [255]: ssh: connect to host mw1178 port 22: Connection timed out
  • 19:23 reedy|webirc: mw1019 returned [127]: bash: sync-common: command not found
  • 19:09 logmsgbot: reedy Started scap: testwiki to 1.24wmf18
  • 18:28 manybubbles: *victim*
  • 18:27 manybubbles: trying to recover from weird Elasticsearch upgrade failure by redoing the upgrade on one node while also blowing away the data directory during the upgrade. elastic1005, you are my first victem.
  • 17:28 cmjohnson1: removing mw1130 from pybal
  • 14:53 hashar: Jenkins: updated PHP CodeSniffer MediaWiki standard on all slaves.
  • 14:36 hashar_: Jenkins: updating mediawiki code sniffer repo bf82117..bc4e590
  • 10:02 hashar: Jenkins installed plugin Throttle Concurrent Builds.
  • 03:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 21 03:20:47 UTC 2014 (duration 20m 46s)
  • 02:36 logmsgbot: LocalisationUpdate completed (1.24wmf17) at 2014-08-21 02:34:56+00:00
  • 02:20 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-21 02:19:26+00:00
  • 00:08 MatmaRex: (manybubbles contd.) …a single node going down but I expect the cluster to stay "yellow" during the process- no alerts.
  • 00:07 manybubbles: bd808 needs to plan a logstash upgrade soon - let it be logged
  • 00:05 manybubbles: if anyone is reading the SAL for fun or sees an error in Elasticsearch cluster in the next 24 hours - we're performing an elasticsearch upgrade. We've set it up this time so its super slow and boring. So boring I'm going to sleep through it. If you see more then transient complaining from icinga about elasticsearch you can call me/have someone with access to the contact list call me. I expect icinga to complain about a
  • 00:00 manybubbles: unattended rolling restart of Elasticsearch cluster is going just fine - adding the 30 minute sleep between servers and turning down the replication rate makes it pretty boring.

August 20

  • 23:07 awight: stopping the Thank You job
  • 22:50 ori: disabled puppet on osmium to debug memory leak
  • 21:46 logmsgbot: marktraceur Synchronized php-1.24wmf17/extensions/MultimediaViewer/: Add disable-by-default option to MultimediaViewer (duration: 00m 07s)
  • 21:09 logmsgbot: marktraceur Synchronized wmf-config: Turn off Media Viewer for logged-in users at Commons. (duration: 00m 07s)
  • 21:06 logmsgbot: marktraceur updated /a/common to I226bd1468: Add item-redirect to OAuth permissions
  • 19:50 hashar: Restarting Zuul to prettify build results bug 66095
  • 19:48 logmsgbot: awight Synchronized php-1.24wmf17/extensions/CentralNotice: push CentralNotice updates, including new hide cookie format (duration: 00m 05s)
  • 19:47 logmsgbot: awight Synchronized php-1.24wmf16/extensions/CentralNotice: push CentralNotice updates, including new hide cookie format (duration: 00m 04s)
  • 19:46 logmsgbot: awight Synchronized php-1.24wmf16/extensions/CentralNotice: push CentralNotice updates, including new hide cookie format (duration: 00m 07s)
  • 16:11 manybubbles: elastic1001 upgrade went well - upgrading elastic1002 now
  • 15:48 hashar: dns: Jenkins will now complain whenever you attempt to send tabs in any file of operations/dns.git bug 69478
  • 15:17 manybubbles: manually lowered elasticsearch recovery speeds to stem off high load caused by healing the restart of elastic1001 - we were slowing down enough that we were filling the pool counter
  • 15:05 logmsgbot: anomie Synchronized wmf-config/CommonSettings.php: SWAT: Add item-redirect to OAuth permissions gerrit:155257 (duration: 00m 09s)
  • 15:01 logmsgbot: anomie Synchronized php-1.24wmf17/extensions/Wikidata/extensions/Wikibase/lib/: SWAT: Touch files on advice of Wikidata folks (duration: 00m 09s)
  • 15:01 logmsgbot: anomie Synchronized wmf-config/Wikibase.php: SWAT: Fix config for specialSiteLinkGroups in Wikibase gerrit:155218 (duration: 00m 09s)
  • 14:49 manybubbles: installing elasticsearch 1.3.2 on elasticsearch1001 only right now as a test
  • 14:47 manybubbles: upgrading elasticsearch plugins on all elasticsearch servers in preparation to upgrade to elasticsearch 1.3 - if we roll back we'll have to redeploy the plugins
  • 14:10 ottomata: changing group ownership and permissions on raw webrequest data in hdfs. Users now must be in the analytics-privatedata-users group to access.
  • 13:47 manybubbles: experimenting with lowering merge factor on enwiki's Cirrus index - should improve query performance at the cost of more background tasks in the Elasticserach cluster
  • 13:36 ottomata: disabling puppet on analytics1027 temporarily
  • 13:10 godog: reboot ms-be1003, xfs errors/panics
  • 12:03 logmsgbot: ori updated /a/common to Ic3fe1ef83: Update all symlinks to /apache
  • 11:36 hashar: Updating Jenkins Job Builder fork 666e953..0268581
  • 11:06 hashar_: mw1019 is missing sync-common causing sync issues.
  • 11:06 logmsgbot: hashar Synchronized wmf-config/InitialiseSettings.php: new domain www.veikkos-archiv.com to wgCopyUploadsDomains 155239 bug 69777 (duration: 00m 03s)
  • 11:05 logmsgbot: hashar Synchronized wmf-config/InitialiseSettings.php: new domain www.veikkos-archiv.com to wgCopyUploadsDomains 155239 bug 69777 (duration: 00m 03s)
  • 10:33 logmsgbot: ori Synchronized w/touch.php: Ic9d8837b1: Canonicalize some remaining references to /apache symlink (duration: 00m 05s)
  • 10:33 logmsgbot: ori Synchronized w/mobilelanding.php: Ic9d8837b1: Canonicalize some remaining references to /apache symlink (duration: 00m 05s)
  • 10:26 logmsgbot: ori updated /a/common to Ic9d8837b1: Canonicalize some remaining references to /apache symlink
  • 10:16 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Id2d5cfa4c: Canonicalize path to $wgSiteMatrixFile (duration: 00m 06s)
  • 09:40 godog: uploaded hhvm_3.3-dev+20140728+wmf5 to carbon
  • 09:27 hashar: restarted Jenkins Gearman plugin.
  • 04:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Aug 20 04:05:41 UTC 2014 (duration 5m 40s)
  • 03:12 logmsgbot: LocalisationUpdate completed (1.24wmf17) at 2014-08-20 03:11:08+00:00
  • 02:40 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-20 02:39:40+00:00

August 19

  • 23:17 logmsgbot: catrope Synchronized php-1.24wmf17/extensions/MobileFrontend: (no message) (duration: 00m 04s)
  • 23:14 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Set wmgWikibaseSiteGroup for wikinews (duration: 00m 05s)
  • 22:59 logmsgbot: aude Synchronized php-1.24wmf17/extensions/Wikidata: Fix badges css on Wikidata (duration: 00m 11s)
  • 22:30 logmsgbot: aude Finished scap: Update Wikidata, WikimediaMessages and ZeroBanner (duration: 22m 02s)
  • 22:08 logmsgbot: aude Started scap: Update Wikidata, WikimediaMessages and ZeroBanner
  • 22:03 logmsgbot: aude Synchronized php-1.24wmf17/extensions/ZeroBanner: Update, per yurik (duration: 00m 18s)
  • 21:22 logmsgbot: aude Synchronized wikidataclient.dblist: Enable Wikibase on Wikinews (duration: 00m 08s)
  • 21:21 logmsgbot: aude Synchronized wmf-config: Config changes to enable Wikibase on Wikinews (duration: 00m 14s)
  • 21:12 aude: added and populated sites table for wikinews
  • 21:05 logmsgbot: aude Synchronized php-1.24wmf17/extensions/Wikidata: Fix badges css and populateSitesTable script in Wikibase (duration: 00m 14s)
  • 20:26 RoanKattouw: Restarting Jenkins, it seems to be stuck
  • 19:58 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: enabling Wikidata to also be a client, e.g. use lua (duration: 00m 09s)
  • 19:58 logmsgbot: aude Synchronized wmf-config/Wikibase.php: enabling Wikidata to also be a client, e.g. use lua (duration: 00m 12s)
  • 19:53 logmsgbot: aude Synchronized php-1.24wmf17/extensions/Wikidata/extensions/Wikibase/lib/resources/: (no message) (duration: 00m 11s)
  • 19:48 logmsgbot: aude Synchronized php-1.24wmf17/extensions/Wikidata/extensions/Wikibase/lib/includes/modules/SitesModule.php: (no message) (duration: 00m 09s)
  • 19:48 logmsgbot: aude Synchronized wmf-config/Wikibase.php: fix config for special site links on Wikidata (duration: 00m 11s)
  • 19:37 logmsgbot: aude Synchronized php-1.24wmf17/extensions/Wikidata/extensions/Wikibase/lib/includes/modules/SitesModule.php: (no message) (duration: 00m 11s)
  • 19:26 logmsgbot: aude Synchronized wmf-config/Wikibase.php: allow adding site links to Wikidata (non-entity) pages on Wikidata (duration: 00m 08s)
  • 19:21 logmsgbot: aude Synchronized wmf-config/Wikibase.php: enable item redirects on Wikidata (duration: 00m 08s)
  • 19:16 logmsgbot: aude Synchronized wmf-config/Wikibase.php: enable badges on Wikidata (duration: 00m 08s)
  • 19:06 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: group1 back to wmf17
  • 19:05 logmsgbot: demon Synchronized php-1.24wmf17/extensions/Wikidata/extensions/Wikibase/lib/includes/changes/EntityChange.php: (no message) (duration: 00m 05s)
  • 18:33 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: all group1 back to wmf16 until WB patch comes
  • 18:22 andrewbogott: added virt1009 to the eqiad virt cluster
  • 18:17 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki back to wmf16
  • 18:13 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to wmf17
  • 15:17 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: CopyUploadDomains for Commons gerrit:154718 (duration: 00m 12s)
  • 15:15 logmsgbot: anomie Synchronized commonsuploads.dblist: SWAT: Remove emlwiki from commonsuploads.dblist gerrit:154714 (duration: 00m 09s)
  • 15:13 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Affiliate namespace on chapcomwiki gerrit:154713 (for real this time) (duration: 00m 09s)
  • 15:12 mark: Completed network migration of BGP confideration renumbering: AS65002 -> AS65001, AS65003 -> AS65004, old AS65001 (pmtpa) is part of eqiad for its remaining lifetime
  • 15:12 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Affiliate namespace on chapcomwiki gerrit:154713 (duration: 00m 09s)
  • 15:10 logmsgbot: anomie Synchronized php-1.24wmf16/includes/parser/Parser.php: SWAT: Fix URL protocol detection regex for file link= parameter gerrit:154844 (duration: 00m 09s)
  • 15:04 logmsgbot: anomie Synchronized php-1.24wmf17/includes/parser/Parser.php: SWAT: Fix URL protocol detection regex for file link= parameter gerrit:154845 (duration: 00m 09s)
  • 14:50 ottomata: starting stat1003 upgrade to trusty
  • 14:37 logmsgbot: demon updated /a/common to I035cebe20: Configure swift-backed snapshots for Cirrus in beta
  • 14:05 logmsgbot: demon Synchronized wmf-config/CirrusSearch-labs.php: beta swift config, no-op (duration: 00m 04s)
  • 13:39 hashar_: Jenkins upgrading hhvm on the Trusty Jenkins slave integration-slave1006-trusty : Unpacking hhvm (3.3-dev+20140728+wmf4) over (3.3-dev+20140728+wmf3)
  • 13:12 logmsgbot: hashar Synchronized wmf-config/InitialiseSettings.php: Fix $wgRestrictionLevels ordering bug 69640 (duration: 00m 04s)
  • 10:19 hashar: Jenkins: bringing back irc bot wmf-insecte in #wikimedia-qa . Will be used to notify failures/fixe of the beta cluster jenkins jobs
  • 09:58 godog: depool mw1019 from appservers, testing trusty+hhvm reinstall RT #8153
  • 07:39 bblack: strontium ok, icinga-wm back
  • 07:17 hashar: Jenkins: manually cleared out a tmpfs partition on lanthanum.eqiad.wmnet which was causing all MediaWiki / extensions jobs to fail completely. bug 69731. We need disk space monitoring which is bug 69733.
  • 07:09 bblack: ... and strontium passenger is failing to start up correctly again. icinga-wm disabled to avoid spam
  • 07:07 bblack: restarted apache2 service on strontium/palladium, expect another small spike of puppet fail->ok
  • 03:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Aug 19 03:20:21 UTC 2014 (duration 20m 20s)
  • 02:37 logmsgbot: LocalisationUpdate completed (1.24wmf17) at 2014-08-19 02:36:21+00:00
  • 02:16 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-19 02:15:14+00:00

August 18

  • 23:03 andrewbogott: isolated virt1006, re-enabling puppet on virt1000 and virt1006
  • 22:36 andrewbogott: disabling puppet on virt1000 and virt1006 while I try to convince the scheduler to overlook virt1006
  • 22:01 bblack: done futzing w/ puppetmasters+neon, all agents enabled and bot back online
  • 21:28 hashar: Zuul processing again. Definitely need to write doc about how to unstuck it
  • 21:02 hashar: Zuul / Jenkins stalled again :-/
  • 21:02 hashar: Zuul / Jenkins stalled again :-/
  • 19:35 bblack: testing new passenger perf params on strontium/palladium. agents on those two and icinga-wm still disabled
  • 19:04 bblack: restarted service apache2 on strontium - passenger for puppet master was dead again
  • 17:00 andrewbogott: added a (yuvi-built) python-txstatsd package to trusty on Carbon.
  • 16:37 bd808: deployment-prep Restarted Apache and HHVM on deployment-mediawiki02 to pick up removal of /etc/php5/conf.d/mail.ini
  • 16:26 logmsgbot: yurik Synchronized php-1.24wmf17/extensions: Syncing JsonConfig,ZeroPortal,ZeroBanner (duration: 01m 13s)
  • 16:22 logmsgbot: yurik Synchronized php-1.24wmf16/extensions: Syncing JsonConfig,ZeroPortal,ZeroBanner (duration: 01m 22s)
  • 16:18 legoktm: migrateAccount.php finished, 2014-08-18 15:42:12 processed 1528652 usernames (22.9/sec), 10 (0.0%) fully migrated, 7938 (0.5%) partially migrated
  • 16:05 hashar: Jenkins tox based jobs are now runnable in parallel 154834
  • 15:36 manybubbles: swat complete
  • 15:29 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - enable cirrus optimization - weighted all fields - on group0 wikis (duration: 00m 07s)
  • 15:29 logmsgbot: manybubbles Synchronized wmf-config/CirrusSearch-common.php: SWAT - drop unused Cirrus parameter (duration: 00m 05s)
  • 15:25 logmsgbot: manybubbles Synchronized php-1.24wmf16/extensions/CentralAuth: SWAT - two centralauth fixes (duration: 00m 05s)
  • 15:22 bblack: resuming slowly wiping varnish caches for mmap update (49 hosts to go), expect small 5xx spikes every ~1.5 hrs for the next few days
  • 15:22 logmsgbot: manybubbles Synchronized wmf-config/: SWAT - noop - sync files adding bouncehandler to betalabs (duration: 00m 04s)
  • 15:19 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - create portal/portal talk namespaces on kowikisource (duration: 00m 04s)
  • 15:18 logmsgbot: manybubbles Synchronized php-1.24wmf17/extensions/CentralAuth/: SWAT - two centralauth fixes (duration: 00m 04s)
  • 15:13 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - create eliminator role on viwiki (duration: 00m 05s)
  • 15:11 logmsgbot: manybubbles Synchronized php-1.24wmf17/extensions/Wikidata/: (no message) (duration: 00m 07s)
  • 15:08 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - Add global-renamer group to metawiki (duration: 00m 04s)
  • 14:50 hashar: Jenkins: reverting PHP CodeSniffer upgrade 154825.We are back to 1.4.7. Previous patch had some issue.
  • 14:42 hashar: Jenkins: upgrading PHP Codesniffer from 1.4.7 to 1.4.8 (thanks to addshore 154053)
  • 14:39 bd808: No apache2.log in fluorine:/a/mw-log; Last file in /a/mw-log/archive is apache2.log-20140816.gz
  • 14:31 bd808: Restarted logstash on logstash1001; event volume was lower than expected
  • 13:49 hashar: restarting zuul. Got stuck again.
  • 13:29 hashar_: Restarted Zuul, some items where stuck in queue. Retrigger your jobs (revote +2 / new patchset / 'recheck' comment)
  • 13:23 logmsgbot: reedy Synchronized php-1.24wmf17/extensions/ExtensionDistributor: Unbreak ExtensionDistributor (duration: 00m 13s)
  • 13:18 hashar: Zuul stuck, looking.
  • 13:06 Reedy: Large amount of incoming traffic to bast1001 is me uploading files
  • 12:11 godog: rebalanced swift object ring in eqiad
  • 09:34 godog: reenabled puppet on neon and started ircecho
  • 09:23 godog: stop ircecho again on neon, disable puppet on neon
  • 09:11 godog: restarted apache2 on strontium
  • 08:58 godog: stopped ircecho on neon while diagnosing puppet failure
  • 03:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Aug 18 03:12:27 UTC 2014 (duration 12m 26s)
  • 03:06 hoo: Ran sync-common on mw1053 to stop "Unrecognized job type 'ChangeNotification'." exceptions
  • 02:31 logmsgbot: LocalisationUpdate completed (1.24wmf17) at 2014-08-18 02:30:17+00:00
  • 02:19 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-18 02:18:52+00:00

August 17

  • 21:07 legoktm: running migrateAccount.php without --safe or --auto on terbium for bug 69291
  • 18:45 hashar: Zuul upgraded
  • 18:41 hashar: Upgrading Zuul to latest version (that is not a friday afterall)
  • 09:22 springle: ongoing schema change wikidatawiki & testwikidatawiki wb_entity_per_page.epp_redirect_target. osc_host.sh processes on terbium ok to kill in emergency
  • 04:34 ottomata: restarted udp2log on oxygen
  • 03:05 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 17 03:04:22 UTC 2014 (duration 4m 21s)
  • 02:49 springle: killed stuff on labsdb1002 using all disk for temp tables. investigating
  • 02:24 logmsgbot: LocalisationUpdate completed (1.24wmf17) at 2014-08-17 02:23:08+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-17 02:13:35+00:00

August 16

  • 18:12 bblack: (amssq33: and yes, removing from fe/be cache pools)
  • 18:11 bblack: powering off amssq33, it's clipping network traffic at peak times due to bad ethernet connection negotiated down to 100Mbps (see existing RT 7933 in esams queue)
  • 18:02 bblack: ms-be1006: syslog indicates it started generating repeated "BUG: soft lockup" 10 minutes before dying, in XFS kernel code again...
  • 17:55 bblack: rebooting ms-be1006, ping-dead in icinga for 23m, console was unresponsive
  • 17:37 bblack: restarted apache2 on palladium... looks like something went horribly wrong with its puppet of itself that somehow killed off puppetmaster service?
  • 03:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 16 03:06:29 UTC 2014 (duration 6m 28s)
  • 02:27 logmsgbot: LocalisationUpdate completed (1.24wmf17) at 2014-08-16 02:26:02+00:00
  • 02:17 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-16 02:16:00+00:00

August 15

  • 20:59 logmsgbot: kaldari Synchronized php-1.24wmf16/extensions/MobileFrontend/less: fixing iOS search bug (duration: 00m 05s)
  • 17:58 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Enable redirects on test.wikidata (duration: 00m 07s)
  • 15:53 logmsgbot: aude Synchronized php-1.24wmf17/extensions/Wikidata: Update test.wikidata (duration: 00m 07s)
  • 15:50 logmsgbot: aude Synchronized php-1.24wmf17/extensions/Wikidata: Fix database error and snak value display on test wikidata (duration: 00m 09s)
  • 15:00 ori: re-enabled puppet on mw1017
  • 13:33 ori: disabling puppet on mw1017 to test rsyslog config
  • 03:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 15 03:50:23 UTC 2014 (duration 50m 22s)
  • 03:04 logmsgbot: LocalisationUpdate completed (1.24wmf17) at 2014-08-15 03:03:49+00:00
  • 02:34 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-15 02:33:21+00:00
  • 00:24 logmsgbot: ori Finished scap: SWAT: cherry picks for TMH and Echo (duration: 14m 38s)
  • 00:09 logmsgbot: ori Started scap: SWAT: cherry picks for TMH and Echo

August 14

  • 23:24 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Bump cache epoch and add badges setting on test.wikidata (duration: 00m 32s)
  • 23:13 logmsgbot: aude Finished scap: Update branch for test.wikidata (duration: 16m 48s)
  • 22:57 logmsgbot: aude Started scap: Update branch for test.wikidata
  • 22:26 logmsgbot: aaron Synchronized php-1.24wmf16/includes/DefaultSettings.php: 67bf481ce1644ff194d7565107d9b8ffe11bf4b7 (duration: 00m 07s)
  • 22:23 logmsgbot: aaron Synchronized wmf-config/CommonSettings.php: Increased wgParsoidCacheUpdateTitlesPerJob to 12 to lower the backlog (duration: 00m 07s)
  • 22:14 logmsgbot: aude scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="test2wiki" --list-file="/a/common/wmf-config/extension-list" --output="/tmp/tmp.kFlVQdKnM2" ' returned non-zero exit status 255 (duration: 00m 40s)
  • 22:13 logmsgbot: aude Started scap: Update branch for test.wikidata
  • 21:49 logmsgbot: reedy Synchronized php-1.24wmf17/includes/context/RequestContext.php: (no message) (duration: 00m 15s)
  • 21:10 godog: restarted hhvm on mw1053
  • 20:47 _joe|away: stopping puppet, jobrunner on mw1053; HHVM is eating memory like godzilla
  • 19:29 bblack: puppeting labmon1001, etc
  • 18:57 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 18:55 logmsgbot: reedy Synchronized database lists: (no message) (duration: 00m 14s)
  • 18:26 mutante: stopped ircecho on neon temporarily
  • 18:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf17
  • 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf16
  • 17:45 AaronSchulz: /srv/deployment/jobrunner updated to 795baf3ca4ce8308597dd74e5242aa5bfbbe961d
  • 17:39 logmsgbot: aaron Synchronized rpc: 6c0ece687bb6ff3fec0ca7e80a587525ebf18a70 (duration: 00m 08s)
  • 16:52 _joe_: uploaded new hhvm package 3.3-dev+20140728+wmf4
  • 16:23 logmsgbot: reedy Synchronized php-1.24wmf17/extensions/CentralAuth/: (no message) (duration: 00m 13s)
  • 16:23 logmsgbot: reedy Synchronized php-1.24wmf16/extensions/CentralAuth/: (no message) (duration: 00m 14s)
  • 15:49 Reedy: Running sync-common on mw1053
  • 15:48 logmsgbot: reedy Finished scap: testwiki to 1.24wmf17 (duration: 33m 13s)
  • 15:47 Jeff_Green: adjust wiki-mail._domainkey DNS record to allow sending from 'wiki*@" addresses, instead of just wiki@
  • 15:23 _joe_: powercycling mw1053, which looks like the victim of hhvm-induced ooms
  • 15:15 logmsgbot: reedy Started scap: testwiki to 1.24wmf17
  • 14:01 _joe_: puppet re-enabled on the appserver
  • 12:38 _joe_: stopping puppet on appservers while deploying a delicate change.
  • 12:12 manybubbles|away: cirrus index rebuilds are still proceeding without issue. Going to continue to let them run and keep half an eye on them. enwiki is nearly done. Commons and wikidata are done. Many of group1 are done - we're up to eswiktionary now - but there are many to go.
  • 09:30 _joe_: the hhvm jobrunner is back in production, seems healthy, see https://logstash.wikimedia.org/#/dashboard/elasticsearch/hhvm_jobrunner
  • 08:09 _joe_: reactivated the jobrunner on mw1053, with promising results. Puppettization pending (in ~ 1 hour)
  • 03:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 14 03:11:33 UTC 2014 (duration 11m 32s)
  • 02:30 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-14 02:29:52+00:00
  • 02:17 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-14 02:16:34+00:00

August 13

  • 21:58 manybubbles: cirrus index rebuild is proceeding without trouble - I'm going to let it continue over night.
  • 21:46 andrewbogott: re-enabled puppetmaster on virt1000; apache changes seem stable now.
  • 21:18 _joe_: stopped puppet on virt1000, our fail
  • 13:23 springle: killed a mass of SpecialWhatLinksHere queries on enwiki
  • 12:51 manybubbles: restarting rebuilding Cirrus indexes to pick up weighted all field
  • 10:35 godog: bump swift weights for ms-be1013 ms-be1014 ms-be1015 to 2500
  • 08:38 hashar: gallium removing some sun-java6* packages coming from old lucid era
  • 07:47 hashar: upgrading Java on contint servers gallium and lanthanum , restarting Jenkins related process
  • 04:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Aug 13 04:02:23 UTC 2014 (duration 2m 22s)
  • 03:12 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-13 03:11:38+00:00
  • 02:41 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-13 02:40:36+00:00

August 12

  • 23:34 logmsgbot: hoo Synchronized tests/multiversion/MWMultiVersionTest.php: (no message) (duration: 00m 11s)
  • 23:32 logmsgbot: hoo Synchronized php-1.24wmf16/skins/Vector/skinStyles/mediawiki.special.preferences.less: Fix missing tab images on Special:Preferences (duration: 00m 10s)
  • 23:26 hoo: Had to abort scap on mw1053 (which is depooled) manually
  • 23:26 logmsgbot: hoo Finished scap: Update WikimediaMessages (superprotect messages for wmf16) (duration: 46m 16s)
  • 22:40 logmsgbot: hoo Started scap: Update WikimediaMessages (superprotect messages for wmf16)
  • 22:21 logmsgbot: hoo Synchronized php-1.24wmf16/extensions/ProofreadPage/: Fix JS error while editing (duration: 00m 10s)
  • 19:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf16
  • 19:06 logmsgbot: reedy Synchronized php-1.24wmf16/includes/specials/SpecialRecentchangeslinked.php: Fix FR bug (duration: 00m 14s)
  • 17:55 AaronSchulz: populateBacklinkNamespace.php finished on all wikis
  • 17:13 springle: restart mysqld on labsdb1002, upgrade to mariadb 10.0.13 for bugfix
  • 16:57 Jeff_Green: removed aluminium.wikimedia.org from production
  • 16:50 springle: restart mysqld on labsdb1001, upgrade to mariadb 10.0.13 for bugfix
  • 15:08 bblack: flipping ulsfo traffic back to ulsfo
  • 11:51 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: Set siteGroup for testwikidata (duration: 00m 11s)
  • 11:21 hashar: Jenkins: clearing up some obsolete symbolic links under gallium.wikimedia.org:/var/lib/jenkins/jobs/*/builds/ Running in a screen as user jenkins
  • 05:01 springle: rsync ~1TB labsdb1001 to labsdb1003, throttled ~25MB/s
  • 04:25 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: s2: repool db1009. s3: repool db1035. (duration: 00m 06s)
  • 03:45 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: s2: depool db1009. repool db1018. adjust db1036 load. (duration: 00m 07s)
  • 03:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Aug 12 03:14:34 UTC 2014 (duration 14m 33s)
  • 02:33 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-12 02:32:09+00:00
  • 02:19 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-12 02:18:36+00:00

August 11

  • 22:33 awight: update CRM schema to wmf_civicrm:7021
  • 21:47 andrewbogott: removed the old puppet-freshness check which should have no effect but may instead produce a torrent of alert spam https://gerrit.wikimedia.org/r/#/c/142560/
  • 04:00 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Aug 11 03:59:17 UTC 2014 (duration 59m 16s)
  • 03:05 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-11 03:04:15+00:00
  • 02:34 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-11 02:33:02+00:00

August 10

  • 23:47 logmsgbot: ori Synchronized php-1.24wmf16/extensions/MassMessage/includes: Revert MassMessage to 9884fbb50a (duration: 00m 06s)
  • 23:36 logmsgbot: ori Synchronized php-1.24wmf16/extensions/MassMessage: Update MassMessage for I840c98dca: Fix MassMessage::getMessengerUser() after Password API changes (duration: 00m 06s)
  • 22:59 logmsgbot: csteipp Finished scap: Deploy Ibe28a69c9fbab00b81c53b1643df722a3f1fbf19 at Eriks request (duration: 26m 13s)
  • 22:33 logmsgbot: csteipp Started scap: Deploy Ibe28a69c9fbab00b81c53b1643df722a3f1fbf19 at Eriks request
  • 16:25 logmsgbot: reedy Finished scap: Rebuild l10n cache for WikimediaMessages (duration: 22m 12s)
  • 16:02 logmsgbot: reedy Started scap: Rebuild l10n cache for WikimediaMessages
  • 15:01 logmsgbot: hoo Synchronized php-1.24wmf16/extensions/CentralAuth/: (no message) (duration: 00m 25s)
  • 15:00 logmsgbot: hoo Synchronized php-1.24wmf15/extensions/CentralAuth/: (no message) (duration: 00m 25s)
  • 13:53 Reedy: Grant staff "superprotect" right per Robla/Erik request
  • 13:02 logmsgbot: tstarling Synchronized wmf-config/InitialiseSettings.php: Idfa21125 (duration: 00m 05s)
  • 13:02 logmsgbot: tstarling Synchronized wmf-config/CommonSettings.php: Idfa21125 (duration: 00m 06s)
  • 12:08 mutante: re-enabling puppet and services on tarin
  • 11:57 mutante: tarin - stopping poolcounterd, gmond,.. (Tampa, should really not be in use)
  • 03:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 10 03:14:52 UTC 2014 (duration 14m 51s)
  • 02:34 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-10 02:33:15+00:00
  • 02:20 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-10 02:19:53+00:00

August 9

  • 15:22 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I313b09ffc: Don't require native CDB support to load {interwiki,trustedxff}.cdb (duration: 00m 05s)
  • 14:25 Reedy: Removed <= MediaWiki 1.24wmf5
  • 13:29 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: move s4 api traffic back to db1042 (duration: 00m 06s)
  • 11:32 mutante: added Ryan Lane to NDA LDAP group
  • 03:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 9 03:20:07 UTC 2014 (duration 20m 6s)
  • 02:37 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-09 02:36:52+00:00
  • 02:20 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-09 02:19:03+00:00

August 8

  • 21:24 Reedy: mw1130 seems to be dead (unresponsive to ping)
  • 21:21 logmsgbot: reedy Synchronized wmf-config/interwiki.cdb: (no message) (duration: 01m 04s)
  • 21:06 awight: deployed crm default/settings.php
  • 17:07 mutante: jenkins/puppet-compiler - granting new LDAP group "nda" the same rights already given to matanya (and wmde even has more)
  • 16:06 bblack: datacenter traffic mapping back to normal, varnish fix/wipe/restart/etc work on pause for the weekend in a stable state
  • 16:02 andrewbogott: merging https://gerrit.wikimedia.org/r/#/c/150273/ which affects every puppet log everywhere...
  • 14:22 mutante: RT - reverted permission change for access requests requestors per robh
  • 13:50 mutante: RT - granted permission to show ticket summary for role requestor in queue access-requests
  • 12:49 akosiaris: uploaded ruby-jsduck 5.3.4-1wmftrusty1 and ruby-rkelly-remix 0.0.6-1trusty1 on apt.wikimedia.org
  • 12:33 ori: testwiki up, judgement poor
  • 12:28 hashar: Jenkins: somehow the ArtifactDeployer plugin got upgraded on Aug 7th 20:57 UTC despite it being broken bug 69197. Attempting manual downgrade
  • 12:13 hashar: reloading Jenkins
  • 12:07 akosiaris: ifconfig br0 0.0.0.0 on platinum to get rid of the IP on that interface and have facter work more reliably. This does not matter right now as it is an evaluation machine but logging it for completeness
  • 12:03 logmsgbot: ori rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
  • 11:32 _joe_: rebooting mw1017
  • 11:29 akosiaris: mw1130 has broken disk
  • 11:09 ori: running rsync-common on mw1017
  • 11:02 logmsgbot: hoo Synchronized php-1.24wmf16/extensions/CentralAuth/: Another shot towards bug 39996 (duration: 01m 04s)
  • 11:01 logmsgbot: hoo Synchronized php-1.24wmf15/extensions/CentralAuth/: Another shot towards bug 39996 (duration: 01m 04s)
  • 09:29 _joe_: reimaging mw1017 aka testwiki.
  • 06:03 springle: ongoing schema changes: rev_content_model, rev_content_format. on terbium, osc_host.sh processes ok to kill in emergency
  • 03:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 8 03:12:21 UTC 2014 (duration 12m 20s)
  • 02:29 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-08 02:28:39+00:00
  • 02:17 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-08 02:16:13+00:00

August 7

  • 19:19 jgage: rebooting analytics1021 for kernel upgrade
  • 18:55 bblack: starting the process of fixing upload cache sizes, there will be periodic slim 5xx spikes...
  • 16:31 Jeff_Green: temporarily disabling icinga notifications for ocg100[123] ocg service check
  • 16:09 logmsgbot: krinkle Synchronized php-1.24wmf16/extensions/GlobalCssJs/GlobalCssJs.hooks.php: 4bbf4e0ed92f9a09 (duration: 00m 05s)
  • 15:48 mutante: zirconium - attempt to fix apache site setup manually
  • 15:46 logmsgbot: reedy Synchronized wmf-config/extension-list-labs: (no message) (duration: 00m 13s)
  • 15:38 logmsgbot: reedy Synchronized php-1.24wmf16/maintenance/findMissingFiles.php: (no message) (duration: 00m 20s)
  • 15:37 logmsgbot: reedy Synchronized php-1.24wmf15/maintenance/findMissingFiles.php: (no message) (duration: 00m 17s)
  • 15:12 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 13s)
  • 14:43 akosiaris: uploaded varnish_3.0.5plus~x-wm7trusty1 on apt.wikimedia.org (for usage in trusty labs machines, notably cxserver)
  • 14:23 mutante: shutting down elastic1018
  • 14:12 ^d: elastic1018: blacklisted from shard allocation since it's dead
  • 14:05 mutante: depooled elastic1018 - service wasnt running and signs of broken hardware (SSD)
  • 13:57 mark: Temporarily set max connections to swift from cp1049 backend varnish from 1000 to 2000
  • 13:56 mutante: starting elasticsearch on elastic1018
  • 12:23 hashar: Zuul upgraded labs branch to match production (i.e. have same version of Zuul cloner)
  • 12:20 hashar: restarting Zuul
  • 11:25 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: I53f76a35ac - No longer allow voyage 'crats to usermerge (duration: 00m 15s)
  • 11:13 akosiaris: removed laner@wikimedia.org entirely. It pointed to rlane@wikimedia.org which no longer exists
  • 11:11 akosiaris: removed rlane from root@wikimedia.org and usability@wikimedia.org
  • 10:45 mutante: iron, bast1001 - installed package upgrades
  • 09:13 hashar: Jenkins: polling a new Jenkins slave using Trusty integration-slave1006-trusty [10.68.17.223] with 4 CPU. Copy pasted from 1004-trusty
  • 08:32 hashar: Jenkins: switching job from https://github.com/wmf-analytics/libcidr/ to https://gerrit.wikimedia.org/r/analytics/libcidr
  • 07:44 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: move s4 api traffic to db1056 (duration: 00m 07s)
  • 07:39 mark: Set OSPF metric 1000 on cr2-eqiad:xe-5/2/2 (GTT link)
  • 05:39 springle: labsdb1002 restart
  • 03:48 springle: labsdb1001 restart
  • 03:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 7 03:08:49 UTC 2014 (duration 8m 48s)
  • 02:28 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-07 02:27:52+00:00
  • 02:16 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-07 02:15:45+00:00

August 6

  • 21:33 hashar: Jenkins: moved mediawiki-core-regression-hhvm-master to run on Trusty instance
  • 20:26 hashar: Jenkins: downgraded ansicolor plugin from 0.4 to 0.3.1 Some colors.js function emits ANSI codes to reset the color which are not properly understood
  • 20:06 hashar: I have broke Zuul/Jenkins :-]
  • 18:53 hashar: Jenkins slow startup is bug 69197
  • 18:50 hashar: restarting jenkins
  • 18:49 hashar: Stopping Jenkins. Reverting upgrade of artifact deployer plugin
  • 18:10 mutante: puppet-catalog-compiler says to "wait while Jenkins is getting ready to work"
  • 17:20 hashar: Jenkins process jobs again, the UI will take a bunch of hours to load though due to some issue when initializing
  • 17:14 hashar: killed Jenkins
  • 17:12 _joe_: stopped the jobrunner on mw1053, was running in fcgi mode unpuppetized and with a broken vhost. Fixed it, it started spawning exceptions. DO NOT enable puppet again
  • 17:02 ^d: jenkins restarted, was stuck
  • 15:52 hashar: Restarted Zuul and Zuul-merger on gallium to tweak logging settings 152118
  • 11:30 logmsgbot: hoo Synchronized wmf-config/CommonSettings.php: Grant 'centralauth-rename' to 'steward' (duration: 00m 24s)
  • 11:26 logmsgbot: demon Synchronized wmf-config/abusefilter.php: (no message) (duration: 00m 19s)
  • 10:10 hashar: Jenkins web interface is back up
  • 09:54 logmsgbot: demon Synchronized wmf-config/abusefilter.php: abuse filter settings for fawiki (duration: 00m 21s)
  • 07:33 hashar: restarting Jenkins. It apparently like to parse the whole history on reload, so aborting that.
  • 07:13 hashar: Upgrading Jenkins plugin and restarting.
  • 07:04 hashar: upgrading Jenkins to latest LTS
  • 03:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Aug 6 03:10:06 UTC 2014 (duration 10m 5s)
  • 02:30 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-06 02:29:00+00:00
  • 02:17 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-06 02:16:38+00:00

August 5

  • 15:07 logmsgbot: root gracefulled all apaches
  • 15:03 logmsgbot: root gracefulled all apaches
  • 12:30 hasharEat: Upgrading python-gear on gallium and restarting zuul and zuul-merger
  • 12:26 akosiaris: uploaded python-gear_0.5.5-1 on apt.wikimedia.org
  • 03:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Aug 5 03:08:30 UTC 2014 (duration 8m 29s)
  • 02:29 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-05 02:27:58+00:00
  • 02:17 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-05 02:16:52+00:00
  • 00:41 springle: ongoing schema changes: ar_content_model, ar_content_format. on terbium, osc_host.sh processes ok to kill in emergency

August 4

  • 22:58 bblack: rebooting ms-be1012
  • 21:47 ottomata: reenabling puppet on analytics1027
  • 21:46 jgage: all kafka brokers upraded to 0.8.1.1 and data replicated: done
  • 20:37 ottomata: stopping puppet on analytics1027 to temporarily disable camus cron job
  • 19:07 ottomata: starting upgrade of kafka cluster
  • 19:02 logmsgbot: maxsem Synchronized php-1.24wmf16/includes/User.php: https://gerrit.wikimedia.org/r/#/c/151691/ (duration: 00m 06s)
  • 18:57 jgage: beginning kafka upgrade: disabling puppet on brokers
  • 13:17 apergos: stopped labs rsync job from dataset1001, mount of labstore1003 was borked, removed 90GB of stuff on /mnt/data (= /) filesystem, restarted nfsd on dataset1001, dumps back to going
  • 03:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Aug 4 03:11:03 UTC 2014 (duration 11m 2s)
  • 02:29 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-04 02:27:58+00:00
  • 02:17 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-04 02:16:46+00:00

August 3

  • 03:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 3 03:28:56 UTC 2014 (duration 28m 55s)
  • 02:28 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-03 02:27:44+00:00
  • 02:17 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-03 02:16:39+00:00

August 2

  • 15:28 godog: reboot ms-be1008, stuck on xfs errors and most processes in D state
  • 14:10 Krinkle: Restarting Zuul
  • 14:08 hashar: Jenkins / Zuul stuck bug 69045
  • 14:00 Krinkle: Restarting Jenkins in attempt to unstuck the clogged Zuul pipeline for gallium
  • 04:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 2 04:20:45 UTC 2014 (duration 20m 44s)
  • 02:33 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-02 02:32:36+00:00
  • 02:21 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-02 02:20:02+00:00
  • 01:49 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1018, replag (duration: 00m 06s)
  • 00:43 Krinkle: Restarting Jenkins on gallium because the pipeline is clogged

August 1

  • 20:25 andrewbogott: shorted the logrotate interval on vanadium; disk space critical should resolve soon
  • 18:10 logmsgbot: csteipp Synchronized php-1.24wmf16/extensions/CentralAuth: Fix for bug 69007 - logins failing for old style hashes (duration: 00m 06s)
  • 17:32 AaronSchulz: Restarted maintenance/populateBacklinkNamespace.php on enwiki
  • 17:31 logmsgbot: aaron Synchronized php-1.24wmf15/maintenance/populateBacklinkNamespace.php: e1cea29342f964cd9a720310185b09ca41eb1a4a (duration: 00m 04s)
  • 17:16 akosiaris: upgraded etherpad-lite on zirconium to 1.4.0-2. Uploaded etherpad-lite_1.4.0-2 on apt.wikimedia.org
  • 17:11 logmsgbot: aaron Synchronized php-1.24wmf15/includes: d218d86dff90a5f0110353c492bd2e8ddaf35497 (duration: 00m 08s)
  • 17:09 logmsgbot: aaron Synchronized php-1.24wmf16/includes: f1a8ff7f802b57cc9f452d47c4c762a185ed93c2 (duration: 00m 06s)
  • 15:48 logmsgbot: reedy Synchronized php-1.24wmf16/includes/specials/SpecialRecentchangeslinked.php: (no message) (duration: 00m 14s)
  • 12:07 apergos: powercycled dataset1001, inaccessible via mgmt console, only visible message was 'mnt.nfs failed'
  • 09:10 _joe_: apache mediawiki::web train finished its run. re-enabling puppet on all appservers
  • 07:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 1 07:46:04 UTC 2014 (duration 46m 3s)
  • 07:24 _joe_: stopping puppet on appservers to deploy a potentially dangerous case
  • 05:16 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: Move enwiki api traffic away from lagging slaves (duration: 00m 07s)
  • 03:12 logmsgbot: LocalisationUpdate completed (1.24wmf16) at 2014-08-01 03:11:14+00:00
  • 02:40 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-08-01 02:38:56+00:00
  • 00:52 logmsgbot: catrope Synchronized php-1.24wmf16/extensions/VisualEditor/lib/ve/modules/ve/ui/inspectors/ve.ui.CommentInspector.js: Fix typo in class name (duration: 00m 10s)

July 31

  • 23:23 logmsgbot: mwalker Synchronized php-1.24wmf16: Updating core and Flow for SWAT (duration: 00m 53s)
  • 23:05 logmsgbot: mwalker Synchronized wmf-config: Updating configuration for 150145 (duration: 00m 05s)
  • 21:17 RobH: blog.wikimedia.org cname changed to migrate over to wp servers
  • 20:22 AaronSchulz: Started populateBacklinkNamespace.php on s1-s3,s5-s7 (commons already running)
  • 20:13 cscott: updated OCG to version d2919c59eb09e09fc87777696411a070620aef45
  • 19:40 hashar: Jenkins build its first hhvm extension \O/ https://integration.wikimedia.org/ci/job/php-FastStringSearch-hhvm-build/2/console
  • 19:24 Coren_away: labsdb1005 had to blow away the postgres slave: was using all the space on / because DB at wrong spot (should have been /srv/postgres)
  • 18:40 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 15s)
  • 18:27 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 15s)
  • 18:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf16
  • 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.24wmf15
  • 17:47 logmsgbot: aaron Synchronized wmf-config/CommonSettings.php: Increased "htmlCacheUpdate" throttle limit (duration: 00m 07s)
  • 17:46 logmsgbot: reedy Finished scap: testwiki to 1.24wmf16 and build l10n cache (duration: 22m 35s)
  • 17:23 logmsgbot: reedy Started scap: testwiki to 1.24wmf16 and build l10n cache
  • 14:57 bblack: added labstore1003 to filter labs-in4 terms allow-labstore-(udp|tcp)4 on cr[12]-eqiad
  • 14:33 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Allow sysops and 'crats on wikimania2014wiki to grant confirmed (duration: 00m 15s)
  • 14:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikivoyages to 1.24wmf15
  • 14:12 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 14s)
  • 14:10 logmsgbot: reedy Finished scap: Rebuild 1.24wmf15 l10n cache for WikimediaMessages updates (duration: 22m 40s)
  • 14:05 bblack: removed labs-in4 and labs-in6 filters on vlan 1117 (labs-hosts1-a-eqiad) on cr[12]-eqiad
  • 13:47 logmsgbot: reedy Started scap: Rebuild 1.24wmf15 l10n cache for WikimediaMessages updates
  • 13:44 logmsgbot: reedy Synchronized php-1.24wmf15/extensions/RelatedSites/: (no message) (duration: 00m 15s)
  • 13:44 logmsgbot: reedy Synchronized php-1.24wmf15/extensions/WikimediaMessages: (no message) (duration: 00m 14s)
  • 12:10 hashar: stopping Jenkins and restarting it
  • 12:04 hashar: reloading Jenkins configuration
  • 11:37 hashar: Jenkins: upgrading almost all jobs to use a new label 'UbuntuPrecise' bug 68340 150785
  • 10:49 hashar: Jenkins: attempting to poll a Trusty slave (integration-slave1004-trusty [10.68.17.148] with label UbuntuTrusty).
  • 10:32 hashar: Jenkins: tweaking jobs labels, that might eventually screw up Zuul/Jenkins entirely.
  • 08:43 _joe_: start rolling reload of nginx to catch up with the new ssl config
  • 06:50 springle: labsdb1001 migration complete, should be all systems go
  • 03:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 31 03:18:07 UTC 2014 (duration 18m 6s)
  • 02:36 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-07-31 02:35:29+00:00
  • 02:20 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-31 02:19:17+00:00
  • 02:06 springle: labsdb1001 migrating to mariadb 10, expect read-only and downtime, see labs-l

July 30

  • 23:27 logmsgbot: maxsem Synchronized php-1.24wmf15/extensions/MwEmbedSupport/: (no message) (duration: 00m 03s)
  • 23:27 logmsgbot: maxsem Synchronized php-1.24wmf15/extensions/Wikidata/: (no message) (duration: 00m 08s)
  • 23:26 logmsgbot: maxsem Synchronized php-1.24wmf15/extensions/SyntaxHighlight_GeSHi/: (no message) (duration: 00m 05s)
  • 23:23 logmsgbot: maxsem Synchronized php-1.24wmf14/extensions/Wikidata: (no message) (duration: 00m 11s)
  • 23:13 logmsgbot: maxsem Synchronized wmf-config: (no message) (duration: 00m 05s)
  • 21:04 AaronSchulz: Started populateBacklinkNamespace.php on wikidata and commons
  • 21:02 bblack: turned icinga email/sms back on
  • 20:24 bblack: icinga back online again
  • 19:57 bblack: shutting off icinga to make some optimizations
  • 19:20 bblack: icinga is now substantially back online. email/sms still disabled for now, and downtimes/acks need to be re-added for known issues
  • 19:06 logmsgbot: csteipp Synchronized php-1.24wmf14/includes/: (no message) (duration: 00m 05s)
  • 19:04 logmsgbot: csteipp Synchronized php-1.24wmf15/includes/: (no message) (duration: 00m 07s)
  • 18:59 bblack: icinga coming back up again for the first time, expect random strangeness to be ignored
  • 18:46 bblack: temporarily hard-disabling email/sms from icinga via 'mv /usr/bin/mail /usr/bin/mail-disabled' on neon to prevent icinga spam on next startup attempt
  • 17:55 bblack: stopping icinga service for now while working out other details
  • 17:25 tacotuesday: repooled elastic1018 and elastic1019 as well
  • 17:21 Coren: labmon1001 rebooting (final check for proper raid+lvm autodetection)
  • 17:08 bblack: working on bringing up new neon install (first puppet run, etc)
  • 17:01 Coren: labmon1001 rebooting (partitioning changes on primary disks)
  • 16:53 tacotuesday: elastic1017 repooled, shards allocating
  • 16:13 bd808: scap and dologmsg from tin won't work until neon is back up and running tcpircbot
  • 16:07 bd808|deploy: Synchronized touch: no-op sync to test scap update (duration: 00m 05s)
  • 16:06 bd808|deploy: scap announce failed -- timeout connecting to tcpircbot on neon.wikimedia.org
  • 16:04 bd808|deploy: Updated scap to 4871208 (rely on $PATH for scap scripts)
  • 15:21 logmsgbot: hoo Synchronized php-1.24wmf15/extensions/Wikidata/extensions/Wikibase/lib/resources/wikibase.js: touch (duration: 00m 20s)
  • 15:17 hashar: upgrading php5 on jenkins slaves
  • 15:07 cmjohnson1: shutting down neon
  • 14:46 logmsgbot: demon Synchronized wmf-config/CirrusSearch-production.php: (no message) (duration: 00m 04s)
  • 14:35 logmsgbot: demon Synchronized wmf-config/PrivateSettings.php: Swift config for Cirrus (duration: 00m 08s)
  • 14:30 godog: rolling restart of ms-fe* to pick up search backup user
  • 14:17 bblack: rebooting neon again, trying to fix the disk situation
  • 14:11 Coren: reinstalling labmon1001 -> change disk partitioning scheme
  • 13:50 springle: neon read-only fs. fsck + reboot
  • 13:16 manybubbles: rebuiding Cirrus index for commons to pick up weighted all field
  • 11:17 _joe_: enabling puppet on all mw* servers
  • 11:15 _joe_: re-enabling puppet on mw1019, last bunch of tests, then re-enabling globally
  • 10:58 _joe_: re-enabling puppet on mw1018, testwiki upgraded to the new config and looks fine
  • 09:25 godog: set weight for ms-be1014 and ms-be1015 to 2300
  • 08:58 _joe_: stopping puppet on the appservers, in preparation for releasing change 148099
  • 08:30 _joe_: powercycling neon, doesn't respond to requests, ssh hangs, console dark
  • 06:41 springle: labsdb1001 work in progress; it may misbehave. see labs-l for updates
  • 04:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 30 04:27:56 UTC 2014 (duration 27m 55s)
  • 03:39 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-07-30 03:38:28+00:00
  • 02:51 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-30 02:50:14+00:00
  • 01:47 bblack: ip addr del for cp4017's ip6_mapped addr on cp4018 (no idea why it was there...)

July 29

  • 23:37 logmsgbot: catrope Finished scap: SWAT updates for wmf15, I'm lazy (duration: 07m 02s)
  • 23:30 AaronSchulz: Updated /srv/jobrunner to d2298139ea22bf8e48de066a73f28024b140ea33
  • 23:30 logmsgbot: catrope Started scap: SWAT updates for wmf15, I'm lazy
  • 23:28 logmsgbot: catrope Synchronized php-1.24wmf14/extensions/VisualEditor: (no message) (duration: 00m 05s)
  • 23:28 logmsgbot: catrope Synchronized php-1.24wmf14/extensions/MobileFrontend: (no message) (duration: 00m 05s)
  • 23:18 logmsgbot: catrope Synchronized wmf-config/: Do not put OCG in sidebar (duration: 00m 04s)
  • 23:11 logmsgbot: catrope Synchronized wmf-config/: Enable TemplateData GUI on nlwiki (duration: 00m 05s)
  • 23:10 bblack: took OCG service IP out of downtime in icinga, it's live
  • 23:06 logmsgbot: mwalker Synchronized wmf-config: Enabling OCG in production (duration: 00m 04s)
  • 23:05 logmsgbot: aaron Synchronized rpc: 0df032d957155aa475d99e2b887ba98b9a4c32fd (duration: 00m 07s)
  • 23:04 logmsgbot: cscott Synchronized wmf-config: (no message) (duration: 00m 12s)
  • 23:03 logmsgbot: cscott updated /a/common to Iae1ac79d5: Enable OCG in production
  • 22:55 cscott: updated OCG to version aeb8623d6ebe41ae7c7e36c57844bd9ea8e6d595
  • 22:50 RoanKattouw: Fixed ownership of slot0/cache on wikitech (virt1000), was root:root but should have been www-data:www-data
  • 22:24 RoanKattouw: Updated lib/ve submodule inside extensions/VisualEditor on virt1000; wikitechwiki was running a Frankenstein version of VE that was part yesterday's code, part code from April
  • 21:47 logmsgbot: ori Synchronized rpc/RunJobs.php: Ia62e9158f: Added a streamlined RunJobs that can be used by redisJobService (2/2) (duration: 00m 03s)
  • 21:47 logmsgbot: ori Synchronized multiversion: Ia62e9158f: Added a streamlined RunJobs that can be used by redisJobService (1/2) (duration: 00m 03s)
  • 21:44 Reedy: cleared bottuzzu@itwiki watchlist
  • 21:32 spagewmf: spage ran `mwscript namespaceDupes.php --wiki=enwiki --prefix Topic`, 5 pages renamed
  • 21:22 logmsgbot: spage Synchronized wmf-config/InitialiseSettings.php: Enable Flow on Wikimania testing page (duration: 00m 13s)
  • 21:22 logmsgbot: ori updated /a/common to Ia62e9158f: Added a streamlined RunJobs that can be used by redisJobService
  • 21:18 logmsgbot: spage updated /a/common to I3b4622e27: Wikivoyages back to 1.24wmf14
  • 20:54 logmsgbot: aaron Synchronized php-1.24wmf14/includes/media: b45248509c07acb8146d6e735ef68dff193ac290 (duration: 00m 07s)
  • 19:46 Krinkle: Reloading Zuul to deploy I7f80ee0b85d29791b7
  • 19:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikivoyages back to 1.24wmf14
  • 19:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikivoyages back to 1.24wmf15...
  • 19:14 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 14s)
  • 19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikivoyages back to 1.24wmf14
  • 18:43 cmjohnson1: power cycling virt1009
  • 18:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf15
  • 18:28 logmsgbot: reedy Synchronized php-1.24wmf15/extensions/Wikidata/extensions/Wikibase/lib/config/WikibaseLib.default.php: touch (duration: 00m 16s)
  • 18:26 bblack: removed "filter { input labs6-in; }" from ae3.1119 (labs-support1-c-eqiad) on cr[12]-eqiad
  • 17:52 logmsgbot: aaron Synchronized php-1.24wmf15/includes/media: 76459cebd9cfbb33e9845f7acd8b8c1382cdae61 (duration: 00m 08s)
  • 16:56 logmsgbot: hoo Synchronized wmf-config/Wikibase.php: Bump $wgCacheEpoch for testwikidata (duration: 00m 08s)
  • 16:52 logmsgbot: hoo Synchronized php-1.24wmf15/extensions/Wikidata/: Touch JS (duration: 00m 10s)
  • 16:52 logmsgbot: hoo Synchronized php-1.24wmf14/extensions/Wikidata/: Touch JS (duration: 00m 11s)
  • 16:50 logmsgbot: hoo Synchronized wmf-config/Wikibase.php: Only declare "special" sitegroups for testwikidata (duration: 00m 07s)
  • 16:48 logmsgbot: hoo Synchronized wmf-config/Wikibase.php: Only declare "special" sitegroups for testwikidata (duration: 00m 08s)
  • 16:47 logmsgbot: hoo Finished scap: Updating Wikidata with various changes for testwikidata and a client bug fix. (duration: 27m 27s)
  • 16:37 cmjohnson1: replacing defective disk virt1009
  • 16:20 logmsgbot: hoo Started scap: Updating Wikidata with various changes for testwikidata and a client bug fix.
  • 16:10 logmsgbot: hoo Synchronized wmf-config/Wikibase.php: Make testwikidata use the "special" sitelink group. Preparations for submodule updates. (duration: 00m 08s)
  • 16:10 bd808: logstash log event volume up after restart
  • 16:09 bd808: restarted logstash on logstash1001.eqiad.wmnet; log volume looked to be down from expected levels
  • 16:08 _joe_: reenabled puppet on mw1053
  • 16:03 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase other projects links per default for ruwiki (duration: 00m 07s)
  • 15:13 manybubbles: building cirrus indexes for group0 wikis in place to turn on the weighted all field we'll use for performance improvements later
  • 15:06 logmsgbot: manybubbles Synchronized wmf-config: SWAT - deploy cirrussearch all field stage 2 part 2 (duration: 00m 04s)
  • 15:06 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - deploy cirrussearch all field stage 2 part 1 (duration: 00m 04s)
  • 13:54 logmsgbot: hashar Synchronized wmf-config/InitialiseSettings.php: added Universiteits Museum Utrecht to the wgCopyUploadsDomains array 150163 (duration: 00m 04s)
  • 13:38 ottomata: restarted gmetad on nickel, seems to have brought ganglia back up
  • 11:30 _joe_: upgrading packages on mw1053, for testing hhvm with pcre-jit enabled
  • 10:35 _joe_: puppet re-enabled on the appservers
  • 10:29 _joe_: temporarily stopping puppet on appservers, releasing a potentially dangerous puppet change
  • 09:10 _joe_: stopping jobrunner on mw1053, disabling puppet as well - running tests
  • 09:02 hashar: restarted zuul-server and zuul-merger on gallium (new version though that is a noop)
  • 09:00 hashar_: Zuul bumping Zuul cloner from patchset 21 to patchset 23. Deploying with tag wmf-deploy-2014-07-29-1
  • 07:51 akosiaris: uploaded PHP 5.3.10-1ubuntu3.13+wmf1 on apt.wikimedia.org. Puppet will upgrade it across the fleet within 20 mins
  • 03:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 29 03:47:39 UTC 2014 (duration 47m 38s)
  • 03:11 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-07-29 03:10:31+00:00
  • 02:36 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-29 02:35:18+00:00
  • 00:44 logmsgbot: aaron Synchronized php-1.24wmf15/maintenance/runJobs.php: fcfa3153e53dc70e6cd190a087e7bd577fe380fb (duration: 00m 03s)
  • 00:27 logmsgbot: aaron Synchronized php-1.24wmf15/maintenance: f754c239ce93fc5f2db19e93f4fe8a1d1ba7bc27 (duration: 00m 04s)
  • 00:27 logmsgbot: aaron Synchronized php-1.24wmf15/includes: f754c239ce93fc5f2db19e93f4fe8a1d1ba7bc27 (duration: 00m 06s)

July 28

  • 23:58 logmsgbot: ori Finished scap: I42c07b64: Update MobileFrontend (duration: 17m 37s)
  • 23:41 logmsgbot: ori Started scap: I42c07b64: Update MobileFrontend
  • 23:33 logmsgbot: ori Synchronized php-1.24wmf15/extensions/VisualEditor: Update VisualEditor to I944f8fbfa (duration: 00m 04s)
  • 23:25 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I369dbad6e: Allow crats to add/remove petitiondata group on foundationWiki (duration: 00m 04s)
  • 23:21 AaronS: Updated /srv/jobrunner to 0bb0ad62dd9240e0f67b2ded4519f125de13dfbc
  • 23:12 mutante: temp. disabled puppet on neon and ircecho
  • 23:06 mutante: graceful apache on palladium
  • 21:12 hashar: Gerrit: allowed JenkinsBot to submit patches on wikimedia/bots (and thus on all child repositories)
  • 20:50 hashar: operations/puppet.git manifests should no more have leading tabulations I69ddc7
  • 20:08 bblack: intermittent 5xx are most likely varnish restarts off and on rest of today
  • 19:51 hashar: Zuul: stopped / started process to clear up obsoletes changes stuck in queue
  • 19:47 hashar: Jenkins/Zuul lost connection somehow. Disabled/Reenabled gearman client in Jenkins
  • 19:44 hashar: Jenkins: updated qunit jobs to roam on both gallium and lanthanum (were previously tied to run only on gallium)
  • 19:42 ottomata: restarted varnishkafka on some esams hosts that have old misconfigured vk processes
  • 19:13 ottomata: restarting varnishkafka on amssq31
  • 19:08 ottomata: restarting varnishkafka on cp3013
  • 17:46 logmsgbot: aaron Synchronized php-1.24wmf14/includes/jobqueue/JobQueueFederated.php: 87e7bfceb795d065d6157ac8ce3381a7814000b5 (duration: 00m 03s)
  • 17:38 logmsgbot: aaron Synchronized php-1.24wmf15/includes/jobqueue/JobQueueFederated.php: 12ce1dc1ec46b06d1160e142ddfaf8dcb1c9f131 (duration: 00m 04s)
  • 16:30 andrewbogott: updated wikitech to 1.24wmf15; turned on OAuth
  • 16:05 Nemo_bis: andrewbogott> Nikerabbit: I'm upgrading it [wikitech wiki], it'll be flaky for a bit
  • 16:00 manybubbles: deone with SWAT
  • 15:57 logmsgbot: manybubbles Synchronized php-1.24wmf14/extensions/VisualEditor/: SWAT - fix visual editor bug - Changes made after reviewing changes are not sent (when caching is enabled) (duration: 00m 07s)
  • 15:46 logmsgbot: manybubbles Synchronized php-1.24wmf15/extensions/VisualEditor/: SWAT - fix visual editor bug - Changes made after reviewing changes are not sent (when caching is enabled) (duration: 00m 08s)
  • 15:41 hoo: Removed all right holders from closed and inaccessible ukwikimedia (bug 68737)
  • 15:39 logmsgbot: manybubbles Synchronized php-1.24wmf15/includes/specials/SpecialRevisiondelete.php: SWAT - fix fatal on revision delete (duration: 00m 08s)
  • 15:33 logmsgbot: manybubbles Synchronized wmf-config/CommonSettings.php: SWAT load Mantle before MobileFrontent (duration: 00m 07s)
  • 15:31 logmsgbot: manybubbles Synchronized php-1.24wmf14/extensions/Echo/: SWAT fix bad variable name in echo (duration: 00m 08s)
  • 15:23 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - update some permissions on eswiki (duration: 00m 08s)
  • 15:17 logmsgbot: manybubbles Synchronized php-1.24wmf15/extensions/Echo/: SWAT - fix incorrect variable name (duration: 00m 08s)
  • 15:14 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - add import sources to bhwiki (duration: 00m 08s)
  • 15:10 logmsgbot: manybubbles Synchronized php-1.24wmf14/extensions/FundraisingTranslateWorkflow/: SWAT update fundraising to fix botched deploy
  • 12:28 hashar: Upgrading our Jenkins Job Builder fork ( d833015..666e953 )
  • 03:00 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 28 02:59:35 UTC 2014 (duration 59m 34s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-07-28 02:25:34+00:00
  • 02:16 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-28 02:15:00+00:00

July 27

  • 05:24 springle: mysqldump s6 dbstore1002 to dbstore1001
  • 02:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 27 02:58:15 UTC 2014 (duration 58m 14s)
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-07-27 02:24:10+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-27 02:13:44+00:00

July 26

  • 21:29 hashar: restarting Zuul to clear up some stalled changes.
  • 02:58 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 26 02:57:52 UTC 2014 (duration 57m 50s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-07-26 02:25:46+00:00
  • 02:16 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-26 02:15:31+00:00

July 25

  • 22:52 mutante: Bugzilla - upgraded to 4.4.5
  • 22:41 mutante: ocg - deleted old log dirs
  • 19:28 hashar: Jenkins : disabling gearman plugin and reenabling it (just uncheck/save/check a box in https://integration.wikimedia.org/ci/configure )
  • 19:25 hashar: zuul@gallium:/etc/zuul/wikimedia$ echo status|nc -q 3 localhost 4730|wc -l ... Yields: 0 . Which mean jobs are no more registered for some reason.
  • 19:24 hashar: Jenkins stalled again yeahhhhh
  • 16:59 mutante: powercycled ms-be1010 - unresponsive to ssh, nothing on mgmt
  • 16:28 MaxSem: Updating PageImages data for mainspace on Commons from terbium
  • 13:09 _joe_: re-enabling puppet, test run on the test host was fine.
  • 13:03 _joe_: stopping puppet on all appservers - will reactivate after testing
  • 11:26 logmsgbot: reedy Synchronized docroot and w: (no message) (duration: 00m 13s)
  • 10:50 hashar: contint: manually cleared /tmp on the 3 labs jenkins slaves.
  • 10:46 hashar: integration-slave1001.eqiad.wmflabs is out of disk space ( / /dev/vda1)
  • 07:29 springle: shutdown tantalum per mwalker request
  • 04:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 25 04:18:45 UTC 2014 (duration 18m 44s)
  • 03:31 logmsgbot: LocalisationUpdate completed (1.24wmf15) at 2014-07-25 03:30:33+00:00
  • 02:48 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-25 02:47:17+00:00
  • 01:21 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ic29ae11fa: On Labs, disable LuaSandbox's profiling feature to isolate bug 68413 (duration: 00m 04s)
  • 00:15 mutante: imported jouncebot from github - https://gerrit.wikimedia.org/r/#/q/project:wikimedia/bots/jouncebot,n,z
  • 00:03 K4-713: updated fundraising civicrm to 0639c11636d9

July 24

  • 23:26 mutante: created gerrit project for jouncebot
  • 23:06 logmsgbot: maxsem Synchronized wmf-config: https://gerrit.wikimedia.org/r/149180 (duration: 00m 05s)
  • 21:53 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 20:41 mutante: rebooted wm-bot instance
  • 20:21 bblack: restarted backend varnish for parsoid on cp1058
  • 20:20 bblack: restarted backend varnish for parsoid on cp1045
  • 20:08 logmsgbot: reedy Synchronized php-1.24wmf15/extensions/Translate: (no message) (duration: 00m 15s)
  • 20:00 logmsgbot: reedy Synchronized php-1.24wmf15: (no message) (duration: 00m 59s)
  • 19:58 logmsgbot: reedy Synchronized php-1.24wmf14: (no message) (duration: 01m 11s)
  • 19:24 hashar: restarted Zuul
  • 18:44 ori: restarted jobrunners for 01c70b1a892ac3944655f84449e89e4508894101
  • 18:41 AaronSchulz: Updated jobrunners to 01c70b1a892ac3944655f84449e89e4508894101
  • 18:39 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 18:34 logmsgbot: aaron Synchronized php-1.24wmf14/includes/jobqueue/aggregator/JobQueueAggregatorRedis.php: ca031131396ee1830e239d0b6a314bb571840c11 (duration: 00m 06s)
  • 18:26 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 14s)
  • 18:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf15
  • 18:23 ori: Purged apache from SSL cluster; provisioned as a side-effect of I0b02a46f3 + I76a0d237f
  • 18:21 godog: updated swift ring to bring ms-be1013 weight to 2300
  • 18:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf14
  • 18:03 logmsgbot: reedy Finished scap: testwiki to 1.24wmf15 and build l10n cache (duration: 31m 12s)
  • 17:32 logmsgbot: reedy Started scap: testwiki to 1.24wmf15 and build l10n cache
  • 16:38 hashar: restarting Jenkins it is broken again
  • 16:10 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Swap wikimania2013wiki to wikimania2014wiki in wmgCentralAuthLoginIcon (duration: 00m 14s)
  • 15:55 bd808|deploy: Fetched de8022b to /a/common on tin; prod no-op change needed for beta
  • 15:40 bd808|deploy: Fetched c7ae85e to /a/common on tin; prod no-op needed for beta
  • 15:39 ottomata: temporarily stopping puppet on analytics1027
  • 15:14 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings-labs.php: Fix reference thumbnail settings syntax (duration: 00m 13s)
  • 15:13 cmjohnson1: swapping disk 8 es1001
  • 15:10 hashar: Clearing out old Zuul references on operations/puppet.git might cause merge errors
  • 15:10 logmsgbot: yurik Synchronized php-1.24wmf14/extensions/ZeroBanner: (no message) (duration: 01m 07s)
  • 15:08 logmsgbot: yurik Synchronized php-1.24wmf13/extensions/ZeroBanner: (no message) (duration: 01m 11s)
  • 14:30 logmsgbot: yurik Synchronized wmf-config/mobile.php: Font for zero banner (duration: 01m 10s)
  • 13:38 hashar: Deleting old Zuul references in the Zuul maintained repository /srv/ssd/zuul/git/mediawiki/core/ on gallium bug 68481 . Should speed up merge operations on that repository.
  • 10:10 hashar: Zuul code being installed on lanthanum.eqiad.wmnet Will let us use a merger daemon there and the Zuul cloner client. 141758
  • 05:44 springle: labsdb1002 work in progress; it may misbehave. see labs-l for updates
  • 03:57 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 24 03:56:32 UTC 2014 (duration 56m 31s)
  • 03:09 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-24 03:08:05+00:00
  • 02:37 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-24 02:36:35+00:00
  • 00:44 ori: installing linux-tools on mw1053 to run perf on jobrunner

July 23

  • 23:59 logmsgbot: maxsem Finished scap: Pick up messages forgotten during Zero deployment (duration: 26m 42s)
  • 23:39 ori: running sync-common on mw1053.eqiad.wmnet
  • 23:32 logmsgbot: maxsem Started scap: Pick up messages forgotten during Zero deployment
  • 23:26 logmsgbot: maxsem Synchronized php-1.24wmf14/extensions/MultimediaViewer/: (no message) (duration: 00m 03s)
  • 23:26 logmsgbot: maxsem Synchronized php-1.24wmf14/extensions/VisualEditor/: (no message) (duration: 00m 04s)
  • 23:19 logmsgbot: maxsem Synchronized php-1.24wmf13/extensions/MobileFrontend/: (no message) (duration: 00m 04s)
  • 23:18 logmsgbot: maxsem Synchronized php-1.24wmf14/extensions/MobileFrontend/: (no message) (duration: 00m 04s)
  • 22:39 mutante: removed platinum from icinga
  • 22:36 _joe_: installed mw1053 as the first hhvm jobrunner, currently stopped. Puppet disabled so that it won't restart the jobrunner automatically
  • 21:49 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I2f366fa93: Use luastandalone on HHVM (duration: 00m 03s)
  • 21:17 hashar: Zuul is all good. It just receives too many patches :-]
  • 20:31 bd808|deploy: Updated /a/common to 07834a9 (beta cluster: use luastandalone); no sync needed
  • 20:30 subbu: deployed parsoid version 47d4bc83
  • 20:27 hashar: Having no idea how to fix zuul. Restarting it and killing the whole queue :-/
  • 20:14 mutante: contacts.wm - set $base_url in default/settings.php to https URL, and $is_https='on' in bootstrap.inc (unpuppetized?)
  • 19:49 logmsgbot: awight Synchronized php-1.24wmf14/extensions/CentralNotice: push many CentralNotice fixes, including GeoIP cookie and hide cookie (duration: 00m 05s)
  • 19:49 logmsgbot: awight Synchronized php-1.24wmf13/extensions/CentralNotice: push many CentralNotice fixes, including GeoIP cookie and hide cookie (duration: 00m 05s)
  • 19:28 logmsgbot: awight Synchronized php-1.24wmf14/extensions/CentralNotice: push many CentralNotice fixes, including GeoIP cookie and hide cookie (duration: 00m 04s)
  • 19:27 logmsgbot: awight Synchronized php-1.24wmf13/extensions/CentralNotice: push many CentralNotice fixes, including GeoIP cookie and hide cookie (duration: 00m 04s)
  • 18:57 hashar: reenabled Gearman plugin in Jenkins. Jobs have been reregistered and seems to be proceeding again
  • 18:55 hashar: back. attempting to fix jenkins
  • 18:38 logmsgbot: yurik Synchronized php-1.24wmf14/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 24s)
  • 18:36 hashar: can't fix jenkins / zuul right now. Will be stalled for at least half an hour
  • 18:35 logmsgbot: yurik Synchronized php-1.24wmf13/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 27s)
  • 18:33 hashar: Jenkins disabled and reenabled Gearman plugin. The jobs were no more registered in Zuul gearman server :-(
  • 18:32 hashar: Jenkins stalled
  • 17:45 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 07s)
  • 17:38 godog: launched a script on ms-fe1001 to collect thumb stats, no impact expected
  • 17:11 logmsgbot: awight Synchronized php-1.24wmf14/extensions/FundraisingTranslateWorkflow: automatic translate workflow fix for Fundraising/ pages on meta.wmo (duration: 00m 04s)
  • 15:38 logmsgbot: reedy Synchronized php-1.24wmf14/extensions/Wikidata: touch (duration: 00m 15s)
  • 15:34 logmsgbot: reedy Synchronized php-1.24wmf14/extensions/Wikidata: Fix css issue in entity suggester on Wikidata (duration: 00m 17s)
  • 15:19 logmsgbot: reedy Synchronized php-1.24wmf14/resources/Resources.php: Fixing forgotten OOUI messages (duration: 00m 15s)
  • 15:11 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Remove flickrApiUrl from (duration: 00m 15s)
  • 14:19 akosiaris: upgraded php5 on mw1017 (test.wikipedia.org) deployment-apache0{1,2} (beta) to 5.3.10-1ubuntu3.13+wmf1
  • 12:42 hashar: upgraded gdnsd on gallium (used to lint operations/dns.git changes)
  • 09:57 hashar: Zuul migrated to zuul user :)
  • 09:43 hashar: zuul changing file ownership on gallium for /srv/ssd/zuul/git from jenkins:root to zuul:zuul
  • 09:42 hashar: breaking zuul
  • 05:29 springle: clone mariadb 10 labsdb1002 to labsdb100[13]
  • 04:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 23 04:09:54 UTC 2014 (duration 9m 53s)
  • 03:21 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-23 03:20:45+00:00
  • 02:50 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-23 02:49:34+00:00

July 22

  • 23:37 logmsgbot: ebernhardson Finished scap: Update flow in wmf/1.24wmf14 (duration: 17m 08s)
  • 23:20 logmsgbot: ebernhardson Started scap: Update flow in wmf/1.24wmf14
  • 21:57 logmsgbot: reedy Synchronized php-1.24wmf14/extensions/WikimediaMessages/: Fix fatal for dumps (duration: 00m 15s)
  • 21:52 logmsgbot: reedy Synchronized php-1.24wmf13/extensions/WikimediaMessages/: Fix fatal for dumps (duration: 00m 13s)
  • 18:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf14 again
  • 18:45 logmsgbot: reedy Synchronized php-1.24wmf14/extensions/CirrusSearch/: Fix fatal (duration: 00m 15s)
  • 18:13 Reedy: Running sync-common on mw1081
  • 18:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias back to 1.24wmf13 due to Wikidata and Cirrus fatals
  • 18:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf14
  • 17:48 logmsgbot: mwalker Finished scap: Deploying Petition extension to the cluster (duration: 28m 27s)
  • 17:19 logmsgbot: mwalker Started scap: Deploying Petition extension to the cluster
  • 17:12 logmsgbot: aude Synchronized php-1.24wmf14/extensions/Wikidata/extensions/ValueView/lib/jquery.ui/jquery.ui.suggester.js: touch jquery.ui.suggester.js for Wikidata (duration: 00m 05s)
  • 17:06 logmsgbot: aude Synchronized php-1.24wmf14/extensions/Wikidata: Update Wikidata submodule for test wikidata, for real! (duration: 00m 06s)
  • 17:02 logmsgbot: aude Synchronized php-1.24wmf14/extensions/Wikidata/extensions/Wikibase/lib/resources/wikibase.js: touch wikibase.js for test wikidata only, fix caching issues (duration: 00m 05s)
  • 16:55 logmsgbot: aude Synchronized php-1.24wmf14/extensions/Wikidata/extensions/ValueView/lib/jquery.ui/jquery.ui.suggester.js: touch jquery.ui.suggester.js for Wikidata (duration: 00m 05s)
  • 16:48 logmsgbot: aude Synchronized php-1.24wmf14/extensions/Wikidata: Update Wikidata: js and json dump fixes (duration: 00m 11s)
  • 16:26 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: enable WikibaseClient on test wikidata (duration: 00m 07s)
  • 16:25 logmsgbot: aude Synchronized wmf-config/Wikibase.php: add settings for enabling WikibaseClient on test wikidata (duration: 00m 04s)
  • 16:18 logmsgbot: reedy Purged l10n cache for 1.24wmf12
  • 16:18 logmsgbot: reedy Purged l10n cache for 1.24wmf11
  • 16:17 logmsgbot: reedy Purged l10n cache for 1.24wmf10
  • 15:12 manybubbles: done with SWAT
  • 15:11 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - touching InitializeSettings.php to make dblist change go (duration: 00m 06s)
  • 15:10 logmsgbot: manybubbles Synchronized commonsuploads.dblist: SWAT add mrwiki to commonsuploads list (duration: 00m 08s)
  • 15:06 logmsgbot: manybubbles Synchronized php-1.24wmf14/extensions/CirrusSearch/: SWAT small cirrus fixes (duration: 00m 08s)
  • 14:48 _joe_: removed old, unused puppet 2.7 packages from reprepro for trusty
  • 14:00 _joe_: reinstalling mw1053 in 5 minutes, downtime on icinga, puppet disabled, setting to 'false' everywhere in pybal
  • 05:31 bblack: authdns servers (mexia, rubidium, eeden) updated to gdnsd-1.11.4~precise1
  • 03:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 22 03:12:10 UTC 2014 (duration 12m 9s)
  • 02:37 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-22 02:36:27+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-22 02:13:40+00:00
  • 00:56 mutante: tungsten,fluorine, search1001-1006 - upgraded libssl

July 21

  • 23:44 mutante: graceful apache on magnesium
  • 23:42 legoktm: cleaned up stalled global rename of Felipegaspars --> L'editeur
  • 23:39 logmsgbot: maxsem Synchronized php-1.24wmf14/extensions/Flow/: https://gerrit.wikimedia.org/r/#/c/148270/ - revert previous change (duration: 00m 04s)
  • 23:36 logmsgbot: maxsem Synchronized php-1.24wmf14/extensions/TimedMediaHandler/: https://gerrit.wikimedia.org/r/#/c/148241/ (duration: 00m 04s)
  • 23:32 logmsgbot: maxsem Synchronized php-1.24wmf14/extensions/Flow/: https://gerrit.wikimedia.org/r/#/c/148249/ (duration: 00m 04s)
  • 23:30 logmsgbot: maxsem Synchronized php-1.24wmf14/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/148128 (duration: 00m 06s)
  • 23:29 logmsgbot: maxsem Synchronized php-1.24wmf14/extensions/AbuseFilter/: https://gerrit.wikimedia.org/r/#/c/148027/ (duration: 00m 06s)
  • 23:27 logmsgbot: maxsem Synchronized php-1.24wmf14/resources/: https://gerrit.wikimedia.org/r/#/c/147854/ (duration: 00m 05s)
  • 22:21 mutante: installing package upgrades on bast1001
  • 21:50 RobH: shutting down ms1002 for reclaim into labstore1003
  • 21:37 ottomata: running kafka preferred-replica-election to rebalance topics
  • 21:27 hashar: beta: removed build timeout from beta-update-databases-eqiad Jenkins jobs. There is a huge schema change being processed by update.php
  • 20:09 subbu: deployed parsoid version 1c9277d6
  • 17:24 mutante: elastic1009,analytics1004,silver, various misc. boxes - upgrading libssl
  • 17:16 mutante: installing package upgrades on iron
  • 16:19 godog: restarted uwsgi on tungsten
  • 16:02 andrewbogott: updated OpenStackManager on wikitech
  • 15:48 logmsgbot: demon Synchronized wmf-config: Undeploying CommunityVoice/ClientSide extensions (duration: 00m 08s)
  • 15:30 logmsgbot: demon Synchronized wmf-config/flaggedrevs.php: ukwiki gets FR for NS_MODULE (duration: 00m 04s)
  • 15:25 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: Set $wgUploadNavigationUrl for plwikisource (duration: 00m 04s)
  • 15:25 logmsgbot: demon Synchronized php-1.24wmf14/extensions/CirrusSearch: CirrusSearch to master for 1.24wmf14 (duration: 00m 07s)
  • 15:22 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: Set $wgForceUIMsgAsContentMsg for zhwikivoyage (duration: 00m 05s)
  • 15:17 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: TemplateData for fiwiki (duration: 00m 06s)
  • 13:41 _joe_: restarted apache on palladium
  • 12:01 apergos: started /usr/local/bin/dumpwikidatajson.sh in root screen session on snapshot1003
  • 03:11 springle: restarted apache on strontium
  • 02:58 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 21 02:57:03 UTC 2014 (duration 57m 2s)
  • 02:50 logmsgbot: krinkle Synchronized wmf-config/InitialiseSettings.php: I27c6f82af5e9b (duration: 00m 06s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-21 02:25:08+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-21 02:13:42+00:00

July 20

  • 02:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 20 02:57:59 UTC 2014 (duration 57m 58s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-20 02:25:54+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-20 02:14:47+00:00
  • 01:40 ori: synced docroot/default/index.html (I005f43b96: Add width/height attributes to img to fix reflow)

July 19

  • 23:33 logmsgbot: aaron Synchronized php-1.24wmf4/maintenance: 926e1997b53f563a4e7f3c540e32b45ddb24b3c5 & 017891ba41cc72987bf3cb441004a847d20105b4 (duration: 00m 08s)
  • 23:33 logmsgbot: aaron Synchronized php-1.24wmf4/includes: 926e1997b53f563a4e7f3c540e32b45ddb24b3c5 & 017891ba41cc72987bf3cb441004a847d20105b4 (duration: 00m 09s)
  • 15:43 bblack: restarted gitblit service on antimony
  • 05:02 Krinkle: Ungracefully restarting Zuul to clear the items stuck in the queue (picked a moment with no real items waiting in the queue).
  • 03:25 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool es1001 (duration: 00m 06s)
  • 02:57 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 19 02:56:30 UTC 2014 (duration 56m 29s)
  • 02:27 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-19 02:26:26+00:00
  • 02:16 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-19 02:15:20+00:00
  • 01:09 logmsgbot: awight Synchronized php-1.24wmf14/extensions/FundraisingTranslateWorkflow: update FundraisingTranslateWorkflow submodule content (take 4) (duration: 00m 04s)
  • 01:08 logmsgbot: awight Synchronized php-1.24wmf13/extensions/FundraisingTranslateWorkflow: update FundraisingTranslateWorkflow submodule content (take 4) (duration: 00m 04s)
  • 01:07 logmsgbot: awight Synchronized php-1.24wmf12/extensions/FundraisingTranslateWorkflow: update FundraisingTranslateWorkflow submodule content (take 4) (duration: 00m 04s)
  • 00:36 logmsgbot: awight Synchronized php-1.24wmf14/extensions/FundraisingTranslateWorkflow: update FundraisingTranslateWorkflow submodule content (take 3) (duration: 00m 04s)
  • 00:36 logmsgbot: awight Synchronized php-1.24wmf13/extensions/FundraisingTranslateWorkflow: update FundraisingTranslateWorkflow submodule content (take 3) (duration: 00m 04s)
  • 00:36 logmsgbot: awight Synchronized php-1.24wmf12/extensions/FundraisingTranslateWorkflow: update FundraisingTranslateWorkflow submodule content (take 3) (duration: 00m 04s)

July 18

  • 22:04 logmsgbot: awight Synchronized php-1.24wmf14: update FundraisingTranslateWorkflow submodule (take 2) (duration: 00m 58s)
  • 22:03 logmsgbot: awight updated /a/common/php-1.24wmf14 to I1036dae02: Update mediawiki/core/vendor to head to 1.24wmf14
  • 21:00 logmsgbot: awight Synchronized php-1.24wmf13: update FundraisingTranslateWorkflow submodule (duration: 01m 04s)
  • 20:58 awight: for the record, I actually updated to ade90e0e22492d87e6069db3a359b22ef56401a6
  • 20:57 logmsgbot: awight updated /a/common/php-1.24wmf13 to Id3462554b: Made --maxtime a soft limit again
  • 20:50 logmsgbot: awight Synchronized php-1.24wmf12: update FundraisingTranslateWorkflow submodule (duration: 00m 49s)
  • 20:48 logmsgbot: awight Synchronized php-1.24wmf12: update FundraisingTranslateWorkflow submodule (duration: 00m 21s)
  • 20:48 logmsgbot: awight updated /a/common/php-1.24wmf12 to Idf3f49941: Updating ZeroBanner
  • 20:41 MaxSem: Load testing GeoData
  • 19:11 mutante: restarted apache on strontium.. sigh
  • 18:17 logmsgbot: aaron Synchronized php-1.24wmf13/maintenance/runJobs.php: ae053860dc36a07f05ab9e31299f2da0d2f66e85 (duration: 00m 03s)
  • 18:16 logmsgbot: aaron Synchronized php-1.24wmf14/maintenance/runJobs.php: 684c21c325370aa3baac631ae9a006fc8861b952 (duration: 00m 03s)
  • 18:05 logmsgbot: aaron Synchronized wmf-config/jobqueue-eqiad.php: Set "daemonized" flag for the redis job queue (duration: 00m 04s)
  • 17:33 cmjohnson: replacing disk 2 es1005
  • 17:25 mutante: temp. stopped icinga-wm to avoid channel spam
  • 17:24 mutante: puppetmaster on strontium had 'Unexpected error in mod_passenger" causing puppet fails all over the place with error 500 on master, resumed normal after graceful
  • 17:21 mutante: graceful'ed apache on strontium
  • 14:37 godog: rolling reload of proxy-server on swift ms-fe1* to pick up changes
  • 13:19 _joe_: re-enabling puppet, applying on a sample of hosts created no change according to my tests.
  • 13:13 _joe_: temporarily disabling puppet on mw servers, will re-enable when I'm done with testing (again) the change
  • 11:20 godog: restart proxy-server on ms-fe1003, as suspected it wasn't running the latest version
  • 11:14 godog: restart proxy-server on ms-fe1003, double checking for a change in numbers reported to graphite
  • 10:04 godog: stagger reload swift {account,object,container} server in ms-be.eqiad to pick up recon changes
  • 06:01 AaronSchulz: Updated /srv/deployment/jobrunner to 4cddd5033efadf431e138c399b5d86542e32f196
  • 03:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 18 03:53:55 UTC 2014 (duration 53m 54s)
  • 03:22 ori: Updated jobrunner to d9520c9 and restarted service on all jobrunners
  • 03:09 logmsgbot: LocalisationUpdate completed (1.24wmf14) at 2014-07-18 03:08:02+00:00
  • 02:45 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: Repool db1021, context RT 7916, warm up (duration: 00m 08s)
  • 02:37 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-18 02:36:54+00:00

July 17

  • 23:49 logmsgbot: mwalker Finished scap: SWAT for 146651, 147102, 146925, 147331, 147332, and 147206
  • 23:19 logmsgbot: mwalker Started scap: SWAT for 146651, 147102, 146925, 147331, 147332, and 147206
  • 21:02 csteipp: deployed fix for bug68187
  • 20:29 ori: updated jobrunner to 71d84ea18d and restarted service
  • 18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf14
  • 18:30 springle: db1021 raid write-cache failure, BBU at 9%
  • 18:14 springle: db1021 disabled sync_binlog, thread tied up on fsync
  • 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf13
  • 18:09 logmsgbot: reedy Synchronized wmf-config/: De-pool db1021 due to increasing replag (duration: 00m 14s)
  • 17:40 logmsgbot: reedy Finished scap: testwiki to 1.24wmf14 take 2 (duration: 33m 02s)
  • 17:30 Jeff_Green: payments1002 dist upgrade & reboot
  • 17:21 mutante: nickel (ganglia) apt-get upgrading packages
  • 17:13 Jeff_Green: dist-upgrade and reboot payments1003
  • 17:07 logmsgbot: reedy Started scap: testwiki to 1.24wmf14 take 2
  • 17:04 logmsgbot: reedy scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/a/common/wmf-config/extension-list" --output="/tmp/tmp.VsoJsYY6Q2" ' returned non-zero exit status 1 (duration: 02m 46s)
  • 17:03 RobH: payments4 is kernel updating (per jgreen)
  • 17:01 logmsgbot: reedy Started scap: testwiki to 1.24wmf14
  • 15:05 logmsgbot: manybubbles Synchronized php-1.24wmf13/extensions/MultimediaViewer/: SWAT - Moving repo icon back to the right-hand side in Media Viewer (duration: 00m 05s)
  • 15:03 logmsgbot: manybubbles Synchronized wmf-config/CommonSettings-labs.php: SWAT deploy to keep us synced, but this is a noop in prod. only anything in beta. (duration: 00m 05s)
  • 07:27 springle: mariadb 10 on labsdb1002:3309 cloning s5 from sanitarium db1054:3308
  • 03:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 17 03:32:25 UTC 2014 (duration 32m 24s)
  • 02:47 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-17 02:46:24+00:00
  • 02:24 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-17 02:23:08+00:00

July 16

  • 23:55 logmsgbot: maxsem Synchronized private: Clean up old mobile cruft (duration: 00m 05s)
  • 23:17 logmsgbot: yurik Synchronized php-1.24wmf13/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 04m 03s)
  • 23:13 logmsgbot: yurik Synchronized php-1.24wmf12/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 04m 14s)
  • 22:34 andrewbogott: temporarily fixed puppet on tin by restarting salt-master and salt-minion. A proper fix would involve upgrading to a salt version that fixes https://github.com/saltstack/salt/issues/6306
  • 22:29 logmsgbot: yurik Synchronized php-1.24wmf13/extensions/ZeroBanner: (no message) (duration: 03m 55s)
  • 22:27 ori: restarted jobrunner service on all job runners
  • 22:18 logmsgbot: yurik Synchronized php-1.24wmf12/extensions/ZeroBanner: (no message) (duration: 04m 31s)
  • 21:50 AaronSchulz: Updated job runners to 186b9b33
  • 21:08 legoktm: clearing Magog the Ogre's watchlist on enwp per request (173668 entries)
  • 21:01 logmsgbot: yurik Synchronized php-1.24wmf13/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 04m 53s)
  • 20:56 logmsgbot: yurik Synchronized php-1.24wmf12/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 04m 54s)
  • 20:22 subbu: deploy parsoid 060dcb54
  • 19:56 ottomata: reenabling puppet on analytics1027
  • 19:21 ottomata: temp disabling puppet on analytics1027
  • 17:57 akosiaris: clean puppet stored config database for osm-db100{1,2}.eqiad.wmnet, updating icinga
  • 16:49 Reedy: Restarted jenkins again
  • 16:12 Reedy: Restarted jenkins
  • 16:11 Reedy: Killed jenkins
  • 14:34 _joe_: moving the stale conf-enabled directory away on jobrunners, or when we upgrade to trusty all hell will break loose
  • 13:06 logmsgbot: oblivian gracefulled all apaches
  • 12:14 logmsgbot: oblivian gracefulled all apaches
  • 12:01 _joe_: removed stale files from /etc/apache2/conf-enabled on all mw hosts
  • 11:25 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: Take Cirrus as default from more wikis while we figure out load issues (duration: 00m 06s)
  • 10:32 _joe_: releasing a new apache config to all mediawikis
  • 08:54 godog: repool ms-fe1004
  • 08:51 godog: repool ms-fe1003 and depool ms-fe1004
  • 08:46 godog: repool ms-fe1002 and depool ms-fe1003
  • 08:39 godog: depool ms-fe1002 for swift upgrade
  • 05:54 springle: resuming page content model schema changes, osc_host.sh processes on terbium ok to kill in emergency
  • 04:22 springle: restarted gitblit on antimony
  • 03:04 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 16 03:03:41 UTC 2014 (duration 3m 40s)
  • 02:27 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-16 02:26:12+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-16 02:14:32+00:00
  • 01:34 manybubbles: moving shards off of elastic101[789]

July 15

  • 23:20 logmsgbot: maxsem Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/146615/ (duration: 00m 04s)
  • 23:16 logmsgbot: maxsem Synchronized php-1.24wmf12/extensions/CirrusSearch/: https://gerrit.wikimedia.org/r/#q,146471,n,z (duration: 00m 05s)
  • 23:14 logmsgbot: maxsem Synchronized php-1.24wmf13/includes/specials/SpecialVersion.php: (no message) (duration: 00m 04s)
  • 23:13 logmsgbot: maxsem Synchronized php-1.24wmf13/extensions/CirrusSearch/: https://gerrit.wikimedia.org/r/#q,146471,n,z (duration: 00m 04s)
  • 22:35 K4-713: synchronized payments to afa12be34769000bf8
  • 21:34 _joe_: disabling puppet on mw1001, tests
  • 21:26 logmsgbot: aude Synchronized php-1.24wmf13/extensions/Wikidata: Update submodule to fix entity search issue on Wikidata (duration: 00m 21s)
  • 21:15 ori: to test r146607, locally modified upstart conf for jobrunner on mw1001 to log to /var/log/mediawiki, and restarted service
  • 20:24 ori: restarted jobrunner on all jobrunners
  • 20:23 AaronSchulz: Deployed /srv/jobrunner to 31e54c564d369e89613db48977eec0a5891b6498
  • 20:21 logmsgbot: reedy Synchronized docroot and w: (no message) (duration: 00m 21s)
  • 20:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf13
  • 20:12 Krinkle: Reloading Zuul to deploy If2312bcf18bdbe8dee
  • 20:12 bd808: log volume up after logstash restart
  • 20:10 bd808: restarted logstash on logstash1001; log volume looked to be down from "normal"
  • 19:55 Reedy: Applied extensions/UploadWizard/UploadWizard.sql to rowiki (re bug 59242)
  • 18:53 manybubbles: bouncing elastic1018 to pick up new merge policy. hopefully that'll help with io thrashing
  • 17:58 ori: _joe_ deployed jobrunner to all job runners
  • 17:40 manybubbles: my last attempt to lower the concurrent traffic for recovery was a failure - tried again and succeeded. that seems to have fixed the echo service disruption from taking elastic1017 out of service
  • 17:37 ori: updated jobrunner to bef32b9120
  • 17:29 manybubbles: elastic1017 went nuts again. just shutting elasticsearch off on it for now
  • 17:17 manybubbles: lowered Elasticsearch concurrent recovery streams to 2 (from 3) and total write rate across those streams to 20MB/sec (from 4MB/sec). This should prevent io thrash on recovery which looked to cause echo distruptions in service while recovering from some other disruption.
  • 16:25 _joe_: all mw servers updated
  • 16:10 _joe_: mw1100 and onwards updated
  • 16:00 _joe_: mw1060-mw1099 updated
  • 15:57 manybubbles: restarting Elasticsearch on elastic1017 - its thrashing the disk again. I'm still not 100% sure why
  • 15:56 _joe_: mw1020-mw1059 updated
  • 15:53 _joe_: mw101[0-9] updated
  • 15:51 manybubbles: elasticsearch1017 is freaking out again - maybe there is something wrong with it. odds aren't good it picked up the same shard again after restart and that shard is somehow poison just for it and not the other two nodes with the same shard....
  • 15:47 _joe_: starting rolling update of all appservers to apache2 2.2.22-1ubuntu1.6, half of them are on 2.2.22-1ubuntu1.5 now
  • 15:42 manybubbles: setting the filter cache on one node in the cluster set it on all. yay, I guess. Anyway, I'm going to let it soak for a while.
  • 15:32 manybubbles: setting filter cache size to 20% on elastic1001 to see if it takes/helps us
  • 15:19 logmsgbot: anomie Synchronized wmf-config/: SWAT: Remove dead ULS variable gerrit:145861 (duration: 00m 10s)
  • 15:18 anomie: anomie actually committed a live hack someone left on tin (removing db1035)
  • 15:16 logmsgbot: anomie updated /a/common to I7ca6a16d5: Switch jawiki back to lsearchd
  • 13:52 manybubbles: after switching jawiki back to lsearchd by default load is mostly recovered. the cluster is still healing from bouncing elastic1017 and that'll take a while. the load will be a bit high during that but searches are coming back in a reasonably amount of time again
  • 13:42 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: jawiki back to lsearchd (duration: 00m 05s)
  • 13:38 manybubbles: elastic1017 had a load average of 60 - was thashing in io. bounced Elasticsearch. lets see if it recovers on its own
  • 09:09 _joe_: restarting mailman on sodium, again, for testing
  • 08:50 godog: restart mailman on sodium after inodes freed
  • 07:27 _joe_: restarted mailman on sodium
  • 07:22 _joe_: stopping mailman on sodium for repairing
  • 06:54 _joe_: killed jenkins stale process on gallium, stuck in a futex while shutting down
  • 04:48 springle: db1035 crash cycle. down for memtest and stuff
  • 03:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 15 03:33:38 UTC 2014 (duration 33m 37s)
  • 03:01 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-15 03:00:03+00:00
  • 02:34 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1035, crashed (duration: 00m 13s)
  • 02:30 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-15 02:29:02+00:00
  • 02:27 springle: powercycle db1035 unresponsive

July 14

  • 23:52 logmsgbot: mwalker Finished scap: Updating for SWAT 146304, 146306, 146149, 146165, 146166, 146282, and 146281. Also finishing awight's deploy of FundraisingTranslateWorkflow. (duration: 19m 42s)
  • 23:32 logmsgbot: mwalker Started scap: Updating for SWAT 146304, 146306, 146149, 146165, 146166, 146282, and 146281. Also finishing awight's deploy of FundraisingTranslateWorkflow.
  • 20:22 cscott: updated Parsoid to version d51e64097bb1b18e356584d4f3ddcfd90a6071ba
  • 19:57 ori: postponing jobrunner deployment to tomorrow; ran over time
  • 19:45 _joe_: doing the same on mw1064, segfaulted for the same reason
  • 19:44 _joe_: killed a lone apache2 child on mw1152, stuck in a futex, after a segfault of another apache process. Restarted apache, now working correctly
  • 19:03 godog: re-enabling mailman on sodium, missing list config restored
  • 18:49 logmsgbot: awight Synchronized wmf-config: Deploying FundraisingTranslateWorkflow on metawiki (t
  • 18:45 logmsgbot: awight Synchronized php-1.24wmf13/extensions/FundraisingTranslateWorkflow: Update FundraisingTranslateWorkflow extension (wmf13) (duration: 00m 05s)
  • 18:43 logmsgbot: awight Synchronized php-1.24wmf12/extensions/FundraisingTranslateWorkflow: Update FundraisingTranslateWorkflow extension (duration: 00m 05s)
  • 18:15 logmsgbot: awight Synchronized wmf-config: Revert: Deploying FundraisingTranslateWorkflow on metawiki (duration: 00m 04s)
  • 18:03 logmsgbot: awight Synchronized wmf-config: Deploying FundraisingTranslateWorkflow on metawiki (duration: 00m 05s)
  • 18:03 logmsgbot: awight updated /a/common to Ie7599fb6e: jawiki gets Cirrus as primary search
  • 17:43 Krinkle: npm-cache for integration slaves got corrupted again. Depooling/Repooling integration-slave100{1,2,3} onoe by one to clear cache and let it warm up again.
  • 17:35 Krinkle: Jenkins slaves in labs are unable to reach zuul.eqiad.wmnet
  • 17:10 andrewbogott: purging old local-* service group entries from labs ldap (via purgeOldServiceGroups.php)
  • 17:05 godog: started mailman on sodium post-reboot
  • 17:04 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: nlwiki getting cirrus as primary (duration: 00m 04s)
  • 15:11 logmsgbot: manybubbles Synchronized wmf-config: SWAT update cirrus settings for commons (duration: 00m 04s)
  • 15:04 logmsgbot: manybubbles Synchronized wmf-config: SWAT update cirrus settings for commons (duration: 00m 04s)
  • 15:01 logmsgbot: manybubbles Synchronized wmf-config: SWAT update cirrus settings for commons (duration: 00m 05s)
  • 14:39 _joe_: rebooted nescio, stuck and with console showing just a truncated log (timestamp only)
  • 14:33 mutante: powercycling sodium
  • 14:02 mutante: stat1002 - "Could not find declared class ::oozie"
  • 09:36 legoktm: ran initSiteStats.php on all wikivoyages for bug 64370
  • 09:02 godog: repool ms-fe1001 after upgrade, basic testing successful
  • 08:33 godog: depool ms-fe1001 for swift icehouse upgrade
  • 02:57 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 14 02:56:22 UTC 2014 (duration 56m 21s)
  • 02:24 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-14 02:23:39+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-14 02:12:54+00:00

July 13

  • 22:12 ori: stopping puppet on rcs1001 to debug nginx issue
  • 21:03 Krinkle: git-deploy: Deploying integration/slave-scripts I7f2b476807465
  • 02:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 13 02:53:33 UTC 2014 (duration 53m 32s)
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-13 02:23:56+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-13 02:13:32+00:00
  • 02:12 legoktm: migratePass0.php finished a while back

July 12

  • 22:21 legoktm: running foreachwiki extensions/CentralAuth/maintenance/migratePass0.php (bug 67350)
  • 22:04 legoktm: checkLocalNames/checkLocalUser finished a few hours ago, I don't have a timestamp (bug 67350)
  • 13:51 godog: reboot ms-be1007, xfs problems on sdn, load at 300+
  • 07:39 legoktm: started running checkLocalUser.php --delete=1 on all CentralAuth wikis for bug 67350
  • 07:37 legoktm: started running checkLocalNames.php --delete=1 on all CentralAuth wikis for bug 67350
  • 02:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 12 02:51:47 UTC 2014 (duration 51m 46s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-12 02:25:47+00:00
  • 02:16 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-12 02:15:33+00:00

July 11

  • 23:44 logmsgbot: awight Synchronized wmf-config: Deploying FundraisingTranslateWorkflow to labs (take 2) (duration: 00m 04s)
  • 23:36 logmsgbot: awight Synchronized wmf-config: Deploying FundraisingTranslateWorkflow to labs (duration: 00m 05s)
  • 23:34 logmsgbot: awight updated /a/common to I862a4afed: Fixup highlightTest.php
  • 22:44 mutante: upgraded libssl on wtp*
  • 22:33 Krinkle: Restarting Jenkins
  • 22:33 Krinkle: Pooled/depooled Jenkins slave on gallium
  • 22:31 Krinkle: jenkins/gallium's weekly w(h)ine hour is here.
  • 21:31 Krinkle: Reloading Zuul to deploy config change I993eba5ab7b70f924a2b925fea7c196db27c4cc3
  • 20:57 ottomata: disabling puppet on analytics1004 (AGH!)
  • 20:51 ottomata: bringing up some hadoop journalnodes (and datanodes)
  • 20:33 mutante: wikitech - graceful apache for ssl cipher list change
  • 18:19 mutante: OTRS - enabled STS, updated SSL cipher list, restarted Apache on iodine
  • 15:15 logmsgbot: hoo Synchronized php-1.24wmf13/extensions/Wikidata/: Fix the wbsearchentities API (duration: 00m 13s)
  • 15:14 logmsgbot: hoo Synchronized php-1.24wmf12/extensions/Wikidata/: Fix the wbsearchentities API (duration: 00m 16s)
  • 13:52 hashar: Jenkins: mediawiki/core change being queued while Jenkins is busy proceeding some history. That is normal, will resume soon ™
  • 12:07 hashar: Jenkins: dropping history of mwext-Wikibase-testextensions-master as well
  • 12:05 hashar_: Jenkins: manually removing history of mwext-Wikibase-client-tests and mwext-Wikibase-repo-tests . They are no more used since January
  • 08:54 hoo: Started rebuildItemsPerSite for wikidatawiki on terbium
  • 03:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 11 03:30:11 UTC 2014 (duration 30m 10s)
  • 03:01 logmsgbot: LocalisationUpdate completed (1.24wmf13) at 2014-07-11 03:00:20+00:00
  • 02:31 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-11 02:30:24+00:00

July 10

  • 23:38 logmsgbot: mwalker Finished scap: Updating Core, VE, and GuidedTour for scap, 145400, 145401, 145431, and 145460 (duration: 16m 26s)
  • 23:22 logmsgbot: mwalker Started scap: Updating Core, VE, and GuidedTour for scap, 145400, 145401, 145431, and 145460
  • 20:00 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 19:46 logmsgbot: reedy Synchronized private: (no message) (duration: 00m 14s)
  • 19:45 csteipp: deployed patch for bug65778
  • 19:43 hashar: Jenkins upgrading Gearman plugin from 0.0.6 to 0.0.7 . That fix the way jobs labels are registered with Gearman
  • 19:16 hashar: Killed jenkins :-(
  • 18:37 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 13s)
  • 18:36 logmsgbot: reedy Synchronized database lists: (no message) (duration: 00m 14s)
  • 18:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf13
  • 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf12
  • 17:25 logmsgbot: hoo Synchronized php-1.24wmf13/extensions/Wikidata/: Fix a UI issue and two API related flaws (same version as for wmf12) (duration: 00m 09s)
  • 17:21 logmsgbot: hoo Synchronized php-1.24wmf12/extensions/Wikidata/: Fix a UI issue and two API related flaws (duration: 00m 14s)
  • 16:04 godog: restarted pdns in turn on virt1000 and virt0 after opendj ulimit change
  • 15:56 hashar: gallium running a rather long du command in a screen. Need to have a good figure at how much disk space each jobs consume
  • 15:50 logmsgbot: reedy Finished scap: testwiki to 1.24wmf13 and build l10n cache (duration: 32m 09s)
  • 15:18 logmsgbot: reedy Started scap: testwiki to 1.24wmf13 and build l10n cache
  • 15:15 ottomata: reinstalling analytics1026 and analytics1027
  • 14:10 godog: ran swift-dispersion-populate on eqiad and esams swift clusters
  • 14:04 godog: cycle-restarting swift proxy-server on ms-fe to apply config updates
  • 13:09 godog: restart pdns on virt1000
  • 12:48 springle: ongoing schema changes: pl_from_namespace gerrit 117373. on terbium, osc_host.sh processes ok to kill in emergency
  • 12:43 godog: restart opendj on virt1000 with higher ulimit -n
  • 12:29 godog: restarted opendj on virt1000, ran out of fd
  • 10:29 godog: restart profiler-to-carbon on tungsten, seemingly cpu spinning
  • 09:48 logmsgbot: reedy Synchronized database lists: (no message) (duration: 00m 13s)
  • 09:30 logmsgbot: oblivian gracefulled all apaches
  • 09:23 _joe_: doing a tagged run to sync apache config
  • 09:07 hashar: gallium err was July 5th and file was from a minute ago ... ignore me
  • 09:06 hashar: gallium deleted /var/lib/puppet/state/agent_catalog_run.lock from July 5th. Was preventing me to run puppet agent -tv
  • 08:02 logmsgbot: oblivian gracefulled all apaches
  • 07:52 _joe_: doing a tagged run of puppet on all appservers to sync apache config
  • 06:40 bblack: all normally-ulsfo traffic is back on ulsfo
  • 05:53 awight: edit CRM Drupal permissions
  • 03:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 10 03:46:36 UTC 2014 (duration 46m 35s)
  • 03:12 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-10 03:11:48+00:00
  • 02:49 mutante: argon,netmon1001, graceful'led apaches
  • 02:48 mutante: netmon1001 - DocumentRoot [/etc/apache2/undef] does not exist
  • 02:42 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-07-10 02:41:29+00:00
  • 02:38 mutante: argon,calcium,iron,rhenium,bast1001,oxygen,netmon1001 - upgraded SSL
  • 01:47 mutante: argon - Ignoring file 'puppet_base_2.7' in directory '/etc/apt/preferences.d/
  • 01:41 awight: update crm schema to wmf_civicrm 7020
  • 01:40 awight: update civicrm from 108802336e4d5f4aab9a6dbfa0ea434bddae0060 to 15cf86cb109a448f1982da9c91215eec73f28499
  • 01:38 mutante: potassium,hydrogen,search1016,nitrogen,analytics1024,chromium - upgrade SSL
  • 01:06 bblack: cleared icinga downtimes for ulsfo (we now have some traffic back there)
  • 00:50 logmsgbot: mattflaschen Synchronized php-1.24wmf11/extensions/GuidedTour/: GuidedTour cherry-pick to 1.24wmf11 in support of GettingStarted anonymous editor acquisition test (duration: 00m 09s)
  • 00:05 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/144938 (duration: 00m 04s)

July 9

  • 23:57 logmsgbot: maxsem Finished scap: SWAT, GettingStarted introduced a new message (duration: 26m 31s)
  • 23:44 mutante: deleted systemusers group on neon & mw1077 (to check it doesnt break anything
  • 23:31 logmsgbot: maxsem Started scap: SWAT, GettingStarted introduced a new message
  • 23:22 logmsgbot: maxsem Synchronized php-1.24wmf11/extensions/GettingStarted/: (no message) (duration: 00m 03s)
  • 23:17 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/144857/ (duration: 00m 04s)
  • 22:42 mark: Enabling PAIX BGP sessions on cr2-ulsfo
  • 22:40 mark: Enabling WMF HQ BGP sessions on cr1-ulsfo
  • 22:38 mark: Enabling TiNet transit links on cr1-ulsfo
  • 22:35 mark: Enabling WMF HQ BGP sessions on cr2-ulsfo
  • 22:34 mark: Enabling NTT and HE transit links on cr2-ulsfo
  • 22:05 mutante: restarted apache on zirconium for config change
  • 20:07 subbu: deployed parsoid 1632288d
  • 18:36 logmsgbot: yurik Synchronized php-1.24wmf12/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 22s)
  • 18:29 logmsgbot: yurik Synchronized php-1.24wmf11/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 24s)
  • 17:17 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: eswiki cirrus (duration: 00m 04s)
  • 16:51 logmsgbot: csteipp Finished scap: Update CentralAuth for Global Rename (duration: 28m 46s)
  • 16:22 logmsgbot: csteipp Started scap: Update CentralAuth for Global Rename
  • 16:17 mark: ulsfo is now offline
  • 16:16 mark: Shutdown NTT BGP sessions on cr2-ulsfo
  • 16:13 mark: Shutdown TiNet BGP sessions on cr1-ulsfo
  • 16:10 mark: Shutdown IXP BGP sessions on cr2-ulsfo
  • 16:10 mark: Shutdown WMF HQ BGP sessions on cr2-ulsfo
  • 16:09 mark: Shutdown WMF HQ BGP sessions on cr1-ulsfo
  • 16:02 logmsgbot: hoo Synchronized php-1.24wmf12/extensions/Wikidata/: Update Wikibase to fix a fatal and various JS things (duration: 00m 14s)
  • 14:13 hashar: Jenkins: bringing back puppet-compiler02.eqiad.wmflabs node online. /tmp get filled when running huge catalog compilations which causes Jenkins to unpool the node :/
  • 13:30 godog: reboot ms-be1005, raid controller confused (?) after disk replacement
  • 12:52 godog: umounted sdg1 on ms-be1005, device disappeared, errors in dmesg
  • 12:35 bblack: enabled amssq47 text frontend cache in pybal for esams
  • 09:39 hashar: Jenkins had a bit of failure earlier due to the massive configuration update of mediawiki-core and mwext jobs. If that fails again the best thing is to stop Jenkins on gallium , wait for it to be killed or force kill -9 the java process then restart Jenkins. Should sort it out
  • 09:30 hashar: restarted Zuul to clear out stalled items in queue
  • 09:12 hashar: Jenkins being slow because the mediawiki-core* jobs history cache has been wiped out while updating their configuration. Jenkins is busy processing the history :(
  • 09:02 hashar: Jenkins killing slave process on lanthanum. Some job is stalled and unrecoverable.
  • 08:53 godog: upgrade ms-be1013/1014/1015 (zone5) to icehouse swift
  • 08:51 hashar: Jenkins migrating jobs to use $ZUUL_URL instead of git://zuul.eqiad.wmnet Preparing to scale out Zuul merger to several nodes
  • 08:19 godog: upgrade ms-be1009/1010/1011 (zone4) to swift icehouse
  • 08:04 hashar: Jenkins: granted matanya the ability to manually trigger builds. Use case: the puppet compiler!
  • 08:02 godog: upgrade ms-be1005/1006/1007 (zone3) to swift icehouse
  • 03:37 mutante: ran puppet on neon - false puppet failure alarms
  • 02:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 9 02:54:37 UTC 2014 (duration 54m 36s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-09 02:25:33+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-07-09 02:14:38+00:00
  • 01:26 mutante: Bugzilla - enabled https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
  • 00:50 mutante: restarted gitblit service

July 8

  • 22:50 mutante: radon (phab)- package and kernel upgrades, rebooting
  • 20:22 legoktm: finished running migrateAccount.php --attachbroken --attachmissing (bug 61876)
  • 20:07 legoktm: finished migrateAccount.php --safe, now starting migrateAccount.php --attachbroken
  • 20:05 mutante: restarted apache on ytterbium
  • 19:47 K4-713: updated payments fraud filters again
  • 19:47 legoktm: running migrateAccount.php --safe for accounts only existing on one wiki (bug 39817)
  • 19:27 mutante: this should have fixed all the services behind misc. varnish now getting an actual "A" rating on ssllabs
  • 19:20 mutante: arr, i meant "nginx", not varnish
  • 19:15 mutante: restarting varnish on cp1043/cp1044 (misc cluster)
  • 18:55 cmjohnson1: disconnecting serial cable from psw1-c2-eqiad
  • 18:50 csteipp: patch for bug66608 deployed to wmf11/12
  • 18:50 K4-713: updated fraud filters on payments cluster
  • 18:28 logmsgbot: reedy Synchronized robots-private.txt: (no message) (duration: 00m 14s)
  • 18:27 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 15s)
  • 18:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.24wmf12
  • 15:22 logmsgbot: reedy Purged l10n cache for 1.24wmf10
  • 15:21 logmsgbot: reedy Purged l10n cache for 1.24wmf9
  • 15:15 logmsgbot: anomie Synchronized php-1.24wmf11/extensions/Scribunto/: SWAT: Fix regression in os.date and os.time at module scope gerrit:144559 (duration: 00m 10s)
  • 15:14 logmsgbot: anomie Synchronized php-1.24wmf12/extensions/Scribunto/: SWAT: Fix regression in os.date and os.time at module scope gerrit:144511 (duration: 00m 11s)
  • 15:10 logmsgbot: anomie Synchronized php-1.24wmf11/extensions/UploadWizard/UploadWizard.config.php: SWAT: Flickr API is https-only now gerrit:144584 (duration: 00m 10s)
  • 15:04 logmsgbot: anomie Synchronized php-1.24wmf12/extensions/UploadWizard/UploadWizard.config.php: SWAT: Flickr API is https-only now gerrit:144583 (duration: 00m 10s)
  • 13:34 springle: slow transaction rollback in progress on db1001 librenms. other databases not affected, but librenms writes are timing out
  • 13:32 cmjohnson1: replacing disk disk 6 ms-be1005
  • 13:30 cmjohnson1: replacing disk 4 ms-be1007
  • 12:38 YuviPanda: disregard previous log message, was meant for labs
  • 12:37 YuviPanda: graphite reduced metrics count from 65k to 25k, monitoring io performance
  • 06:57 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: raise db traffic samplers to normal load (duration: 00m 06s)
  • 05:10 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1010, warm up (duration: 00m 06s)
  • 04:51 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1010 for upgrade (duration: 00m 06s)
  • 03:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 8 03:25:51 UTC 2014 (duration 25m 50s)
  • 03:00 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-08 02:59:33+00:00
  • 02:30 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-07-08 02:29:00+00:00
  • 01:17 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1061, warm up (duration: 00m 06s)
  • 00:57 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1061 for upgrade (duration: 00m 07s)
  • 00:02 ^d: gerrit upgraded to 2.8.1-4-ga1048ce from 2.8.1-2-g724b796, back up. Might be slow for a bit while caches warm.

July 7

  • 23:38 logmsgbot: maxsem Synchronized php-1.24wmf12/extensions/ParserFunctions/: https://gerrit.wikimedia.org/r/#q,144510,n,z (duration: 00m 03s)
  • 23:38 logmsgbot: maxsem Synchronized php-1.24wmf12/includes/StubObject.php: https://gerrit.wikimedia.org/r/#/c/144509/ (duration: 00m 03s)
  • 23:22 logmsgbot: maxsem Synchronized visualeditor-default.dblist: (no message) (duration: 00m 03s)
  • 23:19 logmsgbot: maxsem Synchronized php-1.24wmf12/extensions/GWToolset: (no message) (duration: 00m 03s)
  • 23:18 logmsgbot: maxsem Synchronized php-1.24wmf11/extensions/GWToolset: (no message) (duration: 00m 03s)
  • 23:17 logmsgbot: maxsem Synchronized php-1.24wmf12/extensions/GWToolset: (no message) (duration: 00m 04s)
  • 23:12 logmsgbot: maxsem Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#q,144476,n,z & https://gerrit.wikimedia.org/r/#q,139569,n,z (duration: 00m 05s)
  • 23:04 logmsgbot: maxsem Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#q,144476,n,z & https://gerrit.wikimedia.org/r/#q,139569,n,z (duration: 00m 03s)
  • 22:41 logmsgbot: ori Synchronized wmf-config/mc.php: I8b66e9339: Make app servers connect to nutcracker on port 11212 (duration: 00m 03s)
  • 20:31 logmsgbot: ori Synchronized wmf-config/mc.php: Iea24b092b: Make mw1041 connect to nutcracker on port 11212 (duration: 00m 09s)
  • 20:03 subbu: deployed Parsoid 8ef7b6fe
  • 17:52 legoktm: deleted rows in centralauth's localnames and localuser tables for bug 67548
  • 17:02 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: Cirrus on commons as primary (duration: 00m 04s)
  • 16:34 logmsgbot: aude Finished scap: Update Wikidata to mw1.24-wmf12 branch for group0 wikis (duration: 22m 33s)
  • 16:22 manybubbles: (Cirrus) load tested commons and eswiki over the last hour - both look fine.
  • 16:11 logmsgbot: aude Started scap: Update Wikidata to mw1.24-wmf12 branch for group0 wikis
  • 15:49 bd808: Logstash event volume looks better after restart. Probably related to bug 63490.
  • 15:32 bd808: Restarted logstash on logstash1001 because log volume looked lower than I though it should be.
  • 15:16 cmjohnson1: reseating PEM2 cr1-eqiad
  • 15:08 godog: powercycled ms-be1007, unresponsive on console and remnants of a stack trace
  • 14:49 manybubbles: (Cirrus) Applying cache warmer configuration that went out last Thursday to all wikipedias.
  • 12:11 hashar: Jenkins job builder e1ddd23 fails for us :/ Moving back to parent commit
  • 12:09 hashar: Updated our Jenkins job builder fork 0972985..e1ddd23
  • 09:40 godog: upgrade ms-be1003/1004/1012 (zone2) to swift icehouse
  • 09:16 _joe_: restarting rhenium, pings but no ssh since 2 days, serial console is blank and unresponsive
  • 09:15 godog: upgrade ms-be1002/1008 (zone1) to swift icehouse
  • 02:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 7 02:52:10 UTC 2014 (duration 52m 9s)
  • 02:24 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-07 02:23:48+00:00
  • 02:13 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-07-07 02:12:44+00:00

July 6

  • 02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 6 02:49:21 UTC 2014 (duration 49m 20s)
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-06 02:24:08+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-07-06 02:13:07+00:00

July 5

  • 02:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 5 02:52:04 UTC 2014 (duration 52m 3s)
  • 02:27 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-05 02:26:08+00:00
  • 02:16 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-07-05 02:15:05+00:00
  • 01:22 springle: ongoing osc_host.sh schema change jobs on terbium. fine to kill in an emergency

July 4

  • 20:05 hoo: Ran sync-common on fenari to update the docs on noc.wikimedia.org
  • 15:40 _joe_: restarting salt-minion, killing io hungry job on fenari running since jun 30, 00 AM
  • 12:28 akosiaris: executed dist-upgrade on virt1000. Keystone configure phase failed in keystone-manage db-sync and hence dpkg configure failed. It was trying to create an already existing index in the database. Dropped the index, ran dpkg --configure -a to recreate the index (and whatever else keystone-manage db_sync does). All is back to normal.
  • 03:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 4 03:28:29 UTC 2014 (duration 28m 28s)
  • 03:03 logmsgbot: LocalisationUpdate completed (1.24wmf12) at 2014-07-04 03:02:49+00:00
  • 02:33 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-07-04 02:32:29+00:00
  • 00:28 gwicke: deployed parsoid config change e21a534 to support VE on the OTRS wiki

July 3

  • 23:40 mutante: osmium - libboost-dev : Depends: libboost1.54-dev but it is not going to be installed
  • 23:33 mutante: rhenium (pmacct / flow) Out of memory: Kill process 3123 (pmacctd) score 1 or sacrifice child
  • 23:22 K4-713: updated payments to c5689f385b2f0a7bdc55c5752010e9eb
  • 23:17 logmsgbot: mwalker Synchronized php-1.24wmf12/extensions/VisualEditor/: Updating VisualEditor for 144081 (duration: 00m 12s)
  • 21:07 logmsgbot: oblivian gracefulled all apaches
  • 20:45 mutante: deleted analytics/kraken branch from ops/puppet via gerrit ui, ack'ed by ottomata
  • 20:12 bd808|deploy: Updated scap to ff04431 (restart-nutcracker script)
  • 19:53 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 14s)
  • 19:48 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 19:48 logmsgbot: reedy Synchronized docroot and w: (no message) (duration: 00m 30s)
  • 19:47 logmsgbot: reedy Synchronized database lists: (no message) (duration: 00m 20s)
  • 19:36 jgage: rebooting analytics1012 for bios change: cpufreq governor
  • 19:27 ottomata: disabling puppet on hadoop related analytics nodes, preparing for reinstall
  • 19:21 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf12
  • 19:12 logmsgbot: reedy Synchronized php-1.24wmf11/languages/Language.php: I039547b867b2eab47692dcc018c95b89975bc65d (duration: 00m 40s)
  • 18:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedia to 1.24wmf11
  • 18:41 logmsgbot: reedy Finished scap: testwiki to 1.24wmf12 and build l10n cache (duration: 30m 47s)
  • 18:18 ottomata: doing rolling restarts of zookeeper servers and kafka brokers to load up new zk timeout changes
  • 18:10 logmsgbot: reedy Started scap: testwiki to 1.24wmf12 and build l10n cache
  • 18:10 logmsgbot: reedy scap aborted: testwiki to 1.24wmf12 and build l10n cache (duration: 27m 26s)
  • 17:53 godog: reloading librenms, semi-broke it with a syslog search (again)
  • 17:46 godog: reloading librenms, semi-broke it with a syslog search
  • 17:42 logmsgbot: reedy Started scap: testwiki to 1.24wmf12 and build l10n cache
  • 16:38 logmsgbot: maxsem Synchronized php-1.24wmf11/extensions/EventLogging/: bug 67420 (duration: 00m 35s)
  • 16:34 paravoid: apt: uploading nutcracker backport for precise
  • 08:07 hashar: Jenkins restarted
  • 08:00 hashar: upgrading Jenkins (minor version bump 1.554.2 -> 1.554.3)
  • 03:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 3 03:38:46 UTC 2014 (duration 38m 45s)
  • 03:03 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-07-03 03:02:15+00:00
  • 02:32 logmsgbot: LocalisationUpdate completed (1.24wmf10) at 2014-07-03 02:31:35+00:00

July 2

  • 23:06 logmsgbot: maxsem Synchronized wmf-config/: (no message) (duration: 00m 07s)
  • 22:37 jgage: rebooting analytics1021 to change bios "system profile" from PPW (OS) to PPW (DAPC)
  • 22:19 logmsgbot: ebernhardson Finished scap: (no message) (duration: 36m 25s)
  • 22:16 jgage: rebooting analytics1022 to check bios cpufreq setting
  • 21:43 logmsgbot: ebernhardson Started scap: (no message)
  • 21:42 logmsgbot: ebernhardson Synchronized php-1.24wmf10/extensions/Mantle/: Sync new Mantle extension in 1.24wmf10 (duration: 00m 20s)
  • 21:40 robh: blog updated to newest release, no downtime
  • 21:38 jgage: rebooting analytics1021 to check bios cpufreq setting
  • 20:56 paravoid: pfw1-eqiad: s/mchenry/lead/; all smtp_out rules have [ polonium lead ] as destination-address now
  • 20:49 paravoid: switching non-wikimedia.org MX to polonium/lead (from polonium/mchenry)
  • 20:16 cscott: updated Parsoid to version 6afcb8df
  • 19:08 logmsgbot: yurik Synchronized php-1.24wmf11/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal - bug fix (duration: 01m 40s)
  • 19:00 logmsgbot: yurik Synchronized php-1.24wmf10/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal - bug fix (duration: 01m 15s)
  • 18:42 logmsgbot: yurik Synchronized php-1.24wmf10/extensions/: Reverting previous update to JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 20s)
  • 18:37 logmsgbot: yurik Synchronized php-1.24wmf10/extensions/: Updating JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 55s)
  • 18:33 logmsgbot: yurik Synchronized php-1.24wmf11/extensions/: Updating JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 38s)
  • 18:33 paravoid: reprepro include, trusty-wikimedia (main/universe): nutcracker, libicu 4.8, libzip 0.11, hhvm, {php,hhvm}-wikidiff2, {php,hhvm}-fss, {php,hhvm}-luasandbox, ffmpeg2theora
  • 18:28 yurikR2: yurik ^ was a noop - comment fix
  • 18:28 logmsgbot: yurik Synchronized wmf-config/CommonSettings.php: (no message) (duration: 01m 04s)
  • 18:26 logmsgbot: yurik Synchronized docroot/bits/WikipediaMobileFirefoxOS/: (no message) (duration: 01m 03s)
  • 16:30 mutante: upgrading jenkins to jenkins_1.554.3_all.deb on the apt repo
  • 15:19 manybubbles: done with SWAT for real this time
  • 15:17 logmsgbot: manybubbles Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 28s)
  • 15:16 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 18s)
  • 15:08 manybubbles: *SWAT* complete
  • 15:07 manybubbles: swap complete - logged off of tin
  • 15:04 logmsgbot: manybubbles Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s)
  • 15:04 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s)
  • 15:00 logmsgbot: manybubbles Synchronized wmf-config: SWAT Remove two permissions from some editors on ruwiki (duration: 00m 07s)
  • 13:10 hashar: Jenkins being busy deleting history files
  • 13:02 hashar: Jenkins: dropping history of puppet related jobs after 90 days. 136992
  • 12:18 akosiaris: upgraded PH5 to 5.3.10-1ubuntu3.12+wmf1 on deployment-apache01 and deployment-apache02 (beta)
  • 12:09 akosiaris: upgraded PHP5 to 5.3.10-1ubuntu3.12+wmf1 on test.wikipedia.org
  • 11:05 logmsgbot: hashar Synchronized wmf-config/InitialiseSettings.php: additional upload domain for Erasmus University 143593 bug 67355 (duration: 00m 06s)
  • 08:00 godog: upgrading ms-be1001 to swift icehouse
  • 07:45 godog: umounted (empty and broken) sdk1 from ms-be3003 and wipe its first sectors, no more remounts
  • 03:00 paravoid: rebooting lead
  • 02:57 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 2 02:56:33 UTC 2014 (duration 56m 32s)
  • 02:27 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-07-02 02:26:24+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf10) at 2014-07-02 02:14:53+00:00

July 1

  • 23:49 K4-713: updated payments cluster to c5689f385b2f0a7
  • 23:43 robh: any francium errors can be ignored, as the software doesn't fully deploy from puppet and its not in service
  • 23:36 logmsgbot: maxsem Synchronized wmf-config/FeaturedFeedsWMF.php: https://gerrit.wikimedia.org/r/#/c/136316/ now for realz (duration: 00m 04s)
  • 23:34 logmsgbot: maxsem Synchronized wmf-config/FeaturedFeedsWMF.php: https://gerrit.wikimedia.org/r/#/c/136316/ (duration: 00m 04s)
  • 23:09 logmsgbot: maxsem Synchronized php-1.24wmf11/extensions/CentralAuth/: https://gerrit.wikimedia.org/r/#/c/143473/ (duration: 00m 05s)
  • 23:05 logmsgbot: maxsem Synchronized php-1.24wmf11/resources/: https://gerrit.wikimedia.org/r/#/c/142975/ (duration: 00m 05s)
  • 23:04 logmsgbot: maxsem Synchronized php-1.24wmf10/resources/: https://gerrit.wikimedia.org/r/#/c/142975/ (duration: 00m 19s)
  • 21:39 hoo: Set email for re-renamed dewiki account "Kolimak". Email and password got lost during a screwed rename.
  • 20:36 logmsgbot: reedy Synchronized php-1.24wmf11/extensions/WikimediaMessages: bug 67387 (duration: 00m 15s)
  • 20:31 mutante: restarting apache on mw1217
  • 20:27 manybubbles: Adding cache warmers to all Cirrus indexes for group1 wikis with more then one shard except commons (commons is busy, it'll have to wait:)
  • 19:53 logmsgbot: aude Synchronized wmf-config/Wikibase.php: adjust property suggester setting for wikidata (duration: 00m 11s)
  • 19:14 logmsgbot: ori Synchronized php-1.24wmf10/resources/src/jquery.ui-themes/vector/jquery.ui.core.css: Ib09928248: vector/jquery.ui.core.css: Update rule for .ui-helper-hidden-accessible (bug 67243) (duration: 00m 05s)
  • 19:14 logmsgbot: ori Synchronized php-1.24wmf11/resources/src/jquery.ui-themes/vector/jquery.ui.core.css: Ib09928248: vector/jquery.ui.core.css: Update rule for .ui-helper-hidden-accessible (bug 67243) (duration: 00m 06s)
  • 18:42 andrewbogott: adding virt1008 to labs compute pool
  • 18:41 andrewbogott: switching puppet canary from virt1008 to virt1009
  • 18:38 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable property suggester on Wikidata (duration: 00m 10s)
  • 18:38 logmsgbot: aude Synchronized wmf-config/Wikibase.php: (no message) (duration: 00m 15s)
  • 18:30 logmsgbot: aaron Synchronized wmf-config/PrivateSettings.php: removed obsolete swift tampa config (duration: 00m 07s)
  • 18:15 logmsgbot: reedy Synchronized docroot and w: (no message) (duration: 00m 18s)
  • 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf11
  • 17:54 logmsgbot: demon Synchronized php-1.24wmf10/extensions/Elastica: Updating to master, fixes fatal error (duration: 00m 07s)
  • 17:45 manybubbles: rebuilding cirrus index for commons to put it into fewer shards - it should be faster this way
  • 17:24 mutante: antimony: git.wikimedia.org]: Ensure set to :present but file type is link so no content will be synced
  • 17:24 logmsgbot: hoo Synchronized wmf-config/: Typos typos typso (duration: 00m 08s)
  • 17:21 mutante: restarting apache on antimony
  • 17:21 mutante: fixing svn.wikimedia.org apache site manually
  • 17:08 springle: restarted mysqld on db1046 m2 slave
  • 17:03 logmsgbot: demon Synchronized cirrus.dblist: Move remaining pool 4 lsearchd wikis (except commons) to Cirrus (duration: 00m 07s)
  • 15:09 manybubbles: done with SWAT deploy
  • 15:06 logmsgbot: manybubbles Synchronized php-1.24wmf11/extensions/CirrusSearch/: SWAT code to set up cache warmers (duration: 00m 05s)
  • 15:04 logmsgbot: manybubbles Synchronized wmf-config: SWAT - cirrus settings - cache warmers and shard counts (duration: 00m 06s)
  • 15:04 ottomata: temporarily disabling puppet on hafnium to test an eventlogging alert
  • 14:27 hashar: Stopping Jenkins it has some corrupted threads
  • 13:16 Jeff_Green: dist-upgrade and reboot tellurium
  • 13:08 Jeff_Green: dist-upgrade and reboot boron
  • 12:23 logmsgbot: reedy Synchronized multiversion/: (no message) (duration: 00m 23s)
  • 12:22 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 18s)
  • 12:16 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 20s)
  • 12:03 logmsgbot: reedy Synchronized wmf-config/interwiki.cdb: (no message) (duration: 00m 13s)
  • 12:00 Reedy: Manually created Echo tables on extension1
  • 11:55 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 13s)
  • 11:55 logmsgbot: reedy Synchronized database lists: (no message) (duration: 00m 28s)
  • 11:53 Reedy: Manually created wikimania2015wiki database on 10.64.16.18
  • 11:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
  • 11:48 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 14s)
  • 11:47 logmsgbot: reedy Synchronized database lists: (no message) (duration: 00m 16s)
  • 10:49 _joe_: nginx restarted on all ulsfo hosts as well, we should be PFS-enabled now
  • 10:38 _joe_: esams restart finished, moving to ulsfo
  • 10:30 _joe_: all eqiad SSL terminators are now PFS enabled. Moving to rolling restarting esams
  • 10:09 _joe_: restarting nginx on ssl100* servers in sequence, to activate PFS
  • 08:47 godog: ms-be3003 sdk1 disk to 0 weight
  • 07:22 legoktm: finished running checkLocalNames.php and checkLocalUser.php for some wikivoyages to clean up bug 66535
  • 07:16 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1067 (duration: 00m 12s)
  • 07:06 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1067 during schema changes (duration: 00m 06s)
  • 06:58 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1063 (duration: 00m 06s)
  • 06:40 legoktm: starting to run checkLocalNames.php and checkLocalUser.php for some wikivoyages to clean up bug 66535
  • 06:34 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1063 during schema changes (duration: 00m 06s)
  • 06:31 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1060 (duration: 00m 06s)
  • 06:29 legoktm: ran fixInvalidStudent.php --wiki=enwiki --courseId=359 for bug 66624
  • 06:13 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1060 during schema changes (duration: 00m 07s)
  • 02:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 1 02:50:05 UTC 2014 (duration 50m 4s)
  • 02:24 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-07-01 02:23:49+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf10) at 2014-07-01 02:14:11+00:00

June 30

  • 23:03 awight: update tools from e894f1f77674b6b101ae0e1644e363ca52e319d9 to d605bdc2aaaef2d4b296a4d9567ed2831db86756
  • 23:02 logmsgbot: ori Synchronized wmf-config: Iba41a37a1: Keep thumbnail guessing enabled (duration: 00m 05s)
  • 22:14 mutante: re-enabled puppet on caesium
  • 21:43 mutante: disabling puppet on caesium
  • 21:35 Reedy: running mwscript updateSpecialPages.php --wiki=enwiki --only=Mostlinkedtemplates --override on terbium
  • 21:25 mutante: fixing releases.wikimedia.org Apache site, delete sites-enabled file broken by puppet, add symlink, graceful
  • 21:00 subbu: deployed parsoid 0b365d516
  • 19:44 _joe_: restarting pybal on lvs1005
  • 19:16 awight: updated payments from a04e536b6923f2228bb7f5fbf2caeed64a888742 to 2b6c527617dcde154cc298dd9697c9d57c9f3620
  • 18:41 awight: updated payments from a8138fefd940ba41812e5c07ca6bc74b63cb9bcf to a04e536b6923f2228bb7f5fbf2caeed64a888742
  • 17:38 manybubbles: Cirrus reindex update! all wikipedias finished their in place reindex except ruwiki - that one is running now. all group1 wikis finished their from mediawiki reindex except commons and mgwiktionary which are running now. started from mediawiki reindex of all wikipedias exception for enwiki, itwiki, and cawiki which are already long done.
  • 17:12 logmsgbot: manybubbles Synchronized cirrus.dblist: Enabled CirrusSearch as the default search backend on 30 more wikis - take five (duration: 00m 04s)
  • 17:08 logmsgbot: manybubbles Synchronized wmf-config/: Enable CirrusSearch as the default search backend on 30 more wikis - take four (duration: 00m 04s)
  • 17:08 logmsgbot: manybubbles Synchronized wmf-config/: Enable CirrusSearch as the default search backend on 30 more wikis - for real for real (duration: 00m 04s)
  • 17:07 logmsgbot: manybubbles Synchronized wmf-config/: Enable CirrusSearch as the default search backend on 30 more wikis - for real (duration: 00m 04s)
  • 17:05 logmsgbot: manybubbles Synchronized wmf-config/: Enable CirrusSearch as the default search backend on 30 more wikis (duration: 00m 05s)
  • 15:43 logmsgbot: manybubbles Synchronized php-1.24wmf11/extensions/Wikidata/: (no message) (duration: 00m 09s)
  • 15:35 logmsgbot: manybubbles Synchronized php-1.24wmf11/extensions/VisualEditor/: SWAT Correctly VisualEditor - update full size in MediaSizeWidget (duration: 00m 07s)
  • 15:26 logmsgbot: manybubbles Synchronized wmf-config/: SWAT - disable local uploads on Malay Wiktionary (duration: 00m 04s)
  • 15:23 logmsgbot: manybubbles Synchronized wmf-config/: SWAT - remove completed mediaviewer surveys (duration: 00m 04s)
  • 15:19 _joe_: restarted profiler-to-carbon, stuck since _9_ days, will see that my patch gets deployed.
  • 15:15 logmsgbot: manybubbles Synchronized php-1.24wmf10/extensions/ProofreadPage: SWAT - fix ProofreadPage number of pages (duration: 00m 09s)
  • 14:48 godog: installed new swift ring on esams, decrease ms-be3003/sdk1 weight
  • 14:41 hoo: Cleared out a watchlist with 126652 entries on warwiki to resolve https://bugzilla.wikimedia.org/show_bug.cgi?id=67123
  • 13:31 godog: upgrade ms-fe300[12] to swift icehouse
  • 10:20 hashar: restarting zuul after a puppet change for /etc/zuul/zuul.conf
  • 07:53 godog: upgrading ms-be300[2-4] to swift icehouse
  • 02:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 30 02:48:28 UTC 2014 (duration 48m 27s)
  • 02:28 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1061, warm up (duration: 00m 07s)
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-06-30 02:23:56+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf10) at 2014-06-30 02:14:23+00:00
  • 01:41 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1061 during schema changes (duration: 00m 07s)

June 29

  • 22:26 hoo: Manually cleared a watchlist on shwikt with 819846 entries, see https://bugzilla.wikimedia.org/show_bug.cgi?id=67123#c7
  • 22:10 hoo: Manually cleared a watchlist with 289436 entries, see https://bugzilla.wikimedia.org/show_bug.cgi?id=67123#c5
  • 16:44 hoo: Jenkins/ Zuul not reacting for at least half an hour now
  • 16:43 awight: update tools from 3a35482ab1fede2ccfcc49a64ec661b0cb013b81 to e894f1f77674b6b101ae0e1644e363ca52e319d9
  • 16:09 awight: updated payments from 6d74002f2634f41f7038daa7357ff6de55ee4880 to a8138fefd940ba41812e5c07ca6bc74b63cb9bcf
  • 02:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 29 02:44:35 UTC 2014 (duration 44m 34s)
  • 02:22 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-06-29 02:21:01+00:00

June 28

  • 17:16 ori: restarted lucene on search1016 per _joe_
  • 12:58 manybubbles: Cirrus reindex status: enwiki has almost finished its in place reindex, alphabetical wikipedias are at frwiki, all group1 wikis have finished their in place reindex. all group1 wikis are running from mediawiki reindex. itwiki and cawiki both finished both the in place and from mediawik reindex. Haven't started alphabetical from mediawiki reindex yet for wikipedias. that is the only
  • 10:40 _joe_: restarting lucene on search1015, stuck. again.
  • 02:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 28 02:46:49 UTC 2014 (duration 46m 48s)
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-06-28 02:24:12+00:00
  • 02:16 logmsgbot: LocalisationUpdate completed (1.24wmf10) at 2014-06-28 02:15:38+00:00

June 27

  • 23:15 awight: deploymed payments config
  • 22:57 logmsgbot: csteipp Synchronized php-1.24wmf11/extensions/OAuth/frontend/specialpages/SpecialMWOAuth.php: Fix OAuth Logins for wmf11 (duration: 00m 18s)
  • 20:57 awight: updated crm from 340c43a15a84a9392ad5ef9fc2782243ff140deb to 17439326ca4488ece843a263fc14859b38cff0e9
  • 19:33 hashar: puppet-compiler: removed modules/varnish at root@puppet-compiler02:/opt/wmf/software/compare-puppet-catalogs/external/puppet and resetted repo.
  • 19:07 awight: update crm from e2fe03a9cd51e30206d9a1114d62dfbd6960816b to 340c43a15a84a9392ad5ef9fc2782243ff140deb
  • 18:57 logmsgbot: aaron Synchronized wmf-config/PoolCounterSettings-eqiad.php: Pre-set FileRenderExpensive config
  • 18:34 bblack: updated puppet repo on virt0
  • 18:11 mutante: osmium - hhvm : Depends: libdouble-conversion1 but it is not going to be installed
  • 16:49 bblack: updated carbon repo varnish pkg to 3.0.5plus~x-wm6
  • 14:18 hashar: Updated our Jenkins Job Builder fork: e9db73d..0972985
  • 03:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 27 03:30:00 UTC 2014 (duration 29m 59s)
  • 03:06 logmsgbot: LocalisationUpdate completed (1.24wmf11) at 2014-06-27 03:05:31+00:00
  • 02:36 logmsgbot: LocalisationUpdate completed (1.24wmf10) at 2014-06-27 02:35:20+00:00

June 26

  • 23:32 manybubbles: Cirrus rebuild progress - started large/high cirrus visibility wikis in group2 - enwiki, cawiki, and itwiki.
  • 23:31 manybubbles: Cirrus rebuild progress - alphabetical wikis in group2 are 2/3 of the way done with reindex - from mediawiki rebuild is maybe 20% done there
  • 23:31 manybubbles: Cirrus rebuild progress - big wikis in group1 are finished with in place reindex and well into from mediawiki rebuild.
  • 23:27 ori: Previous scap included I2cfcfaf06 as well
  • 23:23 logmsgbot: ori Finished scap: CirrusSearch updates: Iefe340729, Ie12418e54, Ie21fb352 (duration: 04m 59s)
  • 23:18 logmsgbot: ori Started scap: CirrusSearch updates: Iefe340729, Ie12418e54, Ie21fb352
  • 23:07 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: Ie96265c4f: Add an Erasmus University domain to whitelist (duration: 00m 05s)
  • 23:07 logmsgbot: ori updated /a/common to Ie96265c4f: Add an Erasmus University domain to whitelist
  • 21:51 hashar: Zuul/Jenkins back up and operational.
  • 21:43 hashar: hardkilled Zuul :-( 6 events lost.
  • 21:38 hashar: restarting Zuul it has a bunch of stalled changes
  • 21:32 bblack: enabled cp301[78] frontends in pybal
  • 21:27 hashar: restarting Jenkins
  • 21:26 hashar: Zuul/Jenkins stalled apparently
  • 20:59 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable property suggester on testwikidata (duration: 00m 07s)
  • 20:58 logmsgbot: reedy Synchronized php-1.24wmf11/extensions/WikimediaMessages/: (no message) (duration: 00m 15s)
  • 20:57 logmsgbot: reedy Synchronized php-1.24wmf11/extensions/OAuth/: (no message) (duration: 00m 15s)
  • 20:48 logmsgbot: aude Finished scap: Update Wikidata, for enabling property suggester on testwikidata (duration: 31m 57s)
  • 20:16 logmsgbot: aude Started scap: Update Wikidata, for enabling property suggester on testwikidata
  • 19:18 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 19:14 logmsgbot: reedy Synchronized php-1.24wmf11/extensions/OAuth/: (no message) (duration: 00m 14s)
  • 19:06 RobH: blog is back online after a number of reboots due to raid rebuild issues
  • 18:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf11
  • 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf10
  • 18:15 logmsgbot: reedy Synchronized php-1.24wmf10/includes/api/ApiQueryRecentChanges.php: Id9c316733896a27ce3f6c3e0e5efdf62f7d1ff1b (duration: 00m 14s)
  • 18:08 ottomata: starting new elasticsearch nodes 1017,1018,1019
  • 18:04 RobH: aware of holmium issue (old varnish), in process of repair, blog is down
  • 17:05 logmsgbot: reedy Synchronized php-1.24wmf11/resources/Resources.php: I1237909d7e058137d55e5de9fa4d64fe1f7f9472 (duration: 00m 14s)
  • 17:04 logmsgbot: reedy Finished scap: l10nupdate for 1.24wmf11 for Skins I9395b0e1983122b12bedf003d6398da5ddfd5651 (duration: 16m 35s)
  • 16:48 logmsgbot: reedy Started scap: l10nupdate for 1.24wmf11 for Skins I9395b0e1983122b12bedf003d6398da5ddfd5651
  • 16:46 logmsgbot: reedy Purged l10n cache for 1.24wmf4
  • 16:45 logmsgbot: reedy Purged l10n cache for 1.24wmf5
  • 16:45 logmsgbot: reedy Purged l10n cache for 1.24wmf6
  • 16:44 logmsgbot: reedy Purged l10n cache for 1.24wmf7
  • 16:44 logmsgbot: reedy Purged l10n cache for 1.24wmf8
  • 16:32 logmsgbot: reedy Finished scap: testwiki to 1.24wmf11 and build l10n cache (duration: 27m 20s)
  • 16:05 logmsgbot: reedy Started scap: testwiki to 1.24wmf11 and build l10n cache
  • 16:01 logmsgbot: reedy scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/a/common/wmf-config/extension-list" --output="/tmp/tmp.XetXfk5RPi" ' returned non-zero exit status 1 (duration: 00m 18s)
  • 16:00 logmsgbot: reedy Started scap: testwiki to 1.24wmf11 and build l10n cache
  • 15:56 logmsgbot: reedy scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/a/common/wmf-config/extension-list" --output="/tmp/tmp.9SaYNRzegr" ' returned non-zero exit status 1 (duration: 00m 24s)
  • 15:55 logmsgbot: reedy Started scap: testwiki to 1.24wmf11 and build l10n cache
  • 15:55 logmsgbot: reedy scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/a/common/wmf-config/extension-list" --output="/tmp/tmp.EjEynr9oww" ' returned non-zero exit status 1 (duration: 00m 55s)
  • 15:54 logmsgbot: reedy Started scap: testwiki to 1.24wmf11 and build l10n cache
  • 15:24 cmjohnson1: shutting down holmium to replace disk
  • 14:35 bblack: restarted nova-network on labnet1001
  • 14:26 hashar: updated zuul cloner in git repo and deployed zuul ( tag is wmf-deploy-20140626-1 )
  • 13:54 godog: remounted (broken) sdk1 on ms-be3003
  • 13:32 cmjohnson1: powering down dataset1001 -relocating to 10G rack
  • 13:26 logmsgbot: reedy Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/142142/ No-op for labs (duration: 00m 16s)
  • 12:55 hashar: Jenkins: updates jobs for extensions (phpunit and qunit) to use the mw-run-update-script.sh instead of update.php . That runs update.php twice, the first time logging sql to a file that can be archived. 141851
  • 12:48 mark: Deactivated BGP session to AS13030
  • 11:01 hashar: Replacing operations-puppet-validate job with operations-puppet-pplint-HEAD which is faster and can run concurrently on multiple boxes. 142223
  • 10:52 godog: stopping swift on ms-be3003
  • 10:12 godog: upgrading ms-be3001 to swift icehouse
  • 06:26 springle: ran operations/software maintain-replicas.pl and fedtables.pl on labsdbs for bug 59683
  • 05:54 Tim: on mw1014: reformatted the /tmp partition
  • 05:50 Tim: on mw1014: stopped job runner due to bad /tmp
  • 04:44 ori: mw1014 is sad, has filesystem issues: "Attempt to read block from filesystem resulted in short read while trying to open /tmp". Puppet can't run. Should be depooled.
  • 03:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 26 03:33:19 UTC 2014 (duration 33m 18s)
  • 03:02 logmsgbot: LocalisationUpdate completed (1.24wmf10) at 2014-06-26 03:01:43+00:00
  • 02:32 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-26 02:31:50+00:00

June 25

  • 23:43 awight: updated crm from f3389daa94e9ad924175bdf0d5bc09c4a26aeb8c to e2fe03a9cd51e30206d9a1114d62dfbd6960816b
  • 23:27 logmsgbot: catrope Finished scap: Updating Wikidata and TimedMediaHandler (duration: 04m 24s)
  • 23:23 logmsgbot: catrope Started scap: Updating Wikidata and TimedMediaHandler
  • 21:22 hashar: puppet fixed on gallium / lanthanum . It was missing a group definition. All fixed! Thanks Chase.
  • 20:53 hashar: puppet broken on gallium.wikimedia.org and lanthanum.eqiad.wmnet . That is being looked at.
  • 20:34 subbu: deployed parsoid 4ef9d6be
  • 19:38 manybubbles: restarted Cirrus scripts after incident - the index rebuilds had to be completely restarted - sanity checking was simply paused
  • 18:54 logmsgbot: yurik Synchronized wmf-config/PrivateSettings.php: Removed obsolete ZRMA user/pswd (duration: 01m 06s)
  • 18:46 logmsgbot: yurik Finished scap: Removing ZeroRatedMobileAccess ext settings, depl latest JsonConfig/ZeroBanner/Portal (duration: 29m 09s)
  • 18:17 logmsgbot: yurik Started scap: Removing ZeroRatedMobileAccess ext settings, depl latest JsonConfig/ZeroBanner/Portal
  • 17:41 logmsgbot: demon Synchronized wmf-config/: Cirrus back on for wikis that had it before. Back to square 1 (duration: 00m 04s)
  • 17:29 mwalker: updating fundraising tools from 5f3a7316b636c0723ce3fa353186d4041b662872 to cdc4b73bd59d27c8d386b6df629b1c574cfed85f
  • 17:06 manybubbles: success!
  • 17:06 logmsgbot: manybubbles Synchronized wmf-config/: try to fix cirrus (duration: 00m 04s)
  • 16:51 andrewbogott: restarted apache on palladium -- _that_ helped
  • 16:49 andrewbogott: it didn't help
  • 16:49 andrewbogott: restarting puppetmaster on palladium
  • 16:42 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: Disable Cirrus everywhere but testwiki (duration: 00m 04s)
  • 16:23 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: Roll back previous Cirrus deploy (duration: 00m 05s)
  • 16:23 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: Roll back previous Cirrus deploy (duration: 00m 04s)
  • 16:16 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: I4c54357a: most remaining wikis getting Cirrus as primary (duration: 00m 04s)
  • 16:16 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: I4c54357a: most remaining wikis getting Cirrus as primary (duration: 00m 04s)
  • 15:27 logmsgbot: manybubbles Synchronized php-1.24wmf9/extensions/Wikidata/: SWAT - fix rtl issue (duration: 00m 10s)
  • 15:23 ottomata: reinstalling elastic1017,1018,1019
  • 15:20 logmsgbot: manybubbles Synchronized php-1.24wmf10/extensions/Wikidata/: SWAT - fix rtl issue (duration: 00m 12s)
  • 14:10 Krinkle: Upgrade npm from v1.4.5 to v1.4.16 on integration-slave1001 and integration-slave1002
  • 14:10 Krinkle: Upgraded npm from v1.4.13 to v1.4.16 on integration-slave1003 to fix https://github.com/npm/npm/issues/5472 and repooling
  • 13:30 Krinkle: Depooling integration-slave1003 as almost every other -npm build on this node fails due to corrupted ~/.npm cache
  • 12:52 manybubbles: cirrus rebuild update: starting from mediawiki reindex step for all alphabetical wikis that have finished so far
  • 12:48 manybubbles: cirrus rebuild update: started rebuilding group1's indexes yesterday. commons and wikidata finished their in place pass and started their from mediawiki pass. The remaining wikis are running their in place pass in alphabetical order and currently on frwiktionary.
  • 12:25 hashar: Upgraded Zuul 9839edb..b7fc126 Brings patchset 20 of Zuul cloner ( https://review.openstack.org/#/c/70373/ )
  • 12:02 akosiaris: upgraded etherpad.wikimedia.org to etherpad-lite 1.4.0
  • 11:12 paravoid: switching inbound email for wikimedia.org to polonium/mchenry
  • 10:35 _joe_: restarted lucene on search1016 as it was stuck there as well, once search1015 is up and running
  • 10:06 _joe_: restarted lucene on search1015, it was stuck
  • 07:37 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: incremental LB bump on db1009 and db1021 traffic samplers (duration: 00m 07s)
  • 06:31 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1021 with traffic sampling (duration: 00m 09s)
  • 06:01 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1021, db1049 to normal load (duration: 00m 07s)
  • 05:05 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1049, warm up (duration: 00m 08s)
  • 02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 25 02:48:53 UTC 2014 (duration 48m 52s)
  • 02:39 springle: xtrabackup clone db1005 to db1049
  • 02:27 logmsgbot: LocalisationUpdate completed (1.24wmf10) at 2014-06-25 02:25:57+00:00
  • 02:19 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1049 (duration: 00m 11s)
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-25 02:13:27+00:00
  • 00:57 chasemp: added dns for wikimania 2015 (gerrit 140186)

June 24

  • 23:28 ori: apache-graceful-all was for Ifc9596cc7
  • 23:28 logmsgbot: ori gracefulled all apaches
  • 23:12 logmsgbot: maxsem Synchronized visualeditor.dblist: https://gerrit.wikimedia.org/r/141702 (duration: 00m 03s)
  • 23:11 logmsgbot: maxsem Synchronized php-1.24wmf10/extensions/MultimediaViewer: (no message) (duration: 00m 05s)
  • 23:02 ori: apache graceful done by me for I543efda24, I29b34689e, and I1c269433e
  • 23:00 logmsgbot: root gracefulled all apaches
  • 20:53 hashar: Jenkins / Zuul deploying experimental pipeline 141827
  • 20:29 RoanKattouw: Restarting Apache on mw1220, getting lots of "Unable to allocate memory for pool" errors
  • 20:29 ottomata: rebooting analytics1021
  • 20:25 ottomata: reinitializing varnish topics with replication factor of 3
  • 20:02 hashar: updated our Jenkins Job Builder copy 416ee7d..e9db73d
  • 19:58 hashar: Upgraded Zuul on gallium.wikimedia.org to install the zuul-cloner of doom. 4f9fd51..9839edb Tagged wmf-deploy-20140624-1 in our repo.
  • 19:39 manybubbles: rebuilding search index for group1 wikis after upgrade today
  • 18:27 logmsgbot: reedy Synchronized docroot and w: (no message) (duration: 00m 14s)
  • 18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.24wmf10
  • 17:52 logmsgbot: manybubbles Synchronized wmf-config: Drop Cirrus indexes to five shards on rebuild and switch all wikis to new highlighter (duration: 00m 04s)
  • 17:44 logmsgbot: aaron Synchronized wmf-config/InitialiseSettings.php: Maintenance reports limit incremental increase (duration: 00m 08s)
  • 17:37 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: reduce db1021 load (duration: 00m 10s)
  • 17:06 akosiaris: restarted hadoop yarn on analytics1013
  • 15:36 bblack: VCL compilation is now in-sync everywhere but bits caches...
  • 15:21 logmsgbot: manybubbles Synchronized php-1.24wmf9/extensions/CirrusSearch/: SWAT Stop Cirrus from breaking RandomRootPage (duration: 00m 06s)
  • 15:15 logmsgbot: manybubbles Synchronized php-1.24wmf10/extensions/CirrusSearch/: SWAT Stop Cirrus from breaking RandomRootPage (duration: 00m 04s)
  • 15:06 logmsgbot: manybubbles Synchronized wmf-config/: SWAT - visual editor config changes and retire some beta features (duration: 00m 04s)
  • 15:05 logmsgbot: manybubbles Synchronized visualeditor-default.dblist: SWAT - Enable VisualEditor by default on Wikimania 2014 wiki (duration: 00m 04s)
  • 15:05 logmsgbot: manybubbles Synchronized visualeditor.dblist: SWAT - Enable VisualEditor by default on Wikimania 2014 wiki (duration: 00m 06s)
  • 15:03 logmsgbot: manybubbles Synchronized php-1.24wmf10/includes/config/GlobalVarConfig.php: SWAT - GlobalVarConfig should not throw exceptions for null-valued config settings (duration: 00m 05s)
  • 14:53 logmsgbot: hoo Synchronized wmf-config/CommonSettings.php: Enable Wikibase property suggester on beta (duration: 00m 07s)
  • 14:15 hashar: Jenkins set SMTP server to wiki-mail.wikimedia.org smtp.pmtpa.wmnet got deleted
  • 14:07 hashar: Jenkins is back
  • 13:59 Krinkle: Build logs in Jenkins incorrectly render ansi color codes since it was upgraded to 0.4.0. Downgrading to 0.3.1 and restarting Jenkins.
  • 09:55 godog: removing old salt master cache on palladium, moved yesterday out of the way
  • 06:59 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1049 (duration: 00m 08s)
  • 06:23 Nemo_bis: FYI no gerrit mail since yesterday 15 UTC, https://bugzilla.wikimedia.org/show_bug.cgi?id=67018
  • 02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 24 02:47:14 UTC 2014 (duration 47m 13s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf10) at 2014-06-24 02:25:43+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-24 02:13:38+00:00

June 23

  • 23:12 logmsgbot: maxsem Synchronized php-1.24wmf9/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/141102/ (duration: 00m 06s)
  • 23:12 logmsgbot: maxsem Synchronized php-1.24wmf10/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/141102/ (duration: 00m 05s)
  • 23:03 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/140897/ (duration: 00m 04s)
  • 20:05 subbu: deployed parsoid 392435a2 (deploy sha db94f88c)
  • 19:22 hashar: gallium / zuul : deleting /var/lib/zuul/git old Zuul repositories. They have been migrated to /srv/ssd/zuul/git/ ages ago
  • 19:20 jgage: ms-be3003 full root partition fixed, swift had written to /srv/swift-storage/sdk1 onto root due to umounted sdk1
  • 17:38 bblack: lvs1005:eth3 was negotiated to 100mbps (???) - disable -> enable on switch fixed it
  • 17:36 godog: restarted salt-master on palladium, suspected job cleanup stuck
  • 17:04 bd808: Fixed dangling symlink for /etc/apache2/sites-enabled/logstash.wikimedia.org on logstash1001 by deleting symlink and forcing puppet run
  • 16:49 godog: added mw1149-52 back to pybal apache
  • 16:33 paravoid: switched inbound mail for all non-wikimedia.org domains from mchenry/sodium to polonium/mchenry (~16:00 + <= 1h TTL UTC)
  • 15:13 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Add a Library of Congress domain to wgCopyUploadsDomains gerrit:141308 (duration: 00m 14s)
  • 15:11 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Adjust group rights on ruwiki gerrit:140910 (duration: 00m 14s)
  • 15:10 logmsgbot: anomie Synchronized php-1.24wmf9/includes/api/ApiExpandTemplates.php: SWAT: Fix fatal in API action=expandtemplates with Scribunto gerrit:141416 (duration: 00m 15s)
  • 15:04 logmsgbot: anomie Synchronized php-1.24wmf10/includes/api/ApiExpandTemplates.php: SWAT: Fix fatal in API action=expandtemplates with Scribunto gerrit:141417 (duration: 00m 14s)
  • 14:55 andrewbogott: reenabling puppet on labstore1001, hoping it doesn't break labs
  • 14:38 hashar: Further upgraded Zuul up to upstream b8c24ce + our local hacks. Git tag is wmf-deploy-20140623-4
  • 14:14 hashar: upgraded Zuul by one commit (that introduces swift supports though disabled it on our setup via a custom hack)
  • 13:20 paravoid: switching outbound email to polonium
  • 12:17 manybubbles: rebuilding Cirrus index on group0 wikis to pick up changes like results boosting from categories and wikitext search
  • 10:37 godog: powering down maerlant, decom-med
  • 10:05 godog: hardreset maerlant, stuck on console and no ssh
  • 09:40 paravoid: killing sodium's lighttpd compress cache
  • 07:21 _joe_: powercycled cp4018, stuck with a blank console
  • 02:59 springle: moving lighttpd compressed archives on sodium off / to regain inodes
  • 02:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 23 02:45:24 UTC 2014 (duration 45m 23s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf10) at 2014-06-23 02:25:53+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-23 02:13:53+00:00
  • 00:38 legoktm: mail is stuck, lots of mails queued in exim

June 22

  • 22:25 _joe_: restarted apache on strontium, passenger crashed (again).
  • 21:06 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings-labs.php: For cluster consistency... (duration: 00m 08s)
  • 19:24 godog: silenced LVS healthcheck on rendering.svc until 23:23 UTC
  • 02:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 22 02:41:30 UTC 2014 (duration 41m 29s)
  • 02:24 logmsgbot: LocalisationUpdate completed (1.24wmf10) at 2014-06-22 02:23:50+00:00
  • 02:13 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-22 02:12:50+00:00

June 21

  • 16:12 _joe_: restarted ms-be1012, see http://paste.debian.net/106247/ for console output
  • 02:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 21 02:45:17 UTC 2014 (duration 45m 16s)
  • 02:30 logmsgbot: LocalisationUpdate completed (1.24wmf10) at 2014-06-21 02:28:59+00:00
  • 02:18 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-21 02:17:18+00:00

June 20

  • 22:58 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: Load nostalgia from skins rather than extensions when it exists (duration: 00m 04s)
  • 20:23 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: wfGetIP removal, code cleanup (duration: 00m 04s)
  • 20:22 logmsgbot: demon Synchronized wmf-config/throttle.php: wfGetIP removal, code cleanup (duration: 00m 05s)
  • 17:11 godog: expanded palladium's root to avoid filling up, suspected salt-master (RT #7721)
  • 16:53 bd808: Ran /usr/local/bin/sync-common on fenari to verify fix for bug 66844. It works!
  • 15:16 logmsgbot: reedy Synchronized wmf-config/extension-list-labs: (no message) (duration: 00m 16s)
  • 11:00 _joe_: restarted apache on palladium, passenger was dead and filling error logs
  • 03:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 20 03:34:06 UTC 2014 (duration 34m 5s)
  • 03:19 logmsgbot: LocalisationUpdate completed (1.24wmf10) at 2014-06-20 03:18:36+00:00
  • 02:35 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-20 02:34:14+00:00
  • 00:06 MaxSem: Running clearMessageBlobs.php

June 19

  • 23:52 MaxSem: that was a touch
  • 23:51 logmsgbot: maxsem Synchronized php-1.24wmf9/extensions/MultimediaViewer/: (no message) (duration: 00m 04s)
  • 23:38 logmsgbot: maxsem Finished scap: Mark Traceur made me do it! (duration: 15m 14s)
  • 23:23 logmsgbot: maxsem Started scap: Mark Traceur made me do it!
  • 23:20 logmsgbot: maxsem Synchronized php-1.24wmf10/extensions/MobileFrontend/: (no message) (duration: 00m 03s)
  • 23:19 logmsgbot: maxsem Synchronized php-1.24wmf9/extensions/MobileFrontend/: (no message) (duration: 00m 04s)
  • 23:18 logmsgbot: maxsem Synchronized php-1.24wmf10/extensions/CirrusSearch/: (no message) (duration: 00m 03s)
  • 23:18 logmsgbot: maxsem Synchronized php-1.24wmf10/extensions/VisualEditor/: (no message) (duration: 00m 04s)
  • 23:16 logmsgbot: maxsem Synchronized php-1.24wmf9/extensions/VisualEditor/: (no message) (duration: 00m 04s)
  • 23:14 bd808: Restarted logstash service on logstash1001
  • 23:06 logmsgbot: maxsem Synchronized php-1.24wmf10/extensions/MultimediaViewer/: (no message) (duration: 00m 05s)
  • 23:06 logmsgbot: maxsem Synchronized php-1.24wmf9/extensions/MultimediaViewer/: (no message) (duration: 00m 05s)
  • 22:52 bd808: Updated scap to 792a572
  • 21:21 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Set wmgMediaViewerBeta to false everywhere (duration: 00m 15s)
  • 21:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf10
  • 21:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf9 take 2
  • 19:31 mutante: started mysql on pc1002
  • 19:17 MatmaRex: <RobH> powercycled pc1002
  • 19:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias back to 1.24wmf8
  • 19:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf9
  • 19:00 logmsgbot: reedy Finished scap: scap 1.24wmf10 take 2... (duration: 22m 59s)
  • 18:37 ori: neon, logstash100x, zirconium, stat1001, netmon1001: replaced sites-enabled symlinks with their targets and forced puppet-run to clean up after Iddc778a28
  • 18:37 logmsgbot: reedy Started scap: scap 1.24wmf10 take 2...
  • 18:08 logmsgbot: reedy Started scap: testwiki to 1.24wmf10 and build l10n cache
  • 17:29 logmsgbot: hoo Synchronized php-1.24wmf9/extensions/Wikidata/: Update Wikidata to fix the entity selector (duration: 00m 09s)
  • 15:51 mutante: powercycling elastic1017 (went down and no console output)
  • 15:13 godog: removed old pmtpa swift stats from graphite
  • 15:04 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Put testwiki namespaces in the right place gerrit:140261 (duration: 00m 14s)
  • 15:04 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Put testwiki namespaces in the right place gerrit:140261 (duration: 00m 15s)
  • 15:02 logmsgbot: anomie Synchronized wmf-config/throttle.php: SWAT: Raise account creation limit for Telugu Wikipedia workshop on June 23 gerrit:140669 (duration: 00m 15s)
  • 14:30 cmjohnson1: replacing failed disk slot3 es1006
  • 13:01 _joe_: re-enable puppet on lvs1003
  • 11:26 logmsgbot: reedy Synchronized wmf-config/: touch (duration: 00m 15s)
  • 11:25 logmsgbot: reedy Synchronized commonsuploads.dblist: (no message) (duration: 00m 15s)
  • 11:00 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 14s)
  • 10:53 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 15s)
  • 10:52 logmsgbot: reedy Synchronized docroot and w: (no message) (duration: 00m 15s)
  • 10:28 Reedy: manually ran sync-common tin on fenari
  • 10:09 logmsgbot: reedy Synchronized docroot/noc: (no message) (duration: 00m 15s)
  • 10:07 logmsgbot: reedy Synchronized docroot and w: (no message) (duration: 00m 14s)
  • 10:04 logmsgbot: reedy Synchronized wmf-config/: I248fa7b98a8a0eea943c6643d1bf9c2ed36296b8 (duration: 00m 15s)
  • 03:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 19 03:33:36 UTC 2014 (duration 33m 35s)
  • 02:46 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-19 02:45:51+00:00
  • 02:24 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-19 02:23:42+00:00

June 18

  • 23:09 awight: update crm from 26460d6eaec26861661322df8e9f07a8b0519677 to f3389daa94e9ad924175bdf0d5bc09c4a26aeb8c
  • 23:05 logmsgbot: maxsem Synchronized php-1.24wmf9/extensions/VisualEditor/: https://gerrit.wikimedia.org/r/#/c/140563/ (duration: 00m 03s)
  • 23:03 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/140250/ (duration: 00m 04s)
  • 22:30 bblack: rebooting lvs1004 + lvs1005
  • 22:10 bblack: turning lvs1003 pybal back on
  • 21:52 bblack: disable pybal on lvs1003, since 1006 seems to have all its interfaces :P
  • 21:34 bblack: rebooting lvs1003 for kernel/bios stuff
  • 21:00 bblack: rebooting lvs1006 for kernel/bios stuff
  • 20:23 subbu: deployed Parsoid 88a61f81 (deploy repo sha 470a5ef2)
  • 17:39 logmsgbot: yurik Synchronized docroot/bits/WikipediaMobileFirefoxOS/: (no message) (duration: 01m 09s)
  • 17:35 logmsgbot: yurik Synchronized php-1.24wmf9/extensions/: Updating JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 14s)
  • 17:32 logmsgbot: yurik Synchronized php-1.24wmf8/extensions/: Updating JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 15s)
  • 17:26 logmsgbot: yurik Synchronized docroot/bits/WikipediaMobileFirefoxOS/: (no message) (duration: 01m 04s)
  • 17:10 RobH: magnesium back to proper function
  • 17:09 RobH: apache2ctl restart on magnesium, racktables wasn't working
  • 16:24 bblack: rebooting lvs4001 for kenerl + num_queues
  • 16:19 bblack: rebooting lvs4002 for kenerl + num_queues
  • 15:20 bblack: rebooting lvs4003 for kernel / num_queues updates
  • 15:17 bblack: rebooting lvs4004 for kernel / num_queues updates
  • 15:10 logmsgbot: anomie Synchronized php-1.24wmf9/extensions/Scribunto/engines/LuaCommon/SiteLibrary.php: SWAT: Fix Scribunto-related exceptions on testwiki gerrit:140370 (duration: 00m 14s)
  • 13:40 _joe_: restarted profiler-to-carbon, stuck (again) waiting for mwprof
  • 13:25 springle: script rt-7708.pl hitting m2-master eventlogging from terbium for RT #7708. fine to kill if necessary
  • 10:01 hashar: Updated our Jenkins job builder fork: 8cbc93a..416ee7d
  • 08:26 _joe_: disk is gone, powering down ms-be1007, opening ticket for disk replacement
  • 08:24 _joe_: stopped swift on ms-be1007, unmounting volume to check for repair
  • 06:01 springle: restarted gmetad on nickel while unbreaking the mysql graphs I broke on ganglia
  • 04:30 ori: enabled puppet on polonium (was disabled but nothing in SAL)
  • 02:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 18 02:58:22 UTC 2014 (duration 58m 21s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-18 02:25:03+00:00
  • 02:23 MaxSem: searchidx1001 outta sync - running sync-common
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-18 02:13:34+00:00
  • 02:05 Krinkle: Nevermind, graphite.wikimedia.org going down is due to overload which recovers eventually (it just has). Has become SNAFU/FIXME.
  • 02:02 Krinkle: graphite.wikimedia.org is down with HTTP 502 Bad Gateway errors
  • 01:49 ori: puppet freshness on tungsten and stat1001 can be fixed with https://gerrit.wikimedia.org/r/#/c/140269/

June 17

  • 20:19 logmsgbot: maxsem Synchronized php-1.24wmf9/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/140178/ (duration: 00m 04s)
  • 20:17 logmsgbot: maxsem Synchronized php-1.24wmf8/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/140178/ (duration: 00m 05s)
  • 20:01 logmsgbot: hoo Synchronized php-1.24wmf9/extensions/Wikidata/: Update Wikidata to fix editing site links (duration: 00m 24s)
  • 18:23 logmsgbot: reedy Synchronized docroot and w: (no message) (duration: 00m 16s)
  • 18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.24wmf9
  • 18:05 logmsgbot: demon Synchronized wmf-config/PoolCounterSettings-eqiad.php: Limit regex searches before they start landing on wikis (duration: 00m 04s)
  • 16:32 bblack: enabled amssq31-46 esams text frontend varnishes in pybal (were misconfigured; wrong domainname)
  • 15:18 logmsgbot: manybubbles Synchronized php-1.24wmf8/extensions/CirrusSearch/: SWAT - Fix Cirrus Special:Random (duration: 00m 04s)
  • 15:13 logmsgbot: manybubbles Synchronized php-1.24wmf9/extensions/CirrusSearch/: SWAT - Fix Cirrus Special:Random (duration: 00m 04s)
  • 15:02 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - lower event logging rate for mediaviewer (duration: 00m 05s)
  • 13:51 _joe_: production puppet masters upgraded to puppet 3
  • 07:12 springle: starting updateCollation on s3 frwikinews from tin
  • 07:07 logmsgbot: springle Synchronized wmf-config/InitialiseSettings.php: $wgCategoryCollation to uca-fr on frwikinews (duration: 00m 07s)
  • 03:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 17 03:19:12 UTC 2014 (duration 19m 11s)
  • 02:35 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-17 02:34:09+00:00
  • 02:23 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-17 02:22:46+00:00

June 16

  • 23:12 logmsgbot: maxsem Synchronized php-1.24wmf8/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/139562/ (duration: 00m 05s)
  • 23:11 logmsgbot: maxsem Synchronized php-1.24wmf9/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/139562/ (duration: 00m 06s)
  • 23:05 logmsgbot: maxsem Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/139888/ (duration: 00m 08s)
  • 21:30 ori: upgraded eventlogging to 3012aad
  • 20:45 ori: updated eventlogging to b4b42effc6
  • 17:36 logmsgbot: csteipp Synchronized php-1.24wmf8/extensions/EducationProgram/includes/api/ApiAddStudents.php: Bug66631 (duration: 00m 05s)
  • 17:34 logmsgbot: csteipp Synchronized php-1.24wmf9/extensions/EducationProgram/includes/api/ApiAddStudents.php: (no message) (duration: 00m 05s)
  • 15:59 godog: manually ran update-ubuntu-mirror on carbon, successful
  • 15:57 awight: updated crm from e52a4eb1bfab622f612dc84f687678fff1fdbc04 to 26460d6eaec26861661322df8e9f07a8b0519677
  • 15:30 ottomata: reinstalling analytics1018
  • 13:38 twkozlowski: _joe_ also working on recovering the list which was deleted by mistake
  • 13:37 _joe_: closed wikimedia-de-by list
  • 13:13 _joe_: removing chip-l mailing list as for bug #63877
  • 13:03 godog: restarting swift-proxy-server on ms-fe1001 to test statsd metrics
  • 10:47 godog: restarting swift-proxy-server on ms-fe3002 to test statsd metrics
  • 10:23 hoo: Touched all 1.24wmf8 extension/wikidata files and ran sync-common after that on mw1070
  • 10:18 logmsgbot: hoo Synchronized php-1.24wmf8/extensions/Wikidata/: Update Wikidata to fix a suggester bug (duration: 00m 09s)
  • 10:16 godog: restarting swift-proxy-server on ms-fe3001 to test statsd metrics
  • 10:12 logmsgbot: hoo Synchronized php-1.24wmf9/extensions/Wikidata/: Update Wikidata to fix a suggester bug (duration: 00m 13s)
  • 09:29 apergos: restarted search1015 about 15 mns ago, it's now recovered afaict, restarted search1016, it's doing index setup now
  • 03:00 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 16 02:59:43 UTC 2014 (duration 59m 42s)
  • 02:27 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-16 02:26:05+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-16 02:14:38+00:00

June 15

  • 17:44 logmsgbot: hoo Synchronized php-1.24wmf8/extensions/Wikidata/: Touched various JavaScripts (duration: 00m 09s)
  • 14:26 Reedy: Job runners were restarted on tmh100[12] and are now processing jobs
  • 14:15 godog: extended palladium root partition by +20G
  • 13:50 _joe|away: restarted mw-job-runner on tmh1001
  • 10:02 paravoid: nuked ms-be1001 sdj with zeros, reformatting and placing into production again
  • 02:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 15 02:58:21 UTC 2014 (duration 58m 20s)
  • 02:27 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-15 02:26:03+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-15 02:14:46+00:00

June 14

  • 22:27 bawolff: video scalers seem to have stopped doing webVideoTranscode jobs
  • 20:24 legoktm: ran "delete from ep_students where student_user_id =0 limit 1;" on enwiki for bug 66624
  • 20:10 legoktm: ran "delete from ep_users_per_course where upc_user_id=0 limit 1" on enwiki for bug 66624
  • 19:19 paravoid: unmounting ms-be1001's sdj1, corrupted filesystem
  • 18:46 paravoid: rebooting ms-be1001, XFS: Internal error XFS_WANT_CORRUPTED_RETURN, lots of processes in D
  • 03:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 14 03:07:14 UTC 2014 (duration 7m 13s)
  • 02:37 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-14 02:36:35+00:00
  • 02:36 bblack: enabled amssq43-46 frontends (esams text varnish) in pybal
  • 02:17 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-14 02:16:38+00:00
  • 00:46 bblack: enabled amssq39-42 frontends (esams text varnish) in pybal

June 13

  • 22:01 manybubbles: logstash1002 seems to be properly restoring nodes to itself. I'll monitor it for the next few minutes but I believe my work here is done.
  • 21:55 manybubbles: bouncing logstash1002 because it seems stuck. not sure why. no useful logs.
  • 21:07 bblack: turned on amssq35-38 text frontends in esams (in pybal)
  • 20:57 awight: update crm from c38296add61421f87e12cb5b4f3dd68bdf2340db to e52a4eb1bfab622f612dc84f687678fff1fdbc04
  • 20:23 bblack: turned on amssq31-34 text frontends in esams
  • 18:41 mutante: DNS update - removing manutius' public IP
  • 18:31 mutante: shutting down manutius, decom
  • 18:22 logmsgbot: ori Synchronized php-1.24wmf9/extensions/Math: I498053de4: Fix the VisualEditor parts of Math-wmf9 with a working cherry pick of I7d5e1174 (duration: 00m 08s)
  • 16:55 logmsgbot: hoo Synchronized php-1.24wmf8/extensions/Wikidata/: Update Wikidata to fix JavaScript issues (duration: 00m 09s)
  • 16:45 logmsgbot: hoo Synchronized php-1.24wmf9/extensions/Wikidata/: Update Wikidata to fix JavaScript issues (duration: 00m 10s)
  • 16:31 Reedy: Finished creating mathoid tables on all wikis
  • 16:26 Reedy: Creating mathoid tables on all wikis
  • 16:11 mutante: manutius - decom, delete salt key, puppet cert, stopped services...
  • 15:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 13 15:16:09 UTC 2014 (duration 53m 11s)
  • 14:59 logmsgbot: reedy Synchronized wmf-config/: Disable MW_MATH_SOURCE for now (duration: 00m 15s)
  • 14:46 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-13 14:45:40+00:00
  • 14:36 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-13 14:35:41+00:00
  • 13:00 bblack: moved ge-3/0/0 - 3/0/15 from public to private vlan on cs2-esams (amssq31-46)
  • 10:02 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1049 (duration: 00m 12s)
  • 09:56 paravoid: deactivating eqiad<->HE, excessive packet loss/latency
  • 09:33 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1071 (duration: 00m 07s)
  • 08:10 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1070, depool db1071 (duration: 00m 12s)
  • 07:48 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1066, depool db1070 (duration: 00m 07s)
  • 07:19 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1065, depool db1066 (duration: 00m 13s)
  • 06:51 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1062, depool db1065 (duration: 00m 09s)
  • 06:09 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1062 (duration: 00m 12s)
  • 05:58 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1051 (duration: 00m 14s)
  • 03:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 13 03:53:17 UTC 2014 (duration 53m 16s)
  • 03:12 logmsgbot: LocalisationUpdate completed (1.24wmf9) at 2014-06-13 03:11:28+00:00
  • 02:35 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-13 02:34:41+00:00
  • 00:45 logmsgbot: ori Synchronized php-1.24wmf8/extensions/Math: Reverting Extension:Math to 1bb3bfa3b5656 (duration: 00m 05s)
  • 00:44 logmsgbot: ori Synchronized php-1.24wmf9/extensions/Math: Reverting Extension:Math to 1bb3bfa3b5656 (duration: 00m 06s)
  • 00:41 ori: removed Physikerwelt and Frédéric Wang from extension-Math group in Gerrit pending further inquiry into recent changes
  • 00:38 logmsgbot: ori Finished scap: fix any lingering inconsistencies in the state of the app servers (see https://gerrit.wikimedia.org/r/139089) (duration: 26m 59s)
  • 00:11 logmsgbot: ori Started scap: fix any lingering inconsistencies in the state of the app servers (see https://gerrit.wikimedia.org/r/139089)

June 12

  • 23:35 logmsgbot: ori Synchronized php-1.24wmf8/extensions/MobileFrontend: Re-syncing after submodule update (duration: 00m 06s)
  • 23:34 ori: ran sync-common on mw1151
  • 23:17 logmsgbot: catrope Synchronized php-1.24wmf9/extensions/MobileFrontend: (no message) (duration: 00m 04s)
  • 23:17 logmsgbot: catrope Synchronized php-1.24wmf8/extensions/MobileFrontend: (no message) (duration: 00m 05s)
  • 23:17 logmsgbot: catrope Synchronized php-1.24wmf8/extensions/VisualEditor: (no message) (duration: 00m 04s)
  • 23:07 Krinkle: integration-slave1003 is failing npm-test builds due to a cache corruption (filed as https://github.com/npm/npm/issues/5472). Manually cleared /mnt/home/jenkins-deploy/.npm/async on integration-slave1003.eqiad.wmflabs for now.
  • 23:05 MaxSem: Purging PageImages data from Wikibooks and Wikisource
  • 22:59 logmsgbot: catrope Synchronized wmf-config/: (no message) (duration: 00m 04s)
  • 22:46 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: disable MW_MATH_MATHML until mathoid table is created (BUG 66492) (duration: 00m 04s)
  • 22:31 logmsgbot: ori Synchronized php-1.24wmf8/extensions/WikimediaEvents: Update WikimediaEvents for Ibd36da416 (duration: 00m 03s)
  • 22:30 logmsgbot: ori Synchronized php-1.24wmf9/extensions/WikimediaEvents: Update WikimediaEvents for Ibd36da416 (duration: 00m 03s)
  • 21:11 logmsgbot: yurik Synchronized php-1.24wmf9/extensions/JsonConfig/: JsonConfig ext update, fixing bug 66555 (duration: 01m 03s)
  • 21:10 logmsgbot: yurik Synchronized php-1.24wmf8/extensions/JsonConfig/: JsonConfig ext update, fixing bug 66555 (duration: 01m 04s)
  • 19:25 ottomata: stopping puppet on an18
  • 19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf9
  • 19:19 ottomata: starting hadoop decom of analytics1018. This node will become a Kafka broker
  • 19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.24wmf8
  • 19:04 MaxSem: Dropping old GeoData tables from everywhere
  • 18:52 logmsgbot: reedy Finished scap: 1.24wmf9 staging take 2... (duration: 15m 20s)
  • 18:37 logmsgbot: reedy Started scap: 1.24wmf9 staging take 2...
  • 18:06 logmsgbot: reedy Started scap: testwiki to 1.24wmf9 and build l10n cache
  • 17:49 ottomata: disabling puppet on analytics1012 and analytics1022
  • 17:48 ottomata: starting some kafka failure tests, I have scheduled downtime for some service checks in icinga, hopefully this will not be noisy
  • 17:41 ottomata: restarting elasticsearch on logstash servers
  • 17:34 logmsgbot: yurik Synchronized wmf-config/InitialiseSettings.php: Enabling new zero ext on all wikis (duration: 01m 03s)
  • 17:22 logmsgbot: yurik Synchronized wmf-config/InitialiseSettings.php: Attempting to enable new zero ext on zerowiki & ruwiki - take3 (duration: 01m 04s)
  • 17:06 logmsgbot: yurik Synchronized php-1.24wmf8/extensions/: (no message) (duration: 01m 12s)
  • 17:05 greg-g: yurik's blank sync message could have been: Deploying new JsonConfig,ZeroBanner,ZeroPortal extensions (refactoring ZeroRatedMobileAccess ext)
  • 17:04 logmsgbot: yurik Synchronized php-1.24wmf7/extensions/: (no message) (duration: 01m 15s)
  • 15:31 logmsgbot: manybubbles Synchronized wmf-config/throttle.php: SWAT: Raise account creation limit for eswiki outreach event (duration: 00m 05s)
  • 13:39 bblack: enabling cp301[34] esams mobile frontends in pybal
  • 11:18 hashar: Gerrit: created mediawiki/services/cxserver/deploy repository for Nikerabbit and kart_
  • 05:52 paravoid: cr1-esams/cr2-knams: dismantling amslvs BGP peerings
  • 05:46 paravoid: amslvs[1234]: stopping pybal
  • 03:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 12 03:39:07 UTC 2014 (duration 39m 6s)
  • 03:03 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-12 03:02:09+00:00
  • 02:58 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1051 (duration: 01m 08s)
  • 02:33 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-06-12 02:32:07+00:00
  • 01:47 ori: graceful'd appservers for I0e66ee0a1: 2.4 compat: load mod_filter for AddOutputFilterByType
  • 00:44 bblack: ran "puppetca -s palladium.eqiad.wmnet" on palladium to get agent running again, someone borked/regenerated the key there 6 hours ago?
  • 00:20 mwalker: clearMessageBlobs.php killed because we fixed the problem in a more different way
  • 00:17 logmsgbot: mwalker Synchronized php-1.24wmf8/extensions/MultimediaViewer/resources/mmv/ui/mmv.ui.canvasButtons.js: poking cache for multimediaviewer messages (duration: 00m 04s)
  • 00:05 logmsgbot: aaron Synchronized php-1.24wmf8/includes/EditPage.php: e11d41dd366b039bff79e247368b6bff1245ea5e (duration: 00m 07s)

June 11

  • 23:50 mwalker: clearing resourceloader blobs on commonswiki to try and force a multimediaviewer message "mwscript extensions/WikimediaMaintenance/clearMessageBlobs.php --wiki=commonswiki"
  • 23:49 awight: updated SmashPig from 98b1f348aa55f6a3aac441db08a59ca309fade7a to 22e2923a3a030b17815181574f9ca99b38c5f2dc
  • 23:41 logmsgbot: mwalker Finished scap: SWAT deploy for MultimediaViewer, CentralNotice, and testwiki config (duration: 24m 16s)
  • 23:16 logmsgbot: mwalker Started scap: SWAT deploy for MultimediaViewer, CentralNotice, and testwiki config
  • 23:10 Krinkle: Running deleteEqualMessages.php on trwiki (bug 43917)
  • 22:58 logmsgbot: yurik Synchronized wmf-config/: Restoring to ZRMA for now (duration: 01m 04s)
  • 22:22 logmsgbot: yurik Synchronized wmf-config/InitialiseSettings.php: Attempting to enable new zero ext on zerowiki & ruwiki - take2 (duration: 01m 06s)
  • 22:19 ^d: restarted elasticsearch on logstash1003, complaining about heap.
  • 22:06 logmsgbot: yurik Synchronized wmf-config/InitialiseSettings.php: Attempting to enable new zero ext on zerowiki & ruwiki (duration: 01m 12s)
  • 21:58 logmsgbot: yurik Synchronized php-1.24wmf8/extensions/JsonConfig/: (no message) (duration: 01m 11s)
  • 21:56 logmsgbot: yurik Synchronized php-1.24wmf7/extensions/JsonConfig/: (no message) (duration: 01m 09s)
  • 21:50 logmsgbot: yurik Finished scap: (no message) (duration: 25m 51s)
  • 21:46 ori: Disabling Puppet on mw1149. It's a former bits app server that isn't in PyBal so it isn't getting traffic. Going to stage some proposed changes for apache-config and operations/puppet there.
  • 21:24 logmsgbot: yurik Started scap: (no message)
  • 21:05 logmsgbot: yurik Finished scap: Deploying 3 new ext (JsonConfig, ZeroBanner, ZeroPortal), but they are not enabled anywhere yet (duration: 05m 03s)
  • 21:00 logmsgbot: yurik Started scap: Deploying 3 new ext (JsonConfig, ZeroBanner, ZeroPortal), but they are not enabled anywhere yet
  • 20:07 gwicke: deployed Parsoid 3de0dba15
  • 19:18 bblack: rebooting lvs3003 for 3.13 kernel
  • 19:17 logmsgbot: marktraceur Finished scap: MultimediaViewer fixes for cards 630, 429, and 697 (duration: 18m 45s)
  • 19:17 greg-g: mw1151 *still* giving permission denied errors (publickey), what's the status, yo?
  • 19:03 bblack: rebooting lvs3002 for 3.13 kernel + XPS
  • 18:59 logmsgbot: marktraceur Started scap: MultimediaViewer fixes for cards 630, 429, and 697
  • 18:44 ottomata: disabling puppet on analytics1012 to allow for more replica threads to catch up with current broker replicas...maybe :)
  • 18:41 awight: updated crm from b6815d29de97b80a0ab65db576213a604f0c7cb9 to c38296add61421f87e12cb5b4f3dd68bdf2340db
  • 18:03 Krinkle: Reloading Zuul to deploy I5d154a4002d08
  • 16:43 bblack: shutting off lvs3002.esams pybal to test XPS balancing of live traffic on lvs3004.esams + 3.13
  • 16:30 bblack: rebooting lvs3004 (inactive uploads LVS) for 3.13 again
  • 14:52 hashar: Jenkins restarting (plugin upgrades)
  • 14:48 bblack: rebooting lvs3004.esams (inactive uploads LVS) for 3.13 kernel
  • 14:41 _joe_: manually ran 'planet' on en.planet to restore technews
  • 14:40 hashar: Jenkins updating plugins
  • 13:56 paravoid: upgrading mw1153-mw1160, tmh1001-tmh1002 for USN-2244-1
  • 12:21 _joe_: set up a secondary remote named 'readonly' in /a/common on tin, to use with the icinga check for unmerged commits
  • 11:40 akosiaris: manually cleaning librenms tables. db1001 is going to have increased load for some time. The approach is automatable, see http://jira.observium.org/browse/OBSERVIUM-757
  • 11:32 godog: restarted uwsgi on tungsten, a lot of accesses to reqstats.edits.*.submits
  • 10:45 godog: restarted uwsgi on tungsten, hung on fetching many metrics
  • 09:54 _joe_: restarted apache on palladium - passenger crashed
  • 05:26 paravoid: restarting all swift daemons across the cluster to fix runaway threads due to rsyslog restart
  • 05:04 springle: beginning schema changes bug 49193 page_content_model
  • 03:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 11 03:28:14 UTC 2014 (duration 28m 13s)
  • 02:29 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-11 02:28:18+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-06-11 02:14:43+00:00

June 10

  • 23:34 andrewbogott: updated labs Trusty image w/puppet3, made default
  • 23:19 mutante: rebooting unresponsive ms-be1003
  • 21:09 RobH: montly sms credit check: 1,447.36 SMS credits. will check again in 30 days
  • 19:47 hashar: Jenkins restarted apparently properly. Any breakage would probably be related to the version switch :-D
  • 19:45 ottomata: power cycling analytics1012, attempting to reinstall as kafka broker with new kafka partman recipe
  • 19:42 hashar: Jenkins upgraded from 1.532.2 to 1.554.2 (i.e. bumped to a new LTS version).
  • 19:37 hashar: Broke Jenkins by silently upgrading it  :-(
  • 19:09 Krinkle: git-deploy: Deploying integration/slave-scripts I9521890b911714edf2
  • 18:59 logmsgbot: reedy Synchronized php-1.24wmf8/skins/vector/components/tabs.less: (no message) (duration: 00m 14s)
  • 18:58 mutante: shutting down ekrem
  • 18:18 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Enable data transclusion for wikiquote (duration: 00m 14s)
  • 18:15 logmsgbot: reedy Synchronized docroot and w: Update non Wikipedias to 1.24wmf8 (duration: 00m 16s)
  • 18:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Update non Wikipedias to 1.24wmf8
  • 18:14 logmsgbot: reedy Synchronized php-1.24wmf8/extensions/Wikidata/: (no message) (duration: 00m 16s)
  • 17:28 _joe|away: restarted profiler-to-carbon, stuck waiting data from mwprof
  • 15:21 mutante: ekrem - rm from stored configs/icinga
  • 15:12 mutante: ekrem - revoke salt,puppet keys, stop agents/minion
  • 07:42 springle: enabled pt-slave-delay for dbstore1001, 24h all shards
  • 06:12 springle: xtrabackup clone db1043 to db1048
  • 04:57 springle: db1048 down for upgrade
  • 03:40 springle: switched mchenry to use m2-master/m2-slave for OTRS address lookups
  • 03:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 10 03:24:19 UTC 2014 (duration 24m 18s)
  • 02:29 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-10 02:28:14+00:00
  • 02:27 springle: switched traffic db1048 to db1020. broke gerrit briefly; see ops email
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-06-10 02:14:41+00:00
  • 01:33 chasemp: restarted gerrit on ytterbium
  • 01:01 manybubbles: upgraded all elasticsearch servers in production to 1.2.1. They are just restoring the last few shards on the last node now and they'll spend a few hours tonight rebalancing after the upgrade but otherwise I'm done.
  • 00:41 mwalker: updating donationinterface on payments from b4c5cf1bceb70d65eae28cdd0873036dc33c8992 to 6d74002f2634f41f7038daa7357ff6de55ee4880 for worldpay form error

June 9

  • 23:58 manybubbles: lied - upgrading elastic1014
  • 23:57 manybubbles: upgrading elastic1015
  • 23:30 Krinkle: Reloading Zuul to deploy 6727b8b
  • 23:12 logmsgbot: maxsem Synchronized php-1.24wmf8/extensions/MobileApp: (no message) (duration: 00m 03s)
  • 23:11 logmsgbot: maxsem Synchronized php-1.24wmf7/extensions/MobileApp: (no message) (duration: 00m 03s)
  • 23:03 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://bugzilla.wikimedia.org/66377 (duration: 00m 04s)
  • 20:42 manybubbles: upgraded elastic1007-elastic1010 without issue - starting elastic1010
  • 20:08 subbu: deployed Parsoid 9b673587 (deploy sha 7d0097a1)
  • 19:23 ottomata: disabling puppet on analytics1012
  • 18:59 ottomata: decomissioning analytics1012 in hadoop cluster, this will become a Kafka broker
  • 17:58 manybubbles: elastic1004-1006 upgraded without trouble - cluster is working on filling elatic1006 before moving on to 1007, and the rest
  • 17:04 andrewbogott: switching labs to puppet3
  • 17:03 awight: update crm from b38497a9d0ef75fe2b20b03b649ac13a5e3f47a7 to b6815d29de97b80a0ab65db576213a604f0c7cb9
  • 16:30 manybubbles: upgrading elastic1003 - upgrade is going well so far so I'm going to stop watching it as closely and let it be more automated
  • 15:28 manybubbles: elastic1001 went well, doing 1002 by hand again
  • 15:17 logmsgbot: anomie Synchronized php-1.24wmf8/extensions/Wikidata: SWAT: Wikidata entity suggester bug fixes gerrit:138339 (duration: 00m 16s)
  • 15:12 greg-g: mw1151 still "permission denied" during deploys
  • 15:12 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable TemplateData GUI on Portuguese Wikipedia gerrit:137986 (duration: 00m 14s)
  • 15:09 logmsgbot: anomie Synchronized php-1.24wmf7/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWSaveDialog.js: SWAT: VE fix for focus regression gerrit:137978 (duration: 00m 15s)
  • 15:06 andrewbogott: beta updating all instances to puppet 3 via a cherry-pick of https://gerrit.wikimedia.org/r/#/c/137898/ on deployment-salt
  • 15:05 logmsgbot: anomie Synchronized php-1.24wmf8/extensions/VisualEditor/modules/ve-mw/: SWAT: VE fix for focus regression and alignment issues gerrit:137971 gerrit:138122 (duration: 00m 14s)
  • 15:01 manybubbles: successfully synced plugins, upgrading elastic1001 to make sure everything is working ok with it - then we'll run through the others more quickly
  • 14:57 manybubbles: syncing elasticsearch plugins for 1.2.1 - any elasticsearch restart from here on out needs to come with 1.2.1 or the node will break.
  • 14:54 manybubbles: starting Elasticsearch upgrade with elastic1001
  • 07:14 springle: disabled puppet on analytics1021 to avoid kafka broker restarting with missing mount
  • 05:15 springle: xtrabackup clone db1046 to db1020
  • 04:44 springle: umount /dev/sdf on analytics1021, fs in r/o mode, kafka broker not running. no checks yet
  • 03:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 9 03:23:05 UTC 2014 (duration 23m 4s)
  • 02:29 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-09 02:28:08+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-06-09 02:14:46+00:00

June 8

  • 23:27 p858snake|l: icinga has been shitting in the channel for 9+ hours (before I went to bed) about Varnishkafka, nothing noted in SAL. Here be a note about it.
  • 03:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 8 03:21:28 UTC 2014 (duration 21m 27s)
  • 02:28 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-08 02:27:21+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-06-08 02:14:10+00:00

June 7

  • 23:48 hoo: Fixed four CentralAuth log entries on meta which were logged for WikiSets/0
  • 21:36 manybubbles: that means I turned off puppet and shut down Elasticsearch on elastic1017 - you can expect the cluster to go yellow for half an hour or so while the other nodes take rebuild the redundency that elastic1017 had
  • 21:35 manybubbles: after consulting logs - elastic1017 has had high io wait since it was deployed - I'm taking it out of rotation
  • 21:31 manybubbles: elastic1017 is sick - thrashing to death on io - restarting Elasticsearch to see if it recovers unthrashed
  • 17:56 godog: restarted ES on elastic1017.eqiad.wmnet (at 17:22 UTC)
  • 03:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 7 03:23:32 UTC 2014 (duration 23m 31s)
  • 02:31 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-07 02:29:57+00:00
  • 02:17 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-06-07 02:16:30+00:00

June 6

  • 23:51 Krinkle: Restarted Jenkins, force stopped Zuul, started Zuul, configure Jenkins via web interface (disable Gearman, save, enable German); Seems to be back up now, finally.
  • 22:52 mutante: same for rhenium, titanium, bast1001, calcium, carbon, ytterbium, stat1003
  • 22:42 RoanKattouw: Restarting Jenkins didn't help, jobs still aren't making it across from Zuul into Jenkins
  • 22:36 RoanKattouw: Restarting stuck Jenkins
  • 22:35 mutante: same for holmium, hafnium, silver, netmon1001, magnesium, neon, antimony
  • 22:17 mutante: upgraded ssl packages on zirconium
  • 21:57 Krinkle: Took Jenkins slave on gallium temporarily offline and back online to resolve possible stagnation
  • 20:56 awight_: updated crm from ded541894a70922e098fb3ea48306c8ec0f0f6aa to b38497a9d0ef75fe2b20b03b649ac13a5e3f47a7
  • 18:24 mwalker: updating payments from e823354822c7a35e6c2069d3e72180a45dbc89dc to b4c5cf1bceb70d65eae28cdd0873036dc33c8992 for globalcollect oid hack
  • 14:04 hashar: Gerrit back. chase rebooted it :)
  • 13:55 hashar: Gerrit having some troubles: error: RPC failed; result=22, HTTP code = 503 (while cloning CirrusSearch )
  • 12:58 cmjohnson1: replacing raid controller db1020
  • 06:12 Tim: on osmium installed nodejs for testing
  • 04:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 6 04:23:08 UTC 2014 (duration 23m 7s)
  • 03:13 logmsgbot: LocalisationUpdate completed (1.24wmf8) at 2014-06-06 03:12:19+00:00
  • 02:43 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-06-06 02:42:28+00:00
  • 00:38 bblack: nginx restarted on ssl*
  • 00:16 mutante: fixed permissions on bugzilla's index.cgi, sry

June 5

  • 23:18 logmsgbot: maxsem Synchronized php-1.24wmf7/includes/ChangeTags.php: https://gerrit.wikimedia.org/r/#/c/137563/ (duration: 00m 03s)
  • 23:16 logmsgbot: maxsem Synchronized php-1.24wmf8/includes/ChangeTags.php: https://gerrit.wikimedia.org/r/#/c/137563/ (duration: 00m 03s)
  • 23:06 logmsgbot: maxsem Synchronized php-1.24wmf7/extensions/TemplateData: https://gerrit.wikimedia.org/r/#/c/137751/ (duration: 00m 04s)
  • 22:15 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ife5081549: Put $wgRCFeeds[rcs100x] config behind $wmfRealm check (duration: 00m 04s)
  • 22:12 logmsgbot: ori updated /a/common to Ife5081549: Put $wgRCFeeds['rcs100x'] config behind $wmfRealm check
  • 21:48 ori: updated eventlogging to a8602c1d879f
  • 21:34 MaxSem: Renaming geo_killlist and geo_updates to *_old
  • 18:36 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 14s)
  • 18:35 logmsgbot: reedy Synchronized database lists: (no message) (duration: 00m 13s)
  • 18:17 Reedy: Created FlaggedRevs tables on ckbwiki
  • 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Update group0 to 1.24wmf8
  • 18:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf7
  • 17:00 logmsgbot: reedy Synchronized wmf-config/: Wrap some long lines, add some docs (duration: 00m 26s)
  • 16:43 bblack: rebooting lvs3002
  • 16:36 paravoid: downpref all of amslvs* in favor of lvs30*
  • 16:17 paravoid: downprefing amslvs1, upprefing lvs3001
  • 16:02 mark: Connected cp3018:eth1 to cr1-esams:xe-0/0/3 (unconfigured)
  • 15:59 _joe_: disabling puppet on virt1000 while we test the puppet3 upgrade on virt0
  • 15:48 logmsgbot: reedy Finished scap: 2nd scap for 1.24wmf8, should be effectively a nooop (duration: 12m 33s)
  • 15:35 logmsgbot: reedy Started scap: 2nd scap for 1.24wmf8, should be effectively a nooop
  • 15:21 logmsgbot: anomie Synchronized php-1.24wmf6/extensions/VisualEditor/modules/ve-mw/ui/dialogs/: SWAT: Use <visualeditor-toolbar-cite-label> correctly in the Media and Reference toolbars gerrit:136783 (duration: 00m 15s)
  • 15:18 logmsgbot: anomie Synchronized php-1.24wmf7/extensions/VisualEditor/modules/ve-mw/ui/dialogs/: SWAT: Use <visualeditor-toolbar-cite-label> correctly in the Media and Reference toolbars gerrit:136782 (duration: 00m 12s)
  • 15:04 logmsgbot: anomie Synchronized php-1.24wmf7/extensions/Popups/resources/: SWAT: Hovercard animation fixes gerrit:137530 gerrit:137531 gerrit:137532 (duration: 00m 14s)
  • 14:57 logmsgbot: reedy Finished scap: testwiki to 1.24wmf8 and build l10n cache (duration: 26m 23s)
  • 14:54 hashar: restarting Zuul
  • 14:31 logmsgbot: reedy Started scap: testwiki to 1.24wmf8 and build l10n cache
  • 14:15 logmsgbot: reedy scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/a/common/wmf-config/extension-list" --output="/tmp/tmp.hiiCprts7Z" ' returned non-zero exit status 1 (duration: 00m 17s)
  • 14:14 logmsgbot: reedy Started scap: testwiki to 1.24wmf8 and build l10n cache
  • 14:07 logmsgbot: reedy scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/a/common/wmf-config/extension-list" --output="/tmp/tmp.WtQBrR6JUp" ' returned non-zero exit status 1 (duration: 01m 08s)
  • 14:06 logmsgbot: reedy Started scap: testwiki to 1.24wmf8 and build l10n cahce
  • 14:05 logmsgbot: reedy Purged l10n cache for 1.24wmf5
  • 13:58 hashar: Adding unit tests Jenkins job for most mediawiki extensions 137578
  • 12:05 godog: powercycling ms-be1005, no ssh, no console
  • 10:28 godog: restarted uwsgi on tungsten
  • 09:24 godog: moving bits traffic to the general appserver pool in eqiad
  • 04:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 5 04:09:50 UTC 2014 (duration 9m 49s)
  • 03:03 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-06-05 03:02:00+00:00
  • 02:33 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-06-05 02:32:06+00:00
  • 02:23 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1007 (duration: 01m 26s)
  • 00:46 bblack: lvs3002 (live uploads lb for esams) is running ntpd

June 4

  • 23:43 Tim: on searchidx1001: restarting lsearchd and indexer
  • 23:40 logmsgbot: mwalker Finished scap: Scapping for SWAT; MultiMedia viewer and config changes (duration: 22m 16s)
  • 23:20 Tim: on searchidx1001: as a temporary hack to work around scap disk full errors, set up a bind mount at /usr/local/apache/common-local linking to a directory in /a, by local modification of /etc/fstab
  • 23:18 logmsgbot: mwalker Started scap: Scapping for SWAT; MultiMedia viewer and config changes
  • 21:56 logmsgbot: yurik Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/136503/ (duration: 01m 07s)
  • 21:54 logmsgbot: yurik Synchronized mobilelanding.php: (no message) (duration: 01m 07s)
  • 20:47 MaxSem: Truncating geo_killlist everywhere
  • 20:33 subbu: deployed Parsoid 165a2042 (deploy sha fc1b1ed4)
  • 19:04 bd808|deploy: Restarted elasticsearch on logstash1001; JVM OOM
  • 19:00 logmsgbot: maxsem Synchronized php-1.24wmf7/extensions/GeoData/: (no message) (duration: 00m 04s)
  • 18:58 logmsgbot: maxsem Synchronized php-1.24wmf6/extensions/GeoData/: (no message) (duration: 00m 03s)
  • 18:43 bd808|deploy: mw1151 gave an ssh denied error for MaxSem during sync-dir
  • 18:40 logmsgbot: maxsem Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/136487/ (duration: 00m 04s)
  • 17:54 mutante: shutting down solr1001-1003
  • 17:47 logmsgbot: yurik Synchronized php-1.24wmf7/extensions/ZeroRatedMobileAccess/: (no message) (duration: 01m 07s)
  • 17:44 logmsgbot: yurik Synchronized php-1.24wmf6/extensions/ZeroRatedMobileAccess/: (no message) (duration: 01m 06s)
  • 17:27 mutante: stopping puppet/salt on solr100[13], removed from icinga
  • 16:36 robh: blog.wikimedia.org updated to latest wp version
  • 16:13 mutante: installing package upgrades on bast1001
  • 16:11 mutante: installing package upgrades on iron
  • 15:59 mutante: killing puppet certs,salt keys for solr100[13].eqiad - decom
  • 15:28 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: close wikimania2013wiki for real (duration: 00m 10s)
  • 15:28 logmsgbot: manybubbles Synchronized closed.dblist: close wikimania2013wiki (duration: 00m 09s)
  • 15:23 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: close wikimania2013wiki (duration: 00m 10s)
  • 15:21 logmsgbot: manybubbles Synchronized php-1.24wmf6/extensions/MobileApp/: (no message) (duration: 00m 10s)
  • 15:15 logmsgbot: manybubbles Synchronized php-1.24wmf7/extensions/MobileApp/: (no message) (duration: 00m 08s)
  • 15:07 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT deploy for media viewer (duration: 00m 13s)
  • 14:57 mutante: cleaning up duplicate cronjobs on terbium - all log to /var/log/mediawiki now
  • 12:53 hashar: Zuul upgraded (git tag wmf-deploy-20140604 ). Merges are now done by an indecent process zuul-merger
  • 12:43 hashar: upgrading Zuul to split the merger part to an independent process. Short unscheduled downtime starting in a few minutes
  • 07:51 _joe_: rebooted ms-be1001, host unresponsive to ping, blank console
  • 06:14 springle: starting online schema change, bug 66089 gerrit 137149
  • 04:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 4 04:26:32 UTC 2014 (duration 26m 31s)
  • 03:35 Krinkle: Deploy I882e3fa57b2e5e3de in Zuul and reload config
  • 03:16 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-06-04 03:15:34+00:00
  • 02:47 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-06-04 02:46:06+00:00

June 3

  • 23:14 logmsgbot: ori Synchronized php-1.24wmf7/extensions/MobileApp: SWAT cherry-picks for MobileApp (with patch) (duration: 00m 04s)
  • 23:11 logmsgbot: ori Synchronized php-1.24wmf6/extensions/MobileApp: SWAT cherry-picks for MobileApp (duration: 00m 04s)
  • 23:10 logmsgbot: ori Synchronized php-1.24wmf7/extensions/MobileApp: SWAT cherry-picks for MobileApp (duration: 00m 03s)
  • 23:06 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I9dac0dc6a80: Set $wgIncludejQueryMigrate = true; for all wikis (duration: 00m 03s)
  • 22:41 logmsgbot: marktraceur Finished scap: Update Media Viewer preference string for wmf7 - already backported to wmf6 (duration: 13m 19s)
  • 22:38 Krinkle: git-deploy: Deploying integration/slave-scripts If2e2e675802f
  • 22:27 logmsgbot: marktraceur Started scap: Update Media Viewer preference string for wmf7 - already backported to wmf6
  • 21:49 logmsgbot: marktraceur updated /a/common to I409703a11: Enable MMV by default on dewiki beta.
  • 21:25 logmsgbot: marktraceur Synchronized mediaviewer.dblist: Enable media viewer by default on enwiki (duration: 00m 06s)
  • 21:18 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: Throttle the MMV event logging a bit more for the launch today (duration: 00m 06s)
  • 21:17 logmsgbot: marktraceur updated /a/common to I549906510: Launch Media Viewer for all users on English wikipedia
  • 21:09 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: Touch InitialiseSettings.php because that's what we do (duration: 00m 06s)
  • 21:08 logmsgbot: marktraceur Synchronized mediaviewer.dblist: Add dewiki to the on-by-default list for Media Viewer (duration: 00m 06s)
  • 21:08 logmsgbot: marktraceur updated /a/common to Ie237b0ae1: Launch Media Viewer for all users on German wikipedia
  • 20:51 MaxSem: Disabled GeoData updates on terbium
  • 20:41 hashar: repack command: find /srv/ssd/gerrit/ -type d -name '*.git' -print -exec git --git-dir="{}" repack -afd \; -exec git --git-dir="{}" pack-refs --all \;
  • 20:41 hashar: Jenkins repacking gerritslave replicas on gallium and lanthanum. Running in screen as hashar -> gerritslave
  • 18:14 logmsgbot: reedy Synchronized wmf-config/: Stop sending IRC RC to PMTPA (duration: 00m 17s)
  • 18:07 logmsgbot: reedy Synchronized docroot and w: (no message) (duration: 00m 14s)
  • 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All non wikipedias to 1.24wmf7
  • 15:45 akosiaris: merged https://gerrit.wikimedia.org/r/#/c/133515/ which enabled ferm on hydrogen/chromium
  • 15:41 logmsgbot: anomie Finished scap: SWAT: Update i18n for MultimediaViewer gerrit:136718 (duration: 17m 56s)
  • 15:23 logmsgbot: anomie Started scap: SWAT: Update i18n for MultimediaViewer gerrit:136718
  • 15:03 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Lower MediaViewer sampling for enwiki and dewiki gerrit:136717 (duration: 00m 14s)
  • 13:05 paravoid: salt * start procps
  • 11:13 _joe_: restarted jobrunners as they were blocked by restarting via cron
  • 10:58 godog: try restarting mw-job-runner on mw1012
  • 03:42 springle: revert to lvm snapshot on db1046, xfs being crotchety
  • 03:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 3 03:16:22 UTC 2014 (duration 16m 21s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-06-03 02:25:48+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-06-03 02:14:12+00:00
  • 01:32 logmsgbot: reedy Synchronized wmf-config/CommonSettings.php: wgCentralAuthRC to EQIAD rc ircd (duration: 00m 14s)
  • 00:28 awight: update crm from 5f6217d8f4d750087dcd37faca6b41de82d2362e to ded541894a70922e098fb3ea48306c8ec0f0f6aa

June 2

  • 23:34 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/136428/ (duration: 00m 03s)
  • 23:22 logmsgbot: maxsem Synchronized php-1.24wmf6/extensions/Flow/: (no message) (duration: 00m 04s)
  • 23:21 logmsgbot: maxsem Synchronized php-1.24wmf6/extensions/VisualEditor/: (no message) (duration: 00m 03s)
  • 23:20 logmsgbot: maxsem Synchronized php-1.24wmf7/extensions/VisualEditor/: (no message) (duration: 00m 04s)
  • 23:18 logmsgbot: maxsem Synchronized php-1.24wmf7/extensions/Flow/: https://gerrit.wikimedia.org/r/#/c/136936/ (duration: 00m 05s)
  • 22:49 Krinkle: Repooled integration-slave1003 in Jenkins.
  • 22:37 Krinkle: Hack-patching integration-slave1003.eqiad.wmflabs per https://bugzilla.wikimedia.org/show_bug.cgi?id=61508#c2
  • 21:30 mutante: searchidx1001 - low disk space, gzip MegaSAS.log, delete old kernel headers
  • 21:18 awight: updated crm from b6e004f7349507523423c59170274150a44b0aaf to 5f6217d8f4d750087dcd37faca6b41de82d2362e
  • 20:09 gwicke: deployed Parsoid 04a4bf2b
  • 20:08 hashar: Jenkins unpolled integration-slave1003 npm is outdated there and does not trust npmregistry.org ( bug 61508 )
  • 19:29 awight: updated crm from ce64066316e77f6fc3545c6265e2d81e3ef773c4 to b6e004f7349507523423c59170274150a44b0aaf
  • 19:18 awight: update crm from 5b231163e9e880de5b9787d40b679a6723748aca to ce64066316e77f6fc3545c6265e2d81e3ef773c4
  • 18:58 logmsgbot: csteipp Synchronized php-1.24wmf6/includes/upload/UploadBase.php: (no message) (duration: 00m 04s)
  • 18:51 logmsgbot: csteipp Synchronized php-1.24wmf7/includes/upload/UploadBase.php: (no message) (duration: 00m 06s)
  • 18:41 awight: updated tools from d257e8445e028b758b1d1fa90c857667d4faac62 to cbcd14a84f7bc8682822d3b1910b48bfd932b00d
  • 17:15 chasemp: disabling ircd on ekrem
  • 17:05 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: Cache search suggestions for 3 hours instead of 6 (duration: 00m 04s)
  • 17:03 chasemp: moving irc.wikimedia.org to argon
  • 16:27 ottomata: ran preferred-replica-election to fix vk delivery errors
  • 16:24 logmsgbot: demon Synchronized wmf-config/throttle.php: Library of Israel editathon (duration: 00m 04s)
  • 16:07 manybubbles: rebuilding all english non-wikipedias with unicode normalization
  • 15:36 logmsgbot: manybubbles Synchronized wmf-config/: SWAT deploy - more import sources and upload domains (duration: 00m 04s)
  • 15:34 manybubbles: reindexing all hebrew wikis to switch them from the hebrew analyzer to proper unicode normalization
  • 15:33 ottomata: attempting to powercycle analytics1015, it is not responding to pings, no output on console
  • 15:33 logmsgbot: manybubbles Synchronized wmf-config/: SWAT deploy changing some search settings (duration: 00m 05s)
  • 15:26 hashar: restarted Zuul. All jobs lists :-(
  • 15:25 hashar: Zuul stuck in a loop reporting a change :-(
  • 15:20 hashar: Jenkins/Zuul stuck. Depooling/Repooling some slaves to reregister jobs with Zuul
  • 14:51 ottomata: chown -R datasets /data/xmldatadumps/public/other/pagecounts-ez on dataset1001 to accompany 70a7f61, fixing bug 66005
  • 12:44 akosiaris: manually ran puppet on mw11991
  • 07:21 hashar: restarted Zuul unintentionally
  • 03:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 2 03:11:05 UTC 2014 (duration 11m 4s)
  • 03:04 ori: ..on vanadium.
  • 03:03 ori: moving /var/log/eventlogging/archive/* to /srv/eventlogging-logs to free up space on the root partition. unpuppetized for now, sadly.
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-06-02 02:23:57+00:00
  • 02:13 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-06-02 02:12:53+00:00
  • 02:06 logmsgbot: tstarling Synchronized php-1.24wmf6: Revert "Use square bounding boxes for default-sized thumbnails" (duration: 01m 18s)
  • 02:02 logmsgbot: tstarling Synchronized php-1.24wmf7: (no message) (duration: 01m 31s)

June 1

  • 05:41 awight: updated payments from 7c695e9c4c7386a7585b6067df29b8caaaa089f0 to e823354822c7a35e6c2069d3e72180a45dbc89dc
  • 03:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 1 03:09:28 UTC 2014 (duration 9m 27s)
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-06-01 02:23:58+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-06-01 02:13:23+00:00

May 31

  • 03:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 31 03:13:47 UTC 2014 (duration 13m 46s)
  • 02:27 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-05-31 02:26:20+00:00
  • 02:16 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-05-31 02:15:48+00:00

May 30

  • 22:00 greg-g: ori sync'd out a config change to Add $wgRCFeeds entries for RCStream on rcs100[12].eqiad.wmnet
  • 21:57 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 03s)
  • 21:21 hashar: Zuul restarted (somehow by mistake)
  • 20:39 logmsgbot: bd808 Synchronized php-1.24wmf7/extensions/Elastica/Elastica/test/lib/Elastica/Test/IndexTest.php: Touched to test sync-file (duration: 00m 04s)
  • 20:36 logmsgbot: bd808 Synchronized wmf-config/db-eqiad.php: Touched db-eqiad.php to test sync-file (duration: 00m 04s)
  • 20:35 logmsgbot: bd808 Synchronized wmf-config/throttle.php: Touched to test sync-file (duration: 00m 04s)
  • 20:35 bd808|deploy: Scap updated to 6c0c4f0
  • 19:45 logmsgbot: bd808 Synchronized wmf-config/throttle.php: Touched to test sync-file (duration: 00m 05s)
  • 19:40 logmsgbot: bd808 Synchronized wmf-config/db-eqiad.php: Touched db-eqiad.php to test sync-file (duration: 00m 03s)
  • 19:33 logmsgbot: bd808 Synchronized README: Testing sync-file (duration: 00m 06s)
  • 19:30 bd808|deploy: Scap updated to c4204dd
  • 17:21 andrewbogott: restarted pdns on virt0 and virt1000
  • 16:07 andrewbogott: changes the 'Ops' gid to 700 in ldap
  • 16:02 akosiaris: enabled VT on thallium/mercury for ganeti evaluation purposes
  • 14:54 _joe_: ran scap-rebuild-cdb on mw1163
  • 13:47 hashar: Jenkins: removing label hasBrowserTests from labs slaves 136315
  • 13:43 hashar: Jenkins: removing label hasHhvm from labs slaves 136315
  • 13:42 hashar: Jenkins: removing label hasJenkinsDebianGlue from labs slaves 136315
  • 13:26 hashar: Jenkins lowering number of executors on labs slave from 5 to 4 since they have 4 CPU
  • 13:25 hashar: Jenkins polling a third CI slave integration-slave1003.
  • 09:26 hashar_: Zuul is processing jobs again. For reference bug is bug 63760
  • 09:24 hashar_: Jenkins: disconnecting and reconnecting labs slaves to reregister them with Zuu
  • 09:17 hashar: Jenkins/Zuul locked
  • 03:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 30 03:46:09 UTC 2014 (duration 46m 8s)
  • 03:01 logmsgbot: LocalisationUpdate completed (1.24wmf7) at 2014-05-30 03:00:14+00:00
  • 02:31 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-05-30 02:30:51+00:00

May 29

  • 23:36 logmsgbot: csteipp synchronized php-1.24wmf6/includes/specials/SpecialPasswordReset.php
  • 23:19 ^d: cleaned up /var/cache/apt on searchidx1001, freed up ~20% of the disk, should be fine now
  • 23:09 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I09d92987: Flow-ify mw:Talk:Search'
  • 23:07 logmsgbot: ori synchronized cirrus.dblist 'I3f09dff7e: All Wikipedias with 100k pages or less getting Cirrus as primary'
  • 23:05 logmsgbot: ori updated /a/common to I3f09dff7e: All Wikipedias with 100k pages or less getting Cirrus as primary
  • 20:29 logmsgbot: marktraceur synchronized wmf-config/InitialiseSettings.php 'Enable Media Viewer on all wikisources by default'
  • 20:28 logmsgbot: marktraceur updated /a/common to I95348e0d4: Launch Media Viewer for all users on all Wikisources
  • 20:25 logmsgbot: anomie synchronized php-1.24wmf6/includes/api 'Revert revert of gerrit:120827, underlying bug should be fixed now'
  • 20:19 logmsgbot: anomie synchronized php-1.24wmf6/extensions/EducationProgram/includes/api/ApiListStudents.php 'Backport fix for bugzilla:65906'
  • 20:05 logmsgbot: anomie synchronized php-1.24wmf7/extensions/EducationProgram/includes/api/ApiListStudents.php 'Backport fix for bugzilla:65906'
  • 19:40 ^d: hewiki elastic index was missing geodata mappings. re-map + in place reindex failed spectacularly. rebuilding from scratch now.
  • 19:38 ori: mw1163: mkdir -p /usr/local/apache/common-local && chown mwdeploy:mwdeploy /usr/local/apache/common-local
  • 19:09 cmjohnson1: powering down mw1151 for disk replacement
  • 19:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf7
  • 18:49 cmjohnson1: removing mw1151 from pybal and dsh groups to replace disk and reinstall
  • 18:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf6 take 3
  • 18:44 logmsgbot: reedy synchronized php-1.24wmf6/includes/api/
  • 18:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rv that
  • 18:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf6
  • 15:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
  • 15:42 logmsgbot: reedy Finished scap: testwiki to 1.24wmf7 and build l10n cache (duration: 24m 24s)
  • 15:17 logmsgbot: reedy Started scap: testwiki to 1.24wmf7 and build l10n cache
  • 15:06 logmsgbot: anomie synchronized php-1.24wmf6/extensions/VisualEditor/modules/ve-mw/ 'SWAT: VisualEditor URL decoding and image alignment fixes. gerrit:135922 gerrit:135946'
  • 04:26 logmsgbot: springle synchronized README 'test sync-file'
  • 04:20 ori: updated scap to 9ba9014: Partially revert "Convert sync-dir and sync-file to python"
  • 04:13 ori: re-enabled puppet on tin
  • 03:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 29 03:57:59 UTC 2014 (duration 57m 58s)
  • 03:55 logmsgbot: ori Synchronized README: Debugging sync-file (duration: 00m 06s)
  • 03:51 springle: db1009 mariadb 5.5.37 live trial with low load
  • 03:49 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1009 in s2, take #2'
  • 03:45 ori: disabled puppet on tin and copied sync-common-file from mediawiki/tools/scap@8f2a8356c38 into /usr/local/bin to debug sync issue
  • 03:44 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1009 in s2 (duration: 00m 08s)
  • 03:11 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-05-29 03:10:15+00:00
  • 02:36 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-29 02:35:45+00:00
  • 01:22 awight: updated SmashPig from e9964bfec47b3796dab0a19a9545cc3abb23fde6 to 98b1f348aa55f6a3aac441db08a59ca309fade7a
  • 01:16 awight: (rollback)
  • 01:16 awight: updated SmashPig from 03015f3827fedea9d0f89c791604ad08ec97ba71 to e9964bfec47b3796dab0a19a9545cc3abb23fde6
  • 01:04 awight: update SmashPig from f64f79f13cf4ab560d0bb5bd69690c827a821629 to 03015f3827fedea9d0f89c791604ad08ec97ba71

May 28

  • 23:39 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: Including external CologneBlue/Modern skins, if they exist (duration: 00m 07s)
  • 22:27 awight: updated crm from 65a433b5564f42c3aa4f310cd4bb938ae70f841d to 5b231163e9e880de5b9787d40b679a6723748aca
  • 22:27 awight: updated tools from 1e8029544dc19a84f6d1adf2783266e16d19ef1f to d257e8445e028b758b1d1fa90c857667d4faac62
  • 21:23 RoanKattouw: Restarting Parsoid Varnishes per gwicke's request
  • 20:38 mutante: enabling puppet on osmium
  • 20:18 hashar: Jenkins: killed all phantomjs process on gallium. They were eating all available memory. All three process were VisualEditor qunit tests.
  • 20:13 subbu: deployed parsoid a234af8c0 (deploy sha f17506eb)
  • 20:12 hashar: gallium (Jenkins master) sent to swap somehow :-(
  • 20:02 logmsgbot: bd808 Finished scap: no-op scap deleted.dblist (duration: 10m 40s)
  • 19:52 bd808|deploy: Horrible log message; should be "no-op scap to test code changes"
  • 19:51 logmsgbot: bd808 Started scap: no-op scap deleted.dblist
  • 19:49 logmsgbot: bd808 Synchronized database lists: (no message) (duration: 00m 03s)
  • 19:48 logmsgbot: bd808 Synchronized robots.txt: Testing sync-file in php (duration: 00m 03s)
  • 19:08 bd808|deploy: Symlinks for mergeCdbFileUpdates, mwversionsinuse, refreshCdbJsonFiles, scap-rebuild-cdbs, scap-recompile and sync-common on tin still pointing to /srv/scap/bin instead of /srv/deployment/scap/scap/bin
  • 19:06 robh: mw1053 reinstalling
  • 18:54 logmsgbot: bd808 Synchronized robots.txt: Testing sync-file in php (duration: 00m 05s)
  • 18:28 mutante: running puppet on jobrunners
  • 18:27 ^d: jobrunners back up now, should slowly catch back up
  • 18:24 bd808|deploy: Scap updated to fd7e538; Trebuchet fetch and checkout failed for mw1053.eqiad.wmnet
  • 18:06 bd808|deploy: Restarted logstash on logstash1001; log event volume suspiciously low for the last ~35 minutes
  • 17:59 ^d: all job runners halted at 17:39? graphite shows no jobs being run, runJobs on fluorine also has nothing since the timestamp.
  • 17:41 logmsgbot: yurik synchronized php-1.24wmf6/extensions/ZeroRatedMobileAccess/
  • 17:37 logmsgbot: yurik synchronized php-1.24wmf5/extensions/ZeroRatedMobileAccess/
  • 17:32 mwalker: enabling worldpay in BE (payments from 5136b0b6852f3e949e4dc847f7137f1b7bc3037b to 7c695e9c4c7386a7585b6067df29b8caaaa089f0)
  • 16:47 hashar: Jenkins/Zuul back. Jobs meant to be run on labs instances ended up not being registered anymore with the Zuul Gearman server. That must be a bug in the Jenkins Gearman plugin :-( bug 63760
  • 16:31 hashar: Jenkins / Zuul locked. Looking into it
  • 16:29 _joe_: restarted mwprof/profiler-to-carbon
  • 15:09 logmsgbot: anomie synchronized php-1.24wmf6/extensions/Wikidata 'SWAT: Fix issue with Wikidata rollback gerrit:135767'
  • 13:52 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 13:26 logmsgbot: reedy synchronized wmf-config/ 'Enable REL1_23 in ExtensionDistributor'
  • 13:24 manybubbles: restarting elastic1001 to revert it back to mmapfs - niofs wasn't better. worse, even.
  • 11:23 manybubbles: restarting elastic1001 to try out niofs (instead of mmapfs) on advice from a lucene developer
  • 03:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 28 03:24:24 UTC 2014 (duration 24m 23s)
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-05-28 02:24:55+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-28 02:13:02+00:00

May 27

  • 23:06 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I7fdafede7: Disable the anonymous signup invite experiment'
  • 22:00 mutante: caesium, release files: changed file owner groups mwupld->releasers-mediawiki, mobileupld->releasers-mobile (to match switch to yaml groups)
  • 18:39 logmsgbot: reedy synchronized docroot and w
  • 18:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.24wmf6
  • 18:31 logmsgbot: reedy synchronized php-1.24wmf6/includes/SkinTemplate.php
  • 18:23 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
  • 17:46 logmsgbot: aude synchronized php-1.24wmf6/extensions/Wikidata 'JS fixes for Wikidata'
  • 17:29 mutante: welcome new deployer cscott
  • 16:04 gwicke: restarted parsoids after another surge in load
  • 15:48 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'SWAT: move-categorypages permission changes on fawiki gerrit:135426'
  • 15:42 logmsgbot: anomie synchronized php-1.24wmf5/extensions/UniversalLanguageSelector/resources/ 'SWAT: Update ULS to fix beta feature gerrit:135535'
  • 15:40 logmsgbot: anomie synchronized php-1.24wmf6/extensions/UniversalLanguageSelector/resources/ 'SWAT: Update ULS to fix beta feature gerrit:135310'
  • 15:32 logmsgbot: anomie synchronized php-1.24wmf6/includes/Title.php 'SWAT: Check correct message in category moving gerrit:135211'
  • 15:27 logmsgbot: anomie synchronized php-1.24wmf5/includes/Title.php 'SWAT: Check correct message in category moving gerrit:135210
  • 15:21 logmsgbot: anomie synchronized php-1.24wmf6/includes/HistoryBlob.php 'SWAT: Revert another visibility change that causes errors bugzilla:65665 gerrit:135574'
  • 15:15 logmsgbot: anomie synchronized php-1.24wmf6/includes/revisiondelete/ 'SWAT: Revert another visibility change that causes fatal errors bugzilla:65733 gerrit:135389'
  • 15:13 logmsgbot: anomie synchronized php-1.24wmf5/includes/revisiondelete/ 'SWAT: Revert another visibility change that causes fatal errors bugzilla:65733 gerrit:135388'
  • 15:05 logmsgbot: anomie synchronized php-1.24wmf6/extensions/VisualEditor/modules/ve-mw/ 'SWAT: Fix for VisualEditor image alignment regression gerrit:135171'
  • 12:19 Reedy: Created SecurePoll tables on zerowiki, legalteamwiki, zhwikivoyage, viwikivoyage, tyvwiki
  • 11:40 godog: restart apache2 on tungsten, many report.py hung
  • 05:42 gwicke: restarted parsoids after load surge
  • 03:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 27 03:12:29 UTC 2014 (duration 12m 28s)
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-05-27 02:24:51+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-27 02:13:08+00:00
  • 01:39 springle: starting updateCollation on s2 cs-wiki from tin
  • 01:36 logmsgbot: springle synchronized wmf-config/InitialiseSettings.php '$wgCategoryCollation to uca-cs on cswiki'

May 26

  • 09:17 hashar: bugzilla.bugs_fulltext bug was bug 65762
  • 09:16 _joe_: repaired table bugzilla.bugs_fulltext on db1001 as it was marked as crashed
  • 03:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 26 03:10:02 UTC 2014 (duration 10m 1s)
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-05-26 02:24:39+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-26 02:13:01+00:00

May 25

  • 03:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 25 03:08:47 UTC 2014 (duration 8m 46s)
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-05-25 02:24:10+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-25 02:13:47+00:00

May 24

  • 03:02 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 24 03:01:50 UTC 2014 (duration 1m 49s)
  • 02:21 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-05-24 02:20:11+00:00
  • 02:13 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-24 02:12:25+00:00
  • 00:34 mutante: fixing Aaron's and Ariel's file permissions on fenari

May 23

  • 19:43 logmsgbot: anomie synchronized php-1.24wmf6/includes/HistoryBlob.php 'Backport fix for bug 65665 to 1.24wmf6 gerrit:135089'
  • 19:24 awight: updated fr-tools from 73921d4b4a7ba69b703340ed56e513f8ae8e0bb5 to 1e8029544dc19a84f6d1adf2783266e16d19ef1f
  • 18:37 mwalker: updated paymnets wiki from d99177518b741e7fe18ffda86c83f93c72e164a6 for worldpay
  • 18:22 Jeff_Green: ran authdns-update to merge new wikimedia.community dns zone
  • 17:02 bd808: Starting rolling update of elasticsearch for logstash cluster
  • 16:20 bd808: restarted elasticsearch on logstash1002
  • 16:17 bd808: Elasticsearch on logstash1002 dead due to OOM at 2014-05-23T00:34:03Z
  • 14:52 hashar: killed -9 a remaining Jenkins process
  • 14:21 _joe_: killed zuul server, as was stuck
  • 13:50 _joe_: killed & started jenkins, jvm stuck, unresponsive to jstack
  • 13:17 manybubbles: resarting jenkins because it seems stuck
  • 11:05 mark: Setup BFD on Zayo link between cr2-ulsfo and cr1-eqiad
  • 11:01 mark: Setup BFD on GTT link between cr1-ulsfo and cr2-eqiad
  • 07:30 _joe_: powercycling ms-be1007, unresponsive, console blank, no way to debug
  • 04:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 23 04:16:15 UTC 2014 (duration 16m 14s)
  • 03:23 logmsgbot: LocalisationUpdate completed (1.24wmf6) at 2014-05-23 03:22:07+00:00
  • 02:39 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-23 02:38:33+00:00

May 22

  • 23:22 mutante: osmium,mw1151 fixed UID of mwalker (605->2454)
  • 23:16 bd808: Ran sync-common manually on osmium and mw1151
  • 23:14 mwalker: sync-dir failed for osmium and mw1151
  • 23:14 logmsgbot: mwalker synchronized php-1.24wmf6/extensions/VisualEditor 'Syncing the extension manually because of scap failures on osium, mw1010, mw1070, mw1161, mw1201, and mw1151'
  • 23:11 logmsgbot: mwalker Finished scap: SWAT Update to VisualEditor 134941 (duration: 03m 04s)
  • 23:08 logmsgbot: mwalker Started scap: SWAT Update to VisualEditor 134941
  • 21:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Ifd048bafe0eb4af8765cee20a3d93d7663b1bcdf'
  • 21:33 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
  • 21:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Iecdd8c5e60a142363b40e34d4fe2f27f0e5feef5'
  • 21:22 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'touching'
  • 21:21 logmsgbot: demon synchronized wmf-config/flaggedrevs.php 'Removing FR from mw.org'
  • 21:19 logmsgbot: demon synchronized flaggedrevs.dblist 'Removing FR from mw.org'
  • 20:10 logmsgbot: marktraceur synchronized wmf-config/InitialiseSettings.php 'Sync for mediaviewer.dblist change'
  • 20:06 logmsgbot: marktraceur synchronized mediaviewer.dblist 'Enabling Media Viewer on itwiki and ruwiki by default'
  • 20:04 logmsgbot: marktraceur updated /a/common to I1c658bf65: Remove VE formula editor from BF whitelist (graduated)
  • 19:54 logmsgbot: reedy Purged l10n cache for 1.24wmf4
  • 19:54 logmsgbot: reedy Purged l10n cache for 1.24wmf3
  • 19:54 logmsgbot: reedy Purged l10n cache for 1.24wmf2
  • 19:53 logmsgbot: reedy Purged l10n cache for 1.24wmf1
  • 19:53 logmsgbot: reedy Purged l10n cache for 1.23wmf22
  • 19:52 logmsgbot: reedy Purged l10n cache for 1.23wmf21
  • 19:38 paravoid: cr1/2-ulsfo: BGP peering with AS11820 (WMF Corp HQ)
  • 19:28 Reedy: Ran patch-fr_page_rev-index.sql patch on fawiki
  • 19:27 Reedy: Created flaggedrevs_statistics table on fawiki
  • 19:04 logmsgbot: reedy synchronized wmf-config/
  • 18:30 logmsgbot: reedy synchronized wmf-config/ 'I7a02f2615d98428b6f27514e75d935d36e44fcb1'
  • 18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf6
  • 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf5
  • 17:17 mutante: powercycling ms-be1012
  • 17:05 logmsgbot: reedy Finished scap: testwiki to 1.24wmf6 and build l10n cache (duration: 28m 31s)
  • 16:37 logmsgbot: reedy Started scap: testwiki to 1.24wmf6 and build l10n cache
  • 16:30 mutante: maerlant - it was done for ~8d, old test host that didn't really do anything, revoked salt/pupppet certs, removing from Icinga..
  • 15:52 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'SWAT: Fix typo in MultimediaViewer logging config gerrit:134837'
  • 15:31 logmsgbot: anomie synchronized php-1.24wmf5/extensions/MultimediaViewer/ 'SWAT: Deploy new MultimediaViewer logging to wmf5 wikis gerrit:134804'
  • 15:22 logmsgbot: anomie synchronized wmf-config/CommonSettings.php 'SWAT: Disable old MultimediaViewer logging and pre-enable new logging gerrit:134343'
  • 15:21 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'SWAT: Disable old MultimediaViewer logging and pre-enable new logging gerrit:134343'
  • 15:13 logmsgbot: anomie synchronized php-1.24wmf5/extensions/MultimediaViewer/tests/qunit/mmv/ui/ 'SWAT: Fix qunit tests for MultimediaViewer gerrit:134807'
  • 14:31 paravoid: pushing new swift rings
  • 03:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 22 03:37:31 UTC 2014 (duration 37m 30s)
  • 02:39 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-22 02:38:18+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-22 02:14:49+00:00

May 21

  • 23:28 logmsgbot: maxsem synchronized php-1.24wmf5/extensions/MultimediaViewer 'touch'
  • 23:22 logmsgbot: maxsem synchronized php-1.24wmf5/extensions/MultimediaViewer/ 'https://gerrit.wikimedia.org/r/#/c/134750/'
  • 23:16 logmsgbot: maxsem synchronized php-1.24wmf5/extensions/Flow 'https://gerrit.wikimedia.org/r/#/c/134746/'
  • 22:18 Krinkle: Running deleteEqualMessages.php on guwiki (bug 43917)
  • 21:03 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'GeoData using Elasticsearch everywhere'
  • 20:07 subbu: deployed Parsoid 95929801b (deploy sha ae83633a)
  • 19:56 awight: updated tools from c1f50f6909b04768f3a8faa50b25e88a43f89606 to 73921d4b4a7ba69b703340ed56e513f8ae8e0bb5
  • 19:05 mutante: welcome new deployer tgr
  • 18:22 andrewbogott: restarting gerrit service
  • 15:47 logmsgbot: anomie synchronized php-1.24wmf5/tests/qunit/suites/resources/mediawiki/mediawiki.user.test.js 'May as well sync this too'
  • 15:45 logmsgbot: anomie synchronized php-1.24wmf5/resources/src/mediawiki/mediawiki.user.js 'SWAT: Use mw.log.deprecate to track user() and anonymous()'
  • 15:28 logmsgbot: anomie synchronized php-1.24wmf5/includes/filerepo/file/LocalFile.php 'SWAT: Tweaked timestamp logic in recordUpload2'
  • 15:19 logmsgbot: anomie synchronized php-1.24wmf5/includes/filerepo/file/LocalFile.php 'SWAT: Replace FOR UPDATE with LockManager use in LocalFile::lock()'
  • 15:18 logmsgbot: anomie synchronized php-1.24wmf5/includes/filebackend/FileBackend.php 'SWAT: Replace FOR UPDATE with LockManager use in LocalFile::lock()'
  • 14:11 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'raise db1068 to normal load'
  • 12:30 hashar: Jenkins: updated sysadmin email address from nobody@integration.wikimedia.org to jenkins-bot@wikimedia.org
  • 12:22 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'pool db1068 in s4, warm up'
  • 07:25 awight: updated tools from d2437564c56881f6b879403f2f6f2f554b6b0391 to c1f50f6909b04768f3a8faa50b25e88a43f89606
  • 07:17 awight: updated tools from ee31fc94b17c11a48ddac19aabfcdaab69fd2f72 to d2437564c56881f6b879403f2f6f2f554b6b0391
  • 03:42 springle: resume xtrabackup db1049 to db1068, throttled
  • 03:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 21 03:12:31 UTC 2014 (duration 12m 30s)
  • 03:13 mutante: merging Change-Id: I2827d1ef347 and starting icinga fixed it
  • 03:09 springle: killed db1068 xtrabackup, saturating db1064 network
  • 03:01 mutante: icinga broken on neon due to missing servicegroup 'analytics_eqiad'
  • 02:30 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-21 02:29:03+00:00
  • 02:19 springle: xtrabackup clone db1049 to db1068
  • 02:18 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's4 reduce db1049 load while cloning'
  • 02:16 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-21 02:15:36+00:00
  • 02:04 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's4 raise db1056 to normal load, depool db1011'

May 20

  • 23:53 logmsgbot: maxsem synchronized php-1.24wmf4/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/134504/'
  • 23:52 logmsgbot: maxsem synchronized php-1.24wmf5/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/134504/'
  • 23:40 logmsgbot: maxsem synchronized php-1.24wmf5/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/134517'
  • 23:39 logmsgbot: maxsem synchronized php-1.24wmf4/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/134517'
  • 23:14 logmsgbot: maxsem synchronized php-1.24wmf4/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/134405/'
  • 23:11 logmsgbot: maxsem synchronized php-1.24wmf5/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/134405/'
  • 22:00 logmsgbot: aaron synchronized wmf-config/filebackend.php '69201b4caf703ef1ab52b38be29c80b4e939fdc2 - no-op'
  • 21:57 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Removed old tampa config'
  • 21:41 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Re-enable flow on mediawiki:Talk:Design'
  • 21:39 logmsgbot: bsitu updated /a/common to I037cd0a42: Re-enable flow on Talk:Design ( Removed LQT code )
  • 21:30 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Disable flow on mediawiki:Talk:Design'
  • 21:28 logmsgbot: bsitu updated /a/common to Idde23abd3: Undo "enable flow on Talk:Design"
  • 21:11 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Flow on 3 mediawiki talk pages'
  • 21:08 logmsgbot: bsitu updated /a/common to I549967ca2: Group1 wikis to 1.24wmf5
  • 20:35 Krinkle: Reload zuul to deploy I80496db747a8668be
  • 19:14 hoo: fixed Wikidata for php-1.24wmf5 on mw1138 by manually removing it and then running sync-common
  • 19:09 bd808: ran sync-common on mw1138
  • 19:02 bd808: Updated scap to 7b6fc47
  • 18:44 logmsgbot: aude synchronized php-1.24wmf5/extensions/Wikidata 'Fix jquery error tooltip issue'
  • 18:33 bd808|deploy: Gave up on updating scap with trebuchet
  • 18:32 logmsgbot: bd808 rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.24wmf5
  • 18:22 RobH: restarting salt minions on mw servers
  • 18:12 bd808|deploy: `git deploy sync` for scap ended with "0/230 minions completed fetch"
  • 15:15 andrewbogott: running salt "ms-be*" cmd.run "kill $(ps aux | grep 'find / -user' | awk '{print $2}')" to kill runaway 'finds' on swifts
  • 14:47 andrewbogott: running 'find' commands on many hosts to chown files for users with new UIDs.
  • 13:20 Krinkle: git-deploy: Deploying integration/slave-scripts I4a4e2a4c90fb6
  • 03:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 20 03:37:32 UTC 2014 (duration 37m 31s)
  • 02:34 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-20 02:33:31+00:00
  • 02:20 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-20 02:19:33+00:00

May 19

  • 23:37 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'autopatrolled group on dewikivoyage'
  • 23:26 logmsgbot: catrope synchronized php-1.24wmf5/extensions/VisualEditor
  • 23:25 logmsgbot: catrope synchronized php-1.24wmf5/extensions/MobileFrontend
  • 23:24 logmsgbot: catrope synchronized php-1.24wmf5/extensions/Wikidata
  • 22:19 andrewbogott: Coren restarted opendj and I restarted pdns on virt1000. Opendj was refusing connections for unclear reasons
  • 21:34 Krinkle: Running deleteEqualMessages.php on urwiki (bug 43917)
  • 21:33 Krinkle: Running deleteEqualMessages.php on commonswiki (bug 43917)
  • 20:53 superm401: sync-l10nupdate-1 1.24wmf4 had one error on mw1218
  • 20:27 gwicke: updated Parsoid to 3ac048d7c4b
  • 20:05 csteipp: fix deployed for bug 65501
  • 20:00 Krinkle: Running deleteEqualMessages.php on fowiki (bug 43917)
  • 20:00 Krinkle: Running deleteEqualMessages.php on enwikinews (bug 43917)
  • 19:59 Krinkle: Reloading zuul to deploy I0b8051074da39edcac
  • 19:16 bd808: Added display of exception-json events to fatalmonitor logstash dashboard
  • 19:01 logmsgbot: bd808 Purged l10n cache for 1.24wmf3
  • 19:00 logmsgbot: bd808 Purged l10n cache for 1.23wmf22
  • 18:59 logmsgbot: bd808 Purged l10n cache for 1.23wmf21
  • 18:59 logmsgbot: bd808 Purged l10n cache for 1.23wmf21
  • 18:46 logmsgbot: reedy Finished scap: nooop to test for errors (duration: 02m 45s)
  • 18:44 logmsgbot: reedy Started scap: nooop to test for errors
  • 18:43 Reedy: rm -rf /usr/local/apache/common-local/php-1.23wmf20 against all apaches
  • 18:37 Reedy: Ran sync-common locally on mw1015
  • 18:32 greg-g: mw1010.eqiad.wmnet::common for sync-common, not sure for cdb. (sez superm401)
  • 18:23 superm401: 1 server failed for sync-common. 2 servers failed for sync-rebuild-cdbs
  • 18:18 logmsgbot: mattflaschen Finished scap: Deploy GettingStarted and enable experiment for de, en, fr, and it (duration: 18m 53s)
  • 17:59 logmsgbot: mattflaschen Started scap: Deploy GettingStarted and enable experiment for de, en, fr, and it
  • 15:26 logmsgbot: manybubbles synchronized php-1.24wmf5/resources/lib/oojs-ui/ 'fix panellayout'
  • 15:18 logmsgbot: manybubbles synchronized php-1.24wmf4/extensions/CirrusSearch/ 'adding url parameter to suppress snippets and one to suggest suggestions to cirrus'
  • 15:12 manybubbles: SWAT deployed cirrus update for wmf5 and looks good. doing for wmf4 now.
  • 15:10 logmsgbot: manybubbles synchronized php-1.24wmf5/extensions/CirrusSearch/
  • 14:53 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'touched and synced InitializeSettings.php to make update to cirrus.dblist take hold - resyncing to mw1171'
  • 14:51 _joe_: powercycled mw1171, dead and serial console stuck
  • 14:51 logmsgbot: manybubbles synchronized cirrus.dblist 'Switch cirrus to the primary backend for zh-yue wikipedia - resyncing to mw1171'
  • 14:40 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'touched and synced InitializeSettings.php to make update to cirrus.dblist take hold'
  • 14:38 logmsgbot: manybubbles synchronized cirrus.dblist 'Switch cirrus to the primary backend for zh-yue wikipedia'
  • 13:34 Krinkle: Running deleteEqualMessages.php on mtwiki (bug 43917)
  • 13:17 Krinkle: Running deleteEqualMessages.php on zh_min_nanwiki (bug 43917)
  • 13:07 Krinkle: Running deleteEqualMessages.php on zh_yuewiki (bug 43917)
  • 12:01 Krinkle: Running deleteEqualMessages.php on suwiki (bug 43917)
  • 03:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 19 03:09:00 UTC 2014 (duration 8m 59s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-19 02:25:51+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-19 02:13:37+00:00
  • 00:30 Tim: on osmium: stopping job runners in order to fix cgroup permissions issue

May 18

  • 03:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 18 03:06:03 UTC 2014 (duration 6m 2s)
  • 02:24 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-18 02:23:29+00:00
  • 02:13 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-18 02:12:46+00:00

May 17

  • 03:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 17 03:07:12 UTC 2014 (duration 7m 11s)
  • 02:25 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-17 02:24:46+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-17 02:14:13+00:00
  • 01:58 mutante: powercycling labsdb1003
  • 00:23 ori: varnishadm on cp1056 confirms that varnish recognizes mw1151 as "sick"
  • 00:20 ori: stopping apache and disabling puppet on mw1151 so that varnish stops forwarding reqs to it
  • 00:16 Krinkle: On mw1151, Gadget::loadStructuredList() returns false, memcached has no value for 'enwiki:gadgets-definition:7' and is unable to store it.
  • 00:11 ori: Krinkle identified weird RL responses as all originating in mw1151; dmesg shows ata1 disk troubles: "failed command: READ DMA EXT", "sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed"
  • 00:04 logmsgbot: krinkle synchronized php-1.24wmf4/includes/resourceloader/ResourceLoader.php 'I718fcf23d'

May 16

  • 23:07 awight: updated crm from 0b8aa8aa046935b6cfc67c10ebe10396d5e42745 to 65a433b5564f42c3aa4f310cd4bb938ae70f841d
  • 22:56 logmsgbot: aaron synchronized php-1.24wmf4/includes/db/Database.php '182e42c173b9ab0c2bc5d753879a000b1ff39e77'
  • 22:54 logmsgbot: aaron synchronized php-1.24wmf5/includes/db/Database.php '8829ffc72d3332d348a1a2e58d525e54e126bad5'
  • 22:17 awight: tools updated from 85bb7293d83517086e3609f03365aecde9f58c71 to ee31fc94b17c11a48ddac19aabfcdaab69fd2f72
  • 21:37 logmsgbot: ori synchronized php-1.24wmf4/extensions/MultimediaViewer 'Update MultimediaViewer for I0df067a61: Add sampling to unsampled event logging'
  • 21:33 logmsgbot: ori synchronized php-1.24wmf5/extensions/MultimediaViewer 'Update MultimediaViewer for I0df067a61: Add sampling to unsampled event logging'
  • 21:10 mwalker: updating fundraising smashpig from 2fdf982b20f1cbeaf9f57af64ef21b5b69a36f6e to f64f79f13cf4ab560d0bb5bd69690c827a821629
  • 20:41 awight: update crm from 243641de631b712c4a29ca1f3618771b78dadeae to 0b8aa8aa046935b6cfc67c10ebe10396d5e42745
  • 18:43 awight: update tools from a40c0caa18a0efd93bc5d3f7f68386fbc36bf1fa to 85bb7293d83517086e3609f03365aecde9f58c71
  • 18:12 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I3c453b0949f4e: Tweak MediaViewer sampling settings'
  • 18:07 logmsgbot: ori synchronized wmf-config 'Ia43821231: Add sampling control setting for MediaViewer event'
  • 18:04 logmsgbot: ori updated /a/common to Ia43821231: Add sampling control setting for MediaViewer event logging
  • 17:47 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'GeoData to Elastic for all wikivoyages'
  • 17:11 mwalker: updating staging payments servers as well from 5e24b953dcff5305099e152139e6e93daba8aeec to d99177518b741e7fe18ffda86c83f93c72e164a6
  • 17:10 mwalker: and updated to 1.22.6
  • 17:10 mwalker: moved fundraising wiki from 6a1d4983319038edeb88dc34a1c220ecaec1cbde to d99177518b741e7fe18ffda86c83f93c72e164a6 -- including json i18n changes
  • 16:55 manybubbles: "in place" reindexing (for cirrus) all the wikipedias after the deploy train hit them yesterday
  • 16:53 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Removing old WikiEditor settings'
  • 16:44 RobH: partial zirconium downtime
  • 16:44 RobH: i logged into zirconium, but it had recovered by the time I checked it.
  • 16:33 mwalker: updated fundraising civicrm from 7a23465e620211739421cce3ad57c62597eb8cc3 to 75c1a50b8aa7e7b6f218d7c420932a8fc53a0a34 for an exchange rates fix
  • 16:26 qchris: updated gerrit's hooks-bugzilla plugin to version 2.8.1.2 to allow talking to bugzilla-4.4.4
  • 13:54 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1056 in s4, warm up'
  • 13:17 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'raise db1070 and db1071 to normal load'
  • 10:14 springle: xtrabackup clone db1049 to db1056
  • 10:13 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'reduce db1049 load while cloning'
  • 09:40 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'pool db1070 and db1071 in s1, warm up'
  • 06:20 logmsgbot: ori synchronized php-1.24wmf4/maintenance/compareParserCache.php 'Ica69a3ef2: Added a script to compare current parser output to cache (no impact on prod; syncing for consistency)'
  • 03:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 16 03:54:03 UTC 2014 (duration 54m 2s)
  • 03:09 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-16 03:08:04+00:00
  • 02:39 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-16 02:38:41+00:00
  • 02:07 springle: xtrabackup db1070 to db1071
  • 01:02 logmsgbot: ori synchronized php-1.24wmf5/includes/parser 'I12a60b5cc: Revert "Declare visibility on class properties of includes/parser/"'
  • 00:54 hoo: rebuildItemsPerSite finished running for Wikidata (after about 30h).
  • 00:32 hoo: manually ran rebuildEntityPerPage for Wikidata to fix 2 broken records
  • 00:07 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/131762/'
  • 00:00 logmsgbot: ori synchronized wmf-config/squid.php 'Id188979c1: Use whole subnets in squid.php list for XFF acceptance'

May 15

  • 23:57 logmsgbot: ori updated /a/common to Id188979c1: Use whole subnets in squid.php list for XFF acceptance
  • 23:38 logmsgbot: ori synchronized php-1.24wmf4/includes 'Ia3b12fb9: Speed up CIDR matching from $wgSquidServersNoPurge'
  • 23:19 logmsgbot: ori synchronized php-1.24wmf5/includes 'Ia3b12fb9: Speed up CIDR matching from $wgSquidServersNoPurge'
  • 23:05 logmsgbot: ori synchronized wmf-config/CirrusSearch-production.php 'Iae07852b1: Elasticsearch plugin juggling'
  • 22:54 logmsgbot: ori synchronized wmf-config 'I51a55c4e2, Ia6c01a913, I594848ce0, and I594848ce0'
  • 22:50 logmsgbot: ori updated /a/common to Ifae836de5: Swapping GeoData backend for enwikivoyage
  • 22:28 logmsgbot: ori synchronized php-1.24wmf4/extensions/EventLogging 'Update EventLogging to master for I89819bd943'
  • 22:26 logmsgbot: ori synchronized php-1.24wmf5/extensions/EventLogging 'Update EventLogging to master for I89819bd943'
  • 22:06 awight: updated tools from 93fda5da99674eca221e0abf53ad499583b27cfb to a40c0caa18a0efd93bc5d3f7f68386fbc36bf1fa
  • 22:04 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'GeoData using elasticsearch on enwikivoyage'
  • 21:04 awight: updated all tools from 47407c16d9922b17af70146416913abfe50b728d to 93fda5da99674eca221e0abf53ad499583b27cfb
  • 20:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'I5da4aa5db7b5d3c1843a6fd68d0a7c62a2bbfb4e'
  • 19:56 mwalker: updated fundraising tools repo for screenshots, worldpay auditing, live analysis, and... stomp! from 0eb485c8b6db5f06805976860bce7aa8b0d6444b to 47407c16d9922b17af70146416913abfe50b728d
  • 19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf5
  • 18:55 ori: deploying twemproxy module on mw106*, they may complain for a moment
  • 18:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf4
  • 18:46 logmsgbot: reedy Finished scap: testwiki to 1.24wmf5 and build l10n cache (duration: 27m 47s)
  • 18:32 mutante: mw1053 was already disabled in pybal though and RT 7408,7435
  • 18:31 mutante: mw1053 sits at disk partitioning dialog (via mgmt)
  • 18:29 Reedy: mw1053 is pingable but not ssh-able
  • 18:18 logmsgbot: reedy Started scap: testwiki to 1.24wmf5 and build l10n cache
  • 17:53 Jeff_Green: adjusted exim conf on mchenry to route donate.wm.o mail to barium instead of aluminium
  • 16:43 mwalker: disabled qc and put site_offline and maintenance_mode on civicrm to true
  • 15:20 logmsgbot: anomie synchronized php-1.24wmf4/extensions/MultimediaViewer 'SWAT: Deploy change 133446 to fix bug 65225 in MultimediaViewer'
  • 14:03 springle: xtrabackup clone db1056 to db1070
  • 13:59 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1056 while cloning'
  • 13:44 cmjohnson1: sodium going down again for a different disk replacement
  • 13:16 cmjohnson1: shutting down sodium to replace sdb
  • 12:56 godog: restarting gerrit on ytterbium, clones over https seemingly stuck
  • 12:24 manybubbles|away: "in place" reindexing group1 wikis after the deployment train updated cirrus yesterday. They'll need a full reindex after that is done which will take some time but is required to fix issues with redirects not showing up off of the main namespace
  • 11:56 godog: installed openjdk-7-jdk on ytterbium to attempt gerrit thread dump
  • 10:15 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1009 for raid tests'
  • 06:44 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'move s5 api traffic to db1005'
  • 05:19 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'move s4 commonswiki api traffic to db1042'
  • 04:20 springle: installed db1073
  • 03:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 15 03:14:04 UTC 2014 (duration 14m 3s)
  • 02:27 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-15 02:26:09+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-15 02:14:31+00:00

May 14

  • 23:42 logmsgbot: mwalker synchronized wmf-config/InitialiseSettings.php 'Poking settings to try and apply them'
  • 23:29 logmsgbot: mwalker synchronized visualeditor.dblist 'Another part of 132409 (visual editor)'
  • 23:27 K4-713: updated payments from 78cc4285bdeb6eecba3efc75e4a04c8b886561e4 to 5e24b953dcff5305099e152139e6e93daba8aeec
  • 23:27 logmsgbot: mwalker synchronized wmf-config/ 'SWAT of 132409 (visual editor) and 130274 (abuse filter)'
  • 22:04 logmsgbot: maxsem synchronized php-1.24wmf3/extensions/MobileFrontend/ 'bug 65042'
  • 22:03 marktraceur: cscott deployed a jenkins job change that pushes parsoid git files to beta-labs for version purposes
  • 22:03 logmsgbot: maxsem synchronized php-1.24wmf4/extensions/MobileFrontend/ 'bug 65042'
  • 20:38 awight: updated crm from 3fd3b94834f94529841ad4a695ecd73c98e487bc to 7a23465e620211739421cce3ad57c62597eb8cc3
  • 20:32 bd808: Restarting logstash on logstash1001.eqiad.wmnet due to missing messages from some (all?) logs
  • 19:58 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'No more LQT on wikimania2011wiki'
  • 18:32 Krinkle: integration-slave1001 had its 8GB / /dev/vda1 100% full. Purging /tmp/perf-*.map brought it back to 41%
  • 18:25 Krinkle: integration-slave1001 is having issues writing to disk
  • 17:50 logmsgbot: yurik synchronized php-1.24wmf4/extensions/ZeroRatedMobileAccess/
  • 17:47 logmsgbot: yurik synchronized php-1.24wmf3/extensions/ZeroRatedMobileAccess/
  • 17:30 logmsgbot: yurik synchronized wmf-config/CommonSettings.php
  • 15:44 chasemp: disabling puppet on tungsten to try tweaking carbon settings to affect queue drops (for the better)
  • 14:28 cmjohnson1: mw1053 going down for disk replacement
  • 13:27 bblack: restarting pybals on lvs300x
  • 12:30 _joe_: restarted uwsgi on tungsten
  • 09:46 mark: Started PyBal on lvs300* and established BGP sessions with the routers
  • 09:43 mark: Setup BGP configuration for lvs300* on cr1-esams and cr2-knams, with elevated MEDs to keep them as last resorts
  • 04:18 Tim: deploying apache configuration change with fixes
  • 03:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 14 03:10:36 UTC 2014 (duration 10m 35s)
  • 03:01 Tim: reverting apache change
  • 02:53 Tim: deploying apache configuration change https://gerrit.wikimedia.org/r/106109
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-14 02:25:08+00:00
  • 02:21 springle: upgrade db1043, rebuild as m3 master
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-14 02:13:23+00:00
  • 00:18 manybubbles: bouncing elasticsearch on elastic1015 to pick up gc logging configuration. it might warn but shouldn't cause any service disrubtion.

May 13

  • 23:27 logmsgbot: maxsem synchronized php-1.24wmf4/includes/exception/MWException.php 'https://gerrit.wikimedia.org/r/#/c/133184/'
  • 23:25 logmsgbot: maxsem synchronized php-1.24wmf3/includes/exception/MWException.php 'https://gerrit.wikimedia.org/r/#/c/133183/'
  • 23:21 MaxSem: Ran namespaceDupes after adding new namespaces to zhwikisource - no problems found
  • 23:18 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/127584'
  • 22:43 chasemp: ms-be1009 rebooted as it had locked up, swift seems to have recoverd
  • 22:35 mutante: created new gerrit projects for phabricator,arcanist and libphutil
  • 22:22 ori: restarting tungsten to verify fix for gdash/graphite initialization
  • 21:33 ori: gdash and graphite currently down; chase & ori debugging
  • 21:29 manybubbles: I caused elasticsearch1015 to drop out of the Elasticsearch cluster by tring to take a heap dump on it. don't do that. It stops the application for many seconds.
  • 19:07 logmsgbot: yurik synchronized php-1.24wmf3/extensions/ZeroRatedMobileAccess/
  • 19:02 logmsgbot: yurik synchronized php-1.24wmf4/extensions/ZeroRatedMobileAccess/
  • 18:49 cmjohnson1: replacing failed disk dataset1001
  • 18:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch for I1681addaed690b652822c0296b7a3e9b84de93b6'
  • 18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf4
  • 15:52 logmsgbot: anomie synchronized php-1.24wmf4/resources/Resources.php 'SWAT: Deploy jQuery Migrate to 1.24wmf4'
  • 15:51 logmsgbot: anomie synchronized php-1.24wmf4/resources/lib/jquery/jquery.migrate.js 'SWAT: Deploy jQuery Migrate to 1.24wmf4'
  • 04:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 13 04:07:53 UTC 2014 (duration 7m 52s)
  • 03:01 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-13 03:00:07+00:00
  • 02:31 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-13 02:30:33+00:00

May 12

  • 23:28 logmsgbot: yurik synchronized php-1.24wmf3/extensions/ZeroRatedMobileAccess/
  • 23:24 logmsgbot: yurik synchronized php-1.24wmf4/extensions/ZeroRatedMobileAccess/
  • 20:11 gwicke: deployed Parsoid d1c778ea3
  • 17:00 mutante: re-enabled mw1186 in pybal
  • 16:58 logmsgbot: manybubbles Finished scap: scapping again to get ms1186 synced up (duration: 00m 56s)
  • 16:57 logmsgbot: manybubbles Started scap: scapping again to get ms1186 synced up
  • 16:49 mutante: disabled mw1186 in pybal
  • 16:46 mutante: mw1186 - down, powercycling
  • 16:39 logmsgbot: manybubbles Finished scap: fix php symlink (duration: 04m 25s)
  • 16:35 logmsgbot: manybubbles Started scap: fix php symlink
  • 16:09 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php
  • 16:07 logmsgbot: manybubbles Finished scap: update visual editor for swat deploy (duration: 28m 58s)
  • 15:38 logmsgbot: manybubbles Started scap: update visual editor for swat deploy
  • 11:01 springle: killed bunch of slow Flow\Formatter\ContributionsQuery::queryRevisions queries on flowdb
  • 03:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 12 03:13:40 UTC 2014 (duration 13m 39s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-12 02:25:54+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-12 02:14:23+00:00

May 11

  • 15:26 logmsgbot: reedy synchronized php-1.24wmf3/thumb.php
  • 15:23 logmsgbot: reedy synchronized php-1.24wmf4/thumb.php
  • 14:21 cmjohnson1: power cycling asw-d5-eqiad
  • 03:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 11 03:10:37 UTC 2014 (duration 10m 36s)
  • 02:23 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-11 02:22:02+00:00
  • 02:13 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-11 02:12:49+00:00

May 10

  • 19:58 logmsgbot: hoo synchronized php-1.24wmf3/extensions/Wikidata/ 'Resyncing Wikidata for mw1122'
  • 18:31 hoo: approved an oauth request by Aaron Halfaker by making myself oauth admin for a moment
  • 17:36 logmsgbot: hoo synchronized php-1.24wmf4/extensions/Wikidata/ 'Update Wikidata to fix the JSON dump generation'
  • 17:35 logmsgbot: hoo synchronized php-1.24wmf3/extensions/Wikidata/ 'Update Wikidata to fix the JSON dump generation'
  • 17:11 marktraceur: pushed new uploadwizard qunit job to Jenkins
  • 16:30 logmsgbot: reedy updated /a/common to I415e67919: Memory limit to 235M
  • 15:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'I415e679197b97e2babe50544cf1e8c26c13a598a'
  • 13:59 logmsgbot: bsitu synchronized php-1.24wmf4/extensions/Flow 'Update Flow'
  • 13:46 Krinkle: Reloading Zuul to deploy I403760f1f6dd1bc2
  • 12:07 logmsgbot: hoo synchronized php-1.24wmf4/extensions/Wikidata/ 'Update Wikibase to fix performance issues with dumpJson'
  • 12:05 logmsgbot: hoo synchronized php-1.24wmf3/extensions/Wikidata/ 'Update Wikibase to fix performance issues with dumpJson (2nd run)'
  • 12:03 logmsgbot: hoo synchronized php-1.24wmf3/extensions/Wikidata/ 'Update Wikibase to fix performance issues with dumpJson'
  • 08:53 paravoid: rack D5 down, switch unresponsive; minimal impact (mw1201-1203, 1208-1210)
  • 06:56 logmsgbot: aaron synchronized php-1.24wmf3/img_auth.php '264967c58eccb6dae872ab7345d08f8381ac43a7'
  • 06:46 logmsgbot: aaron synchronized php-1.24wmf4/img_auth.php 'b08af402ef2de7b2c79f71d848c2b8ae98b47be0'
  • 03:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 10 03:12:27 UTC 2014 (duration 12m 26s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-10 02:25:31+00:00
  • 02:16 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-10 02:15:38+00:00
  • 00:02 awight: updated crm from 237d463ed3e275c217a4f497ed30d2f7f20100eb to 3fd3b94834f94529841ad4a695ecd73c98e487bc

May 9

  • 23:09 K4-713: updated payments cluster from 3be44f5d14c00a893a985f3aad86b6b59507a987 to 78cc4285bdeb6eecba3efc75e4a04c8b886561e4
  • 19:24 logmsgbot: reedy synchronized wmf-config/ 'I5265c408443212536a5ed96d910caba50c22e767'
  • 18:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable EducationProgram on ukwiki'
  • 18:52 logmsgbot: reedy updated /a/common to I0d1ea1639: Remove 1.23wmf13 through 1.23wmf20
  • 18:48 logmsgbot: reedy synchronized docroot and w
  • 18:44 logmsgbot: reedy updated /a/common to I25d891030: Do not optimize commons for new highlighter
  • 18:41 Reedy: Created EducationProgram tables on ukwiki
  • 15:45 manybubbles: reindexing commons to unbreak file searches on wikis not using the experimental highlighter
  • 15:43 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'Do not optimize commons for new highlighter on commons'
  • 15:03 manybubbles: rebuilding hewiki's cirrus index so it can pick up hebmorph too
  • 14:52 logmsgbot: spage synchronized php-1.24wmf4/extensions/Flow 'Fix Flow add new topics and reply in 1.24wmf4'
  • 13:40 logmsgbot: reedy updated /a/common to I721c36406: Add en-rtl to wgExtraLanguageNames for beta
  • 13:24 logmsgbot: reedy updated /a/common to I75a80a998: beta: create a RTL english wiki
  • 10:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'raise db1067 and db1066 to normal load. depool db1043'
  • 09:32 logmsgbot: hoo synchronized php-1.24wmf3/extensions/Wikidata/ 'Fix Job injection error handling'
  • 09:24 logmsgbot: hoo synchronized php-1.24wmf3/extensions/Wikidata/ 'Fix Job injection error handling'
  • 08:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'I4657fe64572fb3db22e3b48a87df7112b2248e35'
  • 08:21 hashar: apt-get upgraded apache on gallium and lanthanum
  • 08:18 hashar: Jenkins: un pooled integration-slave1001 and rebooting the instance.
  • 08:16 hashar: Jenkins: un pooled integration-slave1002 and rebooting the instance.
  • 08:16 hashar: restarting Zuul (seems some jobs are not properly registered)
  • 06:55 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'pool db1066 in s2, db1067 in s1, warm up'
  • 05:00 springle: installed db1067, db1070, db1071
  • 04:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 9 04:08:07 UTC 2014 (duration 8m 6s)
  • 03:47 springle: xtrabackup clone db1036 to db1067, db1051 to db1066
  • 03:46 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'reduce db1036 and db1051 load while cloning'
  • 03:24 springle: installed db106[678]
  • 03:09 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-09 03:08:55+00:00
  • 02:41 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-09 02:40:06+00:00
  • 00:03 logmsgbot: maxsem synchronized php-1.24wmf3/extensions/MobileFrontend/ 'bug 65042'
  • 00:01 logmsgbot: maxsem synchronized php-1.24wmf4/extensions/MobileFrontend/ 'bug 65042'

May 8

  • 23:11 logmsgbot: maxsem synchronized php-1.24wmf3/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/132299'
  • 23:08 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'Replica count for commonswiki_file -- syncing with whats already live'
  • 23:07 logmsgbot: maxsem synchronized php-1.24wmf4/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/132299'
  • 21:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgVectorBetaPersonalBar to true for all wikis'
  • 21:28 logmsgbot: reedy updated /a/common to I44f67444c: group0 to 1.24wmf4
  • 20:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf3 and group0 to 1.24wmf4
  • 20:32 logmsgbot: reedy updated /a/common to I11e5ca294: FUTURE: Fifth batch of pilot sites for Media Viewer
  • 20:05 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'touch'
  • 20:04 logmsgbot: demon synchronized mediaviewer.dblist 'mediaviewer for svwiki, eswiki, jawiki, ptwiki'
  • 18:56 DJ-K4: enabled fredge queue consumption + jenkins job
  • 17:10 logmsgbot: reedy Finished scap: Build l10n cache for 1.24wmf4 and move testwiki (duration: 17m 05s)
  • 16:53 logmsgbot: reedy Started scap: Build l10n cache for 1.24wmf4 and move testwiki
  • 16:52 chasemp: rebooted ms-be1006 since it dropped dead
  • 16:52 logmsgbot: reedy Finished scap: Build l10n cache for 1.24wmf4 and move testwiki (duration: 11m 42s)
  • 16:42 manybubbles: reindexing the hebrew wikis other then hewikipedia now that they are on wmf3 so they can have hebmorph
  • 16:40 logmsgbot: reedy Started scap: Build l10n cache for 1.24wmf4 and move testwiki
  • 16:39 manybubbles: rebuilding enwiki's cirrus index to optimize for new highlighter
  • 16:14 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to wmf3
  • 16:13 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'raise db106[45] to normal load'
  • 15:13 logmsgbot: manybubbles synchronized php-1.24wmf3/extensions/CirrusSearch/ 'updating Cirrus to pick up some fixes'
  • 15:08 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'engage new hightlighter on some more wikis'
  • 15:00 manybubbles: rebuilding all hebrew wikis _except_ hebrew wikipedia and hebrew wikisource to pick up hebmorph. hewikisource got it this morning. hewiki will get it this afternoon after the deployment train
  • 13:24 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1064 in s4, db1065 in s1'
  • 12:59 manybubbles: rebuilding cirrus index for hewikisource to pick up hebmorph
  • 10:18 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'reduce db1049 and db1051 load while cloning'
  • 10:16 springle: xtrabackup clone db1051 to db1065
  • 09:35 springle: xtrabackup clone db1049 to db1064
  • 08:55 springle: installed db106[45]
  • 08:35 logmsgbot: reedy synchronized docroot and w
  • 08:18 logmsgbot: reedy synchronized php-1.24wmf4 'staging'
  • 08:01 logmsgbot: reedy updated /a/common to I7f2d2b25d: Allow all users on OfficeWiki to send mass messages
  • 03:44 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 8 03:43:39 UTC 2014 (duration 43m 38s)
  • 02:57 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-08 02:56:02+00:00
  • 02:29 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-08 02:28:17+00:00

May 7

  • 23:36 logmsgbot: demon Finished scap: no-op scap, for ori (duration: 09m 01s)
  • 23:27 logmsgbot: demon Started scap: no-op scap, for ori
  • 23:17 logmsgbot: ori Finished scap: No changes; testing scap to osmium (again) (duration: 03m 25s)
  • 23:14 logmsgbot: ori Started scap: No changes; testing scap to osmium (again)
  • 23:14 logmsgbot: ori scap aborted: No changes; testing scap to osmium (duration: 11m 27s)
  • 23:06 awight: update crm from 1740219e38091ba4f7afe6545ea189b27340bf86 to 237d463ed3e275c217a4f497ed30d2f7f20100eb
  • 23:02 logmsgbot: ori Started scap: No changes; testing scap to osmium
  • 21:37 K4-713: updated payments from a7fa0d64da2c56586c83cf92babb65bac857be2e to 3be44f5d14c00a893a985f3aad86b6b59507a987
  • 21:00 K4-713: synchronized payments from 4811f6d3d80d126c to a7fa0d64da2c56586
  • 20:49 K4-713: revlocked payments to 4811f6d3d80d126c due to strange errors during an attempted deploy of a7fa0d64da2c56586
  • 20:17 awight: drush pm-uninstall wmf_fredge_qc
  • 20:16 awight: bad call: drush en wmf_fredge_qc -- need to rollback the module schema version and try again with "fredge" creds
  • 20:14 awight: crm updated from cfe34fe0b10861167199a8f72bba279b9cac5e6e to 1740219e38091ba4f7afe6545ea189b27340bf86
  • 20:07 subbu: deployed parsoid 71f4e884 (with deploy sha 9a62899d)
  • 17:19 logmsgbot: yurik synchronized php-1.24wmf2/extensions/ZeroRatedMobileAccess/
  • 17:16 logmsgbot: yurik synchronized php-1.24wmf3/extensions/ZeroRatedMobileAccess/
  • 16:33 manybubbles: performing a rolling restart on elasticsearch nodes in production to pick up new plugins: experimental-highlight 0.0.8 and analysis-hebrew 1.1.0
  • 16:30 _joe_: restarted mwprof/profiler-to-carbon on tungsten, stuck somehow
  • 15:40 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'Raised redundancy for commonswiki_file back up, config to match'
  • 15:27 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'SWAT: Remove obsolete $wmgCirrusIsBuilding (no functionality change)'
  • 15:26 logmsgbot: anomie synchronized wmf-config/CirrusSearch-common.php 'SWAT: Remove obsolete $wmgCirrusIsBuilding (no functionality change)'
  • 15:19 anomie: anomie namespaceDupes.php on OfficeWiki done (that was quick)
  • 15:18 anomie: anomie Running maintenance/namespaceDupes.php on OfficeWiki
  • 15:16 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'SWAT: Change wgMetaNamespace for OfficeWiki and add alias'
  • 15:12 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'SWAT: Allow all users on OfficeWiki to send mass messages (for real this time)'
  • 15:09 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'SWAT: Allow all users on OfficeWiki to send mass messages'
  • 15:03 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'SWAT: Set $wgUploadMissingFileUrl for enwiki'
  • 10:21 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'db1049 to full steam'
  • 09:40 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1049 in s4'
  • 07:54 hashar: Jenkins: installing Claim plugin (allow folks to comment on builds and mark them)
  • 06:48 springle: again
  • 05:59 springle: powercycled unresponsive neon, swapdeath + oom killer
  • 03:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 7 03:11:36 UTC 2014 (duration 11m 35s)
  • 02:28 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-07 02:27:23+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-07 02:14:44+00:00
  • 00:02 mutante: upgrading libtiff on imagescalers

May 6

  • 23:18 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/131855'
  • 21:57 mutante: gracefull'ing apaches
  • 20:43 logmsgbot: demon synchronized php-1.24wmf3/extensions/CirrusSearch 'Rolling Cirrus back to known-good state'
  • 20:42 logmsgbot: demon synchronized php-1.24wmf2/extensions/CirrusSearch 'Rolling Cirrus back to known-good state'
  • 20:27 logmsgbot: demon synchronized php-1.24wmf3/extensions/CirrusSearch/includes/Hooks.php 'Fix typehinting'
  • 20:26 logmsgbot: demon synchronized php-1.24wmf2/extensions/CirrusSearch/includes/Hooks.php 'Fix typehinting'
  • 20:19 logmsgbot: demon synchronized php-1.24wmf3/extensions/CirrusSearch/CirrusSearch.php 'I2638b695: fix for page moves'
  • 20:19 logmsgbot: demon synchronized php-1.24wmf2/extensions/CirrusSearch/CirrusSearch.php 'I2638b695: fix for page moves'
  • 20:18 logmsgbot: demon synchronized php-1.24wmf3/extensions/CirrusSearch/includes/Hooks.php 'I2638b695: fix for page moves'
  • 20:17 logmsgbot: demon synchronized php-1.24wmf2/extensions/CirrusSearch/includes/Hooks.php 'I2638b695: fix for page moves'
  • 19:00 mutante: enabling puppet on netmon1001
  • 17:38 hoo: Changed email for global account "ElphiBot".
  • 16:48 logmsgbot: mattflaschen synchronized php-1.24wmf3/extensions/GettingStarted/ 'GettingStarted token and logging deployment'
  • 16:45 logmsgbot: mattflaschen synchronized php-1.24wmf2/extensions/GettingStarted/ 'GettingStarted token and logging deployment'
  • 16:19 hoo: Changed email for global account "Elph".
  • 15:42 akosiaris: killed zuul processes on gallium and restarted the service
  • 09:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 6 09:15:20 UTC 2014
  • 09:08 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Tue May 6 09:07:45 UTC 2014
  • 09:01 logmsgbot: faidon synchronized wmf-config/squid.php 'add Swift to squid.php'
  • 08:59 logmsgbot: faidon updated /a/common to Ica9086dcd: Add Swift frontends to squid.php
  • 08:45 ottomata: re-enabling puppet agent on analytics1022; kafka broker is caught up there and is fully in all ISRs
  • 08:36 ottomata: temporarily disabling puppet on analytics1026 to troubleshoot a camus import problem
  • 06:55 springle: hammering dbstore1001 with dumps in screen session. ignore replag
  • 06:31 ori: ..on vanadium
  • 06:31 ori: deleting rotated logs in /var/log/eventlogging/archive that are older than 90 days
  • 04:47 springle: mydumper/myloader clone db1042 to db1049
  • 04:06 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1049 for maintenance'
  • 03:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 6 03:47:41 UTC 2014 (duration 47m 40s)
  • 03:03 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-06 03:02:03+00:00
  • 02:28 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-06 02:27:21+00:00
  • 00:01 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'Update wgFlowCacheVersion to 4.2'

May 5

  • 23:57 logmsgbot: ori updated /a/common to Id1f2e0acf: Drop wgFlowCacheKey from CommonSettings.php
  • 23:54 logmsgbot: ori Finished scap: SWAT deploy for VisualEditor and Flow cherry-picks (duration: 09m 55s)
  • 23:44 logmsgbot: ori Started scap: SWAT deploy for VisualEditor and Flow cherry-picks
  • 23:26 logmsgbot: ori synchronized php-1.24wmf3/extensions/EventLogging 'Update EventLogging for Id23b37fbe for SWAT.'
  • 23:23 logmsgbot: ori synchronized php-1.24wmf2/extensions/EventLogging 'Update EventLogging for Id23b37fbe for SWAT.'
  • 21:07 jgage: trying on analytics1022: https://wikitech.wikimedia.org/wiki/Analytics/Kraken/Kafka/Administration#Recovering_a_laggy_broker_replica
  • 20:58 RobH: ssl1001-1003 now have updated unified cert in service
  • 20:58 jgage: both kafka brokers back in service
  • 20:54 RobH: cp4001-4020 unified cert and nginx service reloaded, back in service
  • 20:50 RobH: ssl1006 and ssl1009 are responsive to nginx and back in service
  • 20:43 RobH: pybal
  • 20:43 RobH: ssl1009 was refusing connections both before and after my ssl cert update. ssl1006 is presently refusing connections post update. they are set to disabled in pubal
  • 20:39 RobH: ssl1008 back into service, ssl1009 already depooled
  • 20:38 jgage: forced kafka broker reelection
  • 20:34 RobH: ssl1007 going back into service, ssl1008 depooling
  • 20:25 RobH: depooled ssl1006/7 for update
  • 20:25 RobH: ssl1004/5 returned to service (and puppet agents enabled)
  • 20:21 RobH: puppet agent has been re-enabled on ssl1001-1003
  • 20:19 RobH: ssl1004/5 disabled for update
  • 20:18 RobH: putting ssl1002/3 back into service
  • 20:15 subbu: deployed parsoid f2f1f1d7 (with deploy sha 71072f8a)
  • 19:58 RobH: ssl1001 back in service, ssl1002-1003 set to disabled in pybal
  • 19:18 RobH: depooling ssl1001 to test new certs live on system
  • 19:09 RobH: disabled puppet on cp40XX, ssl10XX, and ssl30XX
  • 19:08 logmsgbot: bblack synchronized wmf-config/squid.php 'REVERT: Update wgSquidServersNoPurge to use whole subnets for XFF checking'
  • 19:07 logmsgbot: bblack updated /a/common to Iaf4d57d54: Revert "Use whole subnets in squid.php list for XFF acceptance"
  • 19:03 logmsgbot: bblack synchronized wmf-config/squid.php 'Update wgSquidServersNoPurge to use whole subnets for XFF checking'
  • 19:01 logmsgbot: bblack updated /a/common to I5a2d86ef0: Use whole subnets in squid.php list for XFF acceptance
  • 17:05 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Revert "Increased htmlCacheUpdate throttle"'
  • 16:00 logmsgbot: anomie synchronized php-1.24wmf3/extensions/MobileFrontend/ 'SWAT: Backport change 131237 to 1.24wmf3 to fix bug in MobileFrontend'
  • 15:59 logmsgbot: anomie synchronized php-1.24wmf2/extensions/MobileFrontend/ 'SWAT: Backport change 131237 to 1.24wmf2 to fix bug in MobileFrontend'
  • 15:49 logmsgbot: anomie synchronized php-1.24wmf2/includes/specials/SpecialAllmessages.php 'SWAT: Backport change 131041 to 1.24wmf2 to fix bug in Special:AllMessages'
  • 15:37 logmsgbot: anomie synchronized php-1.24wmf2/includes/specials/SpecialAllmessages.php 'SWAT: Backport change 131041 to 1.24wmf2 to fix bug in Special:AllMessages'
  • 15:24 logmsgbot: anomie synchronized php-1.24wmf3/includes/specials/SpecialAllmessages.php 'SWAT: Backport change 131041 to 1.24wmf3 to fix bug in Special:AllMessages'
  • 15:12 logmsgbot: anomie synchronized php-1.24wmf3/includes/api/ApiLogin.php 'SWAT: Backport change 131056 to 1.24wmf3 to fix bug 64727'
  • 15:10 logmsgbot: anomie synchronized php-1.24wmf2/includes/api/ApiLogin.php 'SWAT: Backport change 131056 to 1.24wmf2 to fix bug 64727'
  • 12:45 akosiaris: removing various sdtpa devices from LibreNMS
  • 03:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 5 03:11:15 UTC 2014 (duration 11m 14s)
  • 02:32 ^demon|away: [gitb]lit's wonkiness but they're certainly not helping matters.
  • 02:32 ^demon|away: antimony: ran very very aggressive repacking on mediawiki/core, operations/puppet, mediawiki/extensions/{UploadWizard,CentralAuth,CentralNotice,DonationInterface,FlaggedRevs,AbuseFilter,BlueSpiceExtensions,Translate,WikimediaMessages,EducationProgram,UniversalLanguageSelector,Wikibase}, pywikibot/{core,compat}, operations/dumps/tests. Basically anything taking up >90MB on disk. Probably not the cause of gitb
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-05 02:25:34+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-05 02:13:18+00:00

May 4

  • 20:57 logmsgbot: aaron synchronized php-1.24wmf3/thumb.php 'c5ebd2aefce9e3fc5b994053078754021176f411'
  • 20:40 logmsgbot: aaron synchronized php-1.24wmf3/thumb.php '6c230cbbc6ffa4d8909e88961ebf75755cf9c9d9'
  • 19:24 logmsgbot: ori updated /a/common to I2916ef3bd: labs: stream recent changes to redis
  • 09:58 _joe_: restarted gitblit, stuck on GC as usual.
  • 08:40 _joe_: restarted apache on tungsten as it was stuck communicating with uwsgi
  • 03:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 4 03:08:41 UTC 2014 (duration 8m 40s)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-04 02:25:06+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-04 02:12:57+00:00

May 3

  • 19:41 logmsgbot: hoo synchronized wmf-config/ 'Documentation only change'
  • 06:02 ori: disabled puppet on osmium to test hhvm build
  • 03:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 3 03:09:40 UTC 2014 (duration 9m 39s)
  • 02:27 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-03 02:26:41+00:00
  • 02:15 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-03 02:14:43+00:00

May 2

  • 23:13 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Made private wikis use thumb_handler.php for thumbnails'
  • 22:30 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Removed useless "handlerUrl" config'
  • 21:21 hashar: Jenkins is back :-]
  • 21:14 hashar: restarting Jenkins (making sure the java process properly disappear)
  • 21:14 hashar: zuul jenkins stuck again :(
  • 20:31 logmsgbot: krinkle synchronized php-1.24wmf3/resources/Resources.php 'Ia12998fb11c686'
  • 20:29 logmsgbot: krinkle synchronized php-1.24wmf2/resources/Resources.php 'Ia12998fb11c686'
  • 17:47 RoanKattouw: Restarting stuck jenkins
  • 16:04 _joe_: depooled mw1053 for hardware problems
  • 15:13 andrewbogott: resetting a bunch more UIDs. Running find-and-chown again, but this time not on the swifts: salt -E '^(?!ms-be|labstore|snapshot).*$'
  • 13:46 paravoid: swift @ eqiad: setting zone 5 (ms-be1013/1014/1015) to weight 2000, i.e. 66%
  • 12:06 hashar: updated our Jenkins Job Builder copy abbf318..8df6bab
  • 06:17 ori: re-enabled puppet on osmium and hafnium
  • 04:02 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 2 04:01:04 UTC 2014 (duration 1m 3s)
  • 03:10 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-02 03:09:16+00:00
  • 02:40 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-02 02:39:39+00:00

May 1

  • 23:43 bd808: Restarted logstash on logstash1001; MaxSem noticed that many recursion-guard logs were not being completely reassembled and JVM had one CPU maxed out.
  • 23:08 logmsgbot: maxsem synchronized php-1.24wmf2/extensions/CommonsMetadata/ 'https://gerrit.wikimedia.org/r/#/c/130971/'
  • 22:58 paravoid: disabling puppet on holmium; manually overriding completely broken varnish config
  • 20:10 bd808: Deployed scap 92ea0e9 via trebuchet (not actively used yet)
  • 19:17 logmsgbot: reedy synchronized wmf-config/
  • 18:55 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf3
  • 18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf2
  • 18:07 manybubbles: upgrading highlighter plugin on elasticsearch machines - the cluster will go yellow for a few hours during the rolling restart
  • 16:43 logmsgbot: reedy Finished scap: testwiki to 1.24wmf3 (duration: 29m 54s)
  • 16:18 subbu: deployed parsoid 5e05c585 (with deploy sha ca2db96d)
  • 16:16 ottomata: reinstalling elastic1008
  • 16:13 logmsgbot: reedy Started scap: testwiki to 1.24wmf3
  • 16:10 logmsgbot: reedy updated /a/common to I832b45db6: Correct a domain in wgCopyUploadsDomains
  • 16:01 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Enable cirrus as a betafeature on all wikis which did not already have it.'
  • 15:51 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'SWAT fix GWtoolset url and add some more logos'
  • 15:40 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'SWAT fix GWtoolset url and add some more logos'
  • 15:35 andrewbogott: reassigning a ton of UIDs in production; running a couple dozen 'find' commands to chown files
  • 15:34 logmsgbot: manybubbles synchronized php-1.24wmf2/includes/Article.php 'SWAT update to prevent fatal in backwards compatibility method'
  • 15:27 logmsgbot: manybubbles synchronized php-1.24wmf2/extensions/VisualEditor/ 'SWAT update for firefox focus'
  • 15:08 logmsgbot: manybubbles synchronized php-1.24wmf2/extensions/Wikidata/ 'SWAT update for time parsing and formatting'
  • 08:15 springle: switching s1-analytics-slave db1047 enwiki to tokudb
  • 03:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 1 03:17:39 UTC 2014 (duration 17m 38s)
  • 02:34 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-01 02:33:34+00:00
  • 02:22 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-05-01 02:21:44+00:00

April 30

  • 23:58 bblack: mobile caches now sync zero carriers/proxies from zero.wm.org rather than noc(fenari) temp hack solution
  • 23:14 logmsgbot: ori synchronized php-1.24wmf2/extensions/VisualEditor 'Ibaf0cc823bfe: Update VisualEditor for cherry-picks'
  • 20:34 logmsgbot: yurik synchronized php-1.24wmf2/extensions/ZeroRatedMobileAccess/
  • 20:31 logmsgbot: yurik synchronized php-1.24wmf1/extensions/ZeroRatedMobileAccess/
  • 18:08 logmsgbot: yurik synchronized wmf-config/mobile.php
  • 17:37 yurikR: yurik Added $wmgZeroRatedMobileAccessApiUserName / password
  • 17:36 logmsgbot: yurik synchronized wmf-config/PrivateSettings.php
  • 17:17 logmsgbot: yurik synchronized php-1.24wmf2/extensions/ZeroRatedMobileAccess/
  • 17:11 logmsgbot: yurik synchronized php-1.24wmf1/extensions/ZeroRatedMobileAccess/
  • 17:10 logmsgbot: aaron synchronized php-1.24wmf2/thumb.php '93a33d733fa81a9a5396083ded6aa28a74f08a98'
  • 17:07 logmsgbot: aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Removed "TMHTransformFrame" pool counter entry'
  • 16:19 Krinkle: restarting stuck Jenkins
  • 16:16 Krinkle: Deploying Id248bd6706f32a on Zuul and reloading service
  • 09:29 springle: dbstore100[12] replicating m2 eventlogging
  • 06:54 logmsgbot: aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Removed GetLocalFileCopy pool counter entry'
  • 03:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 30 03:23:13 UTC 2014 (duration 23m 12s)
  • 02:33 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-04-30 02:33:12+00:00
  • 02:22 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-30 02:22:23+00:00
  • 01:58 Krinkle: Deploying Ia82779635d762a3 on Zuul, reloading services
  • 00:02 logmsgbot: ebernhardson synchronized php-1.24wmf1/extensions/MultimediaViewer/ 'I684d44a0b5'

April 29

  • 23:59 logmsgbot: ebernhardson synchronized php-1.24wmf2/extensions/Wikidata/ 'I84c2283e07'
  • 23:52 logmsgbot: ebernhardson synchronized php-1.24wmf2/extensions/MultimediaViewer/ 'I84f8e347f'
  • 23:37 logmsgbot: ori synchronized php-1.24wmf2/skins 'I66c56c577bad'
  • 23:37 logmsgbot: ori synchronized php-1.24wmf2/extensions/VisualEditor 'I5818dce62'
  • 23:25 logmsgbot: ebernhardson synchronized wmf-config/InitialiseSettings.php 'Enable MediaViewer survey on Spanish and Dutch Wikipedia'
  • 23:22 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I59e1fa87e: Include language-0 categories for betawikiversity'
  • 23:21 logmsgbot: ori updated /a/common to I59e1fa87e: Include language-0 categories for betawikiversity
  • 23:16 logmsgbot: ebernhardson synchronized php-1.24wmf2/extensions/WikiEditor 'Update WikiEditor to 1.24wmf2'
  • 21:59 logmsgbot: spage synchronized wmf-config/InitialiseSettings.php 'enable Flow on two mw talk pages for James_F'
  • 21:25 Krinkle: Running deleteEqualMessages.php on newwiki (bug 43917)
  • 21:12 AaronSchulz: populateImageSha1 fixer script finished on all wikis
  • 20:47 manybubbles: rebuilding search indexes for group1 wikis after the train upgraded cirrus for them
  • 20:13 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I56ae921ca: Change group name on the Persian Wikipedia'
  • 20:13 logmsgbot: ori updated /a/common to I56ae921ca: Change group name on the Persian Wikipedia
  • 18:16 logmsgbot: reedy synchronized multiversion/
  • 18:09 logmsgbot: reedy synchronized wmf-config/ 'I52293b29a87e2c645735b37215e4113e561e47da'
  • 18:04 logmsgbot: reedy synchronized docroot and w
  • 18:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf2
  • 17:13 manybubbles: raising number of replicas of enwiki's cirrus index from 1 to 2. cluster will probably complain while they allocate
  • 16:53 RobH: osmium install complete, ticket resolved, ready for ^d and ori to take over
  • 16:29 logmsgbot: reedy updated /a/common to Idb2a86791: Increased htmlCacheUpdate throttle
  • 16:18 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Increased htmlCacheUpdate throttle'
  • 16:05 logmsgbot: manybubbles synchronized php-1.24wmf2/extensions/Wikidata/ 'SWAT upgrade wikidata for date parsing fixes'
  • 15:47 manybubbles: rebuilding test2wiki's cirrus index after swat deploy
  • 15:45 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'SWAT add autopatrolled group to shwiktionary and draft namespace to chapcomwiki'
  • 15:38 logmsgbot: manybubbles synchronized wmf-config/CirrusSearch-common.php 'SWAT deploy - move group0 wikis to experimental highlighter and give enwiki its redundency back'
  • 15:37 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'SWAT deploy - extra setting for cirrus and new groups and sources for gwtoolset'
  • 15:32 manybubbles: cirrus deploys look good, moving on to twkozlowski's requests
  • 15:31 logmsgbot: manybubbles synchronized wmf-config/CirrusSearch-common.php 'SWAT deploy - move group0 wikis to experimental highlighter and give enwiki its redundency back'
  • 15:31 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'SWAT deploy - extra setting for cirrus'
  • 15:14 logmsgbot: manybubbles synchronized php-1.24wmf2/extensions/CirrusSearch/ 'SWAT upgrade - improves as yet undeployed highlighter config'
  • 14:40 logmsgbot: faidon synchronized php-1.24wmf2/extensions/GettingStarted/ 'Revert GettingStarted anon tokens'
  • 14:39 logmsgbot: faidon synchronized php-1.24wmf1/extensions/GettingStarted/ 'Revert GettingStarted anon tokens'
  • 13:45 Krinkle: Running deleteEqualMessages.php on lnwiki (bug 43917)
  • 13:41 logmsgbot: bblack synchronized wmf-config/InitialiseSettings.php 'Revert "Unset $wgUseXVO"'
  • 13:10 Krinkle: Running deleteEqualMessages.php on nlwiki (bug 43917)
  • 12:35 Krinkle: Running deleteEqualMessages.php on cswikiversity (bug 43917)
  • 12:08 Krinkle: Running deleteEqualMessages.php on cswiktionary (bug 43917)
  • 11:31 logmsgbot: bblack synchronized wmf-config/InitialiseSettings.php 'Revert "Unset $wgUseXVO"'
  • 10:34 akosiaris: restarted gitblit on antimony
  • 10:20 hashar: restarting Zuul
  • 10:14 hashar: Jenkins / Zuul : upgrading python-gear from 0.4.0-1 to 0.5.4-1 . Should fix a bunch of jobs registrations issues in Zuul Gearman. bug 63760
  • 09:59 akosiaris: update python-gear on apt.wikimedia.org to 0.5.4-1
  • 08:30 akosiaris: Published carbon's IPv6 address in DNS. apt.wikimedia.org and ubuntu.wikimedia.org are now IPv6 enabled
  • 05:25 AaronSchulz: Manually removed a few 10000s of duplicate Cyberbot job duplicates
  • 03:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 29 03:48:29 UTC 2014 (duration 48m 28s)
  • 03:02 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-04-29 03:02:55+00:00
  • 02:37 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-29 02:37:21+00:00
  • 02:24 bblack: wiped disk cache (via mkfs) on cp1055 to (hopefully) clear crash-restart cycle, backend back in service now
  • 01:53 Krinkle: Running deleteEqualMessages.php on cswiki (bug 43917)
  • 00:49 Tim: on cp1055: backend varnish is continually panicking and restarting its child, will try to stop/start service
  • 00:22 logmsgbot: aaron synchronized php-1.24wmf1/includes/WikiPage.php '3505cf933d874ea44bd5a3f3ffe210598ef7eec2'
  • 00:14 logmsgbot: aaron synchronized php-1.24wmf2/includes/WikiPage.php '119fd9fc17b3c309b9065b54f4c83ede7d20498b'
  • 00:00 logmsgbot: mwalker Finished scap: SWAT for 129813, 129640, 129708, 129707, and 130246 (duration: 11m 37s)

April 28

  • 23:49 logmsgbot: mwalker Started scap: SWAT for 129813, 129640, 129708, 129707, and 130246
  • 23:26 logmsgbot: aaron synchronized php-1.24wmf2/maintenance/runJobs.php '91dddcaffa58430204e2bf3c612d893b2710f33b'
  • 22:43 jgage: rebooting db1047 due to unpingable and unresponsive on mgmt console
  • 22:28 logmsgbot: mflaschen synchronized php-1.24wmf2/extensions/GettingStarted/ 'Revert token/TrackedPageContentSaveComplete GettingStarted change'
  • 22:28 logmsgbot: mflaschen synchronized php-1.24wmf1/extensions/GettingStarted/ 'Revert token/TrackedPageContentSaveComplete GettingStarted change'
  • 20:47 Krinkle: Running deleteEqualMessages.php on sqwiki (bug 43917)
  • 20:38 paravoid: apache-graceful-all after tuning php.ini's expose_php setting
  • 20:12 logmsgbot: reedy synchronized wmf-config/db-labs.php
  • 20:08 apergos: restarted gmetad on nickel
  • 20:02 gwicke: deployed Parsoid cab9348e using deploy 9e9030d
  • 20:00 logmsgbot: mflaschen synchronized wmf-config/InitialiseSettings.php 'Update GettingStarted config for new format'
  • 19:59 logmsgbot: mflaschen synchronized php-1.24wmf2/extensions/GettingStarted/ 'Sync GettingStarted for Growth team deploy'
  • 19:58 logmsgbot: mflaschen synchronized php-1.24wmf1/extensions/GettingStarted/ 'Sync GettingStarted for Growth team deploy'
  • 19:30 Krinkle: Running deleteEqualMessages.php on simplewiki (bug 43917)
  • 19:25 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I5e0709ef0: Unset $wgUseXVO'
  • 19:25 logmsgbot: ori updated /a/common to I5e0709ef0: Unset $wgUseXVO
  • 19:22 Krinkle: Running deleteEqualMessages.php on rowiktionary (bug 43917)
  • 19:01 Krinkle: Running deleteEqualMessages.php on bat-smgwiki (bug 43917)
  • 18:48 Krinkle: Running deleteEqualMessages.php on afwikiquote (bug 43917)
  • 18:41 hashar: Jenkins disconnected lanthanum slave, killed all jenkins-slave process on it and repooled server.
  • 18:39 Krinkle: Running deleteEqualMessages.php on abwiki (bug 43917)
  • 17:54 manybubbles: deploying a new version of our Elasticsearch highlighter by doing a rolling restart on Elasticsearch machines - should cause no interruption of service
  • 16:51 akosiaris: executed graceful-stop, start for apaches in order to load the new php-luasandbox apache module
  • 15:08 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Add new sources to gwtoolset and namespaces to hewikisource'
  • 12:48 _joe_: restarted apache on wikitech-static
  • 12:29 Krinkle: Running deleteEqualMessages.php on cvwiki (bug 43917)
  • 12:29 Krinkle: Running deleteEqualMessages.php on afwiki (bug 43917)
  • 11:46 Krinkle: Running deleteEqualMessages.php on bpywiki (bug 43917)
  • 11:00 springle: completed schema change, bug 64411, page_props.pp_sortkey
  • 08:49 springle: reloading db1046 from fresh m2 dump
  • 03:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 28 03:12:11 UTC 2014 (duration 12m 10s)
  • 03:00 springle: starting online schema change, bug 64411, page_props.pp_sortkey
  • 02:30 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-04-28 02:30:04+00:00
  • 02:22 springle: powercycle db1046 unresponsive
  • 02:20 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-28 02:20:39+00:00

April 27

  • 14:49 paravoid: stopping pybal on lvs300[1-4] to avoid the logspam
  • 05:31 springle: mariadb sql dump in progress db1048 /a for rebuilding db1046. ok to kill if necessary
  • 03:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 27 03:08:01 UTC 2014 (duration 8m 0s)
  • 02:28 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-04-27 02:28:32+00:00
  • 02:19 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-27 02:19:43+00:00

April 26

  • 23:27 logmsgbot: aaron synchronized php-1.24wmf2/includes/profiler/Profiler.php '7e20cdd2ba0381b81d3b43c8743fa4202a76bd61'
  • 13:43 logmsgbot: hoo synchronized wmf-config/InitialiseSettings-labs.php 'Syncing for cluster consistency'
  • 13:42 logmsgbot: hoo updated /a/common to Ic98928d54: Have Commons on Beta Labs use $stdlogo
  • 13:26 springle: db1016 xfs head behind tail. reverted to last snapshot volume
  • 12:57 springle: powercycle db1016 unresponsive
  • 03:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 26 03:11:16 UTC 2014 (duration 11m 15s)
  • 02:31 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-04-26 02:31:41+00:00
  • 02:22 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-26 02:22:41+00:00

April 25

  • 20:29 logmsgbot: mwalker synchronized php-1.24wmf2/extensions/WikiEditor/ 'Reverting some faulty WikiEditor code for bug 64289'
  • 20:28 logmsgbot: mwalker synchronized php-1.24wmf1/extensions/WikiEditor/ 'Reverting some faulty WikiEditor code for bug 64289'
  • 18:02 K4-713: adjusted antifraud filters on payments
  • 17:08 Jeff_Green: reenabled puppet and notifications for iodine
  • 16:22 manybubbles: Elasticsearch rolling restart complete.
  • 14:46 Jeff_Green: disabled icinga notifications for iodine too...
  • 14:44 Jeff_Green: puppet stopped on iodine, doing manual spamassassin training
  • 12:58 springle: upgrading db1047 (analytics slave) to mariadb 10
  • 12:28 manybubbles: Performing rolling restart of Cirrus's Elasticsearch servers to upgrade a plugin. Low risk because it won't be used by the general public until Mondayish so a Friday push should be ok.
  • 12:07 ottomata: stopping puppet on analytics1026 to test more frequent runs of Camus
  • 12:02 logmsgbot: reedy synchronized wmf-config/
  • 11:35 logmsgbot: reedy synchronized docroot and w
  • 09:41 logmsgbot: reedy synchronized wmf-config/
  • 09:15 logmsgbot: reedy synchronized wmf-config/ 'I4a68dc8321b7b302f5e89b5adafcff096f2ac35b'
  • 09:13 logmsgbot: reedy synchronized multiversion/ 'I4a68dc8321b7b302f5e89b5adafcff096f2ac35b'
  • 08:43 logmsgbot: reedy synchronized docroot and w
  • 08:39 logmsgbot: reedy updated /a/common to I57b6d055e: Update flow cache version to 4.2
  • 07:22 springle: up to 5x pt-table-sync running on db1048 m2 master for eventlogging migration. ok to kill if necessary
  • 03:44 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 25 03:44:16 UTC 2014 (duration 44m 15s)
  • 03:03 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-04-25 03:03:33+00:00
  • 02:37 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-25 02:37:06+00:00
  • 00:47 mwalker: updating payments servers from e6d188f0dfcd57406acb58aa2b5bf45e48117c33 to a7fa0d64da2c56586c83cf92babb65bac857be2e for worldpay
  • 00:29 bd808|deploy: Ori was able to fix permissions and second scap test worked as expected
  • 00:28 logmsgbot: bd808 Finished scap: no-op scap to validate I24149ab and Ie967901 (try 2) (duration: 05m 02s)
  • 00:26 bd808|deploy: File permissions on /srv/scap/scap/*.{py,pyc} were not consistently a+r which is needed for scap-rebuild-cdbs
  • 00:23 logmsgbot: bd808 Started scap: no-op scap to validate I24149ab and Ie967901 (try 2)

April 24

  • 23:55 logmsgbot: bd808 Finished scap: no-op scap to validate I24149ab and Ie967901 (duration: 02m 51s)
  • 23:52 logmsgbot: bd808 Started scap: no-op scap to validate I24149ab and Ie967901
  • 23:46 awight: perform crm schema update 7018
  • 23:36 bd808|deploy: Running scap-rebuild-cdbs on tin to test python port
  • 23:17 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Updating flow configuration 129589'
  • 23:16 logmsgbot: mwalker synchronized php-1.24wmf2/extensions/Flow 'Updating flow for 129589 and 129604'
  • 22:53 AaronSchulz: Running PopulateImageSha1.php for all multi-versioned files on all wikis to fix broken SHA-1s
  • 21:22 springle: eventlogging dump loading on db1048 m2 master in screen. ok to kill if necessary
  • 21:18 hashar: restarting Zuul
  • 21:01 mwalker: updating payments from 4811f6d3d80d126c8b3c89c11d20cc6416cb58f6 to e6d188f0dfcd57406acb58aa2b5bf45e48117c33 for donationinterface / worldpay updates
  • 20:39 paravoid: shutting down sdtpa, cr1-sdtpa, csw1-sdtpa, msw1-sdtpa and other sdtpa hosts gone forever
  • 20:37 Coren: sync-apache for 126969 and 91339
  • 19:59 logmsgbot: reedy synchronized docroot and w
  • 19:54 logmsgbot: aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Removed redundant pool counter config'
  • 19:00 ori: eventlogging data streaming into db1048; db1047 consumer decom'd.
  • 19:00 logmsgbot: reedy synchronized wmf-config/
  • 18:59 logmsgbot: reedy synchronized database lists files:
  • 18:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of group0 to 1.24wmf2
  • 18:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.24wmf1
  • 18:33 springle: begin eventlogging migration db1047 to db1048 (m2), RT #7081
  • 16:42 hashar: restarted both Zuul and Jenkins
  • 16:29 logmsgbot: reedy synchronized php-1.24wmf2/extensions/Wikidata 'I43988505ea0fd7ac6b1278a50237e0e1d3ee0e9e'
  • 16:25 hashar: restarting Zuul. Got asked to be stopped.
  • 16:15 hashar: Jenkins ended up being stalled due to a known unfigured out issue :/
  • 16:13 hashar: Killed a leftover jenkins process on gallium
  • 15:50 akosiaris: scheduled a safe restart of jenkins
  • 15:48 logmsgbot: hoo updated /a/common to I53de8d84b: Add two languages not supported by MediaWiki to testwikidata
  • 15:48 logmsgbot: hoo synchronized wmf-config/InitialiseSettings.php 'Add two languages not supported by MediaWiki to testwikidata'
  • 15:47 andrewbogott: zuul on gallium is dead and I don't know why
  • 15:45 andrewbogott: restarted jenkins and zuul on gallium
  • 14:02 logmsgbot: reedy Finished scap: testwiki to 1.24wmf2 build l10n cache (take 2) (duration: 15m 46s)
  • 13:46 logmsgbot: reedy Started scap: testwiki to 1.24wmf2 build l10n cache (take 2)
  • 13:45 logmsgbot: reedy Finished scap: testwiki to 1.24wmf2 build l10n cache (duration: 08m 16s)
  • 13:37 logmsgbot: reedy Started scap: testwiki to 1.24wmf2 build l10n cache
  • 13:32 logmsgbot: reedy updated /a/common to I543df75e3: Remove $wgDisableTextSearch and $wgDisableSearchUpdate overrides.
  • 12:22 akosiaris: restarted morebots after upgrade of adminbot to 1.7.5
  • 12:00 Krinkle: Running deleteEqualMessages.php on alswiki (bug 43917)
  • 12:00 Krinkle: Running deleteEqualMessages.php on suwiki (bug 43917)
  • 12:00 Krinkle: Running deleteEqualMessages.php on tlwiki (bug 43917)
  • 11:59 Krinkle: Running deleteEqualMessages.php on nahwiktionary (bug 43917)
  • 11:53 paravoid: reenabling ospf3 between cr1-eqiad/cr2-knams
  • 11:50 paravoid: fixing private4/private6 ACLs to be consistent across all routers
  • 06:47 _joe_: also ran puppet node clean to revoke certs, facts, etc (cp3013.esams.wikimedia.org cp3014.esams.wikimedia.org)
  • 06:38 _joe_: ran puppetstoredconfigclean.rb for cp3013.esams.wikimedia.org cp3014.esams.wikimedia.org
  • 05:20 springle: xtrabackup dbstore1001 to dbstore1002
  • 03:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 24 03:29:12 UTC 2014 (duration 29m 11s)
  • 02:46 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-24 02:45:58+00:00
  • 02:29 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-24 02:29:14+00:00

April 23

  • 23:40 logmsgbot: catrope synchronized php-1.24wmf1/extensions/VisualEditor/lib/ve/modules/ve/ui/widgets/ve.ui.SurfaceWidget.js 'Fix surface focusing bug in Firefox'
  • 23:39 logmsgbot: catrope synchronized php-1.24wmf1/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js 'Unbreak badtoken recovery in mobile VE'
  • 23:38 logmsgbot: catrope synchronized php-1.24wmf1/resources/src/jquery/jquery.suggestions.js 'Handle CSS ellipsis when calculating suggestions widths'
  • 23:37 logmsgbot: catrope synchronized php-1.23wmf22/resources/src/jquery/jquery.suggestions.js 'Handle CSS ellipsis when calculating suggestions widths'
  • 21:55 bblack: moved cp301[34] ethernet ports to private1-esams
  • 20:14 Krinkle: Running deleteEqualMessages.php on mlwiki (bug 43917)
  • 20:14 Krinkle: Running deleteEqualMessages.php on miwiki (bug 43917)
  • 20:11 Krinkle: Running deleteEqualMessages.php on gvwiki (bug 43917)
  • 20:11 Krinkle: Running deleteEqualMessages.php on euwiktionary (bug 43917)
  • 20:04 subbu: deployed Parsoid 9c99b0be (deploy SHA cf5eb4d0)
  • 19:51 Krinkle: Running deleteEqualMessages.php on brwiki (bug 43917)
  • 19:51 Krinkle: Running deleteEqualMessages.php on afwiki (bug 43917)
  • 19:50 Krinkle: Running deleteEqualMessages.php on iawiki (bug 43917)
  • 19:37 Krinkle: Running deleteEqualMessages.php on hrwiktionary (bug 43917)
  • 19:37 Krinkle: Running deleteEqualMessages.php on hrwiki (bug 43917)
  • 19:34 Krinkle: Running deleteEqualMessages.php on dawiki (bug 43917)
  • 19:00 logmsgbot: reedy synchronized wmf-config/ 'I543df75e364171a71a48f18429972b662b542894'
  • 18:58 logmsgbot: reedy updated /a/common to I865a08779: Fix $wmgBetaFeaturesWhitelist for labs too
  • 18:58 Krinkle: Running deleteEqualMessages.php on amwiki (bug 43917)
  • 17:59 logmsgbot: demon synchronized wmf-config/InitialiseSettings-labs.php 'no-op in prod, for completeness'
  • 17:52 logmsgbot: demon synchronized all-labs.dblist 'no-op for prod, syncing for completeness'
  • 17:52 logmsgbot: demon synchronized wikiversions-labs.json 'no-op for prod, syncing for completeness'
  • 17:51 logmsgbot: demon updated /a/common to I960a792bc: Override wgSearchTypeAlternatives for beta to remove lucene
  • 15:58 akosiaris: updated adminbot on apt.wikimedia.org to 1.7.5
  • 15:57 manybubbles: rebuilding commons' cirrus search index
  • 15:55 logmsgbot: manybubbles synchronized php-1.24wmf1/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php 'swat update to fix maintenance script'
  • 15:46 ottomata: temporarily disabling puppet on analytics1003 to test some kafkatee settings
  • 15:32 apergos: rebooting dataset2 hoping to detect the arrays on reboot
  • 14:40 akosiaris: unexported /vol/{originals,thumbs} on nas1001-a, nas1-a
  • 14:30 akosiaris: break replication for volumes originals, thumbs on nas1001-a, nas1-a
  • 14:03 paravoid: adding AS path 1257 6830 (Tele2 -> UPC) to avoided paths @ cr1-esams/cr2-knams, multiple users reporting slowness issues
  • 13:52 akosiaris: unmounted /vol/originals and /vol/thumbs on fenari (was /mnt/upload7, /mnt/thumbs2) see RT #7076
  • 12:54 logmsgbot: marc synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
  • 12:50 hashar: Jenkins back
  • 12:46 hashar: restarting jenkins
  • 12:46 hashar: Jenkins: upgrading email-ext and JobConfigHistory plugins (the later now supports slaves configs!)
  • 10:35 hashar: Jenkins: update lanthanum slave agent to use java7
  • 10:32 hashar: Jenkins switching integration-slave1001.eqiad.wmflabs java to use Java 7 . In https://integration.wikimedia.org/ci/computer/integration-slave1001/configure changed JavaPath to /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
  • 07:45 mutante: nfs1 - delete some old kernels and zip mw logs last touched in 2012/13 to free some disk on /
  • 07:32 mutante: nfs1 - re-enabled puppet
  • 07:12 mutante: nfs2 - revoke puppet cert,salt key,stored configs
  • 06:26 mutante: db48,db63 - revoke puppet cert, salt key, kill from storedconfigs
  • 03:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 23 03:55:03 UTC 2014 (duration 55m 2s)
  • 03:09 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-23 03:08:58+00:00
  • 02:46 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-23 02:46:47+00:00
  • 02:25 manybubbles: restarted rebuilding common's Cirrus index after something crashed. going to get more logging out of it if it crashes again. or it'll work. Either way. Like last time the Elasticsearch check might freak out for a bit after it finished because shards are assigning. That can be ignored for an hour or so.

April 22

  • 23:30 logmsgbot: ori Finished scap: I595446dc5, If2c57846f, Iaa232298e (duration: 00m 45s)
  • 23:30 logmsgbot: ori Started scap: I595446dc5, If2c57846f, Iaa232298e
  • 23:28 logmsgbot: ori synchronized php-1.24wmf1/extensions/EventLogging 'Update EventLogging for Iaa232298e: Set line-height for code icon on schema pages (bug 64251)'
  • 23:27 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'If2c57846f: Enable survey option in MediaViewer on a few more wikis'
  • 23:26 logmsgbot: ori updated /a/common to If2c57846f: Enable survey option in MediaViewer on a few more wikis
  • 23:24 logmsgbot: ori synchronized php-1.23wmf22/extensions/MultimediaViewer 'Update MultimediaViewer for I595446dc5: Add more survey languages (fr, de, pt/pr-br)'
  • 23:23 logmsgbot: ori synchronized php-1.24wmf1/extensions/MultimediaViewer 'Update MultimediaViewer for I595446dc5: Add more survey languages (fr, de, pt/pr-br)'
  • 22:28 logmsgbot: aaron synchronized php-1.24wmf1/maintenance/populateImageSha1.php '32d9206'
  • 22:22 logmsgbot: reedy synchronized php-1.24wmf1/extensions/TimedMediaHandler 'I7483c8b7ec75f5149998da2b530ca04'
  • 21:31 logmsgbot: spage synchronized php-1.24wmf1/extensions/Flow/modules/discussion/styles/mixins/collapse.less 'Fix Flow collapsed topics on mw.org'
  • 21:14 logmsgbot: spage synchronized wmf-config/InitialiseSettings.php 'Enable Flow on Compact Personal Bar talk'
  • 21:12 logmsgbot: spage updated /a/common to I851651247: Non wikipedias to 1.24wmf1
  • 20:35 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'No op in prod, disables lsearchd completely for beta'
  • 20:28 ottomata: turning on varnishkafka on text varnishes
  • 20:12 MatmaRex: wikibugs replaced by pywikibugs (https://github.com/valhallasw/pywikibugs) and moved to #wikimedia-dev (at last!)
  • 20:12 manybubbles: rebuilding the search index for a few wikis - might cause the Elasticsearch health check to freak out because it sucks
  • 19:51 MatmaRex: wikibugs is down, let's not bring it back up
  • 19:31 Krinkle: Reloading Zuul to deploy config change I9c2f94b138244ab8
  • 19:05 hashar: Jenkins killed Jenkins java process on deployment-bastion.eqiad.wmflab to free up the executor and threads entirely.
  • 18:55 hashar: restarted Zuul to clean up some stuck jobs from the queue
  • 18:49 hashar: Jenkins deployment-bastion.eqiad.wmflab is back online: Slave successfully connected and online
  • 18:47 logmsgbot: reedy synchronized docroot and w
  • 18:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: non wikipedias to 1.24wmf1
  • 18:43 RobH: tridge back and accessible
  • 18:42 hashar: Jenkins deposing / repooling deployment-bastion.eqiad.wmflabs slave locked up somehow, the executors are no more taken in account by Jenkins master
  • 18:33 RobH: resurrecting tridge in pmtpa
  • 18:00 RobH: tridge is coming dow for relocation, shouldnt disrupt anything but backups in progress
  • 17:52 bblack: disable cp301[34] (mobile varnish frontends) in pybal on fenari
  • 17:21 awight: update crm from 7dafce5 to cfe34fe
  • 16:45 mark: Reenabled Apache and puppet on fenari
  • 16:40 logmsgbot: aaron synchronized php-1.24wmf1/includes/filerepo/file/LocalFile.php 'e9807d0'
  • 16:15 logmsgbot: reedy updated /a/common to I55954c612: Commit updated interwiki.cdb file
  • 15:47 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Collection back on, server move over'
  • 15:14 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Icb6b4bad: Updated $wgForceUIMsgAsContentMsg for commonswiki'
  • 14:59 cmjohnson1: shutting down and relocating virt0 and pdf2
  • 14:50 logmsgbot: marc synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
  • 14:41 logmsgbot: marc synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
  • 14:33 manybubbles: populating cirrus indexes for all remaining wikis
  • 14:19 akosiaris: added bblack account on all junipers
  • 14:18 manybubbles: building new elasticsearch indexes for the last wikis that didn't have them. the cluster may go red as the indexes are assigned. silly nagios check.
  • 14:15 logmsgbot: manybubbles synchronized wmf-config/ 'cirrus for more wikis and disable collection for more'
  • 14:13 logmsgbot: manybubbles synchronized docroot/noc/createTxtFileSymlinks.sh 'noncirrus is removed'
  • 14:09 cmjohnson1: mchenry and sanger going down for server relocation
  • 13:26 mark: Disabled puppet and apache on fenari
  • 13:25 paravoid: second pass of swiftrepl eqiad->esams
  • 12:04 logmsgbot: faidon synchronized wmf-config/squid.php 'add cp3013/cp3014 IPv6 addresses'
  • 12:04 logmsgbot: faidon updated /a/common to If8f39abee: squid.php: add cp3013/cp3014 IPv6 addresses
  • 10:35 akosiaris: upgraded php-luasandbox to 1.9.1 on beta (deployment-apache0{1,2})
  • 10:25 akosiaris: upgraded php-luasandbox to 1.9-1 on test.wikimedia.org
  • 10:18 mutante: harmon - delete salt key
  • 09:04 mutante: hooper - revoked puppet cert
  • 08:47 mutante: upgrading Bugzilla to 4.4.4
  • 06:58 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Unbreak $wmgBetaFeaturesWhitelist'
  • 06:44 mutante: db77 - revoke puppet cert,salt key,rm from monitoring
  • 05:14 springle: db68 down. s1-analytics-slave cname to db1007
  • 04:35 paravoid: reactivate esams<->HE & eqiad<->HE peerings; issues are confirmed to be resolved
  • 03:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 22 03:32:29 UTC 2014 (duration 32m 28s)
  • 02:43 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-22 02:43:01+00:00
  • 02:30 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-22 02:29:34+00:00
  • 01:16 awight: nevermind previous update.
  • 01:11 awight: update from 7dafce5 to cfe34fe
  • 00:18 logmsgbot: catrope synchronized php-1.24wmf1/extensions/MobileFrontend/javascripts/modules/editor/VisualEditorOverlay.js 'Fix JS error on save'
  • 00:15 Reedy: torrus (on manutius) is down
  • 00:05 RoanKattouw: Restarted ircecho on ekrem, IRC working again now
  • 00:00 K4-713: updated payments from 2819549 --> 4811f6d

April 21

  • 23:57 RoanKattouw: Started ircd on ekrem, startup doesn't seem to be puppetized
  • 23:24 logmsgbot: catrope synchronized php-1.23wmf22/extensions/MultimediaViewer 'SWAT deploy cherry-picks'
  • 23:24 logmsgbot: catrope synchronized php-1.24wmf1/extensions/MobileFrontend 'SWAT deploy cherry-picks'
  • 23:23 logmsgbot: catrope synchronized php-1.24wmf1/extensions/MultimediaViewer 'SWAT deploy cherry-picks'
  • 23:23 logmsgbot: catrope synchronized php-1.24wmf1/extensions/VisualEditor 'SWAT deploy cherry-picks'
  • 23:16 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Lithuanian namespace aliases for betawikiversity'
  • 23:11 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'BetaFeatures whitelist'
  • 23:11 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Beta Features whitelist'
  • 22:50 RoanKattouw: Restarted stuck Jenkins
  • 21:33 awight: updated payments: af35b7b --> 2819549
  • 20:45 andrewbogott: rebooted wtp1018
  • 20:45 subbu: deployed Parsoid ec51e5d1 (deploy SHA 0dd607fc)
  • 20:31 manybubbles: rolling restart on remaining Elasticsearch servers to get the plugin (1010, 1011, 1012, 1015)
  • 19:45 manybubbles: rolling restart on more of the elasticsearch servers to pick up plugins (06, 07, 09)
  • 19:15 cmjohnson1: shutting down and relocating dobson
  • 19:15 cmjohnson1: shutting down and relocating linne
  • 17:39 cmjohnson: shutting down and relocating fenari
  • 17:37 cmjohnson: shutting down mexia to relocate to 12th floor
  • 17:25 logmsgbot: aaron synchronized php-1.24wmf1/includes/filerepo/file/LocalFile.php '2026e4a'
  • 17:22 logmsgbot: aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Adjust large file download pool counter config to tie up less workers'
  • 17:18 logmsgbot: aaron synchronized php-1.23wmf22/includes/filerepo/file/LocalFile.php '01ce288'
  • 17:13 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'renderfile-nonstandard throttle config'
  • 17:03 logmsgbot: aaron synchronized php-1.24wmf1/thumb.php '44c4658'
  • 17:02 logmsgbot: aaron synchronized php-1.23wmf22/thumb.php '9591365'
  • 16:34 ottomata: reinstalling elastic1013 (elastic1014 is still coming back online, but I don't want there to an extra eligible master for long)
  • 16:04 ottomata: reinstalling elastic1014
  • 15:22 cmjohnson1: dataset2 going down to be relocated to the 12th floor
  • 13:41 manybubbles: rolling restart on some of the Elasticsearch servers to pick up new plugins. should not cause any trouble.
  • 13:05 Reedy: De-activated status.wm.o monitor for icinga due to false positive from HTTP auth
  • 12:54 paravoid: demoting myself, removing Commons crat/admin rights
  • 12:41 paravoid: escalating myself to Commons bureaucrat/admin, then adding GWToolset privileges
  • 12:40 paravoid: deleting 29 GWToolset XML under Swift's wikipedia-commons-gwtoolset-metadata container for user Fæ/
  • 11:51 logmsgbot: reedy synchronized php-1.23wmf22/extensions/TimedMediaHandler 'I7483c8b7ec75f5149998da2b530ca04'
  • 11:50 paravoid: deactivating esams<->HE peering, >90% packet loss between lon<->nyc
  • 11:49 paravoid: deactivating eqiad<->HE peering, >90% packet loss between lon<->nyc
  • 11:45 logmsgbot: reedy synchronized php-1.23wmf22/extensions/TimedMediaHandler 'I7483c8b7ec75f5149998da2b530ca0467ac70de7'
  • 03:55 springle: reset pc100* slaves previously replicating from pmtpa
  • 03:32 ori: 5.5k fatals over last 20 hrs, of which 3.5k are calls to doTransform() on a non-object at TimedMediaThumbnail.php:201, and 0.9k are Lua API OOMs at LuaSandbox/Engine.php:264
  • 03:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 21 03:30:15 UTC 2014 (duration 30m 14s)
  • 03:26 ori: ap_busy_workers spike on image scalers eqiad, started ~2:55, subsided around ~3:20
  • 02:42 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-21 02:42:30+00:00
  • 02:29 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-21 02:29:49+00:00

April 20

  • 18:51 ori: restarted grrrit-wm by following instructions on https://wikitech.wikimedia.org/wiki/Grrrit-wm#Restarting_the_bot
  • 11:12 Nemo_bis: grrrit dead: 10.28 -!- grrrit-wm [tools.lolr@208.80.155.145] has quit
  • 03:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 20 03:26:48 UTC 2014 (duration 26m 47s)
  • 02:39 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-20 02:39:23+00:00
  • 02:28 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-20 02:28:09+00:00

April 19

  • 03:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 19 03:27:52 UTC 2014 (duration 27m 51s)
  • 02:41 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-19 02:41:02+00:00
  • 02:29 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-19 02:29:33+00:00

April 18

  • 21:00 hashar: Jenkins renamed mw-jenkinsbot irc bot to wmf-insecte (french for "bug"). Updated IRC conf to point to chat.freenode.net:7000 with SSL.
  • 19:02 bblack: enabled cp30[14] varnish mobile frontends in esams pybal
  • 17:50 bblack: cp301[34] reinstalls complete, should stay ok in monitoring
  • 17:48 ottomata: resinsalling elastic1008
  • 16:20 springle: db48 mysqld shutdown for decom
  • 16:20 bblack: ignore cp301[34] msgs, reinstalling them
  • 16:10 springle: db63 mysqld shutdown for decom
  • 15:53 ottomata: reinstalling elastic1007
  • 15:52 springle: db48 mysqld set read_only, disabled m2 repl to db1048
  • 15:51 ottomata: disabling puppet on elasti1007 and elastic1008 for reformatting
  • 15:45 mutante: DNS update - removing Tampa msbe/msfe
  • 15:38 Jeff_Green: switched mchenry to use db1048/db1049 for OTRS address lookups
  • 15:24 mutante: DNS update - removing all the Tampa mw/srv mgmt
  • 15:15 mutante: DNS update - removing lvs1-6
  • 14:54 mutante: es5,es6 - revoke puppet certs, salt keys, icinga
  • 14:51 ottomata: powering down stat1 for decom
  • 14:43 mutante: ms-fe[14] - shutting down
  • 14:41 ottomata: disabling puppet on stat1 for decom
  • 14:37 mutante: ms-be 1-12, Tampa Swift boxes, shutdown
  • 14:24 mutante: ms-fe[14] - stop puppet,revoke certs,remove icinga
  • 13:54 mutante: ms-be1-12 - removing from puppet,salt,icinga
  • 13:06 mutante: Bugzilla Apache, changed SSL cipher suite in I7e9adc182dc ,might cost a a few % performance but zirconium had plenty
  • 11:48 hashar: removing mw-jenkinsbot (the wikimedia jenkins installation) from #wikimedia-labs
  • 10:10 hashar: Jenkins upgraded to 1.532.3.
  • 10:06 hashar: Upgrading Jenkins to latest LTS version 1.532.3
  • 07:57 mutante: DNS update - remove api.svc, arptest.pmtpa ..
  • 06:31 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Next round of wikis done building Cirrus indexes, throw into beta mode'
  • 04:04 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 18 04:04:21 UTC 2014 (duration 4m 20s)
  • 03:06 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-18 03:06:06+00:00
  • 02:39 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-18 02:39:51+00:00
  • 00:21 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'Ie9b265be9: Enable GlobalCssJs on testwiki & test2wiki (2/2)'
  • 00:21 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'Ie9b265be9: Enable GlobalCssJs on testwiki & test2wiki (1/2)'
  • 00:20 logmsgbot: ori updated /a/common to Ie9b265be9: Enable GlobalCssJs on testwiki & test2wiki
  • 00:18 logmsgbot: ori Finished scap: Cherry-pick Ibe8e67ebf for MobileFrontend on 1.23wmf22 and 1.24wmf1; add GlobalCssJs extension to 1.24wmf1 and 1.23wmf22 (duration: 32m 53s)

April 17

  • 23:45 logmsgbot: ori Started scap: Cherry-pick Ibe8e67ebf for MobileFrontend on 1.23wmf22 and 1.24wmf1; add GlobalCssJs extension to 1.24wmf1 and 1.23wmf22
  • 23:39 logmsgbot: ori scap failed: CalledProcessError Command '/usr/local/bin/mw-update-l10n' returned non-zero exit status 1 (duration: 00m 24s)
  • 23:38 logmsgbot: ori Started scap: Cherry-pick Ibe8e67ebf for MobileFrontend on 1.23wmf22 and 1.24wmf1; add GlobalCssJs extension to 1.24wmf1
  • 23:37 logmsgbot: ori updated /a/common to I2a2abd7f3: Add GlobalCssJs to extension-list
  • 23:29 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I52378a4b4: Add meta to legalteamwiki import sources'
  • 23:28 logmsgbot: ori updated /a/common to I52378a4b4: Add meta to legalteamwiki import sources
  • 23:27 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I373df6138: Normalize TextExtracts config handling (2/2)'
  • 23:27 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I373df6138: Normalize TextExtracts config handling (1/2)'
  • 23:26 logmsgbot: ori updated /a/common to I373df6138: Normalize TextExtracts config handling
  • 23:24 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I7841f74b0: Kill all vestiges of $wgMFRemovableClasses (2/2)'
  • 23:24 logmsgbot: ori synchronized wmf-config/mobile.php 'I7841f74b0: Kill all vestiges of $wgMFRemovableClasses (1/2)'
  • 23:23 logmsgbot: ori updated /a/common to I7841f74b0: Kill all vestiges of $wgMFRemovableClasses
  • 23:18 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I1795c70d1: Create a FeaturedFeed for the Tech News bulletin (2/2)'
  • 23:17 logmsgbot: ori synchronized wmf-config/FeaturedFeedsWMF.php 'I1795c70d1: Create a FeaturedFeed for the Tech News bulletin (1/2)'
  • 23:16 logmsgbot: ori updated /a/common to I1795c70d1: Create a FeaturedFeed for the Tech News bulletin
  • 23:14 logmsgbot: ori synchronized php-1.23wmf22/extensions/ApiSandbox 'I9a56b2c5a: Update ApiSandbox'
  • 23:12 K4-713: updates antifraud rules in payments
  • 22:03 andrewbogott: updated default labs precise image (heartbleed fix)
  • 20:41 manybubbles: elastic1016 restarted and not freaking out any more.
  • 20:37 _joe_: restarting gitblit in order to prevent crippling due to the usual memory leak
  • 20:28 manybubbles: restarting elastic1016 - it is freaking out. If it happens again I'll dig deeper, but for now I consider it a fluke of the rolling restarts today....
  • 20:20 RobH: sorry for the misc-web-lb issues folks, they should be resolved at this time (for now)
  • 20:19 paravoid: lvs1002/1005: commenting first resolv.conf entry until we have a more permanent fix, restarting pybal
  • 20:18 paravoid: disabling puppet on lvs1002/lvs1005
  • 19:57 RobH: still working on issue
  • 19:57 RobH: both cp1043 and cp1044 seem online and serving nginx service, but pybal says they are down still working
  • 19:46 ottomata: power off emery
  • 19:40 RobH: replacing ticket.wikimedia.org cert/key, apache may hiccup
  • 19:33 RobH: blog.w.o cert replacement successful
  • 19:30 ottomata: disabling puppet on emery for decommission
  • 19:29 RobH: blog.w.o certificate swap (yes, again ;), apache may hiccup
  • 19:10 logmsgbot: reedy synchronized wmf-config/
  • 19:09 logmsgbot: reedy synchronized database lists files: I6fc44d3eb829d656d352dab652148dd327b06679
  • 19:04 ottomata: reinstalling elastic1001
  • 18:59 logmsgbot: faidon synchronized wmf-config/CommonSettings.php 'reenable CN CrossWiki Hiding'
  • 18:58 logmsgbot: faidon updated /a/common to Ie95165065: Reenable CentralNotice CrossWiki Hiding
  • 18:57 logmsgbot: reedy synchronized wmf-config/
  • 18:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Touch for I0c36c65bb9f405e03b84d3f6c6b93acda522c5c9'
  • 18:33 logmsgbot: reedy synchronized database lists files: I0c36c65bb9f405e03b84d3f6c6b93acda522c5c9
  • 18:30 ottomata: switching erbium udp2log instance from consuming multicast relay to unicast direct from varnishes
  • 18:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf1
  • 18:10 ottomata: stopping puppet on elastic1001 and elastic1002, reinstalling elastic1002
  • 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf22
  • 16:06 logmsgbot: reedy Finished scap: testwiki to 1.24wmf1 and build l10n cache (duration: 26m 06s)
  • 15:47 logmsgbot: anomie synchronized php-1.23wmf22/extensions/VisualEditor 'SWAT: 126913 - backport to wmf22 of critical fixes for the Math extension's VisualEditor tool'
  • 15:47 logmsgbot: anomie synchronized php-1.23wmf22/extensions/Math 'SWAT: 126913 - backport to wmf22 of critical fixes for the Math extension's VisualEditor tool'
  • 15:40 logmsgbot: reedy Started scap: testwiki to 1.24wmf1 and build l10n cache
  • 15:28 logmsgbot: faidon synchronized wmf-config/CommonSettings.php 'disable CN CrossWiki Hiding again'
  • 15:27 logmsgbot: faidon updated /a/common to If74ba5a52: Revert "Enable CentralNotice CrossWiki Hiding"
  • 15:17 manybubbles: updgraded site plugins on Elasticsearch nodes
  • 15:04 ottomata: reinstalling elastic1016
  • 14:02 logmsgbot: reedy updated /a/common to I290bd1ea6: Remove further pmtpa remnants
  • 13:41 manybubbles: synced experimental highlighter to elasticsearch nodes - they'll pick it up on restart
  • 11:05 logmsgbot: reedy synchronized wmf-config/ 'I290bd1ea628563646c02651041fa2cec4a320b56'
  • 10:56 mutante: lvs3,lvs4,lvs5,lvs6 - shutdown
  • 10:42 mutante: lvs1, lvs2 shutdown
  • 10:15 mutante: re-deleting unaccepted salt keys for virt2,5-11
  • 10:11 mutante: lvs1-6 - disable puppet,salt,revoke certs,keys
  • 08:28 mutante: db35,db38 - shutdown
  • 07:47 mutante: db35,db38, stop puppet and salt, revoke certs,keys
  • 07:45 mutante: restarting gitblit
  • 03:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 17 03:49:19 UTC 2014 (duration 49m 18s)
  • 03:06 subbu: deployed Parsoid 0bccf02c (deploy SHA 5e25f3b05) @ 1:30 pm PST, Apr 16th, 2014
  • 03:02 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-17 03:02:25+00:00
  • 02:33 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-17 02:33:47+00:00
  • 02:03 springle: stop mysqld on db35 (m1) for decom
  • 01:42 springle: xtrabackup db63 to db60

April 16

  • 23:19 logmsgbot: mwalker Finished scap: SWAT deploy: configuration change 126223 and multimediaviewer 126852 (duration: 04m 07s)
  • 23:15 logmsgbot: mwalker Started scap: SWAT deploy: configuration change 126223 and multimediaviewer 126852
  • 20:03 RobH: osmium cleared from salt, puppetca, and puppetstoredconfig for reinstall with trusty (ignore any icinga alerts, there are no pages)
  • 18:58 ottomata: reinstalling elastic1015
  • 17:54 logmsgbot: reedy synchronized php-1.23wmf22/includes/jobqueue/ 'I4b4dbe4637dc50cd4630ef19d54f01efba10e138'
  • 17:09 paravoid: starting swiftrepl on copper for eqiad->esams copy
  • 17:08 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I7b6e5c2d7: Enable web fonts by default on Hebrew Wikisource'
  • 17:07 logmsgbot: ori synchronized fc-list 'Ib7b2bc21a: updated fonts list and sorted it, rt #810'
  • 17:06 logmsgbot: ori updated /a/common to I7b6e5c2d7: Enable web fonts by default on Hebrew Wikisource
  • 16:37 logmsgbot: reedy synchronized php-1.23wmf21/includes/jobqueue/JobQueueRedis.php 'I678ab55ae3678b5cd944393f2f2048851625f153'
  • 16:36 logmsgbot: reedy synchronized php-1.23wmf22/includes/jobqueue/JobQueueRedis.php 'I678ab55ae3678b5cd944393f2f2048851625f153'
  • 15:38 ottomata: reinstalling elastic1012
  • 13:50 manybubbles: restarting elastic1009 to suck up new config
  • 13:50 manybubbles: raised the number of replicas for labswiki's search directly in elasticsearch because I can't easilly do for cirrus due to access restrictions
  • 13:45 ottomata: reinstalling elastic1011
  • 13:22 mutante: DNS update - remove virt5-15
  • 12:11 mutante: virt5-11 - shut down
  • 11:40 akosiaris: upgraded python-voluptuous on apt.wikimedia.org to 0.8.2-1wmf1
  • 11:39 hashar: Upgraded Zuul to wmf-deploy-20140416-3 (bring in a84f0e4 - "Make queue processing more efficient" which was much needed)
  • 11:29 hashar: upgraded Zuul to wmf-deploy-20140416-2
  • 11:15 mutante: virt5-11 removing from icinga
  • 11:03 mutante: virt5-11 revoked puppet certs and salt keys
  • 10:56 mutante: stopping puppet on virt5-11
  • 10:47 hashar: Upgraded Zuul on gallium to wmf-deploy-20140416 (depends on python-voluptuous 0.7+ , Alexandros packaged 0.8.2 which I manually installed to validate).
  • 09:26 mutante: disabling mw1163 in pybal
  • 07:03 mutante: zirconium - upgrading apache2, php5 packages
  • 06:07 springle: stop mysqld on db38 (x1) for decom
  • 03:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 16 03:46:23 UTC 2014 (duration 46m 22s)
  • 02:55 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-16 02:55:28+00:00
  • 02:28 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-16 02:28:01+00:00
  • 00:43 K4-713: updated listener credentials on thulium

April 15

  • 23:25 logmsgbot: mwalker synchronized wmf-config/abusefilter.php '126168 more abuse filter configuration fun'
  • 23:21 logmsgbot: mwalker Finished scap: Configuration change 126163 and MultimediaViewer 126158 (duration: 02m 15s)
  • 23:19 logmsgbot: mwalker Started scap: Configuration change 126163 and MultimediaViewer 126158
  • 23:08 logmsgbot: mwalker Finished scap: Configuration changes, 113656, 121834, 126065 (duration: 03m 11s)
  • 23:05 logmsgbot: mwalker Started scap: Configuration changes, 113656, 121834, 126065
  • 23:01 hashar: restarting Zuul to clear leaked file descriptor (know issue, fixed upstream)
  • 22:12 awight: crm updated from e3f2859 to 7dafce5
  • 21:51 manybubbles: restarting elastic1009 again
  • 21:39 hashar: jenkins /var/lib/git cleaned up on gallium
  • 21:16 manybubbles: restarting elastic1009 to test performance changes. cluster will go yellow for a few minutes. might go red (wikitech is busted)
  • 21:15 hashar: Jenkins is processing jobs again
  • 21:14 hashar: cleared /tmp/ on integration-slave1002 (filled up by hhvm job, known issue, bug filled already)
  • 21:12 hashar: Zuul locked again :/ Unpooling and repooling Jenkins slaves.
  • 19:50 RoanKattouw: Restarting stuck Jenkins
  • 19:31 manybubbles: setting refresh interval on elasticsearch indexes to 30s to test effect on load
  • 19:24 logmsgbot: reedy synchronized wmf-config/
  • 19:20 logmsgbot: reedy synchronized php-1.23wmf22/includes/PrefixSearch.php 'I82b5ca65864099c180d915055c43e6839bd4f4a2'
  • 19:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikisources back to 1.23wmf22
  • 19:07 ottomata: reinstalling elastic1010
  • 19:07 logmsgbot: reedy synchronized php-1.23wmf22/extensions/ProofreadPage
  • 18:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikisources back to 1.23wmf21 due to ProofreadPage fatal
  • 18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.23wmf22
  • 17:09 paravoid: stopped pybal on lvs1005
  • 17:06 cmjohnson1: fixing lvs1005 eth1 cable
  • 16:56 cmjohnson1: mw1057 replacing ethernet cable
  • 16:50 manybubbles: raised "new generation" size on elastic1009 to test a performance theory
  • 16:50 cmjohnson1: mw1093 replacing ethernet cable
  • 16:40 cmjohnson1: replacing eth cable on mw1193
  • 16:31 hashar: ... all Jenkins jobs are using /srv/ssd/gerrit instead
  • 16:30 hashar: gallium had two Gerrit replications streams, one of them got removed 122419 thus deleting the target directories under /var/lib/git
  • 16:22 cmjohnson1: shutting down mw1163 to replace DIMM
  • 16:18 cmjohnson1: swapping bad disk slot 4 on dataset1001
  • 16:13 paravoid: moving ms-fe3xxx/ms-be3xxx to private1-esams
  • 16:05 ottomata: reinstalling elastic1009
  • 15:21 logmsgbot: anomie synchronized php-1.23wmf21/extensions/Flow 'SWAT: Flow: Prevent logspam on enwiki 125930'
  • 15:13 logmsgbot: anomie synchronized php-1.23wmf21/extensions/Flow 'SWAT: Flow: Prevent logspam on enwiki 125930'
  • 15:02 mutante: DNS update - removing Tampa service IPs
  • 13:51 hashar: Jenkins compressing console logs of builds. On gallium as user jenkins : find /var/lib/jenkins/jobs -wholename '*/builds/*/log' -type f -exec gzip --best {} \;
  • 13:42 hashar: Command executed (as gerritslave user): find /srv/ssd/gerrit -type d -name '*.git' -exec bash -c 'echo; date; cd {}; echo; pwd; echo; git repack -ad; date;' \;
  • 13:41 hashar: Repacking Gerrit replicated repositories on lanthanum and gallium (both under /srv/ssd/gerrit/ )
  • 13:13 andrewbogott: shutdown and decommissioned virt12
  • 12:19 paravoid: adding ms-be101[345] to Swift eqiad's rings, at 33% weight; old rings kept at ms-fe1001:~/swift-2014-04-14
  • 11:30 mutante: DNS update - removed dbdump.pmtpa.wmnet
  • 11:26 mutante: DNS update - remove db64,db65,db66,db67,db70
  • 10:55 mutante: db64,db67 - powerdown via mgmt
  • 10:51 mutante: db65,db66 - shutdown
  • 10:07 mutante: db70 - powerdown via mgmt
  • 09:47 mutante: db64-67 - puppetstoredconfigclean.rb db${db}.pmtpa.wmnet ; puppetca --clean db${db}.pmtpa.wmnet ; salt-key -d db${db}.pmtpa.wmnet
  • 07:02 springle: shutdown db67 for decom. analytics data is backed up on dbstore1002
  • 06:47 springle: moving pmtpa m1 and x1 slaves to db73 and db69 on 12th floor
  • 03:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 15 03:25:52 UTC 2014 (duration 25m 51s)
  • 02:42 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-15 02:42:48+00:00
  • 02:22 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-15 02:22:54+00:00
  • 00:09 gwicke: enabled wikinews family in Parsoid with temporary live patch to un-break VE deploy

April 14

  • 23:43 logmsgbot: ori Finished scap: (no message) (duration: 04m 31s)
  • 23:39 ori: scap: php-1.23wmf22/extensions/VisualEditor 2b0979f...0652ad2 (I12e5c9751)
  • 23:38 logmsgbot: ori Started scap: (no message)
  • 23:17 logmsgbot: ori synchronized php-1.23wmf22/skins/vector/variables.less 'Ibcdaff017: Revert body font stack to be just sans-serif'
  • 23:15 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I22f25730d: Enable VisualEditor for opt-in on Meta (2/2)'
  • 23:15 logmsgbot: ori synchronized visualeditor.dblist 'I22f25730d: Enable VisualEditor for opt-in on Meta (1/2)'
  • 23:14 logmsgbot: ori updated /a/common to I22f25730d: Enable VisualEditor for opt-in on Meta
  • 23:12 logmsgbot: ori synchronized visualeditor.dblist 'I59f5a6e0b: Enable VisualEditor on French Wikinews'
  • 23:12 logmsgbot: ori updated /a/common to I59f5a6e0b: Enable VisualEditor on French Wikinews
  • 22:56 logmsgbot: aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Limit large (djvu) file downloads for thumbnails'
  • 20:37 mwalker: updating payments wiki for worldpay currencies (from af35b7b to 8a93c17)
  • 20:13 subbu: deployed Parsoid fba548cbf (deploy repo sha d0e12ddf)
  • 17:47 paravoid: fixing /e/n/interfaces for static configuration: gadolinium hafnium labsdb1001 labsdb1002 labsdb1003 labstore1001 searchidx1001 ssl1005 ssl1006 ssl1009 virt1001 ytterbium
  • 17:37 paravoid: fixing /e/n/interfaces for static configuration for cp40xx, lvs40xx
  • 17:14 mutante: brewster - power down, could not revive due to disk or SATA controller fail
  • 16:57 ottomata1: shutting down elastic1006 for reinstall
  • 16:45 mutante: powering brewster back on
  • 16:40 paravoid: powering up brewster
  • 16:13 mutante: deleted old svn apache config on formey, started apache
  • 15:22 paravoid: restarting virt0's salt-master, glance-api, glance-registry, keystone, nova-scheduler
  • 15:11 logmsgbot: manybubbles synchronized wmf-config/CirrusSearch-common.php 'SWAT Cirrus update to improve performance'
  • 15:09 logmsgbot: manybubbles synchronized php-1.23wmf21/extensions/CirrusSearch/ 'SWAT deploy to improve performance'
  • 14:48 paravoid: upgrading all snapshot* hosts
  • 14:38 paravoid: upgrading all packages & staggered restart of all of swift (ms-fe/ms-be)
  • 13:22 logmsgbot: reedy synchronized php-1.23wmf22/includes/api/ApiFeedRecentChanges.php 'I268d0a53067738ba96bee74c593358b0b28cc083'
  • 13:22 logmsgbot: reedy synchronized php-1.23wmf21/includes/api/ApiFeedRecentChanges.php 'I268d0a53067738ba96bee74c593358b0b28cc083'
  • 13:15 paravoid: staggered upgrades for all pending updates on all mw* boxes & restarting apaches/other core services
  • 11:08 mutante: brewster - shut down
  • 10:49 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
  • 10:05 apergos: had to toss extensions/Elastica on virt1000 and run git submodule update --init --recursive seems to be working now
  • 09:26 mutante: deleting huge pybal log on lvs3001
  • 09:01 mutante: brewster - stop lighttpd,bacula-fd,haproxy,dhcp3-server,rsync,nrpe,salt
  • 07:54 mutante: brewster - disabling puppet agent, removed from site.pp, revoke puppet cert
  • 03:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 14 03:22:58 UTC 2014 (duration 22m 57s)
  • 02:42 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-14 02:42:05+00:00
  • 02:23 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-14 02:22:58+00:00

April 13

  • 03:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 13 03:19:45 UTC 2014 (duration 19m 44s)
  • 02:39 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-13 02:39:11+00:00
  • 02:20 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-13 02:20:49+00:00

April 12

  • 05:03 logmsgbot: ori updated /a/common to I5f900190c: Replace $channel with $variant; make it Beta-only
  • 03:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 12 03:21:43 UTC 2014 (duration 21m 42s)
  • 02:42 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-12 02:42:07+00:00
  • 02:23 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-12 02:23:02+00:00

April 11

  • 23:55 RoanKattouw: Restarting stuck Jenkins
  • 23:35 K4-713: synchronized payments to af35b7b
  • 23:25 K4-713: synchronized payments to b321163
  • 19:50 ottomata: upgraded wikitech to MediaWiki 1.23wmf22, applied security patch
  • 18:19 ottomata: rebooting elastic1003
  • 18:14 Krinkle: git-deploy: Deploying integration/slave-scripts I38b90e8c08d7cb
  • 18:08 Krinkle: git-deploy: Deploying integration/slave-scripts I04d8e308daedb3ccb8
  • 17:41 Krinkle: git-deploy: Deploying integration/slave-scripts 'Ia9ee438fa2675170'
  • 14:27 ottomata: reinstalling elastic1005
  • 04:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 11 04:33:40 UTC 2014 (duration 33m 39s)
  • 03:47 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-11 03:47:01+00:00
  • 02:41 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-11 02:41:33+00:00
  • 02:29 ori_: graphite: carbon instance 'f' saturates a cpu core. it's the instance that mediawiki profiling data gets hashed to. collector should probably emit to statsd and have statsd compute per-minute rollups
  • 00:06 marktraceur: leaving MultimediaViewer slightly broken on enwiki based on the fact that logged-in users seem mostly unaffected and other wikis aren't seeing issues, will investigate more tomorrow and fix on Monday

April 10

  • 23:54 bd808: Enabled beta update Jenkins job (https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/)
  • 23:37 logmsgbot: mwalker Finished scap: Attempting to regenerate i18n keys for multimediaviewer (duration: 03m 33s)
  • 23:34 logmsgbot: mwalker Started scap: Attempting to regenerate i18n keys for multimediaviewer
  • 23:16 logmsgbot: mwalker synchronized wmf-config/filebackend.php
  • 23:09 logmsgbot: mwalker synchronized wmf-config/InitialiseSettings.php 'touched to see if that pushes changes to FileBackend.php'
  • 23:03 mwalker: sync-common for 125340 and 125335
  • 22:53 logmsgbot: krinkle synchronized php-1.23wmf21/extensions/VisualEditor/modules/ve-mw/ui/tools/ 'touch *.js'
  • 22:35 logmsgbot: krinkle synchronized php-1.23wmf21/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWReferenceDialogTool.js 'touch'
  • 22:12 logmsgbot: krinkle synchronized php-1.23wmf21/extensions/VisualEditor/lib/ve/modules/ve/ui/ve.ui.Toolbar.js 'touch'
  • 22:10 logmsgbot: krinkle synchronized php-1.23wmf21/extensions/VisualEditor/lib/ve/lib/oojs-ui/oojs-ui.js 'touch'
  • 22:10 logmsgbot: krinkle synchronized php-1.23wmf21/resources/startup.js 'touch'
  • 22:10 logmsgbot: krinkle synchronized php-1.23wmf21/resources/oojs-ui/oojs-ui.js 'touch'
  • 21:50 Krinkle: VisualEditor throws uncaught error on load for 1.23wmf21 wikis (bug 63791)
  • 21:15 bd808: Disabled beta update Jenkins job (https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/) so that scap testing can happen in beta.
  • 19:29 ottomata: reinstalling elastic1004
  • 19:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'I5501078cee871fb9df03e085547b7a047ef5bd7e'
  • 19:16 logmsgbot: ori synchronized wmf-config/InitialiseSettings-labs.php 'Ia79b1b848: Work around bug 63780 by specifying a siteParamsCallback'
  • 19:15 logmsgbot: ori updated /a/common to Ia79b1b848: Work around bug 63780 by specifying a siteParamsCallback
  • 18:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
  • 18:43 logmsgbot: reedy synchronized database lists files: Enable MediaViewer on mediawikiwiki
  • 18:42 logmsgbot: reedy synchronized docroot and w
  • 18:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf22
  • 18:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf21
  • 18:33 logmsgbot: reedy synchronized php-1.23wmf21/extensions/MultimediaViewer
  • 16:55 ottomata: shutting down elastic1003 for reinstall and reformat
  • 16:42 logmsgbot: reedy updated /a/common to Ie72029103: Add/update symlinks
  • 16:40 logmsgbot: reedy Finished scap: testwiki to 1.23wmf22 and build l10n cache (duration: 24m 45s)
  • 16:15 logmsgbot: reedy Started scap: testwiki to 1.23wmf22 and build l10n cache
  • 16:14 logmsgbot: reedy updated /a/common to I2cccebdd7: wikidatawiki back to 1.23wmf21
  • 15:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki back to 1.23wmf21...
  • 15:00 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 10 15:00:01 UTC 2014 (duration 27m 49s)
  • 14:17 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-10 14:17:16+00:00
  • 13:49 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-10 13:49:24+00:00
  • 13:31 logmsgbot: reedy Finished scap: l10n cache update for wikidatawiki (duration: 19m 15s)
  • 13:12 logmsgbot: reedy Started scap: l10n cache update for wikidatawiki
  • 12:42 bblack: removed broken pdns_gmetric cronjob on lvs boxes
  • 09:44 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I107179a27: Move HHVM extension blacklist below extract($globals) so it isn't simply clobbered'
  • 09:42 logmsgbot: ori updated /a/common to I107179a27: Move HHVM extension blacklist below extract($globals) so it isn't simply clobbered
  • 09:39 hashar: Zuul processed its backlog. Had to disconnect/reconnect the labs slaves. There is some weird bug occurring :-(
  • 09:29 hashar: Jenkins: disabling Gearman client in https://integration.wikimedia.org/ci/configure and reenabling it
  • 09:20 hashar: Jenkins unpooling both slave labs using the web interface and killing the Jenkins client running as jenkins-deploy . Will repool so the job can be reregistered properly [[bugzilla:63760|bug 63760]]
  • 09:11 mutante: DNS update - removing ms6
  • 09:04 hashar: Jenkins bunch of jobs are not being triggered properly. Taking traces.
  • 08:55 mutante: ms6 - shutdown -h now
  • 08:42 mutante: forcing Bugzilla logout for all users
  • 08:19 logmsgbot: aude synchronized php-1.23wmf20/extensions/Wikidata
  • 08:09 logmsgbot: aude synchronized php-1.23wmf20/extensions/Wikidata
  • 07:57 logmsgbot: aude rebuilt wikiversions.cdb and synchronized wikiversions files: Rebuild wikiversions and put wikidata on 1.23wmf20
  • 07:53 logmsgbot: aude synchronized wikiversions.json 'Put Wikidata back on 1.23wmf20, due to localisation cache issues'
  • 07:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 10 07:21:16 UTC 2014 (duration 7m 21s)
  • 06:46 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-10 06:45:59+00:00
  • 06:27 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-10 06:27:30+00:00
  • 06:27 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I20bbe05cc: Avoid using bits on beta-hhvm.wmflabs.org'
  • 06:26 logmsgbot: ori updated /a/common to I20bbe05cc: Avoid using bits on beta-hhvm.wmflabs.org
  • 06:15 ori: Some interface messages are missing on wikidata.org. Started a manual l10nupdate.
  • 04:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 10 04:11:47 UTC 2014 (duration 11m 46s)
  • 03:19 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-10 03:19:18+00:00
  • 03:01 logmsgbot: ori synchronized multiversion/MWMultiVersion.php 'Ibdbac982b: Update multiversion regexp for *.beta-hhvm.wmflabs.org'
  • 03:01 logmsgbot: ori updated /a/common to Ibdbac982b: Update multiversion regexp for *.beta-hhvm.wmflabs.org
  • 02:22 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-10 02:22:11+00:00
  • 01:49 logmsgbot: ori synchronized wmf-config/InitialiseSettings-labs.php 'I697f7e4a6: Use to branch on interpreter'
  • 01:48 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I697f7e4a6: Use to branch on interpreter'
  • 01:48 logmsgbot: ori updated /a/common to I697f7e4a6: Use '$channel' to branch on interpreter
  • 01:08 K4-713: updated payments to e1d00b61a703
  • 01:06 Krinkle: git-deploy: Deploying integration/slave-scripts If2539c
  • 01:05 Krinkle: Undid local patch to "grunt-lib-phantomjs/phantomjs/main.js" (for bug 63579) in "/srv/deployment/integration/slave-scripts" on gallium
  • 00:20 logmsgbot: yurik synchronized wmf-config/InitialiseSettings.php
  • 00:08 awight: updated crm from e726e42 to e3f2859
  • 00:06 K4-713: updated payments to 70dce8f4bc7

April 9

  • 23:36 logmsgbot: ebernhardson synchronized php-1.23wmf21/extensions/Math/modules/VisualEditor/ve.ui.MWMathInspectorTool.js 'Update Math VE tool to use a command in 1.23wmf21'
  • 23:32 logmsgbot: ebernhardson synchronized wmf-config/CommonSettings.php 'Update Flow cache version'
  • 23:22 logmsgbot: ebernhardson synchronized php-1.23wmf21/extensions/Flow 'Backport fix DB-to-cache pipeline for 1.23wmd21'
  • 23:05 logmsgbot: ebernhardson synchronized wmf-config/InitialiseSettings-labs.php 'Enable math VE plugin on labs'
  • 23:04 Krinkle: Jenkins and Zuul are back up. Queues have not been preserved.
  • 23:01 ^d: gerrit: reloaded bugzilla plugin to force it to log back in
  • 23:00 Krinkle: Restarting Jenkins because I have no clue what is going on and have no time to investigate yet another random clogging of all jobs. Restart ought to fix it.
  • 22:54 Krinkle: Zuul has lots of queued jobs for npm slaves, but neither Jenkins nor integration-slave1001.eqiad.wmflabs and 1002 themselves have anything queued. They're idle, responsive and waiting for jobs.
  • 22:47 Krinkle: Jenkins slaves in labs seem to be down. Zuul is stacking up jobs for hasNpm nodes (integration slaves in labs). Both slaves have 7/7 executors idle.
  • 22:33 hoo: Logged out all Bugzilla users by deleting all session cookie data from mysql
  • 19:15 logmsgbot: csteipp synchronized php-1.23wmf21/extensions/CentralAuth/maintenance
  • 19:10 logmsgbot: yurik synchronized wmf-config/InitialiseSettings.php
  • 17:38 logmsgbot: yurik synchronized wmf-config/InitialiseSettings.php
  • 17:22 manybubbles: regenerating Elasticsearch index from mediawiki for testwiki to soak up geo changes.
  • 16:48 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/124880'
  • 16:41 manybubbles: reindexed testwiki to soak up geo changes
  • 16:37 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/124876'
  • 16:32 logmsgbot: maxsem synchronized php-1.23wmf21/extensions/GeoData
  • 16:28 manybubbles: fiddling with Elasticsearch cluster balancing options trying to get enwiki better balanced
  • 16:17 logmsgbot: aude synchronized php-1.23wmf21/extensions/Wikidata 'Switch Wikidata back to previous version of Wikibase'
  • 15:52 mutante: ms6 - revoke puppet cert, salt key, remove from icinga
  • 15:02 ottomata: stopped puppet on emery to test sqstat on analytics1003
  • 14:48 ottomata: disabling puppet to test sqstat on analytics1003
  • 14:14 RobH: otrs back up, live hacked apache change, now working permanent puppet change (puppet is disabled on iodine at present)
  • 14:02 RobH: yes, otrs is totally ssl borked, robh is working on it
  • 14:00 mutante: adding filippo to ops/wmf LDAP groups
  • 13:58 RobH: updating otrs cert
  • 09:19 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '[] = 'musees.cg70.fr'; 124754 [[bugzilla:63449|bug 63449]]'
  • 08:39 hashar: Gerrit Letting JenkinsBot submit changes on apps/android/*
  • 03:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 9 03:33:25 UTC 2014 (duration 33m 24s)
  • 02:43 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-09 02:43:52+00:00
  • 02:19 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-09 02:19:25+00:00
  • 01:25 Krinkle: Bug 63579 is still happening occasionally. Leaving patch on gallium in place for now.
  • 01:09 ori: Debugging uWSGI init scripts on tungsten; expect some Graphite / Gdash flapping.
  • 00:15 ori: graphite webapp 502 caused by uwsgi's init script not restarting the service correctly
  • 00:07 Krinkle: graphite.wikimedia.org (e.g. https://graphite.wikimedia.org/render/?) is serving 502 Bad Gateway, ori is investigating
  • 00:04 Krinkle: To investigate bug 63579, manually patched "grunt-lib-phantomjs/phantomjs/main.js" in "/srv/deployment/integration/slave-scripts" on gallium

April 8

  • 23:34 logmsgbot: mwalker synchronized php-1.23wmf21/extensions/MultimediaViewer/ 'Updating MultimediaViewer for 124510'
  • 23:08 logmsgbot: csteipp synchronized php-1.23wmf21/extensions/CentralAuth/maintenance 'Push maintenance script for token reset'
  • 21:21 logmsgbot: bd808 Purged l10n cache for 1.23wmf19
  • 21:20 logmsgbot: bd808 Purged l10n cache for 1.23wmf18
  • 21:12 logmsgbot: bd808 rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.23wmf21
  • 19:58 manybubbles: finished upgrading to Elasticsearch 1.1.0. The process went well with no issues other then some knocking out search in labs 3 times for 30 seconds a piece. And logging lots of nasty warnings to irc. I've started to the process to fix search in labs so it won't happen again.
  • 19:56 manybubbles: upgraded all elasticsearch servers except elastic1008. that is coming now.
  • 18:45 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I4b18e4ce8: Change wgServer and wgCanonicalServer for arbcom wikis'
  • 18:45 logmsgbot: ori updated /a/common to I4b18e4ce8: Change wgServer and wgCanonicalServer for arbcom wikis
  • 18:13 logmsgbot: bd808 Finished scap: group0 wikis to 1.23wmf21 (with patch for bug 63659) (duration: 03m 18s)
  • 18:10 logmsgbot: bd808 Started scap: group0 wikis to 1.23wmf21 (with patch for bug 63659)
  • 18:01 logmsgbot: hoo synchronized wmf-config/InitialiseSettings.php 'Touch to clear config. cache'
  • 17:56 hoo: changed the Wikidata wb_changes_dispatch position of all wikiquote wikis to 118158153
  • 17:37 logmsgbot: hoo synchronized php-1.23wmf20/extensions/Wikidata/extensions/Wikibase/lib/resources/wikibase.Site.js 'touch'
  • 17:37 logmsgbot: aude synchronized wmf-config/Wikibase.php 'bump wgCacheEpoch for wikidata after enabling wikiquote site links'
  • 17:35 ottomata: restarted gmetad on nickel to fix ganglia
  • 17:29 logmsgbot: aude synchronized wikidataclient.dblist 'Enable Wikibase on Wikiquote'
  • 17:29 logmsgbot: aude synchronized wmf-config 'config changes to enable Wikibase on Wikiquote'
  • 17:22 logmsgbot: aude synchronized wmf-config/CirrusSearch-labs.php 'config change for beta, to enable highlighting'
  • 17:16 manybubbles: finished upgrading elastic1001-1006. starting on 1007. yay progress.
  • 17:03 logmsgbot: aude synchronized php-1.23wmf20/extensions/Wikidata 'Update Wikidata build, to allow populating sites table on wikiquote'
  • 16:31 aude: added sites and site_identifiers core tables on wikiquote
  • 16:28 hashar: Jenkins: killed jenkins-slave java process on gallium and repooled gallium slave. It was no more registered in Zuul :-/
  • 14:32 manybubbles: no harm done, just lost time
  • 14:32 manybubbles: woops, just restarted elastic1002. silly me
  • 14:31 manybubbles: upgrading elastic1001
  • 13:54 manybubbles: they'll pick it up during the rolling restart today to upgrade to 1.1.0
  • 13:53 manybubbles: synced first Elasticsearch plugin to production Elasticsearch servers
  • 13:46 RobH: upgraded libssl on holmium
  • 13:39 Jeff_Green: update & reboot tellurium
  • 13:39 RobH: replacing the blog cert, if holmium crashes I didn't do it correctly.
  • 13:37 mutante: restarting gitblit
  • 12:56 logmsgbot: reedy updated /a/common to Id15ddc665: Revert "Group0 wikis to 1.23wmf21"
  • 10:21 Jeff_Green: update & reboot barium
  • 10:15 Jeff_Green: update & reboot samarium
  • 07:47 _joe|away: restarted nginx on cp1044 and cp1043
  • 05:47 apergos: shot many old apache processes running as stats user from 2013, on stat1001 (restarting apache runs it as www-data user)
  • 05:39 apergos: restarted apache on fenari magnesium yterrbium antimony
  • 05:31 _joe_: upgraded openssl on cp10* and cp30* servers as well
  • 04:46 Tim: on dataset1001: upgraded libssl and restarted lighttpd
  • 04:43 Tim: restarted apache on the above list, failed on labs-ns1, virt1000, ytterbium
  • 04:41 Tim: upgraded libssl on zirconium.wikimedia.org,neon.wikimedia.org,netmon1001.wikimedia.org,iodine.wikimedia.org,ytterbium.wikimedia.org,gerrit.wikimedia.org,virt1000.wikimedia.org,labs-ns1.wikimedia.org,stat1001.wikimedia.org
  • 04:38 Ryan_Lane: upgrading libssl on virt0
  • 04:37 Ryan_Lane: upgrading libssl on virt1000
  • 04:15 Tim: also upgraded libssl on cp4001-4019. Restarted nginx on these servers and also the previous list.
  • 04:03 Tim: upgrading libssl on ssl1001,ssl1002,ssl1003,ssl1004,ssl1005,ssl1006,ssl1007,ssl1008,ssl1009,ssl3001.esams.wikimedia.org,ssl3002.esams.wikimedia.org,ssl3003.esams.wikimedia.org
  • 03:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 8 03:11:04 UTC 2014 (duration 11m 3s)
  • 02:34 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-08 02:34:56+00:00
  • 02:16 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-08 02:15:58+00:00
  • 01:06 logmsgbot: bd808 Finished scap: revert group0 to 1.23wmf21 (testwiki still on 1.23wmf21) (duration: 09m 54s)
  • 00:56 logmsgbot: bd808 Started scap: revert group0 to 1.23wmf21 (testwiki still on 1.23wmf21)
  • 00:54 logmsgbot: bd808 scap aborted: group0 to 1.23wmf21 (testing python change for mwversionsinuse) (again) (duration: 00m 25s)
  • 00:54 logmsgbot: bd808 Started scap: group0 to 1.23wmf21 (testing python change for mwversionsinuse) (again)
  • 00:53 logmsgbot: bd808 scap aborted: group0 to 1.23wmf21 (testing python change for mwversionsinuse) (duration: 02m 57s)
  • 00:50 logmsgbot: bd808 Started scap: group0 to 1.23wmf21 (testing python change for mwversionsinuse)
  • 00:25 logmsgbot: catrope synchronized php-1.23wmf21/extensions/VisualEditor 'it helps if you run git submodule update first'
  • 00:24 logmsgbot: catrope synchronized php-1.23wmf20/extensions/VisualEditor 'it helps if you run git submodule update first'

April 7

  • 23:58 logmsgbot: catrope synchronized php-1.23wmf20/extensions/VisualEditor/ 'VisualEditor bug fixes'
  • 23:57 logmsgbot: catrope synchronized php-1.23wmf20/skins/vector/variables.less 'Remove troublesome fonts from font stack'
  • 23:50 logmsgbot: catrope synchronized php-1.23wmf21/resources/oojs-ui/ 'Update OOJS-UI for bug fixes'
  • 23:50 logmsgbot: catrope synchronized php-1.23wmf21/extensions/VisualEditor/ 'VisualEditor bug fixes'
  • 23:49 logmsgbot: catrope synchronized php-1.23wmf21/skins/vector/variables.less 'Remove troublesome fonts from font stack'
  • 23:45 logmsgbot: catrope synchronized php-1.23wmf21/extensions/VisualEditor/ 'VisualEditor bug fixes'
  • 23:42 logmsgbot: catrope synchronized php-1.23wmf21/resources/oojs-ui/ 'Update OOJS-UI for bug fixes'
  • 23:42 logmsgbot: catrope synchronized php-1.23wmf21/skins/vector/variables.less 'Remove troublesome fonts from font stack'
  • 23:17 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'SWAT changes: other projects bar on frwikisource, import sources'
  • 23:06 logmsgbot: bd808 Finished scap: test2wiki to 1.23wmf20 (duration: 16m 48s)
  • 22:49 logmsgbot: bd808 Started scap: test2wiki to 1.23wmf20
  • 22:38 logmsgbot: bd808 Finished scap: test2wiki to 1.23wmf21 (duration: 12m 07s)
  • 22:26 logmsgbot: bd808 Started scap: test2wiki to 1.23wmf21
  • 22:12 logmsgbot: bd808 Finished scap: Testing 1.23wmf21 l10n changes (duration: 01m 31s)
  • 22:10 logmsgbot: bd808 Started scap: Testing 1.23wmf21 l10n changes
  • 22:07 logmsgbot: bd808 Finished scap: Testing 1.23wmf21 l10n changes (duration: 03m 49s)
  • 22:04 logmsgbot: bd808 Started scap: Testing 1.23wmf21 l10n changes
  • 21:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 7 21:13:14 UTC 2014 (duration 1m 59s)
  • 20:40 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-07 20:40:02+00:00
  • 20:23 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-07 20:23:14+00:00
  • 19:08 ottomata: temporatily disabling puppet on analytics 1009, 1010, 1019, 1020 to bring up new journalnodes
  • 18:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 7 18:39:21 UTC 2014 (duration 15m 30s)
  • 18:23 logmsgbot: aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Added "downloadtiff" pool counter config'
  • 18:13 AaronSchulz: shwiki queue finished emptying out in staggered loop on terbium
  • 18:03 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-07 18:03:48+00:00
  • 17:36 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-07 17:36:02+00:00
  • 17:24 bd808: Manually running l10nupdate with new --verbose flag to capture log output
  • 14:34 MaxSem: Rebuilding GeoData index
  • 14:09 hashar: Jenkins cleared swap on gallium (swapoff -a && swapon -a). Makes ganglia graph nicer :D
  • 13:15 apergos: reenabled puppet on dataset2, testing done
  • 12:23 apergos: disabled puppet on dataset2, testing
  • 11:19 logmsgbot: reedy Finished scap: because we're scappy... (rebuilding l10n cache for 1.23wmf21 (duration: 18m 04s)
  • 11:01 logmsgbot: reedy Started scap: because we're scappy... (rebuilding l10n cache for 1.23wmf21
  • 10:45 hashar: integration Getting PHP Composer installed on labs slaves. 124305
  • 09:21 paravoid: reactivating peerings with HE, issues reportedly resolved
  • 09:04 hashar: restarted Zuul
  • 08:54 hashar: gallium killed console-kit-daemon process which was eating a lot of memory
  • 08:42 hashar: Restarting Jenkins, out of Java heap space. Something is leaking memory
  • 08:41 hashar: Jenkins being broken for some reason AGAIN !
  • 05:04 ori: Zuul is stuck: <http://i.imgur.com/o5ghCam.jpg> (617kb image)
  • 02:56 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 7 02:56:12 UTC 2014 (duration 56m 11s)
  • 02:20 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-07 02:20:10+00:00
  • 02:13 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-07 02:13:42+00:00

April 6

  • 02:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 6 02:53:13 UTC 2014 (duration 53m 12s)
  • 02:18 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-06 02:18:43+00:00
  • 02:13 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-06 02:12:57+00:00
  • 01:32 jamesofu_: sugar down for move to labs

April 5

  • 04:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 5 04:15:00 UTC 2014 (duration 50m 39s)
  • 03:56 logmsgbot: andrew rebuilt wikiversions.cdb and synchronized wikiversions files: Revert mw.org, test2wiki and testwikidatawiki to 1.23wmf20 due to localisation issue
  • 03:51 Andrew: Reverting mw.org, test2 and test.wikidata back to 1.23wmf20
  • 03:41 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-05 03:41:36+00:00
  • 03:36 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-05 03:36:04+00:00
  • 03:23 Andrew: Actually, going to rerun l10nupdate first just to check.
  • 03:22 Andrew: Going to revert deployment of 1.23wmf21 again - still broken
  • 03:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 5 03:08:33 UTC 2014 (duration 8m 32s)
  • 02:34 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-05 02:34:54+00:00
  • 02:14 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-05 02:14:07+00:00

April 4

  • 21:34 logmsgbot: bd808 Finished scap: Group0 to 1.23wmf21 (again) (duration: 14m 35s)
  • 21:19 logmsgbot: bd808 Started scap: Group0 to 1.23wmf21 (again)
  • 19:28 hashar: Jenkins: unpooled slave agent on lanthanum, killed it the java agent on it and repooled it.
  • 19:22 hashar: Jenkins is processing jobs again. Queue unchanged so it will resume everything
  • 19:16 hashar: restarting Jenkins
  • 19:07 hashar: Jenkins un pooling gallium slave
  • 19:05 hashar: Zuul / Jenkins stalled again.
  • 18:43 csteipp: redeployed updated patch for bug63251 to fix a reported bug
  • 16:10 _joe_: restarting gitlbit, for the last time today
  • 15:06 _joe_: restarting gitblit as it has eaten up all of its ram again and is trashing cpu
  • 12:32 mutante: hume - shutting down
  • 12:06 mutante: hume - disable puppet/salt/monitoring
  • 11:13 mutante: restarting gitblit with new option to use incremental GC in an attempt to fix timeouts caused by GC eating CPU
  • 08:07 paravoid: deactivating cr1-eqiad<->HE peerings, translantic par2<->ash1 is congested
  • 07:25 mutante: restarting gitblit
  • 05:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 4 05:45:07 UTC 2014 (duration 18m 25s)
  • 04:56 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-04 04:56:06+00:00
  • 04:45 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-04 04:45:01+00:00
  • 04:20 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: unbreak test2.wp and test.wikidata as well
  • 04:17 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: mw.org back to 1.23wmf20
  • 03:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 4 03:43:03 UTC 2014 (duration 43m 2s)
  • 03:28 ori: Interface messages are missing on group0 / 1.23wmf21 wikis (mediawikiwiki, testwiki, test2wiki, and testwikidata)
  • 02:50 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-04 02:50:26+00:00
  • 02:24 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-04 02:24:51+00:00
  • 01:08 logmsgbot: krinkle synchronized php-1.23wmf21/resources 'I6e93d9ab0e4a926c09c'

April 3

  • 22:00 logmsgbot: demon synchronized wmf-config/CirrusSearch-production.php 'lowering cache time, for testing'
  • 21:55 logmsgbot: demon updated /a/common/php-1.23wmf20 to Ic853ebff4: Cherry-pick I550eb4b0a8fa18344e8b0de3ec85d61c2122ffb8
  • 21:54 logmsgbot: demon synchronized php-1.23wmf20/extensions/CirrusSearch 'Cirrus back to master again'
  • 21:50 logmsgbot: ori synchronized multiversion/updateBitsBranchPointers 'updateBitsBranchPointers: get rid of 'static-stable' branch link'
  • 21:50 logmsgbot: ori updated /a/common to Ic1602c045: updateBitsBranchPointers: get rid of 'static-stable' branch link
  • 21:46 logmsgbot: demon synchronized php-1.23wmf20/extensions/CirrusSearch 'Rolling back to 1.23wmf20 branch point from master'
  • 21:38 logmsgbot: demon synchronized php-1.23wmf20/extensions/CirrusSearch 'Updating Cirrus to master'
  • 21:33 logmsgbot: demon synchronized wmf-config/CirrusSearch-production.php 'italian wikis getting interwiki search. they're my favorite beta testers'
  • 19:23 logmsgbot: reedy synchronized docroot and w
  • 19:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf21
  • 19:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias actually to 1.23wmf20
  • 19:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf20
  • 19:09 logmsgbot: reedy Finished scap: testwiki to 1.23wmf21 and build l10n cache (duration: 38m 23s)
  • 18:30 logmsgbot: reedy Started scap: testwiki to 1.23wmf21 and build l10n cache
  • 18:23 logmsgbot: reedy updated /a/common to I835c2b1d5: Depool. See RT 7191.
  • 11:10 paravoid: IPv4 eqiad<->esams private link also elevated by ~15ms but no packet loss observed
  • 11:09 paravoid: affects both IPv6 transit at esams (slowdowns) as well as IPv6 eqiad<->esams
  • 11:08 paravoid: deactivating cr1-esams<->HE peering, latency > 160ms, over at 200ms (congestion?); back to 84ms now;
  • 10:51 akosiaris: temporarily stopped squid on brewster
  • 10:26 hashar: Jenkins job mediawiki-core-phpunit-hhvm is back around thanks to 123573
  • 06:28 paravoid: powercycling ms-be1003, unresponsive, no console output
  • 04:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'return upgraded DB slaves to normal load'
  • 04:11 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's6 repool db1015, warm up'
  • 04:04 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's6 depool db1015 for upgrade'
  • 04:03 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's5 repool db1037, warm up'
  • 03:53 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's5 depool db1037 for upgrade'
  • 03:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 3 03:53:18 UTC 2014 (duration 53m 16s)
  • 03:34 springle: db1020 raid controller dimm ecc errors
  • 03:14 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's4 depool db1020 for upgrade'
  • 03:12 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's3 repool db1019, warm up'
  • 02:57 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's3 depool db1019 for upgrade'
  • 02:56 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's2 repool db1060, warm up'
  • 02:48 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-03 02:48:01+00:00
  • 02:47 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's2 depool db1060 for upgrade'
  • 02:45 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's1 repool db1061, warm up'
  • 02:35 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's1 depool db1061 for upgrade'
  • 02:24 logmsgbot: LocalisationUpdate completed (1.23wmf19) at 2014-04-03 02:24:07+00:00

April 2

  • 23:47 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Bumped wgJobBackoffThrottling for htmlCacheUpdate to 15'
  • 23:47 mwalker: ... deploy was for mobile frontend 123454
  • 23:46 logmsgbot: mwalker synchronized php-1.23wmf20/extensions/MobileFrontend 'SWAT deploy for MaxSem'
  • 20:23 subbu: deployed Parsoid 33471172 with deploy repo sha 5c620e54
  • 19:03 logmsgbot: ori synchronized php-1.23wmf20/extensions/WikimediaEvents 'Update WikimediaEvents for I7fdaa5524: Use simple random sampling to log deprecated usage at 1:100'
  • 19:03 logmsgbot: ori synchronized php-1.23wmf19/extensions/WikimediaEvents 'Update WikimediaEvents for I7fdaa5524: Use simple random sampling to log deprecated usage at 1:100'
  • 17:00 andrewbogott: fixed updating crons on wikitech-status, I think. Time will tell...
  • 16:19 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Lower timeout on prefix searches and make the cirrus.dblist sync I just did take effect.'
  • 16:19 logmsgbot: manybubbles synchronized cirrus.dblist 'Cirrus as primary for most of group1'
  • 16:14 akosiaris: banned tools-exec-03.eqiad.wmflabs. using manual iptables on ytterbium
  • 15:20 ottomata: stopping puppet on stat1
  • 14:27 hashar: Jenkins applying label contintLabsSlave on slaves in labs used for ci (integration-slave1001 and 1002)
  • 14:15 hashar: Jenkins deleting pmtpa slaves (they all have been shutdown and jobs got deleted)
  • 14:00 manybubbles: tried restarting some lsearchd services (carefully) to clear out some crashing when searching for a particular query term. It caused pool queue full errors.... serves me right for trying?
  • 11:20 mutante: running CheckUser/maintenance/purgeOldData.php on all wikis
  • 09:42 akosiaris: rsynced brewster /srv to carbon
  • 09:34 mutante: restarting gitblit on antimony
  • 09:14 mutante: DNS update - removing capella
  • 09:09 mutante: DNS update - removing ms10
  • 05:31 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'normal loads for all upraded slaves'
  • 04:53 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's1 repool db1062, warm up'
  • 04:45 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's1 depool db1062 for upgrade'
  • 04:42 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's7 repool db1039, warm up'
  • 04:27 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's7 depool db1039 for upgrade'
  • 03:56 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's6 repool db1006, warm up'
  • 03:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 2 03:48:31 UTC 2014 (duration 48m 30s)
  • 03:46 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's6 depool db1006 for upgrade'
  • 03:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's5 repool db1045, warm up'
  • 03:27 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's5 depool db1045 for upgrade'
  • 03:21 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's4 repool db1059, warm up'
  • 03:07 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's4 depool db1059 for upgrade'
  • 03:04 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's2 repool db1063, warm up'
  • 02:55 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's2 depool db1063 for upgrade'
  • 02:52 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-02 02:52:48+00:00
  • 02:29 logmsgbot: LocalisationUpdate completed (1.23wmf19) at 2014-04-02 02:29:18+00:00
  • 02:22 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's3 repool db1027, warm up'
  • 02:03 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's3 depool db1027 for upgrade'
  • 01:16 logmsgbot: ori synchronized php-1.23wmf20/extensions/WikimediaEvents 'Undeployed change from earlier SWAT deploy'
  • 01:16 logmsgbot: ori synchronized php-1.23wmf19/extensions/WikimediaEvents 'Undeployed change from earlier SWAT deploy'
  • 01:05 logmsgbot: ori synchronized php-1.23wmf19/extensions/WikimediaEvents
  • 01:04 logmsgbot: ori synchronized php-1.23wmf19/extensions/EventLogging
  • 01:02 logmsgbot: ori synchronized php-1.23wmf20/extensions/WikimediaEvents
  • 01:02 logmsgbot: ori synchronized php-1.23wmf20/extensions/EventLogging

April 1

  • 23:48 logmsgbot: ebernhardson synchronized php-1.23wmf19/extensions/WikimediaEvents/ 'Update WikimediaEvents to master'
  • 23:48 logmsgbot: ebernhardson synchronized php-1.23wmf19/extensions/EventLogging/ 'Update EventLogging to master'
  • 23:47 logmsgbot: ebernhardson synchronized php-1.23wmf20/extensions/EventLogging/ 'Update EventLogging to master'
  • 23:46 logmsgbot: ebernhardson synchronized php-1.23wmf20/extensions/WikimediaEvents/ 'Update WikimediaEvents to master'
  • 23:32 logmsgbot: ebernhardson synchronized docroot and w
  • 21:42 hashar: Ganglia in labs is more or less back in activity: http://ganglia.wmflabs.org/ No clue what it is graphing though
  • 21:27 hashar: jenkins killed stuck build (5 hours+) of beta-update-databases-eqiad . Might have been blocking Jenkins build queue
  • 19:09 Reedy: ori gracefulled mw1018, mw1050, mw1061, mw1070, mw1139, mw1179
  • 18:45 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: zerowiki to 1.23wmf20
  • 18:43 logmsgbot: reedy updated /a/common to If887effe5: Add zerowiki
  • 18:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
  • 18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Add zerowiki
  • 18:41 logmsgbot: reedy synchronized database lists files:
  • 18:37 logmsgbot: reedy synchronized docroot and w
  • 18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: non wikipedias to 1.23wmf20
  • 18:28 mutante: ms10 - shut down
  • 18:19 mutante: ms10 - disable puppet, revoke puppet cert,salt key,icinga..
  • 18:03 logmsgbot: ori synchronized php-1.23wmf20/includes/profiler/ProfilerSimple.php 'Iad91f1d12: Send profiled items under the correct name'
  • 18:02 logmsgbot: ori synchronized php-1.23wmf19/includes/profiler/ProfilerSimple.php 'Iad91f1d12: Send profiled items under the correct name'
  • 17:34 mutante: logging to eqiad wikitech after Andrew switched over
  • 16:05 andrewbogott: switching wikitech to read-only, migrating to eqiad
  • 15:06 logmsgbot: reedy updated /a/common to If3ca3d486: beta: adjust $wgCaptchaDirectory
  • 15:01 hashar: Gerrit super slow again :-(
  • 14:46 mutante: added oblivion to root-auth-keys
  • 14:17 mutante: welcome new shell user oblivion
  • 14:04 hashar: Gerrit flushed a few caches related to user accounts / LDAP
  • 13:43 mutante: adding oblivion to ops and wmf LDAP groups
  • 08:44 mutante: solr1/2 - revoke puppet certs
  • 08:43 mutante: solr3 - delete salt key, puppet cert
  • 03:11 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's7 db1034 full steam'
  • 02:02 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
  • 01:58 springle: killed research queries on db1047. email me
  • 01:35 springle: restarted sanitarium s3 instance for additional private wikis
  • 00:59 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's7 repool db1034, warm up'


2000s

2010s

2020s