Server Admin Log

2017-10-20

10:14 moritzm: upgrading hhvm-wikidiff2 to 1.5.1 on mw2150/mw2151/mw2244/mw2245
09:17 moritzm: upgrading hhvm-wikidiff2 to 1.5.1 on mw2200-mw2223
08:05 Amir1: mwscript createAndPromote.php --wiki=techconductwiki --sysop --bureaucrat 'Ladsgroup' --force
07:39 jynus: stopping db2033, too
07:36 jynus: stopping dbstore1002 and dbstore2002 x1 replication in sync
07:18 _joe_: uploading updated versions of docker base images (wikimedia-jessie, wikimedia-stretch) to docker-registry.wikimedia.org
06:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s)
06:17 marostegui: Stop replication in sync on db1103 and db1078 - T164488
06:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 45s)
06:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 - T164488 (duration: 00m 46s)
05:52 marostegui: Stop replication in sync on db1103 and db1072 - T164488
05:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1051 - T174509 (duration: 00m 47s)
02:19 bblack: upgrading vhtcpd to 0.1.1-1 on: cp1008, cp3030, cp3034, cp4029-32
02:17 bblack: uploaded vhtcpd-0.1.1-1 to reprepro

2017-10-19

22:21 addshore: stop running extra change dispatchers on terbium for wikidatawiki
21:58 addshore: running extra change dispatchers on terbium for wikidatawiki for a short while
21:24 andrewbogott: restarting nginx on nitrogen
21:22 andrewbogott: restarting puppetdb on nitrogen
20:00 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: deploying refinery to fix banner activity false alarms (duration: 00m 14s)
19:59 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: deploying refinery to fix banner activity false alarms
19:53 bblack: reboot cp4029-32 (initial reboot post-puppetization)
19:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.4
18:36 thcipriani@tin: Synchronized portals: SWAT: Bumping portals to master T128546 (duration: 00m 51s)
18:35 thcipriani@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master T128546 (duration: 00m 50s)
18:28 thcipriani@tin: Synchronized wmf-config/abusefilter.php: SWAT: Enable $wgAbuseFilterProfile on ptwiki T177641 (duration: 00m 50s)
18:20 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/Echo/modules/styles/mw.echo.ui.PageNotificationsOptionWidget.less: SWAT: mw.echo.ui.PageNotificationsOptionWidget: Fix CSS after changes in OOjs UI T178439 (duration: 00m 51s)
16:50 hashar: contint1001: installed python3 (pending https://gerrit.wikimedia.org/r/385210 )
15:14 marostegui: Stop replication in sync on db1103 and db1072 - https://phabricator.wikimedia.org/T164488
15:02 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: (no justification provided) (duration: 00m 07s)
15:02 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: (no justification provided)
14:36 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: (no justification provided) (duration: 03m 29s)
14:33 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: (no justification provided)
13:20 moritzm: upgrading mw2120-mw2147 to wikidiff2 1.5.1
12:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034 and db2036 - T178359 (duration: 00m 50s)
11:40 akosiaris: T177196 upload prometheus-postgres-exporter_0.2.0+ds-2 to apt.wikimedia.org/stretch-wikimedia/main and copied over to apt.wikimedia.org/jessie-wikimedia/main
11:10 moritzm: upgrading mw1276-mw1279 (canary servers) to wikidiff2 1.5.1
10:14 godog: upgrade prometheus-hhvm-exporter to 0.4-1
10:07 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
09:54 ppchelko@tin: Finished deploy [restbase/deploy@2001a66]: Enable summary on Cass 3 for all but wikipedia (duration: 07m 32s)
09:50 ariel@tin: Finished deploy [dumps/dumps@2b9326b]: improved job batches for rev content 7z recompression (duration: 00m 02s)
09:50 ariel@tin: Started deploy [dumps/dumps@2b9326b]: improved job batches for rev content 7z recompression
09:48 moritzm: installing pyjwt security updates
09:47 ppchelko@tin: Started deploy [restbase/deploy@2001a66]: Enable summary on Cass 3 for all but wikipedia
08:13 marostegui: Stop replication in sync on db1103 and db1072 - T164488
08:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 - T164488 (duration: 00m 50s)
07:31 marostegui: Stop MySQL on db2034 and db2036 to clone db2092
06:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2034 and db2036 - T178359 (duration: 01m 08s)
06:02 marostegui: Compress revision table across db1044 databases (s3)
05:46 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1102 for s6 - T174509 T177772
05:45 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1064 - T174509 T177772
05:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T174509 (duration: 00m 50s)
05:41 marostegui: Optimize pagelinks and templatelinks on db1051 - T174509
05:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 - T174509 (duration: 00m 50s)
05:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 - T174509 (duration: 00m 50s)
05:29 awight@tin: Started deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support (take 2), T178441
05:08 awight@tin: Finished deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support, T178441 (duration: 00m 18s)
05:08 awight@tin: Started deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support, T178441
03:38 bblack: cp3034 - upgraded vhtcpd to 0.1.0 for testing
03:38 bblack: cp3030 - upgraded vhtcpd to 0.1.0 for testing
03:01 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 19 03:01:54 UTC 2017 (duration 7m 2s)
02:54 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 09m 28s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 09m 40s)
00:01 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/385106/2 (duration: 00m 50s)

2017-10-18

23:51 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/385094/2 (duration: 00m 50s)
23:49 maxsem@tin: Synchronized static/images/project-logos/: https://gerrit.wikimedia.org/r/#/c/385094/2 (duration: 00m 50s)
23:45 maxsem@tin: Synchronized php-1.31.0-wmf.4/extensions/Collection/: https://gerrit.wikimedia.org/r/#/c/384903/ (duration: 00m 50s)
23:44 maxsem@tin: Synchronized php-1.31.0-wmf.3/extensions/Collection/: https://gerrit.wikimedia.org/r/#/c/385005/ (duration: 00m 51s)
23:34 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/385007/ (duration: 00m 51s)
23:05 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.4
23:04 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
22:55 tgr@tin: Synchronized private/: Remove the llama (T150554) (duration: 00m 50s)
22:27 tgr@tin: Synchronized wmf-config/: T174651 / gerrit 384908 - deploy ReadingLists to beta - noop for production (duration: 00m 53s)
20:49 ejegg: updated CiviCRM from 7c26791 to b9d2067
20:16 XioNoX: Changing MTUs on interfaces to NTT
19:19 bblack: uploaded vhtcpd-0.1.0-1 to jessie-wikimedia, testing on cp1008 only for now
19:17 andrewbogott: restarting nodepool
19:15 andrewbogott: stopping nodepool temporarily to give rabbit a chance to catch up
19:01 MaxSem: ran LoginNotify/maintenance/migratePreferences.php on test and test2
18:57 andrewbogott: restarting nova-network on labnet1001
18:42 andrewbogott: restarting rabbitmq-server on labcontrol1001; too many timeouts
16:29 elukey: stop ircecho on einstenium for puppet shower
16:27 herron: restarted apache on puppetmasters
14:23 moritzm: uploaded hhvm 3.18.5+dfsg-1+wmf1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
14:03 chasemp: add static ip to cloud mwstake project per T178012
14:03 chasemp: bump up quota on cyberpower cloud project per T178332
14:03 chasemp: create reading-lists cloud project
13:28 zeljkof: EU SWAT finished!
13:22 zfilipin@tin: Synchronized php-1.31.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: [cirrus] Turn on recall A/B test on enwiki (T177502) (duration: 00m 50s)
13:21 gehel: repooling wdqs1003 now that it has catched up on updates
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: pagePreviews: Restart A/B test on enwiki and dewiki (T176469) (duration: 00m 51s)
13:05 moritzm: uploaded wikidiff2 1.5.1 to apt.wikimedia.org
12:11 mobrovac@tin: Finished deploy [restbase/deploy@3c7abf6]: Bug fix: Remove the space in the summary table name (duration: 07m 13s)
12:04 mobrovac@tin: Started deploy [restbase/deploy@3c7abf6]: Bug fix: Remove the space in the summary table name
11:54 mobrovac@tin: Finished deploy [restbase/deploy@99052c1]: Use Cass3 storage for ruwiki summaries (duration: 08m 59s)
11:47 jynus@tin: Synchronized wmf-config/db-eqiad.php: Add db1105 to help db1071 because db1082 has crashed (duration: 00m 50s)
11:45 mobrovac@tin: Started deploy [restbase/deploy@99052c1]: Use Cass3 storage for ruwiki summaries
11:31 moritzm: upgrading mw1262-mw1265 (canaries) to wikidiff2 1.5.1
11:26 marostegui: Optimize pagelinks and templatelinks on db1102 s7 - T174509
11:21 moritzm: upgrading mw1261 to wikidiff2 1.5.1
11:11 mobrovac@tin: Finished deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now), part #2 (duration: 08m 53s)
11:03 mobrovac@tin: Started deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now), part #2
11:02 mobrovac@tin: Finished deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now) (duration: 05m 23s)
10:57 mobrovac@tin: Started deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now)
10:50 moritzm: rebooting multatuli for kernel update
10:21 moritzm: imported linux 4.9.30-2+deb9u5~bpo8+1 for jessie-wikimedia
09:51 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098, depool db1082 (crashed) (duration: 00m 50s)
09:39 moritzm: installing xserver/xvfb security updates
09:36 jynus: reloading dbproxy1010's haproxy configuration
09:30 elukey: drop MobileWikiAppToCInteraction_10375484_15423246 from the log database on dbstore1002,db1047,db1046 - T177960
08:41 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Re-enable Statement usage tracking on cawiki (T151717) (duration: 00m 50s)
08:25 marostegui: Stop MySQL on db2019 to copy its data to db2084 - T178359
08:23 mobrovac@tin: Finished deploy [changeprop/deploy@065a06e]: Bug fix: Apply ignore_errors on a per-request basis (duration: 01m 17s)
08:22 mobrovac@tin: Started deploy [changeprop/deploy@065a06e]: Bug fix: Apply ignore_errors on a per-request basis
08:09 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable description usage tracking on a few test wikis (T177155) (duration: 00m 50s)
07:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T174509 (duration: 00m 49s)
07:46 marostegui: Optimize pagelinks and templatelinks on db1056 - T174509
07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 - T174509 (duration: 00m 59s)
07:14 gehel: depooling wdqs1003 until it catches up on updates
07:02 marostegui: Stop MySQL on db2079 to copy its data to db2084 - T178359
06:46 _joe_: powercycling wdqs1003
05:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 - T164488 (duration: 00m 49s)
05:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T174509 (duration: 00m 50s)
05:40 marostegui: Optimize pagelinks and templatelinks on db1055 - T174509
05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 - T174509 (duration: 00m 49s)
05:36 marostegui: Optimize pagelinks and templatelinks on db1098 - T174509
05:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1067 - T174509 (duration: 00m 50s)
05:25 marostegui: Optimize pagelinks and templatelinks on db1102 for s2 - T174509
03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 18 03:17:01 UTC 2017 (duration 7m 3s)
03:09 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 14m 58s)
02:33 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 08m 21s)

2017-10-17

23:31 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/Citoid/modules/ve.ui.CiteFromIdInspector.css: SWAT: ve.ui.CiteFromIdInspector: Fix CSS for context menus after changes in OOjs UI T178324 (duration: 00m 50s)
23:29 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/styles/widgets/ve.ui.MWMediaInfoFieldWidget.css: SWAT: ve.ui.MWMediaInfoFieldWidget: Fix positioning of icons T178415 (duration: 00m 50s)
23:22 thcipriani@tin: Synchronized php-1.31.0-wmf.3/includes/shell/Command.php: SWAT: Shell\Command: Better walltime fallback T178314 (duration: 00m 51s)
23:15 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Remove setting no longer in MediaWiki (duration: 00m 50s)
23:06 mutante: disabling Icinga notifications for service recommendation_api on scb hosts - please remember to re-enable once ticket is resolved (T178445) (https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=recommendation_api%20endpoints)
22:28 ejegg: updated CiviCRM from fb513cc to 7c26791
21:55 foks: Removed 2FA from User:Amakuru
21:39 robh: cp4026 being returned to service post memory replacement. puppet runs fine and all icinga checks are green. repooled.
21:23 robh: cp4026 coming down for memory swap, no point in maint moding the system since other cp checks will alert anyhow
20:14 mutante: gerrit2001 - re-enabled puppet, attempting restart with --slave after gerrit:384758
19:54 mutante: cobalt - re-enabled puppet, ran puppet, systemd says gerrit is active(running)
19:51 mutante: cobalt - stopping gerrit service, confirming pid file gets removed, move /etc/init.d/gerrit to /root
19:50 mutante: short gerrit downtime for systemd change coming up
19:23 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.4
19:06 mutante: gerrit2001 - rebooting to verify systemd change and service comes back properly
18:53 mutante: gerrit2001: stopping the gerrit service
18:52 mutante: cobalt: temp disable puppet | gerrit2001: stop puppet
18:22 paravoid: rebooting sodium for kernel/distro upgrade
18:14 paravoid: dist-upgrading sodium to stretch
18:12 demon@tin: Finished scap: bootstrap wmf.4 (duration: 48m 41s)
17:23 demon@tin: Started scap: bootstrap wmf.4
16:49 demon@tin: Pruned MediaWiki: 1.31.0-wmf.2 [keeping static files] (duration: 01m 24s)
16:35 gehel: restart tilerator / tileratorui on all maps cluster after deployment of latest fix - T177389
16:34 gehel@tin: Finished deploy [tilerator/deploy@f3b26f3]: fix tileshell not exiting - T177389 (duration: 01m 42s)
16:32 gehel@tin: Started deploy [tilerator/deploy@f3b26f3]: fix tileshell not exiting - T177389
15:46 _joe_: killed a stuck gerrit process on cobalt
14:47 bd808: wikitech: Cleaned up 'importers' user_group entries (T171682)
14:25 Amir1: end of cleaning up ores_classification table in plwiki, start of ruwiki (T159753)
14:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 - T174509 T177772 (duration: 00m 45s)
14:05 Amir1: start of cleaning up ores_classification table in plwiki (T159753)
14:00 zeljkof: EU SWAT finished
13:59 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: [cirrus] Fix typo in UserTesting config var (take 2) (T177502) (duration: 00m 45s)
13:47 zeljkof: EU SWAT continues
13:31 zeljkof: stopping EU SWAT
13:29 jynus: killing commonswiki script on tin
13:17 zfilipin@tin: Synchronized php-1.31.0-wmf.3/extensions/UniversalLanguageSelector/maintenance/ULSCompactLinksDisablePref.php: SWAT: Remove the 20 edits threshold from ULSCompactLinksDisablePref.php (duration: 00m 46s)
12:53 volans: upgrading cumin masters to cumin 1.2.2-1
12:40 moritzm: upgrading mwdebug* to hhvm-wikidiff 1.5.1
11:47 marostegui: Optimize recentchanges on db1102 for s4 - T174509
11:47 marostegui: Optimize templatelinks and pagelinks on db1102 for s4 - T174509
11:45 Amir1: end of cleaning up ores_classification table in hewiki, start of nlwiki (T159753)
11:39 Amir1: end of cleaning up ores_classification table in fiwiki, start of hewiki (T159753)
11:32 jynus: test-installing proxysql on wasat T175672
11:24 Amir1: end of cleaning up ores_classification table in fawiki, start of fiwiki (T159753)
11:12 gehel: restarting wdqs-updater on wdqs1004 - T175919
10:40 Amir1: end of cleaning up ores_classification table in etwiki, start of fawiki (T159753)
10:36 Amir1: end of cleaning up ores_classification table in cswiki, start of etwiki (T159753)
10:26 Amir1: start of cleaning up ores_classification in cswiki (T159753)
10:16 akosiaris: removing akosiaris's yubikey from network devices
09:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 - T174509 T177772 (duration: 00m 46s)
09:23 marostegui: Stop MySQL on db2084 for multi-instance testing
09:16 mobrovac@tin: Finished deploy [restbase/deploy@7f228d4]: Remove /page/revision and add history metrics end points, part #3 (duration: 07m 34s)
09:08 mobrovac@tin: Started deploy [restbase/deploy@7f228d4]: Remove /page/revision and add history metrics end points, part #3
09:08 mobrovac@tin: Finished deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points, part #2 (duration: 25m 57s)
08:45 gehel: restarting blazegraph on wdqs1004 for GC tuning (adding -XX:+G1PrintRegionLivenessInfo) - T175919
08:42 mobrovac@tin: Started deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points, part #2
08:40 mobrovac@tin: Finished deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points - T158100 T175805 (duration: 01m 05s)
08:39 mobrovac@tin: Started deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points - T158100 T175805
07:54 marostegui: Stop MySQL and reboot db2081 to see if it works fine - T178140
07:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2086 to the config - T170662 (duration: 00m 45s)
07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2086 to the config - T170662 (duration: 00m 45s)
07:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2086 to the config - T170662 (duration: 00m 45s)
07:42 marostegui: Stop MySQL on db2079 to clone db2086 - T170662
07:36 moritzm: rebooting boron for kernel update
07:31 ema: upgrade misc_eqiad to varnish 5 T177233
07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1067 - T174509 (duration: 00m 45s)
07:11 marostegui: Optimize templatelinks and pagelinks on db1067 - T174509
07:10 marostegui: Optimize enwiki.ores_classification on db1067 - T159753
06:55 moritzm: installing libffi security updates on trusty (Debian already fixed)
06:07 marostegui: Stop replication in sync on db1072 and db1103 for data drift fixing - T164488
05:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T174509 (duration: 00m 45s)
05:54 marostegui: Optimize pagelinks and templatelinks on db1034 - T174509
05:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1085 - T174509 T177772 (duration: 00m 45s)
05:46 marostegui: Optimize pagelinks, templatelinks and recentchanges on db1085 T177772 T174509
05:42 marostegui: Optimize pagelinks, templatelinks and ores_classification on db1095 - T174509 T159753
05:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T174509 (duration: 00m 45s)
05:38 marostegui: Optimize recentchanges on db1081 - T177772
05:37 marostegui: Optimize pagelinks and templatelinks on db1081 - T174509
05:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174509 (duration: 00m 45s)
05:32 marostegui: Optimize pagelinks and templatelinks on db1090 - T174509
05:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1073 - T174509 (duration: 00m 46s)
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 17 02:32:30 UTC 2017 (duration 6m 57s)
02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 08m 23s)
00:34 ejegg: updated CiviCRM from 824c2d8 to fb513cc

2017-10-16

23:49 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/CheckUser/specials/SpecialCheckUser.php: SWAT: Revert "Revert "Restore checkuser-userlinks when exist"" T170507 (duration: 00m 46s)
23:46 ebernhardson: increase elasticsearch log levels for transport to debug to help track down T167410
23:39 ejegg: updated civicrm from b953064 to 824c2d8
23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/Echo/includes/ContainmentSet.php: SWAT: ContainmentSet: Use strict comparison for array_search() T177825 (duration: 01m 45s)
20:35 chasemp: disable puppet around cloud things for a refactor merge
20:20 demon@tin: Pruned MediaWiki: 1.30.0-wmf.18 (duration: 02m 59s)
19:59 demon@tin: Synchronized wmf-config/CommonSettings.php: Ief0ea01f (duration: 00m 46s)
19:58 jynus: sudo apt install wmf-mariadb101-client on terbium for debugging an issue
19:57 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Ief0ea01f (duration: 00m 46s)
19:56 demon@tin: Synchronized tests/cirrusTest.php: Ief0ea01f (duration: 00m 47s)
18:18 thcipriani@tin: Synchronized wmf-config/Wikibase.php: SWAT: Add configuration for statement indexing for Wikidata T175199 (duration: 00m 47s)
17:35 ariel@tin: Finished deploy [dumps/dumps@2b9326b]: do the minimum for latestlinks job, no updating other files (duration: 00m 03s)
17:35 ariel@tin: Started deploy [dumps/dumps@2b9326b]: do the minimum for latestlinks job, no updating other files
17:08 gehel@tin: Finished deploy [wdqs/wdqs@b42eb0d]: WDQS GUI uipdates (duration: 02m 20s)
17:06 gehel@tin: Started deploy [wdqs/wdqs@b42eb0d]: WDQS GUI uipdates
16:56 ottomata: deploying LVS for druid-public-broker https://phabricator.wikimedia.org/T176223
16:39 joal@tin: Finished deploy [analytics/aqs/deploy@339674c]: Deploy new mediawiki-history endpoints (alpha version) after monitorin fix (duration: 04m 42s)
16:35 joal@tin: Started deploy [analytics/aqs/deploy@339674c]: Deploy new mediawiki-history endpoints (alpha version) after monitorin fix
16:23 joal@tin: Finished deploy [analytics/aqs/deploy@aa7ef0d]: Deploy new mediawiki-history endpoints (alpha version) after node-modules fix (duration: 06m 56s)
16:16 joal@tin: Started deploy [analytics/aqs/deploy@aa7ef0d]: Deploy new mediawiki-history endpoints (alpha version) after node-modules fix
16:09 ema: upgrade misc_codfw to varnish 5 T177233
16:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 - T174509 T177772 (duration: 00m 46s)
15:51 gehel: killing stuck tileshell process on maps-test2001 - T177389
15:51 gehel@tin: Finished deploy [tilerator/deploy@f3b26f3]: testing tileshell fix - T177389 (duration: 00m 20s)
15:50 gehel@tin: Started deploy [tilerator/deploy@f3b26f3]: testing tileshell fix - T177389
15:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1094 - T174509 (duration: 00m 46s)
15:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1088 - T174509 T177772 (duration: 00m 47s)
14:34 twentyafterfour: deploying hotfix for T178068 and T178107
14:30 moritzm: uploaded wikidiff 1.5.1 to apt.wikimedia.org for jessie/experimental
14:08 hashar: European swat is complete
14:08 hashar@tin: Synchronized wmf-config/abusefilter.php: (T177760) Enable AbuseFilter profiler | (T177761) Allow AbuseFilter to block & grant relevant permissions to permit administrators to manage the restricted option. (duration: 00m 47s)
13:50 moritzm: upgrading deployment-mediawiki0[56] to wikidiff2 1.5.1
13:45 hashar@tin: Synchronized wmf-config/: [cirrus] Prepare profiles for the recall A/B test on enwiki - T177502 (duration: 00m 48s)
13:44 moritzm: upgrading deployment-mediawiki04 to wikidiff2 1.5.1
13:42 hashar@tin: Synchronized wmf-config: [cirrus] Disable A/B test of MLR on testing on 18 wikis with > 1% of search traffic - T177490 (duration: 00m 48s)
13:38 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add additional namespaces to search results for bnwikisource - T178041 (duration: 00m 49s)
12:59 gehel: restart cassandra on maps1001
12:38 ema: upgrading eqiad LVSs to pybal 1.14.2 (T178149, T177815)
11:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 - T174509 (duration: 00m 46s)
11:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1070 - T174509 (duration: 00m 46s)
11:10 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
10:48 Amir1: cleaning up ores_classification table in wikidatawiki (T159753)
10:03 ema: upgrading codfw LVSs to pybal 1.14.2 (T178149, T177815)
09:54 ema: upgrading esams LVSs to pybal 1.14.2 (T178149, T177815)
09:44 ema: upgrading ulsfo LVSs to pybal 1.14.2 (T178149, T177815)
09:43 ema: pybal 1.14.2 uploaded to apt.w.o
09:40 moritzm: restarting cassandra on restbase1007 to pick up NSS security update
09:39 moritzm: restarting cassandra on cerium/xenon/praseodymium to pick up NSS security update
09:37 gehel: restarting elasticsearch on elastic1017
09:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1088 - T174509 T177772 (duration: 00m 46s)
09:07 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1088 - https://phabricator.wikimedia.org/T174509 https://phabricator.wikimedia.org/T177772
09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 - T174509 T177772 (duration: 00m 46s)
08:53 marostegui: Stop MySQL on db1050 as it will be decommissioned - T178162
08:46 marostegui: Remove db1050 from tendril - T178162
08:31 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1050 from config - T178162 (duration: 00m 46s)
08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1050 from config - T178162 (duration: 00m 46s)
07:56 marostegui: Stop db1103 and db1038 at the same position to fix data drifts - T164488
07:42 moritzm: installing NSS security updates
07:30 marostegui: Stop db1103 and db1072 at the same position to fix data drifts - T164488
07:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 - T164488 (duration: 00m 46s)
07:19 marostegui: Optimize table ores_classification on db1073 - T159753
07:14 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2084 to the config - T170662 (duration: 00m 46s)
07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2084 to the config - T170662 (duration: 00m 47s)
06:25 marostegui: Stop MySQL on db2079 to clone db2084 from it - T170662
06:12 marostegui: Optimize pagelinks and templatelinks on db1094 - T174509
06:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1093 - T174509 T177772 (duration: 00m 46s)
06:04 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1093 - T174509 T177772
06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1070 - T174509 (duration: 00m 46s)
05:59 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1070 - T174509
05:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T174509 T177772 (duration: 00m 46s)
05:53 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1084 - T174509 T177772
05:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T174509 (duration: 00m 46s)
05:47 marostegui: Optimize pagelinks and templatelinks on db1074 - T174509
05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1073 - T174509 (duration: 00m 47s)
05:37 marostegui: Optimize templatelinks and pagelinks on db1073 - T174509
05:36 marostegui: Optimize templatelinks and pagelinks on db1073 - 174509
02:35 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 16 02:35:34 UTC 2017 (duration 6m 36s)
02:28 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 07m 44s)

2017-10-15

11:15 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916

2017-10-14

03:52 paravoid: manually cleaning up pdns_metric_cron from all dnsrecursors (hydrogen/chromium/acamar/achernar/maerlant/nescio)

2017-10-13

23:04 mutante: scb1001 - systemctl restart pdfrender (Icinga said: pdfrender Connection refused)
20:36 mutante: gerrit2001 - removing hhvm package (which is now possible without also removing scap, thanks thcipriani), apt-get autoremove to remove more unused packages (T178039)
18:52 joal@tin: Finished deploy [analytics/aqs/deploy@0375c34]: Deploy new mediawiki-history endpoints-alpha version (duration: 15m 02s)
18:40 joal@tin: (no justification provided)
18:37 joal@tin: Started deploy [analytics/aqs/deploy@0375c34]: Deploy new mediawiki-history endpoints-alpha version
16:54 otto@tin: Finished deploy [analytics/refinery@28db253]: (no justification provided) (duration: 05m 50s)
16:48 otto@tin: Started deploy [analytics/refinery@28db253]: (no justification provided)
15:31 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 (duration: 00m 46s)
14:59 mutante: added cparle to wmf LDAP group
14:38 jynus: stopping slave on db1056 and optimize recentchanges
14:25 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1100, depool db1056, fix db1098 weight (duration: 00m 46s)
14:06 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 (duration: 00m 46s)
13:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 (duration: 00m 47s)
12:57 godog: upload scap 3.7.1-1 - T127762
11:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2082 to the config - T170662 (duration: 00m 46s)
11:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2082 to the config - T170662 (duration: 00m 46s)
10:53 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098 (duration: 00m 47s)
10:22 godog: roll-restart pybal to apply https://gerrit.wikimedia.org/r/#/c/383997/ - T178149
10:04 jynus: stopping slave on db1098 and optimize recentchanges
09:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 (duration: 00m 46s)
09:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2082 to the config - T170662 (duration: 00m 46s)
09:22 Amir1: a small clean up of ores_classification table in wikidatawiki (T159753)
09:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2082 to the config - T170662 (duration: 00m 47s)
09:04 godog: restart pybal on lvs1003 to test logstash depool - T178078
08:57 marostegui: Stop db1103 and db1038 in sync for more checksumming - T164488
08:34 ema: upgrade pybal on lvs1003 to 1.14.1 T177815
08:18 ema: upgrade pybal on lvs1006 to 1.14.1 T177815
08:15 jynus: restarting commons wiki recentchanges purge of wb entries T177772
08:11 ema@neodymium: conftool action : set/pooled=yes; selector: name=logstash1001.eqiad.wmnet,service=logstash-gelf,dc=eqiad
08:11 ema@neodymium: conftool action : set/pooled=no; selector: name=logstash1001.eqiad.wmnet,service=logstash-gelf,dc=eqiad
08:10 ema: depool/repool logstash1001 (service=logstash-gelf) T178078
07:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2081 to the config - T170662 (duration: 00m 46s)
07:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2081 to the config - T170662 (duration: 00m 47s)
06:23 marostegui: Stop MySQL on db2080 to copy its data to db2081 - T170662
00:22 mutante: authdns servers: deleted /etc/ganglia/conf.d/gndsd.pyconf & /usr/lib/ganglia/python_modules_gdnsd.py - removing ganglia stats

2017-10-12

23:58 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Restore thanks limit on pl.wikipedia (T169268) (duration: 00m 47s)
23:50 cwd: re-enabled omnimail jobs
23:35 eileen1: update from 2d6668b to b953064
23:25 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Add logging for email blocks (T175419) (duration: 00m 46s)
23:13 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch test wikis to HTML5 fragment mode in links (T152540) (duration: 00m 47s)
22:44 demon@tin: Synchronized php-1.31.0-wmf.3/includes/specialpage/ChangesListSpecialPage.php: silence notices (duration: 00m 47s)
21:36 cwd: disabled omnimail jobs
19:47 ejegg: updated CiviCRM from 47de976 to 2d6668b
19:46 chasemp: disable puppet for lab*services* for merge of 383896
19:18 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.3
18:28 mutante: added Pablo (pgrass) to LDAP group 'wmde' (T177599)
18:15 demon@tin: Synchronized php-1.31.0-wmf.3/extensions/ORES/includes/Hooks.php: Make watchlist queries suck less (duration: 00m 50s)
18:01 demon@tin: Synchronized php-1.31.0-wmf.3/extensions/JsonConfig/includes/JCCache.php: Silence junk warnings (duration: 00m 50s)
15:17 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
15:11 mutante: gerrit2001 - rebooting for kernel upgrade
14:57 mutante: netmon1002 - arming keyholder for rancid deploy
14:52 mutante: netmon1002 (librenms, rancid) - rebooting for kernel upgrade
14:47 moritzm: rebooting kubernetes workers to new 4.9.51 kernel
14:41 moritzm: rebooting kubernetes staging servers to new 4.9.51 kernel
14:29 moritzm: rebooting kubernetes masters to new 4.9.51 kernel
13:54 moritzm: updating image scalers to librsvg 2.40.18
13:48 elukey: deployed the new Analytics Public Druid cluster - T176223
13:45 moritzm: updating remaining thumbor servers to librsvg 2.40.18
13:40 ladsgroup@tin: Synchronized wmf-config/Wikibase.php: SWAT: Remove deprecated config variable for Wikibase (T129475) (duration: 01m 02s)
13:33 moritzm: updating thumbor1001 to librsvg 2.40.18
13:13 moritzm: rolling restart of logstash to pick up thumbor logs (T150734)
13:07 zeljkof: EU SWAT finished
12:56 moritzm: uploaded librsvg 2.40.18 for jessie-wikimedia to apt.wikimedia.org
11:46 ema: upgrading cp3010 to Varnish 5 T177233
11:27 Amir1: finished cleanup of ores_classification in wikidatawiki, still needs more work (T159753)
10:03 ema: lvs1005: upgrade pybal to 1.14.1 T177815
10:02 ema: pybal 1.14.1 uploaded to apt.w.o
09:40 ema: upgrading cp3008 to Varnish 5 T177233
09:06 moritzm: installing nettle update from stretch 9.2 point release
09:01 moritzm: installing ncurses security updates
08:25 moritzm: installing gnutls28 update from stretch 9.2 point release
08:02 Amir1: starting cleanup of ores_classification in wikidatawiki (T159753)
07:51 gehel: reinitialize cassandra on maps-test2004 to test vector tiles - T153282
07:40 moritzm: installing at-spi2-core update from stretch 9.2 point release
07:24 moritzm: installing gnupg2 update from stretch 9.2 point release
07:18 legoktm: purging html5-misnesting linter entries from database on terbium (T178040)
03:25 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 12 03:25:17 UTC 2017 (duration 7m 17s)
03:18 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 15m 57s)
02:41 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 15m 39s)
00:30 twentyafterfour: phabricator update finished (a while ago, I forgot to log it)
00:08 twentyafterfour: upgrading phabricator to #phab-2017-10-11
00:05 mutante: netmon2001 arm keyholder for rancid deploy
00:01 twentyafterfour: Preparing to take phabricator offline for a hopefully momentary upgrade.

2017-10-11

23:55 mutante: netmon2001 - smokeping back up - the reboot, as a side-effect, also resolved the systemd-icinga-alert that was pending in Icinga for a while with comments
23:52 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Vector print logo and print styles on all wikis T169732 PART II (duration: 00m 50s)
23:50 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Enable Vector print logo and print styles on all wikis T169732 PART I (duration: 00m 50s)
23:47 mutante: netmon2001 (smokeping) - kernel upgrade, schedule downtime, reboot
23:40 thcipriani@tin: Synchronized php-1.31.0-wmf.2/skins/Vector/ResourceLoaderLessModule.php: SWAT: Print logo should use an absolute URI T177800 (duration: 00m 49s)
23:38 mutante: netmon1003 - servermon back up - apt-get autoremove to drop unrequired python packages
23:37 thcipriani@tin: Synchronized php-1.31.0-wmf.3/skins/Vector/ResourceLoaderLessModule.php: SWAT: Print logo should use an absolute URI T177800 (duration: 00m 50s)
23:35 mutante: netmon1003 - cant seem to reboot normally, attempting via gnt-instance command
23:31 mutante: netmon1003 (servermon) - upgrading kernel (jessie), schedule icinga downtime, rebooting, short downtime for servermon itself
23:31 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Parser: Switch from Tidy to Remex on nowiki T177989 (duration: 00m 50s)
23:27 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Parser: Switch from Tidy to Remex on fawiki T176150 (duration: 00m 50s)
23:23 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/styles/ve.init.MWVESwitchConfirmDialog.css: SWAT: Fix WikiEditor mode switcher widget (duration: 00m 50s)
23:19 thcipriani@tin: Synchronized php-1.31.0-wmf.2/includes/specials/SpecialUnblock.php: SWAT: Fix unblocking autoblocks T177952 (duration: 00m 50s)
23:18 mutante: releases1001 upgrading kernel, installed linux-image-amd64, took out of cache::misc, upgrade libnss3, scheduled downtime, rebooting ...
23:17 thcipriani@tin: Synchronized php-1.31.0-wmf.3/includes/specials/SpecialUnblock.php: SWAT: Fix unblocking autoblocks T177952 (duration: 00m 51s)
22:01 volans: uploaded cumin_1.2.2-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
21:44 robh: cp4009 shutdown accidental, will boot back up immediately
21:13 mutante: releases2001 - scheduled downtime, rebooting
21:07 mutante: running puppet on cache misc hosts to temp remove codfw backend of releases.wm.org, then rebooting releases2001 for kernel upgrade
20:22 subbu: updated Parsoid to ddf7b293 (T138492, T135667, T177612)
20:18 ssastry@tin: Finished deploy [parsoid/deploy@938e305]: Updating Parsoid to ddf7b293 (duration: 08m 42s)
20:17 hashar@tin: Finished deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044 (duration: 02m 54s)
20:14 hashar@tin: Started deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:13 hashar@tin: Finished deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044 (duration: 00m 02s)
20:13 hashar@tin: Started deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:11 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:10 bsitzmann@tin: Finished deploy [mobileapps/deploy@15a52ba]: Update mobileapps to c045afc (T177301) (duration: 05m 25s)
20:09 ssastry@tin: Started deploy [parsoid/deploy@938e305]: Updating Parsoid to ddf7b293
20:07 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:05 bsitzmann@tin: Started deploy [mobileapps/deploy@15a52ba]: Update mobileapps to c045afc (T177301)
20:04 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:01 chasemp: disable puppet around cloud things to roll refactor out slowly
19:29 mutante: releases1001 - same as 2001, upgraded jenkins to 2.73.2, kept existing config (T177962)
19:27 mutante: releases2001 - upgraded jenkins to 2.73.2, kept existing config (vs overwriting with package config)
19:20 mutante: apt: reprepro copy stretch-wikimedia jessie-wikimedia jenkins
19:16 mutante: releases1001/releases2001 - apt-get autoremove to drop non-required python packages
19:03 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.3
19:02 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
18:47 gehel: killing stuck osm replication on maps-test2001
18:23 demon@tin: Synchronized wmf-config/Wikibase-labs.php: no-op, labs only (duration: 00m 59s)
18:19 demon@tin: Finished scap: wmf.3 bootstrap (duration: 44m 56s)
17:58 herron: ran /home/faidon/update-netboot-stretch.sh on puppetmaster1001
17:49 moritzm: uploaded jenkins 2.73.2 for jessie-wikimedia to apt.wikimedia.org
17:34 demon@tin: Started scap: wmf.3 bootstrap
17:32 demon@tin: Synchronized php: symlink swap to wmf.2, somehow got missed (duration: 00m 45s)
17:29 demon@tin: Synchronized scap/plugins/prep.py: No-op (duration: 01m 49s)
17:09 demon@tin: Pruned MediaWiki: 1.30.0-wmf.17 (duration: 02m 20s)
17:06 demon@tin: Pruned MediaWiki: 1.30.0-wmf.16 (duration: 02m 40s)
16:54 demon@tin: Pruned MediaWiki: 1.31.0-wmf.1 [keeping static files] (duration: 01m 44s)
16:24 hasharAway: Upgrade jenkins Maven integration plugin to 3.0 - T177962
16:15 godog: roll-restart thumbor to apply https://gerrit.wikimedia.org/r/380942 - T150734
16:12 hasharAway: Upgrading Jenkins CI T177962
16:09 gehel: decommission maps-test2004 from its cassandra cluster (free a node to test vector tiles) - T153282
16:09 XioNoX: resuming PDU upgrade
15:39 elukey: drop tables listed in https://phabricator.wikimedia.org/T171629#3674250 from db1046, db1047, dbstore1002
15:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1086 - T174509 (duration: 00m 47s)
15:21 _joe_: stopped redis on ocg* T177931
15:08 _joe_: ocg*: stop ocg; mv /srv/deployment /srv/stale, update-rc.d ocg disable, rm /etc/init/ocg.conf - T177931
14:55 _joe_: manually removing IPVS entries for ocg on eqiad LBs, T177931
14:47 _joe_: rolling restart of low-traffic pybals in eqiad for T177931
14:36 moritzm: installing dbus update from stretch 9.2 point release
13:40 hashar: European SWAT completed
13:39 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Remove compact language links dblist for simplicity (duration: 00m 46s)
13:38 hashar@tin: Synchronized docroot: Remove compact language links dblist for simplicity (duration: 00m 48s)
13:37 hashar@tin: Synchronized dblists: Remove compact language links dblist for simplicity (duration: 00m 47s)
13:36 hashar@tin: Synchronized wmf-config/CommonSettings.php: Remove compact language links dblist for simplicity (duration: 00m 47s)
13:29 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Configure wmgBabelMainCategory for the Dinka Wikipedia (duration: 00m 47s)
13:27 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable mapframe on eswiki - T177695 (duration: 00m 47s)
13:24 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Enable constraint checks on qualifiers and references - T176863 (duration: 00m 47s)
13:21 moritzm: installing db5.3 security updates
13:03 godog: test graphite-web patch on labmon1001 - T177747
12:54 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Disable Statement usage tracking on cawiki (T151717) (duration: 00m 47s)
12:53 marostegui: Start recentchanges purge on s4 primary master T177772
12:48 marostegui: Kill recentchanges purge on s4 primary master - https://phabricator.wikimedia.org/T177772
12:44 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: (temp) Disable Statement usage tracking on cawiki (T151717) (duration: 00m 48s)
12:38 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Statement usage tracking on cawiki and cewiki (T151717) (duration: 00m 47s)
12:32 hoo@tin: Synchronized wmf-config/: Move WB client "disabledUsageAspects" setting into $wmgWikibaseDisabledUsageAspects (duration: 00m 48s)
12:30 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Introduce $wmgWikibaseDisabledUsageAspects (duration: 00m 47s)
12:27 hoo@tin: scap aborted: wmf-config/InitialiseSettings.php Introduce $wmgWikibaseDisabledUsageAspects (duration: 02m 34s)
12:25 hoo@tin: Started scap: wmf-config/InitialiseSettings.php Introduce $wmgWikibaseDisabledUsageAspects
11:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 - T174509 (duration: 00m 50s)
11:49 ema: upgrading cp3007 to Varnish 5 T177233
11:34 moritzm: installing ruby1.9 security updates on trusty hosts
11:19 akosiaris: re-enable OTRS stats group T176221
11:11 jynus: starting purge of ruwiki.recentchanges T177772
10:55 akosiaris: OTRS upgrade done T176221
10:44 akosiaris: starting OTRS upgrade T176221
10:24 jynus: stopping dbstore2001:s3 for maintenance T177908
10:19 moritzm: installing curl security updates
09:32 moritzm: installing libxfont security updates
09:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T174509 (duration: 00m 46s)
09:27 marostegui: Optimize pagelinks and templatelinks on db1087 - T174509
09:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 - T174509 (duration: 00m 47s)
09:04 moritzm: installing curl security updates on app server canaries (along with HHVM restart)
08:50 moritzm: installing ffmpeg security updates
08:38 elukey: reboot kafka-jumbo hosts for kernel updates
08:37 marostegui: Stop replication in sync on db1103 and db1038 to checksum their data - T164488
08:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103 - T164488 (duration: 00m 47s)
07:46 jynus: restart commonswiki recentchanges purging T177772
07:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1072 into the vslow and dump service for s3 - T172679 (duration: 00m 46s)
07:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T174509 (duration: 00m 47s)
07:02 marostegui: Optimize templatelinks and pagelinks on db1086 - T174509
06:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1082 - T174509 (duration: 00m 47s)
06:57 marostegui: Optimize templatelinks and pagelinks on db1082 - T174509
06:14 marostegui: Set db1068 s4 primary master to read-only OFF - T168661
06:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Set s4 to writable mode after maintenance - T168661 (duration: 00m 47s)
06:07 marostegui: Deploy alter table on s4 primary master - T168661
06:01 marostegui: Set read-only on db1068 s4 primary master - T168661
06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Set s4 in read-only for maintenance - T168661 (duration: 00m 47s)
05:48 marostegui: Package update on db1068 s4 primary master - T168661
05:35 marostegui: Disable puppet on db1068 (s4 primary master) - T168661
05:35 marostegui: Disable puppet on db1068 (s4 primary master)
05:13 marostegui: Disable alerts on s4 for 2 hours - T168661
05:06 marostegui: Dump buffer pool on s4 primary master db1068 - T168661
05:03 kartik@tin: Finished deploy [cxserver/deploy@a79b38c]: Update cxserver to 273b515 (duration: 02m 57s)
05:00 kartik@tin: Started deploy [cxserver/deploy@a79b38c]: Update cxserver to 273b515
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 11 02:32:46 UTC 2017 (duration 6m 39s)
02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 08m 52s)

2017-10-10

23:53 thcipriani@tin: Synchronized wmf-config: SWAT: Disable OCG services PART II T177795 (duration: 00m 48s)
23:52 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disable OCG services PART I T177795 (duration: 00m 47s)
23:41 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Collection/RenderingAPI.php: SWAT: Do not request render if renderer not configured T177795 (duration: 00m 47s)
23:39 XioNoX: upgrading ps1-a2-eqiad - T175341
23:35 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: revert Enable Vector print logo on test wiki (duration: 00m 47s)
23:21 thcipriani@tin: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
23:12 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: wikitech: Align "contentadmin" and "sysop" permissions T171208 (duration: 00m 48s)
23:08 eileen1: ran the group clean up script (reduced smart groups)
22:28 eileen1: changed civicrm smart group cache timeout to 5 civicrm/admin/setting/search?reset=1
21:15 bblack: esams repooling - T167840
19:30 krinkle@tin: Synchronized docroot/noc/: Clean up noc base.css (duration: 00m 48s)
19:07 krinkle@tin: Synchronized docroot/noc/: Clean up noc/index CSS (duration: 00m 47s)
19:00 XioNoX: starting the work for T167840
18:41 mutante: logstash*: delete /usr/lib/ganglia/python_modules/elasticsearch_monitoring.py and /etc/ganglia/conf.d/elasticsearch.pyconf via cumin (T177225)
17:32 XioNoX: depooling esams from DNS for T167840
17:26 gehel: shutting down and restarting elasticsearch on relforge1001 for testing - T170378
17:25 krinkle@tin: Synchronized php-1.31.0-wmf.2/resources/lib/jquery.ui/: I717f2580e3aae (duration: 00m 47s)
16:19 milimetric@tin: Finished deploy [analytics/refinery@f4a0a33]: Deploying mostly for the new script in the bin folder (duration: 16m 01s)
16:03 milimetric@tin: Started deploy [analytics/refinery@f4a0a33]: Deploying mostly for the new script in the bin folder
15:48 jynus: starting purge of commonswiki.recentchanges T177772
15:39 thcipriani@tin: Synchronized README: noop deployment to test new logstash-checker.py host (duration: 00m 47s)
15:27 jynus: upgrading neodymium and sarin to mariadb-client 10.1.28
15:23 ema: cp1008 upgraded to varnish 5 T168529
15:00 godog: test stretch 9.2 upgrade on ms-fe2005 - T177739
14:59 cmjohnson1: lvs1007 swapping NIC card
14:53 godog: test stretch 9.2 upgrade on ms-be2013 - T177739
14:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T174509 (duration: 00m 47s)
14:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1080 - T174509 (duration: 00m 47s)
14:31 marostegui: Optimize table ores_classifications on db1080 - T159753
14:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 with low weight - T172679 (duration: 00m 47s)
14:24 aude@tin: Synchronized wmf-config/throttle.php: Add throttle rule T177835 (duration: 00m 47s)
14:22 elukey: add druid public cluster's IPs to analytics-in4 on cr1/cr2 - T177511
13:56 aude@tin: Synchronized wmf-config/throttle.php: Add throttle rule T177835 (duration: 02m 16s)
13:37 aude@tin: Synchronized wmf-config/Wikibase-production.php: Remove test wiki echo configs from Wikibase config (duration: 01m 31s)
13:20 aude@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on gawiki (duration: 01m 50s)
13:11 gehel: restarting rsyslog on mw1180 - T177833
11:32 moritzm: downgrading HHVM on deployment-mediawiki04 to HHVM 3.18.2 temporarily (to further narrow down a problem with the new wikidiff package)
11:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 - T174509 (duration: 05m 02s)
11:31 dcausse: upgrading relforge to elastic 5.5.2
11:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1071 - T174509 (duration: 01m 21s)
10:33 akosiaris: T177511 switch druid100[456] to private1-x-eqiad VLANs
10:26 Amir1: start of cleaning up ores_classification in wikidatawiki (T159753)
08:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2080 to the config - T170662 (duration: 00m 46s)
08:06 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2080 to the config - T170662 (duration: 00m 47s)
07:04 marostegui: Stop MySQL on db2079 to clone db2080 - https://phabricator.wikimedia.org/T170662
06:35 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s3 - T153033
06:27 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s1 - T153033
06:22 marostegui: Stop MySQL on db1038 to transfer its data to db1072 - T172679
06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1038 - T172679 (duration: 00m 47s)
06:05 marostegui: Stop MySQL on db1072 to move it to s3 - T172679
05:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T174509 (duration: 00m 47s)
05:53 marostegui: Optimize templatelinks and pagelinks on db1079 - T174509
05:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1071 - T174509 (duration: 00m 48s)
05:45 marostegui: Optimize templatelinks and pagelinks on db1071 - T174509
05:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174509 (duration: 00m 47s)
05:38 marostegui: Optimize templatelinks and pagelinks on db1076 - T174509
05:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1080 - T174509 (duration: 00m 48s)
05:32 marostegui: Optimize templatelinks and pagelinks on db1080 - T174509
02:35 eileen2: update civicrm from 00da4b0 to 47de976
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 10 02:32:57 UTC 2017 (duration 6m 41s)
02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 08m 49s)
01:21 eileen2: update civicrm from 276cbfd to 00da4b0

2017-10-09

22:18 hoo: Removed empty left-over xmldatadumps/public/other/wikibase/wikidatawiki/20171007 (https://dumps.wikimedia.org/wikidatawiki/entities/20171007/)
19:53 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable new print styles on Vector in test wiki - T169732 (duration: 00m 47s)
19:51 hashar: morning swat completed
19:50 hashar@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo/maintenance/updatePerUserBlacklist.php: Sync a live hack in Echo updatePerUserBlacklist https://gerrit.wikimedia.org/r/#/c/383182/ - T173475 (duration: 00m 47s)
19:48 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable new print styles on Vector in test wiki - T169732 (duration: 00m 47s)
19:22 hashar@tin: Synchronized wmf-config/CommonSettings.php: REVERT: Enable new print styles on Vector - T169732 (duration: 00m 49s)
19:19 hashar@tin: scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
19:17 hashar@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: Reapply "Use User Ids instead of User Names for Echo Mute"" - T173475 (duration: 00m 52s)
18:57 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage for SUL accounts too on hi.wikiversity - T177690 (duration: 00m 47s)
18:52 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage for SUL accounts too on dty.wiki - T177688 (duration: 00m 47s)
18:45 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add autopatrolled user group to sd.wikipedia - T177141 (duration: 00m 47s)
18:00 ottomata: stopping druid services on druid1004 (not 1006)
17:59 ottomata: stopping druid services on druid1006
17:28 ottomata: stopping druid services on druid1005
16:34 ottomata: stopping druid services on druid1006
16:34 ottomata: beginning decom and reinstall process for druid1004-1006 -- T176223
15:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 - T174509 (duration: 00m 47s)
13:53 zeljkof: EU SWAT finished
13:53 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/: SWAT: Revert "[tech-debt] Remove usage of FuzzyLikeThis in favor of simple fuzzy match" (T177727) (duration: 00m 57s)
13:27 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disallow most file types from upload to enwikivoyage (T176647) (duration: 00m 47s)
13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disallow most file types from upload to enwikivoyage (T176647) (duration: 00m 47s)
13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix amwikimedia site name (T176042) (duration: 00m 47s)
13:10 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: New throttle rule (T177737) (duration: 00m 47s)
12:47 ema: restart pybal on eqiad load balancers to pick up bgp-med config change T165584
12:38 ema: restart pybal on codfw load balancers to pick up bgp-med config change T165584
12:31 ema: restart pybal on ulsfo load balancers to pick up bgp-med config change T165584
12:26 ema: restart pybal on esams load balancers to pick up bgp-med config change T165584
11:59 joal@tin: Finished deploy [analytics/refinery@c3812c2]: Analytics regular weekly deploy (duration: 10m 24s)
11:49 joal@tin: Started deploy [analytics/refinery@c3812c2]: Analytics regular weekly deploy
11:42 hashar: Upgrading Jenkins - T168644
11:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174509 (duration: 00m 46s)
11:22 moritzm: installing apt updates from stretch 9.2 point release
11:22 marostegui: Optimize ores_classification table on db1083 - T159753
11:19 reedy@tin: Synchronized wmf-config/Wikibase-production.php: Actually disable injecting RC everywhere but commonswiki and ruwiki (duration: 00m 46s)
11:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 - T174509 (duration: 00m 47s)
11:08 moritzm: installing whois updates from stretch 9.2 point release
11:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1090 - T174509 (duration: 00m 46s)
10:54 moritzm: installing expect updates from stretch 9.2 point release
10:50 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Disable injecing RC records for commonswiki and ruwiki (T171027) (duration: 00m 47s)
10:38 ladsgroup@tin: Synchronized php-1.31.0-wmf.2/extensions/Wikidata/extensions/Wikibase/client: Re-instate $wgWBClientSettings[injectRecentChanges] (T171027) (duration: 00m 56s)
10:35 hoo: Started dumpwikidatajson.sh on snapshot1007 after T169680 disruptions
10:30 volans: rebooting snapshot1007 for NFS stuck - T169680
10:21 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s7 - T153033
10:21 volans: rebooting snapshot1006 for NFS stuck - T169680
10:15 elukey: rebooting snapshot1001 for NFS stuck - T169680
10:00 volans: rebooting snapshot1005 for NFS stuck - T169680
09:51 volans: rebooting dataset1001 for NFS stuck - T169680
09:51 ema: test setting bgp med on lvs3001/3003 T165584
09:50 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s5 - T153033
09:43 hashar: restarting Jenkins. Deadlock in SSHSlave plugin that causes memory to leak quite rapidly - T177749
09:34 moritzm: installing vim security updates
09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2075 - T170662 (duration: 00m 47s)
08:56 volans: disabled puppet on 'stat100[5-6]*,snapshot100[1,5-7]*,dataset1001*' to manually recover from nfsd stuck - T169680
08:52 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s4 - T153033
08:51 marostegui: Optimize pagelinks and templatelinks on db1092 - T174509
08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 - T174509 (duration: 00m 47s)
08:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096 - T174509 (duration: 00m 47s)
08:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1066 on s1 to and move db1072 to s3 instead - T172679 (duration: 00m 47s)
08:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1066 from s1 to s3 - T172679 (duration: 01m 25s)
07:33 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s2 - T153033
07:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2079 to the config - T170662 (duration: 00m 47s)
07:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2079 to the config - T170662 (duration: 00m 47s)
07:01 ema: cp4022 - backend restart, mailbox lag, cache_upload
06:56 marostegui: Stop MySQL on db2075 to use it to clone db2079 - T170662
06:56 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 - T170662 (duration: 00m 46s)
06:25 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s6 - T153033
06:14 marostegui: Optimize pagelinks and templatelinks on db1039 - T174509
06:11 marostegui: Optimize pagelinks and templatelinks on db1050 - T174509
06:08 marostegui: Optimize pagelinks and templatelinks on db1096 - T174509
06:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T174509 (duration: 00m 47s)
05:59 marostegui: Optimize pagelinks and templatelinks on db1091 - T174509
05:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T174509 (duration: 00m 47s)
05:53 marostegui: Optimize pagelinks and templatelinks on db1090 - T174509
05:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174509 (duration: 00m 47s)
05:44 marostegui: Optimize pagelinks and templatelinks on db1083 - T174509
05:38 marostegui: Stop MySQL on db1083 to upgrade it
05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174509 (duration: 00m 47s)
02:48 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 9 02:48:21 UTC 2017 (duration 6m 54s)
02:41 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 13m 49s)

2017-10-08

21:40 hoo: Killed "dumpwikidatardf.sh truthy nt" (wikidata truthy dump) on snapshot1007, got stuck after T169680.
11:28 volans: ack-ed puppet not running on stat100[5-6],snapshot100[1,5-7] due to NFSD stuck on dataset1001 - T169680
08:19 elukey: restart varnish backend on cp4026 to stop 503s

2017-10-07

22:46 reedy@tin: Synchronized php-1.31.0-wmf.2/resources/: T177705 (duration: 00m 48s)
22:44 reedy@tin: Synchronized php-1.31.0-wmf.2/includes/specials/SpecialBlock.php: T177705 (duration: 00m 49s)

2017-10-06

18:58 ejegg: updated CiviCRM from 197b22d to 276cbfd
18:38 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/: Unbreak preference changes (T177524) (duration: 00m 47s)
18:23 arlolra@tin: Finished deploy [parsoid/deploy@e437c93]: Updating Parsoid to 772e11bf (duration: 12m 41s)
18:10 arlolra@tin: Started deploy [parsoid/deploy@e437c93]: Updating Parsoid to 772e11bf
15:30 moritzm: repooling mw2145 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it
15:25 moritzm: repooling mw2203 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it
12:54 moritzm: repooling mw2248-mw2250 (job runners). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool
11:46 bblack: cp1* (eqiad caches) - upgrade to nginx-1.13.5-1+wmf1~jessie1 to match all the other sites
11:34 moritzm: repooling mw2244/mw2245 (image scalers). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool
11:21 moritzm: installing gdk-pixbuf security updates
11:10 elukey: rolling restart of all the druid daemons on druid100[1-6] to pick up new logging changes
11:03 moritzm: installing git security updates on trusty (Debian already fixed)
10:51 moritzm: upgrading HHVM on script runners to HHVM 3.18.5
10:47 moritzm: upgrade remaining video scalers in codfw to HHVM 3.18.5
10:43 elukey: restart replication (mysql+ eventlogging_sync) on dbstore1002 after mysql restart
10:17 moritzm: upgrade remaining job runners in codfw to HHVM 3.18.5
09:53 elukey: stop replication (eventlogging_sync + mysql) on dbstore1002 as prep step for mysql restart
09:17 hashar: Restarted the CI Jenkins for plugin upgrades
09:01 moritzm: upgrade remaining app servers in codfw to HHVM 3.18.5
08:40 moritzm: upgrade remaining image scalers in codfw to HHVM 3.18.5
08:09 ema: upgrade pybal to 1.14.0 on eqiad primary LVSs
07:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1076 original weight - T174054 (duration: 00m 48s)
07:23 ema: upgrade pybal to 1.14.0 on eqiad secondary LVSs
07:18 moritzm: upgrade remaining API servers in codfw to HHVM 3.18.5
07:11 legoktm: live hacking on mwdebug1002
07:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s)
06:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s)
06:35 ema: set max_connections=0 on cp3032 varnish-be
06:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1106 to the config - T172679 (duration: 00m 47s)
06:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1106 to the config - T172679 (duration: 00m 47s)
06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight - T174054 (duration: 00m 46s)
05:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 - T174509 (duration: 00m 48s)
05:35 marostegui: Stop MySQL on db1076 for an upgrade

2017-10-05

23:40 ejegg: updated CiviCRM from cca1a9d to 197b22d
23:37 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/resources/lib/jquery/jquery.migrate.js: sync https://gerrit.wikimedia.org/r/#/c/382530 and https://gerrit.wikimedia.org/r/#/c/382529 (duration: 00m 47s)
23:34 XioNoX: upgrading ps1-d2-eqiad.mgmt.eqiad.wmnet (unused)
23:23 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/BaseInterwikiResolver.php: sync https://gerrit.wikimedia.org/r/#/c/382625/ (duration: 00m 47s)
23:21 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT https://gerrit.wikimedia.org/r/#/c/382611/ (duration: 00m 47s)
22:47 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/: Sync CirrusSearch extension again for good measure (duration: 00m 51s)
22:43 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: Sync CirrusSearch extension again for good measure (duration: 00m 57s)
22:39 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-common.php: sync https://gerrit.wikimedia.org/r/#/c/382474/ (duration: 00m 47s)
22:37 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/: deploy https://gerrit.wikimedia.org/r/#/c/382618/ (duration: 00m 58s)
22:13 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 19s)
22:13 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart
22:04 bd808@tin: Finished deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681) (duration: 00m 33s)
22:04 bd808@tin: Started deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681)
21:35 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358
21:32 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: sync CirrusSearch to deploy https://gerrit.wikimedia.org/r/#/c/382605/ refs T177535 (duration: 01m 06s)
21:22 ejegg: updated CiviCRM from 59e3e00 to cca1a9d
21:13 twentyafterfour: Getting the train back on track: moving to wmf.2 as soon as https://gerrit.wikimedia.org/r/#/c/382605/1 merges
20:47 twentyafterfour: 1.31.0-wmf.2 is now blocked by T177535
20:28 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.1 refs T174358
20:26 twentyafterfour: rolling back to 1.31.0-wmf.1
20:11 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358
20:10 twentyafterfour: Promote all wikis to 1.31.0-wmf.2 refs T174358 (currently there are no blockers and no significant logspam)
19:59 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/Echo: SWAT: revert "Use User Ids instead of User Names for Echo Mute" (duration: 00m 55s)
19:58 andrewbogott: upgrading wikitech-static to REL1_30
19:55 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: SWAT: revert "Use User Ids instead of User Names for Echo Mute" (duration: 00m 52s)
19:49 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/VisualEditor/extension.json: SWAT: Move 'parentheses' message to MWSaveDialog's modules T177446 (duration: 00m 50s)
19:45 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict: SWAT: Dont register simulate page when not enabled as a BF (duration: 00m 52s)
19:05 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/Echo: SWAT: Use User Ids instead of User Names for Echo Mute T173475 (duration: 00m 55s)
18:58 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: SWAT: Use User Ids instead of User Names for Echo Mute T173475 (duration: 00m 56s)
18:53 bblack: cp3* upgrade nginx to 1.13.5-1+wmf1~jessie1
18:38 urandom: restbase-ng: lowering compactionthroughput to 4 (current/5 jbod devices)
18:26 thcipriani@tin: Synchronized wmf-config: SWAT: Enable structured change filters by default on all wikis except FlaggedRevs wikis using non-protection modes T177445 (duration: 00m 54s)
18:13 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable jQuery 3 on all wikis T124742 (duration: 00m 52s)
17:27 cmjohnson1: disable puppet db1024
16:53 moritzm: upgrade remaining app servers in eqiad to HHVM 3.18.5
16:16 XioNoX: failure test for pfw3-codfw
15:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T174054 (duration: 00m 50s)
15:25 marostegui: Pulling out one disk from db1078 - T174054
15:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T174054 (duration: 00m 51s)
15:16 marostegui: Pulling out one disk from db1076 - T174054
14:53 hashar: All Jenkins infra is now on Java 8 - T162828
14:52 marostegui: Sanitize db1095 for amwikimedia - T176043
14:48 hashar: Restarting Jenkins
14:27 moritzm: upgrade remaining eqiad video scalers to HHVM 3.18.5
13:57 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Enable Statement usage tracking on kowiki and trwiki (T151717) (duration: 00m 50s)
13:51 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: (no justification provided) (duration: 00m 50s)
13:49 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Document task reference for amwikimedia logo (T176042) (duration: 00m 54s)
13:43 Amir1: ladsgroup@terbium:~$ mwscript createAndPromote.php --wiki=amwikimedia --createAndPromote=sysop,bureaucrat 'Ladsgroup' "password removed obviously" --force (T176042)
13:35 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Fix amwikimedia logo (T176042) (duration: 00m 50s)
13:26 ladsgroup@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaMessages/i18n/wikimediaprojectnames: (no justification provided) (duration: 00m 49s)
13:15 godog: test upgrade node-exporter in thumbor/swift/services/puppet3-diffs/monitoring projects - T166561
13:11 ladsgroup@tin: Synchronized wmf-config/interwiki.php: Create amwikimedia (T176042) (duration: 00m 51s)
13:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174509 (duration: 00m 50s)
12:58 ladsgroup@tin: Synchronized multiversion/MWMultiVersion.php: Create amwikimedia (T176042) (duration: 00m 50s)
12:53 ladsgroup@tin: Synchronized static/images/project-logos/: Create amwikimedia (T176042) (duration: 00m 50s)
12:52 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Create amwikimedia (T176042) (duration: 00m 50s)
12:50 ladsgroup@tin: rebuilt wikiversions.php and synchronized wikiversions files: Create amwikimedia (T176042)
12:49 ladsgroup@tin: Synchronized dblists: Create amwikimedia (T176042) (duration: 00m 50s)
12:36 ladsgroup@tin: Synchronized dblists: (no justification provided) (duration: 00m 51s)
12:22 moritzm: installing poppler security updates on trusty
11:57 moritzm: upgrade remaining eqiad video scalers to HHVM 3.18.5
11:29 jynus: rebuilding globalimagelinks table on dbstore1001:s4
10:35 hashar: contint2001: switched java to version 8 as well as all the jre/jdk utilities managed by alternatives - T162828
10:26 moritzm: upgrade remaining eqiad API servers to HHVM 3.18.5
10:23 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/TwoColConflictHooks.php: REVERT: Dont register simulate page when not enabled as a BF PT1/3 (duration: 00m 50s)
10:22 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/extension.json: REVERT: Dont register simulate page when not enabled as a BF PT2/3 (duration: 00m 50s)
10:17 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/extension.json: Dont register simulate page when not enabled as a BF PT2/3 (duration: 00m 50s)
10:16 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/TwoColConflictHooks.php: Dont register simulate page when not enabled as a BF PT1/3 (duration: 00m 51s)
10:14 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/SpecialConflictTestPage/SpecialConflictTestPage.php: Fix SimulateTwoColEditConflict when using a page with ns (duration: 00m 50s)
10:09 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/WikimediaEventsHooks.php: 31-wmf.2 WikimediaEvents: WMDE: update $campaignStartTimestamp (duration: 00m 50s)
10:07 addshore@tin: Synchronized php-1.31.0-wmf.1/extensions/WikimediaEvents/WikimediaEventsHooks.php: 31-wmf.1 WikimediaEvents: WMDE: update $campaignStartTimestamp (duration: 00m 52s)
10:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174054 (duration: 00m 50s)
09:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099 - T174509 (duration: 00m 50s)
08:54 marostegui: Stop MySQL on db1104 to clone db1106 - T172679
08:53 gehel: rolling restart of wdqs to pick up new GC options - T175919
08:41 moritzm: upgrade remaining eqiad job runners to HHVM 3.18.5
08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1105 to the config - T172679 (duration: 00m 50s)
08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1105 to the config - T172679 (duration: 00m 52s)
08:29 marostegui: Optimize pagelinks and templatelinks on s5 - db1105 - T174509
08:21 moritzm: upgrade remaining eqiad image scalers to HHVM 3.18.5
08:12 marostegui: Optimize templatelinks and pagelinks on s1 s2 s4 s5 s6 and s7 on labsdb1009 - T174509
08:03 jynus: putting labsdb1009 under maintenance
07:54 moritzm: upgrade mw1221-mw1235 (API servers) to HHVM 3.18.5
07:26 legoktm: running batched query of "DELETE FROM linter WHERE linter_cat=12" on all wikis to clear out mostly bogus html5-misnesting category
07:11 ema: update pybal to 1.14.0 on esams primary LVSs
07:06 ema: update pybal to 1.14.0 on esams secondary LVSs
07:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099 - T174509 (duration: 00m 52s)
07:01 marostegui: Optimize templatelinks and pagelinks on db1099 - T174509
06:56 moritzm: upgrade mw1238-mw1258 (app servers) to HHVM 3.18.5
06:40 marostegui: Stop MySQL on db1104 to clone db1105 from it - T172679
06:32 marostegui: Optimize templatelinks and pagelinks on db1069 - T174509
06:31 marostegui: Optimize templatelinks and pagelinks on db1053 - T174509
06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T174509 (duration: 00m 50s)
06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174509 (duration: 00m 49s)
06:24 marostegui: Optimize templatelinks and pagelinks on db1089 - T174509
06:17 marostegui: Optimize s2.templatelinks and enwiki.pagelinks on codfw master db2017 (without replication) - T174509
06:13 marostegui: Optimize enwiki.templatelinks and enwiki.pagelinks on codfw master db2048 (without replication) - T174509
06:01 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2035 and db2056 - T174509 (duration: 00m 52s)
05:57 elukey: restart varnish backend on cp3040
05:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097 - T174509 (duration: 01m 09s)
05:42 elukey: restart varnish backend on cp3030
02:59 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 5 02:59:18 UTC 2017 (duration 7m 23s)
02:51 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 09m 01s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 09m 51s)
00:06 catrope@tin: Synchronized php-1.31.0-wmf.2/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: T176652 (duration: 00m 49s)
00:04 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T176652 (duration: 00m 50s)

2017-10-04

23:55 eileen: update process_control to 10161f8
23:53 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/jquery.embedPlayer.js: Fix jqmigrate warning (T169385) (duration: 00m 50s)
23:51 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/ORES/includes/Hooks.php: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
23:49 catrope@tin: Synchronized php-1.31.0-wmf.2/resources/src/: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
23:36 catrope@tin: Synchronized php-1.31.0-wmf.2/includes/changes/ChangesListFilter.php: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
23:32 catrope@tin: Synchronized php-1.31.0-wmf.2/includes/changes/ChangesListFilterGroup.php: Fix PHP fatal when transcluding Special:Recentchanges (T176236) (duration: 00m 50s)
23:32 Krinkle: Upgrading xhgui on tungsten (operations/software/xhgui.git) from 0.5.2 to 0.8.1 (matching version used in operations/mediawiki-config vendor) - new feature: flamegraphs
23:29 catrope@tin: Synchronized wmf-config/throttle.php: Remove expired rules (duration: 00m 50s)
23:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile on ptwiki (T177336) (duration: 00m 51s)
23:19 catrope@tin: Synchronized static/images/mobile/copyright/wikipedia-wordmark-en.svg: Remove unnecessary id attributes (T175670) (duration: 00m 50s)
23:18 catrope@tin: Synchronized php-1.31.0-wmf.2/skins/MinervaNeue/: Correct feature phone threshold detection (T176286) (duration: 00m 53s)
22:35 ejegg: changed DonationInterface cURL timeout for all processors to 12 sec
21:09 arlolra: Updated Parsoid to 477942c3 (T177115)
21:07 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
21:03 arlolra@tin: Finished deploy [parsoid/deploy@617cdb3]: Updating Parsoid to 477942c3 (duration: 10m 01s)
20:53 arlolra@tin: Started deploy [parsoid/deploy@617cdb3]: Updating Parsoid to 477942c3
20:33 arlolra@tin: Finished deploy [parsoid/deploy@5b80254]: Updating Parsoid to 477942c3 (duration: 00m 58s)
20:32 arlolra@tin: Started deploy [parsoid/deploy@5b80254]: Updating Parsoid to 477942c3
20:18 pnorman: Start regenerating map tiles on codfw for z13-z14 - T176252
19:55 twentyafterfour@tin: Synchronized php: group1 wikis to 1.31.0-wmf.2 refs T174358 (duration: 00m 48s)
19:55 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.2 refs T174358
19:38 pnorman: Start regenerating map tiles on eqiad for z13-z14 - T176252
19:25 twentyafterfour: Deploying MediaWiki 1.31.0-wmf.2 to Group 1 wikis refs T174358
19:06 bblack: cp2* (all codfw caches): upgrade nginx to 1.13.5-1+wmf1~jessie1
18:39 ottomata: reverting roll out of LVS service druid-analytics.svc.eqiad.wmnet - this wont' work with hosts inside of Analytics VLAN
18:23 niharika29@tin: Synchronized php-1.31.0-wmf.2/skins/Vector/: Do not special-case ULS and Not logged in in RTL in personal bar T48947, T177312 (duration: 00m 50s)
18:22 niharika29@tin: Synchronized php-1.31.0-wmf.1/skins/Vector/: Do not special-case ULS and Not logged in in RTL in personal bar T48947, T177312 (duration: 00m 51s)
18:09 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on most group1 wikis (T124742) (duration: 00m 51s)
17:37 elukey: enabled basic ACLs on the Kafka Jumbo cluster - T173493
17:32 bblack: text@ulsfo - upgrade nginx to 1.13.5-1+wmf1~jessie1
17:32 ottomata: deploying new LVS service for druid-analytics-broker
17:16 moritzm: uploaded icu52 to stretch-wikimedia (co-installable forward port of jessie's ICU for stretch to be used for HHVM)
17:06 bblack: cp4022-26 (now all of upload@ulsfo) - upgrade nginx to 1.13.5-1+wmf1~jessie1
16:58 krinkle@tin: Synchronized php-1.31.0-wmf.2/extensions/TimedMediaHandler/resources/mw.MediaWikiPlayer.loader.js: T175506 (duration: 00m 52s)
16:29 bblack: lvs1001-3: base package upgrades, autoremove, +python-requests
16:24 bblack: recdns: upgrade base packages
16:23 volans: deleted 'R:File = /usr/local/bin/wmf-reimage' on matching hosts T176955
16:20 bblack: maelant: upgrade base packages
16:11 bblack: lvs1004-6: base package upgrades, autoremove, +python-requests
16:06 jynus: performing schema change on labsdb1009/10/11
16:01 ejegg: disabled CiviCRM dedupe jobs
15:40 bblack: acamar (recdns@codfw) - base package upgrades
15:39 bblack: lvs1007-12: base package upgrades, autoremove, +python-requests
15:37 bblack: hydrogen (recdns@eqiad) - base package upgrades
15:35 marostegui: Power off db2010 to decommission it - T175685
15:35 bblack: recdns: apt autoremove -> install python-requests
15:32 bblack: caches: apt autoremove -> install python-requests
15:28 marostegui: Disable puppet on db2010 - it will be decommissioned - T175685
15:24 andrewbogott: rebuilding labvirt1016
14:57 ema: cp4025: restart varnish backend
14:47 XioNoX: replacing faulty VC cable on fasw-c-eqiad
14:45 bblack: base package upgrades (+autoremove) on lvs3001-2
14:43 bblack: base package upgrades (+autoremove) on lvs3004
14:41 bblack: base package upgrades (+autoremove) on lvs3003
14:24 bblack: upgrading nginx to 1.13.5-1+wmf1~jessie1 on cp4021 (first live cache, upload@ulsfo)
14:18 bblack: base package upgrades (+autoremove) on lvs2*
14:10 bblack: base package upgrades (+autoremove) on lvs4*
13:47 zeljkof: EU SWAT finished
13:46 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/tag/PageTranslationHooks.php: SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 50s)
13:45 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/TranslateHooks.php: SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 50s)
13:41 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/TranslateHooks.php: php-1.31.0-wmf.2/extensions/Translate/tag/PageTranslationHooks.php SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 52s)
13:26 zeljkof: one more commit for EU SWAT
13:26 marostegui: Optimize table ores_classification on enwiki codfw master db2048 with replication - might generate lag - T159753
13:22 marostegui: Optimize enwiki.ores_classification on db2069
13:06 zeljkof: EU SWAT finished
13:05 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: New throttle rule (T177370) (duration: 00m 53s)
12:59 jynus: upgrade and reboot labsdb1009
12:56 marostegui: Optimize templatelinks and pagelinks tables on s2 and s7 - labsdb1010 - T174509
12:38 moritzm: upgrading app servers in codfw to HHVM 3.18.5
12:21 moritzm: upgrading image scalers mw1293-mw1295 to HHVM 3.18.5
12:12 moritzm: upgrading HHVM on deployment servers to 3.18.5
12:10 elukey: added two new mediawiki videoscalers - mw1307/1318
12:09 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1318.eqiad.wmnet
12:09 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1307.eqiad.wmnet
12:00 gehel: mediawiki now uses the LVS endpoint for logstash - T175242
11:49 moritzm: upgrading job runners mw1161-mw1167 to HHVM 3.18.5
11:34 moritzm: installing ghostscript security updates
11:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1087 original weight (duration: 00m 47s)
10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1087 weight (duration: 00m 51s)
10:29 jynus: upgrade and reboot labsdb1011
10:22 jynus: draining labsdb1011 connections
10:18 akosiaris: T177374, fully depool wtp1001-wtp1024
10:18 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet
10:03 moritzm: install libidn security updates
10:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 with low weight (duration: 00m 50s)
09:54 _joe_: restarting hhvm on canaries (both appservers and canaries) T107128
09:47 moritzm: upgrading API app servers mw1189-mw1208 to HHVM 3.18.5
09:45 elukey: rolling restart of aqs nodes to pick up the new logstash lvs config
09:39 hoo: Killed remaining dumpRdf runners on snapshot1007 after two crashed due to the db1087 depool. Auto-restart will do the rest.
09:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1104 to the config - T172679 (duration: 00m 50s)
09:32 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1104 to the config - T172679 (duration: 00m 51s)
09:25 moritzm: installing ocaml security updates on trusty
09:16 marostegui: Reboot db1087 to pick up new kernel
09:09 _joe_: restarting hhvm on mwdebug*, T107128
09:07 marostegui: Optimize templatelinks and pagelinks tables on db1097 (s4) - T174509
09:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T174509 (duration: 00m 50s)
08:41 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet
08:40 mobrovac@tin: Finished deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3 (duration: 10m 05s)
08:36 moritzm: upgrading app servers mw1180-mw1188, mw1209-mw1220 to HHVM 3.18.5
08:30 mobrovac@tin: Started deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3
08:28 marostegui: Stop MySQL on db2010 as it will be decommissioned - T175685
07:30 jynus: stopping s5 replication on dbstore1002 and converting wikidatawiki.wb_terms into TokuDB
07:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s)
07:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s)
06:53 marostegui: Stop MySQL on db1087 to clone db1104 from it - T172679
06:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T172679 (duration: 00m 50s)
06:30 marostegui: Optimize templatelinks and pagelinks tables on db1066 - T174509
06:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2035 and db2056 - T174509 (duration: 00m 50s)
06:21 elukey: restart varnish backend on cp3043
06:19 marostegui: Optimize pagelinks and templatelinks on dbstore2002 s1 - T174509
06:16 elukey: restart varnish backend on cp3041
06:13 marostegui: Optimize pagelinks and templatelinks tables on s7 codfw master (db2029) this will generate lag - T174509
05:58 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034, db2055 db2062 - T174509 (duration: 00m 51s)
03:09 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 4 03:09:34 UTC 2017 (duration 7m 15s)
03:02 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 15m 20s)
02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 07m 44s)
00:33 ejegg: re-enabled fundraising queue consumers
00:29 ejegg: re-enabled CiviCRM de-dupe jobs
00:24 ejegg: re-enabled civicrm cron and ingenico jobs
00:20 ejegg: re-enabled fundraising stats and export jobs
00:16 ejegg: re-enabled omnimail import jobs
00:10 ejegg: re-enabled fundraising audit jobs
00:07 ejegg: re-enabled pending queue consumer

2017-10-03

23:52 ejegg: stopped all jobs for process-control upgrade
23:46 pnorman: Start regenerating map tiles on eqiad for z0-z12 - T176252
23:20 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T175971 T176088 Enable RemexHTML on wikitech and eswikiversity (duration: 00m 51s)
23:11 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: Pre-configure cirrussearch for 5.5.x upgrade (duration: 00m 51s)
22:26 bsitzmann@tin: Finished deploy [mobileapps/deploy@82aa7d6]: Update mobileapps to 5dc0c02 (T175762 T177001 T176525 T176517 T176519) (duration: 06m 14s)
22:20 bsitzmann@tin: Started deploy [mobileapps/deploy@82aa7d6]: Update mobileapps to 5dc0c02 (T175762 T177001 T176525 T176517 T176519)
20:29 pnorman: Start regenerating map tiles on codfw for z0-z12 - T176252
20:17 demon@tin: Synchronized php-1.31.0-wmf.2/extensions/GlobalUserPage/includes/Hooks.php: fix broken thing (duration: 00m 51s)
19:58 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.2
19:33 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 51s)
19:17 demon@tin: Finished scap: bootstrap wmf.2 (duration: 33m 50s)
19:01 twentyafterfour: Starting MediaWiki train for 1.30.1-wmf.2
18:43 demon@tin: Started scap: bootstrap wmf.2
18:25 chasemp: change to private vlan for labweb1001 on asw-b-eqiad
18:24 chasemp: change to private vlan for labweb1002 on asw2-d-eqiad
18:24 chasemp: auth-dnsupdate on bahan for 382027
16:51 ejegg: updated payments-wiki from c2a3d43 to b9859f6
15:57 bblack: upgrading basic packages on role::cache::text
15:50 bblack: upgrading basic packages on role::cache::upload
15:42 bblack: upgrading basic packages on role::cache::misc
15:29 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 06m 27s)
15:24 _joe_: uploading blubber to jessie-wikimedia T175296
15:23 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
15:04 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 05m 14s)
14:59 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
14:56 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 04m 25s)
14:56 bblack: upgrading basic packages on cp1065
14:52 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
13:56 zeljkof: EU SWAT finished
13:55 zfilipin@tin: Synchronized php-1.31.0-wmf.1/extensions/Translate/webservices/CxserverWebService.php: SWAT: Make CxserverWebService forward compatible (duration: 00m 45s)
13:46 godog: upgrade grafana to 4.5.2 on krypton - T175980
13:41 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Adjust wgNamespacesToBeSearchedDefault for enwikibooks, fawikibooks and hewikisource (T176906 T176908 T176907) (duration: 00m 46s)
13:34 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Limit thanks for new users at pl.wikipedia to 3 per day" (T176174 T169268) (duration: 00m 46s)
13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ShortUrl Extension on hiwikiversity (T177187) (duration: 00m 46s)
13:25 marostegui: Optimize templatelinks and pagelinks tables on s1, s4 and s6 on labsdb1010 - T174509
13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage Extension on hiwikiversity (T177188) (duration: 00m 46s)
13:04 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2044 - T174764 (duration: 00m 47s)
10:50 mobrovac@tin: Finished deploy [restbase/deploy@1cc530b]: (no justification provided) (duration: 08m 38s)
10:41 mobrovac@tin: Started deploy [restbase/deploy@1cc530b]: (no justification provided)
08:10 Amir1: cleaning up ores_classification tables (T159753)
08:05 marostegui: Deplo alter table on s3 codfw master (db2018) to add PK to the main tables - T163912
08:02 mobrovac@tin: Finished deploy [restbase/deploy@65f519b]: Create new buckets for feed end points (duration: 09m 57s)
07:52 mobrovac@tin: Started deploy [restbase/deploy@65f519b]: Create new buckets for feed end points
07:20 gehel: killing stuck tileshell and cleaning up /srv/osm_expire on maps-test2001 - T175123
07:05 elukey: restart varnish backend on cp3031 (503s)
07:03 marostegui: Optimize pagelinks and templatelinks tables on labsdb1010 for s5 - T174509
07:01 marostegui: Optimize templatelinks and pagelinks tables on s6 codfw master db2028 (this might generate lag) - T174509
06:46 marostegui: Drop now redundant indexes from pagelinks and templatelinks on s7 - T174509
06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1066 - T172679 (duration: 00m 47s)
05:54 marostegui: Optimize templatelinks and pagelinks tables on s5 master codfw (db2023): this will generate lag - T174509
05:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2049 and db2041 - T174509 (duration: 00m 47s)
03:23 demon@tin: Pruned MediaWiki: 1.30.0-wmf.19 [keeping static files] (duration: 01m 30s)
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 3 02:30:30 UTC 2017 (duration 6m 39s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 07m 22s)

2017-10-02

23:38 catrope@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/: T177250 (duration: 00m 47s)
23:36 catrope@tin: Synchronized php-1.31.0-wmf.1/resources/src/mediawiki.rcfilters/dm/mw.rcfilters.dm.FiltersViewModel.js: T177107 (duration: 00m 47s)
23:33 mutante: releases2001 - kill and restart jenkins with systemctl
23:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Update WMF mailing address (T173684) and enable jQuery 3 on Wiktionaries (T124742) (duration: 00m 48s)
23:00 mutante: releases1001 - remove and reinstall jenkins
22:56 Reedy: created newsletter tables on officewiki T176199
19:36 ejegg: turned on CiviMail record creation
18:48 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Change the Turkish Wiktionary logo (T176008) (duration: 00m 46s)
18:46 niharika29@tin: Synchronized static/images/project-logos/: Change the Turkish Wiktionary logo (T176008) (duration: 00m 48s)
17:46 gehel@tin: Finished deploy [wdqs/wdqs@f20b726]: latest wdqs GUI + minor fixes (duration: 01m 48s)
17:44 gehel@tin: Started deploy [wdqs/wdqs@f20b726]: latest wdqs GUI + minor fixes
14:08 zeljkof: EU SWAT finished
14:07 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Sandbox menu to Javanese Wikipedia (T176308) (duration: 00m 46s)
14:01 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:Newsletter on officewiki (T176199) (duration: 00m 46s)
13:59 zeljkof: extending EU SWAT for 5-10 minutes
13:51 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 45s)
13:50 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 46s)
13:44 zfilipin@tin: Synchronized static/images/project-logos/zh_classicalwiki-1.5x.png: static/images/project-logos/zh_classicalwiki-2x.png static/images/project-logos/zh_classicalwiki.png wmf-config/InitialiseSettings.php SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 46s)
13:42 godog: roll restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/381045/ - T166699
13:38 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the Extension:SandboxLink on the nlwikinews (T177170) (duration: 00m 46s)
13:32 zfilipin@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Wikidata: Dont persist description usages (yet) (T177153) (duration: 00m 45s)
13:27 zfilipin@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Wikidata: Dont persist description usages (yet) (T177153) (duration: 00m 46s)
13:20 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add eliminator as a priviliged account (T176554) (duration: 00m 46s)
13:17 zfilipin@tin: Synchronized wmf-config/Wikibase-labs.php: wmf-config/Wikibase.php wmf-config/Wikibase-production.php SWAT: labs: Use redis lock manager for dispatching changes of Wikibase (T175109) (duration: 00m 46s)
13:09 zfilipin@tin: Synchronized wmf-config/Wikibase-labs.php: wmf-config/Wikibase.php wmf-config/Wikibase-production.php SWAT: labs: Use redis lock manager for dispatching changes of Wikibase (T175109) (duration: 00m 47s)
12:56 elukey: added two new mediawiki jobrunners - mw131[01]
12:50 marostegui: Optimize tables pagelinks and templatelinks on s5: dbstore2001
11:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2039 - T174509 (duration: 00m 46s)
10:06 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1311.eqiad.wmnet
10:06 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1310.eqiad.wmnet
10:04 mobrovac: restbase restarting on restbase1007 to start logging error bodies for T176728
09:55 jynus: stopping and upgrading labsdb1010
09:27 Amir1: finished the cleanup of ores_classification table in enwiki (T159753)
09:12 godog: roll-restart thumbor to upgrade to 1.6 - T172556
09:06 elukey: forced remount of /mnt/hdfs on notebook1002
08:47 gehel: deploying latest tilerator on maps-test only, fix for T175123
08:46 gehel@tin: Finished deploy [tilerator/deploy@b02bd9a]: (no justification provided) (duration: 00m 22s)
08:46 gehel@tin: Started deploy [tilerator/deploy@b02bd9a]: (no justification provided)
08:46 marostegui: Poweroff db2044 for HW maintenance - T174764
08:40 marostegui: Stop MySQL on db2044 to get it ready to replace its mainboard - T174764
08:29 Amir1: finished the cleanup of ores_classification table in wikidatawiki and starting the enwiki one (T159753)
08:24 jynus: stopping dbstore1001 for maintenance
08:12 marostegui: Optimize tables pagelinks and templatelinks on s5: db1100 (already depooled) - T174509
07:59 marostegui: Drop now redundant indexes from pagelinks and templatelinks from s5 - T174509
07:57 Amir1: starting a round of cleanup in ores_classification table in wikidatawiki
07:41 marostegui: Stop MySQL on db1036 as it is going to be decommissioned - https://phabricator.wikimedia.org/T176311
07:32 marostegui: Optimize table pagelinks and templatelinks on s6: dbstore2001 - T174509
07:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2039 - T174509 (duration: 00m 46s)
07:20 marostegui: Optimize table pagelinks and templatelinks on s6: db2039 - T174509
07:10 marostegui: Optimize table pagelinks and templatelinks on s1: db2034, db2055 db2062 - T174509
07:06 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2034, db2055 db2062 - T174509 (duration: 00m 46s)
06:55 _joe_: restarting a few API appservers with high cpu load
06:19 elukey: restart varnish backend on cp3040
06:10 elukey: restart varnish backend on cp3033
06:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 46s)
06:07 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 48s)
02:43 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 2 02:43:33 UTC 2017 (duration 6m 39s)
02:36 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 08m 42s)

2017-10-01

21:30 madhuvishy: powercycling labvirt1015
18:58 hashar: restarted Jenkins. Was stuck somehow
13:50 elukey: restart hhvm on mw1167 (jobrunner) - hhvm stuck, dump-debug in /tmp/hhvm.9624.bt.

2017-09-30

20:54 catrope@tin: Synchronized wmf-config/CommonSettings.php: Bump $wgResourceLoaderStorageVersion (T176884) (duration: 00m 47s)
00:37 ejegg: increased PayPal Express Checkout cURL timeout to 12 seconds

2017-09-29

22:12 ejegg: updated CiviCRM from 784756b to 59e3e00
21:19 mutante: labstore2003/2004: closed idle screen sessions
20:27 mutante: mwlog1001, oxygen - closing unused screen session
20:20 mutante: install1002/install2002 - closing unused screen sessions (except a tail -f)
20:11 mutante: restbase2001 - closing screen sessions that weren't running anything anymore
19:23 ebernhardson: completed running cirrussearch category moves on commonswiki
18:19 catrope@tin: Synchronized php-1.31.0-wmf.1/includes/libs/CSSMin.php: T176884 (duration: 00m 47s)
18:08 chasemp: remove 'ToolLabs webservice - lighttpd on precise' catchpoint check for Catchpoint
18:07 chasemp: remove labsdb1002 checks on catchpoint (Toolforge)
18:06 chasemp: remove Labs puppetmaster eqiad catchpoint check (for Toolforge)
18:06 chasemp: remove grid-start-precise catchpoint check
18:04 chasemp: update catchpoing test for stream.wikimedia.org to watch https://stream.wikimedia.org/?doc
17:00 ebernhardson: kicking off extra job runners to process jobs moving commonswiki Category pages from general to content serach index
16:37 andrewbogott: re-imaging labvirt1017, 1018 in order to get it on the standard 4.4.0-81-generic kernel
15:52 andrewbogott: re-imaging labvirt1015 in order to get it on the standard 4.4.0-81-generic kernel
13:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2070, db2069 and db2042 - T174509 (duration: 00m 47s)
13:18 Amir1: starting a round of cleanup in ores_classification table in enwiki (T159753)
13:09 elukey: depool mw1265 (hhvm 3.18.5) - disk filled up
12:43 bblack: removed numa=isolate sys entries manually (runtime + future boots) from cp4021
11:53 bblack: cp4021: reboot for numa mode switch
11:44 elukey: re-enable job runner daemons on mw130[8,9]
11:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2053 db2046 - T174509 (duration: 00m 48s)
11:05 elukey: precautionary stop of jobrunner/jobchron on the new mw130[8,9] - job queue size rapid increase investigation
09:55 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1309.eqiad.wmnet
09:33 ema: upgrade pybal to 1.14.0 on codfw primaries
08:20 ema: upgrade pybal to 1.14.0 on codfw secondaries
05:53 ema: cp4024 repooled T174891
05:21 ema: powerup cp4024 T174891
00:26 ejegg: updated fundraising tools from 69a9627 to 6d4b6f3
00:16 ejegg: updated CiviCRM 2138869 to 784756b
00:07 mutante: releases1001 - starting Jenkins setup wizard with generated admin pass
00:05 mutante: releases1001 - stopped puppet, manually fixing --prefix=/ci setting for jenkins process, killing it, removing init.d file, starting with systemd, jenkins now up T164030

2017-09-28

23:46 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 20s)
23:45 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart
22:18 MaxSem: Deployed fix for T176247
21:30 ejegg: updated CiviCRM from 3f48cb0 to 2138869
21:23 XenoRyet: Updated SmashPig from dc62203 to d056a13
19:36 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.1
19:34 legoktm: mysql:wikiadmin@db1052 [enwiki]> delete from user_groups where ug_user =17629530 limit 1; (T176798)
19:31 bblack: cp4026 - depool->reboot for bios updates, isolcpus fix
19:28 legoktm: mysql:wikiadmin@db1062 [centralauth]> UPDATE localuser SET lu_attached_method="new" WHERE lu_attached_method is NULL AND lu_name="Rua";
19:14 ejegg: updated fundraising tools 5708cf1 Add 'first_donation_date' column to Silverpop export
19:14 robh: cp4024 initial tests done (takes about 2 hours or so) then prompted for more in depth testing (hit yes, no failures so far) T174891
19:00 bblack: cp4025 - depool->reboot for bios updates, isolcpus fix
18:54 thcipriani@tin: Synchronized php-1.31.0-wmf.1/skins/MinervaNeue/includes/skins/SkinMinerva.php: SWAT: Revision::newFromTitle may return null T176882 (duration: 00m 50s)
18:42 bblack: cp4023 - depool->reboot for bios updates, isolcpus fix
18:37 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create rollbacker group at viwiktionary T176979 (duration: 00m 50s)
18:26 thcipriani@tin: Synchronized docroot/noc: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging PART II T104148 (duration: 00m 49s)
18:25 thcipriani@tin: Synchronized wmf-config: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging T104148 (duration: 00m 51s)
17:58 arlolra: Updated Parsoid to 2f4b9a8c (T176363, T176151, T170832)
17:52 mutante: librenms - renewed LE TLS cert ..via puppet, after Icinga alerted and accidentally still had "do_acme: false" in Hiera
17:46 arlolra@tin: Finished deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c (duration: 15m 25s)
17:43 bblack: cp4028 - depool->reboot for bios updates, isolcpus fix
17:40 moritzm: installing apache updates on cobalt
17:32 bblack: cp4022 - depool->reboot for bios updates, isolcpus fix
17:30 arlolra@tin: Started deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c
17:25 marostegui: Run fixStuckGlobalRename.php --wiki=enwiktionary --logwiki=metawiki "CodeCat" "Rua" on terbium - T176985
17:20 bblack: cp4027 - depool->reboot for bios updates, isolcpus fix
17:08 bblack: cp4021 - depool->reboot for bios update
17:05 robh: bios firmware uploaded and awaiting reboot for application on cp402[1235678]
16:48 bd808: Created centralauth.{analytics,web}.db.svc.eqiad.wmflabs Designate CNAME (T176978)
16:26 robh: cp4024 has had ilom and bios firmware updaed per epsa error checking, now re-running epsa utility T174891
16:09 robh: cp4024 bios update in progress (has been for last 5 minutes), letting it alone for a solid 30 to let it finish
16:01 marostegui: Global rename of CodeCat → Rua - T176985
15:22 hoo: Ran scap pull on mwdebug1001 after experiments with T176844
14:54 dcausse: restarting elastic on relforge servers to reinstall prod ltr plugin
14:51 marostegui: Stop MySQL on db1035 as server is going to be decommissioned - T176931
14:45 moritzm: installing apache updates on puppet mastes (will trigger a few failing puppet runs, but ircecho disabled temporarily)
14:42 marostegui: Remove db1035 from tendril - T176931
14:31 moritzm: upgrading mw1276 (API canary) to HHVM 3.18.5
14:28 moritzm: upgrading mwdebug* to HHVM 3.18.5
13:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 49s)
13:55 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 50s)
13:40 zeljkof: EU SWAT finished
13:38 zfilipin@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: SWAT: ve.init.mw.DesktopArticleTarget: Remove hack for reversed tabs in RTL in Vector (T50017) (duration: 00m 50s)
13:37 moritzm: upgrading mw1262 to mw1265 (canary app servers) to HHVM 3.18.5
13:35 moritzm: uploaded HHVM 3.18.5 for jessie-wikimedia to apt.wikimedia.org
13:26 marostegui: Optimize table on s4 codfw master (db2051), with replication (lag might be generated) - T174509
13:13 ema: bounce pybal on ulsfo primaries to pick up bgp-local-ips config change T103882
13:12 marostegui: Drop redundant indexes from pagelinks and templatelinks on s4 - T174509
13:10 moritzm: upgrading mw1261 to HHVM 3.18.5
13:08 ema: bounce pybal on ulsfo secondaries to pick up bgp-local-ips config change T103882
13:02 marostegui: Optimize template links and pagelinks on db2053 and db2046 - T174509
13:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2053 db2046 - T174509 (duration: 00m 49s)
12:48 dcausse: restarting elastic on relforge servers to test a new version of the ltr plugin
12:27 ema: reboot cp402[2-6] to disable numa isolation
10:30 gehel: restart elasticsearch on relforge to validate new logging config - T175242
10:09 elukey: incrementally add mw1312-17 api-appservers to serve live traffic (weights will be raised incrementally)
10:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1308.eqiad.wmnet
09:59 elukey: added new mediawiki jobrunner - mw1308
09:52 ema: upgrade pybal to 1.14.0 on lvs1009-10
09:49 aaron@tin: Synchronized wmf-config/CommonSettings.php: Enable logging of post-send DB updates (duration: 00m 55s)
09:42 ema: upgrade pybal to 1.14.0 on lvs400[13]
08:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1288.eqiad.wmnet
07:27 moritzm: upgrading app servers in deployment-prep to HHVM 3.18.5
07:14 moritzm: installing apache updates on silver/wikitech.wikimedia.org
06:59 moritzm: installing apache updates on graphite* hosts
06:28 elukey: restart varnish backend on cp3032
05:54 marostegui: Optimize tables pagelinks and templatelinks on dbstore2001 (s2) - T174509
05:53 marostegui: Optimize tables pagelinks and templatelinks on dbstore2002 (s2) - T174509
05:43 marostegui: Run maintain-views on labsdb1001, labsdb1003, labsdb1009, labsdb1010, labsdb1011 for hiwikivoyage - T173027
05:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2049 and db2041 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 47s)
05:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2064 db2063 - T174509 (duration: 00m 48s)
05:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2076 db2067 db2060 - T174509 (duration: 00m 48s)
05:19 tstarling@tin: Finished scap: updating Vector skin for gerrit 381153 (duration: 20m 39s)
04:59 tstarling@tin: Started scap: updating Vector skin for gerrit 381153
03:57 bblack: ulsfo dns-depooled at ~03:51
03:26 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 28 03:26:18 UTC 2017 (duration 6m 10s)
03:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 44s)
02:42 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 24s)
00:36 twentyafterfour: upgrading phabricator, expect minimal downtime

2017-09-27

23:39 robh: cp4024 is running hardware testing, leave it alone T174891
23:17 catrope@tin: Synchronized wmf-config/Wikibase.php: Make CirrusSearch default for wbsearchentities on testwikidatawiki T175741 (duration: 00m 48s)
22:34 maxsem@tin: Synchronized php-1.31.0-wmf.1/includes/EditPage.php: https://gerrit.wikimedia.org/r/#/c/381139/1 (duration: 00m 49s)
19:14 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 -> wmf.1
19:13 demon@tin: Synchronized php: symlink bump (duration: 00m 47s)
19:10 demon@tin: Synchronized php-1.31.0-wmf.1/maintenance/backup.inc: unbreak dumps (duration: 00m 49s)
18:46 catrope@tin: Finished scap: SWAT: T176762, T175358, T172023, T174719, and add html5-misnesting to Linter (duration: 22m 55s)
18:39 bblack: re-pooling upload@ulsfo (text@ulsfo remains pooled)
18:23 catrope@tin: Started scap: SWAT: T176762, T175358, T172023, T174719, and add html5-misnesting to Linter
18:21 ejegg: updated payments-wiki from 6104e3d to c2a3d43
17:40 ejegg: updated SmashPig Adyen settings
17:29 bblack: uploaded nginx-1.13.5-1+wmf1 to stretch-wikimedia, and matching nginx-1.13.5-1+wmf1~jessie1 to jessie-wikimedia
16:27 XioNoX: setting local-as to selected transit BGP sessions - T167840
16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.ulsfo.wmnet
16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.ulsfo.wmnet
15:59 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.ulsfo.wmnet
15:58 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.ulsfo.wmnet
15:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1103 with low weight - T172679 (duration: 00m 47s)
15:51 ema: lvs4002: upgrade pybal to 1.14.0
15:49 ema: lvs4004: upgrade pybal to 1.14.0
15:28 _joe_: uploaded prometheus-statsd-exporter to stretch-wikimedia T175539
15:24 bblack: rebooting cp402[2356] (bnx2x oops)
15:15 moritzm: revoking salt certs for WMCS
15:12 mlitn@tin: Synchronized wmf-config/CommonSettings-labs.php: Remove 3D beta config (duration: 00m 47s)
15:11 mlitn@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Remove 3D beta config (duration: 00m 48s)
15:10 mlitn@tin: Synchronized wmf-config/CommonSettings.php: Load 3D extension (duration: 00m 49s)
15:09 mlitn@tin: Synchronized wmf-config/InitialiseSettings.php: Config 3D to be loaded on test, test2 (duration: 00m 48s)
15:00 mlitn@tin: Finished scap: Enable 3D extension (duration: 37m 09s)
14:58 ema: lvs1007: upgrade pybal to 1.14.0
14:45 elukey: rolling restart of all the Yarn nodemanager daemons on analytics1028-1068
14:43 bblack: cp1008/pinkunicorn upgraded to test build of nginx-1.13.5-1+wmf1
14:28 volans: uploaded cumin_1.2.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
14:24 akosiaris: change wtp1001 to wtp1024 weights to 5 from 15 in preparation for deprecation
14:24 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1024.eqiad.wmnet
14:24 mlitn@tin: Started scap: Enable 3D extension
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1023.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1022.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1021.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1020.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1019.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1018.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1017.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1016.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1015.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1014.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1013.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1012.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1011.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1010.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1009.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1008.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1007.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1006.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1005.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1004.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1003.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1002.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1001.eqiad.wmnet
14:17 akosiaris: T175242 restart eventstreams across the fleet to pick up the change in a rolling restart manner with a batch size of 2
14:16 akosiaris: T175242 restart parsoid across the fleet to pick up the change in a rolling restart manner with a batch size of 5
14:03 akosiaris: T175242 restart tilerator, tileratorui, restbase across the fleet to pick up the change in a rolling restart manner with a batch size of 2
13:43 akosiaris: T175242 re-enable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/. Run puppet as well in a batched execution
13:42 anomie@tin: Synchronized php-1.31.0-wmf.1/includes/specials/SpecialWatchlist.php: SWAT: {{gerrit|380968}} Fix watchlist "in the last X hours" display (duration: 00m 48s)
13:42 elukey: raised Hadoop HDFS namenode master daemon max heap size to 6G (prev 4G) on analytics100[12]
13:31 akosiaris: T175242 eventstreams requires manual restart
13:25 akosiaris: T175242 parsoid requires manual restart
13:15 akosiaris: T175242 restbase requires manual restart
13:11 addshore: SWAT done
13:11 addshore@tin: Synchronized php-1.31.0-wmf.1/extensions/TwoColConflict/modules/ext.TwoColConflict.BaseVersionSelector.css: SWAT: Address changes in the label style in OOUI (duration: 00m 46s)
13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add autopatrolled user group to dty.wikipedia (T176709) (duration: 00m 47s)
13:01 akosiaris: T175242 tilerator and tileratorui need manually restart
12:54 akosiaris: T175242 enabled puppet in aqs kafka maps maps-test selected hosts and ran puppet manually.
12:47 akosiaris: T175242 disable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/
12:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1103 to the array of hosts (duration: 00m 48s)
12:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1103 to the array of hosts (duration: 00m 49s)
12:14 moritzm: installing apache updates on einsteinium/tegmen
12:06 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 00m 49s)
12:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 01m 00s)
09:45 marostegui: Stop mysql on db1035 to copy its data to db1103 - T172679
09:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1035 to transfer its data to db1103 - T172679 (duration: 00m 48s)
08:55 mobrovac@tin: Finished deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100 (duration: 01m 26s)
08:53 mobrovac@tin: Started deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100
08:43 mobrovac@tin: Finished deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100 (duration: 10m 01s)
08:34 elukey: raise traffic weights to 30 for mw13[19-28] incrementally - T165519
08:33 mobrovac@tin: Started deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100
08:16 marostegui: Optimize tables pagelinks templatelinks on db2064 and db2063 - T174509
08:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2064 db2063 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
07:55 marostegui: Optimize pagelinks and template links tables on db2076 db2067 db2060 - T174509
07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2076 db2067 db2060 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
07:37 marostegui: Optimize pagelinks and template links tables on db2070 db2069 db2042 db2016 - T174509
07:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2070, db2069 and db2042 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
07:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2071 and db2072 - T174509 (duration: 00m 47s)
06:57 marostegui: Deploy alter table on s6 codfw - T174509
06:55 marostegui: Deploy alter table on db1052 (enwiki master) - T174509
06:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add task number to db2044 line (duration: 00m 48s)
06:37 ema: restart varnish-be on cp3040
06:36 moritzm: installing git security updates (our config is not affected, but still fixing the underlying issue)
06:33 ema: restart varnish-be on cp3041
06:29 marosteg1i: Reboot db2044 after storage failure - T174764
06:15 elukey: restart varnish backend on cp3043
05:41 madhuvishy: service nova-fullstack restart on labnet1001
03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 27 03:17:22 UTC 2017 (duration 6m 22s)
03:11 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 55s)
02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 08m 42s)

2017-09-26

23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWSignatureTool.js: SWAT: Revert MWSignatureTool case (duration: 00m 48s)
23:20 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 48s)
23:19 thcipriani@tin: Synchronized php-1.31.0-wmf.1/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 51s)
22:03 no_justification: gerrit master restart, back momentarily
20:32 mutante: phab1001 - restarted apache after upgrade
20:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2044 - T174764 (duration: 00m 50s)
20:20 mutante: phab1001 (prod phab) - upgrade Apache package
20:15 mutante: phab2001 - install Apache and MariaDB package upgrades
19:16 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.31.0-wmf.1
18:25 mepps: Updated crm from 89b8007 to 3f48cb0
18:00 demon@tin: Finished scap: 1.31.0-wmf.1 bootstrap + testwiki (duration: 47m 56s)
17:12 demon@tin: Started scap: 1.31.0-wmf.1 bootstrap + testwiki
15:50 volans: powecycled cp2012, no ping or ssh console stuck
15:18 XioNoX: starting eqiad frack switch to new infra - T174218
15:06 gehel: restarting blazegraph on all wdqs nodes for heap resize - T175919
14:47 mobrovac@tin: Finished deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940 (duration: 20m 35s)
14:36 marostegui: Optimize tables pagelinks and templatelinks on db2071 and db2072 from s1 - T174509
14:27 mobrovac@tin: Started deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940
14:26 hoo: mwscript refreshLinks.php --wiki elwiki --namespace 0 on terbium has finished (T151717)
14:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2071 and db2072 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 44s)
13:39 marostegui: Deploy alter table on s6 master (with replication enable, so lag will be generated) db2028 - T163979
13:25 hashar@tin: Synchronized wmf-config/filebackend.php: Expose Thumbor swift username - T144479 (duration: 00m 44s)
13:24 hashar@tin: Synchronized wmf-config/CommonSettings.php: Enable asia-specific Navigation Timing metric - T169522 (duration: 00m 47s)
13:22 godog: upgrade grafana to 4.5.2 on labmon1001 - T175980
13:15 hashar@tin: Synchronized wmf-config/CommonSettings.php: Disable RelatedArticles instrumentation on all wikis - T174944 (duration: 00m 45s)
13:02 marostegui: Drop redundant indexes from s6 - T174509
13:00 elukey: add mw132[7,8] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
12:44 gehel: restarting blazegraph on wdqs1004 / wdqs2001 for heap resive - T175919
12:27 marostegui: Drop redundant indexes from s2 - T174509
12:12 marostegui: Drop redundant indexes from enwiki on eqiad - T174509
12:03 kartik@tin: Finished deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction (duration: 03m 03s)
12:00 kartik@tin: Started deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction
11:58 moritzm: uploaded prometheus-apache-exporter 0.3-1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
11:57 marostegui: Drop redundant indexes from enwiki on codfw - T174509
11:49 moritzm: uploaded prometheus-hhvm-exporter 0.3.1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
11:30 kartik@tin: (no justification provided)
11:29 kartik@tin: Finished deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction (duration: 00m 52s)
11:28 kartik@tin: Started deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction
10:52 volans: uploaded cumin_1.1.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
10:34 joal@tin: Finished deploy [analytics/refinery@5d806c6]: Regular analytics deploy (duration: 08m 00s)
10:26 joal@tin: Started deploy [analytics/refinery@5d806c6]: Regular analytics deploy
10:20 _joe_: rebooting copper (again)
09:12 elukey: add mw132[4,5,6] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
09:10 _joe_: rebooting copper to verify docker setup is correct
08:46 moritzm: installing apache updates on mw* app servers
08:44 godog: run puppet on ms-fe* to reduce conntrack generated by swift internal clients towards backends - T173731
08:24 godog: roll-restart thumbor to upgrade to 1.5 - T173804
08:09 godog: roll-restart swift-proxy to apply https://gerrit.wikimedia.org/r/#/c/380483/ - T161535
08:08 marostegui: Drop old temporary tar.gz files from dbstore1001 to get back some disk space
08:07 marostegui: Drop table trackbacks from s3 - T175051
07:48 mattflaschen@tin: Synchronized wmf-config/CommonSettings-labs.php: RCFilters: Beta Cluster only (duration: 00m 46s)
07:46 marostegui: Drop trackbacks table from s7 - T175051
07:28 marostegui: Drop trackbacks table from s6 - T175051
07:16 moritzm: installing perl security updates
06:45 _joe_: restarting varnish backend on cp3033, throwing 503s
06:29 mobrovac@tin: Finished deploy [cpjobqueue/deploy@17c3833]: (no justification provided) (duration: 00m 28s)
06:28 mobrovac@tin: Started deploy [cpjobqueue/deploy@17c3833]: (no justification provided)
06:18 marostegui: Drop table trackbacks from s4 - T175051
06:06 marostegui: Drop table trackbacks from s5 (only exists on dewiki) - T175051
05:48 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 - T176573 (duration: 00m 44s)
05:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove old db1055 comments (duration: 00m 46s)
02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 26 02:29:31 UTC 2017 (duration 5m 52s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 26s)

2017-09-25

23:58 ejegg: updated payments-wiki from d2ce697 to 6104e3d
23:15 mutante: gerrit2001 reinstalled with stretch, revoked old puppet cert, accepted new puppet cert, initial run that will do base and all the gerrit things at once.. (T168562)
22:53 mutante: gerrit2001 - PXE booting
22:34 mutante: gerrit2001 - schedule downtime for reinstall with strech, reboot imminent
20:57 hoo: Manually pruned the sites and site_identifiers table on hiwikivoyage, than ran populateSitesTable.php. (T173030)
20:37 arlolra: Updated Parsoid to 4230c27b (T176425, T173029)
20:30 arlolra@tin: Finished deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b (duration: 16m 52s)
20:13 arlolra@tin: Started deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b
19:58 XioNoX: renumbering ams BGP communities - T167840
19:25 robh@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4021.ulsfo.wmnet
19:02 robh: cp4021 coming down for memory swap
19:02 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4021.ulsfo.wmnet
18:54 MaxSem: started populating ip_changes on group2 wikis
18:38 demon@tin: Synchronized scap/plugins/clean.py: no-op/consistency (duration: 00m 45s)
18:22 ejegg: updated CiviCRM from 45c56cf to 89b8007
17:58 demon@tin: Pruned MediaWiki: 1.30.0-wmf.15 (duration: 02m 46s)
17:07 gehel@tin: Finished deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update (duration: 01m 51s)
17:05 gehel@tin: Started deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update
17:02 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Disable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 46s)
16:37 demon@tin: Synchronized php-1.30.0-wmf.19/maintenance/getConfiguration.php: getConfiguration: Don't bail when a valid variable is set null (duration: 00m 47s)
15:57 urandom: T169940: Converting restbase-ng data tables to size-tiered compaction
14:39 gehel: dropping badly named apifeature index from cirrus elasticsearch codfw/eqiad - T176430
14:31 gehel: rolling restart of logstash for config change - T176430
14:13 gehel: rolling restart of logstash for config change - T176430
13:46 zeljkof: EU SWAT finished
13:45 zfilipin@tin: Synchronized static/images/project-logos/hiwiki.png: SWAT: Make hiwiki logo a little bit bigger (T175721) (duration: 00m 46s)
13:37 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Import sources on hr.wikibooks (T176320) (duration: 00m 46s)
13:30 addshore@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/WikimediaEventsHooks.php: SWAT: gerrit:379749 onBeforeInitializeWMDECampaign, fix early bail condition T174948 (duration: 00m 46s)
13:18 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add several rights to eliminators in fawiki (T176553) (duration: 00m 46s)
12:52 moritzm: removed salt master from neodymium
12:38 moritzm: revoked all salt keys on neodymium
11:58 godog: powercycle ms-be2020 - blk_update_request: I/O error, dev sdb, sector in consol
11:54 moritzm: removing salt-minion/salt-common from production
11:47 Dereckson: Repopulate sites tables for Wikidata clients (T173013)
11:47 Dereckson: Create new database and tables for hi.wikivoyage (T173013)
11:45 dereckson@tin: Synchronized wmf-config/interwiki.php: Interwiki map update (+hiwikivoyage and other changes previous month, T176572) (duration: 00m 46s)
11:39 marostegui: Run redact_sanitarium for hiwikivoyage on db1095 - T173027
11:39 moritzm: upgrading tor to latest stable release (0.3.1.7)
11:37 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Initial configuration for hi.wikivoyage.org (T173013) (duration: 00m 46s)
11:37 dereckson@tin: Synchronized static/images/project-logos/: Logos for hi.wikivoyage.org (T173013) (duration: 00m 45s)
11:34 dereckson@tin: rebuilt wikiversions.php and synchronized wikiversions files: +hiwikivoyage (T173013)
11:32 dereckson@tin: Synchronized dblists: Add hiwikivoyage to the set (T173013) (duration: 00m 46s)
10:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 weight (duration: 00m 46s)
10:14 elukey: add mw1323 to live traffic (mw appserver) - traffic weights will go from 5 to 20 incrementally
09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight and remove main traffic from special slaves (duration: 00m 46s)
09:07 marostegui: Add 50GB to db1036 /srv partition - T176311
09:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1036 and full repool db1101 - T176311 (duration: 00m 46s)
08:49 moritzm: uploaded php-luasandbox 2.0.14, hhvm-tidy 0.1.3 and php-wikidiff2 1.4.1 for stretch-wikimedia to apt.wikimedia.org
08:36 mobrovac@tin: Finished deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233 (duration: 10m 03s)
08:26 mobrovac@tin: Started deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233
08:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 45s)
07:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101 - T176311 (duration: 00m 46s)
07:42 moritzm: uploaded hhvm 3.18.2+dfsg-1+wmf5+deb9u1 for stretch-wikimedia to apt.wikimedia.org
07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 47s)
07:37 elukey: add mw132[0,2] to live traffic (mw appservers) - traffic weights will go from 5 to 20 incrementally
06:38 moritzm: installing apache security updates on logstash*
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 25 02:36:48 UTC 2017 (duration 6m 42s)
02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 17s)

2017-09-24

14:23 jynus: restarting db2047 for kernel upgrade
14:16 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 (duration: 00m 48s)
off: restarted ircecho on einsteinium

2017-09-23

21:12 hoo: Ran scap pull on mwdebug1001 to undo changes to profile https://github.com/wmde/WikibaseDataModel/pull/761/files

2017-09-22

23:36 mutante: gerrit2001 - rebooting .. wave a dead chicken http://www.catb.org/jargon/html/W/wave-a-dead-chicken.html
22:56 mutante: gerrit2001 - trying to manually start gerrit again, as opposed to puppet doing it.. debugging gerrit-ssh issue there, cobalt still untouched
22:34 mutante: gerrit2001 - stopping gerrit with gerrit.sh stop, letting puppet start it again
22:29 mutante: gerrit2001 - systemd says gerrit.service is failed. gerrit.sh start says "Already Running!!" :p - cobalt is fine
18:26 demon@tin: Pruned MediaWiki: 1.30.0-wmf.18 [keeping static files] (duration: 01m 29s)
15:21 hashar: Restarted Jenkins. Out of memory)
14:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1101 with low weight (duration: 00m 47s)
14:43 moritzm: uploaded php-luasandbox build for src:php5.5 (required for CI tests on jessie)
14:42 moritzm: updated tor packages to 0.3.1.7 (new stable series)
13:49 elukey: mw1321 (new appserver) serving traffic (going to increase its weight up to 20)
12:02 hashar: apt-get upgrade on contint1001 / contint2001
09:42 elukey: mw1319 (new appserver) serving traffic (going to increase its weight up to 20)
09:14 jynus: stop mariadb at db1055 for upgrade and maintenance
06:32 moritzm: installing emacs security updates on trusty (Debian already fixed)

2017-09-21

23:56 ejegg: updated payments-wiki from 7ed7243 to d2ce697
23:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/extension.json: T175047: Stop delivering search relevance survey javascript (duration: 00m 46s)
23:18 ebernhardson@tin: Synchronized wmf-config: T175047: Turn off human search relevance survey (duration: 00m 48s)
23:16 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T175047: Turn off human search relevance survey (duration: 00m 46s)
22:59 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1055 (duration: 00m 46s)
22:12 volans: uploaded cumin_1.1.0-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
22:10 ejegg: updated payments-wiki from 9da8482 to 7ed7243
21:07 demon@tin: Synchronized wmf-config/CommonSettings.php: extdist settings (duration: 00m 46s)
21:04 ejegg: updated CiviCRM from fc56901 to 45c56cf
19:32 demon@tin: Synchronized wmf-config/InitialiseSettings.php: adjust logging on csp reports (duration: 00m 46s)
19:20 demon@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: fix user/wgUser mixup (duration: 00m 46s)
19:16 gehel: upgrade to nodejs 6.11 on maps servers (including restart of tilerator / kartotherian) - T171707
19:16 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
19:04 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.19
18:56 urandom: T171772: Applying locally hacked Prometheus exporter config to RESTBase dev Cassandra instances
18:44 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters: SWAT: WLFilters: Do not hide .watchlistDetails while loading T176300 RCFilters: Make the interface not jump around while loading (duration: 00m 49s)
18:41 urandom: T171772: Restarting Cassandra restbase-dev1004-a to apply locally hacked Prometheus exporter config
18:34 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turning off Explore Similar AB test T175649 (duration: 00m 49s)
18:31 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Print instrumentation T176341 (duration: 00m 52s)
14:56 moritzm: uploaded php-defaults and php-redis built against src:php5.5 to component/ci for jessie-wikimedia
13:45 gehel: upgrade to nodejs 6.11 on the full maps-test cluster - T171707
13:12 zeljkof: EU SWAT finished
13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
13:09 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
13:07 zfilipin@tin: Synchronized tests/cirrusTest.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 49s)
12:55 _joe_: stopped long-running runjobs for refreshlinks on commons.
12:50 mobrovac@tin: Finished deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy (duration: 18m 34s)
12:32 mobrovac@tin: Started deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy
11:59 moritzm: upgrading apache on deployment servers and script runners
11:40 moritzm: upgrading apache on canary app servers
11:17 moritzm: installing apache security updates (our configurations are not affected, the upload of 2.4.10-10+deb8u11+wmf1 is mostly for the benefit of whatever people are doing in Cloud VPS, but we should still fix the underlying bug in production as well)
11:14 dcausse: reindexing all hebrew wikis to pickup the new hebmorph analyzer
11:09 dcausse: reindexing ilwikimedia in codfw to pickup and test new hebmorph analyzer
10:26 akosiaris: upload docker-ce_17.06.2~ce-0~debian_amd64.deb to apt.wikimedia.org jessie-wikimedia/thirdparty/ci T175293
09:59 dcausse: reindexing group1 wikis T176397
09:54 dcausse: reindexing group0 wikis T176397
08:35 moritzm: unbreak deployment-puppetmaster02 in deployment-prep (broken by unattended-upgrades update of apache T159254)
08:35 akosiaris: pool all of wtp1025 to wtp1048 T165520
08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1044.eqiad.wmnet
08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1043.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1042.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1041.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1040.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1039.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1038.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1037.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1036.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1035.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1034.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1033.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1032.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1031.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1030.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1029.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1028.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1027.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1026.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:30 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:30 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: name=wtp1025.eqiad.wmnet
08:25 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 03m 18s)
08:22 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
08:21 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 00m 58s)
08:20 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
08:19 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 05m 22s)
08:14 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
08:08 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940 (duration: 10m 28s)
07:57 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940
07:31 _joe_: restarted aphlict on phab1001
07:27 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940 (duration: 00m 30s)
07:27 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940
07:09 ema: bounce pybal on lvs1003 to clear stale alert T176388
06:59 ema: bounce pybal on lvs1006 to clear stale alert
06:55 ema: bounce pybal on lvs1009 to clear stale alert
03:24 bblack: depooled cp4026
03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 21 03:12:41 UTC 2017 (duration 6m 36s)
03:06 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 14m 44s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 50s)
00:41 eileen: update civicrm from da833ac to fc56901
00:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 53s)
00:18 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 49s)
00:09 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on commons (T124742) (duration: 00m 49s)
00:02 catrope@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/: Lazy-load the RCFilters menu (T176250) (duration: 00m 48s)

2017-09-20

23:49 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 48s)
23:47 catrope@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 49s)
23:40 mutante: phabricator: aphlict (notification service) now running on prod server
23:36 mutante: phab1001 - re-enable puppet, run after follow-up fix, adding notification server config
23:31 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add throttle rule for John Michael Kohler Art Center (T176287) (duration: 00m 48s)
23:29 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable user email blacklist on meta (T174694) (duration: 00m 48s)
23:21 catrope@tin: Synchronized dblists/categories-rdf.dblist: Add mlwiki to categories-rdf.dblist (duration: 00m 48s)
23:14 mutante: phab2001 - deluser aphlict, delgroup aphlict, try to let puppet create it again, still fails, just won't create the group
23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add patroller group on metawiki (T176079) (duration: 00m 48s)
23:11 mutante: phab2001 - groupadd aphlict,temp adduser aphlict to debug user creation issue
22:22 ebernhardson: elasticsearch eqiad completely recovery, all green. Returning recovery max_bytes_per_sec to 20mb
21:54 mutante: phab: disable puppet on phab1001 for a minute, apply notification server change on phab2001
21:45 eileen: update civicrm from 0fb4676 to da833ac
21:09 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4015.ulsfo.wmnet
20:58 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4007.ulsfo.wmnet
20:40 bsitzmann@tin: Finished deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263) (duration: 06m 05s)
20:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4014.ulsfo.wmnet
lunch: increase max recovery bytes/s to 80mb in eqiad elasitcsearch cluster
20:34 bsitzmann@tin: Started deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263)
20:33 demon@tin: Synchronized php-1.30.0-wmf.19/includes/diff/DifferenceEngine.php: fix some warnings (duration: 00m 51s)
19:55 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4016.ulsfo.wmnet
19:54 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster (duration: 00m 10s)
19:54 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster
19:53 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4006.ulsfo.wmnet
19:48 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release (duration: 00m 47s)
19:47 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release
19:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4008.ulsfo.wmnet
19:26 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.19
19:24 demon@tin: Synchronized php: symlink swap for wmf.19 (duration: 00m 49s)
19:02 catrope@tin: Finished scap: Patches for T176302, T175962, T173533 (duration: 21m 29s)
18:40 catrope@tin: Started scap: Patches for T176302, T175962, T173533
17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on terbium. Can be KILLed at any time, if needed.
17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on elwiki. Can be KILLed at any time, if needed.
17:13 hoo: Ran mwscript refreshLinks.php --wiki elwiki --namespace 0 -e 30 for testing purposes
17:06 ejegg: removed limit from donation queue consumer
17:05 otto@tin: Finished deploy [eventstreams/deploy@e62ab64]: (no justification provided) (duration: 04m 44s)
17:01 otto@tin: Started deploy [eventstreams/deploy@e62ab64]: (no justification provided)
16:53 madhuvishy: Upgraded and restarted apache2 on labs-puppetmaster.wikimedia.org
16:45 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch
16:42 gehel: rolling restart of logstash ingesters after cirrus eqiad recovery
16:34 gehel: elasticsearch eqiad is finally recovering, after the restart of elastic1051
16:34 gehel: cleanup jar hell and restart elastic1051
16:19 gehel: starting elasticsearch on all nodes on eqiad
16:10 gehel: restarting elasticsearch masters on eqiad (elastic1030, 36 and 40)
16:04 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch
16:03 gehel: depooling all nodes in elasticsearch eqiad to investigate failed cluster restart (traffic is already directed to codfw)
15:46 ejegg: limited donations queue consumer to 100 per run
15:36 ejegg: updated payments-wiki from 896f16a to 9da8482
15:36 gehel: upgrading elasticsearch plugins on elasticsearch eqiad, including cold restart of the cluster - T173231
15:33 ejegg: limited donations queue consumer to 200 per run
14:03 akosiaris@tin: Synchronized wmf-config/InitialiseSettings.php: Whitelist wtp10[25-48] for Linter (duration: 00m 49s)
13:42 otto@tin: Synchronized /srv/mediawiki-staging/wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 00m 49s)
13:24 ema: begin rolling codfw varnish-be restarts, 20m spacing
13:20 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 3/3) (duration: 00m 48s)
13:18 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 2/3) (duration: 00m 49s)
13:17 dereckson@tin: Synchronized tests/cirrusTest.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 1/3, no-op in prod) (duration: 00m 49s)
12:51 ema: restarted varnish-be on cp4028
12:28 gehel: upgrading nodejs to 6.11 on maps-test2003 for testing - T171707
12:19 ema: restarted varnish-be on cp4027
12:18 moritzm: uploaded apache 2.4.10-10+deb8u11+wmf1 to jessie-wikimedia (rebuild of latest security update with our local patches on top)
12:08 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Enable statement usage tracking on elwiki (T151717) (duration: 00m 49s)
11:58 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940 (duration: 07m 20s)
11:56 ema: restarted varnish-be on cp4018
11:51 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940
11:49 jynus: stopping replication on db1101 for faster repartition work T176311
11:45 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1101 (duration: 00m 58s)
11:34 ema: restarted varnish-be on cp4010
11:25 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: (no justification provided) (duration: 00m 02s)
11:25 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: (no justification provided)
11:03 moritzm: rolling out new debdeploy release across the fleet
11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 19s)
11:02 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
11:02 ema: restarted varnish-be on cp4008
11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
11:01 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 03s)
11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
11:00 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
11:00 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
09:55 bblack: restarted varnish-be on cp3042
09:29 bblack: restarted varnish-be on cp1052
09:23 ema: restarted varnish-be on cp3043
09:21 ema: restarted varnish-be on cp3041
09:19 ema: restarted varnish-be on cp3032
09:14 ema: restarted varnish-be on cp3040
08:42 bblack: restarted varnish-be on cp1053
08:41 bblack: restarted varnish-be on cp1068
08:40 bblack: restarted varnish-be on cp1054
08:40 bblack: restarted varnish-be on cp1065
07:11 ema: cp3030 varnish-be restart due to 503s
03:16 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 20 03:16:25 UTC 2017 (duration 7m 31s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 57s)
02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 06s)
00:55 bblack: cp1066 - backend restart, mailbox lag, text
00:54 bblack: cp4016 - backend restart, mailbox lag, text

2017-09-19

23:57 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 49s)
23:56 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 48s)
23:46 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 48s)
23:45 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 50s)
23:39 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
23:36 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
23:29 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 49s)
23:27 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 50s)
23:17 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable jQuery 3 on meta.wikimedia.org T124742 (duration: 00m 50s)
22:58 eileen: manually running Omnirecipient.load from 2017-09-18 with accidentally low throttle setting
22:22 mutante: gerrit2001 - stopped gerrit with init.d script. moved init.d script to /root, started with systemctl
22:21 eileen: update civicrm from 9677cf7 to 0fb4676
22:15 mutante: gerrit2001 /bin/sh /var/lib/gerrit2/review_site/bin/gerrit.sh start
22:02 mutante: gerrit2001 - systemctl start gerrit
21:51 volans: disabled puppet on labpuppetmaster1001, testing cumin T175712
21:27 MaxSem: on terbium: mwscript populateIpChanges.php --wiki=enwikivoyage --force
21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
21:22 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,dc=eqiad,name=cp1066.*
21:18 MaxSem: re-ran populateIpChanges.php on group0 wikis
21:15 maxsem@tin: Synchronized php-1.30.0-wmf.19/maintenance/populateIpChanges.php: https://gerrit.wikimedia.org/r/#/c/379057/1 (duration: 00m 49s)
21:06 ema: cp4009: varnish-be restart, 503 spike
20:04 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4013.ulsfo.wmnet
20:03 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
19:50 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4005.ulsfo.wmnet
19:49 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
19:17 ejegg: updated payments-wiki from 26b16ea to 896f16a
19:08 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.19
18:52 gehel: upgrading nodejs to 6.11 on maps-test2004 for testing - T171707
18:35 urandom: T160570, T169940: Upgrade restbase-ng environment to Cassandra 3.11.0-wmf5
18:26 arlolra: Updated Parsoid to 05a0965 (T122965, T175101, T173384, T173061, T172896, T174977, T159894)
18:17 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984) (duration: 00m 48s)
18:15 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984) (duration: 00m 49s)
18:08 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984) (duration: 00m 50s)
18:07 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984) (duration: 00m 50s)
18:00 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965 (duration: 12m 26s)
17:54 ebernhardson: T173464 start cirrussearch reindex of all wikis starting with zh
17:48 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
17:43 jynus: stopping db2010 and cloning to db2078
17:16 demon@tin: Finished scap: bootstrap wmf.19 (duration: 31m 04s)
16:45 demon@tin: Started scap: bootstrap wmf.19
16:44 demon@tin: scap aborted: bootstrap wmf.19 (duration: 01m 52s)
16:42 demon@tin: Started scap: bootstrap wmf.19
16:38 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1018 (duration: 00m 51s)
16:37 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1018 (duration: 01m 01s)
16:34 demon@tin: Pruned MediaWiki: 1.30.0-wmf.17 [keeping static files] (duration: 01m 18s)
16:31 demon@tin: Pruned MediaWiki: 1.30.0-wmf.14 (duration: 03m 15s)
16:29 jynus: disable puppet on db1009
16:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Trying that again, got an error the first time (duration: 00m 52s)
16:26 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RCFilters by default on cawiki, frwiki and hewiki (T157642) (duration: 00m 59s)
16:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable structured filters on watchlist on all wikis (duration: 00m 45s)
16:11 ema: pybal 1.14.0 uploaded to apt.w.o
15:54 catrope@tin: Finished scap: core, GuidedTour and WikimediaMessages patches for T176191, T167262 and T175765 (duration: 20m 32s)
15:33 catrope@tin: Started scap: core, GuidedTour and WikimediaMessages patches for T176191, T167262 and T175765
15:22 jynus: enabling firewall on db1020 (m2-master)
15:01 ema: restart varnish-be on cp1055, 503 spike
14:55 jynus: deploying firewall to m5 hosts
14:39 jynus: disabling puppet on all misc active mysql hosts
13:35 zeljkof: EU SWAT finished
13:32 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 00m 45s)
13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 55s)
13:19 gehel: upgrading elasticsearch plugins on relforge, including cold restart of the cluster - T173231
13:17 gehel: upgrading elasticsearch plugins on elasticsearch codfw, including cold restart of the cluster - T173231
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 56s)
13:04 gehel: upgrading elasticsearch plugins on elastic2001 - T173231
12:55 moritzm: rebooting kubestage1001
11:25 moritzm: uploaded debdeploy 0.99.1 to apt.wikimedia.org (for trusty, jessie and stretch)
11:23 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 02m 42s)
11:20 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
10:53 volans: restarted nagios-nrpe-server on stat1005
10:14 jynus: shutting down mysql on db1018
09:42 moritzm: installing libxml2 security updates on trusty (Debian already fixed)
09:40 elukey: powercycle analytics1062 - no ssh, console com2 frozen
09:34 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 50s)
09:30 ema: cp1062 - varnish-backend-restart, mbox lag
08:40 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 52s)
08:39 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
08:09 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 13s)
08:09 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
07:58 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 00m 37s)
07:58 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
07:50 moritzm: installing bind9 regression update on trusty servers
07:35 kartik@tin: (no justification provided)
07:35 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 01m 04s)
07:34 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
02:28 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 19 02:28:56 UTC 2017 (duration 7m 6s)
02:21 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 04s)
01:52 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor newline nitpick. I promise that's it (duration: 00m 46s)
01:41 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor spacing nitpick, that's it for today (duration: 00m 45s)
01:17 demon@tin: Synchronized scap/plugins/updatewikiversions.py: add new hotness updatewikiversions.py (duration: 00m 45s)
01:15 demon@tin: Synchronized multiversion/: rm old and busted updateWikiversions (duration: 01m 07s)
00:12 foks: Remove 2FA from users Alan and Magicknight94 (retroactive log)
00:10 eileen: update civicrm from 187238f to b6f7dc3

2017-09-18

23:28 tstarling@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: (no justification provided) (duration: 00m 46s)
23:13 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow: SWAT: Only intercept users who can potentially create articles otherwise T176100 (duration: 00m 46s)
22:32 demon@tin: Synchronized wmf-config/CommonSettings.php: Use PrivateSettings.php directly (duration: 00m 46s)
21:57 bawolff: deployed patch T175900
21:18 krinkle@tin: Synchronized php-1.30.0-wmf.18/extensions/NavigationTiming/modules/ext.navigationTiming.js: Unbreak - T176105 (duration: 00m 48s)
21:16 mutante: repeating import of gerrit_2.13.8+git1-wmf.7 on install1002, followed by reprepro copy stretch-wikimedia jessie-wikimedia gerrit to make it available on stretch
21:06 mutante: uploaded gerrit 2.13.8+git1-wmf.7 for jessie-wikimedia on APT.wm.org
20:42 krinkle@tin: Finished deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148 (duration: 03m 23s)
20:38 krinkle@tin: Started deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148
20:17 arlolra@tin: (no justification provided)
20:15 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965 (duration: 01m 53s)
20:13 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
19:29 XioNoX: restarting pfw3-codfw for software upgrade
19:10 urandom: T160570: Upgrading restbase-dev100[5-6].eqiad.wmnet to Cassandra 3.11.0-wmf5
19:07 smalyshev@tin: Finished deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf (duration: 01m 20s)
19:05 smalyshev@tin: Started deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf
19:03 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4 (duration: 02m 36s)
19:02 no_justification: gerrit: restarting service, back in a moment
19:00 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4
18:56 urandom: T160570: Upgrading restbase-dev1004.eqiad.wmnet to Cassandra 3.11.0-wmf5 (canary)
18:46 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3 (duration: 02m 25s)
18:44 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3
18:37 smalyshev@tin: (no justification provided)
18:36 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 02m 26s)
18:34 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
18:20 maxsem@tin: Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/378357/ https://gerrit.wikimedia.org/r/#/c/376791/ (duration: 00m 48s)
17:42 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 00m 21s)
17:42 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
17:26 demon@tin: Synchronized wmf-config/ProductionServices.php: no-op/comment fix (duration: 00m 45s)
16:46 jynus: shuting down db1100 T175973
16:40 reedy@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: perf improvements T175962 (duration: 00m 46s)
16:05 bblack: cp1063 - backend restart, mailbox lag
15:38 bblack: cp1072 - varnish backend restart, mailbox lag
15:28 bblack: cp1074 - backend restart for mailbox lag
15:25 bblack: cp1050 - backend restart for mailbox lahg
14:27 reedy@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: T175444 (duration: 00m 47s)
13:29 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property (duration: 00m 29s)
13:28 ppchelko@tin: Started deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property
13:25 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Meta(Talk)Namespace configuration for be.wiktionary - T175950 (duration: 00m 46s)
13:18 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: Stop A/B test on enwiki and dewiki - T176068 (duration: 00m 45s)
13:10 hashar@tin: Synchronized php-1.30.0-wmf.18/extensions/BetaFeatures: Temporary log all the executions of the update job. - T175637 (duration: 00m 46s)
13:09 hashar@tin: Synchronized wmf-config: New 'abusefilter-helper' configuration for en.wikipedia - T175684 (duration: 00m 49s)
12:59 moritzm: installing libxen updates
12:36 moritzm: installing tomcat8 security updates
10:40 ema: cp1099 - backend restart, mailbox lag
06:42 moritzm: installing freexl security updates
06:18 TimStarling: testing EtcdConfig in beta cluster
05:40 tstarling@tin: Synchronized wmf-config: just for consistency, should have no effect (gerrit 375108) (duration: 00m 49s)
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 18 02:36:50 UTC 2017 (duration 7m 6s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 04s)

2017-09-17

12:49 _joe_: taking a full debug dump of mw1288 after depooling it
12:40 _joe_: restarting hhvm on some api appservers, to ease the cpu overlock
09:46 elukey: restart varnish backend on cp1052 - recurrent mailbox lag
08:11 elukey: restart varnish backend on cp1053 - recurrent mailbox lag

2017-09-16

22:05 godog: compress older otrs directories to reclaim inodes - T171490
16:53 elukey: restart varnish-backend on cp1073 (cache upload) for mailbox lag
15:40 andrewbogott: rebooting labvirt1017
15:34 andrewbogott: rebooting labvirt1015 for T176044

2017-09-15

21:50 mutante: contint2001 - systemctl start zuul
21:47 mutante: contint1001 - tmp disable puppet - contint2001 - test zuul unit file (gerrit 359016)
21:43 demon@tin: Finished deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though (duration: 00m 08s)
21:43 demon@tin: Started deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though
15:46 bblack: cp1049 - backend restart, mailbox lag
15:40 bblack: cp1099 - backend restart, mailbox lag
15:38 bblack: cp1072 - backend restart, mailbox lag
15:21 _joe_: uploaded td-agent-bit (fluentd-bit) to wikimedia-stretch T175527
13:13 moritzm: pruned hhvm bytecode cache on mw2256, led to log spam for an undefined variable after being out of service for hardware maintenance
13:10 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
13:10 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 2/2 (duration: 00m 45s)
13:08 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 1/2 (duration: 00m 47s)
11:22 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: mw2256.codfw.wmnet
09:10 moritzm: installing tcpdump security updates on trusty hosts (Debian systems already fixed)
08:54 moritzm: installing libbluetooth updates
08:03 jynus: dropping mariadb analytics users on db1031, db1029
07:49 Amir1: running maintenance/updateCollation.php --force on Persian (fa) wikis (T173601)
07:39 gehel: shutting down and masking elasticsearch on elastic1020 - T175951
07:35 gehel: depooling elastic1020 - T175951
07:28 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
00:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 46s)

2017-09-14

23:51 mutante: phabricator - tested gerrit 354247 before merging using apache-fast-test from tin -restarted apache
23:37 catrope@tin: Synchronized wmf-config/: VRS config for Electron (T175868) (duration: 00m 47s)
23:34 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/lib/ve/src/ui/styles/ve.ui.TableLineContext.css: T169389 (duration: 00m 45s)
23:33 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/ContentTranslation/modules/widgets/translator/ext.cx.translator.js: Fix JS error in CX (duration: 00m 46s)
23:29 catrope@tin: Synchronized wmf-config/CommonSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934) (duration: 00m 45s)
23:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934) (duration: 00m 45s)
23:22 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Use RemexHtml on mediawikiwiki and testwiki (T175095) (duration: 00m 46s)
23:19 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org
23:18 mutante: acamar is back up and running - no failed units
23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats to remove accountcreator on mediawikiwiki (T175903) (duration: 00m 46s)
23:12 mutante: acamar - staged BIOS upgrade in progress
23:09 mutante: acamar - done with upgrade - rebooting - it's depooled and removed from resolv.conf - T162850
22:56 mutante: depooled acamar (codfw dns recursor) for BIOS upgrade - installing firmware
22:56 dzahn@neodymium: conftool action : set/pooled=no; selector: name=acamar.*
22:55 maxsem@tin: Synchronized wmf-config/: Labs only: https://gerrit.wikimedia.org/r/378184 https://gerrit.wikimedia.org/r/378187 (duration: 00m 47s)
22:15 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: ACTRIAL going live: https://gerrit.wikimedia.org/r/#/c/378104/ (duration: 00m 46s)
21:22 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/378125/ (duration: 00m 47s)
20:34 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.userpage.icons/userpage.svg: I95d76339 (duration: 00m 45s)
20:18 demon@tin: Synchronized php-1.30.0-wmf.18/includes/api/ApiFeedWatchlist.php: Ibea5bd88 (duration: 00m 46s)
20:04 demon@tin: Synchronized php-1.30.0-wmf.18/extensions/OAuth/backend/MWOAuthDAO.php: I785230df (duration: 00m 46s)
20:02 ppchelko@tin: Finished deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl (duration: 00m 30s)
20:02 ppchelko@tin: Started deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl
19:58 dcausse: banning elastic1020 to see if T175951 is caused by mixed versions of the ltr plugin
19:36 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 46s)
19:31 demon@tin: Synchronized php-1.30.0-wmf.17/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 45s)
19:07 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.18
18:58 tgr@tin: Synchronized wmf-config/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 48s)
18:57 tgr@tin: Synchronized private/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 49s)
18:29 twentyafterfour: Phabricator: Deploying hotfix for T175942
18:10 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Configure enwiki to use CirrusSearch MLR by default T175772 (duration: 00m 50s)
18:03 Jamesofur: disabled oauth validation on metawiki/SUL for Teles after verification
17:57 bblack: cp2001 backend restart, mailbox lag
17:56 bblack: cp1074 backend restart, mailbox lag
17:18 eevans@tin: Started restart [electron-render/deploy@8dd5f13]: (no justification provided)
17:16 gehel@tin: Finished deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3 (duration: 00m 04s)
17:16 gehel@tin: Started deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3
16:59 krinkle@tin: Synchronized php-1.30.0-wmf.17/extensions/NavigationTiming: T104902 (duration: 01m 00s)
15:51 bblack: repool cp1063 (after wiping varnish storage + restarting all 3x main service daemons)
15:51 bblack: cp1064 backend restart, mailbox lag
15:46 bblack: depool cp1063 (upload eqiad) - seems to be having some iowait issues?
14:07 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/CheckUser/specials/SpecialCheckUser.php: Fix Special cu for ip ranges (temp) (T175898) (duration: 00m 49s)
13:34 dcausse@tin: Synchronized wmf-config/CirrusSearch-production.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 49s)
13:33 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 48s)
13:31 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 50s)
11:14 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1049, pool db1100 (duration: 00m 49s)
11:12 akosiaris: upload apertium-crh-tur_0.2.0~r81866-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1049 (duration: 00m 50s)
09:51 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp1025.eqiad.wmnet
09:43 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
09:43 akosiaris@puppetmaster1001: conftool action : set/weight=1; selector: name=wtp1025.eqiad.wmnet
09:41 gehel: re-enabling puppet on all cassandra nodes - T171704
09:41 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 09m 30s)
09:32 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
09:02 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 02m 25s)
09:00 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
08:13 gehel: merging cassandra refactoring for puppet - T171704
02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 14 02:55:10 UTC 2017 (duration 7m 28s)
02:47 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 58s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 09m 27s)
00:46 MaxSem: restarted populateIpChanges to use updated code
00:26 twentyafterfour: everything appears to be up and running. Phabricator upgrade complete.
00:23 twentyafterfour: running puppet on phab1001 to restore files which get lost by scap deploy
00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 01s)
00:21 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 25s)
00:20 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
00:18 twentyafterfour: Begin phabricator upgrade #phab-2017-09-13

2017-09-13

23:48 dereckson@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: Improve ip_changes update script to insert rows in batches (Gerrit:377918) (duration: 00m 49s)
23:29 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/client: Split page set before constructing InjectRCRecordsJob (T175316) (duration: 00m 57s)
23:28 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Echo/modules/api/mw.echo.api.APIHandler.js: (no justification provided) (duration: 00m 48s)
23:26 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Set $wgScoreSafeMode to false (T174413) (duration: 00m 50s)
22:06 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/EventBus/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
22:03 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
21:55 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377874/ (duration: 00m 47s)
21:23 ejegg: updated payments-wiki from ed2e481 to 26b16ea
grrrr: populating ip_changes on group1 wikis
20:43 bblack: re-routed ulsfo->eqiad around possible codfw damage as well - https://gerrit.wikimedia.org/r/#/c/377871/
20:34 bblack: deployed https://gerrit.wikimedia.org/r/#/c/377845/ (no gerrit bot, jenkins lint failing, etc) due to possible codfw routing issues
19:39 andrewbogott: rebooting labtestvirt2001
19:35 ppchelko@tin: Finished deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule (duration: 01m 18s)
19:34 ppchelko@tin: Started deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule
19:21 urandom: T172384: Upgrading restbase-dev100[5-6] to Cassandra 3.11.0-wmf4
19:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.18
19:13 demon@tin: Synchronized php: symlink thingie (duration: 00m 49s)
19:11 urandom: T172384: Upgrading restbase-dev1004 to Cassandra 3.11.0-wmf4 (canary)
18:34 niharika29@tin: Synchronized static/images/project-logos/: Change logo for huwiktionary T175483 (duration: 00m 49s)
18:27 mutante: gerrit2002 - re-enabled puppet - had forgotten to enable it again after recently tryning to test the gerrit->logstash change
18:26 niharika29@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/: Make ve.dm.Change part of core module T175828 (duration: 00m 50s)
18:25 niharika29@tin: Synchronized docroot/mediawiki/ontology/: Add setup for https://www.mediawiki.org/ontology T171807 (duration: 00m 49s)
18:12 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Timeless on frwiki T154371 (duration: 00m 50s)
17:55 gwicke: rolling restart of pdfrender service in equiad after hang T174916 T172815
16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1001.eqiad.wmnet
16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1002.eqiad.wmnet
16:28 gehel: starting decommissioning of wdqs100[12] - T175595
16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1002.eqiad.wmnet
16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1001.eqiad.wmnet
16:22 andrewbogott: rebooting labtestvirt2002
16:08 ottomata: restarting druid-brokers with increase in query cache size
15:23 demon@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: I2ecb6030 (duration: 00m 49s)
15:22 demon@tin: Synchronized php-1.30.0-wmf.18/includes/diff/DifferenceEngine.php: I2ecb6030 (duration: 00m 49s)
15:21 demon@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: I2ecb6030 (duration: 00m 49s)
15:20 demon@tin: Synchronized php-1.30.0-wmf.17/includes/diff/DifferenceEngine.php: I2ecb6030 (duration: 00m 49s)
15:09 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/: backporting I13051038 (duration: 00m 52s)
14:31 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=logstash
14:31 gehel: pooling new logstash servers (logstash100[7-9]) - T175045
14:23 mobrovac@tin: Finished deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210 (duration: 00m 33s)
14:22 mobrovac@tin: Started deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210
13:45 akosiaris: upload apertium-crh_0.1.0~r81872-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
13:30 urandom: T169936: Converting RESTBase dev environment to size-tiered compaction
13:29 elukey: update puppet compiler's facts via ./modules/puppet_compiler/files/compiler-update-facts
13:23 gehel: initial installation of logstash100[7-9] - T175045
13:21 zeljkof: EU SWAT finished
13:13 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Temporary lift account creation limits for WM United Kingdom workshop (T175700) (duration: 00m 49s)
12:14 moritzm: rebooting mc1015 (spare) for some tests
12:07 jynus: stopping wikidata rebuild maintenance script on terbium
12:07 bblack: cp1049 - backend restart, mailbox lag
11:57 jynus@tin: Synchronized wmf-config/db-eqiad.php: Set db1097 as the main api server for s4, Remove regular load from db1070 (duration: 00m 56s)
11:39 jynus: disabling semi-sync replication on the largest s5 database servers
11:03 ariel@tin: Finished deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run (duration: 00m 02s)
11:03 ariel@tin: Started deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run
11:02 joal@tin: Finished deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches) (duration: 03m 12s)
10:58 joal@tin: Started deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches)
10:52 volans: reboot lvs3001
10:06 akosiaris: upload apertium-cat-srd_0.9.0~r82238-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
10:06 akosiaris: upload apertium-srd-ita_0.9.5~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:36 moritzm: installing bind updates from Debian stable/oldstable SUA update (client tools only)
09:25 akosiaris: upload apertium-srd_0.10.0~r82237-1+wmf1_amd64.changes to apt.wikimedia.org/jessie-wikimedia/main T174988
09:24 akosiaris: upload apertium-tur_0.1.0~r81882-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:23 akosiaris: upload apertium-ita_0.10.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:23 akosiaris: upload apertium-cat_2.3.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:05 akosiaris: upload apertium_3.4.2~r68466-2+wmf2 to apt.wikimedia.org/jessie-wikimedia
08:55 elukey: restart varnish backend on cp1072 (upload) - mailbox expiry lag
08:24 moritzm: installing emacs security updates
07:56 akosiaris: bounce varnish on cp1062, mailbox lag
07:47 moritzm: rolling out hhvm-luasandbox 2.0.14 to the remaining hosts in eqiad (along with HHVM restarts) (T173705)
07:08 moritzm: installing tcpdump security updates (fixing 90 CVE IDs...)
05:18 kartik@tin: Finished deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36 (duration: 02m 56s)
05:15 kartik@tin: Started deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36
03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 13 03:12:46 UTC 2017 (duration 7m 30s)
03:05 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 16m 28s)
03:03 jynus: dropping old users from phabricator database
03:00 eileen: update civicrm from ee7dda3 to 187238f
02:41 demon@tin: Synchronized wmf-config/mobile.php: rm unused config var (duration: 00m 46s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 31s)
01:19 jynus: stopping mariadb @ db1048 to clone it to db1059
00:11 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Special:PageLanguage on mul.wikisource Fix correct wiki name for wgPageLanguageUseDB on www.wikisource.org T175622 (duration: 00m 49s)

2017-09-12

23:53 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Do not deploy Timeless on fr.wiktionary for now" (duration: 00m 49s)
23:35 thcipriani@tin: Synchronized wmf-config: SWAT: Enable Flow on Newsletter_talk on mediawiki.org T175502 (duration: 00m 52s)
22:17 chasemp: labvirt1018:~# /sbin/reboot
22:00 bsitzmann@tin: Finished deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 (T174707 T175305) (duration: 04m 28s)
21:55 bsitzmann@tin: Started deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 (T174707 T175305)
21:49 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377527/2 T175725 (duration: 00m 49s)
21:37 MaxSem: Populating ip_changes in group0 wikis
21:33 ejegg: changed contact de-duplication duty cycle to 25 minutes
21:05 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.18
20:33 demon@tin: Finished scap: bootstrap wmf.18 (for real this time) (duration: 26m 23s)
20:06 demon@tin: Started scap: bootstrap wmf.18 (for real this time)
19:53 demon@tin: Finished scap: bootstrap wmf.18 (duration: 08m 14s)
19:45 demon@tin: Started scap: bootstrap wmf.18
19:05 papaul: maintenance complete on mw2256
17:51 bblack: cp3034 - reset fetch_maxchunksize to 0.25G (default) for fe+be
17:40 cwd: turned on dedupe
16:55 niharika29@tin: Finished deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154 (duration: 00m 01s)
16:55 niharika29@tin: Started deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154
16:40 ejegg: updated CiviCRM from 63288ae to ee7dda3
16:35 niharika29@tin: Finished deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134 (duration: 00m 03s)
16:35 niharika29@tin: Started deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134
16:26 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2100.codfw.wmnet
16:18 ejegg: updated SmashPig from dce4f0a to dc62203910
15:43 _joe_: restarted ircecho on einsteinium
15:35 gehel: restart elasticsearch on relforge (last check of new plugin deployment) - T158560
15:22 bblack: manually setting fetch_maxchunksize=16MB via varnishadm on cp3034 fe+be instances
15:06 chasemp: disable puppet for lab* things for designate refactor rolleout (delayed a few minutes for a meeting)
15:04 moritzm: uploaded php5.5 5.5.38 to component/ci of jessie-wikimedia (for use in CI)
14:53 papaul: shutting down mw2256 for maintenance
14:43 elukey: add kafka-jumbo IPs to the kafka term of the analytics-in4 filter on cr1/cr2 eqiad
14:42 ema: cp1074 - backend restart, mailbox lag
14:39 bblack: manually setting fetch_maxchunksize=64MB via varnishadm on cp3034 fe+be instances
14:19 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1059 (duration: 00m 44s)
14:18 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1059 (duration: 00m 45s)
14:10 gehel: restarting elasticsearch on elastic1020 to validate new plugin deployment - T158560
14:07 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/eqiad - T158560
14:06 volans: testing reimage of mw2100 - T166300
14:06 gehel: restarting elasticsearch on elastic2001 to validate new plugin deployment - T158560
14:04 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059, pool db1097 with low load (duration: 00m 45s)
14:02 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/codfw - T158560
13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1005.eqiad.wmnet
13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1004.eqiad.wmnet
13:37 gehel: pooling wdqs100[45] - T171210
13:31 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/CentralAuth/includes/api/ApiSetGlobalAccountStatus.php: API: Unbreak setglobalaccountstatus locked=lock - T175462 (duration: 00m 44s)
13:26 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner for Russian Wikimedia chapter wiki - T175356 (duration: 00m 45s)
13:25 hashar@tin: Synchronized php-1.30.0-wmf.17/includes/changes/ChangesListBooleanFilter.php: WLFilters: Respect default values - T174725 (duration: 00m 45s)
13:21 hashar@tin: Synchronized wmf-config/CommonSettings.php: Add Extension:Newsletter permissions to CommonSettings (duration: 00m 45s)
13:18 hashar@tin: Synchronized wmf-config/throttle.php: Lift account creation restrictions for WM Taiwan 10th anniversary - T175534 (duration: 00m 45s)
13:15 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/Wikidata/extensions/Wikibase/view/resources/jquery/wikibase/themes/default/jquery.wikibase.entityselector.css: Remove red highlighting on all entity selectors - T175525 (duration: 00m 45s)
13:12 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Reduce wikiPageUpdaterDbBatchSize to 20 - T173710 (duration: 00m 45s)
09:33 moritzm: upgrading deployment servers to HHVM-Luasandbox 2.0.14
09:32 moritzm: upgrading mw2* to HHVM-Luasandbox 2.0.14
09:25 moritzm: upgrading mw1293-mw1295 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
09:24 moritzm: upgrading mw1293-mw1205 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
08:38 gehel: deploying elasticsearch plugins on relforge - T158560 (wrong ticket number before)
08:34 gehel: deploying elasticsearch plugins on relforge - T175159
08:28 moritzm: upgrading mw1189-mw1208 (API servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
07:35 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with full weight (duration: 00m 45s)
07:15 moritzm: upgrading mw1180-mw1188, mw1209-1220 (app servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 12 02:29:20 UTC 2017 (duration 6m 45s)
02:22 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 07m 31s)
02:10 bblack: cp1050 - backend restart, mailbox lag
02:08 bblack: cp1099 - backend restart, mailbox lagh
00:48 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 18s)
00:48 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart

2017-09-11

23:06 mutante: jenkins-bot says: java.lang.OutofMemoryError: PermGen space
22:30 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
22:30 dereckson@tin: Synchronized wmf-config/CommonSettings-labs.php: Fix MediaWiki\Auth\AbstractPreAuthenticationProvider case (Gerrit:377281, no-op in prod) (duration: 00m 45s)
21:45 awight@tin: Finished deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster (duration: 13m 47s)
21:41 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.*
21:31 awight@tin: Started deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster
21:03 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.*
20:13 demon@tin: Pruned MediaWiki: 1.30.0-wmf.13 (duration: 02m 49s)
19:11 chasemp: adjust nodepool rate to 10
19:08 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with low load (duration: 00m 46s)
19:01 mutante: restbase-dev1004:/srv# mv /home/eevans/cassandra-1502141526-pid13410.hprof /srv/eevans/ to free disk space
18:52 mutante: restbase - java[12418]: Error: Could not find or load main class org.wikimedia.cassandra.metrics.service.Service
18:40 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable responsive reference columns on wiktionaries, wikivoyages and enwiki https://gerrit.wikimedia.org/r/#/c/376573/, https://gerrit.wikimedia.org/r/#/c/371630/ (duration: 00m 46s)
18:33 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been complete and tested good T174475
18:29 chasemp: bump nodepool rate to 15 and max-servers to 18 (no restart as it picks up config periodically)
18:29 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been started, awaiting completion T174475
18:24 robh: updating firmware on scs-a1-codfw and scs-c1-codfw
18:10 robh: scs-ulsfo has successfully updated firmware T174475
18:03 robh: upgrading firmware on scs-ulsfo
17:09 gehel@tin: Finished deploy [wdqs/wdqs@177d20a]: (no justification provided) (duration: 02m 39s)
17:07 gehel: wdqs weekly deployment (logging and throttling fixes)
17:06 gehel@tin: Started deploy [wdqs/wdqs@177d20a]: (no justification provided)
15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.*
15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4028.*
14:40 godog: roll-restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/377264/ and upgrade to 1.4 - T173580 T174997
14:24 hashar: tin.eqiad.wmnet: reverted patch "WLFilters: Respect default values" that got merged but left undeployed
14:23 hashar: tin.eqiad.wmnet: reverted all four mediawiki-config that got merged but left undeployed and rebased the workspace
14:05 hashar: European SWAT cancelled due to deploy infrastructure issue
13:55 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
13:54 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
13:50 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
13:43 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
13:40 hashar@tin: Synchronized static/images/project-logos/huwiktionary.png: Change logo for huwiktonary - T175483 (duration: 00m 28s)
11:43 moritzm: installing file/libmagic security updates (stretch-specific, does not affect older distros)
10:30 jynus: restarting mysql @ labsdb1001
09:40 godog: roll-restart thumbor to apply changes for gif max animated area - T173580
09:39 mobrovac@tin: Finished deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281 (duration: 00m 20s)
09:39 mobrovac@tin: Started deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281
08:44 mobrovac@tin: Finished deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 - T174992 (duration: 00m 07s)
08:44 mobrovac@tin: Started deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 - T174992
08:16 volans: reimage script tests continue on mc100[1-2] T166300
07:59 moritzm: installing systemd updates from stretch point update
07:41 moritzm: installing remaining libonig security updates
02:41 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 11 02:41:43 UTC 2017 (duration 7m 0s)
02:34 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 10s)

2017-09-10

21:00 bawolff: disabled Jforrester's gerrit account and flush sessions
13:41 elukey: restart Varnish backend on cp1073 (cache::upload) for mailbox expiry lag
13:13 elukey: restart cp1053's varnish backend for mailbox expiry lag and 503s - T175473
00:37 chasemp: labnodepool1001:~# service nodepool restart
00:36 bd808: wmcs: purged rabbitmq ceilometer_notifications.* queues

2017-09-09

23:42 chasemp: labnodepool1001:~# service nodepool start
23:06 chasemp: freeswap && labcontrol1001:/home/rush# service rabbitmq-server restart
22:59 chasemp: restart nova-api and nova-network
22:49 chasemp: labnodepool1001:~# service nodepool stop
22:44 madhuvishy: Running service rabbitmq-server restart on labcontrol1001
20:41 legoktm@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: Fix issues related to comments and files in UI and API - T175443 T175444 (duration: 00m 46s)
01:12 ebernhardson@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: T171740: Reduce annoyance of survey by enforcing minimum 2 days between showing survey to same browser (duration: 00m 46s)

2017-09-08

23:46 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: T171740: Fix inverted sampling rates for human relevance survey (duration: 00m 47s)
23:38 mutante: netmon1002 - terminated unused screen session
23:09 mutante: phab1001, phab2001 - terminating 3 unused screen processes that were from iridium migration / data sync
21:54 mutante: gerrit2001 - restarting gerrit to test logstash config change (while not touching "live" gerrit)
20:24 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: bugs and stuff (duration: 00m 54s)
20:14 mutante: heze - rebooting for firmware upgrade
20:10 mutante: heze (bacula storage) - installing BIOS upgrade (T162850)
20:10 demon@tin: Synchronized scap/scap.cfg: drop git_repo for now, T175041 (duration: 00m 46s)
19:43 mutante: zosma.codfw.wmnet - delete salt key, puppet node clean, puppet node deactivate, remove from Icinga,... (T138650)
19:37 mutante: removing ganeti instance 'zosma' on ganeti2001 (T138650)
18:17 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: no-op (duration: 00m 46s)
16:04 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: T175338, followup (duration: 00m 46s)
16:03 demon@tin: Synchronized php-1.30.0-wmf.17/includes/revisiondelete/RevDelLogList.php: typofix (duration: 00m 46s)
15:53 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4027.*
15:47 ejegg: increased cURL timeout for PayPal IPN confirmation to 14 sec
15:07 akosiaris@tin: Finished deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config (duration: 00m 04s)
15:06 akosiaris@tin: Started deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config
15:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4021.*
14:58 volans: testing wmf-auto-reimage also on mc1002 T166300 T164341
12:36 _joe_: restarting a check of the md2 devices on conf200* T162013 after reducing sync speed
12:36 _joe_: reduced raid resync max speed of raid devices on conf200* to 20000
10:35 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/ProofreadPage: Restore page status buttons - T175304 (duration: 00m 50s)
10:12 _joe_: stopped all md devices syncs on conf200* machines
10:09 _joe_: etcd cluster lost consensus T162013
10:07 volans: testing wmf-auto-reimage on mc1001 T166300 T164341
09:58 _joe_: running the same command on conf2002 too
09:48 _joe_: running `/usr/share/mdadm/checkarray --cron --all --idle --quiet` on conf2001 and conf2003 trying to reproduce consensus issues in T162013
09:25 moritzm: restarting apache on tegmen/einsteinium to pick up security update
09:09 moritzm: installing libonig security updates
09:08 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940 (duration: 01m 01s)
09:07 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940
09:01 moritzm: installing remaining gnutls updates from jessie point release
08:13 moritzm: installing libgd2 security updates
08:11 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster - T169940
08:11 mobrovac: restbase enabled back puppet in the dev cluster - T169940
07:53 moritzm: installing ruby2.3 security updates
07:32 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided) (duration: 07m 02s)
07:25 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided)
07:21 mobrovac@tin: Finished deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided) (duration: 00m 11s)
07:21 mobrovac@tin: Started deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided)
00:42 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: fix comment stuff (duration: 00m 46s)

2017-09-07

23:42 bblack: cp1066 - depooled all traffic services, seems to have some bug causing 503 spikes
23:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.17
23:15 XenoRyet: Updated Smashpig from 8eb98c1 to dce4f0a
22:41 niharika29@tin: Synchronized php-1.30.0-wmf.17/extensions/ArticleCreationWorkflow/: Bump extension version (duration: 00m 49s)
22:03 niharika29@tin: Finished scap: Deploying ArticleCreation Workflow on test2 wiki TT175302 (duration: 23m 00s)
21:40 niharika29@tin: Started scap: Deploying ArticleCreation Workflow on test2 wiki TT175302
21:23 demon@tin: Synchronized php-1.30.0-wmf.17/includes/api/ApiQueryRecentChanges.php: T175307 (duration: 00m 49s)
21:01 urandom: T169940: Disabling puppet and restbase service in RESTBase dev
21:00 urandom: T169940: Disabling changeprop in RESTBase dev environment
19:05 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/MobileFrontend/includes/specials/SpecialMobileHistory.php: T175161 (duration: 00m 49s)
18:48 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.17
18:47 demon@tin: Synchronized php: symlink update -> wmf.17 (duration: 00m 48s)
18:39 legoktm: restarted script to reparse all pages in parsoid for Linter (python3 parsoid_reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php) - T161556
17:55 ejegg: disabled queue consumers for data cleaning job
17:52 ejegg: updated CiviCRM from c1ece1e to 63288ae
17:35 otto@tin: Finished deploy [analytics/refinery@6b60f2c]: (no justification provided) (duration: 03m 23s)
17:32 otto@tin: Started deploy [analytics/refinery@6b60f2c]: (no justification provided)
17:31 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 00m 23s)
17:31 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
17:23 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 03m 25s)
17:20 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
16:13 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Remove Extension:RelatedSites from zhwikivoyage (duration: 00m 50s)
15:32 mobrovac@tin: Finished deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848 (duration: 06m 18s)
15:25 mobrovac@tin: Started deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848
14:45 _joe_: manually running /usr/share/mdadm/checkarray --cron --all --idle --quiet on conf2001, trying to reproduce consensus issues in T162013
14:40 jynus: rebooting and upgrading es1019
13:54 godog: powercycle ms-be2023 - load through the roof and no login possible
13:54 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=jobrunner,name=eqiad
13:14 zeljkof: EU SWAT finished
13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
13:12 zfilipin@tin: Synchronized dblists/wikidataclient.dblist: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
11:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1100 with 0 weight - T172679 (duration: 00m 49s)
09:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 depooled to s5 array - T172679 (duration: 00m 49s)
08:59 ariel@tin: Finished deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps (duration: 00m 02s)
08:58 ariel@tin: Started deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps
08:44 elukey: restart varnish backend on cp1063 - mailbox expiry lag
07:57 elukey: force re-mount of /mnt/hdfs on stat1005
07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 - T172679 (duration: 00m 48s)
07:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1100 - T172679 (duration: 00m 49s)
07:45 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848 (duration: 06m 36s)
07:39 mobrovac@tin: Started deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848
07:35 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint (duration: 01m 43s)
07:33 mobrovac@tin: Started deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint
07:20 oblivian@tin: Synchronized rpc/RunSingleJob.php: Adding the RunSingleJob.php rpc endpoint for jobrunners (duration: 00m 56s)
06:07 eileen: civicrm updated from 73a24b4 to c1ece1e
03:13 eileen: seconds behind=78
03:08 eileen: seconds behind 55
02:52 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 7 02:52:05 UTC 2017 (duration 7m 15s)
02:44 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 06m 15s)
02:43 eileen: restarting groupmember.load after hacking out cache delete calls
02:40 eileen: update civicrm from 764bfe1 to 73a24b4
02:24 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 23s)
02:05 eileen: replag 217 behind
01:56 eileen: replag 197 seconds
01:47 eileen: replag 23 seconds
01:29 eileen: replag 0
01:22 eileen: replag 550 sec behind master
01:13 eileen: replag delay 422
01:12 eileen: job killed
00:49 eileen: 8389 contacts now
00:49 eileen: 7971 contacts so far
00:46 eileen: started Omnigroupmember.load
00:24 twentyafterfour: phabricator update deployed
00:12 twentyafterfour@tin: Finished deploy [phabricator/deployment@c265c1a]: (no justification provided) (duration: 00m 24s)
00:11 twentyafterfour@tin: Started deploy [phabricator/deployment@c265c1a]: (no justification provided)
00:07 twentyafterfour: deploying phabricator update 2017-09-06 https://phabricator.wikimedia.org/project/view/2980/

2017-09-06

23:41 twentyafterfour@tin: Synchronized php-1.30.0-wmf.16/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 to all webservers refs T174387 (duration: 00m 49s)
23:28 twentyafterfour@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 to wmf.17 refs T174387 (duration: 00m 49s)
23:23 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-common.php: CirrusSearch-common.php goes last (duration: 00m 48s)
23:21 twentyafterfour@tin: Synchronized wmf-config/InitialiseSettings.php: Now sync initializesettings (duration: 00m 49s)
23:19 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: sync the added file first (duration: 00m 49s)
23:17 twentyafterfour@tin: scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
23:12 twentyafterfour@tin: Synchronized wmf-config/: roll-back due to huge error rate spike (duration: 00m 51s)
23:10 twentyafterfour@tin: Synchronized wmf-config/: deploy config for CirrusSearch human relevancy survey. Change-Id: I272c69 Bug: T174106 (duration: 00m 52s)
22:18 no_justification: prior deploy was no-op
22:18 demon@tin: Finished deploy [gerrit/gerrit@d4f9a77]: (no justification provided) (duration: 00m 07s)
22:18 demon@tin: Started deploy [gerrit/gerrit@d4f9a77]: (no justification provided)
22:11 bsitzmann@tin: Finished deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808) (duration: 04m 53s)
22:06 bsitzmann@tin: Started deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808)
21:39 niharika29@tin: Synchronized wmf-config/CommonSettings-labs.php: Config for ArticleCreationWorkflow T175054 (duration: 00m 50s)
21:28 ejegg: updated payments-wiki from bf97cd2 to ed2e481
21:24 ejegg: updated CiviCRM from f9c373e to 764bfe1
21:17 ottomata: rebooting kafka-jumbo1004
20:34 arlolra: Updated Parsoid to f9d367ea (T169342)
20:27 arlolra@tin: Finished deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea (duration: 08m 27s)
20:18 arlolra@tin: Started deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea
19:53 ottomata: reimaging kafka-jumbo* with stretch
19:48 chasemp: reboot labvirt1018
18:50 maxsem@tin: Synchronized wmf-config/: just to make sure... (duration: 00m 50s)
18:48 maxsem@tin: Synchronized wmf-config/CommonSettings.php: revert (duration: 00m 49s)
18:33 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/370292/3 (duration: 00m 49s)
18:25 maxsem@tin: Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/370291/2 (duration: 00m 49s)
18:19 maxsem@tin: Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
18:18 maxsem@tin: Synchronized wmf-config/Wikibase-production.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
18:09 maxsem@tin: Synchronized portals: (no justification provided) (duration: 00m 49s)
18:09 ejegg: rolled payments-wiki back to bf97cd2
18:08 maxsem@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 50s)
18:05 ejegg: updated payments-wiki from bf97cd2 to 6911158
17:53 bblack: cp1048 - varnish backend restart for mailbox lag...
17:23 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: code cleanup (duration: 00m 49s)
17:20 mutante: added LDAP user schoenbaechler to WMF group (T175156)
17:16 mutante: modify-ldap-group on terbium is broken
17:02 ejegg: disabled creating CiviMail records for Thank You emails
16:51 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: I284b5aa (duration: 01m 01s)
16:41 chasemp: disable puppet for cloud things to test changes
16:40 bblack: cp1072 - varnish backend restart (mailbox lag)
16:40 bblack: cp1062 - varnish backend restart (mailbox lag)
15:55 ejegg: re-enabled IPN notification at PayPal console (48 minutes ago)
15:53 akosiaris: powercycling lvs3001
15:28 papaul: firmware upgrade on mw2256
15:11 bblack: cp1049 - restart varnish backend (mailbox lag)
15:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool es1019 (duration: 00m 49s)
15:06 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2040 (duration: 00m 49s)
14:17 gehel: wdqs1005 is coming up after a few reimaging issues, expect some icinga noise...
13:19 jynus: restarting and upgrading db2040
13:13 reedy@tin: Synchronized wmf-config/Wikibase-production.php: Move some wikidata config T174962 (duration: 00m 48s)
13:12 reedy@tin: Synchronized wmf-config/Wikibase-labs.php: Move some wikidata config T174962 (duration: 00m 49s)
13:07 reedy@tin: Synchronized wmf-config/throttle.php: Throttle exception T175113 (duration: 00m 49s)
12:08 moritzm: installing libapache2-mod-perl update from jessie 8.9 update
11:41 moritzm: installing gtk+2.0 update from jessie 8.9 update
11:24 elukey: temporarily raise kafka log4j authorizer verbosity to DEBUG on kafka1012 - T173493
11:21 moritzm: installing gnutls update from jessie 8.9 and stretch 9.1 point updates
11:20 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2040 (duration: 00m 49s)
11:20 godog: reimage restbase1008 with cassandra 3 - T169939
11:15 marostegui: Disable puppet on db1100 for mydumper/myloader
10:26 moritzm: installing perl update from stretch 9.1 point release
10:25 moritzm: installing perl update from jessie 8.9 point release
10:12 godog: reimage restbase1010 with cassandra 3 - T169939
09:39 moritzm: installing libgd security updates
09:28 moritzm: installing libonig security uodates#
09:05 jynus: disabling puppet on most db hosts to merge firewall changes safely
08:53 godog: reimage restbase1009 with cassandra 3 - T169939
08:49 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T174948 Add WMDE log channel (duration: 00m 49s)
08:43 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T110170 Enable Newsletter on mediawikiwiki (duration: 00m 51s)
07:15 marostegui: Stop MySQL on db1049 to copy its content to db1100 - https://phabricator.wikimedia.org/T172679
06:46 moritzm: installing php-luasandbox 2.0.14 on API canaries along with HHVM restart (T173705)
06:37 marostegui: Truncate l10n_cache table across production - T150306
04:08 demon@tin: Synchronized wmf-config/throttle.php: no-op (duration: 00m 49s)
03:14 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 6 03:14:16 UTC 2017 (duration 7m 6s)
03:07 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 14m 49s)
02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 18s)
away: disabled IPN messages at the PayPal console
00:14 Reedy: that was T174747 Adjust language icon color
00:14 reedy@tin: Synchronized php-1.30.0-wmf.16/skins/MinervaNeue: (no justification provided) (duration: 00m 49s)
00:10 reedy@tin: Synchronized php-1.30.0-wmf.16/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 49s)
00:09 reedy@tin: Synchronized php-1.30.0-wmf.17/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 50s)

2017-09-05

23:58 reedy@tin: Synchronized wmf-config/: Test ArticleCreationWorkflow extension on the Beta Cluster (duration: 00m 51s)
23:39 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile T161059 (duration: 00m 49s)
23:33 foks: 2FA disabled for Omshivaprakash (T175075)
22:17 robh: rebooting mw1163 as its serial console is locked up
22:07 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: kill some warnings and stuff (duration: 02m 55s)
22:06 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: (no justification provided) (duration: 03m 13s)
20:48 ejegg: re-enabled donation queue consumer and thank you mailer
20:40 ejegg: updated civicrm from 820bd98 to f9c373e
20:37 ejegg: disabled Donation queue consumer and thank-you mail sender
19:38 gilles@tin: Synchronized private/PrivateSettings.php: Add Thumbor username to Swift configuration (duration: 00m 48s)
19:33 jynus: rebuilding querycache table on tewiktionary.dbstore1002, it had crashed
19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wmf.17 on group0
19:05 Reedy: deleted querycache rows where qc_type = '' on all wikis T174513
18:11 ejegg: updated fundraising tools from 9554229 to 8d39806
18:11 demon@tin: Finished scap: wmf.17 bootstrap (duration: 47m 00s)
17:38 bblack: cp1074 - restart varnish backend, mailbox lag
17:24 demon@tin: Started scap: wmf.17 bootstrap
17:14 demon@tin: Pruned MediaWiki: 1.30.0-wmf.12 (duration: 02m 32s)
16:59 andrewbogott: reducing the number of concurrent nova-conductor workers. May cause hiccups as services restart
16:58 elukey: ran authdns-update from ns1.w.o after https://gerrit.wikimedia.org/r/374385 to create electcom.wikimedia.org
16:51 jynus: reloading dbproxy1005 to repoing to db1009 again- things seem stable right now
16:45 elukey: add new virtualhost electcom.wikimedia.org to the appservers apache config - https://gerrit.wikimedia.org/r/374389 (implies apache config reload)
16:33 mattflaschen@tin: Synchronized php-1.30.0-wmf.16/: Prepare to enable RCFilters (WLFilters) on Watchlist, but without i18n changes (reverted) (duration: 12m 41s)
16:14 addshore: addshore@terbium:~$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki mediawikiwiki --extension newsletter
16:08 akosiaris: reboot all kubernetes boxes T170119
15:43 matt_flaschen: 'Scap sync failed on i18n, so I'll deploy just the non-i18n ones'
15:36 mattflaschen@tin: scap aborted: Prepare to enable RCFilters (WLFilters) on Watchlist (duration: 08m 42s)
15:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174509 (duration: 00m 47s)
15:27 mattflaschen@tin: Started scap: Prepare to enable RCFilters (WLFilters) on Watchlist
14:30 reedy@tin: Synchronized wmf-config/: phpcs (duration: 00m 47s)
14:28 reedy@tin: Synchronized tests/: phpcs (duration: 00m 46s)
14:26 reedy@tin: Synchronized errorpages/404.php: phpcs (duration: 00m 45s)
14:25 reedy@tin: Synchronized docroot/noc/conf/highlight.php: Fix links for dblists in highlight.php (duration: 00m 44s)
14:15 zeljkof: EU SWAT finished
14:13 zfilipin@tin: Synchronized php-1.30.0-wmf.16/includes/EditPage.php: SWAT: Re add wpScrolltop id in EditPage (T174723) (duration: 00m 45s)
13:47 zfilipin@tin: Synchronized php-1.30.0-wmf.16/extensions/ContentTranslation/: SWAT: encodeURIComponent title to escape / properly (T174792) (duration: 00m 48s)
13:30 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add abusefilter-view-private to rollbackers in zhwiki (T174978) (duration: 00m 46s)
13:28 godog: reimage restbase2005 - T169939
13:17 marostegui: Drop pr_index tables from where ProofreadPage isn't enabled - T174782
13:14 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logo for sr.wikibooks.org (T172284) (duration: 00m 46s)
12:24 hashar: ci: switched mediawiki/core mediawiki/vendor php5.5 jobs from trusty to jessie - T161882
12:15 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Enable sendEchoNotification for enwiki, dewiki, frwiki (T142102) (duration: 00m 46s)
11:54 marostegui: Drop reader_feedback, reader_feedback_history, reader_feedback_pages tables - T174586
11:07 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: (Actually) Enable the ArticlePlaceholder on sqwiki (T174335) (duration: 00m 45s)
11:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on sqwiki (T174335) (duration: 00m 46s)
10:38 hashar: Nodepool / CI are fully backup and processing queued jobs
10:25 hashar: Restarting Nodepool. I have managed to spawn instances manually for contintcloud tenant
10:18 _joe_: stopping puppet, nginx on mw2249 for experimentation
10:11 ema: reboot cp4024 T174891
10:08 godog: copy python-logstash from jessie-wikimedia to stretch-wikimedia
09:59 hashar: Stopped Nodepool on labnodepool1001.eqiad.wmnet
09:52 akosiaris: reimage kubernetes[12]00[1-4] T170119
08:49 moritzm: upgrading canary app servers to hhvm-luasandbox 2.0.14 (T173705)
08:47 moritzm: uploaded php-luasandbox/hhvm-luasandbox 2.0.14 to apt.wikimedia.org (T173705)
07:14 moritzm: installing libgd security updates on canary app servers (along with hhvm restart)
07:03 _joe_: launching manually 3 workers for refreshLinks jobs on commons, T173710
06:32 marostegui: Deploy alter table on db1083 - T174509
06:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174509 (duration: 00m 46s)
06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174509 (duration: 00m 46s)
06:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174509 (duration: 00m 55s)
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 5 02:30:03 UTC 2017 (duration 6m 37s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 28s)

2017-09-04

23:32 Dereckson: Set e-mail to Hexabot bot account to allow account recovery (T174973)
22:07 legoktm: starting script to reparse all pages in parsoid for Linter (python2 parsoid-reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php) - T161556
17:50 moritzm: uploaded debdeploy 0.0.99-2 to apt.wikimedia.org
17:19 gehel@tin: Finished deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment (duration: 02m 46s)
17:17 gehel@tin: Started deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment
16:29 mobrovac@tin: Finished deploy [restbase/deploy@8c9c436]: (no justification provided) (duration: 06m 23s)
16:23 mobrovac@tin: Started deploy [restbase/deploy@8c9c436]: (no justification provided)
16:10 mobrovac: restbase depooled restbase10(0[89]|10) and restbase2005 for T169939
16:07 _joe_: rolling restart of pybal in codfw/eqiad low-traffic pools for picking up the new jobrunner LVS endpoint
15:42 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,service=nginx
14:51 godog: reimage restbase2003 for use with cassandra 3 / jbod - T169939
14:27 moritzm: enabled LDAP authentication for ganglia.wikimedia.org
13:37 gehel: rolling restart of elasticsearch/logstash to cleanup unused plugins - T174933
13:34 gehel: delete leftover grafana-dashboards index on elasticsearch logstash cluster
13:22 gehel: restart elasticsearch on logstash1006 to test plugin removal - T174933
12:26 moritzm: uploaded debdeploy 0.0.99-1+trusty/trusty-wikimedia to apt.wikimedia.org
12:18 marostegui: Stop MySQL on db1037 as it is going to be decommissioned - T174902
12:10 elukey: rolling restart of zookeeper on conf100[123] for jvm security updates
11:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
11:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
11:45 godog: restart varnish on cp1099 - mailbox lag
10:22 ariel@tin: Finished deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression (duration: 00m 02s)
10:22 ariel@tin: Started deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression
09:51 moritzm: uploaded debdeploy 0.0.99-1+jessie/jessie-wikimedia to apt.wikimedia.org
09:41 mobrovac@tin: Started restart [mathoid/deploy@44ea6d8]: Restart for libxml update
09:36 moritzm: uploaded debdeploy 0.0.99-1/stretch-wikimedia to apt.wikimedia.org
08:52 elukey: restart zookeeper on conf2003 for jvm security updates
08:48 marostegui: Stop MySQL on db1045 as it will be decommissioned - T174806
08:47 moritzm: installing mercurial security updates
08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 46s)
08:39 moritzm: installing libgcrypt20 security updates
08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 48s)
08:33 godog: upload scap 3.7.0-1 to apt.w.o - T127762
07:39 moritzm: installing gnupg security updates
07:03 marostegui: Deploy alter table on s3 codfw master with replication on kpbwiki - T168661
07:02 _joe_: starting additional runJobs instance for htmlcacheupdate on commons T173710
04:52 marostegui: Deploy alter table on db1052 - T168661
02:44 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 4 02:44:15 UTC 2017 (duration 6m 52s)
02:37 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 17s)

2017-09-03

17:52 elukey: depooled cp4024 (ulsfo upload) due to kernel errors in dmesg
15:57 ema: restart varnish backend on cp1073 (mailbox lag)
09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=varnish-fe
09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=nginx
09:25 ema: cp4024 is down, powercycled
09:13 _joe_: manually fixed value of /conftool/v1/pools/eqiad/api_appserver/apache2/mw1208.eqiad.wmnet in codfw to allow replication to restart
00:59 legoktm: legoktm@terbium:~$ mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php --wiki=hiwikiversity --backend=local-multiwrite # T174859

2017-09-01

16:18 bblack: restart varnish backend on cp1074 (mailbox lag)
{{safesubst:SAL entry|1=15:15 urandom: Restarting Cassandra: restbase-dev100[5-6]-{a,b}}}
{{safesubst:SAL entry|1=15:06 urandom: Restarting Cassandra: restbase-dev1004-{a,b}}}
15:00 reedy@tin: Synchronized php-1.30.0-wmf.16/extensions/Popups/: T174724 (duration: 00m 46s)
14:28 marostegui: Add 150G to /srv partition on labsdb1001
13:42 marostegui: Rename table pr_index on enwiki on db1089 - T174782
13:10 marostegui: Stop MySQL on db1026 as it will be decommissioned - T174763
12:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1081 original weight - T168661 (duration: 00m 43s)
12:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)
12:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)
11:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1081 weight - T168661 (duration: 00m 43s)
11:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 with low weight - T168661 (duration: 00m 44s)
10:35 elukey: stop puppet on thorium and disable root rsyncs - T174756
10:31 marostegui: Upgrade MariaDB to 10.0.32 on db1081 - T168661
10:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T168661 (duration: 00m 43s)
09:10 godog: bounce graphite-web on graphite1001, problematic query? using all CPU
09:04 ema: lvs1007 upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
08:59 godog: depool restbase200[135] before reimage - T169939
07:22 elukey: restart apache2 and hue on thorium, Analytics sites down, investigating
07:06 marostegui: Power reset db2044 as it is unresponsive - T174764
03:54 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: I4dfc33f66c3 - Enable jQuery 3 on nlwiki sister projects (duration: 00m 43s)
00:57 tstarling@tin: Synchronized php-1.30.0-wmf.16/includes/parser/Parser.php: (no justification provided) (duration: 00m 44s)
00:13 RainbowSprinkles: gerrit: flushed all non-login caches, things might be sluggish for the next ~15mins or so

2017-08-31

20:03 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: (no justification provided)
19:27 mattflaschen@tin: Finished scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030 (duration: 21m 23s)
19:17 cmjohnson1: powercycled mw1294
19:05 mattflaschen@tin: Started scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030
18:53 mattflaschen@tin: Synchronized docroot/noc/conf/interwiki-labs.php.txt: Interwiki links on Beta Cluster: T69931 (duration: 00m 47s)
18:51 mattflaschen@tin: Synchronized wmf-config: Interwiki links on Beta Cluster: T69931 (duration: 00m 49s)
17:50 demon@tin: Synchronized php-1.30.0-wmf.16/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/watch.svg: sorta impromptu swat thing (duration: 00m 47s)
16:35 ejegg: re-enabled donation queue consumer and thank you mail sender
15:25 ejegg: updated civicrm from f00611d to 820bd98
15:20 ejegg: updated SmashPig from 8f4388d to 8eb98c1
15:19 ejegg: turned off donation queue consumer and thank you job
15:18 elukey: restart zookeeper on conf200[2,3] for jvm security updates
13:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight - T168661 (duration: 00m 47s)
13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T168661 (duration: 00m 47s)
13:07 elukey: restart zookeeper on conf2001 for security updates (canary node)
13:00 moritzm: installing augeas security updates
12:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T168661 (duration: 00m 47s)
12:14 marostegui: Upgrade MariaDB on db1084 - T168661
12:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T168661 (duration: 00m 47s)
12:05 akosiaris: reimage chlorine and kubernetes1004 T170119
12:04 akosiaris: reimage chlorine and kubernetes1004 T10119
11:58 moritzm: restarting nginx on dataset1001/ms1001 to pick up libxml security update
11:21 elukey: restart apache2 on bohrium for libxml + gnutls security updates
11:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1056 original weight - T168661 (duration: 00m 47s)
11:17 moritzm: restarting apache on dbmonitor* to pick up libxml security update
10:27 moritzm: restarting nginx on prometheus* to pick up libxml security update
10:22 moritzm: restarting apache on einsteinium/tegmen (Icinga hosts) to pick up libxml security update
10:14 moritzm: restarting nginx on meitnerium/archiva.wikimedia.org to pick up libxml security update
10:13 moritzm: restarting nginx on debug proxies to pick up libxml security update
10:10 moritzm: restarting apache on hafnium to pick up libxml security update
09:57 addshore: extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php run done, 6326 hashes recalculated, T172987
09:50 addshore: addshore@terbium:~$ mwscript extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php --wiki enwiktionary --batch-size 1000000 # Should upsert 6326 rows T172987
09:44 addshore: addshore@terbium:~$ mwscript extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php --wiki enwiktionary --batch-size 1000000 --dry-run # T172987
09:43 moritzm: restarting apache on auth* to pick up libxml security update
09:40 moritzm: installing libxml2 security updates
09:10 moritzm: installing experimental hhvm-luasandbox package on mw1261 (canary app server) with backported patch for T171392 / T173705
09:04 hashar: contint1001 upgrading git, restarting git-daemon and zuul-merger - T161086
09:01 godog: test reimage of restbase2001 - T169939
08:57 hashar: contint2001 upgrading git, restarting git-daemon and zuul-merger - T161086
08:45 godog: remove /srv/deployment/grafana on naos and tin, unused - T170881
08:41 gehel: restart postgresql / kartotherian / tilerator / tileratorui on maps@eqiad
08:40 addshore@tin: Synchronized php-1.30.0-wmf.16/extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php: T172987 Add waitForReplication to RecalculateCognateNormalizedHashes script (duration: 00m 47s)
08:20 gehel: restart postgresql / kartotherian / tilerator / tileratorui on maps@codfw
08:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic on db1056 - T168661 (duration: 00m 47s)
08:01 marostegui: Rename reader_feedback, reader_feedback_history, reader_feedback_pages tables on dewiki on db1092 - T174586
07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic on db1056 - T168661 (duration: 00m 47s)
06:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 with low weight - T168661 (duration: 00m 46s)
06:19 marostegui: Upgrade MariaDB to 10.0.32 on db1056 - T168661
06:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T168661 (duration: 00m 47s)
06:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T168661 (duration: 00m 47s)
02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 31 02:55:12 UTC 2017 (duration 7m 9s)
02:48 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 06m 40s)
02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 10m 02s)

2017-08-30

23:10 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/Newsletter/includes/specials/SpecialNewsletter.php: T174604 (duration: 00m 47s)
23:09 demon@tin: Synchronized php-1.30.0-wmf.15/extensions/Newsletter/includes/specials/SpecialNewsletter.php: T174604 (duration: 00m 48s)
23:05 demon@tin: Synchronized wmf-config/unitConversionConfig.json: swat (duration: 00m 47s)
21:05 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.30.0-wmf.16
21:03 thcipriani@tin: Synchronized php-1.30.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: ProofreadPage: Avoids a stack overflow T173520 (duration: 00m 47s)
20:45 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.30.0-wmf.15
20:22 ppchelko@tin: Finished deploy [changeprop/deploy@8d5fa29]: Start rejecting deduplicated messages (duration: 01m 05s)
20:21 ppchelko@tin: Started deploy [changeprop/deploy@8d5fa29]: Start rejecting deduplicated messages
20:07 urandom: T169939: Decommission Cassandra: restbase1009-c.eqiad.wmnet
19:09 ppchelko@tin: Finished deploy [changeprop/deploy@ff93ea8]: Don't deduplicate retry messages (duration: 01m 17s)
19:07 ppchelko@tin: Started deploy [changeprop/deploy@ff93ea8]: Don't deduplicate retry messages
19:06 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.16
19:04 demon@tin: Synchronized php: Symlink swap for group1/wmf.16 (duration: 00m 46s)
18:11 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/374399/4 (duration: 00m 47s)
17:29 urandom: T169939: Decommission Cassandra: restbase1009-b.eqiad.wmnet
17:27 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
17:12 ejegg: disabled creating CiviMail records for Thank You emails
16:49 XioNoX: rolling back vrrp priority change for the cr2<->asw-d-codfw link - T174366
16:34 XioNoX: start work on the cr2<->asw-d-codfw link - T174366
16:13 XioNoX: lowering vrrp priority for the cr2<->asw-d-codfw link - T174366
15:56 elukey: re-added analytics1055 among the hdfs/yarn worker after maintenance
15:49 bblack: cp1049 - restart varnish backend, mailbox lag
15:43 urandom: T169939: Decommission Cassandra: restbase1009-a.eqiad.wmnet
15:02 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1294.eqiad.wmnet
14:31 moritzm: restarting nginx on sodium to pick up libxml security update
14:07 elukey: restart java daemons on druid100[456] for jvm security updates
14:04 moritzm: restarting squid on install* servers to pick up libxml security update
14:04 moritzm: restarting squid on installurl downloaders to pick up libxml security update
14:02 moritzm: restarting apache on krypton to pick up libxml security update
13:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1053 original weight - T168661 (duration: 00m 46s)
13:55 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: Scale A/B test bucket sizes by 10 - T172291 (duration: 00m 46s)
13:50 moritzm: restarting squid on url downloaders to pick up libxml security update
13:34 moritzm: installing libxml2 security updates
13:23 moritzm: installing ffmpeg security upadtes
13:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 46s)
13:19 hashar: European SWAT completed
13:16 gehel: restarting wdqs-blazegraph and wdqs-updater on all wdqs nodes for logback config change - T172710
13:11 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on cywiki - T173054 (duration: 00m 48s)
12:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 45s)
12:51 gehel: restart wdqs-updater on wdqs2001 for config change
12:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 47s)
12:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 with low weight - T168661 (duration: 00m 46s)
12:05 marostegui: Upgrade MariaDB to 10.0.32 on db1053 - T168661
12:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T168661 (duration: 00m 47s)
11:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 original weight - T174265 (duration: 00m 46s)
10:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
09:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1059 - T168661 (duration: 00m 47s)
09:07 elukey: restart all jvm daemons on druid100[123] for security updates
08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 48s)
08:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
07:47 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1228.eqiad.wmnet
07:47 marostegui: Upgrade MariaDB on db1059 to 10.0.32 - T168661
07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059 for a MariaDB upgrade - T168661 (duration: 00m 53s)
07:14 moritzm: powering down mw1294 for hardware diagnostics (T1674069
07:12 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1294.eqiad.wmnet
07:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly repool db1055 - T174265 (duration: 00m 52s)
03:16 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 30 03:16:09 UTC 2017 (duration 7m 19s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 15m 42s)
02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 08m 28s)
02:17 demon@tin: Finished deploy [gerrit/gerrit@f33c63f]: Testing, no-op (duration: 00m 07s)
02:17 demon@tin: Started deploy [gerrit/gerrit@f33c63f]: Testing, no-op
01:27 demon@tin: Synchronized wmf-config/InitialiseSettings.php: remove education program from legalteamwiki (duration: 00m 47s)
00:53 demon@tin: Synchronized wmf-config/flaggedrevs.php: killing FR on testwiki (duration: 00m 46s)
00:52 demon@tin: Synchronized dblists/flaggedrevs.dblist: killing FR on testwiki (duration: 00m 47s)
00:05 cwd: turned dedupe off

2017-08-29

23:50 cmjohnson1: reinstalling mw1228
23:34 demon@tin: Synchronized php-1.30.0-wmf.15/extensions/Popups/: swat (duration: 00m 47s)
23:33 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/Popups/: swat (duration: 00m 50s)
21:33 urandom: T169939: Decommission Cassandra: restbase1008-c.eqiad.wmnet
21:16 andrewbogott: restarting designate-sink on labservices1001
21:01 ejegg: updated DjangoBannerStats from 5963e7c to ce95eab
20:56 ejegg: updated payments-wiki from 58d33b8 to bf97cd2
20:49 godog: bounce varnish on cp1074 / cp1099 / cp1072 - mailbox lag
20:38 ppchelko@tin: Finished deploy [changeprop/deploy@a57c79d]: Release a redis-based deduplicator in test mode. Attempt 2 (duration: 01m 17s)
20:37 ppchelko@tin: Started deploy [changeprop/deploy@a57c79d]: Release a redis-based deduplicator in test mode. Attempt 2
20:36 ppchelko@tin: Finished deploy [changeprop/deploy@ed0fadc]: Release a redis-based deduplicator in test mode (duration: 04m 28s)
20:31 ppchelko@tin: Started deploy [changeprop/deploy@ed0fadc]: Release a redis-based deduplicator in test mode
19:47 urandom: T169939: Decommission Cassandra: restbase1008-b.eqiad.wmnet
19:34 ottomata: restarting analytics kafka brokers and all mirror maker processes to apply https://gerrit.wikimedia.org/r/#/c/374610/
19:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.16
19:08 ppchelko@tin: Finished deploy [restbase/deploy@7f2e55f]: Update CXServer endpoints config (duration: 06m 48s)
19:02 ottomata: restarting main-eqiad -> analytics kafka mirror maker processes on analytics kafka brokers, something is not working...
19:01 XenoRyet: updated Smashpig from 98c5516 to 8f4388d
19:01 ppchelko@tin: Started deploy [restbase/deploy@7f2e55f]: Update CXServer endpoints config
18:19 robh: attempting firmware upgrade on scs-a8-eqiad
18:17 Pchelolo: depooling restbase1008,1009,1010,2003,2005 while cluster reshaping is going on
18:15 demon@tin: Finished scap: bootstrap wmf.16 (duration: 45m 27s)
18:15 urandom: T169939: Decommission Cassandra: restbase1008-a.eqiad.wmnet
17:30 demon@tin: Started scap: bootstrap wmf.16
17:23 ejegg: restarted donation queue consumer and thank you mailer
17:20 Reedy: Disabled oathauth for KartikMistry on wikitech
16:33 ejegg: updated civicrm from 7dfea0d to f00611d
16:32 ejegg: turned off donation queue consumer and thank you mailer
16:04 ejegg: restarted ingenico recurring charge job
15:54 urandom: T169939: Decommission Cassandra: restbase2005-c.codfw.wmnet
14:51 elukey: drop log.MobileWebUIClickTracking_10742159_15423246 from db1047 (archived on HDFS) - T172322
14:33 marostegui: Shutdown db1055 to replace its BBU - T174265
13:12 zeljkof: EU SWAT finished
13:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add several HD logos (T150618) (duration: 00m 43s)
13:09 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add several HD logos (T150618) (duration: 00m 43s)
12:37 elukey: restart kafka daemons on kafka1014 for jvm security updates
12:31 godog: bounce pybal on lvs1003 to pick up config changes
11:49 elukey: restart kafka daemons on kafka1013 for jvm security updates
11:43 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1168.eqiad.wmnet
11:30 elukey: restart java daemons on analytics100[1,2] (Hadoop Master nodes) for jvm updates
11:08 marostegui: Stop MariaDB on db1097 to migrate it to file per table - T161088
10:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T161088 (duration: 00m 43s)
10:48 moritzm: installing libxml2 security updates
10:31 moritzm: reimaging mw1168 (video scaler) to jessie
10:16 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1028 (duration: 00m 42s)
10:15 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1028 (duration: 02m 48s)
09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T168661 (duration: 00m 43s)
09:39 marostegui: Update MariaDB on db1064 to 10.0.32 - T168661
09:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 for a mariadb upgrade - T168661 (duration: 00m 43s)
09:33 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1169.eqiad.wmnet
09:29 elukey: re-installed pmacct/librdkafka1/kafkacat on rhenium with stretch versions - T173489
09:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight - T168661 (duration: 00m 43s)
08:53 ema: upgrading cache_text to varnish 4.1.8-1wm1
08:45 akosiaris: reprepro copy calico, calico-cni from jessie-wikimedia to stretch-wikimedia (apt.wikimedia.org) T170119
08:43 hashar: Restarting Jenkins for openjdk update
08:42 elukey: restart yarn/hdfs daemons for openjdk security updates
08:42 akosiaris: upload kubernetes_1.7.4-1 to apt.wikimedia.org/stretch-wikimedia/main T170119
08:42 moritzm: restarting archiva to pick up openjdk security update
08:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight - T168661 (duration: 00m 43s)
08:35 gehel: restart wdqs-updater on wdqs2001
08:32 hashar: apt-get upgrade on contint1001 and contint2001
08:20 moritzm: reimaging mw1169 (video scaler) to jessie
08:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight - T168661 (duration: 00m 43s)
08:06 elukey: drop log.MobileWebUIClickTracking_10742159_15423246 from dbstore1002 to free space (table archived on HDFS) - T172322 T168303
07:41 marostegui: Upgrade MariaDB on db1091 to 10.0.32 - T168661
07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T168661 (duration: 00m 42s)
06:56 moritzm: installing ghostscript security updates on trusty (Debian already fixed)
06:09 demon@tin: Synchronized scap/plugins/clean.py: no-op, for consistency (duration: 00m 43s)
06:05 marostegui: Restart MariaDB on db1102 and db1095 to pick up new replication filters - T174385
05:49 demon@tin: Pruned MediaWiki: 1.30.0-wmf.11 (duration: 02m 50s)
02:34 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 29 02:34:09 UTC 2017 (duration 6m 44s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 08m 00s)
00:25 urandom: T169939: Decommissioning restbase1010-a.eqiad.wmnet

2017-08-28

23:37 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/: consistency (duration: 00m 43s)
23:31 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 43s)
23:27 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 42s)
23:23 reedy@tin: Synchronized wmf-config/CommonSettings.php: abc2ly in firejail (duration: 00m 43s)
23:22 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: Fix escaping (duration: 00m 43s)
23:13 cwd: re-enabled dedupe jobs
23:13 ebernhardson@tin: Synchronized php-1.30.0-wmf.15/extensions/WikimediaEvents/: Turn off two cirrus AB tests T171742 T171214 (duration: 00m 44s)
23:08 ebernhardson@tin: Synchronized docroot/noc/conf/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s)
23:07 ebernhardson@tin: Synchronized dblists/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s)
22:56 ejegg: turned fundraising orphan rectifier back on
22:46 urandom: T169939: Decommissioning restbase1010-b.eqiad.wmnet
21:48 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: I1c9c9c0e - Enable jQuery 3 on nlwiki, svwiki, plwiki (duration: 00m 44s)
21:37 reedy@tin: Synchronized wmf-config/CommonSettings.php: Disable wgScoreAbc2Ly firejail T172582 (duration: 00m 44s)
21:06 ejegg: switched nightly Ingenico audit from wr1 to wx file parsing
21:06 reedy@tin: Synchronized wmf-config/CommonSettings.php: Run Score binaries in firejail T172582 (duration: 00m 43s)
21:01 reedy@tin: Synchronized wmf-config/CommonSettings.php: Move email blacklist to meta:Email_blacklist (duration: 00m 45s)
20:53 XioNoX: pushing "aggregate defaults discard" to all the cr* routers - T174364
20:52 urandom: T169939: Decommissioning restbase1010-c.eqiad.wmnet
20:16 ayounsi@tin: Finished deploy [librenms/librenms@5fea59c]: (no justification provided) (duration: 00m 05s)
20:16 ayounsi@tin: Started deploy [librenms/librenms@5fea59c]: (no justification provided)
20:16 XioNoX: upgrading librenms to 1.31.01
20:08 bd808@tin: Finished deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes) (duration: 00m 26s)
20:07 bd808@tin: Started deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes)
19:24 ejegg: updated CiviCRM from 200f34b to 7dfea0d
19:24 ottomata: restarting eventbus in eqiad to apply https://gerrit.wikimedia.org/r/#/c/372179/
19:23 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki T174357; Enable popups on en and de wiki for A/B test T172291 (duration: 00m 43s)
19:15 herron: scb1001: restarted pdfrender service
19:14 ottomata: restarting eventbus in codfw to verify https://gerrit.wikimedia.org/r/#/c/372179/
19:09 ottomata: rolling restart and rebalances of main-* kafka clusters for https://gerrit.wikimedia.org/r/#/c/372179/
19:03 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/CodeMirror/: Fix exception on some combination of quotes T174060 (duration: 00m 44s)
18:48 gwicke: restarted pdfrender instances in eqiad (T159922)
18:48 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: remove invalidated popup sampling rate variables https://gerrit.wikimedia.org/r/#/c/373171/7 (duration: 00m 43s)
18:47 ayounsi@tin: Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 02s)
18:47 ayounsi@tin: Started deploy [librenms/librenms@8c9da11]: (no justification provided)
18:43 aaron@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: 2d83569397 - Fix old regression in HTMLCacheUpdate de-duplication (duration: 00m 44s)
18:37 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Fix incorrect Special:UserLogin in Popups blacklist https://gerrit.wikimedia.org/r/#/c/373696/ (duration: 00m 43s)
18:33 XioNoX: upgrading librenms to 1.31
18:32 ayounsi@tin: Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 05s)
18:32 ayounsi@tin: Started deploy [librenms/librenms@8c9da11]: (no justification provided)
18:30 ejegg: updated CiviCRM from c30ec13 to 200f34b
18:21 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable wgEchoPerUserBlacklist at all wikis T173838 (duration: 00m 43s)
18:15 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 45s)
18:14 niharika29@tin: Synchronized static/images/mobile/copyright/: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 44s)
17:59 XioNoX: pushing "aggregate defaults discard" to *ams - T174364
17:56 urandom: T169939: Decommission Cassandra: restbase2005-b.codfw.wmnet
17:31 gehel: restarting wdqs-blazegraph and wdqs-updater for config change
17:19 gehel: restarting wdqs-blazegraph and wdqs-updater for config change
17:11 XioNoX: pushing "aggregate defaults discard" to cr2-knams - T174364
17:09 gehel@tin: Finished deploy [wdqs/wdqs@90f4e2d]: (no justification provided) (duration: 02m 27s)
17:07 gehel@tin: Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided)
17:07 gehel@tin: Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided)
16:46 demon@tin: Pruned MediaWiki: 1.30.0-wmf.10 (duration: 02m 41s)
16:39 demon@tin: Pruned MediaWiki: 1.30.0-wmf.14 [keeping static files] (duration: 02m 00s)
14:39 elukey: restart eventlogging_sync on dbstore1002 - issue after drop of old table
14:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099 with normal weight on s5 - T172679 (duration: 00m 44s)
14:31 elukey: drop PageContentSaveComplete_5588433_15423246 from the log database on db1046 (m4-master) - T170720
14:28 zeljkof: EU SWAT finished
14:28 zfilipin@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: SWAT: Disable rebound CDN purges for backlinks in HTMLCacheUpdateJob (T173710) (duration: 00m 45s)
14:01 zeljkof: extending EU SWAT for 10 or so minutes to deploy two more patches
13:55 elukey: restart kafka* daemons on kafka1012 for openjdk security updates (canary)
13:52 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Log WikibaseQualityConstraints (T171281) (duration: 00m 44s)
13:42 zfilipin@tin: Synchronized wmf-config/throttle-analyze.php: SWAT: Automatically include commons and wikidata in $wmgThrottlingExceptions (T163872) (duration: 00m 44s)
13:36 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
13:34 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
13:33 zfilipin@tin: Synchronized wmf-config/throttle-analyze.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow sysops to grant/remove transwiki user group in dtywiki (T174226) (duration: 00m 44s)
13:21 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add 4 URLs to $wgCopyUploadsDomain whitelist (T174152) (duration: 00m 45s)
13:09 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Remove non-transparent background from dty.wiki logos (T174098) (duration: 00m 45s)
12:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s)
12:43 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1259.eqiad.wmnet
12:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s)
12:07 gehel: restarting nginx for upgrade on wdqs*
12:05 gehel: restarting nginx for upgrade on elastic* / relforge*
12:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1099 pooled with low weight to s5 - T172679 (duration: 00m 45s)
11:05 moritzm: installing libxml2 security updates
08:41 moritzm: restarting squid3 on install1002
08:16 marostegui: Ugprade MariaDB on s4 codfw master - db2051 to 10.0.32 - T168661
08:10 marostegui: Stop MySQL on db1045 - T172679
08:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1045 to clone db1099 from it - T172679 (duration: 00m 46s)
07:51 moritzm: reimaging mw1259 to jessie (T145742)
07:22 marostegui: Force re-learn cycle on db1055 - https://phabricator.wikimedia.org/T174265
07:12 moritzm: installing openjdk security updates on notebook* hosts
07:05 moritzm: installing openjdk security updates on meitnerium
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 28 02:36:23 UTC 2017 (duration 6m 54s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 09m 36s)

2017-08-27

20:35 ariel@tin: Finished deploy [dumps/dumps@39f9b52]: write output files to temp location and move into place when complete (duration: 00m 02s)
20:34 ariel@tin: Started deploy [dumps/dumps@39f9b52]: write output files to temp location and move into place when complete
15:46 akosiaris: upload kubernetes_1.4.6-7 to apt.wikimedia.org/jessie-wikimedia/main T170346
05:30 marostegui: Force BBU relearn on db1055 - T174265
00:45 jynus: correction last log s/db1051/db1055/
00:42 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051, hw issues, may get lag (duration: 00m 44s)

2017-08-26

15:46 godog: bounce varnish on cp1073 - mailbox lag

2017-08-25

19:09 gehel: actually not killing the "stuck discovery report-updater process on stat1005", it is already gone - T174110
19:07 gehel: kill stuck discovery report-updater process on stat1005 - T174110
16:03 moritzm: created cn=ciadmin group in LDAP to be used for privileged Jenkins access, initial members in ticket (T169557)
14:29 godog: bounce varnish on cp1049 / cp1072 - mailbox lag
14:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db1097 - T168661 (duration: 00m 44s)
13:53 marostegui: Upgrade MariaDB on db1097 to 10.0.32 - T168661
13:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db1097 - T168661 (duration: 00m 44s)
13:40 marostegui: Upgrade MariaDB to 10.0.32 on db2019 - T168661
13:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2037 - T168661 (duration: 00m 45s)
13:24 marostegui: Upgrade MariaDB on db2037 to 10.0.32 - T168661
13:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2037 - T168661 (duration: 00m 44s)
12:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2044 - T168661 (duration: 00m 44s)
12:29 marostegui: Upgrade MariaDB on db2044 - T168661
12:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2044 - T168661 (duration: 00m 43s)
12:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2058 - T168661 (duration: 00m 48s)
11:48 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1028 (duration: 00m 47s)
09:19 moritzm: restart thumbor to pick up libxml2 security updates
09:12 godog: reimage restbase2001 to test new partman recipe - T169939
08:41 moritzm: restarting apache/hhvm on canary app servers to pick up libxml security update
08:00 volans: upgrading cumin to v1.0.0 on sarin/neodymium
07:55 gehel: reimaging logstash1006 after change of failed disk - T173679
07:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: pool db1069 with more weight (duration: 00m 44s)
07:00 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1033, pool db1069 with low weight (duration: 00m 44s)
06:55 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1033 (duration: 00m 43s)
06:30 jynus@tin: Synchronized wmf-config/db-eqiad.php: formatting fixes (duration: 00m 44s)
06:11 jynus@tin: Synchronized wmf-config/db-codfw.php: repool es2013, formatting fixes (duration: 00m 44s)
06:01 marostegui: Upgrade MariaDB to 10.0.32 on db2058 - T168661
06:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2058 - T168661 (duration: 00m 43s)
05:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 - T168661 (duration: 00m 44s)
05:18 moritzm: re-enable mw1260 (video scaler) for active job processing
05:16 marostegui: Reboot db2065 to pick up new kernel
05:08 marostegui: Upgrade MariaDB on db2065 - T168661
05:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 for a MariaDB upgrade - T168661 (duration: 00m 44s)
01:12 mattflaschen@tin: Synchronized php-1.30.0-wmf.15/extensions/Flow/modules/engine/components/board/features/flow-board-loadmore.js: T173807: Fix Flow infinite scroll (duration: 00m 45s)

2017-08-24

23:17 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler/TimedMediaHandler.php: SWAT: Disable Ogg Theora video transcodes in default config T172445 (duration: 00m 45s)
22:45 demon@tin: Pruned MediaWiki: 1.30.0-wmf.13 [keeping static files] (duration: 02m 01s)
22:24 ejegg: updated civicrm from fa882f2 to c30ec13 (pt-br and ja thank you letters)
21:59 gwicke: restarted pdfrender service instances in eqiad / T159922
21:54 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: https://gerrit.wikimedia.org/r/#/c/373701/1 (duration: 00m 45s)
21:13 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: Logging: https://gerrit.wikimedia.org/r/#/c/373691/ (duration: 00m 44s)
20:07 andrewbogott: stopping, starting, restarting etc. nodepool
19:58 urandom: T169939: Decommission Cassandra: restbase2003-c.codfw.wmnet
19:24 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.15
19:14 andrewbogott: restarting rabbitmq-server on labcontrol1001
18:38 ejegg: updated payments-wiki from bd9f730 to 58d33b8
18:38 bblack: varnish backend restart - cp1074 - mailbox lag
18:24 urandom: T169939: Decommission Cassandra: restbase2003-b.codfw.wmnet
17:33 arlolra@tin: Finished deploy [parsoid/deploy@bd12f8a]: Updating Parsoid to 538dad7f (duration: 12m 24s)
17:21 arlolra@tin: Started deploy [parsoid/deploy@bd12f8a]: Updating Parsoid to 538dad7f
17:01 jynus: stopping and cloning db1033 to db1069
16:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give normal weight to db1096 - T172679 (duration: 00m 47s)
16:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some more weight to db1096 - T172679 (duration: 00m 47s)
16:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some more weight to db1096 - T172679 (duration: 00m 47s)
16:34 urandom: T169939: Decommission Cassandra: restbase2003-a.codfw.wmnet
16:24 andrewbogott: apt-get upgrade and updated to MW 1.29.1 on wikitech-static. Rebooting to pick up kernel updates.
15:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some weight to db1096 - T172679 (duration: 00m 47s)
15:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1096 depooled to s5 - T172679 (duration: 00m 47s)
14:44 ladsgroup@tin: Synchronized php-1.30.0-wmf.15/extensions/Wikidata/extensions/Wikibase/client/includes/Changes/WikiPageUpdater.php: Reduce batch size in WikiPageUpdater (T173710) (duration: 00m 48s)
14:14 phuedx: refreshLinks on bawiki (T173994)
14:13 volans: updated cumin package in our APT to version cumin_1.0.0-1
13:19 phuedx@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T171853: Re-enable Page Previews for enwiki and dewiki on the Beta Cluster (duration: 00m 47s)
13:13 phuedx@tin: Synchronized portals: (no justification provided) (duration: 00m 49s)
13:12 phuedx@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 49s)
12:45 gehel: killing discovery report updater on stats1005 (stuck since Aug 15) - T173333
12:30 marostegui: Stop MySQL on db1026 to clone db1096 from it - T172679
12:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1026 to clone db1096 from it T172679 (duration: 00m 47s)
11:36 jynus: disable puppet on db1069 in preparation for reimage
11:08 marostegui: Global rename Papa1234 → Karl-Heinz Jansen - T173859
09:56 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1078 with full weight (duration: 00m 46s)
09:33 volans: executing https://wikitech.wikimedia.org/wiki/Cumin#Run_Puppet_only_if_last_run_failed to get rid of failed ones
08:54 jynus: 16 minute test run of rebuildTermSqlIndex.php on terbium
08:52 marostegui: Drop tables article_assessment, article_assessment_pages, article_assessment_ratings tables from testwiki - T173590
07:24 Amir1: starting the run for rebuildTermIndex (T171460)
02:50 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 24 02:50:40 UTC 2017 (duration 7m 4s)
02:43 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 05m 56s)
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 09m 50s)
01:46 aaron@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue: Deploy 752580637 (T173710) (duration: 00m 49s)

2017-08-23

23:49 maxsem@tin: Synchronized php-1.30.0-wmf.14/extensions/LoginNotify/: SWAT fixes (duration: 00m 47s)
23:47 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: SWAT fixes (duration: 00m 47s)
23:42 maxsem@tin: Synchronized php-1.30.0-wmf.14/extensions/CodeMirror/: https://gerrit.wikimedia.org/r/#/c/373324/ (duration: 00m 47s)
23:41 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/CodeMirror/: https://gerrit.wikimedia.org/r/#/c/373324/ (duration: 00m 49s)
23:28 maxsem@tin: Synchronized php-1.30.0-wmf.15/resources/: https://gerrit.wikimedia.org/r/#/c/373391/ (duration: 00m 48s)
23:27 maxsem@tin: Synchronized wmf-config/MetaContactPages.php: https://gerrit.wikimedia.org/r/#/c/373369/ (duration: 00m 48s)
20:59 urandom: T169939: Truncating MCS tables
20:26 thcipriani@tin: Synchronized php: group1 wikis to 1.30.0-wmf.15 (duration: 00m 46s)
20:25 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.15
20:13 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1078 with low weight (duration: 00m 47s)
20:12 thcipriani@tin: Finished scap: Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15 T173520 (duration: 04m 11s)
20:08 thcipriani@tin: Started scap: Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15 T173520
19:49 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change: pagePreviews: Enable A/B test (BC-only (duration: 00m 47s)
19:45 joal@tin: Finished deploy [analytics/refinery@f467ce1]: Bug fixing deploy (duration: 03m 16s)
19:44 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/MobileFrontend: Verify the existence of `url` key when parsing lang objects T172316 (duration: 00m 56s)
19:42 joal@tin: Started deploy [analytics/refinery@f467ce1]: Bug fixing deploy
19:36 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler: Enable WebM playback via ogv.js T172444 (duration: 00m 50s)
19:22 kaldari: foreachwiki sql.php /srv/mediawiki/php/maintenance/archives/patch-ip_changes.sql
19:19 kaldari: mwscript sql.php --wiki=testwiki /srv/mediawiki/php/maintenance/archives/patch-ip_changes.sql
19:15 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Remove CitThisPage from blacklist for page previews T173865 (duration: 00m 47s)
19:13 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler/: Enable WebM playback via ogv.js https://gerrit.wikimedia.org/r/#/c/373126/ (duration: 00m 49s)
19:12 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/TimedMediaHandler/: Enable WebM playback via ogv.js https://gerrit.wikimedia.org/r/#/c/373126/ (duration: 00m 50s)
19:00 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/MobileFrontend/: Verify the existence of key when parsing lang objects https://gerrit.wikimedia.org/r/#/c/373292/ (duration: 00m 49s)
18:57 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Log channel for LoginNotify T173888 (duration: 00m 47s)
18:55 ppchelko@tin: Finished deploy [changeprop/deploy@1998c10]: [Config] Disable mobile rerenders for non-wikipedia domains (duration: 01m 33s)
18:54 ppchelko@tin: Started deploy [changeprop/deploy@1998c10]: [Config] Disable mobile rerenders for non-wikipedia domains
18:52 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: Log everything T173888 (duration: 00m 48s)
18:43 gehel: manually running report updater for discovery golden data on stat1005 - T173333
18:40 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Reinforce LoginNotify settings https://gerrit.wikimedia.org/r/#/c/372555/ (duration: 00m 47s)
18:40 gehel: resetting permissions on stat1005:/srv/published-datasets/discovery - T173333
18:39 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Reinforce LoginNotify settings https://gerrit.wikimedia.org/r/#/c/372555/ (duration: 00m 47s)
18:32 thcipriani: jenkins restarted, zuul should pick-up queue
18:26 thcipriani: jenkins appears to be having some issues, trying to dump threads now and then restart to see if it helps
18:13 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Allow crats to add people to accountcreator group on mw.or https://gerrit.wikimedia.org/r/#/c/373317/ (duration: 00m 49s)
17:40 bblack: reboot cp4021
16:38 cwd: disabled all dedupe
16:26 demon@tin: Pruned MediaWiki: 1.30.0-wmf.11 [keeping static files] (duration: 01m 32s)
16:24 bd808@tin: Finished deploy [striker/deploy@2f2dd7c]: Deploying 2f2dd7c "Tool account creation and more" (T128400, T149458, T159044, T164847, T167931, T168480, T173845) (duration: 00m 39s)
16:23 bd808@tin: Started deploy [striker/deploy@2f2dd7c]: Deploying 2f2dd7c "Tool account creation and more" (T128400, T149458, T159044, T164847, T167931, T168480, T173845)
16:22 demon@tin: Pruned MediaWiki: 1.30.0-wmf.9 (duration: 03m 44s)
15:35 legoktm: added addshore to "integration" gerrit group (T173233)
15:10 gehel: shutdown logstash1006 for disk replacement - T173679
14:49 moritzm: installing texlive security updates on trusty (Debian already fixed)
14:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4027.ulsfo.wmnet
14:34 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4027.ulsfo.wmnet
13:31 marostegui: Stop MySQL on db1041 to get it ready for decommission - T173915
13:13 zeljkof: EU SWAT finished
13:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Flow as a Beta feature on wawiki and wawikionary (T172947) (duration: 00m 48s)
12:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1041 to decommission it - T173915 (duration: 00m 48s)
12:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1041 to decommission it - T173915 (duration: 00m 48s)
10:39 jynus: stopping and deleting s1 and s4 from dbstore2001
10:06 joal@tin: Finished deploy [analytics/refinery@84d6ee4]: Regular deploy (1 month since last) (duration: 03m 30s)
10:03 moritzm: upgrading remaining image scalers to luasandbox 2.0.13
10:02 joal@tin: Started deploy [analytics/refinery@84d6ee4]: Regular deploy (1 month since last)
09:36 godog: kill older and running tasks from gerrit queue, it was stuck
09:10 moritzm: upgrading remaining API servers in to luasandbox 2.0.13
09:08 godog: upload prometheus-blackbox-exporter 0.7.0+ds1-1~wmf1 to jessie-wikimedia, backported - T169860
08:35 marostegui: Stop Upgrade MySQL on db2073 to 10.0.32 - T168661
08:22 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2073 - T168661 (duration: 00m 59s)
08:18 moritzm: upgrading remaining job runners to luasandbox 2.0.13
07:12 marostegui: Global rename: Opdire657 → Sakiv - T173834
06:55 moritzm: upgrading remaining app servers in to luasandbox 2.0.13
03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 23 03:15:36 UTC 2017 (duration 7m 6s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 15m 11s)
02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 08m 03s)

2017-08-22

23:39 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/Linter/includes/LintErrorsPager.php: SWAT: Fix up 11f4a97ba6bcd0c1de (duration: 00m 49s)
23:35 thcipriani@tin: Finished scap: SWAT: Fix typo in the notification message (duration: 21m 48s)
23:13 thcipriani@tin: Started scap: SWAT: Fix typo in the notification message
23:10 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/Popups/includes/PopupsHooks.php: SWAT: Remove aborting of BeforePageDisplay hook T173411 (duration: 00m 49s)
21:18 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Retry CodeMirror deployment T170966 (duration: 00m 49s)
20:41 ejegg: updated CiviCRM from 4aa177b to fa882f2
20:25 mutante: restbase-dev* - puppet runs fail due to E: Version '3.11.0' for 'cassandra' was not found
20:00 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.30.0-wmf.15
19:56 urandom: starting cassandra restbase2004-a and restbase2006-c, OOMs
19:46 thcipriani@tin: Finished scap: testwiki to php-1.30.0-wmf.15 and rebuild l10n cache (duration: 45m 55s)
19:00 thcipriani@tin: Started scap: testwiki to php-1.30.0-wmf.15 and rebuild l10n cache
18:50 robh: labservies1002 update completed (not 1003, typo)
18:50 robh: labservies1003 update completed
18:40 robh: firmware update of labservices1002 in progress
17:54 andrewbogott: removing obsolete apache2 and puppetmaster packages from labcontrol boxes for https://phabricator.wikimedia.org/T171786
17:07 thcipriani: starting branch cut for 1.30.0-wmf.15
13:58 godog: bounce varnish on cp1074 / cp1049 / cp1073 - mailbox problems
13:48 zeljkof: EU SWAT finished
13:45 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: relatedArticles: Tidy up config (T165991) (duration: 00m 44s)
13:44 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: relatedArticles: Tidy up config (T165991) (duration: 00m 44s)
13:29 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wmgUseWikimediaShopLink to true for ptwiki (T173768) (duration: 00m 45s)
10:41 moritzm: upgrading hhvm-luasandbox on mw1293-mw1295 (image scalers)
10:21 Amir1: another run of rebuildTermSqlIndex (T171460)
09:54 moritzm: upgrading hhvm-luasandbox on mw1189-mw1208 (API servers)
09:42 marostegui: Renaming user Darwinius → DarwIn - T173159
09:35 moritzm: upgrading hhvm-luasandbox on mw1180-1188 and mw1209-mw1220 (app servers)
09:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T172996 (duration: 00m 44s)
08:59 akosiaris: upload apertium-bel-rus_0.2.0~r81186-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
08:55 moritzm: upgrading hhvm-luasandbox on mw1161-mw1167 (job runners)
08:43 akosiaris: upload apertium-rus_0.1.0~r81184-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
08:43 akosiaris: upload apertium-bel_0.1.0~r81357-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
08:35 godog: bounce varnish on cp1072 - mailbox problems
08:33 godog: bounce varnish on cp4015 - mailbox problems
08:24 marostegui: Drop s4 from db1095 with NO replication - T172996
08:23 moritzm: upgrading hhvm-luasandbox on deployment servers / script runners
07:34 marostegui: Stop replication on db1064 sanitarium2 and sanitarium3 master to move labsdb1009,10 and 11 s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996
07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 - T172996 (duration: 00m 52s)
07:28 moritzm: installing ruby security updates on trusty (Debian already fixed)
07:11 moritzm: installing c-ares security updates on trusty (Debian already fixed)
06:54 moritzm: installing graphite2 (the image library, not the metrics tool) security updates on trusty (Debian already fixed)
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 22 02:30:47 UTC 2017 (duration 6m 55s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 07m 11s)

2017-08-21

22:50 robh: rebooting tin for firmware update since its idle
22:20 mutante: praseodymium - installing BIOS upgrade, reboot
22:11 mutante: xenon - installing BIOS upgrade
21:58 mutante: cerium (cassandra test) - rebooting for firmware upgrade
21:54 mutante: cerium - installing Dell BIOS upgrade (T162850)
21:26 mutante: bast2001 - running Dell BIOS firmware upgrade (T162850)
21:20 arlolra@tin: Finished deploy [parsoid/deploy@2210a38]: Updating Parsoid to 28a9a22b (duration: 10m 59s)
21:09 arlolra@tin: Started deploy [parsoid/deploy@2210a38]: Updating Parsoid to 28a9a22b
18:40 niharika29@tin: Synchronized dblists/pp_stage1.dblist: Roll page previews out to all wikis except en and de wiki T162672 (duration: 00m 44s)
18:34 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Enable OOjs UI EditPage on all wikis https://gerrit.wikimedia.org/r/#/c/366868/ (duration: 00m 44s)
18:33 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable OOjs UI EditPage on all wikis https://gerrit.wikimedia.org/r/#/c/366868/ (duration: 00m 44s)
18:30 niharika29@tin: Synchronized wmf-config/Wikibase.php: Wikibase on deployment-prep: Exclude non-existent wikis from clientDbList T173571 (duration: 00m 44s)
18:25 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Increase AbuseFilter autodisable thresholds for Meta-Wiki T173633 (duration: 00m 44s)
18:22 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Add CookBook and Cookbook Talk NS on hiwikibooks T173398 (duration: 00m 45s)
18:16 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Administrators to add/remove 'transwiki' at nowiktionary T172365 (duration: 00m 45s)
18:12 niharika29@tin: Synchronized static/images/: Set project logo for wikimania2018wiki T173042 (duration: 00m 44s)
18:11 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Set project logo for wikimania2018wiki T173042 (duration: 00m 44s)
17:22 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Bring back cxserver on scb2001 to a stable state - T173038 (duration: 00m 14s)
17:22 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Bring back cxserver on scb2001 to a stable state - T173038
17:12 gehel@tin: Finished deploy [wdqs/wdqs@a1c4f1f]: (no justification provided) (duration: 02m 27s)
17:10 gehel@tin: Started deploy [wdqs/wdqs@a1c4f1f]: (no justification provided)
15:40 bblack: cp1099 - varnish backend restart for mailbox lag
14:55 mobrovac: cxserver depool scb2001 to debug failed checks - T173038
14:54 mobrovac@tin: Finished deploy [cxserver/deploy@1065ffe]: Deploy 1065ffe2 to canary scb2001 for debugging - T173038 (duration: 00m 18s)
14:53 mobrovac@tin: Started deploy [cxserver/deploy@1065ffe]: Deploy 1065ffe2 to canary scb2001 for debugging - T173038
14:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T163190 (duration: 00m 44s)
14:04 zeljkof: EU SWAT finished
14:04 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add HD logos for srwikisource, update them too (T172268) (duration: 00m 44s)
14:02 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add HD logos for srwikisource, update them too (T172268) (duration: 00m 44s)
13:53 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logos for srwikinews, add HD version for them (T172255) (duration: 00m 49s)
13:52 jynus: stop dbstore2001 mariadb@s4, start mariadb@s7
13:52 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logos for srwikinews, add HD version for them (T172255) (duration: 00m 44s)
13:46 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set X-Frame-Options: SAMEORIGIN if UploadWizard enabled (T173631) (duration: 00m 44s)
13:46 ottomata: adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on db1047: T170990
13:37 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logos for srwiktionary, add HD logos for srwiktionary (T172245) (duration: 00m 44s)
13:36 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logos for srwiktionary, add HD logos for srwiktionary (T172245) (duration: 00m 45s)
13:27 jynus: stop dbstore2001 mariadb@s7
13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add new logo for the Baskhir Wikibooks (T173471) (duration: 00m 44s)
13:26 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add new logo for the Baskhir Wikibooks (T173471) (duration: 00m 44s)
13:26 ottomata: adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on dbstore1002: T170990
13:11 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Reopen bawikibooks (T173471) (duration: 00m 44s)
13:10 zfilipin@tin: Synchronized dblists/closed.dblist: SWAT: Reopen bawikibooks (T173471) (duration: 00m 44s)
12:52 marostegui: Stop replication on db1079 and db1041 to compare their data - T163190
12:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T163190 (duration: 00m 45s)
12:35 marostegui: Stop MySQL on db1015 to decommission it - T173570
09:50 jynus: restart dbstore2001 mariadb@s1
09:40 jynus: restart dbstore2001 mariadb@x1
09:29 moritzm: upgrading hhvm-luasandbox on mw1262-1265
09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1015 - T173570 (duration: 00m 44s)
09:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1015 - T173570 (duration: 00m 44s)
09:01 moritzm: upgrading hhvm-luasandbox on mw1261
08:08 marostegui: Rename tables article_assessment, article_assessment_pages, article_assessment_ratings tables from testwiki on db1078 - T173590
07:42 marostegui: Drop cx_drafts table from x1 - T172364
07:13 marostegui: Stop s4 replication thread on db1095 - T172996
07:09 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 (duration: 00m 45s)
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 21 02:36:58 UTC 2017 (duration 7m 3s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 08m 57s)

2017-08-19

13:17 Amir1: another run: (Hotwc3 → HotWC3) (Lamia Bahy → Albedo11) (Monóxido de carbono → Roquetero) (PaulMichaels → PaulBenario) (Rodrigo.dst → RodrigoTavares) (Sadia Tasnim (Moyna) → মুহাম্মদ সুমন মাহমুদ) (Syou 18331322 → Ms3102) (TzvetelinaOOD1 → Tzveti1) (World Para Taekwondo → TKD at World Para Taekwondo) (Yaellerner → Ya1levy777) (平井俊光 → Toshimit) (T173419)
13:00 Amir1: running the script for ("Clopper228" "CGminded") and ("Gregory.lussier" "StevenSmith83473") (T173419)
12:47 Amir1: ladsgroup@terbium:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki "Bolsée" "Kathmandu2017" (T173419)
03:21 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on test.wikidata and mediawiki.org - Id4d42d8c53 (duration: 00m 45s)

2017-08-18

21:05 mutante: gerrit2001 - restarted gerrit again after reverting gerrit:372426, using systemctl commands, not 'service' or init.d
20:55 mutante: restarting gerrit on gerrit2001
20:48 demon@tin: Synchronized scap/plugins/clean.py: Completeness (duration: 00m 44s)
20:43 RainbowSprinkles: prior thing was a no-op, testing
20:42 demon@tin: Pruned MediaWiki: 1.30.0-wmf.9 [keeping static files] (duration: 01m 43s)
19:03 bblack: cp1074 - varnish backend restart (mailbox lag)
19:01 bblack: reboot cp4021-8
18:21 bblack: puppeting cp4021-8 - expect possibility of ipsec alerts, etc...
13:32 Amir1: another run of /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item --from-id $(tail -100 /tmp/rebuildTermSqlIndex.log | grep -E "Processed up to page (\d+?)" | sed -E "s/Processed up to page //; s/ \(Q.+?//" | tail -1) >>/tmp/rebuildTermSqlIndex.log 2>&1
13:26 Amir1: ladsgroup@terbium:~$ mwscript namespaceDupes.php --wiki=hiwikiversity (T172977)
12:44 Amir1: start of ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item
11:36 jynus: change topology of dbstore2001:x1 and dbstore2002:x1
11:00 elukey: reboot dbstore1002 for kernel updates
10:34 elukey: restart mysql on dbstore1002 - attempt to reclaim space after big table drop (stop slaves and el_sync, check running queries, stop mysql, check process, start mysql)
09:55 Amir1: one small pass of ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=entity (T171460)
09:46 Amir1: ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property (T171460)
09:42 moritzm: installing libmspack security updates
09:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2077 - T168409 (duration: 00m 44s)
07:28 marostegui: Stop MySQL on db2077 to copy it to dbstore2001 - T168409
07:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2077 - T168409 (duration: 00m 44s)
07:20 moritzm: installing openjdk-7 security updates on trusty
06:58 jynus: and nginx
06:55 jynus: systemctl restart squid3.service on install2002, apt seem stuck on some servers
06:24 jynus: stopping and upgrading db2033
05:43 jynus: upgrading and restarting all mariadb instances on dbstore2002
afk: updated fundraising dash from 696a3ff to 7dfc969
00:48 ebernhardson@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171213: Increase sampling rate of cirrus satisfaction schema (again) to 1k per bucket per day (duration: 00m 44s)
00:18 ebernhardson@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171213: Increase sampling rate of cirrus satisfaction schema (duration: 00m 44s)

2017-08-17

23:31 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Revert "Disable cirrus MLR ab test" (duration: 00m 44s)
23:21 thcipriani@tin: Synchronized php-1.30.0-wmf.14/resources/src/mediawiki.rcfilters/dm/mw.rcfilters.dm.FilterGroup.js: SWAT: RCFilters: Fix validation for single_option groups T173303 (duration: 00m 44s)
23:14 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Reapply "Remove temporary wgStructuredChangeFiltersEnableExperimentalViews setting" (duration: 00m 45s)
22:10 ppchelko@tin: Finished deploy [changeprop/deploy@2c553a6]: Lower the concurrecy for transcludes to decrease cassandra load during cluster reshaping (duration: 01m 14s)
22:09 thcipriani@tin: Synchronized php-1.30.0-wmf.14/includes/specials/SpecialNewpages.php: Restore the newFromId() approach in SpecialNewpages::feedItemDesc T173541 (duration: 00m 46s)
22:09 ppchelko@tin: Started deploy [changeprop/deploy@2c553a6]: Lower the concurrecy for transcludes to decrease cassandra load during cluster reshaping
22:07 gwicke: restarting pdfrender on scb100* nodes
21:38 ejegg: updated dash from bec0077 to 696a3ff
21:28 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable LoginNotify on sites without Echo too. :|https://gerrit.wikimedia.org/r/#/c/372456/ (duration: 00m 44s)
21:19 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable LoginNotify finally! Yippeeeeee https://gerrit.wikimedia.org/r/#/c/372450/ (duration: 00m 45s)
21:17 bblack: cp1063: varnish backend restart (mailbox lag)
20:07 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.14
19:28 chasemp: disable puppet for cloud things for some careful refactor merging
19:14 thcipriani@tin: Synchronized php: group1 wikis to wmf.14 (duration: 00m 46s)
19:12 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to wmf.14
19:07 thcipriani@tin: Finished scap: ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116 T173520 (duration: 06m 13s)
19:01 thcipriani@tin: Started scap: ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116 T173520
18:28 ebernhardson: restart logstash on logstash1003 to see why its not reading EventError messages from kafka
18:24 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/RevisionSlider/: Revert Reintroduce hover and bar clicking - https://gerrit.wikimedia.org/r/#/c/372384/ (duration: 00m 48s)
18:16 jynus: upgrading and restarting all mariadb instances on dbstore2001
18:12 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/LoginNotify/: Log usage statistics https://gerrit.wikimedia.org/r/#/c/372214/ (duration: 00m 51s)
18:03 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/UploadWizard/UploadWizard.config.php: SWAT: Preserve array keys (language keys) when sorting the language dropdown T173522 (duration: 00m 51s)
17:53 jynus: restarting mariadb (dbstore2001:x1) to test new buffer pool configuration
17:20 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.13 now T173520
17:10 RainbowSprinkles: gerrit: cpu spikes, lots of large gc logs. Looking into it. (nb: things might be a little slow, but it is /up/)
16:54 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.14 now for T164173
16:51 thcipriani@tin: Synchronized php-1.30.0-wmf.14/includes/jobqueue/jobs/RefreshLinksJob.php: Avoid lock acquisition errors for multi-title refreshlinks jobs T173462 (duration: 00m 51s)
16:45 ejegg: updated SmashPig from c501f53 to 98c5516
16:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2077 - T170662 (duration: 00m 49s)
16:24 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2077 - T170662 (duration: 00m 51s)
16:16 urandom: T169939: RESTBase: converting (33) new keyspaces to time-windowed compaction
16:11 urandom: T169939: Lower eqiad compaction throughput from 20MB/s to 15MB/s
15:46 godog: reimage restbase2001 - T169939
15:42 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2001.codfw.wmnet
14:32 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp3036.*
13:30 zeljkof: EU SWAT finished
13:24 zfilipin@tin: Synchronized static/favicon/wikiversity.ico: SWAT: Update wikiversity favicon (T160491) (duration: 00m 50s)
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change $wgArticleCountMethod to any for srwikiquote (T172974) (duration: 00m 51s)
13:08 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Add one throttling exception (T173444) (duration: 00m 51s)
12:38 moritzm: installing libgd/libsoup security updates
12:08 urandom: T169939: Decommissioning Cassandra/restbase2001-c.codfw.wmnet
12:07 urandom: T169939: Decommissioning Cassandra/restbase2001-b.codfw.wmnet
10:29 gilles: Thumbor stress test finished
10:20 gilles: Load testing Thumbor
10:13 godog: roll-restart nginx on thumbor to apply https://gerrit.wikimedia.org/r/372199
08:36 godog: roll-restart thumbor to apply webp support change
08:30 godog: restart varnish on cp1062 to fix mailbox lag
08:05 godog: reboot ms-be2021, unreachable on network
06:24 marostegui: Stop slave on db2047 to fix duplicate keys - T151029
05:55 marostegui: Stop replication on db2077 to fix duplicate entries - T151029
05:48 marostegui: Stop replication in sync on db1078 and db1015 - T164488
03:11 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 17 03:11:13 UTC 2017 (duration 7m 18s)
03:03 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 03m 58s)
02:50 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 08m 33s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 52s)
00:45 urandom: T169939: Decommissioning Cassandra/restbase2001-b.codfw.wmnet

2017-08-16

22:30 urandom: T169939: Decommissioning Cassandra/restbase2001-a.codfw.wmnet
22:14 urandom: T169939: Cleaning up wikipedia parsoid snapshots
22:13 urandom: T169939: Rolling restart of Cassandra complete
21:38 bawolff: deleting private info from securepoll_votes that the script missed due to ref-integ issues for old elections (T173393)
21:37 thcipriani: train is on hold pending resolution of T173462
21:33 bawolff: deleting private info in enwiki arbcom1_vote table (T173393)
21:30 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack d
21:22 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: revert group1 wikis to 1.30.0-wmf.14 for T173462
21:21 thcipriani@tin: Synchronized php: revert group1 wikis to 1.30.0-wmf.14 for T173462 (duration: 00m 47s)
20:46 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack b
20:45 arlolra@tin: Finished deploy [parsoid/deploy@a9dc803]: Updating Parsoid to 1832a78e (duration: 08m 52s)
20:37 arlolra@tin: Started deploy [parsoid/deploy@a9dc803]: Updating Parsoid to 1832a78e
19:57 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack a
19:52 MaxSem: Manually cleaning up PI on enwiki (T173393)
19:37 thcipriani@tin: Synchronized php: group1 wikis to 1.30.0-wmf.14 (duration: 00m 46s)
19:35 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.14
19:06 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack d
18:57 thcipriani@tin: Synchronized wmf-config/CirrusSearch-common.php: SWAT: cirrus Tune ordering of crossproject search results on enwiki T171803 PART II (duration: 00m 50s)
18:56 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: cirrus Tune ordering of crossproject search results on enwiki T171803 PART I (duration: 00m 51s)
18:41 thcipriani@tin: Synchronized wmf-config: SWAT: Apply token count limits to phrase queries on all wikis T172653 (duration: 00m 53s)
18:30 thcipriani@tin: Synchronized php-1.30.0-wmf.13/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Disable cirrus MLR ab test T171214 (duration: 00m 50s)
18:29 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Disable cirrus MLR ab test T171214 (duration: 00m 51s)
18:20 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: JobQueueEventBus: Enable group1 T163380 (duration: 00m 54s)
18:18 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack c
17:31 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack b
17:07 demon@tin: Synchronized php-1.30.0-wmf.14/extensions/AntiSpoof/SpoofUser.php: T173394 (duration: 00m 51s)
15:54 marostegui: Stop MySQL and shutdown db1078 for HW checks - T173365
15:43 elukey: drop PageContentSaveComplete_5588433_15423246 from db1047 and dbstore1002 (analytics-slaves)
15:41 godog: delete outdated CFs cassandra metrics from graphite2002 and graphite1003
15:35 marostegui: Rename cx_drafts table on db1029 - T172364
15:09 godog: restart varnish on cp1072 to clear mailbox lag
14:56 godog: restart varnish on cp1099 to clear mailbox lag
14:48 godog: restart varnish on cp1074 to clear mailbox lag
14:44 godog: restart varnish on cp1049 to clear mailbox lag
14:38 gilles: Thumbor stress test finished
14:29 gilles: Stress testing Thumbor from single IP
13:57 zeljkof: EU SWAT finished
13:35 zfilipin@tin: Synchronized dblists/pp_stage1.dblist: SWAT: pagePreviews: Deploy to next 100 stage 1 wikis (T162672) (duration: 00m 50s)
13:09 zfilipin@tin: Synchronized portals: (no justification provided) (duration: 00m 52s)
13:08 zfilipin@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 52s)
12:25 Amir1: ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki testwikidatawiki --entity-type=property (T172776, T171460)
12:25 moritzm: powercycling cp3036
12:24 marostegui: Compressing InnoDB on db2077 - T168409
12:22 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: cp3036.esams.wmnet
10:46 marostegui: Stop replication in sync on db1015 and db1078 - T164488
10:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2076 - T170662 (duration: 00m 51s)
10:03 godog: copy ubuntu-font-family-sources to stretch-wikimedia - T170817
08:37 marostegui: Drop wikigrok tables from s1, s3 and s5 - T172020
08:12 marostegui: Stop MySQL on db2047 to copy its content to db2077 - T170662
08:10 moritzm: bounced pdfrender on scb1004 (T159922)
08:07 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 - T170662 (duration: 01m 06s)
07:25 marostegui: Stop MySQL on db2076 to copy its content to dbstore2001 - T168409
07:08 elukey: executed sudo find -type f -mtime +30 -exec rm {} \; in /var/log/carbon to free some space
06:40 marostegui: Run pt-table-checksum on s3 for revision table - T164488
06:01 marostegui: Stop replication on db2076 to fix duplicate entries - T151029
03:35 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 16 03:35:43 UTC 2017 (duration 7m 21s)
03:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 16m 32s)
02:51 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 08m 08s)
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 51s)
00:43 dereckson@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata/Wikidata.php: Explicitly load badges extension (Gerrit:372088) (duration: 00m 51s)
00:42 dereckson@tin: Synchronized php-1.30.0-wmf.14/extensions/Wikidata/Wikidata.php: Explicitly load badges extension (Gerrit:372051) (duration: 00m 51s)
00:38 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Timeless on three French wikis (T154371) + Fixes for Wikidata: Remove wbq_evaluation logging, Update Wikidata property blacklist (Gerrit:367913 and Gerrit:370846) (duration: 00m 53s)
00:02 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner on test wikis (T173388) (duration: 00m 51s)

2017-08-15

23:58 dereckson@tin: Synchronized php-1.30.0-wmf.14/skins/Timeless/resources/screen-common.less: Fix messed up recent changes/watchlist legends (T173151) (duration: 00m 50s)
23:55 dereckson@tin: Synchronized php-1.30.0-wmf.13/skins/Timeless/resources/screen-common.less: Fix messed up recent changes/watchlist legends (T173151) (duration: 00m 54s)
23:14 Dereckson: Ran updateArticleCount.php on sr.wikiquote (T172974)
23:12 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Create a few of namespace aliases for hiwikiversity (T172977) (duration: 00m 53s)
22:50 ppchelko@tin: Started restart [changeprop/deploy@444223d]: (no justification provided)
22:45 Pchelolo: scb - codfw: enabling puppet and starting changeprop back T169939
21:40 mobrovac: scb - codfw: disabling puppet and stopping changeprop to release load on Cassandra while dropping old tables - T169939
21:26 urandom: Starting Cassandra restbase2005-b
19:24 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.14
19:19 mobrovac@tin: Finished deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones), part #2 - T169939 (duration: 02m 50s)
19:16 mobrovac@tin: Started deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones), part #2 - T169939
19:14 demon@tin: Finished scap: bootstrap wmf.14 v2.0 (duration: 30m 56s)
19:04 mobrovac@tin: Started deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones) - T169939
19:03 mobrovac@tin: Finished deploy [restbase/deploy@1139d00] (staging): Use only new parsoid tables (before the removal of old ones) - T169939 (duration: 04m 39s)
18:58 mobrovac@tin: Started deploy [restbase/deploy@1139d00] (staging): Use only new parsoid tables (before the removal of old ones) - T169939
18:43 demon@tin: Started scap: bootstrap wmf.14 v2.0
18:39 demon@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_3982832257" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 03m 05s)
18:35 demon@tin: Started scap: bootstrap wmf.14
18:18 demon@tin: Pruned MediaWiki: 1.30.0-wmf.12 [keeping static files] (duration: 02m 47s)
17:59 bsitzmann@tin: Finished deploy [mobileapps/deploy@34a1304]: Update mobileapps to 33b80dd (T172829 T152441 T172021 T103362) (duration: 04m 00s)
17:55 bsitzmann@tin: Started deploy [mobileapps/deploy@34a1304]: Update mobileapps to 33b80dd (T172829 T152441 T172021 T103362)
15:21 mobrovac: restarting pdfrender on scb1001, added some debug messages to help us diagnose T159922
13:42 gehel: cleanup of old leftover logs on deployment-logstash2:/var/log/logstash
13:21 moritzm: bounced pdfrender on scb1001 (T159922)
11:32 moritzm: installing Linux updates on stretch systems (no reboots yet)
10:45 moritzm: installing PHP security updates on trusty
10:09 jynus: rebooting db1078
09:39 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1078 (duration: 00m 49s)
07:56 moritzm: installing cvs security updates
07:18 elukey: restart pdfrender on scb1003
02:56 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 15 02:56:14 UTC 2017 (duration 6m 42s)
02:49 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 07m 53s)
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 58s)

2017-08-14

22:16 Dereckson: Previous log entry is related to Gerrit:371960 and Gerrit:371961.
22:15 dereckson@tin: Synchronized php-1.30.0-wmf.11/extensions/EventBus/JobQueueEventBus.php: JobQueueEventBus: Populate the database field + not set properties are accessed (duration: 00m 48s)
20:44 ppchelko@tin: Finished deploy [restbase/deploy@4d6c706]: Temporary fallback to the new storage buckets before truncation (duration: 09m 35s)
20:35 ppchelko@tin: Started deploy [restbase/deploy@4d6c706]: Temporary fallback to the new storage buckets before truncation
17:09 gehel: resetting mode on stat1005:/srv/published-datasets/discovery recursively
16:29 elukey: execute sudo find -type f -mtime +60 -exec rm {} \; in /var/log/carbon on graphite2001 to free some space in /
13:32 elukey: Execute systemctl mask nfacctd on rhenium.wikimedia.org for T172681
13:08 zeljkof: nothing for EU SWAT
12:21 jynus: stopping replication on all instances of dbstore2001 T169516
11:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2076 - T170662 T151029 (duration: 00m 47s)
11:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2076 - T170662 T151029 (duration: 01m 02s)
10:34 marostegui: Restart MySQL on db1069 to pick up new replcation filters
10:27 marostegui: Restart MySQL on db1095 to pick up new replication filters
09:16 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=thumbor,name=thumbor2001.codfw.wmnet
09:07 _joe_: stopping thumbor on thumbor2001 after depooling it for testing
09:05 oblivian@puppetmaster1001: conftool action : set/pooled=no; selector: service=thumbor,name=thumbor2001.codfw.wmnet
03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 14 03:15:31 UTC 2017 (duration 7m 6s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 06m 47s)
02:39 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 29s)

2017-08-13

22:45 ebernhardson: restart elsaticsearch on elastic1017 after setting md2 readahead to 256 to match md2 on 1032-152
19:36 godog: upload python-thumbor-wikimedia 1.2 - T161719
19:01 godog: bounce pdfrender on scb1001 and scb1003 - T159922
18:46 godog: bounce carbon and uwsgi on graphite1003
17:53 reedy@tin: Synchronized wmf-config/db-eqiad.php: fix comment (duration: 00m 47s)
17:52 reedy@tin: Synchronized wmf-config/db-codfw.php: fix comment (duration: 00m 47s)
17:51 reedy@tin: Synchronized refresh-dblist: phpcs (duration: 00m 48s)

2017-08-12

20:00 krinkle@tin: Synchronized php-1.30.0-wmf.13/includes/jobqueue/JobQueueGroup.php: T171371 - Log job pushes to bogus wikis (duration: 00m 53s)
16:09 Reedy: Deleted some bogus user languages from commonswiki.user_properties
15:25 elukey: powercycle mw2256 (able to use com2 but not to login as root, regular ssh hanging) - T163346

2017-08-11

21:47 legoktm@tin: Finished scap: (no justification provided) (duration: 07m 09s)
21:41 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw2256.codfw.wmnet
21:40 legoktm@tin: Started scap: (no justification provided)
21:40 legoktm@tin: scap failed: LockFailedError Failed to acquire lock "/var/lock/scap.unknown-but-probably-mediawiki.lock"; owner is "legoktm"; reason is "Deploying Timeless (try 2) - T154371" (duration: 00m 00s)
21:02 ebernhardson: unban elastic1017 from elasticsearch cluster
20:52 bblack: varnish backend restart on cp1099
20:46 legoktm@tin: Started scap: Deploying Timeless (try 2) - T154371
20:42 legoktm@tin: scap aborted: Deploying Timeless - T154371 (duration: 05m 10s)
20:36 legoktm@tin: Started scap: Deploying Timeless - T154371
20:19 bblack: varnish backend restart on cp1049 + cp1074 (mailbox lag)
19:59 ebernhardson: ban elastic1017 from eqiad search cluster
18:28 moritzm: installing subversion security updates
17:19 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2075 (duration: 00m 47s)
17:05 twentyafterfour: Deploying phabricator security update
16:21 jynus_: stop db1069:s6 replication and dropping frwiki, jawiki, ruwiki
15:41 jynus_: stopping db2075 to clone it to dbstore2001
15:39 moritzm: installing git security updates on trusty (jessie/stretch already fixed)
15:38 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 (duration: 00m 48s)
13:51 elukey: moved the eventbus scap deployment dirs on kafka[12]00[123] to deploy-service:deploy-service to allow scap to depool/pool - T171506
13:49 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 27s)
13:49 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
13:42 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 13s)
13:42 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
13:25 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 34s)
13:24 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
13:16 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 12s)
13:15 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
13:00 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 17s)
13:00 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
12:51 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 04s)
12:51 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
12:44 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 05s)
12:44 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
12:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2046 - T151029 (duration: 00m 48s)
11:25 jynus: stopping and upgrading labsdb1010
11:19 marostegui: Stop replication on db2046 to fix duplicate entries - T151029
09:28 jynus: stopping and restarting es2013 for upgrade
09:07 jynus: stopping and restarting db2046 for upgrade
07:35 elukey: restart pdfrender on scb1004

2017-08-10

23:43 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/371196/2 (duration: 00m 47s)
23:26 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/370310/ (duration: 00m 47s)
23:25 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/370310/ (duration: 00m 46s)
22:31 twentyafterfour@tin: Synchronized php-1.30.0-wmf.13/includes/specials/SpecialDoubleRedirects.php: Hopefully fix T173045 (duration: 00m 48s)
22:17 moritzm: installing git security-updates
20:51 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2.5 - T137371 (duration: 00m 21s)
20:51 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2.5 - T137371
20:50 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2 - T137371 (duration: 00m 09s)
20:50 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2 - T137371
20:35 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment - T137371 (duration: 00m 05s)
20:34 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment - T137371
20:00 mobrovac@tin: Finished deploy [cassandra/metrics-collector@5db1a43] (staging): First Scap3 deployment - T137371 (duration: 18m 17s)
19:42 mobrovac@tin: Started deploy [cassandra/metrics-collector@5db1a43] (staging): First Scap3 deployment - T137371
19:35 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 47s)
19:33 reedy@tin: Synchronized wmf-config/: Newsletter on testwiki (duration: 00m 49s)
19:31 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (aqs): first Scap3 deployment - T116340 (duration: 01m 08s)
19:30 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (aqs): first Scap3 deployment - T116340
19:30 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa]: first Scap3 deployment - T116340 (duration: 00m 34s)
19:29 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa]: first Scap3 deployment - T116340
19:28 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment (rest of the nodes) - T116340 (duration: 00m 12s)
19:28 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment (rest of the nodes) - T116340
19:27 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: All wikis (except wikidata) to 1.30.0-wmf.13 refs T170631
19:16 godog: restart cassandra instances on xenon to test logstash-logback-encoder deploy
19:13 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment - T116340 (duration: 00m 03s)
19:13 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment - T116340
18:56 Reedy: created newsletter tables on testwiki
18:55 reedy@tin: Synchronized php-1.30.0-wmf.13/extensions/Newsletter/: sql file updates (duration: 00m 52s)
18:53 reedy@tin: Synchronized php-1.30.0-wmf.13/extensions/WikimediaMaintenance/createExtensionTables.php: newsletter (duration: 00m 52s)
18:21 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Fix commons/commonswiki snafu (duration: 00m 52s)
17:53 kartik@tin: Finished deploy [cxserver/deploy@f43ef96]: (no justification provided) (duration: 00m 44s)
17:52 kartik@tin: Started deploy [cxserver/deploy@f43ef96]: (no justification provided)
17:42 kartik@tin: Finished deploy [cxserver/deploy@1065ffe]: Update cxserver to 686f4f3 (duration: 00m 40s)
17:42 kartik@tin: Started deploy [cxserver/deploy@1065ffe]: Update cxserver to 686f4f3
16:23 ema: cp1072: restart varnish backend
15:48 XioNoX: troubleshoting interface errors between pfw3-codfw and fasw-codfw
15:09 marostegui: Stop replication on db2046 to fix duplicate entries - T151029
15:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2046 - T151029 (duration: 00m 50s)
15:03 marostegui: Poweroff es2013 for maintenance - T172265
14:45 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2075 - T170662 (duration: 00m 51s)
14:32 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2045 - T151029 (duration: 00m 51s)
14:11 elukey: restart kafka1012 temporary with some logs to TRACE to debug T172681
away: updated civicrm from 200abc2 to 4aa177b
13:43 marostegui: Compress cebwiki on db1095 - T153058
13:32 zeljkof: EU SWAT finished
13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgArticleCountMethod to any on srwikisource (T172974) (duration: 00m 53s)
13:18 zeljkof: EU SWAT, part two
13:17 marostegui: Drop m3 databases from dbstore1002 - T156758
13:16 zeljkof: EU SWAT finished
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage on knwiki (T172894) (duration: 00m 52s)
12:56 marostegui: Stop MySQL on db2045 to upgrade socket location - T148507
12:07 elukey: restored varnishakafka on cp3032
11:17 elukey: disabled puppet on cp3032 and restarted varnishkafka with debug logging
09:58 jynus: continuing cloning of dbstore2002 to dbstore2001
09:30 gehel: repooling wdqs2001, long after data reload completed
09:30 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2001.codfw.wmnet
09:27 gehel@tin: Finished deploy [wdqs/wdqs@c186e3e]: (no justification provided) (duration: 01m 31s)
09:25 gehel@tin: Started deploy [wdqs/wdqs@c186e3e]: (no justification provided)
09:17 jynus: disabling lag notification on all s4 replicas
08:59 elukey: update librdkafka1 to 0.9.4.1 on eventlog1001
08:15 elukey: add 50G to carbon lv on graphite1003 and 100G on graphite2002
07:32 jynus: enabling semisymc master replication on db1068
07:29 jynus: disabling semisync slave replication on all SSD hosts on s4
06:45 elukey: powercycle mw2256 - T163346
06:38 elukey: restart pdfrender on scb1004
06:04 Dereckson: Removed 2FA for GoldRingChip account (T172878)
03:23 bblack: cp1008, restbase-dev100[456] have puppet disabled, manual "rm /etc/ssh/userkeys/dzahn"
03:20 bblack: batched cumin puppet agent run on all hosts (not forced)
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 09m 12s)

2017-08-09

22:06 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Get baaack you broken CodeMirror! (duration: 00m 51s)
21:20 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Deploy CodeMirror https://gerrit.wikimedia.org/r/#/c/370943/ (duration: 00m 50s)
21:11 maxsem@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror: https://gerrit.wikimedia.org/r/#/c/370904/ (duration: 00m 51s)
21:05 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 07m 30s)
21:01 godog: add 100G to carbon lv on graphite1003 and graphite2002
20:58 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Rollback on canary
20:54 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy (duration: 07m 12s)
20:51 marostegui: Remove m3 replication from dbstore1002 - T156758
20:47 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy
20:43 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3 (duration: 08m 05s)
20:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2046 - T170662 (duration: 00m 49s)
20:35 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3
20:22 twentyafterfour: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: Actually sync all group1 wikis (except wikidata) to 1.30.0-wmf.13 refs T170631
18:57 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 2
18:39 marostegui: Stop replication on db2045 to fix duplicate keys - T151029
18:36 urandom: Restarting Cassandra, restbase2001-b.codfw.wmnet (schema jankiness)
18:27 urandom: Restarting Cassandra, restbase2001-a.codfw.wmnet (schema jankiness)
18:10 ebernhardson: restart es on elastic1018 to test interleaved numa
17:50 marostegui: Add db2076 to tendril - T170662
17:41 marostegui: Compress innodb on s6 on db2076 - T170662
17:31 legoktm: changed mediawiki/vendor to fast forward only
17:24 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 01m 27s)
17:23 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Rollback on canary
16:35 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation (duration: 04m 46s)
16:30 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation
16:29 urandom: T172384: Upgrading Cassandra in RESTBase dev to 3.11.0-wmf2 (patched to disable use of FastThreadLocal)
16:15 marostegui: Stop mysql on db2046 to copy its content to db2076 - T170662
16:14 elukey: rolling restart of eventstream on scb hosts to deploy https://gerrit.wikimedia.org/r/370793
16:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2046 - T170662 (duration: 00m 50s)
16:08 gehel: config update on logstash, a few logs might be lost during restart - T172713
15:19 XioNoX: removing old pfw related config from cr1-codfw - T171970
14:08 mutante: purging req.urls with "^/resources" from varnish cluster, to fix redirect with cached 404
14:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2075 to s5 - T170662 (duration: 00m 50s)
14:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2075 to s5 - T170662 (duration: 00m 51s)
13:55 marostegui: Reboot db2076 for maintenance - T170662
13:21 marostegui: Stop replication on db2075 for maintenance - T170662
13:18 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats remove confirmed user group (T101983) (duration: 00m 51s)
13:10 gehel@tin: Finished deploy [wdqs/wdqs@6620e0f]: (no justification provided) (duration: 01m 34s)
13:09 gehel@tin: Started deploy [wdqs/wdqs@6620e0f]: (no justification provided)
12:41 gehel: restarting updater on wdqs1001 (real fix coming up soon
10:20 jynus: stopping dbstore2002's mysqls and cloning them to dbstore2001
08:39 jynus: disable puppet on dbstore2001, about to be reimaged
07:47 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1065, db1064, db1070 (duration: 01m 04s)
07:12 jynus: stopping replication in sync between db1069 and db1065, db1044, db1064, db1070
07:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1065, db1064, db1070 (duration: 00m 47s)
06:26 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 51s)
06:11 jynus: stopping db1069 (s7) and db1079 replication in sync
06:03 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085, depool db1079 (duration: 00m 51s)
05:42 jynus: stopping db1069 (s6) and db1085 replication in sync
05:32 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060, depool db1085 (duration: 00m 50s)
04:56 jynus: stopping db1069 (s2) and db1060 replication in sync
04:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 (duration: 00m 58s)
04:48 jynus: restarting pdfrender on scb1001, unresponsive
04:09 jynus: powercycling mw2256
03:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 07m 33s)
02:56 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 07m 45s)
02:38 ebernhardson: restart elasticsearch1017 with niofs store
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 38s)
00:27 ebernhardson: test vm.zone_reclaim_mode=0 on elastic1017

2017-08-08

23:23 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats on WMF wikis to grant and remove 'confirmed' (T101983) (duration: 00m 51s)
21:25 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.30.0-wmf.13 refs T170631
21:08 ppchelko@tin: Finished deploy [restbase/deploy@c16fb6b]: Update summary and licensing information for pageviews API (duration: 08m 31s)
21:05 twentyafterfour@tin: Finished scap: again: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631 (duration: 42m 55s)
20:59 ppchelko@tin: Started deploy [restbase/deploy@c16fb6b]: Update summary and licensing information for pageviews API
20:32 twentyafterfour: updated mediawiki.org changelog for 1.30.0-wmf.13
20:22 twentyafterfour@tin: Started scap: again: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631
19:46 ema: restart varnish backend on cp1074
19:45 godog: bounce thumbor-instances on thumbor1003 to make sure all memory limits are applied
19:44 twentyafterfour@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="test2wiki" --outdir="/tmp/scap_l10n_224168097" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 07m 33s)
19:36 twentyafterfour@tin: Started scap: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631
18:28 XioNoX: frack-codfw moved to new infrastructure
18:27 elukey: re-enabled irc-echo after the puppet shower
18:11 elukey: stop ircecho to avoid puppet shower
16:59 twentyafterfour: Branching mediawiki/master to mediawiki/wmf/1.30.0-wmf.13 refs T170631
16:04 elukey: rolling restart of varnishkafka-webrequest to apply https://gerrit.wikimedia.org/r/#/c/370659/ (puppet automatically restarts)
15:03 XioNoX: starting pfw-codfw migration - T171970
14:19 elukey: restart of all the varnishkafka statsv/eventlogging instances on caching hosts to pick up https://gerrit.wikimedia.org/r/370644 (puppet automatic restarts)
14:16 elukey: set mw2256 pooled=inactive + downtime to allow BIOS upgrade - T163346
14:07 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Add copyright info for Wikidata API (T112606) (duration: 00m 47s)
13:56 gehel: dump of task API for elasticsearch eqiad - T169498
13:54 ladsgroup@tin: Synchronized static/images/project-logos: SWAT: Fix srwiki logos (T150618) (duration: 00m 48s)
13:00 elukey: restart varnishkafka-webrequest with kafka.broker.version.fallback=0.9.0.1 + kafka.api.version.request=false on cp3032 (local test, to rollback remove the lines from /etc/varnishkafka/webrequest.conf)
12:49 Amir1: start of ladsgroup@terbium:~$ /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property --rebuild-all-terms (T172776)
12:32 elukey: restart pdfrender on scb1002
12:20 Amir1: start of ladsgroup@terbium:~$ /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property (T172776)
12:15 gehel: restarting wdqs-updater on wdqs1001
12:14 elukey: stop eventlogging on eventlog1001 to test kafka consumer failures
11:22 Amir1: start of ladsgroup@terbium:~$ timeout 3500s /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item >>/tmp/rebuildTermSqlIndex.log 2>&1 (T171460)
10:12 elukey: update librdkafka1* on notebook100[12] and stat1003
09:09 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 as the main recentchanges/watchlist s6 role (duration: 00m 47s)
08:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 at 50% load (duration: 00m 46s)
08:28 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 with limited load (duration: 00m 46s)
07:53 jynus@tin: Synchronized wmf-config/db-codfw.php: Add db1098 (duration: 00m 47s)
07:41 elukey: stop puppet on cp3032 (cache::text) to set varnishkafka-webrequest logging to debug
07:34 Amir1: stopped the script and re-running without --deduplicate-terms (T171460)
07:21 Amir1: start of ladsgroup@terbium:~$ time mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki --entity-type=property --deduplicate-terms (T171460)
06:12 elukey: alert users with big home directories for stat1005 disk alarms (will erase data later on only if they don't answer)
06:12 elukey: restart pdfrender on scb1003
03:55 mutante: phab1001 /usr/local/bin/community_metrics.sh | /usr/local/bin/project_changes.sh creating stats mails to admins (which failed before) (T163938)
03:25 mutante: phab1001 /srv/phab/tools/public_task_dump.py to create dump, was failed cron due to missing /srv/dumps/
02:53 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 8 02:53:13 UTC 2017 (duration 6m 53s)
02:46 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 06m 59s)
02:36 ejegg: enabled donation queue consumer
02:26 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 25s)
00:46 ejegg: disabled donation queue consumer

2017-08-07

21:28 urandom: T172384: Upgrading Cassandra to 3.11.0-wmf1 in dev environment (build patched to disable in-built heap dumping)
19:51 ejegg: updated CiviCRM from f24ba78 to 200abc2
19:05 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171212 - Turn on CirrusSearch MLR AB test (duration: 00m 46s)
19:00 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: T169498 - Enable max token count for phrase rescore on zh lang wikis (step 2) (duration: 00m 46s)
18:59 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T169498 - Enable max token count for phrase rescore on zh lang wikis (step 1) (duration: 00m 46s)
18:53 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-production.php: T171212 - Update CirrusSearch AB test rescore profiles (duration: 00m 46s)
18:44 ebernhardson@tin: Synchronized wmf-config/Wikibase-labs.php: T112606 - beta only - Add copyright info for Wikidata API (duration: 00m 46s)
18:43 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T172630 - Enable wgMinervaEnableSiteNotice for kowiki (duration: 00m 46s)
18:38 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T170687 - Exclude files from Special:ShortPages on commons (duration: 00m 46s)
18:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/CirrusSearch/: T169498 limit phrase token count, T172464 constant boost ltr queries (duration: 00m 58s)
18:10 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T172594 - Translate sitename for nl.wikinews (duration: 00m 47s)
18:05 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Grant autopatrol to editor in en.wikibooks - T172561 (duration: 00m 47s)
17:57 mutante: phab2001 - re-enabling puppet, but closing firewall for 80/443
17:56 jynus: stopping slave and reparitioning db1098
17:10 marostegui: Restart s7 instance on db1102 to pick up new replication filters - T172693
17:05 gehel@tin: Finished deploy [wdqs/wdqs@da33919]: (no justification provided) (duration: 02m 28s)
17:02 gehel@tin: Started deploy [wdqs/wdqs@da33919]: (no justification provided)
16:44 marostegui: Restart s7 instance on db1069 to pick up new replication filters - T172693
16:37 XioNoX: manually restarted varnish on cp1099
15:10 thcipriani: restarting jenkins for plugin upgrade
14:51 gehel: reducing elasticsearch eqiad concurrent rebalance to 4 (from 8)
14:38 elukey: updated librdkafka1 and ++1 to 0.9.4.1 on hafnium
14:32 mutante: phab2001 - stopping Apache,schedule downtime for http and puppet
14:22 herron: mx[1,2]001, fermium: Installed libmail-dkim-perl and restarted spamassassin service - T172689
13:15 jynus: reboot db1098
12:39 _joe_: restarting pdfrender on scb1001, T159922
12:39 elukey: restart kafka on kafka1018 to force it out of the kafka topic leaders - T172681
12:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2074 - T171321 (duration: 00m 45s)
12:08 gehel: deploying https://gerrit.wikimedia.org/r/#/c/299825/ - some logs will be lost during logstash restart
10:02 marostegui: Add dbstore2002:3313 to tendril - T171321
09:47 jynus: stopping db1050's mysql and cloning it to db1089
09:06 elukey: set net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 (was 120) on all the analytics kafka brokers - T136094
09:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 after fixing: linter, page and watchlist tables (duration: 00m 47s)
08:12 marostegui: Force BBU re-learn on db1016 - T166344
07:02 marostegui: Stop replication on db2065 to reimport: page, linter and watchlist tables
07:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 to reimport: page, linter and watchlist tables (duration: 00m 47s)
06:38 marostegui: Stop MySQL on db2074 - T171321
06:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2074 - T171321 (duration: 00m 46s)
06:33 marostegui: Stop replication on db2075 - T170662
06:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2073 - T171321 (duration: 00m 47s)
06:20 marostegui: Force BBU re-learn on db1016 - T166344
02:57 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 7 02:57:42 UTC 2017 (duration 6m 42s)
02:51 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 07m 56s)
02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 10m 16s)

2017-08-06

13:17 elukey: powercycle mw2256 - com2 frozen - T163346
13:13 elukey: restart pdfrender on scb1002
06:18 ebernhardson@tin: Synchronized wmf-config/PoolCounterSettings.php: T169498: Reduce cirrus search pool counter to 200 parallel requests cluster wide (duration: 02m 54s)
01:28 chasemp: conf2002:~# service etcdmirror-conftool-eqiad-wmnet restart (not sure what else to do the service failed)

2017-08-05

14:40 Reedy: created oauth tables on foundationwiki T172591
14:13 reedy@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaMaintenance/createExtensionTables.php: add oauth (duration: 00m 48s)

2017-08-04

23:51 mutante: phab2001 - removed outdated /etc/hosts entries, that fixed rsync, syncing /srv/repos/ from phab1001
23:35 mutante: phab2001 rebooting
23:35 mutante: phab2001 - installing various package upgrades, apt-get autoremove old kernel images
23:12 mutante: "reserved" UID 498 for phd on https://wikitech.wikimedia.org/wiki/UID | phab2001: find -exec chown to fix all the files , restart cron
23:04 mutante: phab2001 - changing UID/GID for phd user from 997:997 to 498:498 to make it match phab1001, to fix rsync breaking permissions. (rsync forces --numeric-ids when fetching from and rsyncd configured with chroot=yes). chown -R phw:www-data /srv/repos/
22:37 ejegg: restarted donations and refund queue consumers
21:44 ejegg: stopped donations and refund queue consumers
21:24 urandom: T172384: Disabling Puppet in dev environment to prevent unattended Cassandra restarts
20:19 mutante: renewing SSL cert for status.wm.org (just like wikitech-static, but that one didnt have monitoring?)
20:02 mutante: wikitech-static-ord - apt-get install certbot
19:41 ejegg: updated CiviCRM from f1fd7f0 to f24ba78
19:38 mutante: renaming graphite varnish director/fixing config, running puppet on cache misc, tested on cp1045
18:17 andrewbogott: switched most cloud instance to new puppetmasters, as per https://phabricator.wikimedia.org/T171786
11:46 marostegui: Deploy schema change directly on s3 master for maiwikimedia - T172485
11:30 marostegui: Deploy schema change directly on s3 master for kbpwiki - T172485
11:14 marostegui: Deploy schema change directly on s3 master for dinwiki - T172485
10:14 marostegui: Deploy schema change directly on s3 master for atjwiki - T172485
10:05 marostegui: Stop replication on db2073 for maintenance
09:22 marostegui: Add dbstore2002 to tendril - T171321
09:19 marostegui: Deploy schema change directly on s3 master for techconductwiki - T172485
08:35 marostegui: Deploy schema change directly on s3 master for hiwikiversity - T172485
08:19 marostegui: Deploy schema change directly on s3 master for wikimania2018wiki - T172485
08:04 marostegui: Sanitize wikimania2018wiki on sanitarium and sanitarium2 - T155041
07:47 marostegui: Stop MySQL on db2073 to copy its data to dbstore2002 - https://phabricator.wikimedia.org/T171321
07:47 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2073 - T171321 (duration: 00m 47s)
07:07 moritzm: installing imagemagick regression security updates on trusty
06:47 marostegui: Sanitize hiwikiversity on sanitarium and sanitarium2 - T171829
05:23 mutante: phab1001 sudo ip addr del 10.64.32.186/32 dev eth0 (T172478)
02:28 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=phab1001-vcs.eqiad.wmnet
02:28 dzahn@neodymium: conftool action : set/pooled=no; selector: name=phab1001-vcs.eqiad.wmnet
02:15 mutante: phab1001 can't talk to mx servers via IPv6, but works via IPv4. iridium and other mailservers can also talk IPv6 to it. why? it did not change even when stopping ferm on client and on server it allows from anywhere. workaround for now was to hardcode IPv4 IP in phab config. (T163938)
02:13 twentyafterfour: outgoing phab mail is working again
01:16 twentyafterfour: twentyafterfour@phab1001:/srv/repos$ sudo chown -R phd:www-data /srv/repos
01:16 twentyafterfour: twentyafterfour@phab1001:/srv/repos$ sudo chmod -x /usr/local/sbin/sync-srv-repos
01:12 mutante: phab1001 - stopped and started exim, which is now running with same options as iridium
00:53 twentyafterfour: starting phd to shut up icinga
00:40 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Turn CirrusSearch MLR test back off (duration: 00m 47s)
00:35 twentyafterfour: phabricator web service is also up
00:34 mutante: phab1001 - service IPs switched - puppet ran - ssh-phab service up
00:30 twentyafterfour: phab1001 chown -R phd:phd /srv/repos
00:28 twentyafterfour: testing phd on phab1001
00:24 mutante: phab1001 sudo ip addr add 2620:0:861:103:10:64:32:186/128 dev eth0
00:24 mutante: phab1001 sudo ip addr add 10.64.32.186/32 dev eth0
00:23 mutante: iridium sudo ip addr del 2620:0:861:ed1a::3:16/128 dev lo
00:23 mutante: iridiium sudo ip addr del 208.80.154.250/32 dev lo
00:23 mutante: iridium sudo ip addr del 2620:0:861:103:10:64:32:186/128 dev eth0
00:23 mutante: iridium sudo ip addr del 10.64.32.186/32 dev eth0
00:07 twentyafterfour: stoped phd and apache on iridium
00:07 twentyafterfour: stopped ssh-phab on iridium
00:03 mutante: phab1001 - /usr/bin/rsync -av rsync://iridium.eqiad.wmnet/srv-repos /srv/repos/
00:02 twentyafterfour: Taking phabricator down for maintenance / migration to a new server: phab1001

2017-08-03

23:43 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171212: Turn on CirrusSearch MLR AB test (duration: 00m 46s)
23:36 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/370117/2 (duration: 00m 47s)
23:14 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror/resources/ext.CodeMirror.js: Only show popup if CodeMirror button exists (duration: 00m 46s)
23:12 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/extension.json: (no justification provided) (duration: 00m 47s)
23:08 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-production.php: CirrusSearch config for MLR test (step 2) (duration: 00m 46s)
23:07 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: CirrusSearch config for MLR test (step 1) (duration: 00m 47s)
22:06 ejegg: restarted donation and refund queue consumers
21:51 maxsem@tin: Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/370096/ (duration: 00m 47s)
21:51 ejegg: enabled ingenico_audit drupal module
21:45 ejegg: updated civicrm from 5c741b1 to f1fd7f0
21:43 ejegg: disabled donation and refund queue consumers
21:31 chasemp: clear out nova-fullstack project so we can monitor fresh on new puppetmaster
21:30 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Enable CodeMirror https://gerrit.wikimedia.org/r/#/c/370072/ (duration: 00m 47s)
21:20 maxsem@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror: https://gerrit.wikimedia.org/r/#/c/370066/ https://gerrit.wikimedia.org/r/#/c/370074/ (duration: 00m 47s)
21:04 herron: copper /var/cache/pbuilder 95% full - grew /dev/copper-vg/pbuilder fs by +5G and tune2fs -m 0. now at 85% full
20:40 milimetric@tin: Finished deploy [analytics/refinery@cc40bf2]: Fix sqoop script with updated scap config (duration: 11m 51s)
20:29 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis except wikidata to 1.30.0-wmf.12, leaving wikidata behind due to T172320
20:28 milimetric@tin: Started deploy [analytics/refinery@cc40bf2]: Fix sqoop script with updated scap config
20:24 milimetric@tin: Finished deploy [analytics/refinery@cc40bf2]: Fix sqoop script (duration: 33m 17s)
19:51 milimetric@tin: Started deploy [analytics/refinery@cc40bf2]: Fix sqoop script
19:21 twentyafterfour@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata/extensions/Wikibase/client/includes/Changes: be sure https://gerrit.wikimedia.org/r/#/c/369847/ is sync'd (duration: 00m 46s)
19:12 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.12 refs T168053
19:08 twentyafterfour: deploying 1.30.0-wmf.12 to group1, will proceed to group2 after verifying the branch is stable.
18:59 arlolra: Updated Parsoid to 6e1a20d5 (T168765, T155038, T165977)
18:53 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Final gerrit 369960: "pagePreviews: Deploy to first 50 of stage 1 wikis" (duration: 00m 46s)
18:52 arlolra@tin: Finished deploy [parsoid/deploy@48be65b]: Updating Parsoid to 6e1a20d5 (duration: 06m 25s)
18:51 demon@tin: Synchronized wmf-config/CommonSettings.php: new dblist for gerrit 369960 (duration: 00m 46s)
18:46 arlolra@tin: Started deploy [parsoid/deploy@48be65b]: Updating Parsoid to 6e1a20d5
18:41 demon@tin: Synchronized docroot/: gerrit 369960 (duration: 00m 47s)
18:39 demon@tin: Synchronized dblists/: gerrit 369960 (duration: 00m 47s)
18:29 demon@tin: Synchronized wmf-config/InitialiseSettings.php: last part of gerrit 351287 (duration: 00m 47s)
18:23 mobrovac@tin: Finished deploy [restbase/deploy@65af18d]: Expose the recommendation API publicly and activate hiwikiversity - T170877 T168765 (duration: 08m 33s)
18:17 demon@tin: Synchronized wmf-config/CommonSettings.php: Add new dblist for page preview stuff, basically no-op (duration: 00m 47s)
18:15 demon@tin: Synchronized docroot/noc/conf/pp_stage0.dblist: tidy up pp config (duration: 00m 46s)
18:15 mobrovac@tin: Started deploy [restbase/deploy@65af18d]: Expose the recommendation API publicly and activate hiwikiversity - T170877 T168765
18:14 demon@tin: Synchronized dblists/pp_stage0.dblist: clean up config (duration: 00m 47s)
18:11 mobrovac@tin: Finished deploy [restbase/deploy@65af18d] (staging): (no justification provided) (duration: 01m 57s)
18:09 mobrovac@tin: Started deploy [restbase/deploy@65af18d] (staging): (no justification provided)
18:03 arlolra: Updated Parsoid to 651f12c2 (T119802, T73386, T170289)
17:55 andrewbogott: restarting rabbitmq-server on labcontrol1001
17:50 arlolra@tin: Finished deploy [parsoid/deploy@612e711]: Updating Parsoid to 651f12c2 (duration: 11m 17s)
17:39 arlolra@tin: Started deploy [parsoid/deploy@612e711]: Updating Parsoid to 651f12c2
16:15 jynus: repointing dbproxy1008 back to db1043
16:06 demon@tin: Finished deploy [gerrit/gerrit@15f1544]: This is a test, disregard (duration: 00m 03s)
16:05 demon@tin: Started deploy [gerrit/gerrit@15f1544]: This is a test, disregard
16:03 demon@tin: Synchronized wmf-config/: doc fixes, no-op (duration: 00m 48s)
16:02 demon@tin: Synchronized scap/plugins/prep.py: doc fixes, no-op (duration: 00m 47s)
16:00 godog: silence cassandra-related alerts on restbase-dev cluster, known OOMs
15:44 urandom: T172384: lower tombstone failure threshold in RESTBase dev to 1000
15:39 reedy@tin: Synchronized wmf-config/interwiki.php: Update for 3 new wikis (duration: 00m 46s)
15:31 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: techconductwiki
15:27 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: techconductwiki (duration: 00m 46s)
15:25 reedy@tin: Synchronized static/images/project-logos/: techconductwiki (duration: 00m 44s)
15:24 reedy@tin: Synchronized dblists/: techconductwiki (duration: 00m 47s)
15:03 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic1028.eqiad.wmnet
15:02 gehel: unbanning and repooling elastic1028 - T168816
14:58 reedy@tin: Synchronized static/images/project-logos/: hiwikiversity (duration: 00m 46s)
14:57 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: hiwikiversity (duration: 00m 48s)
14:55 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: hiwikiversity
14:55 reedy@tin: Synchronized dblists: hiwikiversity (duration: 00m 49s)
14:53 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(29|30|31|32).eqiad.wmnet
14:53 Dereckson: mwscript initSiteStats.php --wiki srwikiquote --update (T172241)
14:52 gehel: unbanning and repooling elastic10(29|30|31|32) - T168816
14:29 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: wikimania2018wiki (duration: 00m 47s)
14:28 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: (no justification provided)
14:27 reedy@tin: Synchronized dblists: wikimania2018wiki (duration: 00m 47s)
14:21 marostegui: Restart MySQL on db2019 to get the new socket location updated
14:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2045 - T170662 (duration: 00m 46s)
14:06 marostegui: Compress s5 on db2075 - T170662
14:02 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(28|29|30|31|32).eqiad.wmnet
14:01 gehel: depooling and shutting down elastic10(28|29|30|31|32) - T168816
14:00 marostegui: Add db2075 to tendril - T170662
13:44 marostegui: Enable gtid back on codfw s4 slaves - T170351
13:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Promote db2051 as s4 codfw master - T170351 (duration: 00m 46s)
13:28 elukey: drop CookieBlock backup tables for T171883
13:26 marostegui: Starting the actual s4 codfw failover db2019 -> db2051 - T170351
13:22 gehel: banning elastic1032 - T168816
13:19 chasemp: disabling puppet acros wmcs things for testing
13:14 marostegui: Start topology change for s4 in codfw, slaves will be moved under db2051 - T170351
13:10 aude@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata: Fix bug in InjectRCRecordsJob (duration: 02m 12s)
13:01 marostegui: Disable gtid on s4 codfw slaves to get ready for the topology change - T170351
12:15 marostegui: Restart MySQL on db2051 - T170351
10:21 elukey: removed /run/pacct_shadow.d on stat1005 to allow atopacct.service restart
10:01 moritzm: installing poppler security updates on trusty
09:57 _joe_: systemctl reset-failed puppetmaster.service on labpuppetmaster1002
09:52 moritzm: installing mysql 5.5 security updates (package as shipped by Debian jessie, not our internal wmf-mariadb package)
09:32 gehel: banning elastic103[01] - T168816
08:56 ema: upgrading cache_upload to varnish 4.1.8-1wm1
08:36 gehel: banning elastic102[89] - T168816
08:35 marostegui: Stop MySQL on db2045 to copy its data to db2075 - T170662
08:34 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2045 - T170662 (duration: 00m 47s)
08:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db2045 - T170662 (duration: 00m 47s)
07:58 marostegui: Add db2073 and db2074 to tendril - T170662
06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2074 to s3 - T170662 (duration: 00m 46s)
06:18 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2074 to s3 - T170662 (duration: 00m 46s)
06:01 marostegui: Compress s7 on db1102 - T172169
05:57 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2051 - T170351 (duration: 00m 54s)
05:27 marostegui: Deploy alter table on enwiki - labsdb1010 - T166204
05:17 marostegui: Stop replication on labsdb1011 for maintenance - T153743
03:09 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 3 03:09:22 UTC 2017 (duration 7m 18s)
03:02 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 05m 47s)
02:33 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 07m 53s)
01:40 twentyafterfour: repositories synced for phabricator migration.
00:28 reedy@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Simply ores (duration: 00m 47s)
00:26 twentyafterfour: scheduled downtime for phabricator migration
00:12 mutante: phab1001 starting ferm service
00:11 mutante: iridium restarted ferm - rsync fragment was in config but not applied, breaking data rsync to phab1001
00:05 mutante: rsyncing /srv/repos from iridium to phab1001 (T163938)

2017-08-02

23:53 ebernhardson@tin: Finished scap: dblists/rtl.dblist T172305: Adding RTL database list for project with default RTL languages (duration: 36m 21s)
23:17 ebernhardson@tin: Started scap: dblists/rtl.dblist T172305: Adding RTL database list for project with default RTL languages
21:48 ejegg: updated SmashPig from f4ca53c to c501f53
20:24 bsitzmann@tin: Finished deploy [mobileapps/deploy@3b61ced]: Update mobileapps to 2d8e8f6 (T170325) (duration: 05m 38s)
20:19 bsitzmann@tin: Started deploy [mobileapps/deploy@3b61ced]: Update mobileapps to 2d8e8f6 (T170325)
19:50 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.11 refs T168053 - rollback due to T172320
19:24 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.12 refs refs T168053
19:23 twentyafterfour: group1 wikis to 1.30.0-wmf.12 refs T168053
18:41 thcipriani@tin: Synchronized php-1.30.0-wmf.12/includes/specials/SpecialRecentchanges.php: SWAT: Follow-up 31be7d0: send tags list if experimental mode is disabled (duration: 00m 47s)
18:21 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: beta-only change Enable HTML5 sections in betalabs (duration: 00m 46s)
18:20 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(24|25|26|27).eqiad.wmnet
18:18 thcipriani@tin: Synchronized php-1.30.0-wmf.12/includes/specials/SpecialUndelete.php: SWAT: Fix Special:Undelete search - use variable and not request param (duration: 00m 46s)
18:18 gehel: un-banning and repooling elastic102[4567] - T168816
18:13 thcipriani@tin: Synchronized wmf-config/jobqueue.php: SWAT: JobQueueEventBus: Enable job events in group0 wikis T163380 Part II (duration: 00m 47s)
18:11 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: JobQueueEventBus: Enable job events in group0 wikis T163380 Part I (duration: 00m 47s)
17:56 ottomata: restart kafka1012 broker with listeners=PLAINTEXT://:9092 to verify https://gerrit.wikimedia.org/r/#/c/356232/ before merge. This should be a functional no-op
17:44 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963, take #2 (duration: 00m 16s)
17:44 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963, take #2
17:44 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963 (duration: 00m 29s)
17:43 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963
17:26 mobrovac@tin: Finished deploy [recommendation-api/deploy@8dfae34]: (no justification provided) (duration: 01m 36s)
17:25 mobrovac@tin: Started deploy [recommendation-api/deploy@8dfae34]: (no justification provided)
17:18 herron: installed libmail-spf-perl and restarted spamassassin service on mx[1,2]001 - T172299
16:45 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(24|25|26|27).eqiad.wmnet
16:43 gehel: depooling and shutting down elastic102[4567] - T168816
15:34 mobrovac@tin: Finished deploy [cxserver/deploy@cf3e280]: (no justification provided) (duration: 00m 22s)
15:34 mobrovac@tin: Started deploy [cxserver/deploy@cf3e280]: (no justification provided)
15:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change db2051 IP - T169501 (duration: 00m 46s)
15:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db2051 IP - T169501 (duration: 00m 46s)
15:17 nschaaf@tin: Finished deploy [recommendation-api/deploy@baa11c0]: source parameter validation (duration: 03m 22s)
15:14 nschaaf@tin: Started deploy [recommendation-api/deploy@baa11c0]: source parameter validation
15:06 marostegui: Poweroff db2051 to get it move to another rack - T169501
14:46 ema: upgrading cache_misc to varnish 4.1.8-1wm1
14:31 ema: varnish 4.1.8-1wm1 (fixes VSV00001, DSA 3924-1) built and uploaded to apt.w.o
14:12 marostegui: Stop MySQL on db2051 in order to get it ready to move to another rack - T170351
13:16 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Write to term_full_entity_id column in wb_terms table in prod too - T167229 (duration: 00m 46s)
13:07 hashar@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata: fix constraint type checks - T169326 (duration: 02m 13s)
12:59 gehel: banning elastic102[67] - T168816
12:41 gehel: banning elastic102[45] - T168816
11:42 jynus: disabling autolearn on newest db1* hosts
11:39 jynus: disabling autolearn on newest db2* hosts
11:33 marostegui: Stop MySQL on db2051 for maintenance - T170351
11:29 jynus: setting all es hosts with disabled auto-learn bbu properties
11:22 kartik@tin: (no justification provided)
11:21 kartik@tin: (no justification provided)
11:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2051 - T170351 (duration: 00m 46s)
11:18 kartik@tin: Finished deploy [cxserver/deploy@cf3e280]: Update cxserver to fe03ad7 (duration: 00m 56s)
11:17 kartik@tin: Started deploy [cxserver/deploy@cf3e280]: Update cxserver to fe03ad7
11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool es2013 (duration: 00m 47s)
11:02 hashar: Upgraded tox on CI to 2.5.0
10:49 marostegui: Force a re-learn cycle on es2013
10:18 marostegui: Rename wikigrok tables on enwiki on db1089 - T172020
09:03 marostegui: Drop table click_tracking_user_properties wherever it exists - T115982
09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2057 - T170662 (duration: 00m 46s)
08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2073 to s4 - T170662 (duration: 00m 47s)
08:50 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2073 to s4 - T170662 (duration: 00m 57s)
08:32 marostegui: Drop table click_tracking wherever it exists - T115982
08:15 godog: bounce pdfrender on scb1001, stuck
07:09 marostegui: Disable BBU auto-learn on es2013
06:58 marostegui: Stop MySQL on db2057 for maintenance - T148507
06:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 - T166204 (duration: 00m 45s)
05:47 demon@tin: Synchronized dblists/: No-op (duration: 00m 47s)
02:59 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 2 02:59:33 UTC 2017 (duration 7m 3s)
02:52 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 05m 54s)
02:36 eileen: update process-control to 3d3978a
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 07m 35s)

2017-08-01

23:47 reedy@tin: Synchronized wmf-config/CommonSettings.php: Make Babel use databasey stuff T145366 (duration: 00m 46s)
23:46 eileen: update process control to33a0262d87aa1512888c82ac6b4ed91de3d3cac5
23:44 reedy@tin: Synchronized wmf-config/CommonSettings.php: wfLoadExtension Scribunto (duration: 00m 46s)
23:43 reedy@tin: Synchronized wmf-config/extension-list: json! (duration: 00m 46s)
23:41 reedy@tin: Synchronized wmf-config/CommonSettings.php: Unbreak ContactPage T172199 (duration: 00m 46s)
23:39 reedy@tin: Synchronized wmf-config/MetaContactPages.php: Unbreak ContactPage T172199 (duration: 00m 46s)
23:36 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: T167071 (duration: 00m 47s)
23:30 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: wikinews - T172211 (duration: 00m 46s)
23:26 reedy@tin: Synchronized php-1.30.0-wmf.12/resources/src/mediawiki.rcfilters/mw.rcfilters.Controller.js: T172156 T171514 (duration: 00m 46s)
23:24 reedy@tin: Synchronized static/images/project-logos/: optimise pngs (duration: 00m 46s)
23:23 eileen: update process_control to ecb6669
23:23 reedy@tin: Synchronized static/apple-touch/: optimise pngs (duration: 00m 46s)
23:21 reedy@tin: Synchronized docroot/noc/css/images/: optimise pngs (duration: 00m 47s)
23:13 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Disable popups on Special pages T170893 (duration: 00m 47s)
23:09 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: New wordmarks T168203 T171769 (duration: 00m 47s)
23:08 reedy@tin: Synchronized static/images/mobile/copyright/: 2 new workmarks (duration: 00m 48s)
22:57 chasemp: push procs off swap for labcontrol1001 w/ swapoff -a & swapon -a
22:20 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/369434/3 (duration: 00m 47s)
22:20 eileen: tools updated from 58bcbf3 to 9554229
22:17 eileen: update process_control to c198b7f
21:48 demon@tin: Started deploy [gerrit/gerrit@15f1544]: Initial deploy x2
21:47 demon@tin: Finished deploy [gerrit/gerrit@15f1544]: Initial deploy (duration: 00m 40s)
21:47 demon@tin: Started deploy [gerrit/gerrit@15f1544]: Initial deploy
21:46 demon@tin: Started deploy [gerrit/gerrit@15f1544]: (no justification provided)
21:45 demon@tin: Finished deploy [gerrit/gerrit@15f1544]: (no justification provided) (duration: 00m 06s)
21:45 demon@tin: Started deploy [gerrit/gerrit@15f1544]: (no justification provided)
21:44 demon@tin: Started deploy [gerrit/gerrit@15f1544]: (no justification provided)
20:40 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.30.0-wmf.12 refs refs T168053
19:33 twentyafterfour@tin: Finished scap: Sync 1.30.0-wmf.12 and build l10n refs T168053 (duration: 29m 58s)
19:19 urandom: restarting Cassandra, restbase1004-a.eqiad.wmnet, aberrant read latency
19:03 twentyafterfour@tin: Started scap: Sync 1.30.0-wmf.12 and build l10n refs T168053
18:49 twentyafterfour@tin: Finished deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment (duration: 00m 02s)
18:49 twentyafterfour@tin: Started deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment
18:47 twentyafterfour@tin: Finished deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment (duration: 00m 16s)
18:47 twentyafterfour@tin: Started deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment
18:43 twentyafterfour@tin: Finished deploy [phabricator/deployment@3d728e1]: (no justification provided) (duration: 00m 48s)
18:42 twentyafterfour@tin: Started deploy [phabricator/deployment@3d728e1]: (no justification provided)
17:23 twentyafterfour: MediaWiki train for 1.30.0-wmf.12 - finished `scap prep` & `scap patch` refs T168053
16:41 ejegg: updated CiviCRM from 23f2bbf to 5c741b1
16:25 twentyafterfour: MediaWiki Train: Creating new branch wmf/1.30.0-wmf.12 from master. See T168053 for deployment blockers.
16:17 dcausse: restarting elastic on relforge100x servers to pick up new version of the plugins
15:54 bblack: varnish backend restart on cp1072 (mailbox lag)
15:43 bblack: rebooting lvs1002
15:42 marostegui: db1069: Migrate trwiktionary.page from TokuDB to InnoDB
15:40 bblack: stopping pybal on lvs1002 for impending reboot
15:39 bblack: stopping pybal on 1002 for impending reboot
15:33 marostegui: Stop s3 on db1069 - replication stuck
15:17 marostegui: Stop MySQL on db1055 for maintenance - https://phabricator.wikimedia.org/T148507
14:56 andrewbogott: rebooting labvirt1016
14:46 marostegui: Deploy InnoDB compression on s3 - db2074 for the following tables (revision, pagelinks and templatelinks) - T170662
14:45 ema: lvs1001-1003 (eqiad primaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
14:28 ema: lvs1004-1006 (eqiad secondaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
14:27 bblack: restart varnish backend on cp1049 (mailbox lag)
14:12 bblack: restart varnish backend on cp1074 (mailbox lag)
13:53 ema@neodymium: conftool action : set/pooled=yes; selector: name=achernar.wikimedia.org,service=pdns_recursor
13:26 ema@neodymium: conftool action : set/pooled=no; selector: name=achernar.wikimedia.org,service=pdns_recursor
13:10 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable OOjs UI EditPage on all wikis except Commons (duration: 00m 44s)
13:06 marostegui: Compress s2 on db1102 - T172169
12:49 elukey: restart hive daemons on analytics1003 to pick up new jvm settings (bigger Xmx, JMX ports)
12:06 dcausse: 100% cpu spike on elastic1023 caused percentiles to jump for a short period of time (T169498)
12:04 elukey: stop eventlogging_sync on analytics-slaves && rename all CookieBlock* tables (log db) to CookieBlock*_backup - T171883
11:52 marostegui: Stop MySQL on db2057 to copy its data to db2074 - T170662
11:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2057 - T170662 (duration: 00m 43s)
11:10 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 - T170662 (duration: 00m 43s)
10:31 hashar: Enabling Zuul/CI again and reenabling puppet on contint1001
10:24 hashar: contint1001 stopped puppet agent to prevent Zuul server to come back up
10:12 hashar: Stopped Zuul / CI for mass mediawiki extension changes
08:55 ema: lvs2001-2003 (codfw primaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
08:32 ema: lvs2004-2006 (codfw secondaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
08:03 ema: lvs3*: upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
07:40 ema: lvs4001, lvs4002 (ulsfo primaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
07:35 ema: lvs4003, lvs4004 (ulsfo secondaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
07:33 ema: pybal 1.13.11 uploaded to apt.w.o T103882
06:20 marostegui: Stop MySQL on db2065 to copy its data to db2073 - T170662
06:13 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 - T170662 (duration: 00m 43s)
05:21 marostegui: Restart MySQL on labsdb1003 as it is totally stuck
03:50 mobrovac@tin: Finished deploy [restbase/deploy@0d12138]: Add nl.wikinews - T171897 (duration: 07m 57s)
03:42 mobrovac@tin: Started deploy [restbase/deploy@0d12138]: Add nl.wikinews - T171897
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 1 02:32:17 UTC 2017 (duration 6m 36s)
02:25 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 07m 46s)