Jump to content

Server Admin Log/Archive 59

From Wikitech

2022-11-15

  • 23:54 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudmetrics[1001-1002].eqiad.wmnet
  • 23:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 23:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 23:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T318605)', diff saved to https://phabricator.wikimedia.org/P39860 and previous config saved to /var/cache/conftool/dbconfig/20221115-234056-ladsgroup.json
  • 23:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 23:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 23:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T321130)', diff saved to https://phabricator.wikimedia.org/P39859 and previous config saved to /var/cache/conftool/dbconfig/20221115-233253-marostegui.json
  • 23:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1118 (T318605)', diff saved to https://phabricator.wikimedia.org/P39858 and previous config saved to /var/cache/conftool/dbconfig/20221115-232600-ladsgroup.json
  • 23:25 brennen@deploy1002: Finished scap: Backport for Feed: Use DerivativeContext and not clone main RequestContext (T323153) (duration: 06m 26s)
  • 23:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P39857 and previous config saved to /var/cache/conftool/dbconfig/20221115-232550-ladsgroup.json
  • 23:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 23:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 23:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T318605)', diff saved to https://phabricator.wikimedia.org/P39856 and previous config saved to /var/cache/conftool/dbconfig/20221115-232532-ladsgroup.json
  • 23:21 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@7762e35]: import_cirrus_indexes: snapshot partitioning should not use dashes (duration: 02m 16s)
  • 23:19 brennen@deploy1002: brennen and tstarling: Backport for Feed: Use DerivativeContext and not clone main RequestContext (T323153) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 23:19 brennen@deploy1002: Started scap: Backport for Feed: Use DerivativeContext and not clone main RequestContext (T323153)
  • 23:18 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@7762e35]: import_cirrus_indexes: snapshot partitioning should not use dashes
  • 23:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P39855 and previous config saved to /var/cache/conftool/dbconfig/20221115-231746-marostegui.json
  • 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P39854 and previous config saved to /var/cache/conftool/dbconfig/20221115-231043-ladsgroup.json
  • 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P39853 and previous config saved to /var/cache/conftool/dbconfig/20221115-231025-ladsgroup.json
  • 23:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P39852 and previous config saved to /var/cache/conftool/dbconfig/20221115-230240-marostegui.json
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T318605)', diff saved to https://phabricator.wikimedia.org/P39851 and previous config saved to /var/cache/conftool/dbconfig/20221115-225537-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P39850 and previous config saved to /var/cache/conftool/dbconfig/20221115-225518-ladsgroup.json
  • 22:52 ejegg: re-enabled civicrm dedupe jobs
  • 22:51 mutante: phab1004 - running public_task_dump.py
  • 22:51 ejegg: civicrm upgraded from d85589e8 to fa71f219
  • 22:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T321130)', diff saved to https://phabricator.wikimedia.org/P39849 and previous config saved to /var/cache/conftool/dbconfig/20221115-224733-marostegui.json
  • 22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T318605)', diff saved to https://phabricator.wikimedia.org/P39848 and previous config saved to /var/cache/conftool/dbconfig/20221115-224011-ladsgroup.json
  • 22:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T321130)', diff saved to https://phabricator.wikimedia.org/P39847 and previous config saved to /var/cache/conftool/dbconfig/20221115-221247-marostegui.json
  • 22:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 22:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 22:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T321130)', diff saved to https://phabricator.wikimedia.org/P39846 and previous config saved to /var/cache/conftool/dbconfig/20221115-221225-marostegui.json
  • 21:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P39845 and previous config saved to /var/cache/conftool/dbconfig/20221115-215719-marostegui.json
  • 21:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P39844 and previous config saved to /var/cache/conftool/dbconfig/20221115-214212-marostegui.json
  • 21:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T321126)', diff saved to https://phabricator.wikimedia.org/P39843 and previous config saved to /var/cache/conftool/dbconfig/20221115-214022-marostegui.json
  • 21:33 cjming: end of UTC late backport window
  • 21:27 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:27 cjming@deploy1002: Finished scap: Backport for tnwiki: Set timezone to Africa/Gaborone (UTC+2) (T318208) (duration: 06m 14s)
  • 21:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T321130)', diff saved to https://phabricator.wikimedia.org/P39842 and previous config saved to /var/cache/conftool/dbconfig/20221115-212706-marostegui.json
  • 21:25 cmooney@cumin1001: START - Cookbook sre.dns.netbox
  • 21:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39841 and previous config saved to /var/cache/conftool/dbconfig/20221115-212516-marostegui.json
  • 21:21 cjming@deploy1002: cjming and stang: Backport for tnwiki: Set timezone to Africa/Gaborone (UTC+2) (T318208) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 21:21 cjming@deploy1002: Started scap: Backport for tnwiki: Set timezone to Africa/Gaborone (UTC+2) (T318208)
  • 21:19 cjming@deploy1002: Finished scap: Backport for logos: Remove duplicated code (T307705) (duration: 04m 31s)
  • 21:15 cjming@deploy1002: cjming and stang: Backport for logos: Remove duplicated code (T307705) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 21:14 cjming@deploy1002: Started scap: Backport for logos: Remove duplicated code (T307705)
  • 21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2130 (T318605)', diff saved to https://phabricator.wikimedia.org/P39840 and previous config saved to /var/cache/conftool/dbconfig/20221115-211314-ladsgroup.json
  • 21:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 21:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 21:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T318605)', diff saved to https://phabricator.wikimedia.org/P39839 and previous config saved to /var/cache/conftool/dbconfig/20221115-211253-ladsgroup.json
  • 21:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39838 and previous config saved to /var/cache/conftool/dbconfig/20221115-211009-marostegui.json
  • 21:09 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:09 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:08 cjming@deploy1002: Finished scap: Backport for EditAttemptStep sampling rate to 1 everywhere (T312016) (duration: 04m 45s)
  • 21:07 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:07 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:04 cjming@deploy1002: cjming and phuedx: Backport for EditAttemptStep sampling rate to 1 everywhere (T312016) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 21:03 cjming@deploy1002: Started scap: Backport for EditAttemptStep sampling rate to 1 everywhere (T312016)
  • 21:02 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:01 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1107 (T318605)', diff saved to https://phabricator.wikimedia.org/P39837 and previous config saved to /var/cache/conftool/dbconfig/20221115-205929-ladsgroup.json
  • 20:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 20:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 20:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T318605)', diff saved to https://phabricator.wikimedia.org/P39836 and previous config saved to /var/cache/conftool/dbconfig/20221115-205907-ladsgroup.json
  • 20:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P39835 and previous config saved to /var/cache/conftool/dbconfig/20221115-205746-ladsgroup.json
  • 20:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T321126)', diff saved to https://phabricator.wikimedia.org/P39834 and previous config saved to /var/cache/conftool/dbconfig/20221115-205503-marostegui.json
  • 20:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2175 (T321126)', diff saved to https://phabricator.wikimedia.org/P39833 and previous config saved to /var/cache/conftool/dbconfig/20221115-205222-marostegui.json
  • 20:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 20:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T321130)', diff saved to https://phabricator.wikimedia.org/P39832 and previous config saved to /var/cache/conftool/dbconfig/20221115-205214-marostegui.json
  • 20:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 20:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 20:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39831 and previous config saved to /var/cache/conftool/dbconfig/20221115-205201-marostegui.json
  • 20:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 20:49 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dbprov2004.mgmt.codfw.wmnet with reboot policy GRACEFUL
  • 20:44 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 20:44 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P39830 and previous config saved to /var/cache/conftool/dbconfig/20221115-204401-ladsgroup.json
  • 20:42 eileen: civicrm upgraded from 16167e9a to d85589e8
  • 20:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P39829 and previous config saved to /var/cache/conftool/dbconfig/20221115-204239-ladsgroup.json
  • 20:39 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy GRACEFUL
  • 20:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39828 and previous config saved to /var/cache/conftool/dbconfig/20221115-203654-marostegui.json
  • 20:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 20:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 20:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T321130)', diff saved to https://phabricator.wikimedia.org/P39827 and previous config saved to /var/cache/conftool/dbconfig/20221115-203521-marostegui.json
  • 20:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P39826 and previous config saved to /var/cache/conftool/dbconfig/20221115-202854-ladsgroup.json
  • 20:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T318605)', diff saved to https://phabricator.wikimedia.org/P39825 and previous config saved to /var/cache/conftool/dbconfig/20221115-202733-ladsgroup.json
  • 20:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39824 and previous config saved to /var/cache/conftool/dbconfig/20221115-202148-marostegui.json
  • 20:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P39823 and previous config saved to /var/cache/conftool/dbconfig/20221115-202015-marostegui.json
  • 20:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T318605)', diff saved to https://phabricator.wikimedia.org/P39822 and previous config saved to /var/cache/conftool/dbconfig/20221115-201348-ladsgroup.json
  • 20:13 eileen: civicrm upgraded from 3eba6ad3 to 16167e9a
  • 20:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39821 and previous config saved to /var/cache/conftool/dbconfig/20221115-200641-marostegui.json
  • 20:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P39820 and previous config saved to /var/cache/conftool/dbconfig/20221115-200508-marostegui.json
  • 20:04 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39819 and previous config saved to /var/cache/conftool/dbconfig/20221115-200358-marostegui.json
  • 20:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 20:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 20:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T321126)', diff saved to https://phabricator.wikimedia.org/P39818 and previous config saved to /var/cache/conftool/dbconfig/20221115-200337-marostegui.json
  • 19:56 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:56 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:54 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:54 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:54 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:51 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2042.codfw.wmnet with OS bullseye
  • 19:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T321130)', diff saved to https://phabricator.wikimedia.org/P39817 and previous config saved to /var/cache/conftool/dbconfig/20221115-195002-marostegui.json
  • 19:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P39816 and previous config saved to /var/cache/conftool/dbconfig/20221115-194830-marostegui.json
  • 19:42 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:42 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T321130)', diff saved to https://phabricator.wikimedia.org/P39815 and previous config saved to /var/cache/conftool/dbconfig/20221115-194037-marostegui.json
  • 19:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T321130)', diff saved to https://phabricator.wikimedia.org/P39814 and previous config saved to /var/cache/conftool/dbconfig/20221115-194016-marostegui.json
  • 19:38 ejegg: turned off CiviCRM dedupe jobs for queue speed measurements
  • 19:38 ejegg: payments-wiki upgraded from a058fdbc to bba997aa
  • 19:34 jbond: updated pcc to 2.5.1
  • 19:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P39813 and previous config saved to /var/cache/conftool/dbconfig/20221115-193324-marostegui.json
  • 19:32 topranks: renumbering overlay vrf loopback interface lsw1-e3-eqiad
  • 19:28 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:28 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P39812 and previous config saved to /var/cache/conftool/dbconfig/20221115-192509-marostegui.json
  • 19:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T321126)', diff saved to https://phabricator.wikimedia.org/P39811 and previous config saved to /var/cache/conftool/dbconfig/20221115-191818-marostegui.json
  • 19:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2148 (T321126)', diff saved to https://phabricator.wikimedia.org/P39810 and previous config saved to /var/cache/conftool/dbconfig/20221115-191536-marostegui.json
  • 19:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 19:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 19:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39809 and previous config saved to /var/cache/conftool/dbconfig/20221115-191514-marostegui.json
  • 19:10 brennen@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.10 refs T320515
  • 19:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P39808 and previous config saved to /var/cache/conftool/dbconfig/20221115-191003-marostegui.json
  • 19:04 brennen: train 1.40.0-wmf.10 (T320515) - no current blockers, rolling to group0.
  • 19:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P39807 and previous config saved to /var/cache/conftool/dbconfig/20221115-190008-marostegui.json
  • 18:58 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp2042.codfw.wmnet with OS bullseye
  • 18:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2116 (T318605)', diff saved to https://phabricator.wikimedia.org/P39806 and previous config saved to /var/cache/conftool/dbconfig/20221115-185619-ladsgroup.json
  • 18:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 18:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 18:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T318605)', diff saved to https://phabricator.wikimedia.org/P39805 and previous config saved to /var/cache/conftool/dbconfig/20221115-185558-ladsgroup.json
  • 18:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T321130)', diff saved to https://phabricator.wikimedia.org/P39804 and previous config saved to /var/cache/conftool/dbconfig/20221115-185457-marostegui.json
  • 18:49 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5032.eqsin.wmnet with OS buster
  • 18:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P39803 and previous config saved to /var/cache/conftool/dbconfig/20221115-184501-marostegui.json
  • 18:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P39802 and previous config saved to /var/cache/conftool/dbconfig/20221115-184051-ladsgroup.json
  • 18:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T318605)', diff saved to https://phabricator.wikimedia.org/P39801 and previous config saved to /var/cache/conftool/dbconfig/20221115-183053-ladsgroup.json
  • 18:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 18:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 18:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 18:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 18:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39800 and previous config saved to /var/cache/conftool/dbconfig/20221115-183025-ladsgroup.json
  • 18:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39799 and previous config saved to /var/cache/conftool/dbconfig/20221115-182955-marostegui.json
  • 18:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39798 and previous config saved to /var/cache/conftool/dbconfig/20221115-182712-marostegui.json
  • 18:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T321126)', diff saved to https://phabricator.wikimedia.org/P39797 and previous config saved to /var/cache/conftool/dbconfig/20221115-182640-marostegui.json
  • 18:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P39796 and previous config saved to /var/cache/conftool/dbconfig/20221115-182545-ladsgroup.json
  • 18:16 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage
  • 18:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P39795 and previous config saved to /var/cache/conftool/dbconfig/20221115-181519-ladsgroup.json
  • 18:13 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage
  • 18:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P39794 and previous config saved to /var/cache/conftool/dbconfig/20221115-181133-marostegui.json
  • 18:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T318605)', diff saved to https://phabricator.wikimedia.org/P39793 and previous config saved to /var/cache/conftool/dbconfig/20221115-181037-ladsgroup.json
  • 18:06 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=idwiki` at mwmaint1002 (T318457)
  • 18:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P39792 and previous config saved to /var/cache/conftool/dbconfig/20221115-180012-ladsgroup.json
  • 17:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P39791 and previous config saved to /var/cache/conftool/dbconfig/20221115-175627-marostegui.json
  • 17:54 jbond: move pcc to 2.5.0
  • 17:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T321130)', diff saved to https://phabricator.wikimedia.org/P39790 and previous config saved to /var/cache/conftool/dbconfig/20221115-174827-marostegui.json
  • 17:48 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 17:48 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 17:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39789 and previous config saved to /var/cache/conftool/dbconfig/20221115-174805-marostegui.json
  • 17:46 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 17:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39788 and previous config saved to /var/cache/conftool/dbconfig/20221115-174506-ladsgroup.json
  • 17:42 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS buster
  • 17:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T321126)', diff saved to https://phabricator.wikimedia.org/P39787 and previous config saved to /var/cache/conftool/dbconfig/20221115-174120-marostegui.json
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2126 (T321126)', diff saved to https://phabricator.wikimedia.org/P39786 and previous config saved to /var/cache/conftool/dbconfig/20221115-173841-marostegui.json
  • 17:38 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T321126)', diff saved to https://phabricator.wikimedia.org/P39785 and previous config saved to /var/cache/conftool/dbconfig/20221115-173804-marostegui.json
  • 17:37 hnowlan@cumin1001: END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0)
  • 17:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P39784 and previous config saved to /var/cache/conftool/dbconfig/20221115-173258-marostegui.json
  • 17:29 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 17:29 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 17:28 mutante: deploy1002:~] $ sudo systemctl start wmf_auto_restart_apache2.service
  • 17:25 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 17:25 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 17:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P39783 and previous config saved to /var/cache/conftool/dbconfig/20221115-172257-marostegui.json
  • 17:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P39782 and previous config saved to /var/cache/conftool/dbconfig/20221115-171752-marostegui.json
  • 17:16 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=enwiki` at mwmaint1002 (T318457)
  • 17:10 godog: add 150G to prometheus/ops in eqiad
  • 17:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P39781 and previous config saved to /var/cache/conftool/dbconfig/20221115-170751-marostegui.json
  • 17:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39780 and previous config saved to /var/cache/conftool/dbconfig/20221115-170245-marostegui.json
  • 17:02 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=fawiki` at mwmaint1002 (T318457)
  • 16:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39779 and previous config saved to /var/cache/conftool/dbconfig/20221115-165323-marostegui.json
  • 16:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 16:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 16:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T321130)', diff saved to https://phabricator.wikimedia.org/P39778 and previous config saved to /var/cache/conftool/dbconfig/20221115-165302-marostegui.json
  • 16:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T321126)', diff saved to https://phabricator.wikimedia.org/P39777 and previous config saved to /var/cache/conftool/dbconfig/20221115-165244-marostegui.json
  • 16:50 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T321126)', diff saved to https://phabricator.wikimedia.org/P39776 and previous config saved to /var/cache/conftool/dbconfig/20221115-165001-marostegui.json
  • 16:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 16:49 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 16:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T321126)', diff saved to https://phabricator.wikimedia.org/P39775 and previous config saved to /var/cache/conftool/dbconfig/20221115-164939-marostegui.json
  • 16:49 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 16:43 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry
  • 16:41 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry
  • 16:40 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 16:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P39774 and previous config saved to /var/cache/conftool/dbconfig/20221115-163755-marostegui.json
  • 16:37 ladsgroup:: Deployed security patch for T320987
  • 16:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2103 (T318605)', diff saved to https://phabricator.wikimedia.org/P39773 and previous config saved to /var/cache/conftool/dbconfig/20221115-163721-ladsgroup.json
  • 16:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 16:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 16:36 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wcqs-public
  • 16:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P39772 and previous config saved to /var/cache/conftool/dbconfig/20221115-163432-marostegui.json
  • 16:34 ladsgroup:: Deployed security patch for T320987
  • 16:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1014.eqiad.wmnet
  • 16:22 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wcqs-public
  • 16:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P39770 and previous config saved to /var/cache/conftool/dbconfig/20221115-162249-marostegui.json
  • 16:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1022.eqiad.wmnet with OS bullseye
  • 16:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet
  • 16:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1014.eqiad.wmnet
  • 16:20 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=arwiki` at mwmaint1002 (T318457)
  • 16:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P39769 and previous config saved to /var/cache/conftool/dbconfig/20221115-161925-marostegui.json
  • 16:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS bullseye
  • 16:15 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wdqs-all
  • 16:15 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 16:14 urandom: initiating Cassandra bootstrap, aqs1019-b -- T307802
  • 16:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet
  • 16:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet
  • 16:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39768 and previous config saved to /var/cache/conftool/dbconfig/20221115-160804-ladsgroup.json
  • 16:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 16:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 16:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T321130)', diff saved to https://phabricator.wikimedia.org/P39767 and previous config saved to /var/cache/conftool/dbconfig/20221115-160742-marostegui.json
  • 16:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1012.eqiad.wmnet
  • 16:06 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wdqs-all
  • 16:05 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 16:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet
  • 16:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T321126)', diff saved to https://phabricator.wikimedia.org/P39766 and previous config saved to /var/cache/conftool/dbconfig/20221115-160419-marostegui.json
  • 16:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
  • 16:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1022.eqiad.wmnet with reason: host reimage
  • 16:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1012.eqiad.wmnet
  • 16:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2104 (T321126)', diff saved to https://phabricator.wikimedia.org/P39765 and previous config saved to /var/cache/conftool/dbconfig/20221115-160140-marostegui.json
  • 16:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 16:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T321126)', diff saved to https://phabricator.wikimedia.org/P39764 and previous config saved to /var/cache/conftool/dbconfig/20221115-160010-marostegui.json
  • 15:59 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1022.eqiad.wmnet with reason: host reimage
  • 15:58 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T321130)', diff saved to https://phabricator.wikimedia.org/P39763 and previous config saved to /var/cache/conftool/dbconfig/20221115-155821-marostegui.json
  • 15:58 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
  • 15:58 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 15:58 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 15:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321130)', diff saved to https://phabricator.wikimedia.org/P39762 and previous config saved to /var/cache/conftool/dbconfig/20221115-155800-marostegui.json
  • 15:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
  • 15:54 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P39761 and previous config saved to /var/cache/conftool/dbconfig/20221115-155237-ladsgroup.json
  • 15:49 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=ptwiki` at mwmaint1002 (T318457)
  • 15:45 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1022.eqiad.wmnet with OS bullseye
  • 15:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P39760 and previous config saved to /var/cache/conftool/dbconfig/20221115-154504-marostegui.json
  • 15:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1022.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 15:44 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1022.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 15:43 moritzm: uploaded cas 6.6.2 to apt.wikimedia.org T311235
  • 15:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P39759 and previous config saved to /var/cache/conftool/dbconfig/20221115-154253-marostegui.json
  • 15:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P39758 and previous config saved to /var/cache/conftool/dbconfig/20221115-153731-ladsgroup.json
  • 15:33 taavi@deploy1002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
  • 15:32 taavi@deploy1002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 15:32 taavi@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
  • 15:31 taavi@deploy1002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
  • 15:30 taavi@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 15:30 taavi@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 15:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P39757 and previous config saved to /var/cache/conftool/dbconfig/20221115-152957-marostegui.json
  • 15:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P39756 and previous config saved to /var/cache/conftool/dbconfig/20221115-152747-marostegui.json
  • 15:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39755 and previous config saved to /var/cache/conftool/dbconfig/20221115-152224-ladsgroup.json
  • 15:15 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=bnwiki` at mwmaint1002 (T318457)
  • 15:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T321126)', diff saved to https://phabricator.wikimedia.org/P39754 and previous config saved to /var/cache/conftool/dbconfig/20221115-151451-marostegui.json
  • 15:14 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS bullseye
  • 15:14 moritzm: installing expat security updates
  • 15:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321130)', diff saved to https://phabricator.wikimedia.org/P39753 and previous config saved to /var/cache/conftool/dbconfig/20221115-151241-marostegui.json
  • 15:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1197 (T321126)', diff saved to https://phabricator.wikimedia.org/P39752 and previous config saved to /var/cache/conftool/dbconfig/20221115-151232-marostegui.json
  • 15:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 15:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 15:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T321126)', diff saved to https://phabricator.wikimedia.org/P39751 and previous config saved to /var/cache/conftool/dbconfig/20221115-151211-marostegui.json
  • 15:10 hnowlan@cumin1001: START - Cookbook sre.postgresql.postgres-init
  • 15:09 hnowlan@cumin1001: END (FAIL) - Cookbook sre.postgresql.postgres-init (exit_code=99)
  • 15:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T318950)', diff saved to https://phabricator.wikimedia.org/P39750 and previous config saved to /var/cache/conftool/dbconfig/20221115-150901-ladsgroup.json
  • 15:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3003.esams.wmnet
  • 15:06 hnowlan@cumin1001: START - Cookbook sre.postgresql.postgres-init
  • 15:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T318955)', diff saved to https://phabricator.wikimedia.org/P39749 and previous config saved to /var/cache/conftool/dbconfig/20221115-150242-ladsgroup.json
  • 15:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 15:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti3003.esams.wmnet
  • 15:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 14:59 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=frwiki` at mwmaint1002 (T318457)
  • 14:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2032.codfw.wmnet
  • 14:57 urbanecm@deploy1002: Finished scap: Backport for updateIsActiveFlagForMentees: Process all mentees (T318457), MentorStore: Use $wgRCMaxAge instead of INACTIVITY_THRESHOLD (T318457) (duration: 05m 44s)
  • 14:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P39748 and previous config saved to /var/cache/conftool/dbconfig/20221115-145704-marostegui.json
  • 14:55 moritzm: installing tomcat security updates
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39747 and previous config saved to /var/cache/conftool/dbconfig/20221115-145355-ladsgroup.json
  • 14:53 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 14:52 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2032.codfw.wmnet
  • 14:52 urbanecm@deploy1002: urbanecm and urbanecm: Backport for updateIsActiveFlagForMentees: Process all mentees (T318457), MentorStore: Use $wgRCMaxAge instead of INACTIVITY_THRESHOLD (T318457) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 14:51 urbanecm@deploy1002: Started scap: Backport for updateIsActiveFlagForMentees: Process all mentees (T318457), MentorStore: Use $wgRCMaxAge instead of INACTIVITY_THRESHOLD (T318457)
  • 14:51 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp5032']
  • 14:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2031.codfw.wmnet
  • 14:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P39746 and previous config saved to /var/cache/conftool/dbconfig/20221115-144736-ladsgroup.json
  • 14:43 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2031.codfw.wmnet
  • 14:43 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5032']
  • 14:43 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cp5032']
  • 14:43 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 14:42 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2003-dev.codfw.wmnet with OS bullseye
  • 14:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P39745 and previous config saved to /var/cache/conftool/dbconfig/20221115-144158-marostegui.json
  • 14:40 moritzm: failover ganeti master in esams to ganeti3001
  • 14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39744 and previous config saved to /var/cache/conftool/dbconfig/20221115-143848-ladsgroup.json
  • 14:33 hnowlan@cumin1001: END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0)
  • 14:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P39743 and previous config saved to /var/cache/conftool/dbconfig/20221115-143229-ladsgroup.json
  • 14:31 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5032']
  • 14:27 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
  • 14:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3002.esams.wmnet
  • 14:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T321126)', diff saved to https://phabricator.wikimedia.org/P39742 and previous config saved to /var/cache/conftool/dbconfig/20221115-142652-marostegui.json
  • 14:25 aborrero@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
  • 14:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1188 (T321126)', diff saved to https://phabricator.wikimedia.org/P39741 and previous config saved to /var/cache/conftool/dbconfig/20221115-142432-marostegui.json
  • 14:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 14:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 14:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T321126)', diff saved to https://phabricator.wikimedia.org/P39740 and previous config saved to /var/cache/conftool/dbconfig/20221115-142411-marostegui.json
  • 14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T318950)', diff saved to https://phabricator.wikimedia.org/P39739 and previous config saved to /var/cache/conftool/dbconfig/20221115-142342-ladsgroup.json
  • 14:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T318950)', diff saved to https://phabricator.wikimedia.org/P39738 and previous config saved to /var/cache/conftool/dbconfig/20221115-142130-ladsgroup.json
  • 14:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 14:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 14:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti3002.esams.wmnet
  • 14:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T318955)', diff saved to https://phabricator.wikimedia.org/P39737 and previous config saved to /var/cache/conftool/dbconfig/20221115-141723-ladsgroup.json
  • 14:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T318955)', diff saved to https://phabricator.wikimedia.org/P39736 and previous config saved to /var/cache/conftool/dbconfig/20221115-141513-ladsgroup.json
  • 14:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 14:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 14:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T321130)', diff saved to https://phabricator.wikimedia.org/P39735 and previous config saved to /var/cache/conftool/dbconfig/20221115-141218-marostegui.json
  • 14:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 14:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 14:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39734 and previous config saved to /var/cache/conftool/dbconfig/20221115-141157-marostegui.json
  • 14:09 aborrero@cumin1001: START - Cookbook sre.hosts.reimage for host cloudgw2003-dev.codfw.wmnet with OS bullseye
  • 14:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P39733 and previous config saved to /var/cache/conftool/dbconfig/20221115-140905-marostegui.json
  • 14:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp5032.mgmt.eqsin.wmnet with reboot policy FORCED
  • 14:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3001.esams.wmnet
  • 13:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P39732 and previous config saved to /var/cache/conftool/dbconfig/20221115-135650-marostegui.json
  • 13:56 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cp5032.mgmt.eqsin.wmnet with reboot policy FORCED
  • 13:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P39731 and previous config saved to /var/cache/conftool/dbconfig/20221115-135358-marostegui.json
  • 13:52 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti3001.esams.wmnet
  • 13:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P39730 and previous config saved to /var/cache/conftool/dbconfig/20221115-134144-marostegui.json
  • 13:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 13:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39729 and previous config saved to /var/cache/conftool/dbconfig/20221115-134036-ladsgroup.json
  • 13:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 13:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 13:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 13:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T321126)', diff saved to https://phabricator.wikimedia.org/P39728 and previous config saved to /var/cache/conftool/dbconfig/20221115-133852-marostegui.json
  • 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T321126)', diff saved to https://phabricator.wikimedia.org/P39727 and previous config saved to /var/cache/conftool/dbconfig/20221115-133631-marostegui.json
  • 13:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 13:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39726 and previous config saved to /var/cache/conftool/dbconfig/20221115-133610-marostegui.json
  • 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4008.ulsfo.wmnet
  • 13:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet
  • 13:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39725 and previous config saved to /var/cache/conftool/dbconfig/20221115-132637-marostegui.json
  • 13:22 sukhe: running homer for Gerrit: 856946 in cr*-ulsfo*
  • 13:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P39724 and previous config saved to /var/cache/conftool/dbconfig/20221115-132103-marostegui.json
  • 13:20 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 13:20 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 13:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 13335
  • 13:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39723 and previous config saved to /var/cache/conftool/dbconfig/20221115-131710-marostegui.json
  • 13:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 13:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 13:08 moritzm: failover ganeti master in ulsfo to ganeti4005
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P39722 and previous config saved to /var/cache/conftool/dbconfig/20221115-130557-marostegui.json
  • 13:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321130)', diff saved to https://phabricator.wikimedia.org/P39721 and previous config saved to /var/cache/conftool/dbconfig/20221115-125950-marostegui.json
  • 12:59 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 13335
  • 12:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
  • 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39720 and previous config saved to /var/cache/conftool/dbconfig/20221115-125050-marostegui.json
  • 12:49 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2003-dev.codfw.wmnet with OS bullseye
  • 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39719 and previous config saved to /var/cache/conftool/dbconfig/20221115-124830-marostegui.json
  • 12:48 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 12:48 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T321126)', diff saved to https://phabricator.wikimedia.org/P39718 and previous config saved to /var/cache/conftool/dbconfig/20221115-124808-marostegui.json
  • 12:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
  • 12:45 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on idp-test1002.wikimedia.org with reason: experiment with CAS 6.6
  • 12:45 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on idp-test1002.wikimedia.org with reason: experiment with CAS 6.6
  • 12:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P39717 and previous config saved to /var/cache/conftool/dbconfig/20221115-124443-marostegui.json
  • 12:43 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=varnish-fe
  • 12:43 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=ats-be
  • 12:43 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=ats-tls
  • 12:37 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39716 and previous config saved to /var/cache/conftool/dbconfig/20221115-123735-root.json
  • 12:36 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39715 and previous config saved to /var/cache/conftool/dbconfig/20221115-123326-root.json
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39714 and previous config saved to /var/cache/conftool/dbconfig/20221115-123302-marostegui.json
  • 12:31 aborrero@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
  • 12:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P39713 and previous config saved to /var/cache/conftool/dbconfig/20221115-122937-marostegui.json
  • 12:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
  • 12:22 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39712 and previous config saved to /var/cache/conftool/dbconfig/20221115-122230-root.json
  • 12:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
  • 12:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39711 and previous config saved to /var/cache/conftool/dbconfig/20221115-121821-root.json
  • 12:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39710 and previous config saved to /var/cache/conftool/dbconfig/20221115-121755-marostegui.json
  • 12:16 aborrero@cumin1001: START - Cookbook sre.hosts.reimage for host cloudgw2003-dev.codfw.wmnet with OS bullseye
  • 12:14 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321130)', diff saved to https://phabricator.wikimedia.org/P39709 and previous config saved to /var/cache/conftool/dbconfig/20221115-121431-marostegui.json
  • 12:13 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:11 hnowlan: resyncing maps2005 replica
  • 12:11 hnowlan@cumin1001: START - Cookbook sre.postgresql.postgres-init
  • 12:07 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39708 and previous config saved to /var/cache/conftool/dbconfig/20221115-120725-root.json
  • 12:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T321130)', diff saved to https://phabricator.wikimedia.org/P39707 and previous config saved to /var/cache/conftool/dbconfig/20221115-120502-marostegui.json
  • 12:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:04 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 12:04 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 12:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39706 and previous config saved to /var/cache/conftool/dbconfig/20221115-120316-root.json
  • 12:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T321126)', diff saved to https://phabricator.wikimedia.org/P39705 and previous config saved to /var/cache/conftool/dbconfig/20221115-120249-marostegui.json
  • 12:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T321126)', diff saved to https://phabricator.wikimedia.org/P39704 and previous config saved to /var/cache/conftool/dbconfig/20221115-120030-marostegui.json
  • 12:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 12:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 12:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T321126)', diff saved to https://phabricator.wikimedia.org/P39703 and previous config saved to /var/cache/conftool/dbconfig/20221115-120009-marostegui.json
  • 11:52 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39702 and previous config saved to /var/cache/conftool/dbconfig/20221115-115220-root.json
  • 11:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39701 and previous config saved to /var/cache/conftool/dbconfig/20221115-114812-root.json
  • 11:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 11:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 11:45 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti6003.drmrs.wmnet
  • 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P39700 and previous config saved to /var/cache/conftool/dbconfig/20221115-114502-marostegui.json
  • 11:42 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2002-dev.codfw.wmnet with OS bullseye
  • 11:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39699 and previous config saved to /var/cache/conftool/dbconfig/20221115-114216-marostegui.json
  • 11:37 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39698 and previous config saved to /var/cache/conftool/dbconfig/20221115-113715-root.json
  • 11:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
  • 11:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39697 and previous config saved to /var/cache/conftool/dbconfig/20221115-113307-root.json
  • 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P39696 and previous config saved to /var/cache/conftool/dbconfig/20221115-112956-marostegui.json
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39695 and previous config saved to /var/cache/conftool/dbconfig/20221115-112709-marostegui.json
  • 11:22 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
  • 11:22 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39694 and previous config saved to /var/cache/conftool/dbconfig/20221115-112210-root.json
  • 11:20 moritzm: failover ganeti master in drmrs/B12 to ganeti6001
  • 11:20 aborrero@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
  • 11:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39693 and previous config saved to /var/cache/conftool/dbconfig/20221115-111802-root.json
  • 11:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T321126)', diff saved to https://phabricator.wikimedia.org/P39692 and previous config saved to /var/cache/conftool/dbconfig/20221115-111449-marostegui.json
  • 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T321126)', diff saved to https://phabricator.wikimedia.org/P39691 and previous config saved to /var/cache/conftool/dbconfig/20221115-111229-marostegui.json
  • 11:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39690 and previous config saved to /var/cache/conftool/dbconfig/20221115-111203-marostegui.json
  • 11:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 11:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39689 and previous config saved to /var/cache/conftool/dbconfig/20221115-111150-marostegui.json
  • 11:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
  • 11:07 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39688 and previous config saved to /var/cache/conftool/dbconfig/20221115-110705-root.json
  • 11:05 aborrero@cumin1001: START - Cookbook sre.hosts.reimage for host cloudgw2002-dev.codfw.wmnet with OS bullseye
  • 11:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39687 and previous config saved to /var/cache/conftool/dbconfig/20221115-110257-root.json
  • 10:59 mfossati@deploy1002: Finished deploy [airflow-dags/platform_eng@3bb99c2]: (no justification provided) (duration: 00m 05s)
  • 10:59 mfossati@deploy1002: Started deploy [airflow-dags/platform_eng@3bb99c2]: (no justification provided)
  • 10:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39686 and previous config saved to /var/cache/conftool/dbconfig/20221115-105657-marostegui.json
  • 10:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P39685 and previous config saved to /var/cache/conftool/dbconfig/20221115-105644-marostegui.json
  • 10:52 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39684 and previous config saved to /var/cache/conftool/dbconfig/20221115-105200-root.json
  • 10:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39683 and previous config saved to /var/cache/conftool/dbconfig/20221115-104752-root.json
  • 10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39682 and previous config saved to /var/cache/conftool/dbconfig/20221115-104435-marostegui.json
  • 10:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 10:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39681 and previous config saved to /var/cache/conftool/dbconfig/20221115-104402-marostegui.json
  • 10:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P39680 and previous config saved to /var/cache/conftool/dbconfig/20221115-104137-marostegui.json
  • 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39679 and previous config saved to /var/cache/conftool/dbconfig/20221115-102856-marostegui.json
  • 10:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39678 and previous config saved to /var/cache/conftool/dbconfig/20221115-102631-marostegui.json
  • 10:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39677 and previous config saved to /var/cache/conftool/dbconfig/20221115-102409-marostegui.json
  • 10:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39676 and previous config saved to /var/cache/conftool/dbconfig/20221115-102319-marostegui.json
  • 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39675 and previous config saved to /var/cache/conftool/dbconfig/20221115-101349-marostegui.json
  • 10:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P39674 and previous config saved to /var/cache/conftool/dbconfig/20221115-100812-marostegui.json
  • 09:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39673 and previous config saved to /var/cache/conftool/dbconfig/20221115-095843-marostegui.json
  • 09:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P39672 and previous config saved to /var/cache/conftool/dbconfig/20221115-095306-marostegui.json
  • 09:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39671 and previous config saved to /var/cache/conftool/dbconfig/20221115-094552-marostegui.json
  • 09:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39670 and previous config saved to /var/cache/conftool/dbconfig/20221115-093758-marostegui.json
  • 09:36 moritzm: draining ganeti1022 for eventual reimage T311687
  • 09:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39669 and previous config saved to /var/cache/conftool/dbconfig/20221115-093539-marostegui.json
  • 09:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 09:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 09:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39668 and previous config saved to /var/cache/conftool/dbconfig/20221115-093518-marostegui.json
  • 09:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P39667 and previous config saved to /var/cache/conftool/dbconfig/20221115-092011-marostegui.json
  • 09:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts bast5001.wikimedia.org
  • 09:14 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:12 jmm@cumin2002: START - Cookbook sre.dns.netbox
  • 09:07 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts bast5001.wikimedia.org
  • 09:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P39666 and previous config saved to /var/cache/conftool/dbconfig/20221115-090505-marostegui.json
  • 09:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2162', diff saved to https://phabricator.wikimedia.org/P39664 and previous config saved to /var/cache/conftool/dbconfig/20221115-090058-root.json
  • 08:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1138.eqiad.wmnet with reason: Maintenance
  • 08:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1138.eqiad.wmnet with reason: Maintenance
  • 08:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39663 and previous config saved to /var/cache/conftool/dbconfig/20221115-084959-marostegui.json
  • 08:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39662 and previous config saved to /var/cache/conftool/dbconfig/20221115-084637-marostegui.json
  • 08:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 08:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 08:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 08:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 08:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 08:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 08:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 08:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 08:06 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1021.eqiad.wmnet to cluster eqiad and group D
  • 08:05 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1021.eqiad.wmnet to cluster eqiad and group D
  • 08:05 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1021.eqiad.wmnet
  • 07:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1021.eqiad.wmnet
  • 07:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39661 and previous config saved to /var/cache/conftool/dbconfig/20221115-072202-marostegui.json
  • 07:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P39660 and previous config saved to /var/cache/conftool/dbconfig/20221115-070655-marostegui.json
  • 06:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P39659 and previous config saved to /var/cache/conftool/dbconfig/20221115-065149-marostegui.json
  • 06:43 robh: all in rack work in eqsin is complete for today
  • 06:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39658 and previous config saved to /var/cache/conftool/dbconfig/20221115-063642-marostegui.json
  • 06:30 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1008.eqiad.wmnet
  • 06:26 robh@cumin1001: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5032
  • 06:26 robh@cumin1001: START - Cookbook sre.network.configure-switch-interfaces for host cp5032
  • 06:25 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 06:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39657 and previous config saved to /var/cache/conftool/dbconfig/20221115-062400-marostegui.json
  • 06:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 06:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 06:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T321130)', diff saved to https://phabricator.wikimedia.org/P39656 and previous config saved to /var/cache/conftool/dbconfig/20221115-062339-marostegui.json
  • 06:23 robh@cumin1001: START - Cookbook sre.dns.netbox
  • 06:20 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1008.eqiad.wmnet
  • 06:19 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1007.eqiad.wmnet
  • 06:18 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1005.eqiad.wmnet
  • 06:15 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-tool1007.eqiad.wmnet
  • 06:13 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host karapace1001.eqiad.wmnet
  • 06:09 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host karapace1001.eqiad.wmnet
  • 06:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P39655 and previous config saved to /var/cache/conftool/dbconfig/20221115-060832-marostegui.json
  • 06:06 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1005.eqiad.wmnet
  • 06:05 stevemunene@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host stat1005.eqiad.wmnet
  • 06:05 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1005.eqiad.wmnet
  • 05:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P39654 and previous config saved to /var/cache/conftool/dbconfig/20221115-055326-marostegui.json
  • 05:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T321130)', diff saved to https://phabricator.wikimedia.org/P39653 and previous config saved to /var/cache/conftool/dbconfig/20221115-053819-marostegui.json
  • 05:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2172 (T321130)', diff saved to https://phabricator.wikimedia.org/P39652 and previous config saved to /var/cache/conftool/dbconfig/20221115-052543-marostegui.json
  • 05:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 05:25 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 05:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321130)', diff saved to https://phabricator.wikimedia.org/P39651 and previous config saved to /var/cache/conftool/dbconfig/20221115-052521-marostegui.json
  • 05:13 robh: ~5AM UTC when plugging a new host into asw1-ulsfo, the virtual chassis crashed and rebooted, causing loss of connectivity to hosts for a very short period
  • 05:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P39650 and previous config saved to /var/cache/conftool/dbconfig/20221115-051015-marostegui.json
  • 04:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P39649 and previous config saved to /var/cache/conftool/dbconfig/20221115-045508-marostegui.json
  • 04:48 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 04:40 mwpresync@deploy1002: Pruned MediaWiki: 1.40.0-wmf.7 (duration: 02m 01s)
  • 04:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321130)', diff saved to https://phabricator.wikimedia.org/P39648 and previous config saved to /var/cache/conftool/dbconfig/20221115-044002-marostegui.json
  • 04:38 mwpresync@deploy1002: Finished scap: testwikis wikis to 1.40.0-wmf.10 refs T320515 (duration: 36m 14s)
  • 04:37 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 04:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2155 (T321130)', diff saved to https://phabricator.wikimedia.org/P39647 and previous config saved to /var/cache/conftool/dbconfig/20221115-042713-marostegui.json
  • 04:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 04:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 04:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 04:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 04:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39646 and previous config saved to /var/cache/conftool/dbconfig/20221115-042647-marostegui.json
  • 04:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 04:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 04:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 04:15 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 04:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P39645 and previous config saved to /var/cache/conftool/dbconfig/20221115-041140-marostegui.json
  • 04:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 04:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 04:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 04:03 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 04:02 mwpresync@deploy1002: Started scap: testwikis wikis to 1.40.0-wmf.10 refs T320515
  • 03:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P39644 and previous config saved to /var/cache/conftool/dbconfig/20221115-035634-marostegui.json
  • 03:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39643 and previous config saved to /var/cache/conftool/dbconfig/20221115-034127-marostegui.json
  • 03:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 03:32 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 03:32 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 03:32 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 03:29 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39642 and previous config saved to /var/cache/conftool/dbconfig/20221115-032929-marostegui.json
  • 03:29 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 03:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 03:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 03:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 03:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39641 and previous config saved to /var/cache/conftool/dbconfig/20221115-031826-marostegui.json
  • 03:16 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 03:12 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 03:12 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 03:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 03:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P39640 and previous config saved to /var/cache/conftool/dbconfig/20221115-030320-marostegui.json
  • 02:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P39639 and previous config saved to /var/cache/conftool/dbconfig/20221115-024813-marostegui.json
  • 02:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39638 and previous config saved to /var/cache/conftool/dbconfig/20221115-023307-marostegui.json
  • 02:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39637 and previous config saved to /var/cache/conftool/dbconfig/20221115-022052-marostegui.json
  • 02:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 02:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 02:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39636 and previous config saved to /var/cache/conftool/dbconfig/20221115-022030-marostegui.json
  • 02:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P39635 and previous config saved to /var/cache/conftool/dbconfig/20221115-020523-marostegui.json
  • 01:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P39634 and previous config saved to /var/cache/conftool/dbconfig/20221115-015017-marostegui.json
  • 01:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39633 and previous config saved to /var/cache/conftool/dbconfig/20221115-013510-marostegui.json
  • 01:28 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS buster
  • 01:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39632 and previous config saved to /var/cache/conftool/dbconfig/20221115-012313-marostegui.json
  • 01:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 01:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 01:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39631 and previous config saved to /var/cache/conftool/dbconfig/20221115-012251-marostegui.json
  • 01:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P39630 and previous config saved to /var/cache/conftool/dbconfig/20221115-010745-marostegui.json
  • 01:05 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 01:02 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 00:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P39629 and previous config saved to /var/cache/conftool/dbconfig/20221115-005238-marostegui.json
  • 00:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39628 and previous config saved to /var/cache/conftool/dbconfig/20221115-003732-marostegui.json
  • 00:33 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS buster
  • 00:29 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host arclamp2001.codfw.wmnet with OS bullseye
  • 00:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39627 and previous config saved to /var/cache/conftool/dbconfig/20221115-002514-marostegui.json
  • 00:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 00:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 00:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T321130)', diff saved to https://phabricator.wikimedia.org/P39626 and previous config saved to /var/cache/conftool/dbconfig/20221115-002441-marostegui.json
  • 00:14 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on arclamp2001.codfw.wmnet with reason: host reimage
  • 00:13 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host puppetdb2003.codfw.wmnet with OS bullseye
  • 00:11 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on arclamp2001.codfw.wmnet with reason: host reimage
  • 00:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P39625 and previous config saved to /var/cache/conftool/dbconfig/20221115-000935-marostegui.json

2022-11-14

  • 23:58 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on puppetdb2003.codfw.wmnet with reason: host reimage
  • 23:55 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on puppetdb2003.codfw.wmnet with reason: host reimage
  • 23:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P39624 and previous config saved to /var/cache/conftool/dbconfig/20221114-235429-marostegui.json
  • 23:52 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host arclamp2001.codfw.wmnet with OS bullseye
  • 23:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T321130)', diff saved to https://phabricator.wikimedia.org/P39623 and previous config saved to /var/cache/conftool/dbconfig/20221114-233922-marostegui.json
  • 23:36 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host puppetdb2003.codfw.wmnet with OS bullseye
  • 23:32 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov2004.codfw.wmnet with OS bullseye
  • 23:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39622 and previous config saved to /var/cache/conftool/dbconfig/20221114-232744-marostegui.json
  • 23:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2119 (T321130)', diff saved to https://phabricator.wikimedia.org/P39621 and previous config saved to /var/cache/conftool/dbconfig/20221114-232714-marostegui.json
  • 23:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 23:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 23:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T321130)', diff saved to https://phabricator.wikimedia.org/P39620 and previous config saved to /var/cache/conftool/dbconfig/20221114-232653-marostegui.json
  • 23:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P39619 and previous config saved to /var/cache/conftool/dbconfig/20221114-231238-marostegui.json
  • 23:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P39618 and previous config saved to /var/cache/conftool/dbconfig/20221114-231146-marostegui.json
  • 23:10 eileen: civicrm upgraded from 93fa3f37 to 3eba6ad3
  • 22:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P39617 and previous config saved to /var/cache/conftool/dbconfig/20221114-225730-marostegui.json
  • 22:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P39616 and previous config saved to /var/cache/conftool/dbconfig/20221114-225638-marostegui.json
  • 22:56 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 22:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 22:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39614 and previous config saved to /var/cache/conftool/dbconfig/20221114-224224-marostegui.json
  • 22:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T321130)', diff saved to https://phabricator.wikimedia.org/P39613 and previous config saved to /var/cache/conftool/dbconfig/20221114-224132-marostegui.json
  • 22:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39612 and previous config saved to /var/cache/conftool/dbconfig/20221114-224006-marostegui.json
  • 22:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 22:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 22:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39611 and previous config saved to /var/cache/conftool/dbconfig/20221114-223945-marostegui.json
  • 22:31 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 22:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2110 (T321130)', diff saved to https://phabricator.wikimedia.org/P39610 and previous config saved to /var/cache/conftool/dbconfig/20221114-222706-marostegui.json
  • 22:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 22:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 22:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T321130)', diff saved to https://phabricator.wikimedia.org/P39609 and previous config saved to /var/cache/conftool/dbconfig/20221114-222644-marostegui.json
  • 22:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P39608 and previous config saved to /var/cache/conftool/dbconfig/20221114-222438-marostegui.json
  • 22:21 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 22:19 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 22:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P39607 and previous config saved to /var/cache/conftool/dbconfig/20221114-221138-marostegui.json
  • 22:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P39606 and previous config saved to /var/cache/conftool/dbconfig/20221114-220932-marostegui.json
  • 22:09 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 22:09 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 22:03 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 21:58 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P39605 and previous config saved to /var/cache/conftool/dbconfig/20221114-215631-marostegui.json
  • 21:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39604 and previous config saved to /var/cache/conftool/dbconfig/20221114-215425-marostegui.json
  • 21:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39603 and previous config saved to /var/cache/conftool/dbconfig/20221114-215204-marostegui.json
  • 21:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 21:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 21:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39602 and previous config saved to /var/cache/conftool/dbconfig/20221114-215143-marostegui.json
  • 21:48 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T321130)', diff saved to https://phabricator.wikimedia.org/P39601 and previous config saved to /var/cache/conftool/dbconfig/20221114-214125-marostegui.json
  • 21:38 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P39600 and previous config saved to /var/cache/conftool/dbconfig/20221114-213636-marostegui.json
  • 21:35 mutante: phab2002 - systemctl start phd, debug why it still fails
  • 21:35 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:29 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2106 (T321130)', diff saved to https://phabricator.wikimedia.org/P39599 and previous config saved to /var/cache/conftool/dbconfig/20221114-212934-marostegui.json
  • 21:29 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 21:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 21:25 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P39598 and previous config saved to /var/cache/conftool/dbconfig/20221114-212130-marostegui.json
  • 21:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 21:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 21:11 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39597 and previous config saved to /var/cache/conftool/dbconfig/20221114-210853-marostegui.json
  • 21:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39596 and previous config saved to /var/cache/conftool/dbconfig/20221114-210623-marostegui.json
  • 21:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39595 and previous config saved to /var/cache/conftool/dbconfig/20221114-210503-marostegui.json
  • 21:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 21:04 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 21:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T321126)', diff saved to https://phabricator.wikimedia.org/P39594 and previous config saved to /var/cache/conftool/dbconfig/20221114-210430-marostegui.json
  • 21:01 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39593 and previous config saved to /var/cache/conftool/dbconfig/20221114-205347-marostegui.json
  • 20:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P39592 and previous config saved to /var/cache/conftool/dbconfig/20221114-204924-marostegui.json
  • 20:47 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 20:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39591 and previous config saved to /var/cache/conftool/dbconfig/20221114-203841-marostegui.json
  • 20:37 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P39590 and previous config saved to /var/cache/conftool/dbconfig/20221114-203417-marostegui.json
  • 20:34 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 20:34 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39589 and previous config saved to /var/cache/conftool/dbconfig/20221114-202334-marostegui.json
  • 20:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T321126)', diff saved to https://phabricator.wikimedia.org/P39588 and previous config saved to /var/cache/conftool/dbconfig/20221114-201911-marostegui.json
  • 20:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T321126)', diff saved to https://phabricator.wikimedia.org/P39587 and previous config saved to /var/cache/conftool/dbconfig/20221114-201650-marostegui.json
  • 20:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 20:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39586 and previous config saved to /var/cache/conftool/dbconfig/20221114-201556-marostegui.json
  • 20:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P39585 and previous config saved to /var/cache/conftool/dbconfig/20221114-200050-marostegui.json
  • 19:55 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: Upgrade horizon to Z to prepare for Openstack upgrades past Wallaby -- T305828 (duration: 04m 41s)
  • 19:50 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: Upgrade horizon to Z to prepare for Openstack upgrades past Wallaby -- T305828
  • 19:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P39583 and previous config saved to /var/cache/conftool/dbconfig/20221114-194543-marostegui.json
  • 19:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39582 and previous config saved to /var/cache/conftool/dbconfig/20221114-193037-marostegui.json
  • 19:29 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS bullseye
  • 19:28 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39581 and previous config saved to /var/cache/conftool/dbconfig/20221114-192816-marostegui.json
  • 19:28 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 19:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 19:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321126)', diff saved to https://phabricator.wikimedia.org/P39580 and previous config saved to /var/cache/conftool/dbconfig/20221114-192754-marostegui.json
  • 19:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39579 and previous config saved to /var/cache/conftool/dbconfig/20221114-192318-marostegui.json
  • 19:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 19:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 19:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321130)', diff saved to https://phabricator.wikimedia.org/P39578 and previous config saved to /var/cache/conftool/dbconfig/20221114-192257-marostegui.json
  • 19:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P39577 and previous config saved to /var/cache/conftool/dbconfig/20221114-191247-marostegui.json
  • 19:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P39576 and previous config saved to /var/cache/conftool/dbconfig/20221114-190750-marostegui.json
  • 18:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P39575 and previous config saved to /var/cache/conftool/dbconfig/20221114-185741-marostegui.json
  • 18:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P39574 and previous config saved to /var/cache/conftool/dbconfig/20221114-185244-marostegui.json
  • 18:50 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:45 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321126)', diff saved to https://phabricator.wikimedia.org/P39573 and previous config saved to /var/cache/conftool/dbconfig/20221114-184235-marostegui.json
  • 18:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T321126)', diff saved to https://phabricator.wikimedia.org/P39572 and previous config saved to /var/cache/conftool/dbconfig/20221114-184014-marostegui.json
  • 18:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 18:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 18:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321126)', diff saved to https://phabricator.wikimedia.org/P39571 and previous config saved to /var/cache/conftool/dbconfig/20221114-183952-marostegui.json
  • 18:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321130)', diff saved to https://phabricator.wikimedia.org/P39570 and previous config saved to /var/cache/conftool/dbconfig/20221114-183738-marostegui.json
  • 18:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1190 (T321130)', diff saved to https://phabricator.wikimedia.org/P39569 and previous config saved to /var/cache/conftool/dbconfig/20221114-182506-marostegui.json
  • 18:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 18:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 18:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P39568 and previous config saved to /var/cache/conftool/dbconfig/20221114-182446-marostegui.json
  • 18:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39567 and previous config saved to /var/cache/conftool/dbconfig/20221114-182445-marostegui.json
  • 18:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 18:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 18:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T322618)', diff saved to https://phabricator.wikimedia.org/P39566 and previous config saved to /var/cache/conftool/dbconfig/20221114-181700-ladsgroup.json
  • 18:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39565 and previous config saved to /var/cache/conftool/dbconfig/20221114-180938-marostegui.json
  • 18:08 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['arclamp2001']
  • 18:07 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['arclamp2001']
  • 18:07 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['arclamp2001']
  • 18:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P39564 and previous config saved to /var/cache/conftool/dbconfig/20221114-180153-ladsgroup.json
  • 17:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39563 and previous config saved to /var/cache/conftool/dbconfig/20221114-175432-marostegui.json
  • 17:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T321126)', diff saved to https://phabricator.wikimedia.org/P39562 and previous config saved to /var/cache/conftool/dbconfig/20221114-175213-marostegui.json
  • 17:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 17:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 17:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 17:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 17:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T321126)', diff saved to https://phabricator.wikimedia.org/P39561 and previous config saved to /var/cache/conftool/dbconfig/20221114-175129-marostegui.json
  • 17:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P39560 and previous config saved to /var/cache/conftool/dbconfig/20221114-174647-ladsgroup.json
  • 17:42 hashar: Restored CI caching mechanism which has been serving stalled caches since March 29th 2022 :-\ T323051
  • 17:42 hashar: Restored CI caching mechanism which has been serving stalled caches since March 29th 2022 :-\ T307334
  • 17:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39559 and previous config saved to /var/cache/conftool/dbconfig/20221114-173925-marostegui.json
  • 17:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P39558 and previous config saved to /var/cache/conftool/dbconfig/20221114-173622-marostegui.json
  • 17:34 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['arclamp2001']
  • 17:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T322618)', diff saved to https://phabricator.wikimedia.org/P39557 and previous config saved to /var/cache/conftool/dbconfig/20221114-173140-ladsgroup.json
  • 17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1197 (T322618)', diff saved to https://phabricator.wikimedia.org/P39556 and previous config saved to /var/cache/conftool/dbconfig/20221114-172929-ladsgroup.json
  • 17:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 17:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 17:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T322618)', diff saved to https://phabricator.wikimedia.org/P39555 and previous config saved to /var/cache/conftool/dbconfig/20221114-172846-ladsgroup.json
  • 17:25 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:24 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 17:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P39554 and previous config saved to /var/cache/conftool/dbconfig/20221114-172116-marostegui.json
  • 17:13 dancy@deploy1002: Installation of scap version "4.28.1" completed for 559 hosts
  • 17:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P39553 and previous config saved to /var/cache/conftool/dbconfig/20221114-171340-ladsgroup.json
  • 17:13 dancy@deploy1002: Installing scap version "4.28.1" for 559 hosts
  • 17:10 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 2:00:00 on wcqs1002.eqiad.wmnet with reason: Reboot for kernel update
  • 17:10 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 2:00:00 on wcqs1002.eqiad.wmnet with reason: Reboot for kernel update
  • 17:09 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs4005.ulsfo.wmnet
  • 17:09 sukhe@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:08 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wcqs[2001-2003].codfw.wmnet with reason: Reboot for kernel update
  • 17:07 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wcqs[2001-2003].codfw.wmnet with reason: Reboot for kernel update
  • 17:07 ryankemper@cumin1001: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on 12 hosts with reason: Reboot for kernel update
  • 17:07 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 12 hosts with reason: Reboot for kernel update
  • 17:07 sukhe@cumin2002: START - Cookbook sre.dns.netbox
  • 17:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T321126)', diff saved to https://phabricator.wikimedia.org/P39552 and previous config saved to /var/cache/conftool/dbconfig/20221114-170609-marostegui.json
  • 17:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1201 (T321126)', diff saved to https://phabricator.wikimedia.org/P39551 and previous config saved to /var/cache/conftool/dbconfig/20221114-170357-marostegui.json
  • 17:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 17:03 jdrewniak@deploy1002: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 03m 45s)
  • 17:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 17:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T321126)', diff saved to https://phabricator.wikimedia.org/P39550 and previous config saved to /var/cache/conftool/dbconfig/20221114-170325-marostegui.json
  • 17:03 sukhe@cumin2002: START - Cookbook sre.hosts.decommission for hosts lvs4005.ulsfo.wmnet
  • 16:59 jdrewniak@deploy1002: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 03m 58s)
  • 16:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P39549 and previous config saved to /var/cache/conftool/dbconfig/20221114-165833-ladsgroup.json
  • 16:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P39548 and previous config saved to /var/cache/conftool/dbconfig/20221114-164818-marostegui.json
  • 16:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T322618)', diff saved to https://phabricator.wikimedia.org/P39547 and previous config saved to /var/cache/conftool/dbconfig/20221114-164327-ladsgroup.json
  • 16:41 sukhe: depooled lvs4005
  • 16:41 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4005.ulsfo.wmnet with reason: downtimed, in the process of decom
  • 16:41 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on lvs4005.ulsfo.wmnet with reason: downtimed, in the process of decom
  • 16:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1188 (T322618)', diff saved to https://phabricator.wikimedia.org/P39546 and previous config saved to /var/cache/conftool/dbconfig/20221114-164015-ladsgroup.json
  • 16:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39545 and previous config saved to /var/cache/conftool/dbconfig/20221114-163954-ladsgroup.json
  • 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39544 and previous config saved to /var/cache/conftool/dbconfig/20221114-163910-marostegui.json
  • 16:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 16:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 16:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2166.codfw.wmnet with reason: Host crashed T323040
  • 16:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2166.codfw.wmnet with reason: Host crashed T323040
  • 16:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P39543 and previous config saved to /var/cache/conftool/dbconfig/20221114-163312-marostegui.json
  • 16:30 sukhe: cr4-ulsfo: set routing-options static route 198.35.26.96/28 next-hop 10.128.0.18 [lvs4005 decomm]
  • 16:29 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 16:28 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 16:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T321130)', diff saved to https://phabricator.wikimedia.org/P39542 and previous config saved to /var/cache/conftool/dbconfig/20221114-162851-marostegui.json
  • 16:28 sukhe: cr3-ulsfo: set routing-options static route 198.35.26.96/28 next-hop 10.128.0.18 [lvs4005 decomm]
  • 16:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P39541 and previous config saved to /var/cache/conftool/dbconfig/20221114-162448-ladsgroup.json
  • 16:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T321126)', diff saved to https://phabricator.wikimedia.org/P39540 and previous config saved to /var/cache/conftool/dbconfig/20221114-161804-marostegui.json
  • 16:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T321126)', diff saved to https://phabricator.wikimedia.org/P39539 and previous config saved to /var/cache/conftool/dbconfig/20221114-161553-marostegui.json
  • 16:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 16:15 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['dbprov2004']
  • 16:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 16:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39537 and previous config saved to /var/cache/conftool/dbconfig/20221114-161520-marostegui.json
  • 16:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P39536 and previous config saved to /var/cache/conftool/dbconfig/20221114-161344-marostegui.json
  • 16:13 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['puppetdb2003']
  • 16:12 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['puppetdb2003']
  • 16:12 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['puppetdb2003']
  • 16:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P39535 and previous config saved to /var/cache/conftool/dbconfig/20221114-160941-ladsgroup.json
  • 16:08 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dbprov2004']
  • 16:08 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['dbprov2004']
  • 16:03 sukhe: reprepro -C main include bullseye-wikimedia varnish-modules_0.15.0-2_amd64.changes: T321309
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'MySQL issues', diff saved to https://phabricator.wikimedia.org/P39534 and previous config saved to /var/cache/conftool/dbconfig/20221114-160140-ladsgroup.json
  • 16:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P39533 and previous config saved to /var/cache/conftool/dbconfig/20221114-160014-marostegui.json
  • 15:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P39532 and previous config saved to /var/cache/conftool/dbconfig/20221114-155838-marostegui.json
  • 15:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39531 and previous config saved to /var/cache/conftool/dbconfig/20221114-155435-ladsgroup.json
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39530 and previous config saved to /var/cache/conftool/dbconfig/20221114-155222-ladsgroup.json
  • 15:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39529 and previous config saved to /var/cache/conftool/dbconfig/20221114-155201-ladsgroup.json
  • 15:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P39528 and previous config saved to /var/cache/conftool/dbconfig/20221114-154507-marostegui.json
  • 15:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T321130)', diff saved to https://phabricator.wikimedia.org/P39527 and previous config saved to /var/cache/conftool/dbconfig/20221114-154331-marostegui.json
  • 15:42 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['puppetdb2003']
  • 15:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T322618)', diff saved to https://phabricator.wikimedia.org/P39526 and previous config saved to /var/cache/conftool/dbconfig/20221114-153903-ladsgroup.json
  • 15:38 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dbprov2004']
  • 15:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P39525 and previous config saved to /var/cache/conftool/dbconfig/20221114-153654-ladsgroup.json
  • 15:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T321130)', diff saved to https://phabricator.wikimedia.org/P39524 and previous config saved to /var/cache/conftool/dbconfig/20221114-153030-marostegui.json
  • 15:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 15:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39523 and previous config saved to /var/cache/conftool/dbconfig/20221114-153001-marostegui.json
  • 15:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 15:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T321130)', diff saved to https://phabricator.wikimedia.org/P39522 and previous config saved to /var/cache/conftool/dbconfig/20221114-152936-marostegui.json
  • 15:28 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1021.eqiad.wmnet with OS bullseye
  • 15:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39521 and previous config saved to /var/cache/conftool/dbconfig/20221114-152749-marostegui.json
  • 15:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 15:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 15:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321126)', diff saved to https://phabricator.wikimedia.org/P39520 and previous config saved to /var/cache/conftool/dbconfig/20221114-152728-marostegui.json
  • 15:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39519 and previous config saved to /var/cache/conftool/dbconfig/20221114-152356-ladsgroup.json
  • 15:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P39518 and previous config saved to /var/cache/conftool/dbconfig/20221114-152148-ladsgroup.json
  • 15:15 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 15:15 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 15:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39516 and previous config saved to /var/cache/conftool/dbconfig/20221114-151428-marostegui.json
  • 15:13 urandom: initiating Cassandra bootstrap, aqs1019-a -- T307802
  • 15:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P39515 and previous config saved to /var/cache/conftool/dbconfig/20221114-151222-marostegui.json
  • 15:11 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 15:10 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 15:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1021.eqiad.wmnet with reason: host reimage
  • 15:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39514 and previous config saved to /var/cache/conftool/dbconfig/20221114-150850-ladsgroup.json
  • 15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39513 and previous config saved to /var/cache/conftool/dbconfig/20221114-150642-ladsgroup.json
  • 15:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39512 and previous config saved to /var/cache/conftool/dbconfig/20221114-150531-ladsgroup.json
  • 15:05 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1021.eqiad.wmnet with reason: host reimage
  • 15:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 15:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 15:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T322618)', diff saved to https://phabricator.wikimedia.org/P39511 and previous config saved to /var/cache/conftool/dbconfig/20221114-150509-ladsgroup.json
  • 14:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39510 and previous config saved to /var/cache/conftool/dbconfig/20221114-145921-marostegui.json
  • 14:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P39509 and previous config saved to /var/cache/conftool/dbconfig/20221114-145715-marostegui.json
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T322618)', diff saved to https://phabricator.wikimedia.org/P39508 and previous config saved to /var/cache/conftool/dbconfig/20221114-145343-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2175 (T322618)', diff saved to https://phabricator.wikimedia.org/P39507 and previous config saved to /var/cache/conftool/dbconfig/20221114-145122-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 14:51 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1021.eqiad.wmnet with OS bullseye
  • 14:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39506 and previous config saved to /var/cache/conftool/dbconfig/20221114-145101-ladsgroup.json
  • 14:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39505 and previous config saved to /var/cache/conftool/dbconfig/20221114-145003-ladsgroup.json
  • 14:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1021.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 14:49 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1021.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 14:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T321130)', diff saved to https://phabricator.wikimedia.org/P39504 and previous config saved to /var/cache/conftool/dbconfig/20221114-144415-marostegui.json
  • 14:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321126)', diff saved to https://phabricator.wikimedia.org/P39503 and previous config saved to /var/cache/conftool/dbconfig/20221114-144209-marostegui.json
  • 14:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T321126)', diff saved to https://phabricator.wikimedia.org/P39502 and previous config saved to /var/cache/conftool/dbconfig/20221114-143957-marostegui.json
  • 14:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 14:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 14:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T321126)', diff saved to https://phabricator.wikimedia.org/P39501 and previous config saved to /var/cache/conftool/dbconfig/20221114-143936-marostegui.json
  • 14:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39500 and previous config saved to /var/cache/conftool/dbconfig/20221114-143554-ladsgroup.json
  • 14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39499 and previous config saved to /var/cache/conftool/dbconfig/20221114-143456-ladsgroup.json
  • 14:33 taavi: ^ correction, starting it on mwmaint1002, not deploy1002
  • 14:32 taavi: START taavi@deploy1002:~$ foreachwikiindblist group1 extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all | tee T315510.log # T315510
  • 14:30 taavi@deploy1002: Finished scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353) (duration: 07m 05s)
  • 14:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P39498 and previous config saved to /var/cache/conftool/dbconfig/20221114-142429-marostegui.json
  • 14:23 taavi@deploy1002: taavi and matmarex: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 14:23 taavi@deploy1002: Started scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353)
  • 14:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39497 and previous config saved to /var/cache/conftool/dbconfig/20221114-142048-ladsgroup.json
  • 14:20 taavi@deploy1002: Finished scap: Backport for Use legacy DiscussionTools heading markup except on beta cluster (T314714), ThreadItemStore: Handle race conditions when finding/inserting outside of transaction (T322701), persistRevisionThreadItems: Print time taken (duration: 06m 14s)
  • 14:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T322618)', diff saved to https://phabricator.wikimedia.org/P39496 and previous config saved to /var/cache/conftool/dbconfig/20221114-141950-ladsgroup.json
  • 14:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T322618)', diff saved to https://phabricator.wikimedia.org/P39495 and previous config saved to /var/cache/conftool/dbconfig/20221114-141738-ladsgroup.json
  • 14:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 14:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T321130)', diff saved to https://phabricator.wikimedia.org/P39494 and previous config saved to /var/cache/conftool/dbconfig/20221114-141731-marostegui.json
  • 14:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 14:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 14:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T322618)', diff saved to https://phabricator.wikimedia.org/P39493 and previous config saved to /var/cache/conftool/dbconfig/20221114-141717-ladsgroup.json
  • 14:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 14:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39492 and previous config saved to /var/cache/conftool/dbconfig/20221114-141710-marostegui.json
  • 14:14 taavi@deploy1002: taavi and matmarex: Backport for Use legacy DiscussionTools heading markup except on beta cluster (T314714), ThreadItemStore: Handle race conditions when finding/inserting outside of transaction (T322701), persistRevisionThreadItems: Print time taken synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmn
  • 14:14 taavi@deploy1002: Started scap: Backport for Use legacy DiscussionTools heading markup except on beta cluster (T314714), ThreadItemStore: Handle race conditions when finding/inserting outside of transaction (T322701), persistRevisionThreadItems: Print time taken
  • 14:13 taavi@deploy1002: backport aborted: (duration: 01m 23s)
  • 14:13 taavi@deploy1002: prep aborted: (duration: 00m 06s)
  • 14:11 taavi@deploy1002: Finished scap: Backport for Separate identifiers from other statements for Lexemes (T318310) (duration: 06m 27s)
  • 14:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P39491 and previous config saved to /var/cache/conftool/dbconfig/20221114-140923-marostegui.json
  • 14:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39490 and previous config saved to /var/cache/conftool/dbconfig/20221114-140541-ladsgroup.json
  • 14:05 taavi@deploy1002: taavi and migr: Backport for Separate identifiers from other statements for Lexemes (T318310) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 14:05 taavi@deploy1002: Started scap: Backport for Separate identifiers from other statements for Lexemes (T318310)
  • 14:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39489 and previous config saved to /var/cache/conftool/dbconfig/20221114-140320-ladsgroup.json
  • 14:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 14:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 14:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T322618)', diff saved to https://phabricator.wikimedia.org/P39488 and previous config saved to /var/cache/conftool/dbconfig/20221114-140259-ladsgroup.json
  • 14:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P39487 and previous config saved to /var/cache/conftool/dbconfig/20221114-140210-ladsgroup.json
  • 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P39486 and previous config saved to /var/cache/conftool/dbconfig/20221114-140203-marostegui.json
  • 13:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T321126)', diff saved to https://phabricator.wikimedia.org/P39485 and previous config saved to /var/cache/conftool/dbconfig/20221114-135416-marostegui.json
  • 13:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T321126)', diff saved to https://phabricator.wikimedia.org/P39484 and previous config saved to /var/cache/conftool/dbconfig/20221114-135204-marostegui.json
  • 13:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T321126)', diff saved to https://phabricator.wikimedia.org/P39483 and previous config saved to /var/cache/conftool/dbconfig/20221114-135114-marostegui.json
  • 13:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P39482 and previous config saved to /var/cache/conftool/dbconfig/20221114-134752-ladsgroup.json
  • 13:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P39481 and previous config saved to /var/cache/conftool/dbconfig/20221114-134704-ladsgroup.json
  • 13:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P39480 and previous config saved to /var/cache/conftool/dbconfig/20221114-134657-marostegui.json
  • 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P39479 and previous config saved to /var/cache/conftool/dbconfig/20221114-133608-marostegui.json
  • 13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P39478 and previous config saved to /var/cache/conftool/dbconfig/20221114-133246-ladsgroup.json
  • 13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T322618)', diff saved to https://phabricator.wikimedia.org/P39477 and previous config saved to /var/cache/conftool/dbconfig/20221114-133157-ladsgroup.json
  • 13:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39476 and previous config saved to /var/cache/conftool/dbconfig/20221114-133150-marostegui.json
  • 13:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P39475 and previous config saved to /var/cache/conftool/dbconfig/20221114-132101-marostegui.json
  • 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39474 and previous config saved to /var/cache/conftool/dbconfig/20221114-132008-marostegui.json
  • 13:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 13:19 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 13:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39473 and previous config saved to /var/cache/conftool/dbconfig/20221114-131946-marostegui.json
  • 13:19 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
  • 13:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T322618)', diff saved to https://phabricator.wikimedia.org/P39472 and previous config saved to /var/cache/conftool/dbconfig/20221114-131740-ladsgroup.json
  • 13:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2148 (T322618)', diff saved to https://phabricator.wikimedia.org/P39471 and previous config saved to /var/cache/conftool/dbconfig/20221114-131519-ladsgroup.json
  • 13:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 13:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 13:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39470 and previous config saved to /var/cache/conftool/dbconfig/20221114-131457-ladsgroup.json
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T321126)', diff saved to https://phabricator.wikimedia.org/P39469 and previous config saved to /var/cache/conftool/dbconfig/20221114-130555-marostegui.json
  • 13:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P39468 and previous config saved to /var/cache/conftool/dbconfig/20221114-130440-marostegui.json
  • 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T321126)', diff saved to https://phabricator.wikimedia.org/P39467 and previous config saved to /var/cache/conftool/dbconfig/20221114-130343-marostegui.json
  • 13:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 13:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39466 and previous config saved to /var/cache/conftool/dbconfig/20221114-130322-marostegui.json
  • 12:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P39465 and previous config saved to /var/cache/conftool/dbconfig/20221114-125951-ladsgroup.json
  • 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P39464 and previous config saved to /var/cache/conftool/dbconfig/20221114-124934-marostegui.json
  • 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P39463 and previous config saved to /var/cache/conftool/dbconfig/20221114-124815-marostegui.json
  • 12:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P39462 and previous config saved to /var/cache/conftool/dbconfig/20221114-124444-ladsgroup.json
  • 12:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39461 and previous config saved to /var/cache/conftool/dbconfig/20221114-123427-marostegui.json
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P39460 and previous config saved to /var/cache/conftool/dbconfig/20221114-123309-marostegui.json
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T322618)', diff saved to https://phabricator.wikimedia.org/P39459 and previous config saved to /var/cache/conftool/dbconfig/20221114-123141-ladsgroup.json
  • 12:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:31 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 12:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 12:31 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 12:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39458 and previous config saved to /var/cache/conftool/dbconfig/20221114-123103-ladsgroup.json
  • 12:30 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 12:30 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 12:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39457 and previous config saved to /var/cache/conftool/dbconfig/20221114-122938-ladsgroup.json
  • 12:28 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39456 and previous config saved to /var/cache/conftool/dbconfig/20221114-122717-ladsgroup.json
  • 12:27 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 12:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T322618)', diff saved to https://phabricator.wikimedia.org/P39455 and previous config saved to /var/cache/conftool/dbconfig/20221114-122655-ladsgroup.json
  • 12:26 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:25 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39454 and previous config saved to /var/cache/conftool/dbconfig/20221114-122214-marostegui.json
  • 12:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 12:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 12:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39453 and previous config saved to /var/cache/conftool/dbconfig/20221114-121802-marostegui.json
  • 12:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P39452 and previous config saved to /var/cache/conftool/dbconfig/20221114-121556-ladsgroup.json
  • 12:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39451 and previous config saved to /var/cache/conftool/dbconfig/20221114-121547-marostegui.json
  • 12:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39450 and previous config saved to /var/cache/conftool/dbconfig/20221114-121525-marostegui.json
  • 12:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 12:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 12:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39449 and previous config saved to /var/cache/conftool/dbconfig/20221114-121202-marostegui.json
  • 12:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P39448 and previous config saved to /var/cache/conftool/dbconfig/20221114-121149-ladsgroup.json
  • 12:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P39447 and previous config saved to /var/cache/conftool/dbconfig/20221114-120043-ladsgroup.json
  • 12:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P39446 and previous config saved to /var/cache/conftool/dbconfig/20221114-120019-marostegui.json
  • 11:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39445 and previous config saved to /var/cache/conftool/dbconfig/20221114-115656-marostegui.json
  • 11:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P39444 and previous config saved to /var/cache/conftool/dbconfig/20221114-115641-ladsgroup.json
  • 11:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39443 and previous config saved to /var/cache/conftool/dbconfig/20221114-114537-ladsgroup.json
  • 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P39442 and previous config saved to /var/cache/conftool/dbconfig/20221114-114512-marostegui.json
  • 11:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39441 and previous config saved to /var/cache/conftool/dbconfig/20221114-114326-ladsgroup.json
  • 11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 11:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 11:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 11:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T322618)', diff saved to https://phabricator.wikimedia.org/P39440 and previous config saved to /var/cache/conftool/dbconfig/20221114-114244-ladsgroup.json
  • 11:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39439 and previous config saved to /var/cache/conftool/dbconfig/20221114-114150-marostegui.json
  • 11:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T322618)', diff saved to https://phabricator.wikimedia.org/P39438 and previous config saved to /var/cache/conftool/dbconfig/20221114-114134-ladsgroup.json
  • 11:40 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 9231
  • 11:40 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 9231
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2126 (T322618)', diff saved to https://phabricator.wikimedia.org/P39437 and previous config saved to /var/cache/conftool/dbconfig/20221114-113913-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T322618)', diff saved to https://phabricator.wikimedia.org/P39436 and previous config saved to /var/cache/conftool/dbconfig/20221114-113837-ladsgroup.json
  • 11:34 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
  • 11:32 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
  • 11:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39435 and previous config saved to /var/cache/conftool/dbconfig/20221114-113006-marostegui.json
  • 11:29 ladsgroup@deploy1002: Finished scap: Backport for Re-add s11 in db config reload callback (T322598) (duration: 05m 01s)
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39434 and previous config saved to /var/cache/conftool/dbconfig/20221114-112750-marostegui.json
  • 11:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 11:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P39433 and previous config saved to /var/cache/conftool/dbconfig/20221114-112736-ladsgroup.json
  • 11:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39432 and previous config saved to /var/cache/conftool/dbconfig/20221114-112729-marostegui.json
  • 11:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39431 and previous config saved to /var/cache/conftool/dbconfig/20221114-112643-marostegui.json
  • 11:24 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for Re-add s11 in db config reload callback (T322598) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 11:24 ladsgroup@deploy1002: Started scap: Backport for Re-add s11 in db config reload callback (T322598)
  • 11:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P39430 and previous config saved to /var/cache/conftool/dbconfig/20221114-112330-ladsgroup.json
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39429 and previous config saved to /var/cache/conftool/dbconfig/20221114-111434-marostegui.json
  • 11:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 11:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T321130)', diff saved to https://phabricator.wikimedia.org/P39428 and previous config saved to /var/cache/conftool/dbconfig/20221114-111412-marostegui.json
  • 11:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P39427 and previous config saved to /var/cache/conftool/dbconfig/20221114-111229-ladsgroup.json
  • 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P39426 and previous config saved to /var/cache/conftool/dbconfig/20221114-111222-marostegui.json
  • 11:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P39425 and previous config saved to /var/cache/conftool/dbconfig/20221114-110824-ladsgroup.json
  • 10:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P39424 and previous config saved to /var/cache/conftool/dbconfig/20221114-105906-marostegui.json
  • 10:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T322618)', diff saved to https://phabricator.wikimedia.org/P39423 and previous config saved to /var/cache/conftool/dbconfig/20221114-105723-ladsgroup.json
  • 10:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P39422 and previous config saved to /var/cache/conftool/dbconfig/20221114-105716-marostegui.json
  • 10:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T322618)', diff saved to https://phabricator.wikimedia.org/P39421 and previous config saved to /var/cache/conftool/dbconfig/20221114-105512-ladsgroup.json
  • 10:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 10:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 10:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39420 and previous config saved to /var/cache/conftool/dbconfig/20221114-105450-ladsgroup.json
  • 10:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T322618)', diff saved to https://phabricator.wikimedia.org/P39419 and previous config saved to /var/cache/conftool/dbconfig/20221114-105317-ladsgroup.json
  • 10:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T322618)', diff saved to https://phabricator.wikimedia.org/P39418 and previous config saved to /var/cache/conftool/dbconfig/20221114-105056-ladsgroup.json
  • 10:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 10:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 10:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T322618)', diff saved to https://phabricator.wikimedia.org/P39417 and previous config saved to /var/cache/conftool/dbconfig/20221114-105034-ladsgroup.json
  • 10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P39416 and previous config saved to /var/cache/conftool/dbconfig/20221114-104400-marostegui.json
  • 10:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39415 and previous config saved to /var/cache/conftool/dbconfig/20221114-104209-marostegui.json
  • 10:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39414 and previous config saved to /var/cache/conftool/dbconfig/20221114-103953-marostegui.json
  • 10:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P39413 and previous config saved to /var/cache/conftool/dbconfig/20221114-103944-ladsgroup.json
  • 10:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P39412 and previous config saved to /var/cache/conftool/dbconfig/20221114-103528-ladsgroup.json
  • 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T321130)', diff saved to https://phabricator.wikimedia.org/P39411 and previous config saved to /var/cache/conftool/dbconfig/20221114-102853-marostegui.json
  • 10:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P39410 and previous config saved to /var/cache/conftool/dbconfig/20221114-102437-ladsgroup.json
  • 10:21 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39409 and previous config saved to /var/cache/conftool/dbconfig/20221114-102155-root.json
  • 10:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P39408 and previous config saved to /var/cache/conftool/dbconfig/20221114-102021-ladsgroup.json
  • 10:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T321130)', diff saved to https://phabricator.wikimedia.org/P39407 and previous config saved to /var/cache/conftool/dbconfig/20221114-101659-marostegui.json
  • 10:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 10:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 10:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39406 and previous config saved to /var/cache/conftool/dbconfig/20221114-101637-marostegui.json
  • 10:12 vgutierrez: upgrading acme-chief on acmechief1001 to version 0.35 (requires disabling puppet on R:acme_chief::cert)
  • 10:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39405 and previous config saved to /var/cache/conftool/dbconfig/20221114-100931-ladsgroup.json
  • 10:09 ladsgroup@deploy1002: Finished scap: Backport for Rework SpecialPagesWithoutScans query (T322849) (duration: 11m 17s)
  • 10:07 vgutierrez: upload acme-chief 0.35 to apt.wm.o (buster-wikimedia) - T244232 T262251
  • 10:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39404 and previous config saved to /var/cache/conftool/dbconfig/20221114-100720-ladsgroup.json
  • 10:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 10:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 10:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 10:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39403 and previous config saved to /var/cache/conftool/dbconfig/20221114-100650-root.json
  • 10:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 10:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T322618)', diff saved to https://phabricator.wikimedia.org/P39402 and previous config saved to /var/cache/conftool/dbconfig/20221114-100515-ladsgroup.json
  • 10:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2104 (T322618)', diff saved to https://phabricator.wikimedia.org/P39401 and previous config saved to /var/cache/conftool/dbconfig/20221114-100254-ladsgroup.json
  • 10:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 10:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 10:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 10:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 10:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P39400 and previous config saved to /var/cache/conftool/dbconfig/20221114-100131-marostegui.json
  • 09:58 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for Rework SpecialPagesWithoutScans query (T322849) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 09:57 ladsgroup@deploy1002: Started scap: Backport for Rework SpecialPagesWithoutScans query (T322849)
  • 09:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 09:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39399 and previous config saved to /var/cache/conftool/dbconfig/20221114-095145-root.json
  • 09:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P39398 and previous config saved to /var/cache/conftool/dbconfig/20221114-094624-marostegui.json
  • 09:44 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
  • 09:39 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
  • 09:38 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 4788
  • 09:38 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 50083
  • 09:38 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 50083
  • 09:37 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 32934
  • 09:36 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39397 and previous config saved to /var/cache/conftool/dbconfig/20221114-093640-root.json
  • 09:35 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 32934
  • 09:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39396 and previous config saved to /var/cache/conftool/dbconfig/20221114-093118-marostegui.json
  • 09:21 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39395 and previous config saved to /var/cache/conftool/dbconfig/20221114-092135-root.json
  • 09:19 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39394 and previous config saved to /var/cache/conftool/dbconfig/20221114-091934-marostegui.json
  • 09:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 09:19 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 09:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321130)', diff saved to https://phabricator.wikimedia.org/P39393 and previous config saved to /var/cache/conftool/dbconfig/20221114-091912-marostegui.json
  • 09:18 ayounsi@cumin1001: END (ERROR) - Cookbook sre.network.peering (exit_code=97) with action 'configure' for AS: 13335
  • 09:18 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 13335
  • 09:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39392 and previous config saved to /var/cache/conftool/dbconfig/20221114-090630-root.json
  • 09:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P39391 and previous config saved to /var/cache/conftool/dbconfig/20221114-090406-marostegui.json
  • 08:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39389 and previous config saved to /var/cache/conftool/dbconfig/20221114-085125-root.json
  • 08:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P39388 and previous config saved to /var/cache/conftool/dbconfig/20221114-084859-marostegui.json
  • 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39387 and previous config saved to /var/cache/conftool/dbconfig/20221114-083620-root.json
  • 08:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321130)', diff saved to https://phabricator.wikimedia.org/P39386 and previous config saved to /var/cache/conftool/dbconfig/20221114-083352-marostegui.json
  • 08:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2145 T322620', diff saved to https://phabricator.wikimedia.org/P39385 and previous config saved to /var/cache/conftool/dbconfig/20221114-082458-root.json
  • 08:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T321130)', diff saved to https://phabricator.wikimedia.org/P39384 and previous config saved to /var/cache/conftool/dbconfig/20221114-082205-marostegui.json
  • 08:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 08:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 08:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39383 and previous config saved to /var/cache/conftool/dbconfig/20221114-082144-marostegui.json
  • 08:20 moritzm: installing php7.4 security updates (as packaged in Debian)
  • 08:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P39381 and previous config saved to /var/cache/conftool/dbconfig/20221114-080637-marostegui.json
  • 08:02 marostegui@deploy1002: Finished scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc1 master" (duration: 04m 34s)
  • 07:57 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "ProductionServices.php: Promote pc2014 to pc1 master" synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 07:57 marostegui@deploy1002: Started scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc1 master"
  • 07:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P39380 and previous config saved to /var/cache/conftool/dbconfig/20221114-075131-marostegui.json
  • 07:50 moritzm: draining ganeti1021 for eventual reimage T311687
  • 07:47 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
  • 07:47 marostegui@deploy1002: Finished scap: Backport for ProductionServices.php: Promote pc2014 to pc1 master (T322295) (duration: 05m 14s)
  • 07:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
  • 07:42 marostegui@deploy1002: marostegui and marostegui: Backport for ProductionServices.php: Promote pc2014 to pc1 master (T322295) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 07:42 marostegui@deploy1002: Started scap: Backport for ProductionServices.php: Promote pc2014 to pc1 master (T322295)
  • 07:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39379 and previous config saved to /var/cache/conftool/dbconfig/20221114-073624-marostegui.json
  • 07:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39378 and previous config saved to /var/cache/conftool/dbconfig/20221114-072203-marostegui.json
  • 07:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 07:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 07:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 07:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 07:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1181.eqiad.wmnet with reason: Maintenance
  • 07:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1181.eqiad.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2118.codfw.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2118.codfw.wmnet with reason: Maintenance
  • 06:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39377 and previous config saved to /var/cache/conftool/dbconfig/20221114-065620-marostegui.json
  • 06:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39376 and previous config saved to /var/cache/conftool/dbconfig/20221114-064113-marostegui.json
  • 06:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39375 and previous config saved to /var/cache/conftool/dbconfig/20221114-062607-marostegui.json
  • 06:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39374 and previous config saved to /var/cache/conftool/dbconfig/20221114-061100-marostegui.json
  • 06:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39373 and previous config saved to /var/cache/conftool/dbconfig/20221114-060847-marostegui.json
  • 06:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 06:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 06:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2173', diff saved to https://phabricator.wikimedia.org/P39372 and previous config saved to /var/cache/conftool/dbconfig/20221114-060207-root.json

2022-11-12

  • 23:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T318605)', diff saved to https://phabricator.wikimedia.org/P39371 and previous config saved to /var/cache/conftool/dbconfig/20221112-233420-ladsgroup.json
  • 23:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P39370 and previous config saved to /var/cache/conftool/dbconfig/20221112-231914-ladsgroup.json
  • 23:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P39369 and previous config saved to /var/cache/conftool/dbconfig/20221112-230407-ladsgroup.json
  • 22:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T318605)', diff saved to https://phabricator.wikimedia.org/P39368 and previous config saved to /var/cache/conftool/dbconfig/20221112-224900-ladsgroup.json
  • 22:46 urandom: initiating bootstrap, aqs1016-b -- T307802
  • 21:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T318605)', diff saved to https://phabricator.wikimedia.org/P39367 and previous config saved to /var/cache/conftool/dbconfig/20221112-210527-ladsgroup.json
  • 20:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39366 and previous config saved to /var/cache/conftool/dbconfig/20221112-205020-ladsgroup.json
  • 20:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39365 and previous config saved to /var/cache/conftool/dbconfig/20221112-203514-ladsgroup.json
  • 20:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T318605)', diff saved to https://phabricator.wikimedia.org/P39364 and previous config saved to /var/cache/conftool/dbconfig/20221112-202007-ladsgroup.json
  • off: uploaded python3-gjson_0.4.0 to apt.wikimedia.org bullseye-wikimedia
  • 17:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2179 (T318605)', diff saved to https://phabricator.wikimedia.org/P39363 and previous config saved to /var/cache/conftool/dbconfig/20221112-171705-ladsgroup.json
  • 17:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 17:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 17:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T318605)', diff saved to https://phabricator.wikimedia.org/P39362 and previous config saved to /var/cache/conftool/dbconfig/20221112-171643-ladsgroup.json
  • 17:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P39361 and previous config saved to /var/cache/conftool/dbconfig/20221112-170137-ladsgroup.json
  • 16:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P39360 and previous config saved to /var/cache/conftool/dbconfig/20221112-164630-ladsgroup.json
  • 16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T318605)', diff saved to https://phabricator.wikimedia.org/P39359 and previous config saved to /var/cache/conftool/dbconfig/20221112-163124-ladsgroup.json
  • 14:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1199 (T318605)', diff saved to https://phabricator.wikimedia.org/P39358 and previous config saved to /var/cache/conftool/dbconfig/20221112-144302-ladsgroup.json
  • 14:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 14:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 14:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T318605)', diff saved to https://phabricator.wikimedia.org/P39357 and previous config saved to /var/cache/conftool/dbconfig/20221112-144240-ladsgroup.json
  • 14:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P39356 and previous config saved to /var/cache/conftool/dbconfig/20221112-142734-ladsgroup.json
  • 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P39355 and previous config saved to /var/cache/conftool/dbconfig/20221112-141227-ladsgroup.json
  • 13:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T318605)', diff saved to https://phabricator.wikimedia.org/P39354 and previous config saved to /var/cache/conftool/dbconfig/20221112-135721-ladsgroup.json
  • 10:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2172 (T318605)', diff saved to https://phabricator.wikimedia.org/P39353 and previous config saved to /var/cache/conftool/dbconfig/20221112-105847-ladsgroup.json
  • 10:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 10:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 10:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T318605)', diff saved to https://phabricator.wikimedia.org/P39352 and previous config saved to /var/cache/conftool/dbconfig/20221112-105825-ladsgroup.json
  • 10:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P39351 and previous config saved to /var/cache/conftool/dbconfig/20221112-104319-ladsgroup.json
  • 10:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P39350 and previous config saved to /var/cache/conftool/dbconfig/20221112-102812-ladsgroup.json
  • 10:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T318605)', diff saved to https://phabricator.wikimedia.org/P39349 and previous config saved to /var/cache/conftool/dbconfig/20221112-101306-ladsgroup.json
  • 08:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1190 (T318605)', diff saved to https://phabricator.wikimedia.org/P39348 and previous config saved to /var/cache/conftool/dbconfig/20221112-082623-ladsgroup.json
  • 08:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 08:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 08:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T318605)', diff saved to https://phabricator.wikimedia.org/P39347 and previous config saved to /var/cache/conftool/dbconfig/20221112-082601-ladsgroup.json
  • 08:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39346 and previous config saved to /var/cache/conftool/dbconfig/20221112-081055-ladsgroup.json
  • 07:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39345 and previous config saved to /var/cache/conftool/dbconfig/20221112-075548-ladsgroup.json
  • 07:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T318605)', diff saved to https://phabricator.wikimedia.org/P39344 and previous config saved to /var/cache/conftool/dbconfig/20221112-074042-ladsgroup.json
  • 04:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2155 (T318605)', diff saved to https://phabricator.wikimedia.org/P39343 and previous config saved to /var/cache/conftool/dbconfig/20221112-043203-ladsgroup.json
  • 04:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 04:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 04:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 04:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 04:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39342 and previous config saved to /var/cache/conftool/dbconfig/20221112-043137-ladsgroup.json
  • 04:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P39341 and previous config saved to /var/cache/conftool/dbconfig/20221112-041631-ladsgroup.json
  • 04:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P39340 and previous config saved to /var/cache/conftool/dbconfig/20221112-040124-ladsgroup.json
  • 03:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39339 and previous config saved to /var/cache/conftool/dbconfig/20221112-034618-ladsgroup.json
  • 02:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T321130)', diff saved to https://phabricator.wikimedia.org/P39338 and previous config saved to /var/cache/conftool/dbconfig/20221112-022827-marostegui.json
  • 02:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T318605)', diff saved to https://phabricator.wikimedia.org/P39337 and previous config saved to /var/cache/conftool/dbconfig/20221112-022535-ladsgroup.json
  • 02:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 02:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 02:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P39336 and previous config saved to /var/cache/conftool/dbconfig/20221112-021321-marostegui.json
  • 01:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P39335 and previous config saved to /var/cache/conftool/dbconfig/20221112-015814-marostegui.json
  • 01:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T321130)', diff saved to https://phabricator.wikimedia.org/P39334 and previous config saved to /var/cache/conftool/dbconfig/20221112-014308-marostegui.json
  • 01:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2182 (T321130)', diff saved to https://phabricator.wikimedia.org/P39333 and previous config saved to /var/cache/conftool/dbconfig/20221112-013650-marostegui.json
  • 01:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 01:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 01:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39332 and previous config saved to /var/cache/conftool/dbconfig/20221112-013628-marostegui.json
  • 01:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P39331 and previous config saved to /var/cache/conftool/dbconfig/20221112-012122-marostegui.json
  • 01:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P39330 and previous config saved to /var/cache/conftool/dbconfig/20221112-010615-marostegui.json
  • 00:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39329 and previous config saved to /var/cache/conftool/dbconfig/20221112-005107-marostegui.json
  • 00:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39328 and previous config saved to /var/cache/conftool/dbconfig/20221112-004443-marostegui.json
  • 00:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 00:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 00:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39327 and previous config saved to /var/cache/conftool/dbconfig/20221112-004422-marostegui.json
  • 00:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P39326 and previous config saved to /var/cache/conftool/dbconfig/20221112-002915-marostegui.json
  • 00:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P39325 and previous config saved to /var/cache/conftool/dbconfig/20221112-001408-marostegui.json

2022-11-11

  • 23:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39324 and previous config saved to /var/cache/conftool/dbconfig/20221111-235902-marostegui.json
  • 23:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2168:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39323 and previous config saved to /var/cache/conftool/dbconfig/20221111-235235-marostegui.json
  • 23:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 23:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 23:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T321130)', diff saved to https://phabricator.wikimedia.org/P39322 and previous config saved to /var/cache/conftool/dbconfig/20221111-235214-marostegui.json
  • 23:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P39321 and previous config saved to /var/cache/conftool/dbconfig/20221111-233707-marostegui.json
  • 23:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P39320 and previous config saved to /var/cache/conftool/dbconfig/20221111-232201-marostegui.json
  • 23:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T321130)', diff saved to https://phabricator.wikimedia.org/P39319 and previous config saved to /var/cache/conftool/dbconfig/20221111-230654-marostegui.json
  • 23:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2159 (T321130)', diff saved to https://phabricator.wikimedia.org/P39318 and previous config saved to /var/cache/conftool/dbconfig/20221111-230037-marostegui.json
  • 23:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 23:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 23:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 23:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 23:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T321130)', diff saved to https://phabricator.wikimedia.org/P39317 and previous config saved to /var/cache/conftool/dbconfig/20221111-230000-marostegui.json
  • 22:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P39316 and previous config saved to /var/cache/conftool/dbconfig/20221111-224454-marostegui.json
  • 22:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P39315 and previous config saved to /var/cache/conftool/dbconfig/20221111-222948-marostegui.json
  • 22:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T321130)', diff saved to https://phabricator.wikimedia.org/P39314 and previous config saved to /var/cache/conftool/dbconfig/20221111-221441-marostegui.json
  • 22:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39313 and previous config saved to /var/cache/conftool/dbconfig/20221111-220939-ladsgroup.json
  • 22:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 22:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 22:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2150 (T321130)', diff saved to https://phabricator.wikimedia.org/P39312 and previous config saved to /var/cache/conftool/dbconfig/20221111-220820-marostegui.json
  • 22:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 22:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 22:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T321130)', diff saved to https://phabricator.wikimedia.org/P39311 and previous config saved to /var/cache/conftool/dbconfig/20221111-220758-marostegui.json
  • 21:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P39310 and previous config saved to /var/cache/conftool/dbconfig/20221111-215252-marostegui.json
  • 21:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P39309 and previous config saved to /var/cache/conftool/dbconfig/20221111-213745-marostegui.json
  • 21:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T321130)', diff saved to https://phabricator.wikimedia.org/P39308 and previous config saved to /var/cache/conftool/dbconfig/20221111-212239-marostegui.json
  • 21:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2122 (T321130)', diff saved to https://phabricator.wikimedia.org/P39307 and previous config saved to /var/cache/conftool/dbconfig/20221111-211611-marostegui.json
  • 21:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 21:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 21:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39306 and previous config saved to /var/cache/conftool/dbconfig/20221111-211550-marostegui.json
  • 21:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P39305 and previous config saved to /var/cache/conftool/dbconfig/20221111-210043-marostegui.json
  • 20:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 20:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 20:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T318605)', diff saved to https://phabricator.wikimedia.org/P39304 and previous config saved to /var/cache/conftool/dbconfig/20221111-205919-ladsgroup.json
  • 20:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P39303 and previous config saved to /var/cache/conftool/dbconfig/20221111-204536-marostegui.json
  • 20:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P39302 and previous config saved to /var/cache/conftool/dbconfig/20221111-204413-ladsgroup.json
  • 20:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39301 and previous config saved to /var/cache/conftool/dbconfig/20221111-203030-marostegui.json
  • 20:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P39300 and previous config saved to /var/cache/conftool/dbconfig/20221111-202906-ladsgroup.json
  • 20:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39299 and previous config saved to /var/cache/conftool/dbconfig/20221111-202413-marostegui.json
  • 20:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 20:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 20:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T321130)', diff saved to https://phabricator.wikimedia.org/P39298 and previous config saved to /var/cache/conftool/dbconfig/20221111-202351-marostegui.json
  • 20:21 mutante: phab1001,phab1004,phab2002 - systemctl reset-failed
  • 20:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T318605)', diff saved to https://phabricator.wikimedia.org/P39297 and previous config saved to /var/cache/conftool/dbconfig/20221111-201400-ladsgroup.json
  • 20:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P39296 and previous config saved to /var/cache/conftool/dbconfig/20221111-200845-marostegui.json
  • 19:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P39295 and previous config saved to /var/cache/conftool/dbconfig/20221111-195338-marostegui.json
  • 19:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T321130)', diff saved to https://phabricator.wikimedia.org/P39294 and previous config saved to /var/cache/conftool/dbconfig/20221111-193832-marostegui.json
  • 19:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2120 (T321130)', diff saved to https://phabricator.wikimedia.org/P39293 and previous config saved to /var/cache/conftool/dbconfig/20221111-193214-marostegui.json
  • 19:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 19:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 19:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T321130)', diff saved to https://phabricator.wikimedia.org/P39292 and previous config saved to /var/cache/conftool/dbconfig/20221111-193152-marostegui.json
  • 19:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P39291 and previous config saved to /var/cache/conftool/dbconfig/20221111-191646-marostegui.json
  • 19:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P39290 and previous config saved to /var/cache/conftool/dbconfig/20221111-190139-marostegui.json
  • 18:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T321130)', diff saved to https://phabricator.wikimedia.org/P39289 and previous config saved to /var/cache/conftool/dbconfig/20221111-184633-marostegui.json
  • 18:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2108 (T321130)', diff saved to https://phabricator.wikimedia.org/P39288 and previous config saved to /var/cache/conftool/dbconfig/20221111-184017-marostegui.json
  • 18:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 18:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 18:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 18:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T321130)', diff saved to https://phabricator.wikimedia.org/P39287 and previous config saved to /var/cache/conftool/dbconfig/20221111-182640-marostegui.json
  • 18:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39286 and previous config saved to /var/cache/conftool/dbconfig/20221111-181134-marostegui.json
  • 17:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39285 and previous config saved to /var/cache/conftool/dbconfig/20221111-175627-marostegui.json
  • 17:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T321130)', diff saved to https://phabricator.wikimedia.org/P39284 and previous config saved to /var/cache/conftool/dbconfig/20221111-174121-marostegui.json
  • 17:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T321130)', diff saved to https://phabricator.wikimedia.org/P39283 and previous config saved to /var/cache/conftool/dbconfig/20221111-173907-marostegui.json
  • 17:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T321130)', diff saved to https://phabricator.wikimedia.org/P39282 and previous config saved to /var/cache/conftool/dbconfig/20221111-173846-marostegui.json
  • 17:34 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=varnish-fe
  • 17:34 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-be
  • 17:34 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-tls
  • 17:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39281 and previous config saved to /var/cache/conftool/dbconfig/20221111-172339-marostegui.json
  • 17:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39280 and previous config saved to /var/cache/conftool/dbconfig/20221111-170833-marostegui.json
  • 16:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T321130)', diff saved to https://phabricator.wikimedia.org/P39279 and previous config saved to /var/cache/conftool/dbconfig/20221111-165326-marostegui.json
  • 16:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1194 (T321130)', diff saved to https://phabricator.wikimedia.org/P39278 and previous config saved to /var/cache/conftool/dbconfig/20221111-165113-marostegui.json
  • 16:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 16:50 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 16:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39277 and previous config saved to /var/cache/conftool/dbconfig/20221111-165051-marostegui.json
  • 16:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39275 and previous config saved to /var/cache/conftool/dbconfig/20221111-163545-marostegui.json
  • 16:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39274 and previous config saved to /var/cache/conftool/dbconfig/20221111-162038-marostegui.json
  • 16:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 16:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 16:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39273 and previous config saved to /var/cache/conftool/dbconfig/20221111-161528-ladsgroup.json
  • 16:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39272 and previous config saved to /var/cache/conftool/dbconfig/20221111-160532-marostegui.json
  • 16:05 vgutierrez: restart varnish in cp2042
  • 16:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P39271 and previous config saved to /var/cache/conftool/dbconfig/20221111-160022-ladsgroup.json
  • 15:58 vgutierrez: rolling restart of varnish in cp4045 - cp4050 - T322903
  • 15:57 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:56 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS buster
  • 15:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P39270 and previous config saved to /var/cache/conftool/dbconfig/20221111-154515-ladsgroup.json
  • 15:43 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS buster
  • 15:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39269 and previous config saved to /var/cache/conftool/dbconfig/20221111-153009-ladsgroup.json
  • 15:21 moritzm: installing node-end-of-stream security updates
  • 15:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39268 and previous config saved to /var/cache/conftool/dbconfig/20221111-150516-marostegui.json
  • 15:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 15:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 15:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T321130)', diff saved to https://phabricator.wikimedia.org/P39267 and previous config saved to /var/cache/conftool/dbconfig/20221111-150454-marostegui.json
  • 14:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P39266 and previous config saved to /var/cache/conftool/dbconfig/20221111-144948-marostegui.json
  • 14:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T318605)', diff saved to https://phabricator.wikimedia.org/P39265 and previous config saved to /var/cache/conftool/dbconfig/20221111-144047-ladsgroup.json
  • 14:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 14:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 14:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T318605)', diff saved to https://phabricator.wikimedia.org/P39264 and previous config saved to /var/cache/conftool/dbconfig/20221111-144025-ladsgroup.json
  • 14:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P39263 and previous config saved to /var/cache/conftool/dbconfig/20221111-143441-marostegui.json
  • 14:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39262 and previous config saved to /var/cache/conftool/dbconfig/20221111-142519-ladsgroup.json
  • 14:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T321130)', diff saved to https://phabricator.wikimedia.org/P39261 and previous config saved to /var/cache/conftool/dbconfig/20221111-141935-marostegui.json
  • 14:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T321130)', diff saved to https://phabricator.wikimedia.org/P39260 and previous config saved to /var/cache/conftool/dbconfig/20221111-141721-marostegui.json
  • 14:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 14:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 14:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 14:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 14:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39259 and previous config saved to /var/cache/conftool/dbconfig/20221111-141233-marostegui.json
  • 14:12 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2003-dev.codfw.wmnet with OS bullseye
  • 14:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39258 and previous config saved to /var/cache/conftool/dbconfig/20221111-141012-ladsgroup.json
  • 13:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P39257 and previous config saved to /var/cache/conftool/dbconfig/20221111-135727-marostegui.json
  • 13:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T318605)', diff saved to https://phabricator.wikimedia.org/P39256 and previous config saved to /var/cache/conftool/dbconfig/20221111-135506-ladsgroup.json
  • 13:51 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 13:50 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 13:49 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2003-dev.codfw.wmnet with reason: host reimage
  • 13:47 aborrero@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2003-dev.codfw.wmnet with reason: host reimage
  • 13:45 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 13:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P39255 and previous config saved to /var/cache/conftool/dbconfig/20221111-134221-marostegui.json
  • 13:42 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:42 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:30 moritzm: installing procmail security updates
  • 13:30 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirt2003-dev.codfw.wmnet with OS bullseye
  • 13:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39254 and previous config saved to /var/cache/conftool/dbconfig/20221111-132714-marostegui.json
  • 13:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39253 and previous config saved to /var/cache/conftool/dbconfig/20221111-132105-marostegui.json
  • 13:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T321130)', diff saved to https://phabricator.wikimedia.org/P39252 and previous config saved to /var/cache/conftool/dbconfig/20221111-132043-marostegui.json
  • 13:20 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:13 jnuche@deploy1002: sync-world aborted: (no justification provided) (duration: 17m 49s)
  • 13:13 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
  • 13:13 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 13:13 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 13:13 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 13:13 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 13:13 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 13:13 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:12 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 13:10 jnuche@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 13:10 jnuche@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
  • 13:08 jnuche@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 13:08 jnuche@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
  • 13:08 jnuche@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 13:07 jnuche@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
  • 13:06 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 13:06 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
  • 13:06 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 13:06 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 13:06 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 13:06 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 13:06 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:06 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 13:05 jnuche@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P39251 and previous config saved to /var/cache/conftool/dbconfig/20221111-130537-marostegui.json
  • 13:05 jnuche@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:01 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:01 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 12:55 jnuche@deploy1002: Started scap: (no justification provided)
  • 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P39249 and previous config saved to /var/cache/conftool/dbconfig/20221111-125030-marostegui.json
  • 12:42 moritzm: installing debootstrap bugfix updates from buster point release
  • 12:37 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 12:35 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T321130)', diff saved to https://phabricator.wikimedia.org/P39248 and previous config saved to /var/cache/conftool/dbconfig/20221111-123524-marostegui.json
  • 12:35 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:34 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host ganeti1033.eqiad.wmnet
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T321130)', diff saved to https://phabricator.wikimedia.org/P39247 and previous config saved to /var/cache/conftool/dbconfig/20221111-123310-marostegui.json
  • 12:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 12:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 12:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39246 and previous config saved to /var/cache/conftool/dbconfig/20221111-123232-marostegui.json
  • 12:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P39245 and previous config saved to /var/cache/conftool/dbconfig/20221111-121725-marostegui.json
  • 12:14 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 12:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet
  • 12:10 aborrero@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 12:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P39244 and previous config saved to /var/cache/conftool/dbconfig/20221111-120219-marostegui.json
  • 11:53 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 11:51 aborrero@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 11:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39243 and previous config saved to /var/cache/conftool/dbconfig/20221111-114712-marostegui.json
  • 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39242 and previous config saved to /var/cache/conftool/dbconfig/20221111-114458-marostegui.json
  • 11:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 11:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 11:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T321130)', diff saved to https://phabricator.wikimedia.org/P39241 and previous config saved to /var/cache/conftool/dbconfig/20221111-114437-marostegui.json
  • 11:42 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P39240 and previous config saved to /var/cache/conftool/dbconfig/20221111-112931-marostegui.json
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P39239 and previous config saved to /var/cache/conftool/dbconfig/20221111-111424-marostegui.json
  • 11:03 moritzm: installing wireshark security updates
  • 10:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T321130)', diff saved to https://phabricator.wikimedia.org/P39238 and previous config saved to /var/cache/conftool/dbconfig/20221111-105918-marostegui.json
  • 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T321130)', diff saved to https://phabricator.wikimedia.org/P39237 and previous config saved to /var/cache/conftool/dbconfig/20221111-105305-marostegui.json
  • 10:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 10:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 10:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39236 and previous config saved to /var/cache/conftool/dbconfig/20221111-105244-marostegui.json
  • 10:52 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 10:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P39235 and previous config saved to /var/cache/conftool/dbconfig/20221111-103738-marostegui.json
  • 10:22 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 10:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P39234 and previous config saved to /var/cache/conftool/dbconfig/20221111-102231-marostegui.json
  • 10:18 aborrero@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 10:15 elukey@cumin1001: END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES eqiad cluster: Roll restart of ORES's daemons.
  • 10:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39233 and previous config saved to /var/cache/conftool/dbconfig/20221111-100725-marostegui.json
  • 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39232 and previous config saved to /var/cache/conftool/dbconfig/20221111-100054-marostegui.json
  • 10:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 10:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39231 and previous config saved to /var/cache/conftool/dbconfig/20221111-100033-marostegui.json
  • 09:55 elukey@cumin1001: START - Cookbook sre.ores.roll-restart-workers for ORES eqiad cluster: Roll restart of ORES's daemons.
  • 09:54 elukey@cumin1001: END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES codfw cluster: Roll restart of ORES's daemons.
  • 09:45 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 09:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P39230 and previous config saved to /var/cache/conftool/dbconfig/20221111-094526-marostegui.json
  • 09:35 elukey@cumin1001: START - Cookbook sre.ores.roll-restart-workers for ORES codfw cluster: Roll restart of ORES's daemons.
  • 09:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P39229 and previous config saved to /var/cache/conftool/dbconfig/20221111-093020-marostegui.json
  • 09:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39228 and previous config saved to /var/cache/conftool/dbconfig/20221111-092503-ladsgroup.json
  • 09:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 09:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 09:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39227 and previous config saved to /var/cache/conftool/dbconfig/20221111-092441-ladsgroup.json
  • 09:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39226 and previous config saved to /var/cache/conftool/dbconfig/20221111-091514-marostegui.json
  • 09:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P39225 and previous config saved to /var/cache/conftool/dbconfig/20221111-090935-ladsgroup.json
  • 09:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39224 and previous config saved to /var/cache/conftool/dbconfig/20221111-090846-marostegui.json
  • 09:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:07 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1020.eqiad.wmnet to cluster eqiad and group D
  • 09:06 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1130.eqiad.wmnet with reason: Maintenance
  • 09:06 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1130.eqiad.wmnet with reason: Maintenance
  • 09:06 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1020.eqiad.wmnet to cluster eqiad and group D
  • 09:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2113.codfw.wmnet with reason: Maintenance
  • 09:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2113.codfw.wmnet with reason: Maintenance
  • 09:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1020.eqiad.wmnet
  • 09:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 09:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 09:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2112.codfw.wmnet with reason: Maintenance
  • 09:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2112.codfw.wmnet with reason: Maintenance
  • 08:55 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1020.eqiad.wmnet
  • 08:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P39223 and previous config saved to /var/cache/conftool/dbconfig/20221111-085428-ladsgroup.json
  • 08:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1020.eqiad.wmnet with OS bullseye
  • 08:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39222 and previous config saved to /var/cache/conftool/dbconfig/20221111-083922-ladsgroup.json
  • 08:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T318605)', diff saved to https://phabricator.wikimedia.org/P39221 and previous config saved to /var/cache/conftool/dbconfig/20221111-083611-ladsgroup.json
  • 08:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 08:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 08:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39220 and previous config saved to /var/cache/conftool/dbconfig/20221111-083549-ladsgroup.json
  • 08:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1020.eqiad.wmnet with reason: host reimage
  • 08:28 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1020.eqiad.wmnet with reason: host reimage
  • 08:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P39219 and previous config saved to /var/cache/conftool/dbconfig/20221111-082042-ladsgroup.json
  • 08:14 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1020.eqiad.wmnet with OS bullseye
  • 08:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1020.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 08:09 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1020.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 08:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P39218 and previous config saved to /var/cache/conftool/dbconfig/20221111-080536-ladsgroup.json
  • 07:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39217 and previous config saved to /var/cache/conftool/dbconfig/20221111-075028-ladsgroup.json
  • 06:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T321123)', diff saved to https://phabricator.wikimedia.org/P39216 and previous config saved to /var/cache/conftool/dbconfig/20221111-063240-marostegui.json
  • 06:22 vgutierrez: restart varnish on cp4047 to clear VarnishChildRestarted alert - T322903
  • 06:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P39215 and previous config saved to /var/cache/conftool/dbconfig/20221111-061733-marostegui.json
  • 06:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P39214 and previous config saved to /var/cache/conftool/dbconfig/20221111-060227-marostegui.json
  • 05:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T321123)', diff saved to https://phabricator.wikimedia.org/P39213 and previous config saved to /var/cache/conftool/dbconfig/20221111-054720-marostegui.json
  • 05:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2176 (T321123)', diff saved to https://phabricator.wikimedia.org/P39212 and previous config saved to /var/cache/conftool/dbconfig/20221111-054511-marostegui.json
  • 05:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 05:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 05:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T321123)', diff saved to https://phabricator.wikimedia.org/P39211 and previous config saved to /var/cache/conftool/dbconfig/20221111-054449-marostegui.json
  • 05:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P39210 and previous config saved to /var/cache/conftool/dbconfig/20221111-052943-marostegui.json
  • 05:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P39209 and previous config saved to /var/cache/conftool/dbconfig/20221111-051436-marostegui.json
  • 04:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T321123)', diff saved to https://phabricator.wikimedia.org/P39208 and previous config saved to /var/cache/conftool/dbconfig/20221111-045930-marostegui.json
  • 04:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2174 (T321123)', diff saved to https://phabricator.wikimedia.org/P39207 and previous config saved to /var/cache/conftool/dbconfig/20221111-045720-marostegui.json
  • 04:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 04:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 04:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T321123)', diff saved to https://phabricator.wikimedia.org/P39206 and previous config saved to /var/cache/conftool/dbconfig/20221111-045659-marostegui.json
  • 04:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P39205 and previous config saved to /var/cache/conftool/dbconfig/20221111-044152-marostegui.json
  • 04:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P39204 and previous config saved to /var/cache/conftool/dbconfig/20221111-042646-marostegui.json
  • 04:15 ejegg: civicrm upgraded from fd60273a to 93fa3f37
  • 04:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T321123)', diff saved to https://phabricator.wikimedia.org/P39203 and previous config saved to /var/cache/conftool/dbconfig/20221111-041139-marostegui.json
  • 04:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2173 (T321123)', diff saved to https://phabricator.wikimedia.org/P39202 and previous config saved to /var/cache/conftool/dbconfig/20221111-041030-marostegui.json
  • 04:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 04:10 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 04:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 04:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 04:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39201 and previous config saved to /var/cache/conftool/dbconfig/20221111-040953-marostegui.json
  • 03:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P39200 and previous config saved to /var/cache/conftool/dbconfig/20221111-035447-marostegui.json
  • 03:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P39199 and previous config saved to /var/cache/conftool/dbconfig/20221111-033940-marostegui.json
  • 03:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39198 and previous config saved to /var/cache/conftool/dbconfig/20221111-032434-marostegui.json
  • 03:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39197 and previous config saved to /var/cache/conftool/dbconfig/20221111-032224-marostegui.json
  • 03:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 03:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 03:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39196 and previous config saved to /var/cache/conftool/dbconfig/20221111-032203-marostegui.json
  • 03:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39195 and previous config saved to /var/cache/conftool/dbconfig/20221111-031358-ladsgroup.json
  • 03:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P39194 and previous config saved to /var/cache/conftool/dbconfig/20221111-030656-marostegui.json
  • 02:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P39193 and previous config saved to /var/cache/conftool/dbconfig/20221111-025851-ladsgroup.json
  • 02:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P39192 and previous config saved to /var/cache/conftool/dbconfig/20221111-025150-marostegui.json
  • 02:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P39191 and previous config saved to /var/cache/conftool/dbconfig/20221111-024345-ladsgroup.json
  • 02:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39190 and previous config saved to /var/cache/conftool/dbconfig/20221111-023643-marostegui.json
  • 02:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2167:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39189 and previous config saved to /var/cache/conftool/dbconfig/20221111-023534-marostegui.json
  • 02:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 02:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 02:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T321123)', diff saved to https://phabricator.wikimedia.org/P39188 and previous config saved to /var/cache/conftool/dbconfig/20221111-023513-marostegui.json
  • 02:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39187 and previous config saved to /var/cache/conftool/dbconfig/20221111-023252-ladsgroup.json
  • 02:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 02:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 02:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T318605)', diff saved to https://phabricator.wikimedia.org/P39186 and previous config saved to /var/cache/conftool/dbconfig/20221111-023231-ladsgroup.json
  • 02:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39185 and previous config saved to /var/cache/conftool/dbconfig/20221111-022838-ladsgroup.json
  • 02:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39184 and previous config saved to /var/cache/conftool/dbconfig/20221111-022619-ladsgroup.json
  • 02:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 02:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 02:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39183 and previous config saved to /var/cache/conftool/dbconfig/20221111-022557-ladsgroup.json
  • 02:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P39182 and previous config saved to /var/cache/conftool/dbconfig/20221111-022006-marostegui.json
  • 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39181 and previous config saved to /var/cache/conftool/dbconfig/20221111-021738-ladsgroup.json
  • 02:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P39180 and previous config saved to /var/cache/conftool/dbconfig/20221111-021725-ladsgroup.json
  • 02:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39179 and previous config saved to /var/cache/conftool/dbconfig/20221111-021717-ladsgroup.json
  • 02:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P39178 and previous config saved to /var/cache/conftool/dbconfig/20221111-021051-ladsgroup.json
  • 02:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P39177 and previous config saved to /var/cache/conftool/dbconfig/20221111-020500-marostegui.json
  • 02:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P39176 and previous config saved to /var/cache/conftool/dbconfig/20221111-020218-ladsgroup.json
  • 02:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P39175 and previous config saved to /var/cache/conftool/dbconfig/20221111-020211-ladsgroup.json
  • 01:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P39174 and previous config saved to /var/cache/conftool/dbconfig/20221111-015544-ladsgroup.json
  • 01:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T321123)', diff saved to https://phabricator.wikimedia.org/P39173 and previous config saved to /var/cache/conftool/dbconfig/20221111-014953-marostegui.json
  • 01:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2153 (T321123)', diff saved to https://phabricator.wikimedia.org/P39172 and previous config saved to /var/cache/conftool/dbconfig/20221111-014744-marostegui.json
  • 01:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 01:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 01:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T321123)', diff saved to https://phabricator.wikimedia.org/P39171 and previous config saved to /var/cache/conftool/dbconfig/20221111-014722-marostegui.json
  • 01:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T318605)', diff saved to https://phabricator.wikimedia.org/P39170 and previous config saved to /var/cache/conftool/dbconfig/20221111-014712-ladsgroup.json
  • 01:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P39169 and previous config saved to /var/cache/conftool/dbconfig/20221111-014704-ladsgroup.json
  • 01:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39168 and previous config saved to /var/cache/conftool/dbconfig/20221111-014037-ladsgroup.json
  • 01:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39167 and previous config saved to /var/cache/conftool/dbconfig/20221111-013818-ladsgroup.json
  • 01:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 01:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 01:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39166 and previous config saved to /var/cache/conftool/dbconfig/20221111-013756-ladsgroup.json
  • 01:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P39165 and previous config saved to /var/cache/conftool/dbconfig/20221111-013209-marostegui.json
  • 01:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39164 and previous config saved to /var/cache/conftool/dbconfig/20221111-013157-ladsgroup.json
  • 01:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P39163 and previous config saved to /var/cache/conftool/dbconfig/20221111-012250-ladsgroup.json
  • 01:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P39162 and previous config saved to /var/cache/conftool/dbconfig/20221111-011703-marostegui.json
  • 01:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P39161 and previous config saved to /var/cache/conftool/dbconfig/20221111-010743-ladsgroup.json
  • 01:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T321123)', diff saved to https://phabricator.wikimedia.org/P39160 and previous config saved to /var/cache/conftool/dbconfig/20221111-010156-marostegui.json
  • 00:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2146 (T321123)', diff saved to https://phabricator.wikimedia.org/P39159 and previous config saved to /var/cache/conftool/dbconfig/20221111-005947-marostegui.json
  • 00:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 00:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 00:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T321123)', diff saved to https://phabricator.wikimedia.org/P39158 and previous config saved to /var/cache/conftool/dbconfig/20221111-005925-marostegui.json
  • 00:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39157 and previous config saved to /var/cache/conftool/dbconfig/20221111-005237-ladsgroup.json
  • 00:50 jclark@cumin1001: START - Cookbook sre.hosts.provision for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2168:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39156 and previous config saved to /var/cache/conftool/dbconfig/20221111-005017-ladsgroup.json
  • 00:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 00:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 00:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T322618)', diff saved to https://phabricator.wikimedia.org/P39155 and previous config saved to /var/cache/conftool/dbconfig/20221111-004945-ladsgroup.json
  • 00:47 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 00:45 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 00:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P39154 and previous config saved to /var/cache/conftool/dbconfig/20221111-004419-marostegui.json
  • 00:43 jclark@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:43 jclark@cumin1001: START - Cookbook sre.hosts.provision for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:42 jclark@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:38 jclark@cumin1001: START - Cookbook sre.hosts.provision for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P39153 and previous config saved to /var/cache/conftool/dbconfig/20221111-003438-ladsgroup.json
  • 00:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39152 and previous config saved to /var/cache/conftool/dbconfig/20221111-003141-ladsgroup.json
  • 00:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 00:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 00:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P39151 and previous config saved to /var/cache/conftool/dbconfig/20221111-002913-marostegui.json
  • 00:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P39150 and previous config saved to /var/cache/conftool/dbconfig/20221111-001932-ladsgroup.json
  • 00:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T321123)', diff saved to https://phabricator.wikimedia.org/P39149 and previous config saved to /var/cache/conftool/dbconfig/20221111-001406-marostegui.json
  • 00:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2145 (T321123)', diff saved to https://phabricator.wikimedia.org/P39148 and previous config saved to /var/cache/conftool/dbconfig/20221111-001156-marostegui.json
  • 00:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2145.codfw.wmnet with reason: Maintenance
  • 00:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2145.codfw.wmnet with reason: Maintenance
  • 00:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 00:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 00:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T321123)', diff saved to https://phabricator.wikimedia.org/P39147 and previous config saved to /var/cache/conftool/dbconfig/20221111-001056-marostegui.json
  • 00:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T322618)', diff saved to https://phabricator.wikimedia.org/P39146 and previous config saved to /var/cache/conftool/dbconfig/20221111-000425-ladsgroup.json
  • 00:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2159 (T322618)', diff saved to https://phabricator.wikimedia.org/P39145 and previous config saved to /var/cache/conftool/dbconfig/20221111-000206-ladsgroup.json
  • 00:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 00:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 00:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 00:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 00:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T322618)', diff saved to https://phabricator.wikimedia.org/P39144 and previous config saved to /var/cache/conftool/dbconfig/20221111-000118-ladsgroup.json

2022-11-10

  • 23:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P39143 and previous config saved to /var/cache/conftool/dbconfig/20221110-235549-marostegui.json
  • 23:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P39142 and previous config saved to /var/cache/conftool/dbconfig/20221110-234612-ladsgroup.json
  • 23:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P39141 and previous config saved to /var/cache/conftool/dbconfig/20221110-234043-marostegui.json
  • 23:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P39140 and previous config saved to /var/cache/conftool/dbconfig/20221110-233105-ladsgroup.json
  • 23:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T321123)', diff saved to https://phabricator.wikimedia.org/P39139 and previous config saved to /var/cache/conftool/dbconfig/20221110-232536-marostegui.json
  • 23:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2130 (T321123)', diff saved to https://phabricator.wikimedia.org/P39138 and previous config saved to /var/cache/conftool/dbconfig/20221110-232327-marostegui.json
  • 23:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 23:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 23:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T321123)', diff saved to https://phabricator.wikimedia.org/P39137 and previous config saved to /var/cache/conftool/dbconfig/20221110-232305-marostegui.json
  • 23:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T322618)', diff saved to https://phabricator.wikimedia.org/P39136 and previous config saved to /var/cache/conftool/dbconfig/20221110-231558-ladsgroup.json
  • 23:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2150 (T322618)', diff saved to https://phabricator.wikimedia.org/P39135 and previous config saved to /var/cache/conftool/dbconfig/20221110-231339-ladsgroup.json
  • 23:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 23:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 23:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T322618)', diff saved to https://phabricator.wikimedia.org/P39134 and previous config saved to /var/cache/conftool/dbconfig/20221110-231306-ladsgroup.json
  • 23:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P39133 and previous config saved to /var/cache/conftool/dbconfig/20221110-230759-marostegui.json
  • 22:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P39132 and previous config saved to /var/cache/conftool/dbconfig/20221110-225759-ladsgroup.json
  • 22:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P39131 and previous config saved to /var/cache/conftool/dbconfig/20221110-225253-marostegui.json
  • 22:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P39130 and previous config saved to /var/cache/conftool/dbconfig/20221110-224253-ladsgroup.json
  • 22:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T321123)', diff saved to https://phabricator.wikimedia.org/P39129 and previous config saved to /var/cache/conftool/dbconfig/20221110-223746-marostegui.json
  • 22:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2116 (T321123)', diff saved to https://phabricator.wikimedia.org/P39128 and previous config saved to /var/cache/conftool/dbconfig/20221110-223537-marostegui.json
  • 22:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 22:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 22:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T321123)', diff saved to https://phabricator.wikimedia.org/P39127 and previous config saved to /var/cache/conftool/dbconfig/20221110-223515-marostegui.json
  • 22:27 eileen: thank you back on config revision changed from bbdd4315 to 2bb73bb1
  • 22:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T322618)', diff saved to https://phabricator.wikimedia.org/P39126 and previous config saved to /var/cache/conftool/dbconfig/20221110-222746-ladsgroup.json
  • 22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2122 (T322618)', diff saved to https://phabricator.wikimedia.org/P39125 and previous config saved to /var/cache/conftool/dbconfig/20221110-222526-ladsgroup.json
  • 22:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 22:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T322618)', diff saved to https://phabricator.wikimedia.org/P39124 and previous config saved to /var/cache/conftool/dbconfig/20221110-222505-ladsgroup.json
  • 22:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P39123 and previous config saved to /var/cache/conftool/dbconfig/20221110-222009-marostegui.json
  • 22:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P39122 and previous config saved to /var/cache/conftool/dbconfig/20221110-220958-ladsgroup.json
  • 22:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P39121 and previous config saved to /var/cache/conftool/dbconfig/20221110-220502-marostegui.json
  • 21:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P39120 and previous config saved to /var/cache/conftool/dbconfig/20221110-215452-ladsgroup.json
  • 21:53 jgleeson: payments updated from 17cd1956 to a058fdbc
  • 21:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T321123)', diff saved to https://phabricator.wikimedia.org/P39119 and previous config saved to /var/cache/conftool/dbconfig/20221110-214956-marostegui.json
  • 21:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2103 (T321123)', diff saved to https://phabricator.wikimedia.org/P39118 and previous config saved to /var/cache/conftool/dbconfig/20221110-214746-marostegui.json
  • 21:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 21:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 21:47 greg-g: 3:43:33 <eileen> !civicrm upgraded from 6c2e07e0 to fd60273a
  • 21:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 21:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 21:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 21:42 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 21:42 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 21:42 eileen: process-control config revision changed from 4e438cf5 to bbdd4315 (disable thank you)
  • 21:42 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 21:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T321123)', diff saved to https://phabricator.wikimedia.org/P39117 and previous config saved to /var/cache/conftool/dbconfig/20221110-214240-marostegui.json
  • 21:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T322618)', diff saved to https://phabricator.wikimedia.org/P39116 and previous config saved to /var/cache/conftool/dbconfig/20221110-213945-ladsgroup.json
  • 21:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2121 (T322618)', diff saved to https://phabricator.wikimedia.org/P39115 and previous config saved to /var/cache/conftool/dbconfig/20221110-213726-ladsgroup.json
  • 21:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 21:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 21:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T322618)', diff saved to https://phabricator.wikimedia.org/P39114 and previous config saved to /var/cache/conftool/dbconfig/20221110-213704-ladsgroup.json
  • 21:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P39113 and previous config saved to /var/cache/conftool/dbconfig/20221110-212734-marostegui.json
  • 21:27 dancy@deploy1002: Installation of scap version "4.28.0" completed for 559 hosts
  • 21:26 dancy@deploy1002: Installing scap version "4.28.0" for 559 hosts
  • 21:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P39112 and previous config saved to /var/cache/conftool/dbconfig/20221110-212158-ladsgroup.json
  • 21:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P39111 and previous config saved to /var/cache/conftool/dbconfig/20221110-211227-marostegui.json
  • 21:10 mutante: deploy1002 - armed the keyholder with deployment keys - 2 hours ago alerts started that it was not armed (does it notify people?) - got pinged that deployers got scap problems - unknown why it was disarmed - now it is armed again
  • 21:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P39110 and previous config saved to /var/cache/conftool/dbconfig/20221110-210651-ladsgroup.json
  • 21:04 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:03 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 21:03 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕞🍵 sudo cumin -b 8 A:cp 'run-puppet-agent --enable T306580'
  • 20:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T321123)', diff saved to https://phabricator.wikimedia.org/P39109 and previous config saved to /var/cache/conftool/dbconfig/20221110-205720-marostegui.json
  • 20:56 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1196 (T321123)', diff saved to https://phabricator.wikimedia.org/P39108 and previous config saved to /var/cache/conftool/dbconfig/20221110-205613-marostegui.json
  • 20:56 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 20:55 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 20:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T321123)', diff saved to https://phabricator.wikimedia.org/P39107 and previous config saved to /var/cache/conftool/dbconfig/20221110-205552-marostegui.json
  • 20:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T322618)', diff saved to https://phabricator.wikimedia.org/P39106 and previous config saved to /var/cache/conftool/dbconfig/20221110-205145-ladsgroup.json
  • 20:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2120 (T322618)', diff saved to https://phabricator.wikimedia.org/P39105 and previous config saved to /var/cache/conftool/dbconfig/20221110-204925-ladsgroup.json
  • 20:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 20:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 20:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T322618)', diff saved to https://phabricator.wikimedia.org/P39104 and previous config saved to /var/cache/conftool/dbconfig/20221110-204904-ladsgroup.json
  • 20:43 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:42 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 20:41 dancy@deploy1002: Installing scap version "4.28.0" for 559 hosts
  • 20:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P39103 and previous config saved to /var/cache/conftool/dbconfig/20221110-204045-marostegui.json
  • 20:40 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:39 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 20:36 cdanis: ✔️ cdanis@cp3053.esams.wmnet ~ 🕞🍵 sudo run-puppet-agent --enable T306580
  • 20:36 cdanis: ✔️ cdanis@cp3052.esams.wmnet ~ 🕞🍵 sudo run-puppet-agent --enable T306580
  • 20:35 cdanis: ✔️ cdanis@cp2027.codfw.wmnet ~ 🕞🍵 sudo run-puppet-agent --enable T306580
  • 20:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P39102 and previous config saved to /var/cache/conftool/dbconfig/20221110-203357-ladsgroup.json
  • 20:33 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕞🍵 sudo cumin A:cp 'disable-puppet T306580'
  • 20:32 dancy@deploy1002: Installing scap version "4.28.0" for 559 hosts
  • 20:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T321130)', diff saved to https://phabricator.wikimedia.org/P39101 and previous config saved to /var/cache/conftool/dbconfig/20221110-202744-marostegui.json
  • 20:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P39100 and previous config saved to /var/cache/conftool/dbconfig/20221110-202539-marostegui.json
  • 20:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P39099 and previous config saved to /var/cache/conftool/dbconfig/20221110-201851-ladsgroup.json
  • 20:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P39098 and previous config saved to /var/cache/conftool/dbconfig/20221110-201237-marostegui.json
  • 20:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T321123)', diff saved to https://phabricator.wikimedia.org/P39097 and previous config saved to /var/cache/conftool/dbconfig/20221110-201032-marostegui.json
  • 20:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1186 (T321123)', diff saved to https://phabricator.wikimedia.org/P39096 and previous config saved to /var/cache/conftool/dbconfig/20221110-200924-marostegui.json
  • 20:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 20:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 20:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T321123)', diff saved to https://phabricator.wikimedia.org/P39095 and previous config saved to /var/cache/conftool/dbconfig/20221110-200903-marostegui.json
  • 20:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T322618)', diff saved to https://phabricator.wikimedia.org/P39094 and previous config saved to /var/cache/conftool/dbconfig/20221110-200344-ladsgroup.json
  • 20:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2108 (T322618)', diff saved to https://phabricator.wikimedia.org/P39093 and previous config saved to /var/cache/conftool/dbconfig/20221110-200125-ladsgroup.json
  • 20:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 20:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 20:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 20:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 20:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 20:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 19:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 19:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 19:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T322618)', diff saved to https://phabricator.wikimedia.org/P39092 and previous config saved to /var/cache/conftool/dbconfig/20221110-195938-ladsgroup.json
  • 19:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P39091 and previous config saved to /var/cache/conftool/dbconfig/20221110-195731-marostegui.json
  • 19:57 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:55 dzahn@cumin2002: START - Cookbook sre.dns.netbox
  • 19:54 mutante: netbox - deleting special case phab2001-vcs.codfw.wmnet IPv4 (10.192.32.149) and IPv6 (2620:0:860:103:10:192:32:149) - T296022 - T322250
  • 19:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P39090 and previous config saved to /var/cache/conftool/dbconfig/20221110-195357-marostegui.json
  • 19:52 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:51 robh@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1033.eqiad.wmnet with OS bullseye
  • 19:51 dzahn@cumin2002: START - Cookbook sre.dns.netbox
  • 19:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39089 and previous config saved to /var/cache/conftool/dbconfig/20221110-194431-ladsgroup.json
  • 19:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T321130)', diff saved to https://phabricator.wikimedia.org/P39088 and previous config saved to /var/cache/conftool/dbconfig/20221110-194224-marostegui.json
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2178 (T321130)', diff saved to https://phabricator.wikimedia.org/P39087 and previous config saved to /var/cache/conftool/dbconfig/20221110-194031-marostegui.json
  • 19:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39086 and previous config saved to /var/cache/conftool/dbconfig/20221110-194009-marostegui.json
  • 19:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P39085 and previous config saved to /var/cache/conftool/dbconfig/20221110-193850-marostegui.json
  • 19:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2136 (T318605)', diff saved to https://phabricator.wikimedia.org/P39084 and previous config saved to /var/cache/conftool/dbconfig/20221110-193459-ladsgroup.json
  • 19:34 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1033.eqiad.wmnet with reason: host reimage
  • 19:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 19:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 19:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T318605)', diff saved to https://phabricator.wikimedia.org/P39083 and previous config saved to /var/cache/conftool/dbconfig/20221110-193437-ladsgroup.json
  • 19:32 damilare: civicrm upgraded from 07fdeed5 to 6c2e07e0
  • 19:31 robh@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1033.eqiad.wmnet with reason: host reimage
  • 19:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39082 and previous config saved to /var/cache/conftool/dbconfig/20221110-192925-ladsgroup.json
  • 19:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P39081 and previous config saved to /var/cache/conftool/dbconfig/20221110-192503-marostegui.json
  • 19:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T321123)', diff saved to https://phabricator.wikimedia.org/P39080 and previous config saved to /var/cache/conftool/dbconfig/20221110-192343-marostegui.json
  • 19:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1184 (T321123)', diff saved to https://phabricator.wikimedia.org/P39079 and previous config saved to /var/cache/conftool/dbconfig/20221110-192236-marostegui.json
  • 19:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 19:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 19:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T321123)', diff saved to https://phabricator.wikimedia.org/P39078 and previous config saved to /var/cache/conftool/dbconfig/20221110-192215-marostegui.json
  • 19:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P39077 and previous config saved to /var/cache/conftool/dbconfig/20221110-191930-ladsgroup.json
  • 19:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 19:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 19:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39076 and previous config saved to /var/cache/conftool/dbconfig/20221110-191900-ladsgroup.json
  • 19:18 robh@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1033.eqiad.wmnet with OS bullseye
  • 19:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T322618)', diff saved to https://phabricator.wikimedia.org/P39075 and previous config saved to /var/cache/conftool/dbconfig/20221110-191418-ladsgroup.json
  • 19:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T322618)', diff saved to https://phabricator.wikimedia.org/P39074 and previous config saved to /var/cache/conftool/dbconfig/20221110-191208-ladsgroup.json
  • 19:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 19:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 19:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P39073 and previous config saved to /var/cache/conftool/dbconfig/20221110-191146-ladsgroup.json
  • 19:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P39072 and previous config saved to /var/cache/conftool/dbconfig/20221110-190957-marostegui.json
  • 19:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P39071 and previous config saved to /var/cache/conftool/dbconfig/20221110-190708-marostegui.json
  • 19:05 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts phab2001.codfw.wmnet
  • 19:05 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P39070 and previous config saved to /var/cache/conftool/dbconfig/20221110-190424-ladsgroup.json
  • 19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39069 and previous config saved to /var/cache/conftool/dbconfig/20221110-190353-ladsgroup.json
  • 19:03 dzahn@cumin2002: START - Cookbook sre.dns.netbox
  • 18:58 mutante: phabricator - running decom cookbook on phab2001 - T322250
  • 18:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39068 and previous config saved to /var/cache/conftool/dbconfig/20221110-185640-ladsgroup.json
  • 18:55 dzahn@cumin2002: START - Cookbook sre.hosts.decommission for hosts phab2001.codfw.wmnet
  • 18:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39067 and previous config saved to /var/cache/conftool/dbconfig/20221110-185450-marostegui.json
  • 18:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P39066 and previous config saved to /var/cache/conftool/dbconfig/20221110-185202-marostegui.json
  • 18:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39065 and previous config saved to /var/cache/conftool/dbconfig/20221110-185135-marostegui.json
  • 18:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 18:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 18:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39064 and previous config saved to /var/cache/conftool/dbconfig/20221110-185103-marostegui.json
  • 18:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T318605)', diff saved to https://phabricator.wikimedia.org/P39063 and previous config saved to /var/cache/conftool/dbconfig/20221110-184917-ladsgroup.json
  • 18:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39062 and previous config saved to /var/cache/conftool/dbconfig/20221110-184847-ladsgroup.json
  • 18:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39061 and previous config saved to /var/cache/conftool/dbconfig/20221110-184133-ladsgroup.json
  • 18:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T321123)', diff saved to https://phabricator.wikimedia.org/P39060 and previous config saved to /var/cache/conftool/dbconfig/20221110-183655-marostegui.json
  • 18:36 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4008.ulsfo.wmnet with reason: downtimed as we are resolving issues with LVS configuration
  • 18:36 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on lvs4008.ulsfo.wmnet with reason: downtimed as we are resolving issues with LVS configuration
  • 18:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P39059 and previous config saved to /var/cache/conftool/dbconfig/20221110-183556-marostegui.json
  • 18:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1169 (T321123)', diff saved to https://phabricator.wikimedia.org/P39058 and previous config saved to /var/cache/conftool/dbconfig/20221110-183548-marostegui.json
  • 18:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 18:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 18:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T321123)', diff saved to https://phabricator.wikimedia.org/P39057 and previous config saved to /var/cache/conftool/dbconfig/20221110-183455-marostegui.json
  • 18:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39056 and previous config saved to /var/cache/conftool/dbconfig/20221110-183340-ladsgroup.json
  • 18:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P39055 and previous config saved to /var/cache/conftool/dbconfig/20221110-182627-ladsgroup.json
  • 18:26 ladsgroup@deploy1002: Synchronized portals: (no justification provided) (duration: 03m 38s)
  • 18:22 ladsgroup@deploy1002: Synchronized portals/wikipedia.org/assets: (no justification provided) (duration: 03m 46s)
  • 18:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P39054 and previous config saved to /var/cache/conftool/dbconfig/20221110-182049-marostegui.json
  • 18:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P39053 and previous config saved to /var/cache/conftool/dbconfig/20221110-181948-marostegui.json
  • 18:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:18 ladsgroup@deploy1002: Finished scap: Backport for Bump portals to HEAD (T273179) (duration: 05m 14s)
  • 18:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:17 dcausse@deploy1002: Finished deploy [wikimedia/discovery/analytics@a030f5f]: T320656: convert_to_esbulk: fix typo in config (duration: 02m 22s)
  • 18:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:15 dcausse@deploy1002: Started deploy [wikimedia/discovery/analytics@a030f5f]: T320656: convert_to_esbulk: fix typo in config
  • 18:13 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for Bump portals to HEAD (T273179) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 18:12 ladsgroup@deploy1002: Started scap: Backport for Bump portals to HEAD (T273179)
  • 18:09 volans: upgrading spicerack to 5.0.0 on cumin hosts
  • 18:05 volans: uploaded spicerack_5.0.0 to apt.wikimedia.org bullseye-wikimedia
  • 18:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39052 and previous config saved to /var/cache/conftool/dbconfig/20221110-180543-marostegui.json
  • 18:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P39051 and previous config saved to /var/cache/conftool/dbconfig/20221110-180442-marostegui.json
  • 18:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39050 and previous config saved to /var/cache/conftool/dbconfig/20221110-180228-marostegui.json
  • 18:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 18:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 18:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39049 and previous config saved to /var/cache/conftool/dbconfig/20221110-180206-marostegui.json
  • 18:01 volans: uploaded python3-gjson_0.3.0 to apt.wikimedia.org bullseye-wikimedia,unstable-wikimedia
  • 17:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T321123)', diff saved to https://phabricator.wikimedia.org/P39048 and previous config saved to /var/cache/conftool/dbconfig/20221110-174935-marostegui.json
  • 17:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1135 (T321123)', diff saved to https://phabricator.wikimedia.org/P39047 and previous config saved to /var/cache/conftool/dbconfig/20221110-174828-marostegui.json
  • 17:48 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 17:48 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 17:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T321123)', diff saved to https://phabricator.wikimedia.org/P39046 and previous config saved to /var/cache/conftool/dbconfig/20221110-174806-marostegui.json
  • 17:47 dcausse@deploy1002: Finished deploy [wikimedia/discovery/analytics@84dd7b5]: T320656: image_suggestions: schedule ad hoc dataset to fix improper suggestions (duration: 02m 18s)
  • 17:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P39045 and previous config saved to /var/cache/conftool/dbconfig/20221110-174659-marostegui.json
  • 17:44 dcausse@deploy1002: Started deploy [wikimedia/discovery/analytics@84dd7b5]: T320656: image_suggestions: schedule ad hoc dataset to fix improper suggestions
  • 17:37 sukhe: [done] running sukhe@cumin2002:~$ homer "cr*-ulsfo*" commit "Gerrit 855583: sites.yaml: add lvs4008 (ulsfo hardware refresh)"
  • 17:36 sukhe: running sukhe@cumin2002:~$ homer "cr*-ulsfo*" commit "Gerrit 855583: sites.yaml: add lvs4008 (ulsfo hardware refresh)"
  • 17:35 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4008.ulsfo.wmnet with OS buster
  • 17:34 rzl: rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_growthexperiments-userImpactDelete.service # test run for T322706 T322541
  • 17:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P39044 and previous config saved to /var/cache/conftool/dbconfig/20221110-173300-marostegui.json
  • 17:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P39043 and previous config saved to /var/cache/conftool/dbconfig/20221110-173153-marostegui.json
  • 17:28 urandom: restarting bootstrap of aqs1016-a -- T307802
  • 17:26 urandom: increasing stream throughput to 400mbit, aqs1011-{a,b} & aqs1013-{a,b} -- T307802
  • 17:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P39042 and previous config saved to /var/cache/conftool/dbconfig/20221110-172611-ladsgroup.json
  • 17:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 17:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 17:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T322618)', diff saved to https://phabricator.wikimedia.org/P39041 and previous config saved to /var/cache/conftool/dbconfig/20221110-172549-ladsgroup.json
  • 17:23 rzl: rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_growthexperiments-userImpactUpdateRecentlyEdited.service # test run for T322706 T322541
  • 17:18 robh@cumin1001: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ganeti1033']
  • 17:18 rzl: rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_growthexperiments-userImpactUpdateRecentlyRegistered.service # test run for T322706 T322541
  • 17:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P39040 and previous config saved to /var/cache/conftool/dbconfig/20221110-171753-marostegui.json
  • 17:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39039 and previous config saved to /var/cache/conftool/dbconfig/20221110-171646-marostegui.json
  • 17:13 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 17:13 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39038 and previous config saved to /var/cache/conftool/dbconfig/20221110-171329-marostegui.json
  • 17:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 17:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 17:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T321130)', diff saved to https://phabricator.wikimedia.org/P39037 and previous config saved to /var/cache/conftool/dbconfig/20221110-171308-marostegui.json
  • 17:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39036 and previous config saved to /var/cache/conftool/dbconfig/20221110-171043-ladsgroup.json
  • 17:10 robh@cumin1001: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1033']
  • 17:09 robh@cumin1001: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ganeti1033']
  • 17:09 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 17:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T321123)', diff saved to https://phabricator.wikimedia.org/P39035 and previous config saved to /var/cache/conftool/dbconfig/20221110-170247-marostegui.json
  • 17:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1134 (T321123)', diff saved to https://phabricator.wikimedia.org/P39034 and previous config saved to /var/cache/conftool/dbconfig/20221110-170139-marostegui.json
  • 17:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1134.eqiad.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1134.eqiad.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1133.eqiad.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1133.eqiad.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 (T321123)', diff saved to https://phabricator.wikimedia.org/P39033 and previous config saved to /var/cache/conftool/dbconfig/20221110-170102-marostegui.json
  • 16:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P39032 and previous config saved to /var/cache/conftool/dbconfig/20221110-165802-marostegui.json
  • 16:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39031 and previous config saved to /var/cache/conftool/dbconfig/20221110-165536-ladsgroup.json
  • 16:53 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host lvs4008.ulsfo.wmnet with OS buster
  • 16:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P39030 and previous config saved to /var/cache/conftool/dbconfig/20221110-164556-marostegui.json
  • 16:44 sukhe@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs4008.ulsfo.wmnet with OS buster
  • 16:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P39029 and previous config saved to /var/cache/conftool/dbconfig/20221110-164255-marostegui.json
  • 16:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T322618)', diff saved to https://phabricator.wikimedia.org/P39028 and previous config saved to /var/cache/conftool/dbconfig/20221110-164030-ladsgroup.json
  • 16:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T322618)', diff saved to https://phabricator.wikimedia.org/P39027 and previous config saved to /var/cache/conftool/dbconfig/20221110-163819-ladsgroup.json
  • 16:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 16:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 16:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T322618)', diff saved to https://phabricator.wikimedia.org/P39026 and previous config saved to /var/cache/conftool/dbconfig/20221110-163758-ladsgroup.json
  • 16:37 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 16:34 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 16:33 robh@cumin1001: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1033']
  • 16:33 robh@cumin1001: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti1033']
  • 16:32 robh@cumin1001: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1033']
  • 16:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P39025 and previous config saved to /var/cache/conftool/dbconfig/20221110-163049-marostegui.json
  • 16:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T321130)', diff saved to https://phabricator.wikimedia.org/P39024 and previous config saved to /var/cache/conftool/dbconfig/20221110-162749-marostegui.json
  • 16:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2128 (T321130)', diff saved to https://phabricator.wikimedia.org/P39023 and previous config saved to /var/cache/conftool/dbconfig/20221110-162453-marostegui.json
  • 16:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 16:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 16:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 16:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 16:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T321130)', diff saved to https://phabricator.wikimedia.org/P39022 and previous config saved to /var/cache/conftool/dbconfig/20221110-162416-marostegui.json
  • 16:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P39021 and previous config saved to /var/cache/conftool/dbconfig/20221110-162251-ladsgroup.json
  • 16:16 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host lvs4008.ulsfo.wmnet with OS buster
  • 16:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 (T321123)', diff saved to https://phabricator.wikimedia.org/P39020 and previous config saved to /var/cache/conftool/dbconfig/20221110-161543-marostegui.json
  • 16:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1132 (T321123)', diff saved to https://phabricator.wikimedia.org/P39019 and previous config saved to /var/cache/conftool/dbconfig/20221110-161435-marostegui.json
  • 16:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1132.eqiad.wmnet with reason: Maintenance
  • 16:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1132.eqiad.wmnet with reason: Maintenance
  • 16:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128 (T321123)', diff saved to https://phabricator.wikimedia.org/P39018 and previous config saved to /var/cache/conftool/dbconfig/20221110-161413-marostegui.json
  • 16:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P39017 and previous config saved to /var/cache/conftool/dbconfig/20221110-160906-marostegui.json
  • 16:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P39016 and previous config saved to /var/cache/conftool/dbconfig/20221110-160745-ladsgroup.json
  • 16:03 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2002-dev.codfw.wmnet with OS bullseye
  • 15:59 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 15:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P39015 and previous config saved to /var/cache/conftool/dbconfig/20221110-155907-marostegui.json
  • 15:56 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-eqiad
  • 15:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P39014 and previous config saved to /var/cache/conftool/dbconfig/20221110-155400-marostegui.json
  • 15:53 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T322618)', diff saved to https://phabricator.wikimedia.org/P39013 and previous config saved to /var/cache/conftool/dbconfig/20221110-155238-ladsgroup.json
  • 15:50 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
  • 15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T322618)', diff saved to https://phabricator.wikimedia.org/P39012 and previous config saved to /var/cache/conftool/dbconfig/20221110-154827-ladsgroup.json
  • 15:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 15:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 15:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 15:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 15:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39011 and previous config saved to /var/cache/conftool/dbconfig/20221110-154746-ladsgroup.json
  • 15:47 aborrero@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
  • 15:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P39010 and previous config saved to /var/cache/conftool/dbconfig/20221110-154400-marostegui.json
  • 15:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T321130)', diff saved to https://phabricator.wikimedia.org/P39009 and previous config saved to /var/cache/conftool/dbconfig/20221110-153853-marostegui.json
  • 15:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2123 (T321130)', diff saved to https://phabricator.wikimedia.org/P39008 and previous config saved to /var/cache/conftool/dbconfig/20221110-153559-marostegui.json
  • 15:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 15:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 15:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T321130)', diff saved to https://phabricator.wikimedia.org/P39007 and previous config saved to /var/cache/conftool/dbconfig/20221110-153537-marostegui.json
  • 15:33 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudgw2002-dev.codfw.wmnet with OS bullseye
  • 15:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P39006 and previous config saved to /var/cache/conftool/dbconfig/20221110-153240-ladsgroup.json
  • 15:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128 (T321123)', diff saved to https://phabricator.wikimedia.org/P39005 and previous config saved to /var/cache/conftool/dbconfig/20221110-152854-marostegui.json
  • 15:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P39004 and previous config saved to /var/cache/conftool/dbconfig/20221110-152031-marostegui.json
  • 15:19 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1128 (T321123)', diff saved to https://phabricator.wikimedia.org/P39003 and previous config saved to /var/cache/conftool/dbconfig/20221110-151945-marostegui.json
  • 15:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1128.eqiad.wmnet with reason: Maintenance
  • 15:19 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1128.eqiad.wmnet with reason: Maintenance
  • 15:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T321123)', diff saved to https://phabricator.wikimedia.org/P39002 and previous config saved to /var/cache/conftool/dbconfig/20221110-151924-marostegui.json
  • 15:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P39001 and previous config saved to /var/cache/conftool/dbconfig/20221110-151733-ladsgroup.json
  • 15:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 15:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 15:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 15:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 15:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P39000 and previous config saved to /var/cache/conftool/dbconfig/20221110-150524-marostegui.json
  • 15:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P38999 and previous config saved to /var/cache/conftool/dbconfig/20221110-150417-marostegui.json
  • 15:03 kharlan@deploy1002: Finished scap: Backport for refreshUserImpactData: Add option to use job queue (T322706), refreshUserImpactData: Add feature flag (T313395) (duration: 04m 47s)
  • 15:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38998 and previous config saved to /var/cache/conftool/dbconfig/20221110-150226-ladsgroup.json
  • 15:02 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 15:01 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 15:01 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 15:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38997 and previous config saved to /var/cache/conftool/dbconfig/20221110-150015-ladsgroup.json
  • 15:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 15:00 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 14:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38996 and previous config saved to /var/cache/conftool/dbconfig/20221110-145953-ladsgroup.json
  • 14:58 kharlan@deploy1002: kharlan and kharlan: Backport for refreshUserImpactData: Add option to use job queue (T322706), refreshUserImpactData: Add feature flag (T313395) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 14:58 kharlan@deploy1002: Started scap: Backport for refreshUserImpactData: Add option to use job queue (T322706), refreshUserImpactData: Add feature flag (T313395)
  • 14:51 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-codfw
  • 14:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T321130)', diff saved to https://phabricator.wikimedia.org/P38995 and previous config saved to /var/cache/conftool/dbconfig/20221110-145018-marostegui.json
  • 14:49 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp6016.drmrs.wmnet
  • 14:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P38994 and previous config saved to /var/cache/conftool/dbconfig/20221110-144911-marostegui.json
  • 14:46 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet
  • 14:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2111 (T321130)', diff saved to https://phabricator.wikimedia.org/P38993 and previous config saved to /var/cache/conftool/dbconfig/20221110-144602-marostegui.json
  • 14:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 14:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 14:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P38992 and previous config saved to /var/cache/conftool/dbconfig/20221110-144447-ladsgroup.json
  • 14:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 14:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 14:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 14:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 14:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T321130)', diff saved to https://phabricator.wikimedia.org/P38991 and previous config saved to /var/cache/conftool/dbconfig/20221110-144121-marostegui.json
  • 14:39 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:38 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:38 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:38 kharlan@deploy1002: backport aborted: (duration: 02m 16s)
  • 14:37 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:35 kharlan@deploy1002: Finished scap: Backport for GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled (T313395) (duration: 04m 57s)
  • 14:34 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
  • 14:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38990 and previous config saved to /var/cache/conftool/dbconfig/20221110-143404-marostegui.json
  • 14:33 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 14:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38989 and previous config saved to /var/cache/conftool/dbconfig/20221110-143256-marostegui.json
  • 14:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1119.eqiad.wmnet with reason: Maintenance
  • 14:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1119.eqiad.wmnet with reason: Maintenance
  • 14:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 (T321123)', diff saved to https://phabricator.wikimedia.org/P38988 and previous config saved to /var/cache/conftool/dbconfig/20221110-143235-marostegui.json
  • 14:31 kharlan@deploy1002: kharlan and kharlan: Backport for GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled (T313395) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 14:30 kharlan@deploy1002: Started scap: Backport for GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled (T313395)
  • 14:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P38987 and previous config saved to /var/cache/conftool/dbconfig/20221110-142938-ladsgroup.json
  • 14:29 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 14:28 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 14:28 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
  • 14:27 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
  • 14:27 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
  • 14:26 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 14:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P38986 and previous config saved to /var/cache/conftool/dbconfig/20221110-142614-marostegui.json
  • 14:25 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
  • 14:24 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 14:24 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 14:22 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 14:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:21 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:21 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:21 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 14:19 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 14:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P38985 and previous config saved to /var/cache/conftool/dbconfig/20221110-141728-marostegui.json
  • 14:14 daniel@deploy1002: Finished scap: Backport for mediawiki.org: set VE to new direct mode (duration: 08m 17s)
  • 14:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38984 and previous config saved to /var/cache/conftool/dbconfig/20221110-141431-ladsgroup.json
  • 14:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38983 and previous config saved to /var/cache/conftool/dbconfig/20221110-141220-ladsgroup.json
  • 14:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 14:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 14:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 14:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 14:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T322618)', diff saved to https://phabricator.wikimedia.org/P38982 and previous config saved to /var/cache/conftool/dbconfig/20221110-141141-ladsgroup.json
  • 14:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P38981 and previous config saved to /var/cache/conftool/dbconfig/20221110-141106-marostegui.json
  • 14:06 daniel@deploy1002: daniel and daniel: Backport for mediawiki.org: set VE to new direct mode synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 14:06 daniel@deploy1002: Started scap: Backport for mediawiki.org: set VE to new direct mode
  • 14:04 moritzm: rolling restart of FPM and Apache on mw canaries to pick up expat security update
  • 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P38980 and previous config saved to /var/cache/conftool/dbconfig/20221110-140222-marostegui.json
  • 14:00 moritzm: drain ganeti1020 for eventual reimage to bullseye T311687
  • 13:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P38979 and previous config saved to /var/cache/conftool/dbconfig/20221110-135635-ladsgroup.json
  • 13:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T321130)', diff saved to https://phabricator.wikimedia.org/P38978 and previous config saved to /var/cache/conftool/dbconfig/20221110-135600-marostegui.json
  • 13:53 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D
  • 13:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1200 (T321130)', diff saved to https://phabricator.wikimedia.org/P38977 and previous config saved to /var/cache/conftool/dbconfig/20221110-135334-marostegui.json
  • 13:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 13:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 13:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T321130)', diff saved to https://phabricator.wikimedia.org/P38976 and previous config saved to /var/cache/conftool/dbconfig/20221110-135313-marostegui.json
  • 13:53 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 13:52 moritzm: installing expat securiy updates
  • 13:50 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
  • 13:48 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 (T321123)', diff saved to https://phabricator.wikimedia.org/P38975 and previous config saved to /var/cache/conftool/dbconfig/20221110-134715-marostegui.json
  • 13:46 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 13:46 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 13:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1118 (T321123)', diff saved to https://phabricator.wikimedia.org/P38974 and previous config saved to /var/cache/conftool/dbconfig/20221110-134608-marostegui.json
  • 13:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 13:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 13:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T321123)', diff saved to https://phabricator.wikimedia.org/P38973 and previous config saved to /var/cache/conftool/dbconfig/20221110-134546-marostegui.json
  • 13:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P38972 and previous config saved to /var/cache/conftool/dbconfig/20221110-134128-ladsgroup.json
  • 13:40 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D
  • 13:39 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 13:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38971 and previous config saved to /var/cache/conftool/dbconfig/20221110-133806-marostegui.json
  • 13:37 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 13:37 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 13:36 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 13:36 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 13:35 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 13:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P38970 and previous config saved to /var/cache/conftool/dbconfig/20221110-133040-marostegui.json
  • 13:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T322618)', diff saved to https://phabricator.wikimedia.org/P38969 and previous config saved to /var/cache/conftool/dbconfig/20221110-132622-ladsgroup.json
  • 13:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38968 and previous config saved to /var/cache/conftool/dbconfig/20221110-132300-marostegui.json
  • 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T322618)', diff saved to https://phabricator.wikimedia.org/P38967 and previous config saved to /var/cache/conftool/dbconfig/20221110-132010-ladsgroup.json
  • 13:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 13:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T322618)', diff saved to https://phabricator.wikimedia.org/P38966 and previous config saved to /var/cache/conftool/dbconfig/20221110-131949-ladsgroup.json
  • 13:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P38965 and previous config saved to /var/cache/conftool/dbconfig/20221110-131533-marostegui.json
  • 13:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T321130)', diff saved to https://phabricator.wikimedia.org/P38964 and previous config saved to /var/cache/conftool/dbconfig/20221110-130753-marostegui.json
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1185 (T321130)', diff saved to https://phabricator.wikimedia.org/P38963 and previous config saved to /var/cache/conftool/dbconfig/20221110-130527-marostegui.json
  • 13:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 13:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T321130)', diff saved to https://phabricator.wikimedia.org/P38962 and previous config saved to /var/cache/conftool/dbconfig/20221110-130506-marostegui.json
  • 13:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38961 and previous config saved to /var/cache/conftool/dbconfig/20221110-130443-ladsgroup.json
  • 13:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D
  • 13:04 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D
  • 13:01 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet
  • 13:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T321123)', diff saved to https://phabricator.wikimedia.org/P38960 and previous config saved to /var/cache/conftool/dbconfig/20221110-130027-marostegui.json
  • 12:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1107 (T321123)', diff saved to https://phabricator.wikimedia.org/P38959 and previous config saved to /var/cache/conftool/dbconfig/20221110-125919-marostegui.json
  • 12:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38958 and previous config saved to /var/cache/conftool/dbconfig/20221110-125858-marostegui.json
  • 12:53 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet
  • 12:51 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 12:51 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 12:51 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:51 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38957 and previous config saved to /var/cache/conftool/dbconfig/20221110-124959-marostegui.json
  • 12:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38956 and previous config saved to /var/cache/conftool/dbconfig/20221110-124936-ladsgroup.json
  • 12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2119 (T318605)', diff saved to https://phabricator.wikimedia.org/P38955 and previous config saved to /var/cache/conftool/dbconfig/20221110-124805-ladsgroup.json
  • 12:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 12:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T318605)', diff saved to https://phabricator.wikimedia.org/P38954 and previous config saved to /var/cache/conftool/dbconfig/20221110-124743-ladsgroup.json
  • 12:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P38953 and previous config saved to /var/cache/conftool/dbconfig/20221110-124352-marostegui.json
  • 12:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38952 and previous config saved to /var/cache/conftool/dbconfig/20221110-123527-ladsgroup.json
  • 12:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38951 and previous config saved to /var/cache/conftool/dbconfig/20221110-123453-marostegui.json
  • 12:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T322618)', diff saved to https://phabricator.wikimedia.org/P38950 and previous config saved to /var/cache/conftool/dbconfig/20221110-123428-ladsgroup.json
  • 12:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P38949 and previous config saved to /var/cache/conftool/dbconfig/20221110-123237-ladsgroup.json
  • 12:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T322618)', diff saved to https://phabricator.wikimedia.org/P38948 and previous config saved to /var/cache/conftool/dbconfig/20221110-123215-ladsgroup.json
  • 12:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 12:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38947 and previous config saved to /var/cache/conftool/dbconfig/20221110-123153-ladsgroup.json
  • 12:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P38946 and previous config saved to /var/cache/conftool/dbconfig/20221110-122845-marostegui.json
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P38945 and previous config saved to /var/cache/conftool/dbconfig/20221110-122720-ladsgroup.json
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T318605)', diff saved to https://phabricator.wikimedia.org/P38944 and previous config saved to /var/cache/conftool/dbconfig/20221110-122708-ladsgroup.json
  • 12:26 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:26 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:26 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 12:26 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 12:25 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 12:25 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 12:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38943 and previous config saved to /var/cache/conftool/dbconfig/20221110-122020-ladsgroup.json
  • 12:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T321130)', diff saved to https://phabricator.wikimedia.org/P38942 and previous config saved to /var/cache/conftool/dbconfig/20221110-121946-marostegui.json
  • 12:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P38941 and previous config saved to /var/cache/conftool/dbconfig/20221110-121730-ladsgroup.json
  • 12:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P38940 and previous config saved to /var/cache/conftool/dbconfig/20221110-121647-ladsgroup.json
  • 12:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T321130)', diff saved to https://phabricator.wikimedia.org/P38939 and previous config saved to /var/cache/conftool/dbconfig/20221110-121601-marostegui.json
  • 12:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 12:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38938 and previous config saved to /var/cache/conftool/dbconfig/20221110-121339-marostegui.json
  • 12:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 12:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 12:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38937 and previous config saved to /var/cache/conftool/dbconfig/20221110-121313-marostegui.json
  • 12:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P38936 and previous config saved to /var/cache/conftool/dbconfig/20221110-121202-ladsgroup.json
  • 12:08 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry
  • 12:06 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry
  • 12:06 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:06 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 100%: Index rebuilt', diff saved to https://phabricator.wikimedia.org/P38935 and previous config saved to /var/cache/conftool/dbconfig/20221110-120537-ladsgroup.json
  • 12:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38934 and previous config saved to /var/cache/conftool/dbconfig/20221110-120513-ladsgroup.json
  • 12:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T318605)', diff saved to https://phabricator.wikimedia.org/P38933 and previous config saved to /var/cache/conftool/dbconfig/20221110-120224-ladsgroup.json
  • 12:02 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:02 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:01 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:01 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P38932 and previous config saved to /var/cache/conftool/dbconfig/20221110-120140-ladsgroup.json
  • 11:59 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 11:59 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 11:58 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 11:58 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 11:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P38931 and previous config saved to /var/cache/conftool/dbconfig/20221110-115807-marostegui.json
  • 11:57 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 11:57 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 11:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P38930 and previous config saved to /var/cache/conftool/dbconfig/20221110-115655-ladsgroup.json
  • 11:55 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 11:52 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 75%: Index rebuilt', diff saved to https://phabricator.wikimedia.org/P38929 and previous config saved to /var/cache/conftool/dbconfig/20221110-115032-ladsgroup.json
  • 11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38928 and previous config saved to /var/cache/conftool/dbconfig/20221110-115007-ladsgroup.json
  • 11:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38927 and previous config saved to /var/cache/conftool/dbconfig/20221110-114634-ladsgroup.json
  • 11:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38926 and previous config saved to /var/cache/conftool/dbconfig/20221110-114422-ladsgroup.json
  • 11:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 11:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 11:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38925 and previous config saved to /var/cache/conftool/dbconfig/20221110-114400-ladsgroup.json
  • 11:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P38924 and previous config saved to /var/cache/conftool/dbconfig/20221110-114300-marostegui.json
  • 11:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T318605)', diff saved to https://phabricator.wikimedia.org/P38923 and previous config saved to /var/cache/conftool/dbconfig/20221110-114149-ladsgroup.json
  • 11:41 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wdqs-all
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38922 and previous config saved to /var/cache/conftool/dbconfig/20221110-113958-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38921 and previous config saved to /var/cache/conftool/dbconfig/20221110-113948-ladsgroup.json
  • 11:38 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38920 and previous config saved to /var/cache/conftool/dbconfig/20221110-113853-root.json
  • 11:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 25%: Index rebuilt', diff saved to https://phabricator.wikimedia.org/P38919 and previous config saved to /var/cache/conftool/dbconfig/20221110-113526-ladsgroup.json
  • 11:31 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wdqs-all
  • 11:31 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wcqs-public
  • 11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P38918 and previous config saved to /var/cache/conftool/dbconfig/20221110-112854-ladsgroup.json
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38917 and previous config saved to /var/cache/conftool/dbconfig/20221110-112753-marostegui.json
  • 11:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38916 and previous config saved to /var/cache/conftool/dbconfig/20221110-112441-ladsgroup.json
  • 11:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38915 and previous config saved to /var/cache/conftool/dbconfig/20221110-112403-marostegui.json
  • 11:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 11:23 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38914 and previous config saved to /var/cache/conftool/dbconfig/20221110-112348-root.json
  • 11:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 11:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38913 and previous config saved to /var/cache/conftool/dbconfig/20221110-112342-marostegui.json
  • 11:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 10%: Index rebuilt', diff saved to https://phabricator.wikimedia.org/P38912 and previous config saved to /var/cache/conftool/dbconfig/20221110-112022-ladsgroup.json
  • 11:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P38911 and previous config saved to /var/cache/conftool/dbconfig/20221110-111347-ladsgroup.json
  • 11:13 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wcqs-public
  • 11:13 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38910 and previous config saved to /var/cache/conftool/dbconfig/20221110-111323-marostegui.json
  • 11:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 11:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 11:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38909 and previous config saved to /var/cache/conftool/dbconfig/20221110-111244-marostegui.json
  • 11:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38908 and previous config saved to /var/cache/conftool/dbconfig/20221110-110935-ladsgroup.json
  • 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38907 and previous config saved to /var/cache/conftool/dbconfig/20221110-110843-root.json
  • 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P38906 and previous config saved to /var/cache/conftool/dbconfig/20221110-110835-marostegui.json
  • 11:00 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 10:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38905 and previous config saved to /var/cache/conftool/dbconfig/20221110-105841-ladsgroup.json
  • 10:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P38904 and previous config saved to /var/cache/conftool/dbconfig/20221110-105738-marostegui.json
  • 10:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38903 and previous config saved to /var/cache/conftool/dbconfig/20221110-105628-ladsgroup.json
  • 10:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 10:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 10:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38902 and previous config saved to /var/cache/conftool/dbconfig/20221110-105428-ladsgroup.json
  • 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38901 and previous config saved to /var/cache/conftool/dbconfig/20221110-105338-root.json
  • 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P38900 and previous config saved to /var/cache/conftool/dbconfig/20221110-105329-marostegui.json
  • 10:53 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 10:50 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38899 and previous config saved to /var/cache/conftool/dbconfig/20221110-104919-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 10:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 10:47 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 10:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P38898 and previous config saved to /var/cache/conftool/dbconfig/20221110-104231-marostegui.json
  • 10:38 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38897 and previous config saved to /var/cache/conftool/dbconfig/20221110-103827-root.json
  • 10:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38896 and previous config saved to /var/cache/conftool/dbconfig/20221110-103822-marostegui.json
  • 10:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38895 and previous config saved to /var/cache/conftool/dbconfig/20221110-103533-marostegui.json
  • 10:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T321130)', diff saved to https://phabricator.wikimedia.org/P38894 and previous config saved to /var/cache/conftool/dbconfig/20221110-103512-marostegui.json
  • 10:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38893 and previous config saved to /var/cache/conftool/dbconfig/20221110-102725-marostegui.json
  • 10:26 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38892 and previous config saved to /var/cache/conftool/dbconfig/20221110-102617-marostegui.json
  • 10:26 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 10:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 10:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38891 and previous config saved to /var/cache/conftool/dbconfig/20221110-102556-marostegui.json
  • 10:25 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1013.eqiad.wmnet to cluster eqiad and group B
  • 10:24 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1013.eqiad.wmnet to cluster eqiad and group B
  • 10:23 moritzm: installing libxml2 security updates
  • 10:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1013.eqiad.wmnet
  • 10:23 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 10:22 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 10:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P38890 and previous config saved to /var/cache/conftool/dbconfig/20221110-102005-marostegui.json
  • 10:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1013.eqiad.wmnet
  • 10:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P38889 and previous config saved to /var/cache/conftool/dbconfig/20221110-101050-marostegui.json
  • 10:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P38888 and previous config saved to /var/cache/conftool/dbconfig/20221110-100459-marostegui.json
  • 09:57 marostegui@cumin1001: dbctl commit (dc=all): 'Reduce es4 master weight', diff saved to https://phabricator.wikimedia.org/P38887 and previous config saved to /var/cache/conftool/dbconfig/20221110-095724-marostegui.json
  • 09:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P38886 and previous config saved to /var/cache/conftool/dbconfig/20221110-095543-marostegui.json
  • 09:53 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38885 and previous config saved to /var/cache/conftool/dbconfig/20221110-095313-root.json
  • 09:52 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 09:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T321130)', diff saved to https://phabricator.wikimedia.org/P38884 and previous config saved to /var/cache/conftool/dbconfig/20221110-094952-marostegui.json
  • 09:49 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 09:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T321130)', diff saved to https://phabricator.wikimedia.org/P38883 and previous config saved to /var/cache/conftool/dbconfig/20221110-094604-marostegui.json
  • 09:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1110.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1110.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100 (T321130)', diff saved to https://phabricator.wikimedia.org/P38882 and previous config saved to /var/cache/conftool/dbconfig/20221110-094542-marostegui.json
  • 09:44 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 09:43 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 09:43 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 09:42 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 09:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38881 and previous config saved to /var/cache/conftool/dbconfig/20221110-094037-marostegui.json
  • 09:39 marostegui@deploy1002: Finished scap: Backport for Revert "db-production.php: Disable es5 writes" (duration: 04m 20s)
  • 09:39 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 09:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38880 and previous config saved to /var/cache/conftool/dbconfig/20221110-093929-marostegui.json
  • 09:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 09:36 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-eqiad
  • 09:35 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "db-production.php: Disable es5 writes" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 09:35 marostegui@deploy1002: Started scap: Backport for Revert "db-production.php: Disable es5 writes"
  • 09:34 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
  • 09:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es1023 T322187', diff saved to https://phabricator.wikimedia.org/P38879 and previous config saved to /var/cache/conftool/dbconfig/20221110-093354-root.json
  • 09:32 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es1024 to es5 primary T322187', diff saved to https://phabricator.wikimedia.org/P38878 and previous config saved to /var/cache/conftool/dbconfig/20221110-093243-root.json
  • 09:32 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 09:32 marostegui: Starting es5 eqiad failover from es1023 to es1024 T322187
  • 09:31 marostegui@deploy1002: Finished scap: Backport for db-production.php: Disable es5 writes (T322187) (duration: 04m 39s)
  • 09:31 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 09:31 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 09:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P38877 and previous config saved to /var/cache/conftool/dbconfig/20221110-093036-marostegui.json
  • 09:30 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 09:27 marostegui@deploy1002: marostegui and marostegui: Backport for db-production.php: Disable es5 writes (T322187) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 09:27 marostegui@deploy1002: Started scap: Backport for db-production.php: Disable es5 writes (T322187)
  • 09:23 marostegui@cumin1001: dbctl commit (dc=all): 'Set es1024 with weight 0 T322187', diff saved to https://phabricator.wikimedia.org/P38876 and previous config saved to /var/cache/conftool/dbconfig/20221110-092336-root.json
  • 09:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322187
  • 09:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322187
  • 09:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38875 and previous config saved to /var/cache/conftool/dbconfig/20221110-092215-marostegui.json
  • 09:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1013.eqiad.wmnet with OS bullseye
  • 09:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P38874 and previous config saved to /var/cache/conftool/dbconfig/20221110-091530-marostegui.json
  • 09:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38873 and previous config saved to /var/cache/conftool/dbconfig/20221110-090708-marostegui.json
  • 09:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1013.eqiad.wmnet with reason: host reimage
  • 09:00 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1013.eqiad.wmnet with reason: host reimage
  • 09:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100 (T321130)', diff saved to https://phabricator.wikimedia.org/P38872 and previous config saved to /var/cache/conftool/dbconfig/20221110-090023-marostegui.json
  • 08:58 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-codfw
  • 08:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1100 (T321130)', diff saved to https://phabricator.wikimedia.org/P38871 and previous config saved to /var/cache/conftool/dbconfig/20221110-085756-marostegui.json
  • 08:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 08:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 08:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38870 and previous config saved to /var/cache/conftool/dbconfig/20221110-085735-marostegui.json
  • 08:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38869 and previous config saved to /var/cache/conftool/dbconfig/20221110-085201-marostegui.json
  • 08:46 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1013.eqiad.wmnet with OS bullseye
  • 08:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P38868 and previous config saved to /var/cache/conftool/dbconfig/20221110-084229-marostegui.json
  • 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38867 and previous config saved to /var/cache/conftool/dbconfig/20221110-083655-marostegui.json
  • 08:30 hashar@deploy1002: Finished deploy [gerrit/gerrit@84648b3]: Gerrit to 3.4.8 on gerrit1001 # T322724 (duration: 00m 08s)
  • 08:30 hashar@deploy1002: Started deploy [gerrit/gerrit@84648b3]: Gerrit to 3.4.8 on gerrit1001 # T322724
  • 08:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P38866 and previous config saved to /var/cache/conftool/dbconfig/20221110-082722-marostegui.json
  • 08:26 hashar@deploy1002: Finished deploy [gerrit/gerrit@84648b3]: Gerrit to 3.4.8 on gerrit2002 # T322724 (duration: 00m 10s)
  • 08:26 hashar@deploy1002: Started deploy [gerrit/gerrit@84648b3]: Gerrit to 3.4.8 on gerrit2002 # T322724
  • 08:17 moritzm: installing pixman security updates on buster
  • 08:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38865 and previous config saved to /var/cache/conftool/dbconfig/20221110-081216-marostegui.json
  • 08:09 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 136933
  • 08:08 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 136933
  • 08:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38864 and previous config saved to /var/cache/conftool/dbconfig/20221110-080823-marostegui.json
  • 08:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 08:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 08:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38863 and previous config saved to /var/cache/conftool/dbconfig/20221110-080746-marostegui.json
  • 08:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 08:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 07:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1123.eqiad.wmnet with reason: Maintenance
  • 07:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1122.eqiad.wmnet with reason: Maintenance
  • 07:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1123.eqiad.wmnet with reason: Maintenance
  • 07:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1122.eqiad.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2107.codfw.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 07:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2107.codfw.wmnet with reason: Maintenance
  • 07:00 ayounsi@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 06:45 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1011.eqiad.wmnet
  • 06:41 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-tool1011.eqiad.wmnet
  • 06:40 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1096.eqiad.wmnet
  • 06:32 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1096.eqiad.wmnet
  • 06:30 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1097.eqiad.wmnet
  • 06:22 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1097.eqiad.wmnet
  • 06:21 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1098.eqiad.wmnet
  • 06:13 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1098.eqiad.wmnet
  • 04:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T318605)', diff saved to https://phabricator.wikimedia.org/P38862 and previous config saved to /var/cache/conftool/dbconfig/20221110-044050-ladsgroup.json
  • 04:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 04:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 04:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T318605)', diff saved to https://phabricator.wikimedia.org/P38861 and previous config saved to /var/cache/conftool/dbconfig/20221110-044028-ladsgroup.json
  • 04:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38860 and previous config saved to /var/cache/conftool/dbconfig/20221110-042522-ladsgroup.json
  • 04:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38859 and previous config saved to /var/cache/conftool/dbconfig/20221110-041016-ladsgroup.json
  • 03:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T318605)', diff saved to https://phabricator.wikimedia.org/P38858 and previous config saved to /var/cache/conftool/dbconfig/20221110-035509-ladsgroup.json
  • 00:03 tzatziki: removing 1 file for legal compliance

2022-11-09

  • 23:57 tzatziki: removing 1 file for legal compliance
  • 23:44 tzatziki: removing 2 files for legal compliance
  • 23:22 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1034.eqiad.wmnet with OS bullseye
  • 23:17 tzatziki: removing 1 file for legal compliance
  • 23:07 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1034.eqiad.wmnet with reason: host reimage
  • 23:04 robh@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1034.eqiad.wmnet with reason: host reimage
  • 23:03 aikochou@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 23:00 aikochou@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 22:51 robh@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1034.eqiad.wmnet with OS bullseye
  • 22:34 damilare: civicrm upgraded from f2017495 to 07fdeed5
  • 22:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T318605)', diff saved to https://phabricator.wikimedia.org/P38857 and previous config saved to /var/cache/conftool/dbconfig/20221109-221551-ladsgroup.json
  • 22:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 22:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 22:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T318605)', diff saved to https://phabricator.wikimedia.org/P38856 and previous config saved to /var/cache/conftool/dbconfig/20221109-221529-ladsgroup.json
  • 22:06 robh@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti1034.eqiad.wmnet with OS bullseye
  • 22:01 robh@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1034.eqiad.wmnet with OS bullseye
  • 22:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38855 and previous config saved to /var/cache/conftool/dbconfig/20221109-220023-ladsgroup.json
  • 21:55 robh@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti1034.eqiad.wmnet with OS bullseye
  • 21:48 robh@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1034.eqiad.wmnet with OS bullseye
  • 21:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38854 and previous config saved to /var/cache/conftool/dbconfig/20221109-214516-ladsgroup.json
  • 21:37 TheresNoTime: closing UTC late backport window
  • 21:35 samtar@deploy1002: Finished scap: Backport for Fix TOC misaligned when max width option is disable (T322162) (duration: 08m 48s)
  • 21:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T318605)', diff saved to https://phabricator.wikimedia.org/P38853 and previous config saved to /var/cache/conftool/dbconfig/20221109-213010-ladsgroup.json
  • 21:30 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:29 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:29 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:26 samtar@deploy1002: samtar and nray: Backport for Fix TOC misaligned when max width option is disable (T322162) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 21:26 samtar@deploy1002: Started scap: Backport for Fix TOC misaligned when max width option is disable (T322162)
  • 21:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2110 (T318605)', diff saved to https://phabricator.wikimedia.org/P38852 and previous config saved to /var/cache/conftool/dbconfig/20221109-211613-ladsgroup.json
  • 21:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 21:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 21:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T318605)', diff saved to https://phabricator.wikimedia.org/P38851 and previous config saved to /var/cache/conftool/dbconfig/20221109-211551-ladsgroup.json
  • 21:15 samtar@deploy1002: Finished scap: Backport for Only Enable LBFactory config callback in CLI in production (T298485) (duration: 05m 41s)
  • 21:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:10 samtar@deploy1002: samtar and dancy: Backport for Only Enable LBFactory config callback in CLI in production (T298485) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 21:09 samtar@deploy1002: Started scap: Backport for Only Enable LBFactory config callback in CLI in production (T298485)
  • 21:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:09 root@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 21:08 samtar@deploy1002: Finished scap: Backport for Add no=>nb to wgInterlanguageLinkCodeMap for some multilingual wikis (T322696) (duration: 06m 06s)
  • 21:02 samtar@deploy1002: samtar and jhsoby: Backport for Add no=>nb to wgInterlanguageLinkCodeMap for some multilingual wikis (T322696) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 21:02 samtar@deploy1002: Started scap: Backport for Add no=>nb to wgInterlanguageLinkCodeMap for some multilingual wikis (T322696)
  • 21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P38850 and previous config saved to /var/cache/conftool/dbconfig/20221109-210044-ladsgroup.json
  • 20:45 root@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1023.eqiad.wmnet with reason: host reimage
  • 20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P38849 and previous config saved to /var/cache/conftool/dbconfig/20221109-204538-ladsgroup.json
  • 20:42 root@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1023.eqiad.wmnet with reason: host reimage
  • 20:41 robh@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:35 mutante: gerrit1001 (gerrit) - restarting gerrit service to disable aggressive garbage collection. gerrit:854514 - T237807
  • 20:30 mutante: gerrit2002 (gerrit-replica) - restarting gerrit service
  • 20:30 root@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 20:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T318605)', diff saved to https://phabricator.wikimedia.org/P38848 and previous config saved to /var/cache/conftool/dbconfig/20221109-203031-ladsgroup.json
  • 20:26 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:26 robh@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1034.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:23 root@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 20:12 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1034.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:05 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:04 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:01 root@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 20:01 root@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 20:01 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1034.mgmt.eqiad.wmnet with reboot policy FORCED
  • 19:59 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1034.mgmt.eqiad.wmnet with reboot policy FORCED
  • 19:52 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 19:50 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 19:47 robh@cumin1001: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1034
  • 19:47 robh@cumin1001: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1034
  • 19:47 robh@cumin1001: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1033
  • 19:47 robh@cumin1001: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1033
  • 19:37 root@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:35 root@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:35 root@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1013.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 19:13 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1013.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 19:07 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:05 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:01 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:00 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:49 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:49 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:41 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:40 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:35 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:33 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:20 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:19 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 17:57 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wcqs1003.eqiad.wmnet with reason: data reload
  • 17:55 bking@cumin2002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on wcqs1003.eqiad.wmnet with reason: data reload
  • 17:45 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 17:40 volans@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Converted existing STAGED hosts to ACTIVE - volans@cumin1001 - T320696"
  • 17:37 volans@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Converted existing STAGED hosts to ACTIVE - volans@cumin1001 - T320696"
  • 17:29 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 17:28 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 17:17 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 17:16 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 17:09 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 17:09 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:55 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 8218
  • 16:52 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 8218
  • 16:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T322618)', diff saved to https://phabricator.wikimedia.org/P38843 and previous config saved to /var/cache/conftool/dbconfig/20221109-162849-ladsgroup.json
  • 16:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38842 and previous config saved to /var/cache/conftool/dbconfig/20221109-161343-ladsgroup.json
  • 15:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38841 and previous config saved to /var/cache/conftool/dbconfig/20221109-155836-ladsgroup.json
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T318605)', diff saved to https://phabricator.wikimedia.org/P38840 and previous config saved to /var/cache/conftool/dbconfig/20221109-154933-ladsgroup.json
  • 15:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 15:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 15:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38839 and previous config saved to /var/cache/conftool/dbconfig/20221109-154911-ladsgroup.json
  • 15:47 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet
  • 15:47 aikochou@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T322618)', diff saved to https://phabricator.wikimedia.org/P38838 and previous config saved to /var/cache/conftool/dbconfig/20221109-154330-ladsgroup.json
  • 15:42 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet
  • 15:42 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1099.eqiad.wmnet
  • 15:42 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-stretch2002.codfw.wmnet
  • 15:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T322618)', diff saved to https://phabricator.wikimedia.org/P38837 and previous config saved to /var/cache/conftool/dbconfig/20221109-153922-ladsgroup.json
  • 15:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 15:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 15:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T322618)', diff saved to https://phabricator.wikimedia.org/P38836 and previous config saved to /var/cache/conftool/dbconfig/20221109-153901-ladsgroup.json
  • 15:34 aikochou@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38835 and previous config saved to /var/cache/conftool/dbconfig/20221109-153405-ladsgroup.json
  • 15:33 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host kafka-stretch2002.codfw.wmnet
  • 15:32 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1099.eqiad.wmnet
  • 15:31 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1100.eqiad.wmnet
  • 15:24 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1100.eqiad.wmnet
  • 15:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38834 and previous config saved to /var/cache/conftool/dbconfig/20221109-152354-ladsgroup.json
  • 15:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38833 and previous config saved to /var/cache/conftool/dbconfig/20221109-151858-ladsgroup.json
  • 15:18 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-stretch2001.codfw.wmnet
  • 15:12 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1101.eqiad.wmnet
  • 15:11 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host kafka-stretch2001.codfw.wmnet
  • 15:10 stevemunene@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host kafka-stretch2001.codfw.wmnet
  • 15:10 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host kafka-stretch2001.codfw.wmnet
  • 15:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38832 and previous config saved to /var/cache/conftool/dbconfig/20221109-150848-ladsgroup.json
  • 15:04 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet
  • 15:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38831 and previous config saved to /var/cache/conftool/dbconfig/20221109-150351-ladsgroup.json
  • 15:03 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1101.eqiad.wmnet
  • 15:02 stevemunene@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host an-worker1101.eqiad.wmnet
  • 15:02 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1101.eqiad.wmnet
  • 15:02 moritzm: installing pixman security updates on buster
  • 14:57 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T322618)', diff saved to https://phabricator.wikimedia.org/P38830 and previous config saved to /var/cache/conftool/dbconfig/20221109-145341-ladsgroup.json
  • 14:50 moritzm: rolling restart of mw canaries to pick up libxml security update
  • 14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T322618)', diff saved to https://phabricator.wikimedia.org/P38829 and previous config saved to /var/cache/conftool/dbconfig/20221109-144933-ladsgroup.json
  • 14:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 14:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T322618)', diff saved to https://phabricator.wikimedia.org/P38828 and previous config saved to /var/cache/conftool/dbconfig/20221109-144912-ladsgroup.json
  • 14:46 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=frwiki` in a tmux at mwmaint1002 (locally applied shorter MentorStore::INACTIVITY_THRESHOLD; T318457)
  • 14:43 sukhe: reprepro remove bullseye-wikimedia trafficserver: T321309
  • 14:40 moritzm: installing libxml2 security updates
  • 14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38827 and previous config saved to /var/cache/conftool/dbconfig/20221109-143404-ladsgroup.json
  • 14:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2106 (T318605)', diff saved to https://phabricator.wikimedia.org/P38826 and previous config saved to /var/cache/conftool/dbconfig/20221109-143050-ladsgroup.json
  • 14:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 14:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 14:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38825 and previous config saved to /var/cache/conftool/dbconfig/20221109-141858-ladsgroup.json
  • 14:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:12 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:12 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:11 Lucas_WMDE: UTC afternoon backport+config window done
  • 14:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:09 lucaswerkmeister-wmde@deploy1002: Finished scap: Backport for Enable show nearby feature for ruwiki (T321548) (duration: 05m 42s)
  • 14:04 lucaswerkmeister-wmde@deploy1002: lucaswerkmeister-wmde and lilients: Backport for Enable show nearby feature for ruwiki (T321548) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 14:04 lucaswerkmeister-wmde@deploy1002: Started scap: Backport for Enable show nearby feature for ruwiki (T321548)
  • 14:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T322618)', diff saved to https://phabricator.wikimedia.org/P38824 and previous config saved to /var/cache/conftool/dbconfig/20221109-140351-ladsgroup.json
  • 13:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T322618)', diff saved to https://phabricator.wikimedia.org/P38823 and previous config saved to /var/cache/conftool/dbconfig/20221109-135943-ladsgroup.json
  • 13:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 13:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 13:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T322618)', diff saved to https://phabricator.wikimedia.org/P38822 and previous config saved to /var/cache/conftool/dbconfig/20221109-135911-ladsgroup.json
  • 13:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38821 and previous config saved to /var/cache/conftool/dbconfig/20221109-134404-ladsgroup.json
  • 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38820 and previous config saved to /var/cache/conftool/dbconfig/20221109-132858-ladsgroup.json
  • 13:24 moritzm: drain ganeti1013 for eventual reimage to bullseye T311687
  • 13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T322618)', diff saved to https://phabricator.wikimedia.org/P38818 and previous config saved to /var/cache/conftool/dbconfig/20221109-131903-ladsgroup.json
  • 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf1004.eqiad.wmnet
  • 13:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T322618)', diff saved to https://phabricator.wikimedia.org/P38817 and previous config saved to /var/cache/conftool/dbconfig/20221109-131351-ladsgroup.json
  • 13:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf1004.eqiad.wmnet
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T322618)', diff saved to https://phabricator.wikimedia.org/P38816 and previous config saved to /var/cache/conftool/dbconfig/20221109-130944-ladsgroup.json
  • 13:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 13:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38815 and previous config saved to /var/cache/conftool/dbconfig/20221109-130923-ladsgroup.json
  • 13:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38814 and previous config saved to /var/cache/conftool/dbconfig/20221109-130357-ladsgroup.json
  • 12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38813 and previous config saved to /var/cache/conftool/dbconfig/20221109-125416-ladsgroup.json
  • 12:54 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 63199
  • 12:51 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply
  • 12:50 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 63199
  • 12:50 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-main: apply
  • 12:49 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 29169
  • 12:48 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 29169
  • 12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38812 and previous config saved to /var/cache/conftool/dbconfig/20221109-124850-ladsgroup.json
  • 12:43 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply
  • 12:42 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-main: apply
  • 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38811 and previous config saved to /var/cache/conftool/dbconfig/20221109-123910-ladsgroup.json
  • 12:33 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
  • 12:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T322618)', diff saved to https://phabricator.wikimedia.org/P38810 and previous config saved to /var/cache/conftool/dbconfig/20221109-123344-ladsgroup.json
  • 12:33 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
  • 12:31 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:30 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:28 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:28 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:26 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
  • 12:26 hashar@deploy1002: Finished deploy [gerrit/gerrit@b83625a]: gerrit1001: Gerrit JavaScript plugins as standalone files # T319378 (duration: 00m 09s)
  • 12:26 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
  • 12:25 hashar@deploy1002: Started deploy [gerrit/gerrit@b83625a]: gerrit1001: Gerrit JavaScript plugins as standalone files # T319378
  • 12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T322618)', diff saved to https://phabricator.wikimedia.org/P38809 and previous config saved to /var/cache/conftool/dbconfig/20221109-122528-ladsgroup.json
  • 12:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 12:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T322618)', diff saved to https://phabricator.wikimedia.org/P38808 and previous config saved to /var/cache/conftool/dbconfig/20221109-122507-ladsgroup.json
  • 12:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38807 and previous config saved to /var/cache/conftool/dbconfig/20221109-122403-ladsgroup.json
  • 12:18 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply
  • 12:17 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply
  • 12:16 hashar@deploy1002: Finished deploy [gerrit/gerrit@b83625a]: gerrit2002: Gerrit JavaScript plugins as standalone files # T319378 (duration: 00m 10s)
  • 12:16 hashar@deploy1002: Started deploy [gerrit/gerrit@b83625a]: gerrit2002: Gerrit JavaScript plugins as standalone files # T319378
  • 12:11 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-logging-external: apply
  • 12:10 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-logging-external: apply
  • 12:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38806 and previous config saved to /var/cache/conftool/dbconfig/20221109-121001-ladsgroup.json
  • 12:03 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply
  • 12:03 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply
  • 11:56 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply
  • 11:56 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply
  • 11:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38805 and previous config saved to /var/cache/conftool/dbconfig/20221109-115454-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T322618)', diff saved to https://phabricator.wikimedia.org/P38804 and previous config saved to /var/cache/conftool/dbconfig/20221109-113948-ladsgroup.json
  • 11:38 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host datahubsearch1003.eqiad.wmnet
  • 11:36 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 11:34 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host datahubsearch1003.eqiad.wmnet
  • 11:33 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-eqiad
  • 11:32 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd1001.eqiad.wmnet
  • 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T322618)', diff saved to https://phabricator.wikimedia.org/P38803 and previous config saved to /var/cache/conftool/dbconfig/20221109-113144-ladsgroup.json
  • 11:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 11:31 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
  • 11:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 11:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 11:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P38802 and previous config saved to /var/cache/conftool/dbconfig/20221109-113108-ladsgroup.json
  • 11:28 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1001.eqiad.wmnet
  • 11:28 stevemunene@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host dse-k8s-etcd1001.eqiad.wmnet
  • 11:28 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1001.eqiad.wmnet
  • 11:25 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host datahubsearch1002.eqiad.wmnet
  • 11:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38801 and previous config saved to /var/cache/conftool/dbconfig/20221109-112347-ladsgroup.json
  • 11:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38800 and previous config saved to /var/cache/conftool/dbconfig/20221109-112326-ladsgroup.json
  • 11:21 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host datahubsearch1002.eqiad.wmnet
  • 11:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38799 and previous config saved to /var/cache/conftool/dbconfig/20221109-111601-ladsgroup.json
  • 11:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38798 and previous config saved to /var/cache/conftool/dbconfig/20221109-110819-ladsgroup.json
  • 11:05 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-codfw
  • 11:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38797 and previous config saved to /var/cache/conftool/dbconfig/20221109-110055-ladsgroup.json
  • 10:59 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1024.eqiad.wmnet to cluster eqiad and group C
  • 10:58 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1024.eqiad.wmnet to cluster eqiad and group C
  • 10:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38796 and previous config saved to /var/cache/conftool/dbconfig/20221109-105313-ladsgroup.json
  • 10:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P38794 and previous config saved to /var/cache/conftool/dbconfig/20221109-104548-ladsgroup.json
  • 10:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38793 and previous config saved to /var/cache/conftool/dbconfig/20221109-103806-ladsgroup.json
  • 10:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P38792 and previous config saved to /var/cache/conftool/dbconfig/20221109-103722-ladsgroup.json
  • 10:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 10:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 10:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 10:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 10:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T322618)', diff saved to https://phabricator.wikimedia.org/P38791 and previous config saved to /var/cache/conftool/dbconfig/20221109-103026-ladsgroup.json
  • 10:22 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host datahubsearch1001.eqiad.wmnet
  • 10:17 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1018.eqiad.wmnet to cluster eqiad and group B
  • 10:17 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host datahubsearch1001.eqiad.wmnet
  • 10:16 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1018.eqiad.wmnet to cluster eqiad and group B
  • 10:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38788 and previous config saved to /var/cache/conftool/dbconfig/20221109-101519-ladsgroup.json
  • 10:02 volans: set Netbox status to Active for 299 devices with role=server, tenant=none, status=staged - T320696
  • 10:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38787 and previous config saved to /var/cache/conftool/dbconfig/20221109-100013-ladsgroup.json
  • 09:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1024.eqiad.wmnet
  • 09:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1024.eqiad.wmnet
  • 09:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T322618)', diff saved to https://phabricator.wikimedia.org/P38786 and previous config saved to /var/cache/conftool/dbconfig/20221109-094506-ladsgroup.json
  • 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1018.eqiad.wmnet
  • 09:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38785 and previous config saved to /var/cache/conftool/dbconfig/20221109-093751-ladsgroup.json
  • 09:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 09:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T322618)', diff saved to https://phabricator.wikimedia.org/P38784 and previous config saved to /var/cache/conftool/dbconfig/20221109-093650-ladsgroup.json
  • 09:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T322618)', diff saved to https://phabricator.wikimedia.org/P38783 and previous config saved to /var/cache/conftool/dbconfig/20221109-093629-ladsgroup.json
  • 09:35 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1018.eqiad.wmnet
  • 09:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 09:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 09:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T322618)', diff saved to https://phabricator.wikimedia.org/P38782 and previous config saved to /var/cache/conftool/dbconfig/20221109-093454-ladsgroup.json
  • 09:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38781 and previous config saved to /var/cache/conftool/dbconfig/20221109-092122-ladsgroup.json
  • 09:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38780 and previous config saved to /var/cache/conftool/dbconfig/20221109-091947-ladsgroup.json
  • 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1024.eqiad.wmnet with OS bullseye
  • 09:07 moritzm: installing nodejs security updates
  • 09:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38779 and previous config saved to /var/cache/conftool/dbconfig/20221109-090616-ladsgroup.json
  • 09:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38778 and previous config saved to /var/cache/conftool/dbconfig/20221109-090441-ladsgroup.json
  • 09:03 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 54994
  • 08:59 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 54994
  • 08:57 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1024.eqiad.wmnet with reason: host reimage
  • 08:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38777 and previous config saved to /var/cache/conftool/dbconfig/20221109-085542-ladsgroup.json
  • 08:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 08:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 08:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 08:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 08:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 08:54 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1024.eqiad.wmnet with reason: host reimage
  • 08:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 08:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T322618)', diff saved to https://phabricator.wikimedia.org/P38776 and previous config saved to /var/cache/conftool/dbconfig/20221109-085109-ladsgroup.json
  • 08:51 kartik@deploy1002: Finished scap: Backport for Add channel for MessageBundle feature of Translate extension (T322430) (duration: 11m 19s)
  • 08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T322618)', diff saved to https://phabricator.wikimedia.org/P38775 and previous config saved to /var/cache/conftool/dbconfig/20221109-084934-ladsgroup.json
  • 08:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T322618)', diff saved to https://phabricator.wikimedia.org/P38774 and previous config saved to /var/cache/conftool/dbconfig/20221109-084525-ladsgroup.json
  • 08:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 08:45 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 08:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 08:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 08:44 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:44 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:43 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T322618)', diff saved to https://phabricator.wikimedia.org/P38773 and previous config saved to /var/cache/conftool/dbconfig/20221109-084254-ladsgroup.json
  • 08:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 08:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 08:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1024.eqiad.wmnet with OS bullseye
  • 08:40 kartik@deploy1002: kartik and abi: Backport for Add channel for MessageBundle feature of Translate extension (T322430) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
  • 08:39 kartik@deploy1002: Started scap: Backport for Add channel for MessageBundle feature of Translate extension (T322430)
  • 08:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1018.eqiad.wmnet with OS bullseye
  • 08:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:33 kartik@deploy1002: Finished scap: Backport for Update Metrics Platform streams (T322277) (duration: 08m 17s)
  • 08:32 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:32 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:31 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:30 ayounsi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1182.eqiad.wmnet with reason: paged then depooled
  • 08:30 ayounsi@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1182.eqiad.wmnet with reason: paged then depooled
  • 08:25 kartik@deploy1002: kartik and phuedx: Backport for Update Metrics Platform streams (T322277) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 08:24 kartik@deploy1002: Started scap: Backport for Update Metrics Platform streams (T322277)
  • 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1018.eqiad.wmnet with reason: host reimage
  • 08:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:21 kartik@deploy1002: Finished scap: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016) (duration: 08m 10s)
  • 08:20 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1018.eqiad.wmnet with reason: host reimage
  • 08:20 ayounsi@cumin1001: dbctl commit (dc=all): 'Depool db1182', diff saved to https://phabricator.wikimedia.org/P38772 and previous config saved to /var/cache/conftool/dbconfig/20221109-082045-ayounsi.json
  • 08:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:19 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:13 kartik@deploy1002: kartik and phuedx: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 08:12 kartik@deploy1002: Started scap: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016)
  • 08:06 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1018.eqiad.wmnet with OS bullseye
  • 07:24 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 6461
  • 07:22 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 6461
  • 07:22 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 8218
  • 07:22 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 8218
  • 07:21 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
  • 07:21 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 37271
  • 07:20 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37662
  • 07:20 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 37662
  • 07:20 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 29608
  • 07:19 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 29608
  • 07:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 8309
  • 07:18 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 8309
  • 07:17 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 54994
  • 07:16 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 54994
  • 07:16 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37693
  • 07:15 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 37693
  • 07:14 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 23889
  • 07:14 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 23889
  • 07:11 ayounsi@cumin1001: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 23889
  • 07:11 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 23889
  • 07:11 ayounsi@cumin1001: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 23889
  • 07:11 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 23889
  • 07:10 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 3225
  • 07:10 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 3225
  • 07:10 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 15412
  • 07:09 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 15412
  • 07:09 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 29169
  • 07:08 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 29169
  • 07:08 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37613
  • 07:08 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 37613
  • 07:08 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30990
  • 07:07 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 30990
  • 07:06 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 61955
  • 07:06 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 61955
  • 07:05 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 8220
  • 07:04 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 8220
  • 07:04 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 6774
  • 07:02 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 6774
  • 06:42 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 11404
  • 06:41 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 11404

2022-11-08

  • 22:00 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:59 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:59 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:59 urbanecm: UTC late evening B&C window done
  • 21:58 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:58 urbanecm@deploy1002: Finished scap: Backport for Revert "Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis" (duration: 05m 04s)
  • 21:53 urbanecm@deploy1002: urbanecm and urbanecm: Backport for Revert "Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis" synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 21:53 urbanecm@deploy1002: Started scap: Backport for Revert "Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis"
  • 21:43 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:42 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:42 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:41 urbanecm@deploy1002: Finished scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353) (duration: 06m 36s)
  • 21:41 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:35 urbanecm@deploy1002: urbanecm and matmarex: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 21:35 urbanecm@deploy1002: Started scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353)
  • 21:32 urbanecm@deploy1002: Finished scap: Backport for Bump sampling rate to 0.2 for various editing schemas on a/b test wikis (T321734), ThreadItemStore: Fix setting parent IDs when parent already existed (T322599) (duration: 05m 45s)
  • 21:31 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:30 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:30 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:26 urbanecm@deploy1002: urbanecm and kemayo and matmarex: Backport for Bump sampling rate to 0.2 for various editing schemas on a/b test wikis (T321734), ThreadItemStore: Fix setting parent IDs when parent already existed (T322599) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 21:26 urbanecm@deploy1002: Started scap: Backport for Bump sampling rate to 0.2 for various editing schemas on a/b test wikis (T321734), ThreadItemStore: Fix setting parent IDs when parent already existed (T322599)
  • 21:25 urbanecm@deploy1002: backport aborted: (duration: 01m 16s)
  • 21:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:23 urbanecm@deploy1002: Finished scap: Backport for Keep DiscussionTools "Share feedback..." links on WMF wikis for now (T322494) (duration: 04m 14s)
  • 21:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:19 urbanecm@deploy1002: Started scap: Backport for Keep DiscussionTools "Share feedback..." links on WMF wikis for now (T322494)
  • 21:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:17 urbanecm@deploy1002: Finished scap: Backport for ABtest for mobile, logged in users (T320993), ABtest for mobile, logged out users (T320993) (duration: 04m 10s)
  • 21:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:15 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:13 urbanecm@deploy1002: Started scap: Backport for ABtest for mobile, logged in users (T320993), ABtest for mobile, logged out users (T320993)
  • 21:13 urbanecm@deploy1002: backport aborted: (duration: 00m 01s)
  • 21:13 urbanecm@deploy1002: Finished scap: Backport for Enable history page visual diffs on beta cluster (T314588), Update wgSpecialContributeSkinsDisabled → wgSpecialContributeSkinsEnabled (T319327) (duration: 04m 33s)
  • 21:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:08 urbanecm@deploy1002: Started scap: Backport for Enable history page visual diffs on beta cluster (T314588), Update wgSpecialContributeSkinsDisabled → wgSpecialContributeSkinsEnabled (T319327)
  • 20:58 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:58 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:57 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:57 reedy@deploy1002: Synchronized wmf-config/LabsServices.php: T322667 (duration: 04m 02s)
  • 20:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T321130)', diff saved to https://phabricator.wikimedia.org/P38770 and previous config saved to /var/cache/conftool/dbconfig/20221108-203111-marostegui.json
  • 20:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P38769 and previous config saved to /var/cache/conftool/dbconfig/20221108-201604-marostegui.json
  • 20:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P38768 and previous config saved to /var/cache/conftool/dbconfig/20221108-200058-marostegui.json
  • 19:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T321130)', diff saved to https://phabricator.wikimedia.org/P38767 and previous config saved to /var/cache/conftool/dbconfig/20221108-194551-marostegui.json
  • 19:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 19:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 19:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T322618)', diff saved to https://phabricator.wikimedia.org/P38766 and previous config saved to /var/cache/conftool/dbconfig/20221108-194206-ladsgroup.json
  • 19:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2175 (T321130)', diff saved to https://phabricator.wikimedia.org/P38765 and previous config saved to /var/cache/conftool/dbconfig/20221108-193907-marostegui.json
  • 19:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 19:38 cstone: civicrm upgraded from b95f46bb to f2017495
  • 19:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 19:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38764 and previous config saved to /var/cache/conftool/dbconfig/20221108-193845-marostegui.json
  • 19:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P38763 and previous config saved to /var/cache/conftool/dbconfig/20221108-192659-ladsgroup.json
  • 19:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P38762 and previous config saved to /var/cache/conftool/dbconfig/20221108-192339-marostegui.json
  • 19:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P38761 and previous config saved to /var/cache/conftool/dbconfig/20221108-191152-ladsgroup.json
  • 19:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P38760 and previous config saved to /var/cache/conftool/dbconfig/20221108-190832-marostegui.json
  • 19:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T321123)', diff saved to https://phabricator.wikimedia.org/P38759 and previous config saved to /var/cache/conftool/dbconfig/20221108-190827-marostegui.json
  • 18:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1024.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 18:58 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1024.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 18:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T322618)', diff saved to https://phabricator.wikimedia.org/P38758 and previous config saved to /var/cache/conftool/dbconfig/20221108-185646-ladsgroup.json
  • 18:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1201 (T322618)', diff saved to https://phabricator.wikimedia.org/P38757 and previous config saved to /var/cache/conftool/dbconfig/20221108-185437-ladsgroup.json
  • 18:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 18:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 18:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T322618)', diff saved to https://phabricator.wikimedia.org/P38756 and previous config saved to /var/cache/conftool/dbconfig/20221108-185416-ladsgroup.json
  • 18:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38755 and previous config saved to /var/cache/conftool/dbconfig/20221108-185326-marostegui.json
  • 18:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38754 and previous config saved to /var/cache/conftool/dbconfig/20221108-185320-marostegui.json
  • 18:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38753 and previous config saved to /var/cache/conftool/dbconfig/20221108-184642-marostegui.json
  • 18:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 18:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 18:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T321130)', diff saved to https://phabricator.wikimedia.org/P38752 and previous config saved to /var/cache/conftool/dbconfig/20221108-184620-marostegui.json
  • 18:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P38751 and previous config saved to /var/cache/conftool/dbconfig/20221108-183909-ladsgroup.json
  • 18:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38750 and previous config saved to /var/cache/conftool/dbconfig/20221108-183814-marostegui.json
  • 18:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P38749 and previous config saved to /var/cache/conftool/dbconfig/20221108-183114-marostegui.json
  • 18:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P38748 and previous config saved to /var/cache/conftool/dbconfig/20221108-182403-ladsgroup.json
  • 18:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T321123)', diff saved to https://phabricator.wikimedia.org/P38747 and previous config saved to /var/cache/conftool/dbconfig/20221108-182307-marostegui.json
  • 18:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P38746 and previous config saved to /var/cache/conftool/dbconfig/20221108-181607-marostegui.json
  • 18:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T322618)', diff saved to https://phabricator.wikimedia.org/P38745 and previous config saved to /var/cache/conftool/dbconfig/20221108-180856-ladsgroup.json
  • 18:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T321123)', diff saved to https://phabricator.wikimedia.org/P38744 and previous config saved to /var/cache/conftool/dbconfig/20221108-180808-marostegui.json
  • 18:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 18:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 18:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T321123)', diff saved to https://phabricator.wikimedia.org/P38743 and previous config saved to /var/cache/conftool/dbconfig/20221108-180747-marostegui.json
  • 18:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T322618)', diff saved to https://phabricator.wikimedia.org/P38742 and previous config saved to /var/cache/conftool/dbconfig/20221108-180648-ladsgroup.json
  • 18:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 18:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 18:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38741 and previous config saved to /var/cache/conftool/dbconfig/20221108-180626-ladsgroup.json
  • 17:58 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2148 (T321130)', diff saved to https://phabricator.wikimedia.org/P38739 and previous config saved to /var/cache/conftool/dbconfig/20221108-175425-marostegui.json
  • 17:58 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 17:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38738 and previous config saved to /var/cache/conftool/dbconfig/20221108-175404-marostegui.json
  • 17:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38737 and previous config saved to /var/cache/conftool/dbconfig/20221108-175240-marostegui.json
  • 17:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P38736 and previous config saved to /var/cache/conftool/dbconfig/20221108-175120-ladsgroup.json
  • 17:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1015.eqiad.wmnet
  • 17:40 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1015.eqiad.wmnet
  • 17:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1014.eqiad.wmnet
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P38735 and previous config saved to /var/cache/conftool/dbconfig/20221108-173857-marostegui.json
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38734 and previous config saved to /var/cache/conftool/dbconfig/20221108-173734-marostegui.json
  • 17:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P38733 and previous config saved to /var/cache/conftool/dbconfig/20221108-173613-ladsgroup.json
  • 17:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1014.eqiad.wmnet
  • 17:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1013.eqiad.wmnet
  • 17:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P38732 and previous config saved to /var/cache/conftool/dbconfig/20221108-172351-marostegui.json
  • 17:23 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1013.eqiad.wmnet
  • 17:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T321123)', diff saved to https://phabricator.wikimedia.org/P38731 and previous config saved to /var/cache/conftool/dbconfig/20221108-172227-marostegui.json
  • 17:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38730 and previous config saved to /var/cache/conftool/dbconfig/20221108-172107-ladsgroup.json
  • 17:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38729 and previous config saved to /var/cache/conftool/dbconfig/20221108-171857-ladsgroup.json
  • 17:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 17:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 17:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T322618)', diff saved to https://phabricator.wikimedia.org/P38728 and previous config saved to /var/cache/conftool/dbconfig/20221108-171835-ladsgroup.json
  • 17:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38727 and previous config saved to /var/cache/conftool/dbconfig/20221108-170844-marostegui.json
  • 17:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T321123)', diff saved to https://phabricator.wikimedia.org/P38726 and previous config saved to /var/cache/conftool/dbconfig/20221108-170752-marostegui.json
  • 17:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 17:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 17:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 17:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 17:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38725 and previous config saved to /var/cache/conftool/dbconfig/20221108-170715-marostegui.json
  • 17:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P38724 and previous config saved to /var/cache/conftool/dbconfig/20221108-170329-ladsgroup.json
  • 17:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38723 and previous config saved to /var/cache/conftool/dbconfig/20221108-170157-marostegui.json
  • 17:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T321130)', diff saved to https://phabricator.wikimedia.org/P38722 and previous config saved to /var/cache/conftool/dbconfig/20221108-170136-marostegui.json
  • 17:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet
  • 16:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet
  • 16:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2002.codfw.wmnet
  • 16:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38721 and previous config saved to /var/cache/conftool/dbconfig/20221108-165208-marostegui.json
  • 16:49 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2002.codfw.wmnet
  • 16:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P38720 and previous config saved to /var/cache/conftool/dbconfig/20221108-164822-ladsgroup.json
  • 16:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2002.codfw.wmnet
  • 16:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P38719 and previous config saved to /var/cache/conftool/dbconfig/20221108-164629-marostegui.json
  • 16:41 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow2002.codfw.wmnet
  • 16:39 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 16:38 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:37 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 16:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38718 and previous config saved to /var/cache/conftool/dbconfig/20221108-163702-marostegui.json
  • 16:36 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow5002.eqsin.wmnet
  • 16:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T322618)', diff saved to https://phabricator.wikimedia.org/P38717 and previous config saved to /var/cache/conftool/dbconfig/20221108-163316-ladsgroup.json
  • 16:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P38716 and previous config saved to /var/cache/conftool/dbconfig/20221108-163122-marostegui.json
  • 16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T322618)', diff saved to https://phabricator.wikimedia.org/P38715 and previous config saved to /var/cache/conftool/dbconfig/20221108-163107-ladsgroup.json
  • 16:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 16:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 16:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T322618)', diff saved to https://phabricator.wikimedia.org/P38714 and previous config saved to /var/cache/conftool/dbconfig/20221108-163045-ladsgroup.json
  • 16:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow5002.eqsin.wmnet
  • 16:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow6001.drmrs.wmnet
  • 16:24 moritzm: drain ganeti1024 for eventual reimage to bullseye T311687
  • 16:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1018.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 16:22 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1018.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 16:22 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow6001.drmrs.wmnet
  • 16:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38713 and previous config saved to /var/cache/conftool/dbconfig/20221108-162155-marostegui.json
  • 16:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T321130)', diff saved to https://phabricator.wikimedia.org/P38712 and previous config saved to /var/cache/conftool/dbconfig/20221108-161616-marostegui.json
  • 16:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P38711 and previous config saved to /var/cache/conftool/dbconfig/20221108-161538-ladsgroup.json
  • 16:13 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2126 (T321130)', diff saved to https://phabricator.wikimedia.org/P38710 and previous config saved to /var/cache/conftool/dbconfig/20221108-161338-marostegui.json
  • 16:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T321130)', diff saved to https://phabricator.wikimedia.org/P38709 and previous config saved to /var/cache/conftool/dbconfig/20221108-161312-marostegui.json
  • 16:10 Emperor: upload wmf-beamer-style version 0.4 to apt
  • 16:06 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38708 and previous config saved to /var/cache/conftool/dbconfig/20221108-160632-marostegui.json
  • 16:06 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:06 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P38707 and previous config saved to /var/cache/conftool/dbconfig/20221108-160032-ladsgroup.json
  • 15:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P38706 and previous config saved to /var/cache/conftool/dbconfig/20221108-155805-marostegui.json
  • 15:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 15:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 15:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T321123)', diff saved to https://phabricator.wikimedia.org/P38705 and previous config saved to /var/cache/conftool/dbconfig/20221108-155229-marostegui.json
  • 15:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T322618)', diff saved to https://phabricator.wikimedia.org/P38704 and previous config saved to /var/cache/conftool/dbconfig/20221108-154525-ladsgroup.json
  • 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T322618)', diff saved to https://phabricator.wikimedia.org/P38703 and previous config saved to /var/cache/conftool/dbconfig/20221108-154317-ladsgroup.json
  • 15:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 15:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P38702 and previous config saved to /var/cache/conftool/dbconfig/20221108-154259-marostegui.json
  • 15:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T322618)', diff saved to https://phabricator.wikimedia.org/P38701 and previous config saved to /var/cache/conftool/dbconfig/20221108-154221-ladsgroup.json
  • 15:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38700 and previous config saved to /var/cache/conftool/dbconfig/20221108-153722-marostegui.json
  • 15:35 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:35 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:34 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:34 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T321130)', diff saved to https://phabricator.wikimedia.org/P38699 and previous config saved to /var/cache/conftool/dbconfig/20221108-152752-marostegui.json
  • 15:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P38698 and previous config saved to /var/cache/conftool/dbconfig/20221108-152715-ladsgroup.json
  • 15:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38697 and previous config saved to /var/cache/conftool/dbconfig/20221108-152548-ladsgroup.json
  • 15:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38696 and previous config saved to /var/cache/conftool/dbconfig/20221108-152216-marostegui.json
  • 15:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T321130)', diff saved to https://phabricator.wikimedia.org/P38695 and previous config saved to /var/cache/conftool/dbconfig/20221108-152037-marostegui.json
  • 15:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 15:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 15:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T321130)', diff saved to https://phabricator.wikimedia.org/P38694 and previous config saved to /var/cache/conftool/dbconfig/20221108-152016-marostegui.json
  • 15:18 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:18 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:17 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:16 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P38693 and previous config saved to /var/cache/conftool/dbconfig/20221108-151208-ladsgroup.json
  • 15:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P38692 and previous config saved to /var/cache/conftool/dbconfig/20221108-151041-ladsgroup.json
  • 15:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T321123)', diff saved to https://phabricator.wikimedia.org/P38691 and previous config saved to /var/cache/conftool/dbconfig/20221108-150709-marostegui.json
  • 15:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P38690 and previous config saved to /var/cache/conftool/dbconfig/20221108-150509-marostegui.json
  • 15:04 daniel@deploy1002: Finished scap: Backport for Stash original wikitext when rendering unsaved content. (T321862) (duration: 07m 25s)
  • 15:03 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:03 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:01 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 15:01 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:01 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:01 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 15:01 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 15:00 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:57 daniel@deploy1002: daniel and daniel: Backport for Stash original wikitext when rendering unsaved content. (T321862) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 14:57 daniel@deploy1002: Started scap: Backport for Stash original wikitext when rendering unsaved content. (T321862)
  • 14:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T322618)', diff saved to https://phabricator.wikimedia.org/P38689 and previous config saved to /var/cache/conftool/dbconfig/20221108-145702-ladsgroup.json
  • 14:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P38688 and previous config saved to /var/cache/conftool/dbconfig/20221108-145535-ladsgroup.json
  • 14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T322618)', diff saved to https://phabricator.wikimedia.org/P38687 and previous config saved to /var/cache/conftool/dbconfig/20221108-145453-ladsgroup.json
  • 14:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 14:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38686 and previous config saved to /var/cache/conftool/dbconfig/20221108-145432-ladsgroup.json
  • 14:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T321123)', diff saved to https://phabricator.wikimedia.org/P38685 and previous config saved to /var/cache/conftool/dbconfig/20221108-145210-marostegui.json
  • 14:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 14:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 14:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T321123)', diff saved to https://phabricator.wikimedia.org/P38684 and previous config saved to /var/cache/conftool/dbconfig/20221108-145148-marostegui.json
  • 14:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P38683 and previous config saved to /var/cache/conftool/dbconfig/20221108-145003-marostegui.json
  • 14:43 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 14:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38682 and previous config saved to /var/cache/conftool/dbconfig/20221108-144028-ladsgroup.json
  • 14:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P38681 and previous config saved to /var/cache/conftool/dbconfig/20221108-143925-ladsgroup.json
  • 14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38680 and previous config saved to /var/cache/conftool/dbconfig/20221108-143815-ladsgroup.json
  • 14:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 14:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 14:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38679 and previous config saved to /var/cache/conftool/dbconfig/20221108-143743-ladsgroup.json
  • 14:37 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 14:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38678 and previous config saved to /var/cache/conftool/dbconfig/20221108-143642-marostegui.json
  • 14:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T321130)', diff saved to https://phabricator.wikimedia.org/P38677 and previous config saved to /var/cache/conftool/dbconfig/20221108-143457-marostegui.json
  • 14:33 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 14:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2104 (T321130)', diff saved to https://phabricator.wikimedia.org/P38676 and previous config saved to /var/cache/conftool/dbconfig/20221108-143220-marostegui.json
  • 14:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 14:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 14:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 14:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 14:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P38675 and previous config saved to /var/cache/conftool/dbconfig/20221108-142419-ladsgroup.json
  • 14:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 14:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 14:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T321130)', diff saved to https://phabricator.wikimedia.org/P38674 and previous config saved to /var/cache/conftool/dbconfig/20221108-142247-marostegui.json
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P38673 and previous config saved to /var/cache/conftool/dbconfig/20221108-142236-ladsgroup.json
  • 14:22 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 14:22 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 14:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38672 and previous config saved to /var/cache/conftool/dbconfig/20221108-142135-marostegui.json
  • 14:21 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 14:21 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 14:11 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 14:11 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 14:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38671 and previous config saved to /var/cache/conftool/dbconfig/20221108-140912-ladsgroup.json
  • 14:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38670 and previous config saved to /var/cache/conftool/dbconfig/20221108-140803-ladsgroup.json
  • 14:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 14:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 14:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P38669 and previous config saved to /var/cache/conftool/dbconfig/20221108-140741-marostegui.json
  • 14:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P38668 and previous config saved to /var/cache/conftool/dbconfig/20221108-140730-ladsgroup.json
  • 14:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T321123)', diff saved to https://phabricator.wikimedia.org/P38667 and previous config saved to /var/cache/conftool/dbconfig/20221108-140628-marostegui.json
  • 13:56 dcausse@deploy1002: Finished deploy [wikimedia/discovery/analytics@248d897]: import_cirrus_indexes: increase driver mem (duration: 02m 23s)
  • 13:54 dcausse@deploy1002: Started deploy [wikimedia/discovery/analytics@248d897]: import_cirrus_indexes: increase driver mem
  • 13:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P38666 and previous config saved to /var/cache/conftool/dbconfig/20221108-135234-marostegui.json
  • 13:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38665 and previous config saved to /var/cache/conftool/dbconfig/20221108-135224-ladsgroup.json
  • 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T321123)', diff saved to https://phabricator.wikimedia.org/P38664 and previous config saved to /var/cache/conftool/dbconfig/20221108-135129-marostegui.json
  • 13:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 13:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38663 and previous config saved to /var/cache/conftool/dbconfig/20221108-135011-ladsgroup.json
  • 13:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 13:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 13:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38662 and previous config saved to /var/cache/conftool/dbconfig/20221108-134949-ladsgroup.json
  • 13:49 btullis@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 13:49 btullis@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 13:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 13:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 13:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T321123)', diff saved to https://phabricator.wikimedia.org/P38661 and previous config saved to /var/cache/conftool/dbconfig/20221108-134656-marostegui.json
  • 13:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P38660 and previous config saved to /var/cache/conftool/dbconfig/20221108-133730-ladsgroup.json
  • 13:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T321130)', diff saved to https://phabricator.wikimedia.org/P38659 and previous config saved to /var/cache/conftool/dbconfig/20221108-133721-marostegui.json
  • 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1197 (T321130)', diff saved to https://phabricator.wikimedia.org/P38658 and previous config saved to /var/cache/conftool/dbconfig/20221108-133505-marostegui.json
  • 13:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 13:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P38657 and previous config saved to /var/cache/conftool/dbconfig/20221108-133442-ladsgroup.json
  • 13:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 13:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T321130)', diff saved to https://phabricator.wikimedia.org/P38656 and previous config saved to /var/cache/conftool/dbconfig/20221108-133433-marostegui.json
  • 13:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38655 and previous config saved to /var/cache/conftool/dbconfig/20221108-133149-marostegui.json
  • 13:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow3002.esams.wmnet
  • 13:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38654 and previous config saved to /var/cache/conftool/dbconfig/20221108-132223-ladsgroup.json
  • 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38653 and previous config saved to /var/cache/conftool/dbconfig/20221108-132014-ladsgroup.json
  • 13:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 13:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38652 and previous config saved to /var/cache/conftool/dbconfig/20221108-131952-ladsgroup.json
  • 13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P38651 and previous config saved to /var/cache/conftool/dbconfig/20221108-131936-ladsgroup.json
  • 13:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P38650 and previous config saved to /var/cache/conftool/dbconfig/20221108-131927-marostegui.json
  • 13:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow3002.esams.wmnet
  • 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4002.ulsfo.wmnet
  • 13:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38649 and previous config saved to /var/cache/conftool/dbconfig/20221108-131643-marostegui.json
  • 13:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow4002.ulsfo.wmnet
  • 13:07 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 15557
  • 13:05 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 15557
  • 13:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38648 and previous config saved to /var/cache/conftool/dbconfig/20221108-130446-ladsgroup.json
  • 13:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38647 and previous config saved to /var/cache/conftool/dbconfig/20221108-130429-ladsgroup.json
  • 13:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P38646 and previous config saved to /var/cache/conftool/dbconfig/20221108-130420-marostegui.json
  • 13:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38645 and previous config saved to /var/cache/conftool/dbconfig/20221108-130216-ladsgroup.json
  • 13:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 13:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 13:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38644 and previous config saved to /var/cache/conftool/dbconfig/20221108-130205-ladsgroup.json
  • 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T321123)', diff saved to https://phabricator.wikimedia.org/P38643 and previous config saved to /var/cache/conftool/dbconfig/20221108-130136-marostegui.json
  • 12:55 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T321123)', diff saved to https://phabricator.wikimedia.org/P38642 and previous config saved to /var/cache/conftool/dbconfig/20221108-125529-marostegui.json
  • 12:55 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T321123)', diff saved to https://phabricator.wikimedia.org/P38641 and previous config saved to /var/cache/conftool/dbconfig/20221108-125508-marostegui.json
  • 12:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38640 and previous config saved to /var/cache/conftool/dbconfig/20221108-124939-ladsgroup.json
  • 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T321130)', diff saved to https://phabricator.wikimedia.org/P38639 and previous config saved to /var/cache/conftool/dbconfig/20221108-124914-marostegui.json
  • 12:47 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P38638 and previous config saved to /var/cache/conftool/dbconfig/20221108-124659-ladsgroup.json
  • 12:47 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1188 (T321130)', diff saved to https://phabricator.wikimedia.org/P38637 and previous config saved to /var/cache/conftool/dbconfig/20221108-124658-marostegui.json
  • 12:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 12:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 12:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T321130)', diff saved to https://phabricator.wikimedia.org/P38636 and previous config saved to /var/cache/conftool/dbconfig/20221108-124636-marostegui.json
  • 12:42 ladsgroup@deploy1002: Finished scap: Backport for Include core PSR-4 classes in the generated classmap (T274041) (duration: 05m 29s)
  • 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki2002.codfw.wmnet
  • 12:40 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:40 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38635 and previous config saved to /var/cache/conftool/dbconfig/20221108-124001-marostegui.json
  • 12:38 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:38 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 12:37 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for Include core PSR-4 classes in the generated classmap (T274041) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 12:37 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 12:37 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 12:37 ladsgroup@deploy1002: Started scap: Backport for Include core PSR-4 classes in the generated classmap (T274041)
  • 12:36 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 12:36 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host pki2002.codfw.wmnet
  • 12:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38634 and previous config saved to /var/cache/conftool/dbconfig/20221108-123433-ladsgroup.json
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P38633 and previous config saved to /var/cache/conftool/dbconfig/20221108-123152-ladsgroup.json
  • 12:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P38632 and previous config saved to /var/cache/conftool/dbconfig/20221108-123130-marostegui.json
  • 12:29 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp1002.wikimedia.org
  • 12:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38631 and previous config saved to /var/cache/conftool/dbconfig/20221108-122923-ladsgroup.json
  • 12:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 12:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 12:28 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:27 sukhe: reprepro -C main include bullseye-wikimedia libvmod-re2_1.5.3-3_amd64.changes: T321309
  • 12:25 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host idp1002.wikimedia.org
  • 12:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38630 and previous config saved to /var/cache/conftool/dbconfig/20221108-122455-marostegui.json
  • 12:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38629 and previous config saved to /var/cache/conftool/dbconfig/20221108-121646-ladsgroup.json
  • 12:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P38628 and previous config saved to /var/cache/conftool/dbconfig/20221108-121623-marostegui.json
  • 12:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38627 and previous config saved to /var/cache/conftool/dbconfig/20221108-121433-ladsgroup.json
  • 12:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 12:14 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 12:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 12:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 12:14 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 12:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 12:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T322618)', diff saved to https://phabricator.wikimedia.org/P38626 and previous config saved to /var/cache/conftool/dbconfig/20221108-121347-ladsgroup.json
  • 12:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T321123)', diff saved to https://phabricator.wikimedia.org/P38625 and previous config saved to /var/cache/conftool/dbconfig/20221108-120949-marostegui.json
  • 12:09 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:08 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:02 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T321123)', diff saved to https://phabricator.wikimedia.org/P38624 and previous config saved to /var/cache/conftool/dbconfig/20221108-120143-marostegui.json
  • 12:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 12:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38623 and previous config saved to /var/cache/conftool/dbconfig/20221108-120122-marostegui.json
  • 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T321130)', diff saved to https://phabricator.wikimedia.org/P38622 and previous config saved to /var/cache/conftool/dbconfig/20221108-120117-marostegui.json
  • 11:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P38621 and previous config saved to /var/cache/conftool/dbconfig/20221108-115840-ladsgroup.json
  • 11:54 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T321130)', diff saved to https://phabricator.wikimedia.org/P38620 and previous config saved to /var/cache/conftool/dbconfig/20221108-115452-marostegui.json
  • 11:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 11:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 11:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38619 and previous config saved to /var/cache/conftool/dbconfig/20221108-115430-marostegui.json
  • 11:52 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 11:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38618 and previous config saved to /var/cache/conftool/dbconfig/20221108-114615-marostegui.json
  • 11:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P38617 and previous config saved to /var/cache/conftool/dbconfig/20221108-114333-ladsgroup.json
  • 11:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P38616 and previous config saved to /var/cache/conftool/dbconfig/20221108-113924-marostegui.json
  • 11:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38615 and previous config saved to /var/cache/conftool/dbconfig/20221108-113109-marostegui.json
  • 11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T322618)', diff saved to https://phabricator.wikimedia.org/P38614 and previous config saved to /var/cache/conftool/dbconfig/20221108-112825-ladsgroup.json
  • 11:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2129 (T322618)', diff saved to https://phabricator.wikimedia.org/P38613 and previous config saved to /var/cache/conftool/dbconfig/20221108-112612-ladsgroup.json
  • 11:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 11:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 11:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T322618)', diff saved to https://phabricator.wikimedia.org/P38612 and previous config saved to /var/cache/conftool/dbconfig/20221108-112551-ladsgroup.json
  • 11:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P38611 and previous config saved to /var/cache/conftool/dbconfig/20221108-112417-marostegui.json
  • 11:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38610 and previous config saved to /var/cache/conftool/dbconfig/20221108-111602-marostegui.json
  • 11:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P38609 and previous config saved to /var/cache/conftool/dbconfig/20221108-111044-ladsgroup.json
  • 11:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38608 and previous config saved to /var/cache/conftool/dbconfig/20221108-110956-marostegui.json
  • 11:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 11:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 11:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T321123)', diff saved to https://phabricator.wikimedia.org/P38607 and previous config saved to /var/cache/conftool/dbconfig/20221108-110934-marostegui.json
  • 11:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38606 and previous config saved to /var/cache/conftool/dbconfig/20221108-110911-marostegui.json
  • 11:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38605 and previous config saved to /var/cache/conftool/dbconfig/20221108-110243-marostegui.json
  • 11:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 11:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 11:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T321130)', diff saved to https://phabricator.wikimedia.org/P38604 and previous config saved to /var/cache/conftool/dbconfig/20221108-110232-marostegui.json
  • 11:00 moritzm: drain ganeti1024 for eventual reimage to bullseye T311687
  • 11:00 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 10:58 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 10:57 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 10:55 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 10:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P38603 and previous config saved to /var/cache/conftool/dbconfig/20221108-105538-ladsgroup.json
  • 10:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38602 and previous config saved to /var/cache/conftool/dbconfig/20221108-105428-marostegui.json
  • 10:52 btullis: added stevemunene to wmf and ops LDAP groups T322339
  • 10:51 moritzm: installing batik security updates
  • 10:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P38601 and previous config saved to /var/cache/conftool/dbconfig/20221108-104726-marostegui.json
  • 10:42 moritzm: installing ntfs-3g security updates
  • 10:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T322618)', diff saved to https://phabricator.wikimedia.org/P38600 and previous config saved to /var/cache/conftool/dbconfig/20221108-104031-ladsgroup.json
  • 10:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38599 and previous config saved to /var/cache/conftool/dbconfig/20221108-103921-marostegui.json
  • 10:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T322618)', diff saved to https://phabricator.wikimedia.org/P38598 and previous config saved to /var/cache/conftool/dbconfig/20221108-103817-ladsgroup.json
  • 10:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 10:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 10:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P38597 and previous config saved to /var/cache/conftool/dbconfig/20221108-103756-ladsgroup.json
  • 10:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P38596 and previous config saved to /var/cache/conftool/dbconfig/20221108-103219-marostegui.json
  • 10:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T321123)', diff saved to https://phabricator.wikimedia.org/P38595 and previous config saved to /var/cache/conftool/dbconfig/20221108-102415-marostegui.json
  • 10:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P38594 and previous config saved to /var/cache/conftool/dbconfig/20221108-102249-ladsgroup.json
  • 10:18 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T321123)', diff saved to https://phabricator.wikimedia.org/P38593 and previous config saved to /var/cache/conftool/dbconfig/20221108-101806-marostegui.json
  • 10:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 10:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 10:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38592 and previous config saved to /var/cache/conftool/dbconfig/20221108-101745-marostegui.json
  • 10:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T321130)', diff saved to https://phabricator.wikimedia.org/P38591 and previous config saved to /var/cache/conftool/dbconfig/20221108-101713-marostegui.json
  • 10:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T321130)', diff saved to https://phabricator.wikimedia.org/P38590 and previous config saved to /var/cache/conftool/dbconfig/20221108-101457-marostegui.json
  • 10:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 10:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 10:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T321130)', diff saved to https://phabricator.wikimedia.org/P38589 and previous config saved to /var/cache/conftool/dbconfig/20221108-101435-marostegui.json
  • 10:10 moritzm: installing glibc security updates on buster
  • 10:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P38588 and previous config saved to /var/cache/conftool/dbconfig/20221108-100743-ladsgroup.json
  • 10:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38587 and previous config saved to /var/cache/conftool/dbconfig/20221108-100239-marostegui.json
  • 09:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P38586 and previous config saved to /var/cache/conftool/dbconfig/20221108-095928-marostegui.json
  • 09:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P38585 and previous config saved to /var/cache/conftool/dbconfig/20221108-095236-ladsgroup.json
  • 09:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P38584 and previous config saved to /var/cache/conftool/dbconfig/20221108-095026-ladsgroup.json
  • 09:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 09:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 09:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Partially done', diff saved to https://phabricator.wikimedia.org/P38583 and previous config saved to /var/cache/conftool/dbconfig/20221108-094950-ladsgroup.json
  • 09:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38582 and previous config saved to /var/cache/conftool/dbconfig/20221108-094732-marostegui.json
  • 09:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P38581 and previous config saved to /var/cache/conftool/dbconfig/20221108-094655-ladsgroup.json
  • 09:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 09:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 09:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P38580 and previous config saved to /var/cache/conftool/dbconfig/20221108-094422-marostegui.json
  • 09:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38579 and previous config saved to /var/cache/conftool/dbconfig/20221108-093226-marostegui.json
  • 09:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T321130)', diff saved to https://phabricator.wikimedia.org/P38578 and previous config saved to /var/cache/conftool/dbconfig/20221108-092915-marostegui.json
  • 09:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T321130)', diff saved to https://phabricator.wikimedia.org/P38577 and previous config saved to /var/cache/conftool/dbconfig/20221108-092256-marostegui.json
  • 09:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 09:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 09:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38576 and previous config saved to /var/cache/conftool/dbconfig/20221108-092229-marostegui.json
  • 09:22 moritzm: installing ffmpeg security updates on buster
  • 09:17 moritzm: drain ganeti1018 for eventual reimage to bullseye T311687
  • 09:09 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1004.eqiad.wmnet to plain
  • 09:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1004.eqiad.wmnet to plain
  • 09:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P38575 and previous config saved to /var/cache/conftool/dbconfig/20221108-090722-marostegui.json
  • 08:54 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1004.eqiad.wmnet to drbd
  • 08:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P38574 and previous config saved to /var/cache/conftool/dbconfig/20221108-085216-marostegui.json
  • 08:48 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:47 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:47 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:44 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1004.eqiad.wmnet to drbd
  • 08:44 kartik@deploy1002: Finished scap: Backport for Revert "EditAttemptStep sampling rate to 1 for group1 wikis" (duration: 04m 26s)
  • 08:40 kartik@deploy1002: kartik and trainbranchbot: Backport for Revert "EditAttemptStep sampling rate to 1 for group1 wikis" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 08:39 kartik@deploy1002: Started scap: Backport for Revert "EditAttemptStep sampling rate to 1 for group1 wikis"
  • 08:37 kartik@deploy1002: Sync cancelled.
  • 08:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38573 and previous config saved to /var/cache/conftool/dbconfig/20221108-083709-marostegui.json
  • 08:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:35 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:35 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:34 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1006.eqiad.wmnet to plain
  • 08:33 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1006.eqiad.wmnet to plain
  • 08:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38572 and previous config saved to /var/cache/conftool/dbconfig/20221108-083210-marostegui.json
  • 08:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 08:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 08:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321123)', diff saved to https://phabricator.wikimedia.org/P38571 and previous config saved to /var/cache/conftool/dbconfig/20221108-083148-marostegui.json
  • 08:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38570 and previous config saved to /var/cache/conftool/dbconfig/20221108-083037-marostegui.json
  • 08:30 kartik@deploy1002: kartik and phuedx: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 08:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 08:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 08:30 kartik@deploy1002: Started scap: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016)
  • 08:26 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 08:25 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 08:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38569 and previous config saved to /var/cache/conftool/dbconfig/20221108-082546-marostegui.json
  • 08:24 kartik@deploy1002: Finished scap: Backport for Enable Content and Section translation on 6 Wikipedias (T319175) (duration: 08m 20s)
  • 08:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1006.eqiad.wmnet to drbd
  • 08:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:16 kartik@deploy1002: kartik and kartik: Backport for Enable Content and Section translation on 6 Wikipedias (T319175) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 08:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38568 and previous config saved to /var/cache/conftool/dbconfig/20221108-081641-marostegui.json
  • 08:16 kartik@deploy1002: Started scap: Backport for Enable Content and Section translation on 6 Wikipedias (T319175)
  • 08:14 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1006.eqiad.wmnet to drbd
  • 08:12 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P38567 and previous config saved to /var/cache/conftool/dbconfig/20221108-081040-marostegui.json
  • 08:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:09 kartik@deploy1002: Finished scap: Backport for Enable Content and Section translation in Bambara and Goan Konkani Wikipedias (T314557) (duration: 06m 44s)
  • 08:03 kartik@deploy1002: kartik and kartik: Backport for Enable Content and Section translation in Bambara and Goan Konkani Wikipedias (T314557) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 08:03 kartik@deploy1002: Started scap: Backport for Enable Content and Section translation in Bambara and Goan Konkani Wikipedias (T314557)
  • 08:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38566 and previous config saved to /var/cache/conftool/dbconfig/20221108-080135-marostegui.json
  • 07:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P38565 and previous config saved to /var/cache/conftool/dbconfig/20221108-075533-marostegui.json
  • 07:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321123)', diff saved to https://phabricator.wikimedia.org/P38564 and previous config saved to /var/cache/conftool/dbconfig/20221108-074628-marostegui.json
  • 07:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38563 and previous config saved to /var/cache/conftool/dbconfig/20221108-074027-marostegui.json
  • 07:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T321123)', diff saved to https://phabricator.wikimedia.org/P38562 and previous config saved to /var/cache/conftool/dbconfig/20221108-074022-marostegui.json
  • 07:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 07:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 07:37 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38561 and previous config saved to /var/cache/conftool/dbconfig/20221108-073711-marostegui.json
  • 07:37 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 07:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 07:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38560 and previous config saved to /var/cache/conftool/dbconfig/20221108-073649-marostegui.json
  • 07:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 07:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 07:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321123)', diff saved to https://phabricator.wikimedia.org/P38559 and previous config saved to /var/cache/conftool/dbconfig/20221108-073549-marostegui.json
  • 07:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T318605)', diff saved to https://phabricator.wikimedia.org/P38558 and previous config saved to /var/cache/conftool/dbconfig/20221108-072648-ladsgroup.json
  • 07:22 XioNoX: push pfw policies - T322613
  • 07:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P38557 and previous config saved to /var/cache/conftool/dbconfig/20221108-072143-marostegui.json
  • 07:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38556 and previous config saved to /var/cache/conftool/dbconfig/20221108-072042-marostegui.json
  • 07:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P38555 and previous config saved to /var/cache/conftool/dbconfig/20221108-071142-ladsgroup.json
  • 07:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P38554 and previous config saved to /var/cache/conftool/dbconfig/20221108-070636-marostegui.json
  • 07:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38553 and previous config saved to /var/cache/conftool/dbconfig/20221108-070536-marostegui.json
  • 06:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P38552 and previous config saved to /var/cache/conftool/dbconfig/20221108-065635-ladsgroup.json
  • 06:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38551 and previous config saved to /var/cache/conftool/dbconfig/20221108-065130-marostegui.json
  • 06:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321123)', diff saved to https://phabricator.wikimedia.org/P38550 and previous config saved to /var/cache/conftool/dbconfig/20221108-065029-marostegui.json
  • 06:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38549 and previous config saved to /var/cache/conftool/dbconfig/20221108-064447-marostegui.json
  • 06:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 06:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 06:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T321123)', diff saved to https://phabricator.wikimedia.org/P38548 and previous config saved to /var/cache/conftool/dbconfig/20221108-064422-marostegui.json
  • 06:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 06:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 06:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 06:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 06:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T318605)', diff saved to https://phabricator.wikimedia.org/P38547 and previous config saved to /var/cache/conftool/dbconfig/20221108-064129-ladsgroup.json
  • 06:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 06:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 06:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 06:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 06:38 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 63199
  • 06:36 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 63199
  • 06:35 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 22381
  • 06:35 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 22381
  • 06:34 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 4817
  • 06:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 06:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 06:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 06:33 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 4817
  • 06:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1138.eqiad.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1138.eqiad.wmnet with reason: Maintenance
  • 06:32 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 13150
  • 06:31 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 13150
  • 06:30 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 30058
  • 06:29 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 30058
  • 06:27 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 46416
  • 06:27 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 06:27 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 46416
  • 05:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2182 (T318605)', diff saved to https://phabricator.wikimedia.org/P38546 and previous config saved to /var/cache/conftool/dbconfig/20221108-052025-ladsgroup.json
  • 05:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 05:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 05:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38545 and previous config saved to /var/cache/conftool/dbconfig/20221108-052004-ladsgroup.json
  • 05:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P38544 and previous config saved to /var/cache/conftool/dbconfig/20221108-050457-ladsgroup.json
  • 04:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P38543 and previous config saved to /var/cache/conftool/dbconfig/20221108-044951-ladsgroup.json
  • 04:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38542 and previous config saved to /var/cache/conftool/dbconfig/20221108-043444-ladsgroup.json
  • 03:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 03:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 03:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T318605)', diff saved to https://phabricator.wikimedia.org/P38541 and previous config saved to /var/cache/conftool/dbconfig/20221108-034607-ladsgroup.json
  • 03:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P38540 and previous config saved to /var/cache/conftool/dbconfig/20221108-033101-ladsgroup.json
  • 03:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20221108-031550-ladsgroup.json
  • 03:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38539 and previous config saved to /var/cache/conftool/dbconfig/20221108-031102-ladsgroup.json
  • 03:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 03:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 03:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38538 and previous config saved to /var/cache/conftool/dbconfig/20221108-031041-ladsgroup.json
  • 03:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 03:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 03:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 03:03 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 03:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T318605)', diff saved to https://phabricator.wikimedia.org/P38537 and previous config saved to /var/cache/conftool/dbconfig/20221108-030043-ladsgroup.json
  • 02:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P38536 and previous config saved to /var/cache/conftool/dbconfig/20221108-025533-ladsgroup.json
  • 02:47 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host contint1002.mgmt.eqiad.wmnet with reboot policy FORCED
  • 02:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T318605)', diff saved to https://phabricator.wikimedia.org/P38535 and previous config saved to /var/cache/conftool/dbconfig/20221108-024600-ladsgroup.json
  • 02:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 02:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 02:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T318605)', diff saved to https://phabricator.wikimedia.org/P38534 and previous config saved to /var/cache/conftool/dbconfig/20221108-024539-ladsgroup.json
  • 02:44 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host contint1002.mgmt.eqiad.wmnet with reboot policy FORCED
  • 02:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P38533 and previous config saved to /var/cache/conftool/dbconfig/20221108-024027-ladsgroup.json
  • 02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P38532 and previous config saved to /var/cache/conftool/dbconfig/20221108-023032-ladsgroup.json
  • 02:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38531 and previous config saved to /var/cache/conftool/dbconfig/20221108-022520-ladsgroup.json
  • 02:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P38530 and previous config saved to /var/cache/conftool/dbconfig/20221108-021525-ladsgroup.json
  • 02:14 eileen: civicrm upgraded from 72fccce1 to b95f46bb
  • 02:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T318605)', diff saved to https://phabricator.wikimedia.org/P38529 and previous config saved to /var/cache/conftool/dbconfig/20221108-020019-ladsgroup.json
  • 01:47 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 01:47 tgr@deploy1002: Finished scap: Backport for Add UserRegistrationLookupHelper (duration: 04m 36s)
  • 01:46 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 01:46 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 01:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 01:43 tgr@deploy1002: tgr and tgr: Backport for Add UserRegistrationLookupHelper synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 01:42 tgr@deploy1002: Started scap: Backport for Add UserRegistrationLookupHelper
  • 01:39 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 01:38 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 01:37 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
  • 01:25 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 01:22 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
  • 01:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2168:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38528 and previous config saved to /var/cache/conftool/dbconfig/20221108-010245-ladsgroup.json
  • 01:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 01:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 01:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T318605)', diff saved to https://phabricator.wikimedia.org/P38527 and previous config saved to /var/cache/conftool/dbconfig/20221108-010224-ladsgroup.json
  • 00:55 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1194 (T318605)', diff saved to https://phabricator.wikimedia.org/P38526 and previous config saved to /var/cache/conftool/dbconfig/20221108-005338-ladsgroup.json
  • 00:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 00:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 00:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T318605)', diff saved to https://phabricator.wikimedia.org/P38525 and previous config saved to /var/cache/conftool/dbconfig/20221108-005317-ladsgroup.json
  • 00:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P38524 and previous config saved to /var/cache/conftool/dbconfig/20221108-004717-ladsgroup.json
  • 00:47 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:40 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 00:39 tgr@deploy1002: Finished scap: Backport for createExtensionTables.php: Remove closeConnection() (duration: 04m 43s)
  • 00:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38523 and previous config saved to /var/cache/conftool/dbconfig/20221108-003934-marostegui.json
  • 00:39 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 00:39 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 00:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 00:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P38522 and previous config saved to /var/cache/conftool/dbconfig/20221108-003810-ladsgroup.json
  • 00:36 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 00:35 tgr@deploy1002: tgr and tgr: Backport for createExtensionTables.php: Remove closeConnection() synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 00:35 tgr@deploy1002: Started scap: Backport for createExtensionTables.php: Remove closeConnection()
  • 00:34 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
  • 00:33 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 00:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P38521 and previous config saved to /var/cache/conftool/dbconfig/20221108-003210-ladsgroup.json
  • 00:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P38520 and previous config saved to /var/cache/conftool/dbconfig/20221108-002428-marostegui.json
  • 00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P38519 and previous config saved to /var/cache/conftool/dbconfig/20221108-002304-ladsgroup.json
  • 00:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T318605)', diff saved to https://phabricator.wikimedia.org/P38518 and previous config saved to /var/cache/conftool/dbconfig/20221108-001704-ladsgroup.json
  • 00:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P38517 and previous config saved to /var/cache/conftool/dbconfig/20221108-000922-marostegui.json
  • 00:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T318605)', diff saved to https://phabricator.wikimedia.org/P38516 and previous config saved to /var/cache/conftool/dbconfig/20221108-000757-ladsgroup.json
  • 00:02 tgr: running foreachwikiindblist growthexperiments.dblist extensions/WikimediaMaintenance/createExtensionTables.php growthexperiments

2022-11-07

  • 23:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T318605)', diff saved to https://phabricator.wikimedia.org/P38515 and previous config saved to /var/cache/conftool/dbconfig/20221107-235526-ladsgroup.json
  • 23:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 23:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 23:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T318605)', diff saved to https://phabricator.wikimedia.org/P38514 and previous config saved to /var/cache/conftool/dbconfig/20221107-235505-ladsgroup.json
  • 23:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38513 and previous config saved to /var/cache/conftool/dbconfig/20221107-235415-marostegui.json
  • 23:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38512 and previous config saved to /var/cache/conftool/dbconfig/20221107-235206-marostegui.json
  • 23:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 23:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 23:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T321123)', diff saved to https://phabricator.wikimedia.org/P38511 and previous config saved to /var/cache/conftool/dbconfig/20221107-235144-marostegui.json
  • 23:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P38510 and previous config saved to /var/cache/conftool/dbconfig/20221107-233637-marostegui.json
  • 23:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P38509 and previous config saved to /var/cache/conftool/dbconfig/20221107-232447-ladsgroup.json
  • 23:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P38508 and previous config saved to /var/cache/conftool/dbconfig/20221107-232131-marostegui.json
  • 23:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T318605)', diff saved to https://phabricator.wikimedia.org/P38507 and previous config saved to /var/cache/conftool/dbconfig/20221107-230940-ladsgroup.json
  • 23:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T321123)', diff saved to https://phabricator.wikimedia.org/P38506 and previous config saved to /var/cache/conftool/dbconfig/20221107-230624-marostegui.json
  • 23:04 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2172 (T321123)', diff saved to https://phabricator.wikimedia.org/P38505 and previous config saved to /var/cache/conftool/dbconfig/20221107-230414-marostegui.json
  • 23:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 23:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 23:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321123)', diff saved to https://phabricator.wikimedia.org/P38504 and previous config saved to /var/cache/conftool/dbconfig/20221107-230353-marostegui.json
  • 22:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38503 and previous config saved to /var/cache/conftool/dbconfig/20221107-225943-marostegui.json
  • 22:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2159 (T318605)', diff saved to https://phabricator.wikimedia.org/P38502 and previous config saved to /var/cache/conftool/dbconfig/20221107-225602-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T318605)', diff saved to https://phabricator.wikimedia.org/P38501 and previous config saved to /var/cache/conftool/dbconfig/20221107-225536-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T318605)', diff saved to https://phabricator.wikimedia.org/P38500 and previous config saved to /var/cache/conftool/dbconfig/20221107-225525-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 22:53 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on phab2001.codfw.wmnet with reason: T322250
  • 22:53 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on phab2001.codfw.wmnet with reason: T322250
  • 22:51 mutante: phab2001 - removing from production puppet role - removes ssh access, ferm rules, exim config and more T322250
  • 22:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P38499 and previous config saved to /var/cache/conftool/dbconfig/20221107-224847-marostegui.json
  • 22:44 maryum: Deployed patches for T316414 and T315123
  • 22:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P38498 and previous config saved to /var/cache/conftool/dbconfig/20221107-224437-marostegui.json
  • 22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P38497 and previous config saved to /var/cache/conftool/dbconfig/20221107-224029-ladsgroup.json
  • 22:36 ejegg: fundraising CiviCRM upgraded from c0db8f34 to 72fccce1
  • 22:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P38496 and previous config saved to /var/cache/conftool/dbconfig/20221107-223340-marostegui.json
  • 22:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P38495 and previous config saved to /var/cache/conftool/dbconfig/20221107-222930-marostegui.json
  • 22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P38494 and previous config saved to /var/cache/conftool/dbconfig/20221107-222523-ladsgroup.json
  • 22:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321123)', diff saved to https://phabricator.wikimedia.org/P38493 and previous config saved to /var/cache/conftool/dbconfig/20221107-221834-marostegui.json
  • 22:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 22:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 22:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 22:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2155 (T321123)', diff saved to https://phabricator.wikimedia.org/P38492 and previous config saved to /var/cache/conftool/dbconfig/20221107-221624-marostegui.json
  • 22:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 22:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 22:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 22:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 22:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38491 and previous config saved to /var/cache/conftool/dbconfig/20221107-221557-marostegui.json
  • 22:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38490 and previous config saved to /var/cache/conftool/dbconfig/20221107-221423-marostegui.json
  • 22:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 22:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38489 and previous config saved to /var/cache/conftool/dbconfig/20221107-221209-marostegui.json
  • 22:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 22:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 22:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38488 and previous config saved to /var/cache/conftool/dbconfig/20221107-221148-marostegui.json
  • 22:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T318605)', diff saved to https://phabricator.wikimedia.org/P38487 and previous config saved to /var/cache/conftool/dbconfig/20221107-221016-ladsgroup.json
  • 22:07 mutante: [apt1001:~] $ sudo -E reprepro --verbose --component thirdparty/terraform update bullseye-wikimedia - T322344
  • 22:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P38486 and previous config saved to /var/cache/conftool/dbconfig/20221107-220051-marostegui.json
  • 21:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P38485 and previous config saved to /var/cache/conftool/dbconfig/20221107-215641-marostegui.json
  • 21:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P38484 and previous config saved to /var/cache/conftool/dbconfig/20221107-214545-marostegui.json
  • 21:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 21:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 21:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38483 and previous config saved to /var/cache/conftool/dbconfig/20221107-214254-ladsgroup.json
  • 21:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P38482 and previous config saved to /var/cache/conftool/dbconfig/20221107-214135-marostegui.json
  • 21:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38481 and previous config saved to /var/cache/conftool/dbconfig/20221107-213038-marostegui.json
  • 21:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318955)', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20221107-212900-ladsgroup.json
  • 21:28 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38480 and previous config saved to /var/cache/conftool/dbconfig/20221107-212828-marostegui.json
  • 21:28 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38479 and previous config saved to /var/cache/conftool/dbconfig/20221107-212800-marostegui.json
  • 21:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P38478 and previous config saved to /var/cache/conftool/dbconfig/20221107-212748-ladsgroup.json
  • 21:26 mutante: DNS - removing phab1001-aphlict.eqiad.wmnet - should have no effect because we use aphlict.discovery.wmnet - but if it does, then it's Phabricator realtime notifications - T280597
  • 21:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38477 and previous config saved to /var/cache/conftool/dbconfig/20221107-212628-marostegui.json
  • 21:26 urbanecm: Start [urbanecm@mwmaint1002 /srv/mediawiki]$ foreachwikiindblist group0 extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all # T315510, running in mwmaint1002 at a tmux session under my name
  • 21:25 mutante: DNS - removing phab1001-aphlict.eqiad.wmnet - should have no effect because we use aphlict.discovery.wmnet - but if it does, then it's Phabricator realtime notifications
  • 21:23 urbanecm@deploy1002: Finished scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353) (duration: 05m 47s)
  • 21:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38476 and previous config saved to /var/cache/conftool/dbconfig/20221107-212156-marostegui.json
  • 21:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 21:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 21:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38475 and previous config saved to /var/cache/conftool/dbconfig/20221107-212135-marostegui.json
  • 21:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:19 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:17 urbanecm@deploy1002: urbanecm and matmarex: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 21:17 urbanecm@deploy1002: Started scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353)
  • 21:16 urbanecm@deploy1002: Finished scap: Backport for ThreadItemStore: Update existing rows if possible rather than insert+delete (T321121) (duration: 07m 30s)
  • 21:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38474 and previous config saved to /var/cache/conftool/dbconfig/20221107-211353-ladsgroup.json
  • 21:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P38473 and previous config saved to /var/cache/conftool/dbconfig/20221107-211253-marostegui.json
  • 21:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P38472 and previous config saved to /var/cache/conftool/dbconfig/20221107-211241-ladsgroup.json
  • 21:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:09 urbanecm@deploy1002: urbanecm and matmarex: Backport for ThreadItemStore: Update existing rows if possible rather than insert+delete (T321121) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 21:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:09 urbanecm@deploy1002: Started scap: Backport for ThreadItemStore: Update existing rows if possible rather than insert+delete (T321121)
  • 21:08 urbanecm@deploy1002: Finished scap: Backport for Simplify some redundant settings, Clean up wgDiscussionToolsABTest config for beta cluster (duration: 04m 40s)
  • 21:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P38471 and previous config saved to /var/cache/conftool/dbconfig/20221107-210628-marostegui.json
  • 21:03 urbanecm@deploy1002: Started scap: Backport for Simplify some redundant settings, Clean up wgDiscussionToolsABTest config for beta cluster
  • 21:02 urbanecm@deploy1002: backport aborted: (duration: 00m 02s)
  • 20:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38470 and previous config saved to /var/cache/conftool/dbconfig/20221107-205847-ladsgroup.json
  • 20:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P38469 and previous config saved to /var/cache/conftool/dbconfig/20221107-205747-marostegui.json
  • 20:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38468 and previous config saved to /var/cache/conftool/dbconfig/20221107-205735-ladsgroup.json
  • 20:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P38467 and previous config saved to /var/cache/conftool/dbconfig/20221107-205122-marostegui.json
  • 20:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2150 (T318605)', diff saved to https://phabricator.wikimedia.org/P38466 and previous config saved to /var/cache/conftool/dbconfig/20221107-204827-ladsgroup.json
  • 20:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 20:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 20:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T318605)', diff saved to https://phabricator.wikimedia.org/P38465 and previous config saved to /var/cache/conftool/dbconfig/20221107-204805-ladsgroup.json
  • 20:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38464 and previous config saved to /var/cache/conftool/dbconfig/20221107-204340-ladsgroup.json
  • 20:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38463 and previous config saved to /var/cache/conftool/dbconfig/20221107-204240-marostegui.json
  • 20:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38462 and previous config saved to /var/cache/conftool/dbconfig/20221107-204131-marostegui.json
  • 20:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 20:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 20:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38461 and previous config saved to /var/cache/conftool/dbconfig/20221107-204110-marostegui.json
  • 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38460 and previous config saved to /var/cache/conftool/dbconfig/20221107-203626-ladsgroup.json
  • 20:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38459 and previous config saved to /var/cache/conftool/dbconfig/20221107-203615-marostegui.json
  • 20:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 20:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38458 and previous config saved to /var/cache/conftool/dbconfig/20221107-203609-ladsgroup.json
  • 20:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P38457 and previous config saved to /var/cache/conftool/dbconfig/20221107-203258-ladsgroup.json
  • 20:31 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38456 and previous config saved to /var/cache/conftool/dbconfig/20221107-203138-marostegui.json
  • 20:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 20:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 20:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T321130)', diff saved to https://phabricator.wikimedia.org/P38455 and previous config saved to /var/cache/conftool/dbconfig/20221107-203116-marostegui.json
  • 20:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P38454 and previous config saved to /var/cache/conftool/dbconfig/20221107-202603-marostegui.json
  • 20:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38453 and previous config saved to /var/cache/conftool/dbconfig/20221107-202102-ladsgroup.json
  • 20:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P38452 and previous config saved to /var/cache/conftool/dbconfig/20221107-201752-ladsgroup.json
  • 20:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P38451 and previous config saved to /var/cache/conftool/dbconfig/20221107-201610-marostegui.json
  • 20:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P38450 and previous config saved to /var/cache/conftool/dbconfig/20221107-201057-marostegui.json
  • 20:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38449 and previous config saved to /var/cache/conftool/dbconfig/20221107-200556-ladsgroup.json
  • 20:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T318605)', diff saved to https://phabricator.wikimedia.org/P38448 and previous config saved to /var/cache/conftool/dbconfig/20221107-200245-ladsgroup.json
  • 20:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P38447 and previous config saved to /var/cache/conftool/dbconfig/20221107-200103-marostegui.json
  • 19:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38446 and previous config saved to /var/cache/conftool/dbconfig/20221107-195550-marostegui.json
  • 19:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38445 and previous config saved to /var/cache/conftool/dbconfig/20221107-195340-marostegui.json
  • 19:53 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 19:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 19:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T321123)', diff saved to https://phabricator.wikimedia.org/P38444 and previous config saved to /var/cache/conftool/dbconfig/20221107-195319-marostegui.json
  • 19:51 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 19:51 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host sretest2002
  • 19:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38443 and previous config saved to /var/cache/conftool/dbconfig/20221107-195049-ladsgroup.json
  • 19:50 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2002
  • 19:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T321130)', diff saved to https://phabricator.wikimedia.org/P38442 and previous config saved to /var/cache/conftool/dbconfig/20221107-194557-marostegui.json
  • 19:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38441 and previous config saved to /var/cache/conftool/dbconfig/20221107-194335-ladsgroup.json
  • 19:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 19:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 19:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 19:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 19:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38440 and previous config saved to /var/cache/conftool/dbconfig/20221107-194319-ladsgroup.json
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T321130)', diff saved to https://phabricator.wikimedia.org/P38439 and previous config saved to /var/cache/conftool/dbconfig/20221107-194026-marostegui.json
  • 19:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 19:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 19:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P38438 and previous config saved to /var/cache/conftool/dbconfig/20221107-193813-marostegui.json
  • 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38437 and previous config saved to /var/cache/conftool/dbconfig/20221107-193646-ladsgroup.json
  • 19:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T318605)', diff saved to https://phabricator.wikimedia.org/P38436 and previous config saved to /var/cache/conftool/dbconfig/20221107-193625-ladsgroup.json
  • 19:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 19:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 19:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38435 and previous config saved to /var/cache/conftool/dbconfig/20221107-193604-marostegui.json
  • 19:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38434 and previous config saved to /var/cache/conftool/dbconfig/20221107-192813-ladsgroup.json
  • 19:25 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P38433 and previous config saved to /var/cache/conftool/dbconfig/20221107-192306-marostegui.json
  • 19:24 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:24 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P38432 and previous config saved to /var/cache/conftool/dbconfig/20221107-192119-ladsgroup.json
  • 19:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P38431 and previous config saved to /var/cache/conftool/dbconfig/20221107-192058-marostegui.json
  • 19:16 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38430 and previous config saved to /var/cache/conftool/dbconfig/20221107-191306-ladsgroup.json
  • 19:10 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T321123)', diff saved to https://phabricator.wikimedia.org/P38429 and previous config saved to /var/cache/conftool/dbconfig/20221107-190800-marostegui.json
  • 19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P38428 and previous config saved to /var/cache/conftool/dbconfig/20221107-190612-ladsgroup.json
  • 19:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P38427 and previous config saved to /var/cache/conftool/dbconfig/20221107-190551-marostegui.json
  • 19:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2136 (T321123)', diff saved to https://phabricator.wikimedia.org/P38426 and previous config saved to /var/cache/conftool/dbconfig/20221107-190550-marostegui.json
  • 19:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 19:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 19:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38425 and previous config saved to /var/cache/conftool/dbconfig/20221107-190528-marostegui.json
  • 18:58 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38424 and previous config saved to /var/cache/conftool/dbconfig/20221107-185800-ladsgroup.json
  • 18:57 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T318605)', diff saved to https://phabricator.wikimedia.org/P38423 and previous config saved to /var/cache/conftool/dbconfig/20221107-185105-ladsgroup.json
  • 18:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38422 and previous config saved to /var/cache/conftool/dbconfig/20221107-185044-marostegui.json
  • 18:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38421 and previous config saved to /var/cache/conftool/dbconfig/20221107-185035-ladsgroup.json
  • 18:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 18:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 18:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P38420 and previous config saved to /var/cache/conftool/dbconfig/20221107-185022-marostegui.json
  • 18:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38419 and previous config saved to /var/cache/conftool/dbconfig/20221107-184510-marostegui.json
  • 18:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 18:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 18:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38418 and previous config saved to /var/cache/conftool/dbconfig/20221107-184502-ladsgroup.json
  • 18:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 18:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321130)', diff saved to https://phabricator.wikimedia.org/P38417 and previous config saved to /var/cache/conftool/dbconfig/20221107-184448-marostegui.json
  • 18:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T318605)', diff saved to https://phabricator.wikimedia.org/P38416 and previous config saved to /var/cache/conftool/dbconfig/20221107-183722-ladsgroup.json
  • 18:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 18:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 18:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 18:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 18:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T318605)', diff saved to https://phabricator.wikimedia.org/P38415 and previous config saved to /var/cache/conftool/dbconfig/20221107-183643-ladsgroup.json
  • 18:36 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P38414 and previous config saved to /var/cache/conftool/dbconfig/20221107-183515-marostegui.json
  • 18:33 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host arclamp2001.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38413 and previous config saved to /var/cache/conftool/dbconfig/20221107-182956-ladsgroup.json
  • 18:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P38412 and previous config saved to /var/cache/conftool/dbconfig/20221107-182941-marostegui.json
  • 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2122 (T318605)', diff saved to https://phabricator.wikimedia.org/P38411 and previous config saved to /var/cache/conftool/dbconfig/20221107-182704-ladsgroup.json
  • 18:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 18:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 18:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38410 and previous config saved to /var/cache/conftool/dbconfig/20221107-182642-ladsgroup.json
  • 18:25 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host arclamp2001.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:24 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host puppetdb2003.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P38409 and previous config saved to /var/cache/conftool/dbconfig/20221107-182137-ladsgroup.json
  • 18:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38408 and previous config saved to /var/cache/conftool/dbconfig/20221107-182009-marostegui.json
  • 18:18 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host puppetdb2003.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:18 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38407 and previous config saved to /var/cache/conftool/dbconfig/20221107-181759-marostegui.json
  • 18:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 18:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 18:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T321123)', diff saved to https://phabricator.wikimedia.org/P38406 and previous config saved to /var/cache/conftool/dbconfig/20221107-181737-marostegui.json
  • 18:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38405 and previous config saved to /var/cache/conftool/dbconfig/20221107-181449-ladsgroup.json
  • 18:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P38404 and previous config saved to /var/cache/conftool/dbconfig/20221107-181435-marostegui.json
  • 18:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P38403 and previous config saved to /var/cache/conftool/dbconfig/20221107-181135-ladsgroup.json
  • 18:11 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host arclamp2001
  • 18:10 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host arclamp2001
  • 18:09 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host puppetdb2003
  • 18:08 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host puppetdb2003
  • 18:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P38402 and previous config saved to /var/cache/conftool/dbconfig/20221107-180630-ladsgroup.json
  • 18:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P38401 and previous config saved to /var/cache/conftool/dbconfig/20221107-180230-marostegui.json
  • 18:00 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38400 and previous config saved to /var/cache/conftool/dbconfig/20221107-175943-ladsgroup.json
  • 17:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321130)', diff saved to https://phabricator.wikimedia.org/P38399 and previous config saved to /var/cache/conftool/dbconfig/20221107-175928-marostegui.json
  • 17:58 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P38398 and previous config saved to /var/cache/conftool/dbconfig/20221107-175629-ladsgroup.json
  • 17:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T321130)', diff saved to https://phabricator.wikimedia.org/P38397 and previous config saved to /var/cache/conftool/dbconfig/20221107-175357-marostegui.json
  • 17:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 17:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 17:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321130)', diff saved to https://phabricator.wikimedia.org/P38396 and previous config saved to /var/cache/conftool/dbconfig/20221107-175335-marostegui.json
  • 17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38395 and previous config saved to /var/cache/conftool/dbconfig/20221107-175228-ladsgroup.json
  • 17:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 17:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38394 and previous config saved to /var/cache/conftool/dbconfig/20221107-175217-ladsgroup.json
  • 17:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T318605)', diff saved to https://phabricator.wikimedia.org/P38393 and previous config saved to /var/cache/conftool/dbconfig/20221107-175124-ladsgroup.json
  • 17:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P38392 and previous config saved to /var/cache/conftool/dbconfig/20221107-174724-marostegui.json
  • 17:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38391 and previous config saved to /var/cache/conftool/dbconfig/20221107-174123-ladsgroup.json
  • 17:41 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 17:38 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P38390 and previous config saved to /var/cache/conftool/dbconfig/20221107-173829-marostegui.json
  • 17:37 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 17:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38389 and previous config saved to /var/cache/conftool/dbconfig/20221107-173711-ladsgroup.json
  • 17:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T321123)', diff saved to https://phabricator.wikimedia.org/P38388 and previous config saved to /var/cache/conftool/dbconfig/20221107-173217-marostegui.json
  • 17:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2110 (T321123)', diff saved to https://phabricator.wikimedia.org/P38387 and previous config saved to /var/cache/conftool/dbconfig/20221107-173007-marostegui.json
  • 17:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 17:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 17:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38386 and previous config saved to /var/cache/conftool/dbconfig/20221107-172946-marostegui.json
  • 17:24 krinkle@deploy1002: Finished deploy [performance/arc-lamp@e1ac118]: https://gerrit.wikimedia.org/r/c/825870 - T322561, T315056 (duration: 00m 07s)
  • 17:24 krinkle@deploy1002: Started deploy [performance/arc-lamp@e1ac118]: https://gerrit.wikimedia.org/r/c/825870 - T322561, T315056
  • 17:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P38385 and previous config saved to /var/cache/conftool/dbconfig/20221107-172322-marostegui.json
  • 17:22 sukhe: reprepro -C main include bullseye-wikimedia purged_0.19_amd64.changes: T321309
  • 17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38384 and previous config saved to /var/cache/conftool/dbconfig/20221107-172204-ladsgroup.json
  • 17:22 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 17:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P38383 and previous config saved to /var/cache/conftool/dbconfig/20221107-171439-marostegui.json
  • 17:13 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 17:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321130)', diff saved to https://phabricator.wikimedia.org/P38382 and previous config saved to /var/cache/conftool/dbconfig/20221107-170816-marostegui.json
  • 17:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38381 and previous config saved to /var/cache/conftool/dbconfig/20221107-170658-ladsgroup.json
  • 17:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T321130)', diff saved to https://phabricator.wikimedia.org/P38380 and previous config saved to /var/cache/conftool/dbconfig/20221107-170247-marostegui.json
  • 17:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 17:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 16:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38379 and previous config saved to /var/cache/conftool/dbconfig/20221107-165943-ladsgroup.json
  • 16:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P38378 and previous config saved to /var/cache/conftool/dbconfig/20221107-165933-marostegui.json
  • 16:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 16:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 16:59 filippo@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host dispatch-be2001.codfw.wmnet
  • 16:59 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 16:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 16:58 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 16:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T321130)', diff saved to https://phabricator.wikimedia.org/P38377 and previous config saved to /var/cache/conftool/dbconfig/20221107-165847-marostegui.json
  • 16:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T318605)', diff saved to https://phabricator.wikimedia.org/P38376 and previous config saved to /var/cache/conftool/dbconfig/20221107-165108-ladsgroup.json
  • 16:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T318605)', diff saved to https://phabricator.wikimedia.org/P38375 and previous config saved to /var/cache/conftool/dbconfig/20221107-165046-ladsgroup.json
  • 16:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T318955)', diff saved to https://phabricator.wikimedia.org/P38374 and previous config saved to /var/cache/conftool/dbconfig/20221107-165036-ladsgroup.json
  • 16:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38373 and previous config saved to /var/cache/conftool/dbconfig/20221107-164427-marostegui.json
  • 16:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P38372 and previous config saved to /var/cache/conftool/dbconfig/20221107-164340-marostegui.json
  • 16:42 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38371 and previous config saved to /var/cache/conftool/dbconfig/20221107-164217-marostegui.json
  • 16:42 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321123)', diff saved to https://phabricator.wikimedia.org/P38370 and previous config saved to /var/cache/conftool/dbconfig/20221107-164122-marostegui.json
  • 16:38 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:35 filippo@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dispatch-be2001.codfw.wmnet on all recursors
  • 16:35 filippo@cumin1001: START - Cookbook sre.dns.wipe-cache dispatch-be2001.codfw.wmnet on all recursors
  • 16:35 filippo@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38369 and previous config saved to /var/cache/conftool/dbconfig/20221107-163540-ladsgroup.json
  • 16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38368 and previous config saved to /var/cache/conftool/dbconfig/20221107-163529-ladsgroup.json
  • 16:33 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P38367 and previous config saved to /var/cache/conftool/dbconfig/20221107-162834-marostegui.json
  • 16:26 filippo@cumin1001: START - Cookbook sre.dns.netbox
  • 16:26 filippo@cumin1001: START - Cookbook sre.ganeti.makevm for new host dispatch-be2001.codfw.wmnet
  • 16:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P38366 and previous config saved to /var/cache/conftool/dbconfig/20221107-162616-marostegui.json
  • 16:23 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:21 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:21 volans@cumin1001: START - Cookbook sre.dns.netbox
  • 16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38365 and previous config saved to /var/cache/conftool/dbconfig/20221107-162033-ladsgroup.json
  • 16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38364 and previous config saved to /var/cache/conftool/dbconfig/20221107-162023-ladsgroup.json
  • 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38363 and previous config saved to /var/cache/conftool/dbconfig/20221107-161837-ladsgroup.json
  • 16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 16:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T318605)', diff saved to https://phabricator.wikimedia.org/P38362 and previous config saved to /var/cache/conftool/dbconfig/20221107-161816-ladsgroup.json
  • 16:14 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T321130)', diff saved to https://phabricator.wikimedia.org/P38361 and previous config saved to /var/cache/conftool/dbconfig/20221107-161327-marostegui.json
  • 16:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1201 (T321130)', diff saved to https://phabricator.wikimedia.org/P38360 and previous config saved to /var/cache/conftool/dbconfig/20221107-161118-marostegui.json
  • 16:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P38359 and previous config saved to /var/cache/conftool/dbconfig/20221107-161109-marostegui.json
  • 16:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 16:10 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 16:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T321130)', diff saved to https://phabricator.wikimedia.org/P38358 and previous config saved to /var/cache/conftool/dbconfig/20221107-161050-marostegui.json
  • 16:06 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 16:06 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@e51ff67]: import_cirrus_indexes: set executor cores to 1 (duration: 02m 19s)
  • 16:05 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 16:05 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T318605)', diff saved to https://phabricator.wikimedia.org/P38357 and previous config saved to /var/cache/conftool/dbconfig/20221107-160527-ladsgroup.json
  • 16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T318955)', diff saved to https://phabricator.wikimedia.org/P38356 and previous config saved to /var/cache/conftool/dbconfig/20221107-160516-ladsgroup.json
  • 16:04 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 16:03 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1010.eqiad.wmnet to cluster eqiad and group C
  • 16:03 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@e51ff67]: import_cirrus_indexes: set executor cores to 1
  • 16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P38355 and previous config saved to /var/cache/conftool/dbconfig/20221107-160310-ladsgroup.json
  • 16:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1010.eqiad.wmnet to cluster eqiad and group C
  • 16:02 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:02 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 16:02 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T318955)', diff saved to https://phabricator.wikimedia.org/P38354 and previous config saved to /var/cache/conftool/dbconfig/20221107-160124-ladsgroup.json
  • 16:01 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 16:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 16:01 claime: cleaning up stale mwdebug kubernetes config
  • 16:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T318955)', diff saved to https://phabricator.wikimedia.org/P38353 and previous config saved to /var/cache/conftool/dbconfig/20221107-160102-ladsgroup.json
  • 16:00 cgoubert@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 15:59 cgoubert@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 15:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321123)', diff saved to https://phabricator.wikimedia.org/P38352 and previous config saved to /var/cache/conftool/dbconfig/20221107-155603-marostegui.json
  • 15:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P38351 and previous config saved to /var/cache/conftool/dbconfig/20221107-155544-marostegui.json
  • 15:55 elukey: upgrade istioctl to 1.15.3 on apt1001 for {buster,bullseye}-wikimedia - T322193
  • 15:54 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1199 (T321123)', diff saved to https://phabricator.wikimedia.org/P38350 and previous config saved to /var/cache/conftool/dbconfig/20221107-155455-marostegui.json
  • 15:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 15:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 15:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321123)', diff saved to https://phabricator.wikimedia.org/P38349 and previous config saved to /var/cache/conftool/dbconfig/20221107-155434-marostegui.json
  • 15:51 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:50 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:50 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:49 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P38348 and previous config saved to /var/cache/conftool/dbconfig/20221107-154803-ladsgroup.json
  • 15:46 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:46 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38347 and previous config saved to /var/cache/conftool/dbconfig/20221107-154556-ladsgroup.json
  • 15:44 sukhe: reprepro -C main include bullseye-wikimedia libvmod-querysort_0.3_amd64.changes: T321309
  • 15:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P38346 and previous config saved to /var/cache/conftool/dbconfig/20221107-154037-marostegui.json
  • 15:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P38345 and previous config saved to /var/cache/conftool/dbconfig/20221107-153927-marostegui.json
  • 15:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T318605)', diff saved to https://phabricator.wikimedia.org/P38344 and previous config saved to /var/cache/conftool/dbconfig/20221107-153257-ladsgroup.json
  • 15:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38343 and previous config saved to /var/cache/conftool/dbconfig/20221107-153049-ladsgroup.json
  • 15:27 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:27 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T321130)', diff saved to https://phabricator.wikimedia.org/P38342 and previous config saved to /var/cache/conftool/dbconfig/20221107-152531-marostegui.json
  • 15:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P38341 and previous config saved to /var/cache/conftool/dbconfig/20221107-152421-marostegui.json
  • 15:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T321130)', diff saved to https://phabricator.wikimedia.org/P38340 and previous config saved to /var/cache/conftool/dbconfig/20221107-152322-marostegui.json
  • 15:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 15:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 15:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38339 and previous config saved to /var/cache/conftool/dbconfig/20221107-152301-marostegui.json
  • 15:18 sukhe: reprepro -C main include bullseye-wikimedia libvmod-netmapper_1.9-2_amd64.changes: T321309
  • 15:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T318955)', diff saved to https://phabricator.wikimedia.org/P38338 and previous config saved to /var/cache/conftool/dbconfig/20221107-151543-ladsgroup.json
  • 15:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T318955)', diff saved to https://phabricator.wikimedia.org/P38337 and previous config saved to /var/cache/conftool/dbconfig/20221107-151151-ladsgroup.json
  • 15:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 15:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 15:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T318955)', diff saved to https://phabricator.wikimedia.org/P38336 and previous config saved to /var/cache/conftool/dbconfig/20221107-151130-ladsgroup.json
  • 15:09 sukhe: reprepro -C main include bullseye-wikimedia varnishkafka_1.1.0-2_amd64.changes: T321309
  • 15:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321123)', diff saved to https://phabricator.wikimedia.org/P38335 and previous config saved to /var/cache/conftool/dbconfig/20221107-150914-marostegui.json
  • 15:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1190 (T321123)', diff saved to https://phabricator.wikimedia.org/P38334 and previous config saved to /var/cache/conftool/dbconfig/20221107-150807-marostegui.json
  • 15:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 15:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P38333 and previous config saved to /var/cache/conftool/dbconfig/20221107-150754-marostegui.json
  • 15:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 15:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321123)', diff saved to https://phabricator.wikimedia.org/P38332 and previous config saved to /var/cache/conftool/dbconfig/20221107-150745-marostegui.json
  • 14:57 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1010.eqiad.wmnet
  • 14:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38331 and previous config saved to /var/cache/conftool/dbconfig/20221107-145623-ladsgroup.json
  • 14:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P38330 and previous config saved to /var/cache/conftool/dbconfig/20221107-145248-marostegui.json
  • 14:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P38329 and previous config saved to /var/cache/conftool/dbconfig/20221107-145239-marostegui.json
  • 14:48 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1010.eqiad.wmnet
  • 14:43 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38328 and previous config saved to /var/cache/conftool/dbconfig/20221107-144117-ladsgroup.json
  • 14:40 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:40 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38327 and previous config saved to /var/cache/conftool/dbconfig/20221107-143741-marostegui.json
  • 14:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P38326 and previous config saved to /var/cache/conftool/dbconfig/20221107-143732-marostegui.json
  • 14:37 urbanecm: UTC afternoon B&C window done
  • 14:36 urbanecm@deploy1002: Finished scap: Backport for MentorHooks: Add missing check for GEMentorshipUseIsActiveFlag (T322538), Rename QuitMentorship to ReassignMentees (T321382), ReassignMentees: Pass the actual performer to ChangeMentor (T321382), ManageMentorsRemoveMentor: Reassign mentees to a different mentor (T321382) (duration: 14m 56s)
  • 14:35 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38324 and previous config saved to /var/cache/conftool/dbconfig/20221107-143526-marostegui.json
  • 14:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 14:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 14:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321130)', diff saved to https://phabricator.wikimedia.org/P38323 and previous config saved to /var/cache/conftool/dbconfig/20221107-143504-marostegui.json
  • 14:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T318605)', diff saved to https://phabricator.wikimedia.org/P38322 and previous config saved to /var/cache/conftool/dbconfig/20221107-142904-ladsgroup.json
  • 14:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 14:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 14:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38321 and previous config saved to /var/cache/conftool/dbconfig/20221107-142832-ladsgroup.json
  • 14:26 urbanecm@deploy1002: urbanecm and urbanecm: Backport for MentorHooks: Add missing check for GEMentorshipUseIsActiveFlag (T322538), Rename QuitMentorship to ReassignMentees (T321382), ReassignMentees: Pass the actual performer to ChangeMentor (T321382), ManageMentorsRemoveMentor: Reassign mentees to a different mentor (T321382) synced to the testse
  • 14:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T318955)', diff saved to https://phabricator.wikimedia.org/P38320 and previous config saved to /var/cache/conftool/dbconfig/20221107-142610-ladsgroup.json
  • 14:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321123)', diff saved to https://phabricator.wikimedia.org/P38319 and previous config saved to /var/cache/conftool/dbconfig/20221107-142226-marostegui.json
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T318955)', diff saved to https://phabricator.wikimedia.org/P38318 and previous config saved to /var/cache/conftool/dbconfig/20221107-142219-ladsgroup.json
  • 14:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 14:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T318955)', diff saved to https://phabricator.wikimedia.org/P38317 and previous config saved to /var/cache/conftool/dbconfig/20221107-142157-ladsgroup.json
  • 14:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1010.eqiad.wmnet with OS bullseye
  • 14:21 urbanecm@deploy1002: Started scap: Backport for MentorHooks: Add missing check for GEMentorshipUseIsActiveFlag (T322538), Rename QuitMentorship to ReassignMentees (T321382), ReassignMentees: Pass the actual performer to ChangeMentor (T321382), ManageMentorsRemoveMentor: Reassign mentees to a different mentor (T321382)
  • 14:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T321123)', diff saved to https://phabricator.wikimedia.org/P38316 and previous config saved to /var/cache/conftool/dbconfig/20221107-142118-marostegui.json
  • 14:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 14:21 urbanecm@deploy1002: Backport cancelled.
  • 14:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 14:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 14:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 14:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38315 and previous config saved to /var/cache/conftool/dbconfig/20221107-142041-marostegui.json
  • 14:20 urbanecm@deploy1002: Finished scap: Backport for Set timezone for knwiki , knwiktionary , knwikiquote and knwikisource (T322471) (duration: 06m 01s)
  • 14:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P38314 and previous config saved to /var/cache/conftool/dbconfig/20221107-141958-marostegui.json
  • 14:19 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:19 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:18 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:14 urbanecm@deploy1002: urbanecm and anzx: Backport for Set timezone for knwiki , knwiktionary , knwikiquote and knwikisource (T322471) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 14:14 urbanecm@deploy1002: Started scap: Backport for Set timezone for knwiki , knwiktionary , knwikiquote and knwikisource (T322471)
  • 14:13 urbanecm@deploy1002: Finished scap: Backport for Enable flood flag on knwiki (T322472) (duration: 05m 10s)
  • 14:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38313 and previous config saved to /var/cache/conftool/dbconfig/20221107-141344-ladsgroup.json
  • 14:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P38312 and previous config saved to /var/cache/conftool/dbconfig/20221107-141326-ladsgroup.json
  • 14:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2120 (T318605)', diff saved to https://phabricator.wikimedia.org/P38311 and previous config saved to /var/cache/conftool/dbconfig/20221107-141224-ladsgroup.json
  • 14:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 14:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T318605)', diff saved to https://phabricator.wikimedia.org/P38310 and previous config saved to /var/cache/conftool/dbconfig/20221107-141203-ladsgroup.json
  • 14:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:09 xcollazo@deploy1002: Finished deploy [airflow-dags/platform_eng@3bb99c2]: Deploying to Airflow platform_eng instance (duration: 00m 20s)
  • 14:09 urbanecm@deploy1002: urbanecm and anzx: Backport for Enable flood flag on knwiki (T322472) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 14:09 xcollazo@deploy1002: Started deploy [airflow-dags/platform_eng@3bb99c2]: Deploying to Airflow platform_eng instance
  • 14:08 urbanecm@deploy1002: Started scap: Backport for Enable flood flag on knwiki (T322472)
  • 14:08 urbanecm@deploy1002: Finished scap: Backport for testwiki: Add config for Visual Editor Feature Use instrument (T309602) (duration: 06m 43s)
  • 14:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38309 and previous config saved to /var/cache/conftool/dbconfig/20221107-140651-ladsgroup.json
  • 14:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P38308 and previous config saved to /var/cache/conftool/dbconfig/20221107-140535-marostegui.json
  • 14:05 btullis@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:05 btullis@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 14:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:05 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1010.eqiad.wmnet with reason: host reimage
  • 14:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P38307 and previous config saved to /var/cache/conftool/dbconfig/20221107-140451-marostegui.json
  • 14:02 urbanecm@deploy1002: urbanecm and cjming: Backport for testwiki: Add config for Visual Editor Feature Use instrument (T309602) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 14:01 urbanecm@deploy1002: Started scap: Backport for testwiki: Add config for Visual Editor Feature Use instrument (T309602)
  • 14:01 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1010.eqiad.wmnet with reason: host reimage
  • 13:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38306 and previous config saved to /var/cache/conftool/dbconfig/20221107-135837-ladsgroup.json
  • 13:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P38305 and previous config saved to /var/cache/conftool/dbconfig/20221107-135819-ladsgroup.json
  • 13:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P38304 and previous config saved to /var/cache/conftool/dbconfig/20221107-135656-ladsgroup.json
  • 13:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38303 and previous config saved to /var/cache/conftool/dbconfig/20221107-135144-ladsgroup.json
  • 13:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P38302 and previous config saved to /var/cache/conftool/dbconfig/20221107-135028-marostegui.json
  • 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321130)', diff saved to https://phabricator.wikimedia.org/P38301 and previous config saved to /var/cache/conftool/dbconfig/20221107-134944-marostegui.json
  • 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T321130)', diff saved to https://phabricator.wikimedia.org/P38300 and previous config saved to /var/cache/conftool/dbconfig/20221107-134735-marostegui.json
  • 13:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 13:47 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1010.eqiad.wmnet with OS bullseye
  • 13:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T321130)', diff saved to https://phabricator.wikimedia.org/P38299 and previous config saved to /var/cache/conftool/dbconfig/20221107-134714-marostegui.json
  • 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38298 and previous config saved to /var/cache/conftool/dbconfig/20221107-134331-ladsgroup.json
  • 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38297 and previous config saved to /var/cache/conftool/dbconfig/20221107-134313-ladsgroup.json
  • 13:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P38296 and previous config saved to /var/cache/conftool/dbconfig/20221107-134150-ladsgroup.json
  • 13:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T318955)', diff saved to https://phabricator.wikimedia.org/P38295 and previous config saved to /var/cache/conftool/dbconfig/20221107-133638-ladsgroup.json
  • 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38294 and previous config saved to /var/cache/conftool/dbconfig/20221107-133522-marostegui.json
  • 13:34 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38293 and previous config saved to /var/cache/conftool/dbconfig/20221107-133414-marostegui.json
  • 13:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 13:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T321123)', diff saved to https://phabricator.wikimedia.org/P38292 and previous config saved to /var/cache/conftool/dbconfig/20221107-133353-marostegui.json
  • 13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T318955)', diff saved to https://phabricator.wikimedia.org/P38291 and previous config saved to /var/cache/conftool/dbconfig/20221107-133246-ladsgroup.json
  • 13:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 13:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T318955)', diff saved to https://phabricator.wikimedia.org/P38290 and previous config saved to /var/cache/conftool/dbconfig/20221107-133225-ladsgroup.json
  • 13:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P38289 and previous config saved to /var/cache/conftool/dbconfig/20221107-133208-marostegui.json
  • 13:29 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1010.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 13:29 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1010.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38288 and previous config saved to /var/cache/conftool/dbconfig/20221107-132824-ladsgroup.json
  • 13:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T318605)', diff saved to https://phabricator.wikimedia.org/P38287 and previous config saved to /var/cache/conftool/dbconfig/20221107-132643-ladsgroup.json
  • 13:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38286 and previous config saved to /var/cache/conftool/dbconfig/20221107-132109-ladsgroup.json
  • 13:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 13:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38285 and previous config saved to /var/cache/conftool/dbconfig/20221107-132048-ladsgroup.json
  • 13:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P38284 and previous config saved to /var/cache/conftool/dbconfig/20221107-131846-marostegui.json
  • 13:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38283 and previous config saved to /var/cache/conftool/dbconfig/20221107-131718-ladsgroup.json
  • 13:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P38282 and previous config saved to /var/cache/conftool/dbconfig/20221107-131701-marostegui.json
  • 13:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38281 and previous config saved to /var/cache/conftool/dbconfig/20221107-130541-ladsgroup.json
  • 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P38280 and previous config saved to /var/cache/conftool/dbconfig/20221107-130340-marostegui.json
  • 13:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38279 and previous config saved to /var/cache/conftool/dbconfig/20221107-130212-ladsgroup.json
  • 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T321130)', diff saved to https://phabricator.wikimedia.org/P38278 and previous config saved to /var/cache/conftool/dbconfig/20221107-130155-marostegui.json
  • 12:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T321130)', diff saved to https://phabricator.wikimedia.org/P38277 and previous config saved to /var/cache/conftool/dbconfig/20221107-125946-marostegui.json
  • 12:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T321130)', diff saved to https://phabricator.wikimedia.org/P38276 and previous config saved to /var/cache/conftool/dbconfig/20221107-125529-marostegui.json
  • 12:51 XioNoX: Add NAT for frmon2001 - T321735
  • 12:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38275 and previous config saved to /var/cache/conftool/dbconfig/20221107-125035-ladsgroup.json
  • 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T321123)', diff saved to https://phabricator.wikimedia.org/P38274 and previous config saved to /var/cache/conftool/dbconfig/20221107-124833-marostegui.json
  • 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T318955)', diff saved to https://phabricator.wikimedia.org/P38273 and previous config saved to /var/cache/conftool/dbconfig/20221107-124706-ladsgroup.json
  • 12:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T321123)', diff saved to https://phabricator.wikimedia.org/P38272 and previous config saved to /var/cache/conftool/dbconfig/20221107-124526-marostegui.json
  • 12:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 12:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 12:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38271 and previous config saved to /var/cache/conftool/dbconfig/20221107-124504-marostegui.json
  • 12:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P38270 and previous config saved to /var/cache/conftool/dbconfig/20221107-124022-marostegui.json
  • 12:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38269 and previous config saved to /var/cache/conftool/dbconfig/20221107-123528-ladsgroup.json
  • 12:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P38268 and previous config saved to /var/cache/conftool/dbconfig/20221107-122957-marostegui.json
  • 12:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38267 and previous config saved to /var/cache/conftool/dbconfig/20221107-122814-ladsgroup.json
  • 12:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38266 and previous config saved to /var/cache/conftool/dbconfig/20221107-122737-ladsgroup.json
  • 12:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P38265 and previous config saved to /var/cache/conftool/dbconfig/20221107-122516-marostegui.json
  • 12:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P38264 and previous config saved to /var/cache/conftool/dbconfig/20221107-121451-marostegui.json
  • 12:13 sukhe: reprepro -C main include bullseye-wikimedia prometheus-varnishkafka-exporter_0.1-2_amd64.changes: T321309
  • 12:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38263 and previous config saved to /var/cache/conftool/dbconfig/20221107-121230-ladsgroup.json
  • 12:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T321130)', diff saved to https://phabricator.wikimedia.org/P38262 and previous config saved to /var/cache/conftool/dbconfig/20221107-121009-marostegui.json
  • 12:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38261 and previous config saved to /var/cache/conftool/dbconfig/20221107-121004-ladsgroup.json
  • 12:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 12:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 12:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38260 and previous config saved to /var/cache/conftool/dbconfig/20221107-120942-ladsgroup.json
  • 12:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T321130)', diff saved to https://phabricator.wikimedia.org/P38259 and previous config saved to /var/cache/conftool/dbconfig/20221107-120800-marostegui.json
  • 12:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 12:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 12:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38258 and previous config saved to /var/cache/conftool/dbconfig/20221107-120739-marostegui.json
  • 12:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2108 (T318605)', diff saved to https://phabricator.wikimedia.org/P38257 and previous config saved to /var/cache/conftool/dbconfig/20221107-120614-ladsgroup.json
  • 12:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 12:05 vgutierrez: testing acme-chief 0.35 in acmechief-test1001
  • 12:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 12:05 volans@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts ganeti4003.ulsfo.wmnet
  • 12:05 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:04 volans@cumin1001: START - Cookbook sre.dns.netbox
  • 12:00 volans@cumin1001: START - Cookbook sre.hosts.decommission for hosts ganeti4003.ulsfo.wmnet
  • 11:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38256 and previous config saved to /var/cache/conftool/dbconfig/20221107-115944-marostegui.json
  • 11:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38255 and previous config saved to /var/cache/conftool/dbconfig/20221107-115737-marostegui.json
  • 11:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 11:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38254 and previous config saved to /var/cache/conftool/dbconfig/20221107-115723-ladsgroup.json
  • 11:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 11:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38253 and previous config saved to /var/cache/conftool/dbconfig/20221107-115715-marostegui.json
  • 11:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P38252 and previous config saved to /var/cache/conftool/dbconfig/20221107-115436-ladsgroup.json
  • 11:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P38251 and previous config saved to /var/cache/conftool/dbconfig/20221107-115232-marostegui.json
  • 11:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T318955)', diff saved to https://phabricator.wikimedia.org/P38250 and previous config saved to /var/cache/conftool/dbconfig/20221107-114649-ladsgroup.json
  • 11:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T318955)', diff saved to https://phabricator.wikimedia.org/P38249 and previous config saved to /var/cache/conftool/dbconfig/20221107-114628-ladsgroup.json
  • 11:44 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 112
  • 11:44 ayounsi@cumin1001: START - Cookbook sre.network.debug for Netbox circuit ID 112
  • 11:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38248 and previous config saved to /var/cache/conftool/dbconfig/20221107-114217-ladsgroup.json
  • 11:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P38247 and previous config saved to /var/cache/conftool/dbconfig/20221107-114209-marostegui.json
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P38246 and previous config saved to /var/cache/conftool/dbconfig/20221107-113929-ladsgroup.json
  • 11:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P38245 and previous config saved to /var/cache/conftool/dbconfig/20221107-113726-marostegui.json
  • 11:36 arturo: running homer on cr-eqiad/cr-codfw for https://gerrit.wikimedia.org/r/853947 (T321220, T309407)
  • 11:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38244 and previous config saved to /var/cache/conftool/dbconfig/20221107-113452-ladsgroup.json
  • 11:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 11:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 11:32 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38243 and previous config saved to /var/cache/conftool/dbconfig/20221107-113224-root.json
  • 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38242 and previous config saved to /var/cache/conftool/dbconfig/20221107-113122-ladsgroup.json
  • 11:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 11:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38241 and previous config saved to /var/cache/conftool/dbconfig/20221107-112857-ladsgroup.json
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P38240 and previous config saved to /var/cache/conftool/dbconfig/20221107-112702-marostegui.json
  • 11:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38239 and previous config saved to /var/cache/conftool/dbconfig/20221107-112423-ladsgroup.json
  • 11:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38238 and previous config saved to /var/cache/conftool/dbconfig/20221107-112219-marostegui.json
  • 11:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd1002.eqiad.wmnet to plain
  • 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd1002.eqiad.wmnet to plain
  • 11:17 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38237 and previous config saved to /var/cache/conftool/dbconfig/20221107-111719-root.json
  • 11:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38236 and previous config saved to /var/cache/conftool/dbconfig/20221107-111637-marostegui.json
  • 11:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 11:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 11:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38235 and previous config saved to /var/cache/conftool/dbconfig/20221107-111616-marostegui.json
  • 11:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38234 and previous config saved to /var/cache/conftool/dbconfig/20221107-111615-ladsgroup.json
  • 11:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38233 and previous config saved to /var/cache/conftool/dbconfig/20221107-111351-ladsgroup.json
  • 11:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38232 and previous config saved to /var/cache/conftool/dbconfig/20221107-111156-marostegui.json
  • 11:06 arturo: running homer on cr-eqiad/cr-codfw for https://gerrit.wikimedia.org/r/c/operations/homer/public/+/853374 (T321220, T309407)
  • 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd1002.eqiad.wmnet to drbd
  • 11:04 _joe_: manually started dump_cloud_ip_ranges.service
  • 11:02 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38231 and previous config saved to /var/cache/conftool/dbconfig/20221107-110215-root.json
  • 11:01 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 11:01 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 11:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P38230 and previous config saved to /var/cache/conftool/dbconfig/20221107-110110-marostegui.json
  • 11:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T318955)', diff saved to https://phabricator.wikimedia.org/P38229 and previous config saved to /var/cache/conftool/dbconfig/20221107-110109-ladsgroup.json
  • 10:59 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 10:59 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 10:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38228 and previous config saved to /var/cache/conftool/dbconfig/20221107-105844-ladsgroup.json
  • 10:57 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 10:57 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 10:56 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd1002.eqiad.wmnet to drbd
  • 10:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T318955)', diff saved to https://phabricator.wikimedia.org/P38227 and previous config saved to /var/cache/conftool/dbconfig/20221107-105015-ladsgroup.json
  • 10:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 10:47 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38226 and previous config saved to /var/cache/conftool/dbconfig/20221107-104710-root.json
  • 10:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P38225 and previous config saved to /var/cache/conftool/dbconfig/20221107-104603-marostegui.json
  • 10:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38224 and previous config saved to /var/cache/conftool/dbconfig/20221107-104338-ladsgroup.json
  • 10:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 10:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 10:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T318955)', diff saved to https://phabricator.wikimedia.org/P38223 and previous config saved to /var/cache/conftool/dbconfig/20221107-104101-ladsgroup.json
  • 10:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38222 and previous config saved to /var/cache/conftool/dbconfig/20221107-103622-ladsgroup.json
  • 10:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 10:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 10:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38221 and previous config saved to /var/cache/conftool/dbconfig/20221107-103549-ladsgroup.json
  • 10:32 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 5398
  • 10:32 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38220 and previous config saved to /var/cache/conftool/dbconfig/20221107-103205-root.json
  • 10:31 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 5398
  • 10:31 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30103
  • 10:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38219 and previous config saved to /var/cache/conftool/dbconfig/20221107-103056-marostegui.json
  • 10:30 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 30103
  • 10:29 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 399338
  • 10:29 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 399338
  • 10:28 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 59796
  • 10:28 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 59796
  • 10:27 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4817
  • 10:26 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 4817
  • 10:26 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 35598
  • 10:26 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 35598
  • 10:26 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30058
  • 10:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38218 and previous config saved to /var/cache/conftool/dbconfig/20221107-102555-ladsgroup.json
  • 10:25 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 30058
  • 10:25 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 46416
  • 10:25 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 46416
  • 10:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38217 and previous config saved to /var/cache/conftool/dbconfig/20221107-102509-marostegui.json
  • 10:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 10:25 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 10:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38216 and previous config saved to /var/cache/conftool/dbconfig/20221107-102458-marostegui.json
  • 10:24 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 3214
  • 10:24 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 3214
  • 10:21 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 17511
  • 10:21 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 17511
  • 10:21 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 12400
  • 10:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38215 and previous config saved to /var/cache/conftool/dbconfig/20221107-102043-ladsgroup.json
  • 10:20 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 12400
  • 10:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 7459
  • 10:19 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 7459
  • 10:17 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 6661
  • 10:17 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 6661
  • 10:17 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 5398
  • 10:17 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 5398
  • 10:17 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38214 and previous config saved to /var/cache/conftool/dbconfig/20221107-101700-root.json
  • 10:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38213 and previous config saved to /var/cache/conftool/dbconfig/20221107-101140-marostegui.json
  • 10:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38212 and previous config saved to /var/cache/conftool/dbconfig/20221107-101102-marostegui.json
  • 10:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38211 and previous config saved to /var/cache/conftool/dbconfig/20221107-101048-ladsgroup.json
  • 10:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38210 and previous config saved to /var/cache/conftool/dbconfig/20221107-100952-marostegui.json
  • 10:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38209 and previous config saved to /var/cache/conftool/dbconfig/20221107-100536-ladsgroup.json
  • 10:01 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38208 and previous config saved to /var/cache/conftool/dbconfig/20221107-100155-root.json
  • 09:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P38207 and previous config saved to /var/cache/conftool/dbconfig/20221107-095556-marostegui.json
  • 09:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T318955)', diff saved to https://phabricator.wikimedia.org/P38206 and previous config saved to /var/cache/conftool/dbconfig/20221107-095542-ladsgroup.json
  • 09:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38205 and previous config saved to /var/cache/conftool/dbconfig/20221107-095445-marostegui.json
  • 09:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T318955)', diff saved to https://phabricator.wikimedia.org/P38204 and previous config saved to /var/cache/conftool/dbconfig/20221107-095149-ladsgroup.json
  • 09:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 09:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 09:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38203 and previous config saved to /var/cache/conftool/dbconfig/20221107-095030-ladsgroup.json
  • 09:49 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 09:48 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 09:48 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudmetrics1004.eqiad.wmnet
  • 09:46 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38202 and previous config saved to /var/cache/conftool/dbconfig/20221107-094650-root.json
  • 09:46 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 09:46 cgoubert@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 09:45 cgoubert@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 09:44 cgoubert@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 09:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38201 and previous config saved to /var/cache/conftool/dbconfig/20221107-094315-ladsgroup.json
  • 09:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 09:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 09:42 cgoubert@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 09:42 ladsgroup@deploy1002: Finished scap: Backport for pruneRevData: Make it reload config (duration: 08m 10s)
  • 09:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 09:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 09:41 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 09:40 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 09:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P38200 and previous config saved to /var/cache/conftool/dbconfig/20221107-094050-marostegui.json
  • 09:40 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 09:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 09:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38199 and previous config saved to /var/cache/conftool/dbconfig/20221107-093939-marostegui.json
  • 09:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudmetrics1004.eqiad.wmnet
  • 09:38 elukey: restart rsyslog on ml-serve2001
  • 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudmetrics1003.eqiad.wmnet
  • 09:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38198 and previous config saved to /var/cache/conftool/dbconfig/20221107-093629-ladsgroup.json
  • 09:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:34 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for pruneRevData: Make it reload config synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 09:34 ladsgroup@deploy1002: Started scap: Backport for pruneRevData: Make it reload config
  • 09:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38197 and previous config saved to /var/cache/conftool/dbconfig/20221107-093352-marostegui.json
  • 09:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 09:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 09:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudmetrics1003.eqiad.wmnet
  • 09:29 moritzm: installing Django security updates
  • 09:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38196 and previous config saved to /var/cache/conftool/dbconfig/20221107-092543-marostegui.json
  • 09:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38195 and previous config saved to /var/cache/conftool/dbconfig/20221107-092436-marostegui.json
  • 09:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 09:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 09:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T321123)', diff saved to https://phabricator.wikimedia.org/P38194 and previous config saved to /var/cache/conftool/dbconfig/20221107-092414-marostegui.json
  • 09:18 moritzm: draining ganeti1010 for eventual reimage T311687
  • 09:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P38193 and previous config saved to /var/cache/conftool/dbconfig/20221107-090908-marostegui.json
  • 09:08 Emperor: set thanos ring replicas to 3.30 T311690
  • 09:06 urbanecm@deploy1002: Finished scap: Backport for Set wmgVisualEditorAccessRestbaseDirectly = false for testwiki and dewiki.beta (T320531) (duration: 09m 07s)
  • 09:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 09:03 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 09:03 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 09:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:57 urbanecm@deploy1002: urbanecm and daniel: Backport for Set wmgVisualEditorAccessRestbaseDirectly = false for testwiki and dewiki.beta (T320531) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 08:57 urbanecm@deploy1002: Started scap: Backport for Set wmgVisualEditorAccessRestbaseDirectly = false for testwiki and dewiki.beta (T320531)
  • 08:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P38192 and previous config saved to /var/cache/conftool/dbconfig/20221107-085402-marostegui.json
  • 08:52 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:51 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:51 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:50 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:49 marostegui@deploy1002: Finished scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc3 master" (duration: 04m 22s)
  • 08:45 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "ProductionServices.php: Promote pc2014 to pc3 master" synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 08:45 marostegui@deploy1002: Started scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc3 master"
  • 08:40 urbanecm: UTC morning B&C window done
  • 08:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T321123)', diff saved to https://phabricator.wikimedia.org/P38191 and previous config saved to /var/cache/conftool/dbconfig/20221107-083855-marostegui.json
  • 08:38 urbanecm@deploy1002: Finished scap: Backport for ContentTranslation: Move haw, ps and xh Wikipedias out of Beta (duration: 06m 56s)
  • 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T321123)', diff saved to https://phabricator.wikimedia.org/P38190 and previous config saved to /var/cache/conftool/dbconfig/20221107-083648-marostegui.json
  • 08:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 08:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321123)', diff saved to https://phabricator.wikimedia.org/P38189 and previous config saved to /var/cache/conftool/dbconfig/20221107-083626-marostegui.json
  • 08:35 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:34 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:33 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:31 urbanecm@deploy1002: urbanecm and kartik: Backport for ContentTranslation: Move haw, ps and xh Wikipedias out of Beta synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 08:31 urbanecm@deploy1002: Started scap: Backport for ContentTranslation: Move haw, ps and xh Wikipedias out of Beta
  • 08:30 urbanecm@deploy1002: Finished scap: Backport for Set ContentTranslation MT threshold to 75 in Japanese WP (T321819) (duration: 06m 24s)
  • 08:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:24 urbanecm@deploy1002: urbanecm and kartik: Backport for Set ContentTranslation MT threshold to 75 in Japanese WP (T321819) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 08:24 urbanecm@deploy1002: Started scap: Backport for Set ContentTranslation MT threshold to 75 in Japanese WP (T321819)
  • 08:23 urbanecm@deploy1002: Finished scap: Backport for Set VisualEditorDefaultParsoidClient for dewiki-beta mad testwiki (T320531) (duration: 18m 54s)
  • 08:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38188 and previous config saved to /var/cache/conftool/dbconfig/20221107-082120-marostegui.json
  • 08:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38185 and previous config saved to /var/cache/conftool/dbconfig/20221107-080613-marostegui.json
  • 08:05 urbanecm@deploy1002: urbanecm and daniel: Backport for Set VisualEditorDefaultParsoidClient for dewiki-beta mad testwiki (T320531) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 08:04 urbanecm@deploy1002: Started scap: Backport for Set VisualEditorDefaultParsoidClient for dewiki-beta mad testwiki (T320531)
  • 08:03 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:01 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:00 marostegui@deploy1002: Finished scap: Backport for ProductionServices.php: Promote pc2014 to pc3 master (duration: 04m 18s)
  • 07:56 marostegui@deploy1002: marostegui and marostegui: Backport for ProductionServices.php: Promote pc2014 to pc3 master synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 07:55 marostegui@deploy1002: Started scap: Backport for ProductionServices.php: Promote pc2014 to pc3 master
  • 07:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 07:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321123)', diff saved to https://phabricator.wikimedia.org/P38184 and previous config saved to /var/cache/conftool/dbconfig/20221107-075106-marostegui.json
  • 07:50 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 07:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 07:50 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T321123)', diff saved to https://phabricator.wikimedia.org/P38183 and previous config saved to /var/cache/conftool/dbconfig/20221107-074959-marostegui.json
  • 07:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 07:49 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 07:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321123)', diff saved to https://phabricator.wikimedia.org/P38182 and previous config saved to /var/cache/conftool/dbconfig/20221107-074938-marostegui.json
  • 07:47 marostegui@deploy1002: Finished scap: Backport for Revert "ProductionServices.php: Promote pc1014 to pc1 master" (duration: 04m 53s)
  • 07:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 07:44 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=frwiki` in a tmux at mwmaint1002 (T318457)
  • 07:42 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "ProductionServices.php: Promote pc1014 to pc1 master" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 07:42 marostegui@deploy1002: Started scap: Backport for Revert "ProductionServices.php: Promote pc1014 to pc1 master"
  • 07:37 elukey: `elukey@aux-k8s-worker1002:~$ sudo systemctl reset-failed ifup@ens13.service`
  • 07:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38181 and previous config saved to /var/cache/conftool/dbconfig/20221107-073431-marostegui.json
  • 07:31 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 07:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 07:28 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 07:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 07:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 07:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38180 and previous config saved to /var/cache/conftool/dbconfig/20221107-071925-marostegui.json
  • 07:17 marostegui@deploy1002: Finished scap: Backport for ProductionServices.php: Promote pc1014 to pc1 master (T322295) (duration: 04m 29s)
  • 07:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 07:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 07:13 marostegui@deploy1002: marostegui and marostegui: Backport for ProductionServices.php: Promote pc1014 to pc1 master (T322295) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
  • 07:13 marostegui@deploy1002: Started scap: Backport for ProductionServices.php: Promote pc1014 to pc1 master (T322295)
  • 07:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 07:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2011.codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover
  • 07:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on pc2011.codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover
  • 07:05 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=bnwiki` in a tmux at mwmaint1002 (T318457)
  • 07:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321123)', diff saved to https://phabricator.wikimedia.org/P38179 and previous config saved to /var/cache/conftool/dbconfig/20221107-070418-marostegui.json
  • 07:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T321123)', diff saved to https://phabricator.wikimedia.org/P38178 and previous config saved to /var/cache/conftool/dbconfig/20221107-070311-marostegui.json
  • 07:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 07:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 07:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38177 and previous config saved to /var/cache/conftool/dbconfig/20221107-070249-marostegui.json
  • 07:02 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=cswiki` in a tmux at mwmaint1002 (T318457)
  • 07:01 urbanecm@deploy1002: Finished scap: Backport for Add support for gemm_mentee_is_active (T318457), Calculate mentorship-related metrics (T318684) (duration: 06m 27s)
  • 06:55 urbanecm@deploy1002: urbanecm and urbanecm: Backport for Add support for gemm_mentee_is_active (T318457), Calculate mentorship-related metrics (T318684) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 06:55 urbanecm@deploy1002: Started scap: Backport for Add support for gemm_mentee_is_active (T318457), Calculate mentorship-related metrics (T318684)
  • 06:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es2024 T322406', diff saved to https://phabricator.wikimedia.org/P38176 and previous config saved to /var/cache/conftool/dbconfig/20221107-065251-root.json
  • 06:50 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es2023 to es5 primary and set section read-write T322406', diff saved to https://phabricator.wikimedia.org/P38175 and previous config saved to /var/cache/conftool/dbconfig/20221107-065048-root.json
  • 06:49 marostegui: Starting es5 codfw failover from es2024 to es2023 - T322406
  • 06:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38174 and previous config saved to /var/cache/conftool/dbconfig/20221107-064743-marostegui.json
  • 06:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322406
  • 06:46 marostegui@cumin1001: dbctl commit (dc=all): 'Set es2023 with weight 0 T322406', diff saved to https://phabricator.wikimedia.org/P38173 and previous config saved to /var/cache/conftool/dbconfig/20221107-064608-root.json
  • 06:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322406
  • 06:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38172 and previous config saved to /var/cache/conftool/dbconfig/20221107-063236-marostegui.json
  • 06:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38171 and previous config saved to /var/cache/conftool/dbconfig/20221107-061730-marostegui.json
  • 06:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38170 and previous config saved to /var/cache/conftool/dbconfig/20221107-061019-marostegui.json
  • 06:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 06:10 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 3292
  • 06:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 06:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 06:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 06:09 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 3292
  • 06:06 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 61461
  • 06:05 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 61461
  • 06:01 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 25091
  • 05:59 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 25091
  • 05:54 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 20115
  • 05:54 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 20115
  • 05:53 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 7843
  • 05:53 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 7843

2022-11-06

  • 08:23 elukey: restart rsyslog on centralog2002
  • 08:19 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
  • 08:19 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
  • 08:17 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
  • 08:17 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
  • 07:50 elukey: restart kube-apiserver on ml-serve-ctrl1001
  • 07:48 elukey: restart kube-apiserver on ml-serve-ctrl1002 - high HTTP 409 registered since days ago

2022-11-05

  • 12:56 mfossati@deploy1002: Finished deploy [airflow-dags/platform_eng@c849762]: (no justification provided) (duration: 00m 49s)
  • 12:55 mfossati@deploy1002: Started deploy [airflow-dags/platform_eng@c849762]: (no justification provided)
  • 09:39 elukey: reinstall kubernetes-node on ml-staging200[12] to allow puppet to run (cleanup after yesterday issue, worker nodes had master role applied)
  • 09:32 elukey: restart kube-apiserver on ml-staging-ctrl2001
  • 09:31 elukey: restart kube-apiserver on ml-staging-ctrl2002

2022-11-04

  • 18:31 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS buster
  • 18:12 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 18:09 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 17:44 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS buster
  • 17:25 fnegri@cumin1001: conftool action : set/pooled=yes; selector: name=dbproxy1019.eqiad.wmnet,service=wikireplicas-a
  • 17:19 fnegri@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host dbproxy1018.eqiad.wmnet
  • 17:08 fnegri@cumin1001: START - Cookbook sre.hosts.reboot-single for host dbproxy1018.eqiad.wmnet
  • 17:06 fnegri@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host dbproxy1018.eqiad.wmnet
  • 17:06 fnegri@cumin1001: START - Cookbook sre.hosts.reboot-single for host dbproxy1018.eqiad.wmnet
  • 17:04 fnegri@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host dbproxy1018.eqiad.wmnet
  • 17:04 fnegri@cumin1001: START - Cookbook sre.hosts.reboot-single for host dbproxy1018.eqiad.wmnet
  • 17:01 mvernon@cumin2002: conftool action : set/weight=40; selector: service=nginx,name=moss-fe2001.codfw.wmnet
  • 17:01 mvernon@cumin2002: conftool action : set/weight=40; selector: service=swift-fe,name=moss-fe2001.codfw.wmnet
  • 17:00 mvernon@cumin2002: conftool action : set/weight=40; selector: service=nginx,name=moss-fe1001.eqiad.wmnet
  • 17:00 mvernon@cumin2002: conftool action : set/weight=40; selector: service=swift-fe,name=moss-fe1001.eqiad.wmnet
  • 16:58 Emperor: rolling restart of swift-proxies to bring moss-fe{1,2}001 into service T322424
  • 16:55 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe2001.codfw.wmnet
  • 16:53 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe1001.eqiad.wmnet
  • 16:48 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host moss-fe2001.codfw.wmnet
  • 16:48 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host moss-fe1001.eqiad.wmnet
  • 16:41 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe2001.codfw.wmnet
  • 16:41 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe1001.eqiad.wmnet
  • 16:35 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host moss-fe2001.codfw.wmnet
  • 16:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp4052']
  • 16:34 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host moss-fe1001.eqiad.wmnet
  • 16:34 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 16:33 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cp4052']
  • 16:29 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe2001.codfw.wmnet with OS bullseye
  • 16:26 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe1001.eqiad.wmnet with OS bullseye
  • 16:13 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe2001.codfw.wmnet with reason: host reimage
  • 16:11 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: host reimage
  • 16:10 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe2001.codfw.wmnet with reason: host reimage
  • 16:07 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: host reimage
  • 16:06 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 16:06 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 15:57 Emperor: repool ms-fe{1,2}009
  • 15:55 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host moss-fe2001.codfw.wmnet with OS bullseye
  • 15:54 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host moss-fe1001.eqiad.wmnet with OS bullseye
  • 15:48 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 15:43 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:41 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 15:00 elukey: `elukey@cumin1001:~$ sudo cumin 'ms-fe2*' 'systemctl restart swift-proxy' -b 1 -s 20`
  • 14:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 14:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 14:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T318955)', diff saved to https://phabricator.wikimedia.org/P38159 and previous config saved to /var/cache/conftool/dbconfig/20221104-145225-ladsgroup.json
  • 14:52 vgutierrez@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=swift,name=eqiad
  • 14:51 Emperor: restart swift-proxy on ms-fe1012
  • 14:48 elukey: restart swift-proxy on ms-fe1011
  • 14:44 Emperor: restart swift-proxy on ms-fe1010
  • 14:41 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 14:37 vgutierrez@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=swift,name=eqiad
  • 14:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P38158 and previous config saved to /var/cache/conftool/dbconfig/20221104-143718-ladsgroup.json
  • 14:28 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 14:26 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp4052']
  • 14:25 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 14:24 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 14:23 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp4052']
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P38157 and previous config saved to /var/cache/conftool/dbconfig/20221104-142212-ladsgroup.json
  • 14:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T318955)', diff saved to https://phabricator.wikimedia.org/P38156 and previous config saved to /var/cache/conftool/dbconfig/20221104-140705-ladsgroup.json
  • 14:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1196 (T318955)', diff saved to https://phabricator.wikimedia.org/P38155 and previous config saved to /var/cache/conftool/dbconfig/20221104-140427-ladsgroup.json
  • 14:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 14:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 14:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T318955)', diff saved to https://phabricator.wikimedia.org/P38154 and previous config saved to /var/cache/conftool/dbconfig/20221104-140405-ladsgroup.json
  • 13:58 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 13:58 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dbprov2004
  • 13:57 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dbprov2004
  • 13:56 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:54 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 13:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P38153 and previous config saved to /var/cache/conftool/dbconfig/20221104-134859-ladsgroup.json
  • 13:45 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 13:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P38152 and previous config saved to /var/cache/conftool/dbconfig/20221104-133353-ladsgroup.json
  • 13:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T318955)', diff saved to https://phabricator.wikimedia.org/P38151 and previous config saved to /var/cache/conftool/dbconfig/20221104-131846-ladsgroup.json
  • 13:17 sukhe: reprepro -C main include bullseye-wikimedia python-logstash_0.4.6-3_amd64.changes: T321309
  • 13:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1186 (T318955)', diff saved to https://phabricator.wikimedia.org/P38150 and previous config saved to /var/cache/conftool/dbconfig/20221104-131607-ladsgroup.json
  • 13:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 13:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 13:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T318955)', diff saved to https://phabricator.wikimedia.org/P38149 and previous config saved to /var/cache/conftool/dbconfig/20221104-131546-ladsgroup.json
  • 13:11 sukhe: reprepro -C main include bullseye-wikimedia prometheus-rdkafka-exporter_0.3_amd64.changes: T321309
  • 13:10 sukhe: reprepro -C main include bullseye-wikimedia file-read-backwards_2.0.0-3_amd64.changes: T321309
  • 13:09 sukhe: reprepro -C main include bullseye-wikimedia fifo-log-demux_0.6.3_amd64.changes: T321309
  • 13:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P38148 and previous config saved to /var/cache/conftool/dbconfig/20221104-130039-ladsgroup.json
  • 12:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P38147 and previous config saved to /var/cache/conftool/dbconfig/20221104-124533-ladsgroup.json
  • 12:36 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 100%: After reboot', diff saved to https://phabricator.wikimedia.org/P38146 and previous config saved to /var/cache/conftool/dbconfig/20221104-123606-root.json
  • 12:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T318955)', diff saved to https://phabricator.wikimedia.org/P38145 and previous config saved to /var/cache/conftool/dbconfig/20221104-123026-ladsgroup.json
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1184 (T318955)', diff saved to https://phabricator.wikimedia.org/P38144 and previous config saved to /var/cache/conftool/dbconfig/20221104-122747-ladsgroup.json
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T318955)', diff saved to https://phabricator.wikimedia.org/P38143 and previous config saved to /var/cache/conftool/dbconfig/20221104-122726-ladsgroup.json
  • 12:21 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 75%: After reboot', diff saved to https://phabricator.wikimedia.org/P38142 and previous config saved to /var/cache/conftool/dbconfig/20221104-122101-root.json
  • 12:19 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:18 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T318955)', diff saved to https://phabricator.wikimedia.org/P38141 and previous config saved to /var/cache/conftool/dbconfig/20221104-121848-ladsgroup.json
  • 12:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P38140 and previous config saved to /var/cache/conftool/dbconfig/20221104-121219-ladsgroup.json
  • 12:05 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 50%: After reboot', diff saved to https://phabricator.wikimedia.org/P38139 and previous config saved to /var/cache/conftool/dbconfig/20221104-120556-root.json
  • 12:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P38138 and previous config saved to /var/cache/conftool/dbconfig/20221104-120342-ladsgroup.json
  • 11:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P38137 and previous config saved to /var/cache/conftool/dbconfig/20221104-115713-ladsgroup.json
  • 11:50 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 25%: After reboot', diff saved to https://phabricator.wikimedia.org/P38136 and previous config saved to /var/cache/conftool/dbconfig/20221104-115051-root.json
  • 11:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P38135 and previous config saved to /var/cache/conftool/dbconfig/20221104-114835-ladsgroup.json
  • 11:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T318955)', diff saved to https://phabricator.wikimedia.org/P38134 and previous config saved to /var/cache/conftool/dbconfig/20221104-114207-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1169 (T318955)', diff saved to https://phabricator.wikimedia.org/P38133 and previous config saved to /var/cache/conftool/dbconfig/20221104-113929-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 11:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T318955)', diff saved to https://phabricator.wikimedia.org/P38132 and previous config saved to /var/cache/conftool/dbconfig/20221104-113725-ladsgroup.json
  • 11:35 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 10%: After reboot', diff saved to https://phabricator.wikimedia.org/P38131 and previous config saved to /var/cache/conftool/dbconfig/20221104-113546-root.json
  • 11:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T318955)', diff saved to https://phabricator.wikimedia.org/P38130 and previous config saved to /var/cache/conftool/dbconfig/20221104-113329-ladsgroup.json
  • 11:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2176 (T318955)', diff saved to https://phabricator.wikimedia.org/P38129 and previous config saved to /var/cache/conftool/dbconfig/20221104-113048-ladsgroup.json
  • 11:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 11:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 11:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T318955)', diff saved to https://phabricator.wikimedia.org/P38128 and previous config saved to /var/cache/conftool/dbconfig/20221104-113027-ladsgroup.json
  • 11:27 elukey: restart kube-apiserver on ml-serve-ctrl2002 - high latencies for LIST (knative resources)
  • 11:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P38127 and previous config saved to /var/cache/conftool/dbconfig/20221104-112218-ladsgroup.json
  • 11:20 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 5%: After reboot', diff saved to https://phabricator.wikimedia.org/P38125 and previous config saved to /var/cache/conftool/dbconfig/20221104-112041-root.json
  • 11:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P38124 and previous config saved to /var/cache/conftool/dbconfig/20221104-111521-ladsgroup.json
  • 11:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P38123 and previous config saved to /var/cache/conftool/dbconfig/20221104-110712-ladsgroup.json
  • 11:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P38122 and previous config saved to /var/cache/conftool/dbconfig/20221104-110014-ladsgroup.json
  • 10:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T318955)', diff saved to https://phabricator.wikimedia.org/P38121 and previous config saved to /var/cache/conftool/dbconfig/20221104-105205-ladsgroup.json
  • 10:50 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 1%: After reboot', diff saved to https://phabricator.wikimedia.org/P38120 and previous config saved to /var/cache/conftool/dbconfig/20221104-105031-root.json
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1135 (T318955)', diff saved to https://phabricator.wikimedia.org/P38119 and previous config saved to /var/cache/conftool/dbconfig/20221104-104927-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 10:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T318955)', diff saved to https://phabricator.wikimedia.org/P38117 and previous config saved to /var/cache/conftool/dbconfig/20221104-104508-ladsgroup.json
  • 10:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2174 (T318955)', diff saved to https://phabricator.wikimedia.org/P38116 and previous config saved to /var/cache/conftool/dbconfig/20221104-104227-ladsgroup.json
  • 10:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 10:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 07:27 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P38115 and previous config saved to /var/cache/conftool/dbconfig/20221104-072722-root.json
  • 07:12 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P38114 and previous config saved to /var/cache/conftool/dbconfig/20221104-071217-root.json
  • 06:57 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P38113 and previous config saved to /var/cache/conftool/dbconfig/20221104-065712-root.json
  • 06:42 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P38112 and previous config saved to /var/cache/conftool/dbconfig/20221104-064207-root.json
  • 06:32 marostegui@cumin1001: dbctl commit (dc=all): 'Give weight to es2021', diff saved to https://phabricator.wikimedia.org/P38111 and previous config saved to /var/cache/conftool/dbconfig/20221104-063250-root.json
  • 06:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es2020 T322389', diff saved to https://phabricator.wikimedia.org/P38110 and previous config saved to /var/cache/conftool/dbconfig/20221104-063224-root.json
  • 06:31 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es2021 to es4 primary and set section read-write T322389', diff saved to https://phabricator.wikimedia.org/P38109 and previous config saved to /var/cache/conftool/dbconfig/20221104-063128-root.json
  • 06:30 marostegui: Starting es4 codfw failover from es2020 to es2021 - T322389
  • 06:27 marostegui@cumin1001: dbctl commit (dc=all): 'Set es2021 with weight 0 T322389', diff saved to https://phabricator.wikimedia.org/P38108 and previous config saved to /var/cache/conftool/dbconfig/20221104-062740-root.json
  • 06:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es4 T322389
  • 06:27 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 10%: After schema change', diff saved to https://phabricator.wikimedia.org/P38107 and previous config saved to /var/cache/conftool/dbconfig/20221104-062702-root.json
  • 06:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es4 T322389
  • 06:11 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 5%: After schema change', diff saved to https://phabricator.wikimedia.org/P38106 and previous config saved to /var/cache/conftool/dbconfig/20221104-061157-root.json
  • 05:56 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 3%: After schema change', diff saved to https://phabricator.wikimedia.org/P38105 and previous config saved to /var/cache/conftool/dbconfig/20221104-055652-root.json
  • 05:41 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 1%: After schema change', diff saved to https://phabricator.wikimedia.org/P38104 and previous config saved to /var/cache/conftool/dbconfig/20221104-054147-root.json
  • 03:24 ejegg: payments-wiki upgraded from 1c8f522f to 8baa6bb5
  • 01:29 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 46s)
  • 01:28 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)

2022-11-03

  • 22:45 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 01m 00s)
  • 22:44 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 22:43 krinkle@deploy1002: Finished deploy [integration/docroot@44f1640]: (no justification provided) (duration: 00m 29s)
  • 22:42 krinkle@deploy1002: Started deploy [integration/docroot@44f1640]: (no justification provided)
  • 22:40 cwhite: logstash eqiad - opensearch 2.2.0 upgrade complete T304440
  • 22:13 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 02m 07s)
  • 22:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 22:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 22:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T318605)', diff saved to https://phabricator.wikimedia.org/P38102 and previous config saved to /var/cache/conftool/dbconfig/20221103-221329-ladsgroup.json
  • 22:11 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 22:11 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 08s)
  • 22:11 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P38101 and previous config saved to /var/cache/conftool/dbconfig/20221103-215823-ladsgroup.json
  • 21:51 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 16s)
  • 21:51 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 21:48 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 50s)
  • 21:47 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 21:47 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 05s)
  • 21:47 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 21:46 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 50s)
  • 21:45 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P38100 and previous config saved to /var/cache/conftool/dbconfig/20221103-214317-ladsgroup.json
  • 21:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T318605)', diff saved to https://phabricator.wikimedia.org/P38099 and previous config saved to /var/cache/conftool/dbconfig/20221103-212810-ladsgroup.json
  • 21:26 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 02m 31s)
  • 21:24 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 21:22 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:22 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:22 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:22 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:19 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:19 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:19 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:19 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:08 ryankemper: [WCQS] Pooled `wcqs100[1,2]`
  • 21:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:03 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:03 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1200 (T318605)', diff saved to https://phabricator.wikimedia.org/P38098 and previous config saved to /var/cache/conftool/dbconfig/20221103-205855-ladsgroup.json
  • 20:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 20:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 20:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T318605)', diff saved to https://phabricator.wikimedia.org/P38097 and previous config saved to /var/cache/conftool/dbconfig/20221103-205823-ladsgroup.json
  • 20:57 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:56 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:56 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:56 urbanecm@deploy1002: Finished scap: Backport for ApiSetMenteeStatus: Check GEMentorshipEnabled in wiki config (T321805), SpecialManageMentors: Do not include explanatory text on transclusion (T321773) (duration: 06m 34s)
  • 20:55 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:52 bking@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
  • 20:50 urbanecm@deploy1002: urbanecm and urbanecm: Backport for ApiSetMenteeStatus: Check GEMentorshipEnabled in wiki config (T321805), SpecialManageMentors: Do not include explanatory text on transclusion (T321773) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 20:50 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 20:50 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:49 urbanecm@deploy1002: Started scap: Backport for ApiSetMenteeStatus: Check GEMentorshipEnabled in wiki config (T321805), SpecialManageMentors: Do not include explanatory text on transclusion (T321773)
  • 20:47 bking@cumin1001: START - Cookbook sre.wdqs.data-transfer
  • 20:46 samtar@deploy1002: Finished scap: Backport for Update lv and bn wordmarks (T319223) (duration: 06m 36s)
  • 20:45 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:44 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:44 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:43 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38096 and previous config saved to /var/cache/conftool/dbconfig/20221103-204316-ladsgroup.json
  • 20:39 samtar@deploy1002: samtar and jdlrobson: Backport for Update lv and bn wordmarks (T319223) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 20:39 samtar@deploy1002: Started scap: Backport for Update lv and bn wordmarks (T319223)
  • 20:35 ryankemper: T322037 Rolling changes in https://gerrit.wikimedia.org/r/c/operations/puppet/+/852885 and https://gerrit.wikimedia.org/r/853006 out to query service fleet, 4 hosts at a time: `ryankemper@cumin1001:~$ sudo -E cumin -b 4 'A:wcqs-public or A:wdqs-all' 'run-puppet-agent --force'`
  • 20:35 samtar@deploy1002: Finished scap: Backport for Finish moving to Page Tools naming convention (duration: 07m 50s)
  • 20:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38094 and previous config saved to /var/cache/conftool/dbconfig/20221103-202810-ladsgroup.json
  • 20:27 samtar@deploy1002: samtar and jdlrobson: Backport for Finish moving to Page Tools naming convention synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 20:27 samtar@deploy1002: Started scap: Backport for Finish moving to Page Tools naming convention
  • 20:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:21 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:20 thcipriani@deploy1002: Finished scap: Backport for Enable parsoid cache warming on testwiki. (T320535) (duration: 10m 30s)
  • 20:16 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:15 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:15 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:14 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T318605)', diff saved to https://phabricator.wikimedia.org/P38093 and previous config saved to /var/cache/conftool/dbconfig/20221103-201303-ladsgroup.json
  • 20:10 thcipriani@deploy1002: thcipriani and daniel: Backport for Enable parsoid cache warming on testwiki. (T320535) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 20:10 thcipriani@deploy1002: Started scap: Backport for Enable parsoid cache warming on testwiki. (T320535)
  • 20:06 samtar@deploy1002: backport aborted: (duration: 05m 23s)
  • 19:56 ryankemper: Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/852885; disabled puppet on query service fleet via `ryankemper@cumin1001:~$ sudo -E cumin 'A:wcqs-public or A:wdqs-all' 'sudo disable-puppet "T322037"'`; testing change on `wdqs1009`
  • 19:49 cstone: civicrm upgraded from d1f286f0 to c0db8f34
  • 19:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1185 (T318605)', diff saved to https://phabricator.wikimedia.org/P38092 and previous config saved to /var/cache/conftool/dbconfig/20221103-194839-ladsgroup.json
  • 19:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 19:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 19:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38091 and previous config saved to /var/cache/conftool/dbconfig/20221103-194818-ladsgroup.json
  • 19:40 brennen@deploy1002: Finished deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 (duration: 02m 24s)
  • 19:37 brennen@deploy1002: Started deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004
  • 19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38090 and previous config saved to /var/cache/conftool/dbconfig/20221103-193311-ladsgroup.json
  • 19:29 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 48s)
  • 19:28 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 19:26 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 48s)
  • 19:25 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 19:22 brennen@deploy1002: Finished deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 (duration: 00m 25s)
  • 19:22 brennen@deploy1002: Started deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004
  • 19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38089 and previous config saved to /var/cache/conftool/dbconfig/20221103-191805-ladsgroup.json
  • 19:06 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 01m 10s)
  • 19:05 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 19:03 ladsgroup@deploy1002: Finished scap: Backport for WikiExporter: Avoid calling reload in processing every row (T298485 T322360) (duration: 04m 24s)
  • 19:03 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 19:03 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 19:03 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38088 and previous config saved to /var/cache/conftool/dbconfig/20221103-190258-ladsgroup.json
  • 19:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:59 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for WikiExporter: Avoid calling reload in processing every row (T298485 T322360) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 18:59 ladsgroup@deploy1002: Started scap: Backport for WikiExporter: Avoid calling reload in processing every row (T298485 T322360)
  • 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38087 and previous config saved to /var/cache/conftool/dbconfig/20221103-182756-ladsgroup.json
  • 18:27 jynus@cumin1001: dbctl commit (dc=all): 'increase db1144:3315 load', diff saved to https://phabricator.wikimedia.org/P38086 and previous config saved to /var/cache/conftool/dbconfig/20221103-182750-jynus.json
  • 18:23 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 02m 56s)
  • 18:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 18:20 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)
  • 18:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 18:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 18:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 18:16 bblack: lvs1017: restart pybal to clear etcd error states
  • 18:15 bblack: lvs1018: restart pybal to clear etcd error states
  • 18:14 bblack: lvs1019: restart pybal to clear etcd error states
  • 18:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:09 bblack: lvs1020: restart pybal to hopefully clear etcd error states
  • 18:07 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: all wikis to 1.40.0-wmf.8 refs T320513
  • 17:46 vgutierrez: vgutierrez@conf1007:~$ sudo -i systemctl start etcd
  • 17:46 fnegri@cumin1001: conftool action : set/pooled=inactive; selector: name=dbproxy1018.eqiad.wmnet,service=wikireplicas-b
  • 17:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 17:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 17:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P38084 and previous config saved to /var/cache/conftool/dbconfig/20221103-174306-marostegui.json
  • 17:39 elukey: `sudo truncate -s 20G /var/log/nginx/etcd_access.log.1` on conf100[7-9], root partition full
  • 17:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P38083 and previous config saved to /var/cache/conftool/dbconfig/20221103-173843-ladsgroup.json
  • 17:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P38082 and previous config saved to /var/cache/conftool/dbconfig/20221103-173022-ladsgroup.json
  • 17:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P38081 and previous config saved to /var/cache/conftool/dbconfig/20221103-172759-marostegui.json
  • 17:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T318955)', diff saved to https://phabricator.wikimedia.org/P38080 and previous config saved to /var/cache/conftool/dbconfig/20221103-172338-ladsgroup.json
  • 17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T318955)', diff saved to https://phabricator.wikimedia.org/P38079 and previous config saved to /var/cache/conftool/dbconfig/20221103-172235-ladsgroup.json
  • 17:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P38078 and previous config saved to /var/cache/conftool/dbconfig/20221103-172100-ladsgroup.json
  • 17:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1134 (T318955)', diff saved to https://phabricator.wikimedia.org/P38077 and previous config saved to /var/cache/conftool/dbconfig/20221103-171959-ladsgroup.json
  • 17:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2173 (T318955)', diff saved to https://phabricator.wikimedia.org/P38076 and previous config saved to /var/cache/conftool/dbconfig/20221103-171952-ladsgroup.json
  • 17:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1134.eqiad.wmnet with reason: Maintenance
  • 17:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 17:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 17:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 17:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1134.eqiad.wmnet with reason: Maintenance
  • 17:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 17:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P38075 and previous config saved to /var/cache/conftool/dbconfig/20221103-171925-ladsgroup.json
  • 17:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1133.eqiad.wmnet with reason: Maintenance
  • 17:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1133.eqiad.wmnet with reason: Maintenance
  • 17:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 (T318955)', diff saved to https://phabricator.wikimedia.org/P38074 and previous config saved to /var/cache/conftool/dbconfig/20221103-171850-ladsgroup.json
  • 17:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P38073 and previous config saved to /var/cache/conftool/dbconfig/20221103-171512-ladsgroup.json
  • 17:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38072 and previous config saved to /var/cache/conftool/dbconfig/20221103-171250-marostegui.json
  • 17:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38071 and previous config saved to /var/cache/conftool/dbconfig/20221103-171028-marostegui.json
  • 17:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 17:10 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 17:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T321123)', diff saved to https://phabricator.wikimedia.org/P38070 and previous config saved to /var/cache/conftool/dbconfig/20221103-171004-marostegui.json
  • 17:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T318605)', diff saved to https://phabricator.wikimedia.org/P38069 and previous config saved to /var/cache/conftool/dbconfig/20221103-170553-ladsgroup.json
  • 17:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P38068 and previous config saved to /var/cache/conftool/dbconfig/20221103-170417-ladsgroup.json
  • 17:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P38067 and previous config saved to /var/cache/conftool/dbconfig/20221103-170341-ladsgroup.json
  • 17:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38066 and previous config saved to /var/cache/conftool/dbconfig/20221103-170003-ladsgroup.json
  • 16:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P38065 and previous config saved to /var/cache/conftool/dbconfig/20221103-165456-marostegui.json
  • 16:53 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 16:52 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 16:52 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 16:51 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 16:49 fnegri@cumin1001: conftool action : set/pooled=no; selector: name=dbproxy1018.eqiad.wmnet,service=wikireplicas-b
  • 16:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P38063 and previous config saved to /var/cache/conftool/dbconfig/20221103-164909-ladsgroup.json
  • 16:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P38062 and previous config saved to /var/cache/conftool/dbconfig/20221103-164833-ladsgroup.json
  • 16:48 fnegri@cumin1001: conftool action : set/pooled=yes; selector: name=dbproxy1019.eqiad.wmnet,service=wikireplicas-b
  • 16:42 fnegri@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1019.eqiad.wmnet with OS bullseye
  • 16:41 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 16:40 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P38056 and previous config saved to /var/cache/conftool/dbconfig/20221103-163947-marostegui.json
  • 16:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P38055 and previous config saved to /var/cache/conftool/dbconfig/20221103-163402-ladsgroup.json
  • 16:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 (T318955)', diff saved to https://phabricator.wikimedia.org/P38054 and previous config saved to /var/cache/conftool/dbconfig/20221103-163324-ladsgroup.json
  • 16:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P38053 and previous config saved to /var/cache/conftool/dbconfig/20221103-163219-ladsgroup.json
  • 16:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 16:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P38052 and previous config saved to /var/cache/conftool/dbconfig/20221103-163156-ladsgroup.json
  • 16:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1132 (T318955)', diff saved to https://phabricator.wikimedia.org/P38051 and previous config saved to /var/cache/conftool/dbconfig/20221103-163041-ladsgroup.json
  • 16:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1132.eqiad.wmnet with reason: Maintenance
  • 16:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1132.eqiad.wmnet with reason: Maintenance
  • 16:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128 (T318955)', diff saved to https://phabricator.wikimedia.org/P38050 and previous config saved to /var/cache/conftool/dbconfig/20221103-163016-ladsgroup.json
  • 16:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2157 (T318605)', diff saved to https://phabricator.wikimedia.org/P38049 and previous config saved to /var/cache/conftool/dbconfig/20221103-162927-ladsgroup.json
  • 16:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 16:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 16:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38048 and previous config saved to /var/cache/conftool/dbconfig/20221103-162904-ladsgroup.json
  • 16:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T321123)', diff saved to https://phabricator.wikimedia.org/P38047 and previous config saved to /var/cache/conftool/dbconfig/20221103-162437-marostegui.json
  • 16:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2120 (T321123)', diff saved to https://phabricator.wikimedia.org/P38046 and previous config saved to /var/cache/conftool/dbconfig/20221103-162214-marostegui.json
  • 16:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 16:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 16:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T321123)', diff saved to https://phabricator.wikimedia.org/P38045 and previous config saved to /var/cache/conftool/dbconfig/20221103-162152-marostegui.json
  • 16:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38044 and previous config saved to /var/cache/conftool/dbconfig/20221103-162141-ladsgroup.json
  • 16:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 16:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 16:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38043 and previous config saved to /var/cache/conftool/dbconfig/20221103-162118-ladsgroup.json
  • 16:20 fnegri@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1019.eqiad.wmnet with reason: host reimage
  • 16:18 fnegri@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy1019.eqiad.wmnet with reason: host reimage
  • 16:18 sukhe: reprepro -C main include bullseye-wikimedia trafficserver_9.1.3-1wm3_amd64.changes: T321309
  • 16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P38042 and previous config saved to /var/cache/conftool/dbconfig/20221103-161648-ladsgroup.json
  • 16:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P38041 and previous config saved to /var/cache/conftool/dbconfig/20221103-161507-ladsgroup.json
  • 16:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P38039 and previous config saved to /var/cache/conftool/dbconfig/20221103-161356-ladsgroup.json
  • 16:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 16:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 16:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 16:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 16:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P38037 and previous config saved to /var/cache/conftool/dbconfig/20221103-160645-marostegui.json
  • 16:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P38036 and previous config saved to /var/cache/conftool/dbconfig/20221103-160611-ladsgroup.json
  • 16:02 fnegri@cumin1001: START - Cookbook sre.hosts.reimage for host dbproxy1019.eqiad.wmnet with OS bullseye
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P38035 and previous config saved to /var/cache/conftool/dbconfig/20221103-160141-ladsgroup.json
  • 15:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P38034 and previous config saved to /var/cache/conftool/dbconfig/20221103-155958-ladsgroup.json
  • 15:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P38033 and previous config saved to /var/cache/conftool/dbconfig/20221103-155847-ladsgroup.json
  • 15:54 sukhe: sudo -i reprepro -C main include bullseye-wikimedia varnish_6.0.10-1wm2_amd64.changes: T321309
  • 15:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P38032 and previous config saved to /var/cache/conftool/dbconfig/20221103-155136-marostegui.json
  • 15:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P38031 and previous config saved to /var/cache/conftool/dbconfig/20221103-155101-ladsgroup.json
  • 15:48 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:46 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 15:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P38030 and previous config saved to /var/cache/conftool/dbconfig/20221103-154631-ladsgroup.json
  • 15:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128 (T318955)', diff saved to https://phabricator.wikimedia.org/P38029 and previous config saved to /var/cache/conftool/dbconfig/20221103-154449-ladsgroup.json
  • 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2167:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P38028 and previous config saved to /var/cache/conftool/dbconfig/20221103-154347-ladsgroup.json
  • 15:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38027 and previous config saved to /var/cache/conftool/dbconfig/20221103-154339-ladsgroup.json
  • 15:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T318955)', diff saved to https://phabricator.wikimedia.org/P38026 and previous config saved to /var/cache/conftool/dbconfig/20221103-154325-ladsgroup.json
  • 15:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1128 (T318955)', diff saved to https://phabricator.wikimedia.org/P38025 and previous config saved to /var/cache/conftool/dbconfig/20221103-154209-ladsgroup.json
  • 15:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1128.eqiad.wmnet with reason: Maintenance
  • 15:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1128.eqiad.wmnet with reason: Maintenance
  • 15:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T318955)', diff saved to https://phabricator.wikimedia.org/P38024 and previous config saved to /var/cache/conftool/dbconfig/20221103-154145-ladsgroup.json
  • 15:38 fnegri@cumin1001: conftool action : set/pooled=inactive; selector: name=dbproxy1019.eqiad.wmnet
  • 15:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T321123)', diff saved to https://phabricator.wikimedia.org/P38023 and previous config saved to /var/cache/conftool/dbconfig/20221103-153628-marostegui.json
  • 15:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38022 and previous config saved to /var/cache/conftool/dbconfig/20221103-153553-ladsgroup.json
  • 15:34 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2108 (T321123)', diff saved to https://phabricator.wikimedia.org/P38021 and previous config saved to /var/cache/conftool/dbconfig/20221103-153404-marostegui.json
  • 15:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 15:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 15:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 15:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 15:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 15:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 15:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 15:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 15:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T321123)', diff saved to https://phabricator.wikimedia.org/P38020 and previous config saved to /var/cache/conftool/dbconfig/20221103-153224-marostegui.json
  • 15:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P38019 and previous config saved to /var/cache/conftool/dbconfig/20221103-152817-ladsgroup.json
  • 15:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P38018 and previous config saved to /var/cache/conftool/dbconfig/20221103-152638-ladsgroup.json
  • 15:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 15:17 Emperor: comment out www-data crontab on cloudmetrics100{1,2} T297712
  • 15:17 otto@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Set destination_event_service for rc0.mediawiki.page_content_change to fix canary producer job (duration: 03m 36s)
  • 15:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P38017 and previous config saved to /var/cache/conftool/dbconfig/20221103-151716-marostegui.json
  • 15:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 15:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 15:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 15:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P38016 and previous config saved to /var/cache/conftool/dbconfig/20221103-151307-ladsgroup.json
  • 15:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P38015 and previous config saved to /var/cache/conftool/dbconfig/20221103-151129-ladsgroup.json
  • 15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38014 and previous config saved to /var/cache/conftool/dbconfig/20221103-150633-ladsgroup.json
  • 15:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 15:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T318605)', diff saved to https://phabricator.wikimedia.org/P38013 and previous config saved to /var/cache/conftool/dbconfig/20221103-150610-ladsgroup.json
  • 15:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P38012 and previous config saved to /var/cache/conftool/dbconfig/20221103-150208-marostegui.json
  • 14:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T318955)', diff saved to https://phabricator.wikimedia.org/P38011 and previous config saved to /var/cache/conftool/dbconfig/20221103-145759-ladsgroup.json
  • 14:56 fnegri@cumin1001: conftool action : set/pooled=yes; selector: name=dbproxy1018.eqiad.wmnet
  • 14:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T318955)', diff saved to https://phabricator.wikimedia.org/P38010 and previous config saved to /var/cache/conftool/dbconfig/20221103-145620-ladsgroup.json
  • 14:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2153 (T318955)', diff saved to https://phabricator.wikimedia.org/P38009 and previous config saved to /var/cache/conftool/dbconfig/20221103-145516-ladsgroup.json
  • 14:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 14:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T318955)', diff saved to https://phabricator.wikimedia.org/P38008 and previous config saved to /var/cache/conftool/dbconfig/20221103-145453-ladsgroup.json
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1119 (T318955)', diff saved to https://phabricator.wikimedia.org/P38007 and previous config saved to /var/cache/conftool/dbconfig/20221103-145339-ladsgroup.json
  • 14:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1119.eqiad.wmnet with reason: Maintenance
  • 14:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1119.eqiad.wmnet with reason: Maintenance
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 (T318955)', diff saved to https://phabricator.wikimedia.org/P38006 and previous config saved to /var/cache/conftool/dbconfig/20221103-145316-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P38005 and previous config saved to /var/cache/conftool/dbconfig/20221103-145133-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 14:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T318605)', diff saved to https://phabricator.wikimedia.org/P38004 and previous config saved to /var/cache/conftool/dbconfig/20221103-145110-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P38003 and previous config saved to /var/cache/conftool/dbconfig/20221103-145101-ladsgroup.json
  • 14:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T321123)', diff saved to https://phabricator.wikimedia.org/P38002 and previous config saved to /var/cache/conftool/dbconfig/20221103-144658-marostegui.json
  • 14:45 fnegri@cumin1001: conftool action : set/pooled=no; selector: name=dbproxy1019.eqiad.wmnet
  • 14:41 fnegri@cumin1001: conftool action : set/pooled=no; selector: service=wikireplicas-b,name=dbproxy1019.eqiad.wmnet
  • 14:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P38001 and previous config saved to /var/cache/conftool/dbconfig/20221103-143943-ladsgroup.json
  • 14:38 fnegri@cumin1001: conftool action : set/pooled=no; selector: service=wikireplicas-b,name=dbproxy1019
  • 14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P38000 and previous config saved to /var/cache/conftool/dbconfig/20221103-143809-ladsgroup.json
  • 14:37 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T321123)', diff saved to https://phabricator.wikimedia.org/P37999 and previous config saved to /var/cache/conftool/dbconfig/20221103-143745-marostegui.json
  • 14:37 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 14:37 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 14:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T321123)', diff saved to https://phabricator.wikimedia.org/P37998 and previous config saved to /var/cache/conftool/dbconfig/20221103-143722-marostegui.json
  • 14:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P37997 and previous config saved to /var/cache/conftool/dbconfig/20221103-143603-ladsgroup.json
  • 14:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P37996 and previous config saved to /var/cache/conftool/dbconfig/20221103-143552-ladsgroup.json
  • 14:26 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 14:26 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 14:26 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 14:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P37995 and previous config saved to /var/cache/conftool/dbconfig/20221103-142434-ladsgroup.json
  • 14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P37994 and previous config saved to /var/cache/conftool/dbconfig/20221103-142301-ladsgroup.json
  • 14:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P37993 and previous config saved to /var/cache/conftool/dbconfig/20221103-142215-marostegui.json
  • 14:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P37992 and previous config saved to /var/cache/conftool/dbconfig/20221103-142055-ladsgroup.json
  • 14:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T318605)', diff saved to https://phabricator.wikimedia.org/P37991 and previous config saved to /var/cache/conftool/dbconfig/20221103-142044-ladsgroup.json
  • 14:16 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 14:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:11 claime: Sunsetting search.wikimedia.org, starting a 2 week grace period before decommission - T316296
  • 14:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T318955)', diff saved to https://phabricator.wikimedia.org/P37990 and previous config saved to /var/cache/conftool/dbconfig/20221103-140926-ladsgroup.json
  • 14:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 (T318955)', diff saved to https://phabricator.wikimedia.org/P37989 and previous config saved to /var/cache/conftool/dbconfig/20221103-140753-ladsgroup.json
  • 14:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P37988 and previous config saved to /var/cache/conftool/dbconfig/20221103-140703-marostegui.json
  • 14:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2146 (T318955)', diff saved to https://phabricator.wikimedia.org/P37987 and previous config saved to /var/cache/conftool/dbconfig/20221103-140643-ladsgroup.json
  • 14:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 14:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 14:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T318955)', diff saved to https://phabricator.wikimedia.org/P37986 and previous config saved to /var/cache/conftool/dbconfig/20221103-140621-ladsgroup.json
  • 14:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T318605)', diff saved to https://phabricator.wikimedia.org/P37985 and previous config saved to /var/cache/conftool/dbconfig/20221103-140541-ladsgroup.json
  • 14:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1118 (T318955)', diff saved to https://phabricator.wikimedia.org/P37984 and previous config saved to /var/cache/conftool/dbconfig/20221103-140509-ladsgroup.json
  • 14:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 14:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 14:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T318955)', diff saved to https://phabricator.wikimedia.org/P37983 and previous config saved to /var/cache/conftool/dbconfig/20221103-140447-ladsgroup.json
  • 13:58 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:57 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:57 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:56 Lucas_WMDE: UTC afternoon backport+config window done
  • 13:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:56 lucaswerkmeister-wmde@deploy1002: Finished scap: Backport for Media border option applies to the media element, not the wrapper (T318300) (duration: 06m 31s)
  • 13:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2128 (T318605)', diff saved to https://phabricator.wikimedia.org/P37982 and previous config saved to /var/cache/conftool/dbconfig/20221103-135522-ladsgroup.json
  • 13:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 13:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 13:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 13:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 13:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T318605)', diff saved to https://phabricator.wikimedia.org/P37981 and previous config saved to /var/cache/conftool/dbconfig/20221103-135454-ladsgroup.json
  • 13:52 fnegri@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on dbproxy1019.eqiad.wmnet with reason: T313445
  • 13:52 fnegri@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on dbproxy1019.eqiad.wmnet with reason: T313445
  • 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T321123)', diff saved to https://phabricator.wikimedia.org/P37980 and previous config saved to /var/cache/conftool/dbconfig/20221103-135155-marostegui.json
  • 13:51 Lucas_WMDE: Finished scap: Backport for Enable the CampaignEvents extension on test(2)wiki and officewiki (T318592) (duration: 20m 46s) (originally at 13:38:40 UTC; logmsgbot dropped the message)
  • 13:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P37979 and previous config saved to /var/cache/conftool/dbconfig/20221103-135113-ladsgroup.json
  • 13:50 lucaswerkmeister-wmde@deploy1002: lucaswerkmeister-wmde and arlolra: Backport for Media border option applies to the media element, not the wrapper (T318300) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 13:50 lucaswerkmeister-wmde@deploy1002: Started scap: Backport for Media border option applies to the media element, not the wrapper (T318300)
  • 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1194 (T321123)', diff saved to https://phabricator.wikimedia.org/P37978 and previous config saved to /var/cache/conftool/dbconfig/20221103-134943-marostegui.json
  • 13:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P37977 and previous config saved to /var/cache/conftool/dbconfig/20221103-134935-ladsgroup.json
  • 13:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 13:49 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321123)', diff saved to https://phabricator.wikimedia.org/P37976 and previous config saved to /var/cache/conftool/dbconfig/20221103-134920-marostegui.json
  • 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1107 (T318955)', diff saved to https://phabricator.wikimedia.org/P37962 and previous config saved to /var/cache/conftool/dbconfig/20221103-131638-ladsgroup.json
  • 13:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: labstore1006.wikimedia.org
  • 13:28 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: labstore1006.wikimedia.org
  • 13:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T318955)', diff saved to https://phabricator.wikimedia.org/P37961 and previous config saved to /var/cache/conftool/dbconfig/20221103-131614-ladsgroup.json
  • 13:28 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: labstore1007.wikimedia.org
  • 13:28 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: labstore1007.wikimedia.org
  • 13:28 lucaswerkmeister-wmde@deploy1002: Finished scap: Backport for Remove $wgCampaignEventsDatabaseName (T318592) (duration: 08m 44s)
  • 13:20 Lucas_WMDE: lucaswerkmeister-wmde and daimona: Backport for Enable the CampaignEvents extension on test(2)wiki and officewiki (T318592) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet (on behalf of scap – log message got lost?)
  • 13:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P37960 and previous config saved to /var/cache/conftool/dbconfig/20221103-131103-ladsgroup.json
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T318605)', diff saved to https://phabricator.wikimedia.org/P37959 and previous config saved to /var/cache/conftool/dbconfig/20221103-130931-ladsgroup.json
  • 13:07 lucaswerkmeister-wmde@deploy1002: lucaswerkmeister-wmde and daimona: Backport for Remove $wgCampaignEventsDatabaseName (T318592) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 13:06 lucaswerkmeister-wmde@deploy1002: Started scap: Backport for Remove $wgCampaignEventsDatabaseName (T318592)
  • 13:06 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1025.eqiad.wmnet to cluster eqiad and group A
  • 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321123)', diff saved to https://phabricator.wikimedia.org/P37958 and previous config saved to /var/cache/conftool/dbconfig/20221103-130352-marostegui.json
  • 13:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1025.eqiad.wmnet to cluster eqiad and group A
  • 13:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P37957 and previous config saved to /var/cache/conftool/dbconfig/20221103-130153-ladsgroup.json
  • 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T321123)', diff saved to https://phabricator.wikimedia.org/P37956 and previous config saved to /var/cache/conftool/dbconfig/20221103-130140-marostegui.json
  • 13:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 13:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T321123)', diff saved to https://phabricator.wikimedia.org/P37955 and previous config saved to /var/cache/conftool/dbconfig/20221103-130117-marostegui.json
  • 13:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P37954 and previous config saved to /var/cache/conftool/dbconfig/20221103-130106-ladsgroup.json
  • 12:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P37953 and previous config saved to /var/cache/conftool/dbconfig/20221103-125555-ladsgroup.json
  • 12:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P37952 and previous config saved to /var/cache/conftool/dbconfig/20221103-124646-ladsgroup.json
  • 12:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P37951 and previous config saved to /var/cache/conftool/dbconfig/20221103-124607-marostegui.json
  • 12:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P37950 and previous config saved to /var/cache/conftool/dbconfig/20221103-124557-ladsgroup.json
  • 12:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2123 (T318605)', diff saved to https://phabricator.wikimedia.org/P37949 and previous config saved to /var/cache/conftool/dbconfig/20221103-124516-ladsgroup.json
  • 12:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 12:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 12:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T318605)', diff saved to https://phabricator.wikimedia.org/P37948 and previous config saved to /var/cache/conftool/dbconfig/20221103-124454-ladsgroup.json
  • 12:44 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry
  • 12:43 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on gitlab1004.wikimedia.org with reason: upgrade gitlab1004 to new version
  • 12:43 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 0:20:00 on gitlab1004.wikimedia.org with reason: upgrade gitlab1004 to new version
  • 12:41 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry
  • 12:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100 (T318605)', diff saved to https://phabricator.wikimedia.org/P37947 and previous config saved to /var/cache/conftool/dbconfig/20221103-124047-ladsgroup.json
  • 12:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1025.eqiad.wmnet
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T318955)', diff saved to https://phabricator.wikimedia.org/P37946 and previous config saved to /var/cache/conftool/dbconfig/20221103-123137-ladsgroup.json
  • 12:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P37945 and previous config saved to /var/cache/conftool/dbconfig/20221103-123101-marostegui.json
  • 12:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T318955)', diff saved to https://phabricator.wikimedia.org/P37944 and previous config saved to /var/cache/conftool/dbconfig/20221103-123048-ladsgroup.json
  • 12:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P37943 and previous config saved to /var/cache/conftool/dbconfig/20221103-122944-ladsgroup.json
  • 12:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2130 (T318955)', diff saved to https://phabricator.wikimedia.org/P37942 and previous config saved to /var/cache/conftool/dbconfig/20221103-122854-ladsgroup.json
  • 12:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 12:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 12:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T318955)', diff saved to https://phabricator.wikimedia.org/P37941 and previous config saved to /var/cache/conftool/dbconfig/20221103-122831-ladsgroup.json
  • 12:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1025.eqiad.wmnet
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T318955)', diff saved to https://phabricator.wikimedia.org/P37940 and previous config saved to /var/cache/conftool/dbconfig/20221103-122709-ladsgroup.json
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 12:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 12:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P37939 and previous config saved to /var/cache/conftool/dbconfig/20221103-122640-ladsgroup.json
  • 12:16 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:16 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T321123)', diff saved to https://phabricator.wikimedia.org/P37938 and previous config saved to /var/cache/conftool/dbconfig/20221103-121553-marostegui.json
  • 12:15 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:15 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1100 (T318605)', diff saved to https://phabricator.wikimedia.org/P37937 and previous config saved to /var/cache/conftool/dbconfig/20221103-121458-ladsgroup.json
  • 12:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 12:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P37936 and previous config saved to /var/cache/conftool/dbconfig/20221103-121436-ladsgroup.json
  • 12:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 12:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P37935 and previous config saved to /var/cache/conftool/dbconfig/20221103-121423-ladsgroup.json
  • 12:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P37934 and previous config saved to /var/cache/conftool/dbconfig/20221103-121320-ladsgroup.json
  • 12:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P37932 and previous config saved to /var/cache/conftool/dbconfig/20221103-121133-ladsgroup.json
  • 12:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1025.eqiad.wmnet with OS bullseye
  • 12:08 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:08 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:07 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:06 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:05 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 11:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T318605)', diff saved to https://phabricator.wikimedia.org/P37931 and previous config saved to /var/cache/conftool/dbconfig/20221103-115928-ladsgroup.json
  • 11:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P37930 and previous config saved to /var/cache/conftool/dbconfig/20221103-115916-ladsgroup.json
  • 11:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P37929 and previous config saved to /var/cache/conftool/dbconfig/20221103-115813-ladsgroup.json
  • 11:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P37928 and previous config saved to /var/cache/conftool/dbconfig/20221103-115624-ladsgroup.json
  • 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1025.eqiad.wmnet with reason: host reimage
  • 11:55 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 11:52 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1025.eqiad.wmnet with reason: host reimage
  • 11:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P37924 and previous config saved to /var/cache/conftool/dbconfig/20221103-114408-ladsgroup.json
  • 11:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T318955)', diff saved to https://phabricator.wikimedia.org/P37923 and previous config saved to /var/cache/conftool/dbconfig/20221103-114304-ladsgroup.json
  • 11:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T321123)', diff saved to https://phabricator.wikimedia.org/P37922 and previous config saved to /var/cache/conftool/dbconfig/20221103-114135-marostegui.json
  • 11:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 11:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P37921 and previous config saved to /var/cache/conftool/dbconfig/20221103-114116-ladsgroup.json
  • 11:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 11:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 11:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 11:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37920 and previous config saved to /var/cache/conftool/dbconfig/20221103-114054-marostegui.json
  • 11:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2116 (T318955)', diff saved to https://phabricator.wikimedia.org/P37919 and previous config saved to /var/cache/conftool/dbconfig/20221103-114021-ladsgroup.json
  • 11:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T318955)', diff saved to https://phabricator.wikimedia.org/P37918 and previous config saved to /var/cache/conftool/dbconfig/20221103-113956-ladsgroup.json
  • 11:39 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye
  • 11:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P37917 and previous config saved to /var/cache/conftool/dbconfig/20221103-113833-ladsgroup.json
  • 11:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P37916 and previous config saved to /var/cache/conftool/dbconfig/20221103-113809-ladsgroup.json
  • 11:37 volans@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1025.mgmt.eqiad.wmnet with reboot policy GRACEFUL
  • 11:35 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1025.eqiad.wmnet with OS bullseye
  • 11:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P37914 and previous config saved to /var/cache/conftool/dbconfig/20221103-112900-ladsgroup.json
  • 11:28 volans@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1025.mgmt.eqiad.wmnet with reboot policy GRACEFUL
  • 11:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P37913 and previous config saved to /var/cache/conftool/dbconfig/20221103-112546-marostegui.json
  • 11:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P37912 and previous config saved to /var/cache/conftool/dbconfig/20221103-112448-ladsgroup.json
  • 11:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2111 (T318605)', diff saved to https://phabricator.wikimedia.org/P37911 and previous config saved to /var/cache/conftool/dbconfig/20221103-112343-ladsgroup.json
  • 11:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P37910 and previous config saved to /var/cache/conftool/dbconfig/20221103-112300-ladsgroup.json
  • 11:16 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 11:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P37909 and previous config saved to /var/cache/conftool/dbconfig/20221103-111037-marostegui.json
  • 11:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P37908 and previous config saved to /var/cache/conftool/dbconfig/20221103-110939-ladsgroup.json
  • 11:09 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P37907 and previous config saved to /var/cache/conftool/dbconfig/20221103-110751-ladsgroup.json
  • 11:06 volans@cumin1001: START - Cookbook sre.dns.netbox
  • 11:06 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye
  • 11:04 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1025.eqiad.wmnet with OS bullseye
  • 10:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37906 and previous config saved to /var/cache/conftool/dbconfig/20221103-105527-marostegui.json
  • 10:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T318955)', diff saved to https://phabricator.wikimedia.org/P37905 and previous config saved to /var/cache/conftool/dbconfig/20221103-105429-ladsgroup.json
  • 10:53 jmm@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti1025.eqiad.wmnet']
  • 10:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P37904 and previous config saved to /var/cache/conftool/dbconfig/20221103-105243-ladsgroup.json
  • 10:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2103 (T318955)', diff saved to https://phabricator.wikimedia.org/P37903 and previous config saved to /var/cache/conftool/dbconfig/20221103-105148-ladsgroup.json
  • 10:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 10:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 10:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 10:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 10:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 (T318955)', diff saved to https://phabricator.wikimedia.org/P37902 and previous config saved to /var/cache/conftool/dbconfig/20221103-104957-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T318605)', diff saved to https://phabricator.wikimedia.org/P37901 and previous config saved to /var/cache/conftool/dbconfig/20221103-104942-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 10:43 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37900 and previous config saved to /var/cache/conftool/dbconfig/20221103-104313-marostegui.json
  • 10:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 10:42 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 10:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T321123)', diff saved to https://phabricator.wikimedia.org/P37899 and previous config saved to /var/cache/conftool/dbconfig/20221103-104239-marostegui.json
  • 10:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P37898 and previous config saved to /var/cache/conftool/dbconfig/20221103-102730-marostegui.json
  • 10:19 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1025.eqiad.wmnet']
  • 10:12 jmm@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ganeti1025.eqiad.wmnet']
  • 10:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P37897 and previous config saved to /var/cache/conftool/dbconfig/20221103-101222-marostegui.json
  • 09:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T321123)', diff saved to https://phabricator.wikimedia.org/P37896 and previous config saved to /var/cache/conftool/dbconfig/20221103-095715-marostegui.json
  • 09:55 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T321123)', diff saved to https://phabricator.wikimedia.org/P37895 and previous config saved to /var/cache/conftool/dbconfig/20221103-095501-marostegui.json
  • 09:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 09:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 09:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T321123)', diff saved to https://phabricator.wikimedia.org/P37894 and previous config saved to /var/cache/conftool/dbconfig/20221103-095409-marostegui.json
  • 09:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P37893 and previous config saved to /var/cache/conftool/dbconfig/20221103-093901-marostegui.json
  • 09:36 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1025.eqiad.wmnet']
  • 09:26 elukey@cumin1001: END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES codfw cluster: Roll restart of ORES's daemons.
  • 09:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P37892 and previous config saved to /var/cache/conftool/dbconfig/20221103-092353-marostegui.json
  • 09:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T321123)', diff saved to https://phabricator.wikimedia.org/P37891 and previous config saved to /var/cache/conftool/dbconfig/20221103-090844-marostegui.json
  • 09:06 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T321123)', diff saved to https://phabricator.wikimedia.org/P37890 and previous config saved to /var/cache/conftool/dbconfig/20221103-090631-marostegui.json
  • 09:06 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 09:06 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 09:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T321123)', diff saved to https://phabricator.wikimedia.org/P37889 and previous config saved to /var/cache/conftool/dbconfig/20221103-090607-marostegui.json
  • 09:05 elukey@cumin1001: START - Cookbook sre.ores.roll-restart-workers for ORES codfw cluster: Roll restart of ORES's daemons.
  • 09:02 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye
  • 09:02 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1025.eqiad.wmnet with OS bullseye
  • 08:56 elukey@cumin1001: END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES eqiad cluster: Roll restart of ORES's daemons.
  • 08:53 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye
  • 08:53 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1025.eqiad.wmnet with OS bullseye
  • 08:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P37888 and previous config saved to /var/cache/conftool/dbconfig/20221103-085059-marostegui.json
  • 08:44 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye
  • 08:43 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti1025.eqiad.wmnet with OS bullseye
  • 08:39 moritzm: installing ruby-nokogiri security updates
  • 08:37 elukey@cumin1001: START - Cookbook sre.ores.roll-restart-workers for ORES eqiad cluster: Roll restart of ORES's daemons.
  • 08:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P37887 and previous config saved to /var/cache/conftool/dbconfig/20221103-083549-marostegui.json
  • 08:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T321123)', diff saved to https://phabricator.wikimedia.org/P37886 and previous config saved to /var/cache/conftool/dbconfig/20221103-082040-marostegui.json
  • 08:18 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T321123)', diff saved to https://phabricator.wikimedia.org/P37885 and previous config saved to /var/cache/conftool/dbconfig/20221103-081827-marostegui.json
  • 08:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 08:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 08:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37884 and previous config saved to /var/cache/conftool/dbconfig/20221103-081805-marostegui.json
  • 08:17 moritzm: installing glibc security updates on buster
  • 08:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P37883 and previous config saved to /var/cache/conftool/dbconfig/20221103-080257-marostegui.json
  • 08:01 moritzm: installing exim4 security updates
  • 07:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye
  • 07:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1025.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 07:55 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1025.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 07:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P37882 and previous config saved to /var/cache/conftool/dbconfig/20221103-074748-marostegui.json
  • 07:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37881 and previous config saved to /var/cache/conftool/dbconfig/20221103-073240-marostegui.json
  • 07:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37880 and previous config saved to /var/cache/conftool/dbconfig/20221103-073028-marostegui.json
  • 07:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 07:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 07:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37879 and previous config saved to /var/cache/conftool/dbconfig/20221103-073004-marostegui.json
  • 07:15 marostegui: Create idm and idm_staging databases on m5 T320426
  • 07:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P37878 and previous config saved to /var/cache/conftool/dbconfig/20221103-071455-marostegui.json
  • 06:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P37877 and previous config saved to /var/cache/conftool/dbconfig/20221103-065946-marostegui.json
  • 06:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37876 and previous config saved to /var/cache/conftool/dbconfig/20221103-064438-marostegui.json
  • 06:42 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37875 and previous config saved to /var/cache/conftool/dbconfig/20221103-064225-marostegui.json
  • 06:42 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 06:42 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 06:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1130.eqiad.wmnet with reason: Maintenance
  • 06:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1130.eqiad.wmnet with reason: Maintenance
  • 06:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2113.codfw.wmnet with reason: Maintenance
  • 06:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2113.codfw.wmnet with reason: Maintenance

2022-11-02

  • 23:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37874 and previous config saved to /var/cache/conftool/dbconfig/20221102-232540-ladsgroup.json
  • 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P37873 and previous config saved to /var/cache/conftool/dbconfig/20221102-231031-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P37872 and previous config saved to /var/cache/conftool/dbconfig/20221102-225523-ladsgroup.json
  • 22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37871 and previous config saved to /var/cache/conftool/dbconfig/20221102-224014-ladsgroup.json
  • 21:58 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp4052']
  • 21:57 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 21:53 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp4052']
  • 21:53 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 21:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp4052.mgmt.ulsfo.wmnet with reboot policy FORCED
  • 21:31 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cp4052.mgmt.ulsfo.wmnet with reboot policy FORCED
  • 21:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T318605)', diff saved to https://phabricator.wikimedia.org/P37869 and previous config saved to /var/cache/conftool/dbconfig/20221102-211342-ladsgroup.json
  • 20:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P37868 and previous config saved to /var/cache/conftool/dbconfig/20221102-205833-ladsgroup.json
  • 20:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P37867 and previous config saved to /var/cache/conftool/dbconfig/20221102-204325-ladsgroup.json
  • 20:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37866 and previous config saved to /var/cache/conftool/dbconfig/20221102-203621-ladsgroup.json
  • 20:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 20:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 20:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318605)', diff saved to https://phabricator.wikimedia.org/P37865 and previous config saved to /var/cache/conftool/dbconfig/20221102-203547-ladsgroup.json
  • 20:33 TheresNoTime: UTC late backport window
  • 20:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T318605)', diff saved to https://phabricator.wikimedia.org/P37864 and previous config saved to /var/cache/conftool/dbconfig/20221102-202815-ladsgroup.json
  • 20:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:21 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P37863 and previous config saved to /var/cache/conftool/dbconfig/20221102-202037-ladsgroup.json
  • 20:18 samtar@deploy1002: Finished scap: Backport for Fix remaining Wikipedia logos (T319223) (duration: 05m 46s)
  • 20:16 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:15 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:14 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:13 samtar@deploy1002: samtar and jdlrobson: Backport for Fix remaining Wikipedia logos (T319223) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 20:12 samtar@deploy1002: Started scap: Backport for Fix remaining Wikipedia logos (T319223)
  • 20:10 samtar@deploy1002: Finished scap: Backport for Remove Research Incentive survey from enwiki (T318333) (duration: 05m 45s)
  • 20:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T318605)', diff saved to https://phabricator.wikimedia.org/P37862 and previous config saved to /var/cache/conftool/dbconfig/20221102-200610-ladsgroup.json
  • 20:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 20:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 20:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T318605)', diff saved to https://phabricator.wikimedia.org/P37861 and previous config saved to /var/cache/conftool/dbconfig/20221102-200537-ladsgroup.json
  • 20:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P37860 and previous config saved to /var/cache/conftool/dbconfig/20221102-200528-ladsgroup.json
  • 20:05 samtar@deploy1002: samtar and dani: Backport for Remove Research Incentive survey from enwiki (T318333) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 20:05 samtar@deploy1002: Started scap: Backport for Remove Research Incentive survey from enwiki (T318333)
  • 20:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:01 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 19:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P37859 and previous config saved to /var/cache/conftool/dbconfig/20221102-195029-ladsgroup.json
  • 19:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318605)', diff saved to https://phabricator.wikimedia.org/P37858 and previous config saved to /var/cache/conftool/dbconfig/20221102-195019-ladsgroup.json
  • 19:45 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp4052
  • 19:44 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp4052
  • 19:39 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:37 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 19:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P37857 and previous config saved to /var/cache/conftool/dbconfig/20221102-193522-ladsgroup.json
  • 19:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T321123)', diff saved to https://phabricator.wikimedia.org/P37856 and previous config saved to /var/cache/conftool/dbconfig/20221102-192039-marostegui.json
  • 19:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T318605)', diff saved to https://phabricator.wikimedia.org/P37855 and previous config saved to /var/cache/conftool/dbconfig/20221102-192014-ladsgroup.json
  • 19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T318605)', diff saved to https://phabricator.wikimedia.org/P37854 and previous config saved to /var/cache/conftool/dbconfig/20221102-191623-ladsgroup.json
  • 19:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 19:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 19:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 19:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 19:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318605)', diff saved to https://phabricator.wikimedia.org/P37853 and previous config saved to /var/cache/conftool/dbconfig/20221102-191557-ladsgroup.json
  • 19:11 dzahn@cumin2002: conftool action : set/pooled=no; selector: dc=codfw,name=phab2001-vcs.codfw.wmnet
  • 19:11 dzahn@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,name=phab2001-vcs.codfw.wmnet
  • 19:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P37852 and previous config saved to /var/cache/conftool/dbconfig/20221102-190531-marostegui.json
  • 19:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P37851 and previous config saved to /var/cache/conftool/dbconfig/20221102-190048-ladsgroup.json
  • 18:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T318605)', diff saved to https://phabricator.wikimedia.org/P37850 and previous config saved to /var/cache/conftool/dbconfig/20221102-185627-ladsgroup.json
  • 18:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 18:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 18:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T318605)', diff saved to https://phabricator.wikimedia.org/P37849 and previous config saved to /var/cache/conftool/dbconfig/20221102-185604-ladsgroup.json
  • 18:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P37848 and previous config saved to /var/cache/conftool/dbconfig/20221102-185023-marostegui.json
  • 18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P37847 and previous config saved to /var/cache/conftool/dbconfig/20221102-184538-ladsgroup.json
  • 18:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P37846 and previous config saved to /var/cache/conftool/dbconfig/20221102-184056-ladsgroup.json
  • 18:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T321123)', diff saved to https://phabricator.wikimedia.org/P37845 and previous config saved to /var/cache/conftool/dbconfig/20221102-183514-marostegui.json
  • 18:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2178 (T321123)', diff saved to https://phabricator.wikimedia.org/P37844 and previous config saved to /var/cache/conftool/dbconfig/20221102-183327-marostegui.json
  • 18:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 18:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 18:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37843 and previous config saved to /var/cache/conftool/dbconfig/20221102-183305-marostegui.json
  • 18:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318605)', diff saved to https://phabricator.wikimedia.org/P37842 and previous config saved to /var/cache/conftool/dbconfig/20221102-183031-ladsgroup.json
  • 18:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P37841 and previous config saved to /var/cache/conftool/dbconfig/20221102-182548-ladsgroup.json
  • 18:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P37840 and previous config saved to /var/cache/conftool/dbconfig/20221102-181753-marostegui.json
  • 18:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T318605)', diff saved to https://phabricator.wikimedia.org/P37839 and previous config saved to /var/cache/conftool/dbconfig/20221102-181039-ladsgroup.json
  • 18:10 jhuneidi@deploy1002: Synchronized php: group1 wikis to 1.40.0-wmf.8 refs T320513 (duration: 03m 43s)
  • 18:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:07 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.8 refs T320513
  • 18:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P37838 and previous config saved to /var/cache/conftool/dbconfig/20221102-180245-marostegui.json
  • 18:01 ejegg: updated fundraising python tools from 4c143d97 to 72570bdd
  • 17:52 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 17:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37837 and previous config saved to /var/cache/conftool/dbconfig/20221102-174737-marostegui.json
  • 17:46 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cp4052.ulsfo.wmnet
  • 17:46 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37836 and previous config saved to /var/cache/conftool/dbconfig/20221102-174504-marostegui.json
  • 17:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 17:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 17:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T321123)', diff saved to https://phabricator.wikimedia.org/P37835 and previous config saved to /var/cache/conftool/dbconfig/20221102-174451-marostegui.json
  • 17:43 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 17:41 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 17:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 17:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 17:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T318955)', diff saved to https://phabricator.wikimedia.org/P37834 and previous config saved to /var/cache/conftool/dbconfig/20221102-174138-ladsgroup.json
  • 17:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T318605)', diff saved to https://phabricator.wikimedia.org/P37833 and previous config saved to /var/cache/conftool/dbconfig/20221102-174110-ladsgroup.json
  • 17:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 17:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 17:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T318605)', diff saved to https://phabricator.wikimedia.org/P37832 and previous config saved to /var/cache/conftool/dbconfig/20221102-174048-ladsgroup.json
  • 17:40 mutante: clouddumps1002 - /usr/local/bin/dump-fetch-phabdumps.sh T322221
  • 17:39 pt1979@cumin2002: START - Cookbook sre.hosts.decommission for hosts cp4052.ulsfo.wmnet
  • 17:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P37831 and previous config saved to /var/cache/conftool/dbconfig/20221102-172944-marostegui.json
  • 17:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P37830 and previous config saved to /var/cache/conftool/dbconfig/20221102-172630-ladsgroup.json
  • 17:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P37829 and previous config saved to /var/cache/conftool/dbconfig/20221102-172540-ladsgroup.json
  • 17:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P37828 and previous config saved to /var/cache/conftool/dbconfig/20221102-171436-marostegui.json
  • 17:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P37827 and previous config saved to /var/cache/conftool/dbconfig/20221102-171122-ladsgroup.json
  • 17:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P37826 and previous config saved to /var/cache/conftool/dbconfig/20221102-171032-ladsgroup.json
  • 17:06 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 17:04 hashar@deploy1002: Finished deploy [integration/docroot@8d2f4a0]: Remove .zuul-change font-weight - T322168 (duration: 00m 10s)
  • 17:04 hashar@deploy1002: Started deploy [integration/docroot@8d2f4a0]: Remove .zuul-change font-weight - T322168
  • 16:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T321123)', diff saved to https://phabricator.wikimedia.org/P37825 and previous config saved to /var/cache/conftool/dbconfig/20221102-165927-marostegui.json
  • 16:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2157 (T321123)', diff saved to https://phabricator.wikimedia.org/P37824 and previous config saved to /var/cache/conftool/dbconfig/20221102-165656-marostegui.json
  • 16:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 16:56 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:56 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 16:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37823 and previous config saved to /var/cache/conftool/dbconfig/20221102-165631-marostegui.json
  • 16:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T318955)', diff saved to https://phabricator.wikimedia.org/P37822 and previous config saved to /var/cache/conftool/dbconfig/20221102-165614-ladsgroup.json
  • 16:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T318605)', diff saved to https://phabricator.wikimedia.org/P37821 and previous config saved to /var/cache/conftool/dbconfig/20221102-165523-ladsgroup.json
  • 16:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1197 (T318955)', diff saved to https://phabricator.wikimedia.org/P37820 and previous config saved to /var/cache/conftool/dbconfig/20221102-165400-ladsgroup.json
  • 16:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 16:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 16:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T318955)', diff saved to https://phabricator.wikimedia.org/P37819 and previous config saved to /var/cache/conftool/dbconfig/20221102-165337-ladsgroup.json
  • 16:52 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 16:46 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 16:45 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 16:44 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 16:43 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
  • 16:42 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:41 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
  • 16:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P37818 and previous config saved to /var/cache/conftool/dbconfig/20221102-164123-marostegui.json
  • 16:40 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
  • 16:40 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 16:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P37817 and previous config saved to /var/cache/conftool/dbconfig/20221102-163829-ladsgroup.json
  • 16:37 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 16:36 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 16:35 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 16:35 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
  • 16:34 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
  • 16:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T318605)', diff saved to https://phabricator.wikimedia.org/P37816 and previous config saved to /var/cache/conftool/dbconfig/20221102-163334-ladsgroup.json
  • 16:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 16:33 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
  • 16:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 16:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T318605)', diff saved to https://phabricator.wikimedia.org/P37815 and previous config saved to /var/cache/conftool/dbconfig/20221102-163300-ladsgroup.json
  • 16:32 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 16:31 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 16:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T318605)', diff saved to https://phabricator.wikimedia.org/P37814 and previous config saved to /var/cache/conftool/dbconfig/20221102-162629-ladsgroup.json
  • 16:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P37813 and previous config saved to /var/cache/conftool/dbconfig/20221102-162614-marostegui.json
  • 16:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P37812 and previous config saved to /var/cache/conftool/dbconfig/20221102-162320-ladsgroup.json
  • 16:22 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 16:22 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:21 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 16:21 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 16:20 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
  • 16:19 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
  • 16:19 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
  • 16:19 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 16:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P37811 and previous config saved to /var/cache/conftool/dbconfig/20221102-161753-ladsgroup.json
  • 16:12 Daimona: Creating schema for the CampaignEvents extension on testwiki, test2wiki and officewiki # T318595
  • 16:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37810 and previous config saved to /var/cache/conftool/dbconfig/20221102-161104-marostegui.json
  • 16:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37809 and previous config saved to /var/cache/conftool/dbconfig/20221102-160834-marostegui.json
  • 16:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 16:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 16:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T321123)', diff saved to https://phabricator.wikimedia.org/P37808 and previous config saved to /var/cache/conftool/dbconfig/20221102-160809-marostegui.json
  • 16:06 moritzm: installing glibc security updates on buster
  • 16:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1188 (T318955)', diff saved to https://phabricator.wikimedia.org/P37807 and previous config saved to /var/cache/conftool/dbconfig/20221102-160600-ladsgroup.json
  • 16:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T318955)', diff saved to https://phabricator.wikimedia.org/P37806 and previous config saved to /var/cache/conftool/dbconfig/20221102-160537-ladsgroup.json
  • 16:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P37805 and previous config saved to /var/cache/conftool/dbconfig/20221102-160243-ladsgroup.json
  • 16:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 16:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T318950)', diff saved to https://phabricator.wikimedia.org/P37804 and previous config saved to /var/cache/conftool/dbconfig/20221102-160136-ladsgroup.json
  • 15:53 dcausse: restarting blazegraph on wdqs1007 (BlazegraphFreeAllocatorsDecreasingRapidly)
  • 15:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P37803 and previous config saved to /var/cache/conftool/dbconfig/20221102-155302-marostegui.json
  • 15:52 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 15:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P37802 and previous config saved to /var/cache/conftool/dbconfig/20221102-155026-ladsgroup.json
  • 15:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T318605)', diff saved to https://phabricator.wikimedia.org/P37801 and previous config saved to /var/cache/conftool/dbconfig/20221102-154736-ladsgroup.json
  • 15:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P37800 and previous config saved to /var/cache/conftool/dbconfig/20221102-154628-ladsgroup.json
  • 15:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P37799 and previous config saved to /var/cache/conftool/dbconfig/20221102-153754-marostegui.json
  • 15:35 otto@cumin1001: END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
  • 15:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P37798 and previous config saved to /var/cache/conftool/dbconfig/20221102-153519-ladsgroup.json
  • 15:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P37797 and previous config saved to /var/cache/conftool/dbconfig/20221102-153121-ladsgroup.json
  • 15:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T321123)', diff saved to https://phabricator.wikimedia.org/P37796 and previous config saved to /var/cache/conftool/dbconfig/20221102-152244-marostegui.json
  • 15:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2128 (T321123)', diff saved to https://phabricator.wikimedia.org/P37795 and previous config saved to /var/cache/conftool/dbconfig/20221102-152113-marostegui.json
  • 15:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 15:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 15:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 15:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 15:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T321123)', diff saved to https://phabricator.wikimedia.org/P37794 and previous config saved to /var/cache/conftool/dbconfig/20221102-152045-marostegui.json
  • 15:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T318955)', diff saved to https://phabricator.wikimedia.org/P37793 and previous config saved to /var/cache/conftool/dbconfig/20221102-152012-ladsgroup.json
  • 15:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T318950)', diff saved to https://phabricator.wikimedia.org/P37792 and previous config saved to /var/cache/conftool/dbconfig/20221102-151613-ladsgroup.json
  • 15:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1201 (T318950)', diff saved to https://phabricator.wikimedia.org/P37791 and previous config saved to /var/cache/conftool/dbconfig/20221102-151403-ladsgroup.json
  • 15:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 15:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 15:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T318950)', diff saved to https://phabricator.wikimedia.org/P37790 and previous config saved to /var/cache/conftool/dbconfig/20221102-151341-ladsgroup.json
  • 15:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P37789 and previous config saved to /var/cache/conftool/dbconfig/20221102-150538-marostegui.json
  • 15:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T318955)', diff saved to https://phabricator.wikimedia.org/P37788 and previous config saved to /var/cache/conftool/dbconfig/20221102-150508-ladsgroup.json
  • 15:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37787 and previous config saved to /var/cache/conftool/dbconfig/20221102-150444-ladsgroup.json
  • 14:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P37786 and previous config saved to /var/cache/conftool/dbconfig/20221102-145833-ladsgroup.json
  • 14:53 otto@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
  • 14:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P37785 and previous config saved to /var/cache/conftool/dbconfig/20221102-145030-marostegui.json
  • 14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P37784 and previous config saved to /var/cache/conftool/dbconfig/20221102-144937-ladsgroup.json
  • 14:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T318605)', diff saved to https://phabricator.wikimedia.org/P37783 and previous config saved to /var/cache/conftool/dbconfig/20221102-144719-ladsgroup.json
  • 14:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 14:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 14:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T318605)', diff saved to https://phabricator.wikimedia.org/P37782 and previous config saved to /var/cache/conftool/dbconfig/20221102-144657-ladsgroup.json
  • 14:45 moritzm: installing ffmpeg security updates on bullseye
  • 14:43 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 14:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P37781 and previous config saved to /var/cache/conftool/dbconfig/20221102-144325-ladsgroup.json
  • 14:41 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 14:38 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:37 jhathaway@cumin1001: START - Cookbook sre.dns.netbox
  • 14:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T321123)', diff saved to https://phabricator.wikimedia.org/P37780 and previous config saved to /var/cache/conftool/dbconfig/20221102-143522-marostegui.json
  • 14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P37779 and previous config saved to /var/cache/conftool/dbconfig/20221102-143430-ladsgroup.json
  • 14:34 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 14:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2123 (T321123)', diff saved to https://phabricator.wikimedia.org/P37778 and previous config saved to /var/cache/conftool/dbconfig/20221102-143350-marostegui.json
  • 14:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 14:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 14:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T321123)', diff saved to https://phabricator.wikimedia.org/P37777 and previous config saved to /var/cache/conftool/dbconfig/20221102-143324-marostegui.json
  • 14:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P37776 and previous config saved to /var/cache/conftool/dbconfig/20221102-143150-ladsgroup.json
  • 14:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T318950)', diff saved to https://phabricator.wikimedia.org/P37775 and previous config saved to /var/cache/conftool/dbconfig/20221102-142818-ladsgroup.json
  • 14:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T318950)', diff saved to https://phabricator.wikimedia.org/P37774 and previous config saved to /var/cache/conftool/dbconfig/20221102-142605-ladsgroup.json
  • 14:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 14:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 14:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T318950)', diff saved to https://phabricator.wikimedia.org/P37773 and previous config saved to /var/cache/conftool/dbconfig/20221102-142540-ladsgroup.json
  • 14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T318950)', diff saved to https://phabricator.wikimedia.org/P37772 and previous config saved to /var/cache/conftool/dbconfig/20221102-142345-ladsgroup.json
  • 14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T318955)', diff saved to https://phabricator.wikimedia.org/P37771 and previous config saved to /var/cache/conftool/dbconfig/20221102-142331-ladsgroup.json
  • 14:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 14:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318605)', diff saved to https://phabricator.wikimedia.org/P37770 and previous config saved to /var/cache/conftool/dbconfig/20221102-142258-ladsgroup.json
  • 14:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37769 and previous config saved to /var/cache/conftool/dbconfig/20221102-141922-ladsgroup.json
  • 14:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P37768 and previous config saved to /var/cache/conftool/dbconfig/20221102-141815-marostegui.json
  • 14:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P37767 and previous config saved to /var/cache/conftool/dbconfig/20221102-141640-ladsgroup.json
  • 14:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P37766 and previous config saved to /var/cache/conftool/dbconfig/20221102-141029-ladsgroup.json
  • 14:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P37765 and previous config saved to /var/cache/conftool/dbconfig/20221102-140837-ladsgroup.json
  • 14:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P37764 and previous config saved to /var/cache/conftool/dbconfig/20221102-140822-ladsgroup.json
  • 14:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P37763 and previous config saved to /var/cache/conftool/dbconfig/20221102-140749-ladsgroup.json
  • 14:03 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 100%: After reboot', diff saved to https://phabricator.wikimedia.org/P37762 and previous config saved to /var/cache/conftool/dbconfig/20221102-140355-root.json
  • 14:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P37761 and previous config saved to /var/cache/conftool/dbconfig/20221102-140307-marostegui.json
  • 14:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T318605)', diff saved to https://phabricator.wikimedia.org/P37760 and previous config saved to /var/cache/conftool/dbconfig/20221102-140133-ladsgroup.json
  • 14:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37759 and previous config saved to /var/cache/conftool/dbconfig/20221102-135746-ladsgroup.json
  • 13:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T318955)', diff saved to https://phabricator.wikimedia.org/P37758 and previous config saved to /var/cache/conftool/dbconfig/20221102-135723-ladsgroup.json
  • 13:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P37757 and previous config saved to /var/cache/conftool/dbconfig/20221102-135521-ladsgroup.json
  • 13:54 vgutierrez: re-enabled puppet in A:cp - T321776
  • 13:54 moritzm: import puppetdb 7.11.2-1 to component/puppetdb7 for bookworm-wikimedia T321783
  • 13:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P37756 and previous config saved to /var/cache/conftool/dbconfig/20221102-135328-ladsgroup.json
  • 13:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P37755 and previous config saved to /var/cache/conftool/dbconfig/20221102-135315-ladsgroup.json
  • 13:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P37754 and previous config saved to /var/cache/conftool/dbconfig/20221102-135240-ladsgroup.json
  • 13:48 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 75%: After reboot', diff saved to https://phabricator.wikimedia.org/P37753 and previous config saved to /var/cache/conftool/dbconfig/20221102-134849-root.json
  • 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T321123)', diff saved to https://phabricator.wikimedia.org/P37752 and previous config saved to /var/cache/conftool/dbconfig/20221102-134758-marostegui.json
  • 13:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2111 (T321123)', diff saved to https://phabricator.wikimedia.org/P37751 and previous config saved to /var/cache/conftool/dbconfig/20221102-134527-marostegui.json
  • 13:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 13:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 13:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 13:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 13:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 13:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 13:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T321123)', diff saved to https://phabricator.wikimedia.org/P37750 and previous config saved to /var/cache/conftool/dbconfig/20221102-134404-marostegui.json
  • 13:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P37749 and previous config saved to /var/cache/conftool/dbconfig/20221102-134216-ladsgroup.json
  • 13:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T318950)', diff saved to https://phabricator.wikimedia.org/P37748 and previous config saved to /var/cache/conftool/dbconfig/20221102-134012-ladsgroup.json
  • 13:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T318950)', diff saved to https://phabricator.wikimedia.org/P37747 and previous config saved to /var/cache/conftool/dbconfig/20221102-133819-ladsgroup.json
  • 13:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T318955)', diff saved to https://phabricator.wikimedia.org/P37746 and previous config saved to /var/cache/conftool/dbconfig/20221102-133807-ladsgroup.json
  • 13:38 vgutierrez: uploaded trafficserver 9.1.3-1wm2 to apt.wm.o (buster-wikimedia) - T321776
  • 13:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318605)', diff saved to https://phabricator.wikimedia.org/P37745 and previous config saved to /var/cache/conftool/dbconfig/20221102-133733-ladsgroup.json
  • 13:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T318605)', diff saved to https://phabricator.wikimedia.org/P37744 and previous config saved to /var/cache/conftool/dbconfig/20221102-133637-ladsgroup.json
  • 13:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 13:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 13:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T318950)', diff saved to https://phabricator.wikimedia.org/P37743 and previous config saved to /var/cache/conftool/dbconfig/20221102-133559-ladsgroup.json
  • 13:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2175 (T318955)', diff saved to https://phabricator.wikimedia.org/P37742 and previous config saved to /var/cache/conftool/dbconfig/20221102-133549-ladsgroup.json
  • 13:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 13:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 13:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 13:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37741 and previous config saved to /var/cache/conftool/dbconfig/20221102-133533-ladsgroup.json
  • 13:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 13:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37740 and previous config saved to /var/cache/conftool/dbconfig/20221102-133526-ladsgroup.json
  • 13:35 vgutierrez: vgutierrez@apt1001:~$ sudo -i reprepro --delete clearvanished - T321776
  • 13:35 vgutierrez: vgutierrez@apt1001:~$ sudo -i reprepro clearvanished - T321776
  • 13:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T318950)', diff saved to https://phabricator.wikimedia.org/P37739 and previous config saved to /var/cache/conftool/dbconfig/20221102-133402-ladsgroup.json
  • 13:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 50%: After reboot', diff saved to https://phabricator.wikimedia.org/P37738 and previous config saved to /var/cache/conftool/dbconfig/20221102-133343-root.json
  • 13:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 13:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T318950)', diff saved to https://phabricator.wikimedia.org/P37737 and previous config saved to /var/cache/conftool/dbconfig/20221102-133338-ladsgroup.json
  • 13:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P37736 and previous config saved to /var/cache/conftool/dbconfig/20221102-132855-marostegui.json
  • 13:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P37735 and previous config saved to /var/cache/conftool/dbconfig/20221102-132707-ladsgroup.json
  • 13:22 vgutierrez: disable puppet on A:cp before merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/850087 - T321776
  • 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P37734 and previous config saved to /var/cache/conftool/dbconfig/20221102-132025-ladsgroup.json
  • 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P37733 and previous config saved to /var/cache/conftool/dbconfig/20221102-132017-ladsgroup.json
  • 13:18 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 25%: After reboot', diff saved to https://phabricator.wikimedia.org/P37732 and previous config saved to /var/cache/conftool/dbconfig/20221102-131837-root.json
  • 13:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P37731 and previous config saved to /var/cache/conftool/dbconfig/20221102-131830-ladsgroup.json
  • 13:16 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:15 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:15 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:14 Lucas_WMDE: UTC afternoon backport+config window done
  • 13:14 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P37730 and previous config saved to /var/cache/conftool/dbconfig/20221102-131348-marostegui.json
  • 13:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T318955)', diff saved to https://phabricator.wikimedia.org/P37729 and previous config saved to /var/cache/conftool/dbconfig/20221102-131159-ladsgroup.json
  • 13:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T318955)', diff saved to https://phabricator.wikimedia.org/P37728 and previous config saved to /var/cache/conftool/dbconfig/20221102-130948-ladsgroup.json
  • 13:10 cjming@deploy1002: Finished scap: Backport for testwiki: Add mediawiki.visual_editor_feature_use stream (T309602) (duration: 04m 56s)
  • 13:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 13:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T318955)', diff saved to https://phabricator.wikimedia.org/P37727 and previous config saved to /var/cache/conftool/dbconfig/20221102-130923-ladsgroup.json
  • 13:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P37726 and previous config saved to /var/cache/conftool/dbconfig/20221102-130518-ladsgroup.json
  • 13:05 cjming@deploy1002: cjming and cjming: Backport for testwiki: Add mediawiki.visual_editor_feature_use stream (T309602) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 13:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P37725 and previous config saved to /var/cache/conftool/dbconfig/20221102-130509-ladsgroup.json
  • 13:04 cjming@deploy1002: Started scap: Backport for testwiki: Add mediawiki.visual_editor_feature_use stream (T309602)
  • 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 10%: After reboot', diff saved to https://phabricator.wikimedia.org/P37724 and previous config saved to /var/cache/conftool/dbconfig/20221102-130331-root.json
  • 13:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P37723 and previous config saved to /var/cache/conftool/dbconfig/20221102-130322-ladsgroup.json
  • 12:59 moritzm: draining ganeti1025 for eventual reimage T311687
  • 12:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T321123)', diff saved to https://phabricator.wikimedia.org/P37722 and previous config saved to /var/cache/conftool/dbconfig/20221102-125840-marostegui.json
  • 12:56 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1200 (T321123)', diff saved to https://phabricator.wikimedia.org/P37721 and previous config saved to /var/cache/conftool/dbconfig/20221102-125607-marostegui.json
  • 12:56 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T321123)', diff saved to https://phabricator.wikimedia.org/P37720 and previous config saved to /var/cache/conftool/dbconfig/20221102-125544-marostegui.json
  • 12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P37719 and previous config saved to /var/cache/conftool/dbconfig/20221102-125415-ladsgroup.json
  • 12:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37718 and previous config saved to /var/cache/conftool/dbconfig/20221102-125009-ladsgroup.json
  • 12:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37717 and previous config saved to /var/cache/conftool/dbconfig/20221102-125001-ladsgroup.json
  • 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 5%: After reboot', diff saved to https://phabricator.wikimedia.org/P37716 and previous config saved to /var/cache/conftool/dbconfig/20221102-124824-root.json
  • 12:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T318950)', diff saved to https://phabricator.wikimedia.org/P37715 and previous config saved to /var/cache/conftool/dbconfig/20221102-124812-ladsgroup.json
  • 12:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37714 and previous config saved to /var/cache/conftool/dbconfig/20221102-124754-ladsgroup.json
  • 12:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37713 and previous config saved to /var/cache/conftool/dbconfig/20221102-124743-ladsgroup.json
  • 12:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 12:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37712 and previous config saved to /var/cache/conftool/dbconfig/20221102-124732-ladsgroup.json
  • 12:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T318955)', diff saved to https://phabricator.wikimedia.org/P37711 and previous config saved to /var/cache/conftool/dbconfig/20221102-124720-ladsgroup.json
  • 12:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T318950)', diff saved to https://phabricator.wikimedia.org/P37710 and previous config saved to /var/cache/conftool/dbconfig/20221102-124602-ladsgroup.json
  • 12:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 12:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 12:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T318950)', diff saved to https://phabricator.wikimedia.org/P37709 and previous config saved to /var/cache/conftool/dbconfig/20221102-124537-ladsgroup.json
  • 12:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P37708 and previous config saved to /var/cache/conftool/dbconfig/20221102-124037-marostegui.json
  • 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P37707 and previous config saved to /var/cache/conftool/dbconfig/20221102-123906-ladsgroup.json
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 3%: After reboot', diff saved to https://phabricator.wikimedia.org/P37706 and previous config saved to /var/cache/conftool/dbconfig/20221102-123319-root.json
  • 12:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P37705 and previous config saved to /var/cache/conftool/dbconfig/20221102-123224-ladsgroup.json
  • 12:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P37704 and previous config saved to /var/cache/conftool/dbconfig/20221102-123213-ladsgroup.json
  • 12:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P37701 and previous config saved to /var/cache/conftool/dbconfig/20221102-123029-ladsgroup.json
  • 12:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P37700 and previous config saved to /var/cache/conftool/dbconfig/20221102-122529-marostegui.json
  • 12:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T318955)', diff saved to https://phabricator.wikimedia.org/P37699 and previous config saved to /var/cache/conftool/dbconfig/20221102-122356-ladsgroup.json
  • 12:18 marostegui@cumin1001: dbctl commit (dc=all): 'es1020 (re)pooling @ 1%: After reboot', diff saved to https://phabricator.wikimedia.org/P37698 and previous config saved to /var/cache/conftool/dbconfig/20221102-121812-root.json
  • 12:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P37697 and previous config saved to /var/cache/conftool/dbconfig/20221102-121716-ladsgroup.json
  • 12:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P37696 and previous config saved to /var/cache/conftool/dbconfig/20221102-121704-ladsgroup.json
  • 12:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P37695 and previous config saved to /var/cache/conftool/dbconfig/20221102-121521-ladsgroup.json
  • 12:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 12:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 12:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 12:10 marostegui@deploy1002: Finished scap: Backport for Revert "db-production: Disable writes one es4" (duration: 04m 37s)
  • 12:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T321123)', diff saved to https://phabricator.wikimedia.org/P37694 and previous config saved to /var/cache/conftool/dbconfig/20221102-121020-marostegui.json
  • 12:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 12:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1185 (T321123)', diff saved to https://phabricator.wikimedia.org/P37693 and previous config saved to /var/cache/conftool/dbconfig/20221102-120805-marostegui.json
  • 12:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 12:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 12:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T321123)', diff saved to https://phabricator.wikimedia.org/P37692 and previous config saved to /var/cache/conftool/dbconfig/20221102-120742-marostegui.json
  • 12:08 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "db-production: Disable writes one es4" synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 12:06 marostegui@deploy1002: Started scap: Backport for Revert "db-production: Disable writes one es4"
  • 12:06 marostegui@deploy1002: Backport cancelled.
  • 12:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T318955)', diff saved to https://phabricator.wikimedia.org/P37691 and previous config saved to /var/cache/conftool/dbconfig/20221102-120505-ladsgroup.json
  • 12:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 12:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 12:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37690 and previous config saved to /var/cache/conftool/dbconfig/20221102-120436-ladsgroup.json
  • 12:04 marostegui@cumin1001: dbctl commit (dc=all): 'Add some weight to es4 master', diff saved to https://phabricator.wikimedia.org/P37689 and previous config saved to /var/cache/conftool/dbconfig/20221102-120233-marostegui.json
  • 12:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37688 and previous config saved to /var/cache/conftool/dbconfig/20221102-120209-ladsgroup.json
  • 12:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T318955)', diff saved to https://phabricator.wikimedia.org/P37687 and previous config saved to /var/cache/conftool/dbconfig/20221102-120157-ladsgroup.json
  • 12:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T318950)', diff saved to https://phabricator.wikimedia.org/P37686 and previous config saved to /var/cache/conftool/dbconfig/20221102-120013-ladsgroup.json
  • 12:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37685 and previous config saved to /var/cache/conftool/dbconfig/20221102-115948-ladsgroup.json
  • 12:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 12:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2148 (T318955)', diff saved to https://phabricator.wikimedia.org/P37684 and previous config saved to /var/cache/conftool/dbconfig/20221102-115940-ladsgroup.json
  • 12:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 11:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 11:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T318950)', diff saved to https://phabricator.wikimedia.org/P37683 and previous config saved to /var/cache/conftool/dbconfig/20221102-115925-ladsgroup.json
  • 11:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 11:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37682 and previous config saved to /var/cache/conftool/dbconfig/20221102-115855-ladsgroup.json
  • 11:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T318950)', diff saved to https://phabricator.wikimedia.org/P37681 and previous config saved to /var/cache/conftool/dbconfig/20221102-115802-ladsgroup.json
  • 11:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 11:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 11:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 11:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 11:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 11:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 11:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T318950)', diff saved to https://phabricator.wikimedia.org/P37680 and previous config saved to /var/cache/conftool/dbconfig/20221102-115705-ladsgroup.json
  • 11:54 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es1020', diff saved to https://phabricator.wikimedia.org/P37679 and previous config saved to /var/cache/conftool/dbconfig/20221102-115448-root.json
  • 11:53 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es1021 to es4 primary T322181', diff saved to https://phabricator.wikimedia.org/P37678 and previous config saved to /var/cache/conftool/dbconfig/20221102-115313-root.json
  • 11:52 marostegui: Starting es4 eqiad failover from es1020 to es1021 - T322181
  • 11:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P37677 and previous config saved to /var/cache/conftool/dbconfig/20221102-115233-marostegui.json
  • 11:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 11:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T318605)', diff saved to https://phabricator.wikimedia.org/P37676 and previous config saved to /var/cache/conftool/dbconfig/20221102-115023-ladsgroup.json
  • 11:49 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 11:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P37675 and previous config saved to /var/cache/conftool/dbconfig/20221102-114927-ladsgroup.json
  • 11:48 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 11:48 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 11:47 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 11:47 marostegui@deploy1002: Finished scap: Backport for db-production: Disable writes one es4 (T322181) (duration: 04m 43s)
  • 11:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P37674 and previous config saved to /var/cache/conftool/dbconfig/20221102-114416-ladsgroup.json
  • 11:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P37673 and previous config saved to /var/cache/conftool/dbconfig/20221102-114347-ladsgroup.json
  • 11:43 marostegui@deploy1002: marostegui and marostegui: Backport for db-production: Disable writes one es4 (T322181) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 11:42 marostegui@deploy1002: Started scap: Backport for db-production: Disable writes one es4 (T322181)
  • 11:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P37672 and previous config saved to /var/cache/conftool/dbconfig/20221102-114157-ladsgroup.json
  • 11:42 marostegui@cumin1001: dbctl commit (dc=all): 'Set es1021 with weight 0 T322181', diff saved to https://phabricator.wikimedia.org/P37671 and previous config saved to /var/cache/conftool/dbconfig/20221102-114107-root.json
  • 11:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es4 T322181
  • 11:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es4 T322181
  • 11:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P37670 and previous config saved to /var/cache/conftool/dbconfig/20221102-113726-marostegui.json
  • 11:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T318605)', diff saved to https://phabricator.wikimedia.org/P37669 and previous config saved to /var/cache/conftool/dbconfig/20221102-113542-ladsgroup.json
  • 11:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 11:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P37668 and previous config saved to /var/cache/conftool/dbconfig/20221102-113515-ladsgroup.json
  • 11:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 11:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318605)', diff saved to https://phabricator.wikimedia.org/P37667 and previous config saved to /var/cache/conftool/dbconfig/20221102-113506-ladsgroup.json
  • 11:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P37666 and previous config saved to /var/cache/conftool/dbconfig/20221102-113419-ladsgroup.json
  • 11:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P37665 and previous config saved to /var/cache/conftool/dbconfig/20221102-112909-ladsgroup.json
  • 11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P37664 and previous config saved to /var/cache/conftool/dbconfig/20221102-112839-ladsgroup.json
  • 11:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P37663 and previous config saved to /var/cache/conftool/dbconfig/20221102-112648-ladsgroup.json
  • 11:24 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
  • 11:24 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
  • 11:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T321123)', diff saved to https://phabricator.wikimedia.org/P37662 and previous config saved to /var/cache/conftool/dbconfig/20221102-112217-marostegui.json
  • 11:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P37661 and previous config saved to /var/cache/conftool/dbconfig/20221102-112008-ladsgroup.json
  • 11:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P37660 and previous config saved to /var/cache/conftool/dbconfig/20221102-111958-ladsgroup.json
  • 11:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37659 and previous config saved to /var/cache/conftool/dbconfig/20221102-111911-ladsgroup.json
  • 11:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T318950)', diff saved to https://phabricator.wikimedia.org/P37658 and previous config saved to /var/cache/conftool/dbconfig/20221102-111400-ladsgroup.json
  • 11:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37657 and previous config saved to /var/cache/conftool/dbconfig/20221102-111331-ladsgroup.json
  • 11:13 vgutierrez: pool cp1075, cp2027 and cp3050 running HAProxy 2.6.6 - T321775
  • 11:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T318950)', diff saved to https://phabricator.wikimedia.org/P37656 and previous config saved to /var/cache/conftool/dbconfig/20221102-111147-ladsgroup.json
  • 11:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T318950)', diff saved to https://phabricator.wikimedia.org/P37655 and previous config saved to /var/cache/conftool/dbconfig/20221102-111141-ladsgroup.json
  • 11:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 11:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 11:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 11:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 11:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 11:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37654 and previous config saved to /var/cache/conftool/dbconfig/20221102-111113-ladsgroup.json
  • 11:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 11:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 11:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T318950)', diff saved to https://phabricator.wikimedia.org/P37653 and previous config saved to /var/cache/conftool/dbconfig/20221102-111059-ladsgroup.json
  • 11:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 11:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T318955)', diff saved to https://phabricator.wikimedia.org/P37652 and previous config saved to /var/cache/conftool/dbconfig/20221102-111051-ladsgroup.json
  • 11:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T318950)', diff saved to https://phabricator.wikimedia.org/P37651 and previous config saved to /var/cache/conftool/dbconfig/20221102-110931-ladsgroup.json
  • 11:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 11:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 11:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37650 and previous config saved to /var/cache/conftool/dbconfig/20221102-110909-ladsgroup.json
  • 11:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P37649 and previous config saved to /var/cache/conftool/dbconfig/20221102-110451-ladsgroup.json
  • 11:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37648 and previous config saved to /var/cache/conftool/dbconfig/20221102-110314-ladsgroup.json
  • 11:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 11:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:57 vgutierrez: depool cp1075, cp2027 and cp3050 prior to HAProxy 2.6 upgrade - T321775
  • 10:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P37647 and previous config saved to /var/cache/conftool/dbconfig/20221102-105551-ladsgroup.json
  • 10:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P37646 and previous config saved to /var/cache/conftool/dbconfig/20221102-105544-ladsgroup.json
  • 10:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P37645 and previous config saved to /var/cache/conftool/dbconfig/20221102-105400-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318605)', diff saved to https://phabricator.wikimedia.org/P37644 and previous config saved to /var/cache/conftool/dbconfig/20221102-104942-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T318955)', diff saved to https://phabricator.wikimedia.org/P37643 and previous config saved to /var/cache/conftool/dbconfig/20221102-104932-ladsgroup.json
  • 10:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P37642 and previous config saved to /var/cache/conftool/dbconfig/20221102-104042-ladsgroup.json
  • 10:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P37641 and previous config saved to /var/cache/conftool/dbconfig/20221102-104034-ladsgroup.json
  • 10:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P37640 and previous config saved to /var/cache/conftool/dbconfig/20221102-103851-ladsgroup.json
  • 10:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T321123)', diff saved to https://phabricator.wikimedia.org/P37639 and previous config saved to /var/cache/conftool/dbconfig/20221102-103555-marostegui.json
  • 10:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 10:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 10:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37638 and previous config saved to /var/cache/conftool/dbconfig/20221102-103453-marostegui.json
  • 10:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P37637 and previous config saved to /var/cache/conftool/dbconfig/20221102-103424-ladsgroup.json
  • 10:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T318605)', diff saved to https://phabricator.wikimedia.org/P37636 and previous config saved to /var/cache/conftool/dbconfig/20221102-103400-ladsgroup.json
  • 10:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 10:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 10:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 10:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 10:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T318950)', diff saved to https://phabricator.wikimedia.org/P37635 and previous config saved to /var/cache/conftool/dbconfig/20221102-102533-ladsgroup.json
  • 10:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T318955)', diff saved to https://phabricator.wikimedia.org/P37634 and previous config saved to /var/cache/conftool/dbconfig/20221102-102527-ladsgroup.json
  • 10:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37633 and previous config saved to /var/cache/conftool/dbconfig/20221102-102342-ladsgroup.json
  • 10:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2129 (T318950)', diff saved to https://phabricator.wikimedia.org/P37632 and previous config saved to /var/cache/conftool/dbconfig/20221102-102320-ladsgroup.json
  • 10:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 10:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2126 (T318955)', diff saved to https://phabricator.wikimedia.org/P37631 and previous config saved to /var/cache/conftool/dbconfig/20221102-102310-ladsgroup.json
  • 10:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 10:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 10:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 10:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 10:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T318950)', diff saved to https://phabricator.wikimedia.org/P37630 and previous config saved to /var/cache/conftool/dbconfig/20221102-102256-ladsgroup.json
  • 10:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 10:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T318955)', diff saved to https://phabricator.wikimedia.org/P37629 and previous config saved to /var/cache/conftool/dbconfig/20221102-102243-ladsgroup.json
  • 10:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37628 and previous config saved to /var/cache/conftool/dbconfig/20221102-102233-ladsgroup.json
  • 10:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 10:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 10:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37627 and previous config saved to /var/cache/conftool/dbconfig/20221102-102221-ladsgroup.json
  • 10:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P37626 and previous config saved to /var/cache/conftool/dbconfig/20221102-101946-marostegui.json
  • 10:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P37625 and previous config saved to /var/cache/conftool/dbconfig/20221102-101916-ladsgroup.json
  • 10:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P37624 and previous config saved to /var/cache/conftool/dbconfig/20221102-100746-ladsgroup.json
  • 10:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P37623 and previous config saved to /var/cache/conftool/dbconfig/20221102-100736-ladsgroup.json
  • 10:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P37622 and previous config saved to /var/cache/conftool/dbconfig/20221102-100713-ladsgroup.json
  • 10:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P37621 and previous config saved to /var/cache/conftool/dbconfig/20221102-100438-marostegui.json
  • 10:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T318955)', diff saved to https://phabricator.wikimedia.org/P37620 and previous config saved to /var/cache/conftool/dbconfig/20221102-100408-ladsgroup.json
  • 10:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T318955)', diff saved to https://phabricator.wikimedia.org/P37619 and previous config saved to /var/cache/conftool/dbconfig/20221102-100156-ladsgroup.json
  • 10:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 10:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 10:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37618 and previous config saved to /var/cache/conftool/dbconfig/20221102-100133-ladsgroup.json
  • 09:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P37617 and previous config saved to /var/cache/conftool/dbconfig/20221102-095237-ladsgroup.json
  • 09:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P37616 and previous config saved to /var/cache/conftool/dbconfig/20221102-095225-ladsgroup.json
  • 09:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P37615 and previous config saved to /var/cache/conftool/dbconfig/20221102-095205-ladsgroup.json
  • 09:51 moritzm: installing exim4 security updates
  • 09:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37614 and previous config saved to /var/cache/conftool/dbconfig/20221102-094928-marostegui.json
  • 09:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37613 and previous config saved to /var/cache/conftool/dbconfig/20221102-094709-marostegui.json
  • 09:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 09:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 09:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37612 and previous config saved to /var/cache/conftool/dbconfig/20221102-094644-marostegui.json
  • 09:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P37611 and previous config saved to /var/cache/conftool/dbconfig/20221102-094622-ladsgroup.json
  • 09:41 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1016.eqiad.wmnet to cluster eqiad and group B
  • 09:40 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1016.eqiad.wmnet to cluster eqiad and group B
  • 09:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T318950)', diff saved to https://phabricator.wikimedia.org/P37610 and previous config saved to /var/cache/conftool/dbconfig/20221102-093730-ladsgroup.json
  • 09:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 09:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T318955)', diff saved to https://phabricator.wikimedia.org/P37609 and previous config saved to /var/cache/conftool/dbconfig/20221102-093717-ladsgroup.json
  • 09:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 09:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37608 and previous config saved to /var/cache/conftool/dbconfig/20221102-093657-ladsgroup.json
  • 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1016.eqiad.wmnet
  • 09:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T318950)', diff saved to https://phabricator.wikimedia.org/P37607 and previous config saved to /var/cache/conftool/dbconfig/20221102-093517-ladsgroup.json
  • 09:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance
  • 09:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 09:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance
  • 09:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T318955)', diff saved to https://phabricator.wikimedia.org/P37606 and previous config saved to /var/cache/conftool/dbconfig/20221102-093459-ladsgroup.json
  • 09:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 09:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T318950)', diff saved to https://phabricator.wikimedia.org/P37605 and previous config saved to /var/cache/conftool/dbconfig/20221102-093453-ladsgroup.json
  • 09:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 09:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37604 and previous config saved to /var/cache/conftool/dbconfig/20221102-093443-ladsgroup.json
  • 09:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 09:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T318955)', diff saved to https://phabricator.wikimedia.org/P37603 and previous config saved to /var/cache/conftool/dbconfig/20221102-093436-ladsgroup.json
  • 09:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37602 and previous config saved to /var/cache/conftool/dbconfig/20221102-093420-ladsgroup.json
  • 09:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P37601 and previous config saved to /var/cache/conftool/dbconfig/20221102-093135-marostegui.json
  • 09:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P37600 and previous config saved to /var/cache/conftool/dbconfig/20221102-093115-ladsgroup.json
  • 09:28 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1016.eqiad.wmnet
  • 09:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P37599 and previous config saved to /var/cache/conftool/dbconfig/20221102-091942-ladsgroup.json
  • 09:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P37598 and previous config saved to /var/cache/conftool/dbconfig/20221102-091928-ladsgroup.json
  • 09:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P37597 and previous config saved to /var/cache/conftool/dbconfig/20221102-091912-ladsgroup.json
  • 09:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P37596 and previous config saved to /var/cache/conftool/dbconfig/20221102-091628-marostegui.json
  • 09:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37595 and previous config saved to /var/cache/conftool/dbconfig/20221102-091606-ladsgroup.json
  • 09:06 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1028.eqiad.wmnet to cluster eqiad and group C
  • 09:05 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1028.eqiad.wmnet to cluster eqiad and group C
  • 09:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P37594 and previous config saved to /var/cache/conftool/dbconfig/20221102-090435-ladsgroup.json
  • 09:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P37593 and previous config saved to /var/cache/conftool/dbconfig/20221102-090418-ladsgroup.json
  • 09:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P37592 and previous config saved to /var/cache/conftool/dbconfig/20221102-090404-ladsgroup.json
  • 09:03 mvernon@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=swift,name=eqiad
  • 09:01 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet
  • 09:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37591 and previous config saved to /var/cache/conftool/dbconfig/20221102-090119-marostegui.json
  • 09:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T318955)', diff saved to https://phabricator.wikimedia.org/P37590 and previous config saved to /var/cache/conftool/dbconfig/20221102-090007-ladsgroup.json
  • 08:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 08:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 08:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37589 and previous config saved to /var/cache/conftool/dbconfig/20221102-085903-marostegui.json
  • 08:58 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 08:58 Emperor: repool ms-fe10-12
  • 08:58 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 08:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T321123)', diff saved to https://phabricator.wikimedia.org/P37588 and previous config saved to /var/cache/conftool/dbconfig/20221102-085838-marostegui.json
  • 08:55 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet
  • 08:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:54 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:54 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:53 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1016.eqiad.wmnet with OS bullseye
  • 08:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T318950)', diff saved to https://phabricator.wikimedia.org/P37587 and previous config saved to /var/cache/conftool/dbconfig/20221102-084927-ladsgroup.json
  • 08:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T318955)', diff saved to https://phabricator.wikimedia.org/P37586 and previous config saved to /var/cache/conftool/dbconfig/20221102-084910-ladsgroup.json
  • 08:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37585 and previous config saved to /var/cache/conftool/dbconfig/20221102-084853-ladsgroup.json
  • 08:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T318950)', diff saved to https://phabricator.wikimedia.org/P37584 and previous config saved to /var/cache/conftool/dbconfig/20221102-084713-ladsgroup.json
  • 08:51 marostegui@deploy1002: Finished scap: Backport for Revert "ProductionServices.php: Promote pc1014 to pc2 master" (duration: 04m 16s)
  • 08:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 08:50 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2104 (T318955)', diff saved to https://phabricator.wikimedia.org/P37583 and previous config saved to /var/cache/conftool/dbconfig/20221102-084653-ladsgroup.json
  • 08:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 08:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T318950)', diff saved to https://phabricator.wikimedia.org/P37582 and previous config saved to /var/cache/conftool/dbconfig/20221102-084643-ladsgroup.json
  • 08:49 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti1028.eqiad.wmnet
  • 08:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 08:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 08:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 08:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 08:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 08:47 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T318605)', diff saved to https://phabricator.wikimedia.org/P37581 and previous config saved to /var/cache/conftool/dbconfig/20221102-084540-ladsgroup.json
  • 08:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 08:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 08:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P37580 and previous config saved to /var/cache/conftool/dbconfig/20221102-084330-marostegui.json
  • 08:43 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "ProductionServices.php: Promote pc1014 to pc2 master" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 08:42 marostegui@deploy1002: Started scap: Backport for Revert "ProductionServices.php: Promote pc1014 to pc2 master"
  • 08:36 moritzm: draining ganeti1020 for eventual reimage T311687
  • 08:34 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet
  • 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1016.eqiad.wmnet with reason: host reimage
  • 08:33 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti1028.eqiad.wmnet
  • 08:30 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1016.eqiad.wmnet with reason: host reimage
  • 08:29 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 100%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37579 and previous config saved to /var/cache/conftool/dbconfig/20221102-082942-root.json
  • 08:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P37578 and previous config saved to /var/cache/conftool/dbconfig/20221102-082822-marostegui.json
  • 08:25 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:24 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:24 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:23 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet
  • 08:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:18 marostegui@deploy1002: Finished scap: Backport for ProductionServices.php: Promote pc1014 to pc2 master (duration: 04m 43s)
  • 08:18 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:15 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1016.eqiad.wmnet with OS bullseye
  • 08:14 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 75%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37577 and previous config saved to /var/cache/conftool/dbconfig/20221102-081437-root.json
  • 08:14 marostegui@deploy1002: marostegui and marostegui: Backport for ProductionServices.php: Promote pc1014 to pc2 master synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 08:13 marostegui@deploy1002: Started scap: Backport for ProductionServices.php: Promote pc1014 to pc2 master
  • 08:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T321123)', diff saved to https://phabricator.wikimedia.org/P37576 and previous config saved to /var/cache/conftool/dbconfig/20221102-081313-marostegui.json
  • 08:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T321123)', diff saved to https://phabricator.wikimedia.org/P37575 and previous config saved to /var/cache/conftool/dbconfig/20221102-081059-marostegui.json
  • 08:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1110.eqiad.wmnet with reason: Maintenance
  • 08:10 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1110.eqiad.wmnet with reason: Maintenance
  • 08:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100 (T321123)', diff saved to https://phabricator.wikimedia.org/P37574 and previous config saved to /var/cache/conftool/dbconfig/20221102-081034-marostegui.json
  • 08:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:05 marostegui@deploy1002: Finished scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc2 master" (duration: 05m 01s)
  • 08:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:03 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:01 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "ProductionServices.php: Promote pc2014 to pc2 master" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 08:00 marostegui@deploy1002: Started scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc2 master"
  • 07:59 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 50%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37573 and previous config saved to /var/cache/conftool/dbconfig/20221102-075930-root.json
  • 07:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P37572 and previous config saved to /var/cache/conftool/dbconfig/20221102-075527-marostegui.json
  • 07:48 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 07:47 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 07:47 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 07:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 07:44 marostegui@deploy1002: Finished scap: Backport for ProductionServices.php: Promote pc2014 to pc2 master (duration: 04m 13s)
  • 07:44 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 25%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37571 and previous config saved to /var/cache/conftool/dbconfig/20221102-074422-root.json
  • 07:40 marostegui@deploy1002: marostegui and marostegui: Backport for ProductionServices.php: Promote pc2014 to pc2 master synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 07:40 marostegui@deploy1002: Started scap: Backport for ProductionServices.php: Promote pc2014 to pc2 master
  • 07:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P37570 and previous config saved to /var/cache/conftool/dbconfig/20221102-074016-marostegui.json
  • 07:29 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 10%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37569 and previous config saved to /var/cache/conftool/dbconfig/20221102-072916-root.json
  • 07:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100 (T321123)', diff saved to https://phabricator.wikimedia.org/P37568 and previous config saved to /var/cache/conftool/dbconfig/20221102-072508-marostegui.json
  • 07:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1100 (T321123)', diff saved to https://phabricator.wikimedia.org/P37567 and previous config saved to /var/cache/conftool/dbconfig/20221102-072254-marostegui.json
  • 07:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 07:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 07:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37566 and previous config saved to /var/cache/conftool/dbconfig/20221102-072220-marostegui.json
  • 07:14 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 5%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37565 and previous config saved to /var/cache/conftool/dbconfig/20221102-071410-root.json
  • 07:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 07:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 07:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 07:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 07:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P37564 and previous config saved to /var/cache/conftool/dbconfig/20221102-070712-marostegui.json
  • 07:06 urbanecm@deploy1002: Finished scap: Backport for Deploy GrowthExperiments to 100% users at all wikis but dewiki (T320876) (duration: 04m 38s)
  • 07:02 urbanecm@deploy1002: urbanecm and urbanecm: Backport for Deploy GrowthExperiments to 100% users at all wikis but dewiki (T320876) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 07:01 urbanecm@deploy1002: Started scap: Backport for Deploy GrowthExperiments to 100% users at all wikis but dewiki (T320876)
  • 06:59 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 3%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37563 and previous config saved to /var/cache/conftool/dbconfig/20221102-065903-root.json
  • 06:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P37562 and previous config saved to /var/cache/conftool/dbconfig/20221102-065203-marostegui.json
  • 06:43 marostegui@cumin1001: dbctl commit (dc=all): 'db1143 (re)pooling @ 1%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37561 and previous config saved to /var/cache/conftool/dbconfig/20221102-064357-root.json
  • 06:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37560 and previous config saved to /var/cache/conftool/dbconfig/20221102-063653-marostegui.json
  • 06:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37559 and previous config saved to /var/cache/conftool/dbconfig/20221102-063038-marostegui.json
  • 06:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 06:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 05:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 05:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 05:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T318605)', diff saved to https://phabricator.wikimedia.org/P37558 and previous config saved to /var/cache/conftool/dbconfig/20221102-054747-ladsgroup.json
  • 05:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P37557 and previous config saved to /var/cache/conftool/dbconfig/20221102-053238-ladsgroup.json
  • 05:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P37556 and previous config saved to /var/cache/conftool/dbconfig/20221102-051730-ladsgroup.json
  • 05:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T318605)', diff saved to https://phabricator.wikimedia.org/P37555 and previous config saved to /var/cache/conftool/dbconfig/20221102-050222-ladsgroup.json
  • 04:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1203 (T318605)', diff saved to https://phabricator.wikimedia.org/P37554 and previous config saved to /var/cache/conftool/dbconfig/20221102-042904-ladsgroup.json
  • 04:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 04:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 04:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T318605)', diff saved to https://phabricator.wikimedia.org/P37553 and previous config saved to /var/cache/conftool/dbconfig/20221102-042838-ladsgroup.json
  • 04:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P37552 and previous config saved to /var/cache/conftool/dbconfig/20221102-041330-ladsgroup.json
  • 03:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P37551 and previous config saved to /var/cache/conftool/dbconfig/20221102-035821-ladsgroup.json
  • 03:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T318605)', diff saved to https://phabricator.wikimedia.org/P37550 and previous config saved to /var/cache/conftool/dbconfig/20221102-034312-ladsgroup.json
  • 03:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1193 (T318605)', diff saved to https://phabricator.wikimedia.org/P37549 and previous config saved to /var/cache/conftool/dbconfig/20221102-030959-ladsgroup.json
  • 03:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 03:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 03:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T318605)', diff saved to https://phabricator.wikimedia.org/P37548 and previous config saved to /var/cache/conftool/dbconfig/20221102-030934-ladsgroup.json
  • 02:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P37547 and previous config saved to /var/cache/conftool/dbconfig/20221102-025427-ladsgroup.json
  • 02:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P37546 and previous config saved to /var/cache/conftool/dbconfig/20221102-023919-ladsgroup.json
  • 02:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T318605)', diff saved to https://phabricator.wikimedia.org/P37545 and previous config saved to /var/cache/conftool/dbconfig/20221102-022408-ladsgroup.json
  • 01:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1192 (T318605)', diff saved to https://phabricator.wikimedia.org/P37544 and previous config saved to /var/cache/conftool/dbconfig/20221102-015019-ladsgroup.json
  • 01:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 01:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 01:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T318605)', diff saved to https://phabricator.wikimedia.org/P37543 and previous config saved to /var/cache/conftool/dbconfig/20221102-014955-ladsgroup.json
  • 01:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P37542 and previous config saved to /var/cache/conftool/dbconfig/20221102-013447-ladsgroup.json
  • 01:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P37541 and previous config saved to /var/cache/conftool/dbconfig/20221102-011937-ladsgroup.json
  • 01:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T318605)', diff saved to https://phabricator.wikimedia.org/P37540 and previous config saved to /var/cache/conftool/dbconfig/20221102-010958-ladsgroup.json
  • 01:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T318605)', diff saved to https://phabricator.wikimedia.org/P37539 and previous config saved to /var/cache/conftool/dbconfig/20221102-010430-ladsgroup.json
  • 00:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P37538 and previous config saved to /var/cache/conftool/dbconfig/20221102-005451-ladsgroup.json
  • 00:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P37537 and previous config saved to /var/cache/conftool/dbconfig/20221102-003941-ladsgroup.json
  • 00:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1178 (T318605)', diff saved to https://phabricator.wikimedia.org/P37536 and previous config saved to /var/cache/conftool/dbconfig/20221102-002936-ladsgroup.json
  • 00:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 00:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 00:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37535 and previous config saved to /var/cache/conftool/dbconfig/20221102-002913-ladsgroup.json
  • 00:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T318605)', diff saved to https://phabricator.wikimedia.org/P37534 and previous config saved to /var/cache/conftool/dbconfig/20221102-002433-ladsgroup.json
  • 00:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P37533 and previous config saved to /var/cache/conftool/dbconfig/20221102-001401-ladsgroup.json

2022-11-01

  • 23:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P37532 and previous config saved to /var/cache/conftool/dbconfig/20221101-235853-ladsgroup.json
  • 23:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2181 (T318605)', diff saved to https://phabricator.wikimedia.org/P37531 and previous config saved to /var/cache/conftool/dbconfig/20221101-234957-ladsgroup.json
  • 23:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
  • 23:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
  • 23:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37530 and previous config saved to /var/cache/conftool/dbconfig/20221101-234935-ladsgroup.json
  • 23:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37529 and previous config saved to /var/cache/conftool/dbconfig/20221101-234346-ladsgroup.json
  • 23:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3318', diff saved to https://phabricator.wikimedia.org/P37528 and previous config saved to /var/cache/conftool/dbconfig/20221101-233427-ladsgroup.json
  • 23:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3318', diff saved to https://phabricator.wikimedia.org/P37527 and previous config saved to /var/cache/conftool/dbconfig/20221101-231919-ladsgroup.json
  • 23:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37526 and previous config saved to /var/cache/conftool/dbconfig/20221101-230833-ladsgroup.json
  • 23:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 23:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 23:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T318605)', diff saved to https://phabricator.wikimedia.org/P37525 and previous config saved to /var/cache/conftool/dbconfig/20221101-230811-ladsgroup.json
  • 23:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37524 and previous config saved to /var/cache/conftool/dbconfig/20221101-230411-ladsgroup.json
  • 22:55 Emperor: depool ms-fe2009
  • 22:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P37523 and previous config saved to /var/cache/conftool/dbconfig/20221101-225303-ladsgroup.json
  • 22:43 krinkle@deploy1002: Finished deploy [integration/docroot@2ddd7d9]: (no justification provided) (duration: 00m 33s)
  • 22:43 krinkle@deploy1002: Started deploy [integration/docroot@2ddd7d9]: (no justification provided)
  • 22:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P37522 and previous config saved to /var/cache/conftool/dbconfig/20221101-223754-ladsgroup.json
  • 22:32 jhathaway@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host aux-k8s-worker1002.eqiad.wmnet
  • 22:32 Emperor: rolling restart of eqiad swift front-ends
  • 22:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2168:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37521 and previous config saved to /var/cache/conftool/dbconfig/20221101-222858-ladsgroup.json
  • 22:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 22:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 22:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37520 and previous config saved to /var/cache/conftool/dbconfig/20221101-222835-ladsgroup.json
  • 22:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T318605)', diff saved to https://phabricator.wikimedia.org/P37519 and previous config saved to /var/cache/conftool/dbconfig/20221101-222247-ladsgroup.json
  • 22:20 jhathaway@cumin1001: conftool action : set/pooled=false; selector: dnsdisc=swift,name=eqiad
  • 22:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3318', diff saved to https://phabricator.wikimedia.org/P37518 and previous config saved to /var/cache/conftool/dbconfig/20221101-221328-ladsgroup.json
  • 22:09 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aux-k8s-worker1002.eqiad.wmnet on all recursors
  • 22:09 jhathaway@cumin1001: START - Cookbook sre.dns.wipe-cache aux-k8s-worker1002.eqiad.wmnet on all recursors
  • 22:08 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3318', diff saved to https://phabricator.wikimedia.org/P37517 and previous config saved to /var/cache/conftool/dbconfig/20221101-215820-ladsgroup.json
  • 21:53 jhathaway@cumin1001: START - Cookbook sre.dns.netbox
  • 21:53 jhathaway@cumin1001: START - Cookbook sre.ganeti.makevm for new host aux-k8s-worker1002.eqiad.wmnet
  • 21:52 jhathaway@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host aux-k8s-worker1001.eqiad.wmnet
  • 21:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1172 (T318605)', diff saved to https://phabricator.wikimedia.org/P37516 and previous config saved to /var/cache/conftool/dbconfig/20221101-214659-ladsgroup.json
  • 21:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 21:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37515 and previous config saved to /var/cache/conftool/dbconfig/20221101-214311-ladsgroup.json
  • 21:29 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aux-k8s-worker1001.eqiad.wmnet on all recursors
  • 21:29 jhathaway@cumin1001: START - Cookbook sre.dns.wipe-cache aux-k8s-worker1001.eqiad.wmnet on all recursors
  • 21:28 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:21 jhathaway@cumin1001: START - Cookbook sre.dns.netbox
  • 21:20 jhathaway@cumin1001: START - Cookbook sre.ganeti.makevm for new host aux-k8s-worker1001.eqiad.wmnet
  • 21:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 21:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 21:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T318605)', diff saved to https://phabricator.wikimedia.org/P37514 and previous config saved to /var/cache/conftool/dbconfig/20221101-211013-ladsgroup.json
  • 21:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2167:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37513 and previous config saved to /var/cache/conftool/dbconfig/20221101-210658-ladsgroup.json
  • 21:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 21:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 21:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T318605)', diff saved to https://phabricator.wikimedia.org/P37512 and previous config saved to /var/cache/conftool/dbconfig/20221101-210622-ladsgroup.json
  • 20:56 ryankemper: T322037 Re-enabled puppet across `A:wdqs-all` and `A:wcqs-public`
  • 20:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P37511 and previous config saved to /var/cache/conftool/dbconfig/20221101-205505-ladsgroup.json
  • 20:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P37510 and previous config saved to /var/cache/conftool/dbconfig/20221101-205115-ladsgroup.json
  • 20:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P37509 and previous config saved to /var/cache/conftool/dbconfig/20221101-203957-ladsgroup.json
  • 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P37508 and previous config saved to /var/cache/conftool/dbconfig/20221101-203607-ladsgroup.json
  • 20:34 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:34 cjming: end of UTC late backport window
  • 20:33 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:33 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:33 cjming@deploy1002: Finished scap: Backport for Update Edit Attempt Step sampling rate to 1 for group 0 wikis (T312016) (duration: 04m 29s)
  • 20:32 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:29 cjming@deploy1002: cjming and cjming: Backport for Update Edit Attempt Step sampling rate to 1 for group 0 wikis (T312016) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 20:29 cjming@deploy1002: Started scap: Backport for Update Edit Attempt Step sampling rate to 1 for group 0 wikis (T312016)
  • 20:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:28 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:26 cjming@deploy1002: Finished scap: Backport for Add MP stream for VisualEditorFeatureUse instrument (T309602) (duration: 04m 36s)
  • 20:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T318605)', diff saved to https://phabricator.wikimedia.org/P37507 and previous config saved to /var/cache/conftool/dbconfig/20221101-202449-ladsgroup.json
  • 20:21 cjming@deploy1002: cjming and cjming: Backport for Add MP stream for VisualEditorFeatureUse instrument (T309602) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 20:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T318605)', diff saved to https://phabricator.wikimedia.org/P37506 and previous config saved to /var/cache/conftool/dbconfig/20221101-202059-ladsgroup.json
  • 20:21 cjming@deploy1002: Started scap: Backport for Add MP stream for VisualEditorFeatureUse instrument (T309602)
  • 20:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:18 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:17 cjming@deploy1002: Finished scap: Backport for Enable DiscussionTools visual enhancements beta feature at jawiki (T318127) (duration: 04m 55s)
  • 20:16 jhathaway: restarting pybal on lvs1019.eqiad.net
  • 20:12 cjming@deploy1002: cjming and matmarex: Backport for Enable DiscussionTools visual enhancements beta feature at jawiki (T318127) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 20:12 cjming@deploy1002: Started scap: Backport for Enable DiscussionTools visual enhancements beta feature at jawiki (T318127)
  • 20:10 jhathaway: restarting pybal on lvs1020.eqiad.net
  • 20:10 ejegg: civicrm upgraded from 6f511710 to 97b9d830
  • 20:09 cjming@deploy1002: Finished scap: Backport for Enable DiscussionTools mobile visual enhancements at jawiki (T318870) (duration: 05m 31s)
  • 20:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:06 ryankemper: T322037 Disabled puppet across `A:wdqs-all` and `A:wcqs-public`
  • 19:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2166 (T318605)', diff saved to https://phabricator.wikimedia.org/P37505 and previous config saved to /var/cache/conftool/dbconfig/20221101-194718-ladsgroup.json
  • 19:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
  • 19:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
  • 19:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T318605)', diff saved to https://phabricator.wikimedia.org/P37504 and previous config saved to /var/cache/conftool/dbconfig/20221101-194655-ladsgroup.json
  • 19:41 jhathaway@puppetmaster1001: conftool action : set/pooled=yes:weight=1; selector: cluster=aux-k8s,service=kubemaster
  • 19:36 otto@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Enable rc0.mediawiki.page_change on group0 wikis - T311129 (duration: 03m 38s)
  • 19:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1167 (T318605)', diff saved to https://phabricator.wikimedia.org/P37503 and previous config saved to /var/cache/conftool/dbconfig/20221101-193404-ladsgroup.json
  • 19:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 19:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 19:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 19:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1126 (T318605)', diff saved to https://phabricator.wikimedia.org/P37502 and previous config saved to /var/cache/conftool/dbconfig/20221101-193323-ladsgroup.json
  • 19:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P37501 and previous config saved to /var/cache/conftool/dbconfig/20221101-193148-ladsgroup.json
  • 19:26 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 19:25 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 19:25 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 19:24 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 19:19 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 19:19 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 19:19 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1126', diff saved to https://phabricator.wikimedia.org/P37500 and previous config saved to /var/cache/conftool/dbconfig/20221101-191815-ladsgroup.json
  • 19:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P37499 and previous config saved to /var/cache/conftool/dbconfig/20221101-191639-ladsgroup.json
  • 19:15 otto@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Declare rc0.mediawiki.page_content_change stream - T307959 T308017 (duration: 03m 42s)
  • 19:14 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1126', diff saved to https://phabricator.wikimedia.org/P37498 and previous config saved to /var/cache/conftool/dbconfig/20221101-190307-ladsgroup.json
  • 19:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T318605)', diff saved to https://phabricator.wikimedia.org/P37497 and previous config saved to /var/cache/conftool/dbconfig/20221101-190132-ladsgroup.json
  • 18:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1126 (T318605)', diff saved to https://phabricator.wikimedia.org/P37496 and previous config saved to /var/cache/conftool/dbconfig/20221101-184758-ladsgroup.json
  • 18:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 18:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 18:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T318955)', diff saved to https://phabricator.wikimedia.org/P37495 and previous config saved to /var/cache/conftool/dbconfig/20221101-183920-ladsgroup.json
  • 18:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2164 (T318605)', diff saved to https://phabricator.wikimedia.org/P37494 and previous config saved to /var/cache/conftool/dbconfig/20221101-182734-ladsgroup.json
  • 18:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 18:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 18:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
  • 18:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
  • 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T318605)', diff saved to https://phabricator.wikimedia.org/P37493 and previous config saved to /var/cache/conftool/dbconfig/20221101-182655-ladsgroup.json
  • 18:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 18:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 18:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T318950)', diff saved to https://phabricator.wikimedia.org/P37492 and previous config saved to /var/cache/conftool/dbconfig/20221101-182421-ladsgroup.json
  • 18:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P37491 and previous config saved to /var/cache/conftool/dbconfig/20221101-182412-ladsgroup.json
  • 18:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:24 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:22 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=varnish-fe
  • 18:22 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-be
  • 18:22 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-tls
  • 18:21 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:18 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.8 refs T320513
  • 18:14 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4052.ulsfo.wmnet
  • 18:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1126 (T318605)', diff saved to https://phabricator.wikimedia.org/P37490 and previous config saved to /var/cache/conftool/dbconfig/20221101-181310-ladsgroup.json
  • 18:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1126.eqiad.wmnet with reason: Maintenance
  • 18:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1126.eqiad.wmnet with reason: Maintenance
  • 18:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P37489 and previous config saved to /var/cache/conftool/dbconfig/20221101-181148-ladsgroup.json
  • 18:11 jhuneidi@deploy1002: Finished scap: testwikis wikis to 1.40.0-wmf.8 refs T320513 (duration: 04m 18s)
  • 18:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P37488 and previous config saved to /var/cache/conftool/dbconfig/20221101-180913-ladsgroup.json
  • 18:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P37487 and previous config saved to /var/cache/conftool/dbconfig/20221101-180902-ladsgroup.json
  • 18:07 jhuneidi@deploy1002: Started scap: testwikis wikis to 1.40.0-wmf.8 refs T320513
  • 18:05 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4052.ulsfo.wmnet
  • 18:03 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4051.ulsfo.wmnet
  • 17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P37486 and previous config saved to /var/cache/conftool/dbconfig/20221101-175639-ladsgroup.json
  • 17:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P37479 and previous config saved to /var/cache/conftool/dbconfig/20221101-173712-ladsgroup.json
  • 17:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1197 (T318950)', diff saved to https://phabricator.wikimedia.org/P37478 and previous config saved to /var/cache/conftool/dbconfig/20221101-173636-ladsgroup.json
  • 17:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 17:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 17:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T318950)', diff saved to https://phabricator.wikimedia.org/P37477 and previous config saved to /var/cache/conftool/dbconfig/20221101-173624-ladsgroup.json
  • 17:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T318955)', diff saved to https://phabricator.wikimedia.org/P37476 and previous config saved to /var/cache/conftool/dbconfig/20221101-173607-ladsgroup.json
  • 17:35 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4049.ulsfo.wmnet
  • 17:34 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4048.ulsfo.wmnet
  • 17:24 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4048.ulsfo.wmnet
  • 17:24 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4047.ulsfo.wmnet
  • 17:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1114', diff saved to https://phabricator.wikimedia.org/P37475 and previous config saved to /var/cache/conftool/dbconfig/20221101-172341-ladsgroup.json
  • 17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P37474 and previous config saved to /var/cache/conftool/dbconfig/20221101-172204-ladsgroup.json
  • 17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P37473 and previous config saved to /var/cache/conftool/dbconfig/20221101-172116-ladsgroup.json
  • 17:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P37472 and previous config saved to /var/cache/conftool/dbconfig/20221101-172058-ladsgroup.json
  • 17:14 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4047.ulsfo.wmnet
  • 17:14 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4046.ulsfo.wmnet
  • 17:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1114', diff saved to https://phabricator.wikimedia.org/P37471 and previous config saved to /var/cache/conftool/dbconfig/20221101-170832-ladsgroup.json
  • 17:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2163 (T318605)', diff saved to https://phabricator.wikimedia.org/P37470 and previous config saved to /var/cache/conftool/dbconfig/20221101-170752-ladsgroup.json
  • 17:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
  • 17:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
  • 17:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37469 and previous config saved to /var/cache/conftool/dbconfig/20221101-170730-ladsgroup.json
  • 17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T318955)', diff saved to https://phabricator.wikimedia.org/P37468 and previous config saved to /var/cache/conftool/dbconfig/20221101-170656-ladsgroup.json
  • 17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P37467 and previous config saved to /var/cache/conftool/dbconfig/20221101-170608-ladsgroup.json
  • 17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P37466 and previous config saved to /var/cache/conftool/dbconfig/20221101-170550-ladsgroup.json
  • 17:06 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4046.ulsfo.wmnet
  • 17:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T318955)', diff saved to https://phabricator.wikimedia.org/P37465 and previous config saved to /var/cache/conftool/dbconfig/20221101-170447-ladsgroup.json
  • 17:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 17:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 17:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T318955)', diff saved to https://phabricator.wikimedia.org/P37464 and previous config saved to /var/cache/conftool/dbconfig/20221101-170424-ladsgroup.json
  • 16:58 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4045.ulsfo.wmnet
  • 16:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1114 (T318605)', diff saved to https://phabricator.wikimedia.org/P37463 and previous config saved to /var/cache/conftool/dbconfig/20221101-165323-ladsgroup.json
  • 16:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P37462 and previous config saved to /var/cache/conftool/dbconfig/20221101-165221-ladsgroup.json
  • 16:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T318950)', diff saved to https://phabricator.wikimedia.org/P37461 and previous config saved to /var/cache/conftool/dbconfig/20221101-165100-ladsgroup.json
  • 16:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T318955)', diff saved to https://phabricator.wikimedia.org/P37460 and previous config saved to /var/cache/conftool/dbconfig/20221101-165042-ladsgroup.json
  • 16:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T318955)', diff saved to https://phabricator.wikimedia.org/P37459 and previous config saved to /var/cache/conftool/dbconfig/20221101-164930-ladsgroup.json
  • 16:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P37458 and previous config saved to /var/cache/conftool/dbconfig/20221101-164914-ladsgroup.json
  • 16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37457 and previous config saved to /var/cache/conftool/dbconfig/20221101-164907-ladsgroup.json
  • 16:50 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4045.ulsfo.wmnet
  • 16:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1188 (T318950)', diff saved to https://phabricator.wikimedia.org/P37456 and previous config saved to /var/cache/conftool/dbconfig/20221101-164845-ladsgroup.json
  • 16:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T318950)', diff saved to https://phabricator.wikimedia.org/P37455 and previous config saved to /var/cache/conftool/dbconfig/20221101-164832-ladsgroup.json
  • 16:42 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:41 jhathaway@cumin1001: START - Cookbook sre.dns.netbox
  • 16:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P37454 and previous config saved to /var/cache/conftool/dbconfig/20221101-163713-ladsgroup.json
  • 16:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T318950)', diff saved to https://phabricator.wikimedia.org/P37453 and previous config saved to /var/cache/conftool/dbconfig/20221101-163706-ladsgroup.json
  • 16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P37452 and previous config saved to /var/cache/conftool/dbconfig/20221101-163407-ladsgroup.json
  • 16:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P37451 and previous config saved to /var/cache/conftool/dbconfig/20221101-163358-ladsgroup.json
  • 16:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P37450 and previous config saved to /var/cache/conftool/dbconfig/20221101-163324-ladsgroup.json
  • 16:27 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:25 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 16:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37449 and previous config saved to /var/cache/conftool/dbconfig/20221101-162206-ladsgroup.json
  • 16:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P37448 and previous config saved to /var/cache/conftool/dbconfig/20221101-162158-ladsgroup.json
  • 16:21 jhathaway@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:19 jhathaway@cumin1001: START - Cookbook sre.dns.netbox
  • 16:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T318955)', diff saved to https://phabricator.wikimedia.org/P37447 and previous config saved to /var/cache/conftool/dbconfig/20221101-161859-ladsgroup.json
  • 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P37446 and previous config saved to /var/cache/conftool/dbconfig/20221101-161851-ladsgroup.json
  • 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P37445 and previous config saved to /var/cache/conftool/dbconfig/20221101-161816-ladsgroup.json
  • 16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T318955)', diff saved to https://phabricator.wikimedia.org/P37444 and previous config saved to /var/cache/conftool/dbconfig/20221101-161648-ladsgroup.json
  • 16:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1114 (T318605)', diff saved to https://phabricator.wikimedia.org/P37443 and previous config saved to /var/cache/conftool/dbconfig/20221101-161636-ladsgroup.json
  • 16:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1114.eqiad.wmnet with reason: Maintenance
  • 16:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T318955)', diff saved to https://phabricator.wikimedia.org/P37442 and previous config saved to /var/cache/conftool/dbconfig/20221101-161625-ladsgroup.json
  • 16:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1114.eqiad.wmnet with reason: Maintenance
  • 16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1111 (T318605)', diff saved to https://phabricator.wikimedia.org/P37441 and previous config saved to /var/cache/conftool/dbconfig/20221101-161614-ladsgroup.json
  • 16:10 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4044.ulsfo.wmnet
  • 16:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P37440 and previous config saved to /var/cache/conftool/dbconfig/20221101-160649-ladsgroup.json
  • 16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37439 and previous config saved to /var/cache/conftool/dbconfig/20221101-160344-ladsgroup.json
  • 16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T318950)', diff saved to https://phabricator.wikimedia.org/P37438 and previous config saved to /var/cache/conftool/dbconfig/20221101-160308-ladsgroup.json
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P37437 and previous config saved to /var/cache/conftool/dbconfig/20221101-160116-ladsgroup.json
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1111', diff saved to https://phabricator.wikimedia.org/P37436 and previous config saved to /var/cache/conftool/dbconfig/20221101-160106-ladsgroup.json
  • 16:00 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4044.ulsfo.wmnet
  • 15:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T318950)', diff saved to https://phabricator.wikimedia.org/P37435 and previous config saved to /var/cache/conftool/dbconfig/20221101-155458-ladsgroup.json
  • 15:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37434 and previous config saved to /var/cache/conftool/dbconfig/20221101-155446-ladsgroup.json
  • 15:53 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4043.ulsfo.wmnet
  • 15:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T318950)', diff saved to https://phabricator.wikimedia.org/P37433 and previous config saved to /var/cache/conftool/dbconfig/20221101-155142-ladsgroup.json
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37432 and previous config saved to /var/cache/conftool/dbconfig/20221101-155002-ladsgroup.json
  • 15:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 15:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 15:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37431 and previous config saved to /var/cache/conftool/dbconfig/20221101-154938-ladsgroup.json
  • 15:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2175 (T318950)', diff saved to https://phabricator.wikimedia.org/P37430 and previous config saved to /var/cache/conftool/dbconfig/20221101-154919-ladsgroup.json
  • 15:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 15:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 15:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37429 and previous config saved to /var/cache/conftool/dbconfig/20221101-154907-ladsgroup.json
  • 15:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37428 and previous config saved to /var/cache/conftool/dbconfig/20221101-154844-ladsgroup.json
  • 15:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance
  • 15:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance
  • 15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T318605)', diff saved to https://phabricator.wikimedia.org/P37427 and previous config saved to /var/cache/conftool/dbconfig/20221101-154819-ladsgroup.json
  • 15:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P37426 and previous config saved to /var/cache/conftool/dbconfig/20221101-154607-ladsgroup.json
  • 15:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1111', diff saved to https://phabricator.wikimedia.org/P37425 and previous config saved to /var/cache/conftool/dbconfig/20221101-154557-ladsgroup.json
  • 15:42 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4043.ulsfo.wmnet
  • 15:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P37424 and previous config saved to /var/cache/conftool/dbconfig/20221101-153938-ladsgroup.json
  • 15:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P37423 and previous config saved to /var/cache/conftool/dbconfig/20221101-153430-ladsgroup.json
  • 15:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P37422 and previous config saved to /var/cache/conftool/dbconfig/20221101-153400-ladsgroup.json
  • 15:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P37421 and previous config saved to /var/cache/conftool/dbconfig/20221101-153311-ladsgroup.json
  • 15:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T318955)', diff saved to https://phabricator.wikimedia.org/P37420 and previous config saved to /var/cache/conftool/dbconfig/20221101-153059-ladsgroup.json
  • 15:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1111 (T318605)', diff saved to https://phabricator.wikimedia.org/P37419 and previous config saved to /var/cache/conftool/dbconfig/20221101-153049-ladsgroup.json
  • 15:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T318955)', diff saved to https://phabricator.wikimedia.org/P37418 and previous config saved to /var/cache/conftool/dbconfig/20221101-152850-ladsgroup.json
  • 15:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 15:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 15:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T318955)', diff saved to https://phabricator.wikimedia.org/P37417 and previous config saved to /var/cache/conftool/dbconfig/20221101-152827-ladsgroup.json
  • 15:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P37416 and previous config saved to /var/cache/conftool/dbconfig/20221101-152430-ladsgroup.json
  • 15:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P37415 and previous config saved to /var/cache/conftool/dbconfig/20221101-151923-ladsgroup.json
  • 15:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P37414 and previous config saved to /var/cache/conftool/dbconfig/20221101-151853-ladsgroup.json
  • 15:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P37413 and previous config saved to /var/cache/conftool/dbconfig/20221101-151803-ladsgroup.json
  • 15:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 15:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 15:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 15:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P37412 and previous config saved to /var/cache/conftool/dbconfig/20221101-151320-ladsgroup.json
  • 15:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 15:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37411 and previous config saved to /var/cache/conftool/dbconfig/20221101-150922-ladsgroup.json
  • 15:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37410 and previous config saved to /var/cache/conftool/dbconfig/20221101-150711-ladsgroup.json
  • 15:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 15:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 15:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T318950)', diff saved to https://phabricator.wikimedia.org/P37409 and previous config saved to /var/cache/conftool/dbconfig/20221101-150659-ladsgroup.json
  • 15:06 dancy@deploy1002: Pruned MediaWiki: 1.40.0-wmf.6 (duration: 01m 47s)
  • 15:04 dancy@deploy1002: Finished scap: testwikis wikis to 1.40.0-wmf.8 refs T320513 (duration: 05m 05s)
  • 15:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37408 and previous config saved to /var/cache/conftool/dbconfig/20221101-150415-ladsgroup.json
  • 15:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37407 and previous config saved to /var/cache/conftool/dbconfig/20221101-150345-ladsgroup.json
  • 15:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T318605)', diff saved to https://phabricator.wikimedia.org/P37406 and previous config saved to /var/cache/conftool/dbconfig/20221101-150255-ladsgroup.json
  • 15:02 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4042.ulsfo.wmnet
  • 15:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37405 and previous config saved to /var/cache/conftool/dbconfig/20221101-150122-ladsgroup.json
  • 15:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 15:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 15:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T318950)', diff saved to https://phabricator.wikimedia.org/P37404 and previous config saved to /var/cache/conftool/dbconfig/20221101-150107-ladsgroup.json
  • 14:59 dancy@deploy1002: Started scap: testwikis wikis to 1.40.0-wmf.8 refs T320513
  • 14:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P37403 and previous config saved to /var/cache/conftool/dbconfig/20221101-145813-ladsgroup.json
  • 14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P37402 and previous config saved to /var/cache/conftool/dbconfig/20221101-145152-ladsgroup.json
  • 14:54 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4042.ulsfo.wmnet
  • 14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1111 (T318605)', diff saved to https://phabricator.wikimedia.org/P37401 and previous config saved to /var/cache/conftool/dbconfig/20221101-145026-ladsgroup.json
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37400 and previous config saved to /var/cache/conftool/dbconfig/20221101-145019-ladsgroup.json
  • 14:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1111.eqiad.wmnet with reason: Maintenance
  • 14:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 14:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1111.eqiad.wmnet with reason: Maintenance
  • 14:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1104 (T318605)', diff saved to https://phabricator.wikimedia.org/P37399 and previous config saved to /var/cache/conftool/dbconfig/20221101-145004-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T318955)', diff saved to https://phabricator.wikimedia.org/P37398 and previous config saved to /var/cache/conftool/dbconfig/20221101-144954-ladsgroup.json
  • 14:48 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4041.ulsfo.wmnet
  • 14:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P37397 and previous config saved to /var/cache/conftool/dbconfig/20221101-144559-ladsgroup.json
  • 14:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T318955)', diff saved to https://phabricator.wikimedia.org/P37396 and previous config saved to /var/cache/conftool/dbconfig/20221101-144302-ladsgroup.json
  • 14:40 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4041.ulsfo.wmnet
  • 14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P37394 and previous config saved to /var/cache/conftool/dbconfig/20221101-143645-ladsgroup.json
  • 14:38 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging EJoseph out of all services on: 803 hosts
  • 14:38 jmm@cumin2002: START - Cookbook sre.idm.logout Logging EJoseph out of all services on: 803 hosts
  • 14:38 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging EJoseph out of all services on: 1202 hosts
  • 14:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1104', diff saved to https://phabricator.wikimedia.org/P37393 and previous config saved to /var/cache/conftool/dbconfig/20221101-143453-ladsgroup.json
  • 14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P37392 and previous config saved to /var/cache/conftool/dbconfig/20221101-143445-ladsgroup.json
  • 14:34 jmm@cumin2002: START - Cookbook sre.idm.logout Logging EJoseph out of all services on: 1202 hosts
  • 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1028.eqiad.wmnet with OS bullseye
  • 14:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P37391 and previous config saved to /var/cache/conftool/dbconfig/20221101-143051-ladsgroup.json
  • 14:31 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4040.ulsfo.wmnet
  • 14:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 14:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2161 (T318605)', diff saved to https://phabricator.wikimedia.org/P37390 and previous config saved to /var/cache/conftool/dbconfig/20221101-142854-ladsgroup.json
  • 14:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance
  • 14:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 14:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T318955)', diff saved to https://phabricator.wikimedia.org/P37389 and previous config saved to /var/cache/conftool/dbconfig/20221101-142842-ladsgroup.json
  • 14:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance
  • 14:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T318605)', diff saved to https://phabricator.wikimedia.org/P37388 and previous config saved to /var/cache/conftool/dbconfig/20221101-142832-ladsgroup.json
  • 14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T318950)', diff saved to https://phabricator.wikimedia.org/P37387 and previous config saved to /var/cache/conftool/dbconfig/20221101-142136-ladsgroup.json
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1104', diff saved to https://phabricator.wikimedia.org/P37386 and previous config saved to /var/cache/conftool/dbconfig/20221101-141945-ladsgroup.json
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P37385 and previous config saved to /var/cache/conftool/dbconfig/20221101-141936-ladsgroup.json
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T318950)', diff saved to https://phabricator.wikimedia.org/P37384 and previous config saved to /var/cache/conftool/dbconfig/20221101-141924-ladsgroup.json
  • 14:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 14:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 14:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T318950)', diff saved to https://phabricator.wikimedia.org/P37383 and previous config saved to /var/cache/conftool/dbconfig/20221101-141913-ladsgroup.json
  • 14:19 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4040.ulsfo.wmnet
  • 14:19 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:18 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4039.ulsfo.wmnet
  • 14:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T318950)', diff saved to https://phabricator.wikimedia.org/P37382 and previous config saved to /var/cache/conftool/dbconfig/20221101-141544-ladsgroup.json
  • 14:14 otto@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Use eventgate-analytics-external for rc0.mediawiki.page_change stream - T311129 (duration: 03m 42s)
  • 14:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P37381 and previous config saved to /var/cache/conftool/dbconfig/20221101-141335-ladsgroup.json
  • 14:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2148 (T318950)', diff saved to https://phabricator.wikimedia.org/P37380 and previous config saved to /var/cache/conftool/dbconfig/20221101-141322-ladsgroup.json
  • 14:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P37379 and previous config saved to /var/cache/conftool/dbconfig/20221101-141321-ladsgroup.json
  • 14:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 14:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 14:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37378 and previous config saved to /var/cache/conftool/dbconfig/20221101-141308-ladsgroup.json
  • 14:12 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:10 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti1028.eqiad.wmnet with reason: host reimage
  • 14:10 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1028.eqiad.wmnet with reason: host reimage
  • 14:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P37375 and previous config saved to /var/cache/conftool/dbconfig/20221101-140402-ladsgroup.json
  • 14:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:02 urbanecm@deploy1002: Finished scap: Backport for [GrowthExperiments] Remove wmgGEFeaturesMayBeAvailableToNewcomers (duration: 04m 32s)
  • 14:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P37374 and previous config saved to /var/cache/conftool/dbconfig/20221101-135827-ladsgroup.json
  • 13:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P37373 and previous config saved to /var/cache/conftool/dbconfig/20221101-135811-ladsgroup.json
  • 13:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P37372 and previous config saved to /var/cache/conftool/dbconfig/20221101-135800-ladsgroup.json
  • 13:59 moritzm: draining ganeti1016 for eventual reimage T311687
  • 13:59 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1028.eqiad.wmnet with OS bullseye
  • 13:59 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
  • 13:59 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4038.ulsfo.wmnet
  • 13:59 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
  • 13:57 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4037.ulsfo.wmnet
  • 13:56 moritzm: installing exim4 security updates on buster
  • 13:55 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:54 urbanecm@deploy1002: Started scap: Backport for [GrowthExperiments] Remove wmgGEFeaturesMayBeAvailableToNewcomers
  • 13:53 urbanecm@deploy1002: Finished scap: Backport for Copy reverse-proxy-staging.php to reverse-proxy-labs.php, "reverse-proxy-staging.php" -> "reverse-staging-labs.php", Delete "reverse-proxy-staging.php" (duration: 04m 30s)
  • 13:53 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:53 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:53 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T318955)', diff saved to https://phabricator.wikimedia.org/P37371 and previous config saved to /var/cache/conftool/dbconfig/20221101-135120-ladsgroup.json
  • 13:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 13:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 13:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 13:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 13:49 urbanecm@deploy1002: urbanecm and zabe: Backport for Copy reverse-proxy-staging.php to reverse-proxy-labs.php, "reverse-proxy-staging.php" -> "reverse-staging-labs.php", Delete "reverse-proxy-staging.php" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 13:49 urbanecm@deploy1002: Started scap: Backport for Copy reverse-proxy-staging.php to reverse-proxy-labs.php, "reverse-proxy-staging.php" -> "reverse-staging-labs.php", Delete "reverse-proxy-staging.php"
  • 13:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P37370 and previous config saved to /var/cache/conftool/dbconfig/20221101-134854-ladsgroup.json
  • 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T318955)', diff saved to https://phabricator.wikimedia.org/P37369 and previous config saved to /var/cache/conftool/dbconfig/20221101-134318-ladsgroup.json
  • 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T318605)', diff saved to https://phabricator.wikimedia.org/P37368 and previous config saved to /var/cache/conftool/dbconfig/20221101-134302-ladsgroup.json
  • 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P37367 and previous config saved to /var/cache/conftool/dbconfig/20221101-134252-ladsgroup.json
  • 13:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T318955)', diff saved to https://phabricator.wikimedia.org/P37366 and previous config saved to /var/cache/conftool/dbconfig/20221101-134108-ladsgroup.json
  • 13:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 13:42 sukhe@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp4037.ulsfo.wmnet
  • 13:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 13:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37365 and previous config saved to /var/cache/conftool/dbconfig/20221101-134045-ladsgroup.json
  • 13:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 13:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 13:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T318955)', diff saved to https://phabricator.wikimedia.org/P37364 and previous config saved to /var/cache/conftool/dbconfig/20221101-133857-ladsgroup.json
  • 13:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T318950)', diff saved to https://phabricator.wikimedia.org/P37363 and previous config saved to /var/cache/conftool/dbconfig/20221101-133346-ladsgroup.json
  • 13:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T318950)', diff saved to https://phabricator.wikimedia.org/P37362 and previous config saved to /var/cache/conftool/dbconfig/20221101-133132-ladsgroup.json
  • 13:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 13:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 13:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 13:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 13:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37361 and previous config saved to /var/cache/conftool/dbconfig/20221101-133113-ladsgroup.json
  • 13:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1104 (T318605)', diff saved to https://phabricator.wikimedia.org/P37360 and previous config saved to /var/cache/conftool/dbconfig/20221101-132841-ladsgroup.json
  • 13:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1104.eqiad.wmnet with reason: Maintenance
  • 13:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1104.eqiad.wmnet with reason: Maintenance
  • 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37359 and previous config saved to /var/cache/conftool/dbconfig/20221101-132817-ladsgroup.json
  • 13:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37358 and previous config saved to /var/cache/conftool/dbconfig/20221101-132745-ladsgroup.json
  • 13:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P37357 and previous config saved to /var/cache/conftool/dbconfig/20221101-132537-ladsgroup.json
  • 13:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37356 and previous config saved to /var/cache/conftool/dbconfig/20221101-132523-ladsgroup.json
  • 13:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 13:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 13:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T318950)', diff saved to https://phabricator.wikimedia.org/P37355 and previous config saved to /var/cache/conftool/dbconfig/20221101-132500-ladsgroup.json
  • 13:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P37354 and previous config saved to /var/cache/conftool/dbconfig/20221101-132348-ladsgroup.json
  • 13:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:22 urbanecm: UTC afternoon B&C window done
  • 13:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:20 urbanecm@deploy1002: Finished scap: Backport for zhwikivoyage: Add wordmark (T322133) (duration: 06m 36s)
  • 13:20 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1028.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 13:17 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1028.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 13:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P37353 and previous config saved to /var/cache/conftool/dbconfig/20221101-131605-ladsgroup.json
  • 13:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:14 urbanecm@deploy1002: urbanecm and stang: Backport for zhwikivoyage: Add wordmark (T322133) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 13:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1012.eqiad.wmnet
  • 13:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:14 urbanecm@deploy1002: Started scap: Backport for zhwikivoyage: Add wordmark (T322133)
  • 13:13 urbanecm@deploy1002: Finished scap: Backport for viwiki: Increase autoconfirmed edit count to 10 (T322105) (duration: 10m 35s)
  • 13:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P37352 and previous config saved to /var/cache/conftool/dbconfig/20221101-131309-ladsgroup.json
  • 13:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P37351 and previous config saved to /var/cache/conftool/dbconfig/20221101-131026-ladsgroup.json
  • 13:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P37350 and previous config saved to /var/cache/conftool/dbconfig/20221101-130952-ladsgroup.json
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2154 (T318605)', diff saved to https://phabricator.wikimedia.org/P37349 and previous config saved to /var/cache/conftool/dbconfig/20221101-130919-ladsgroup.json
  • 13:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
  • 13:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
  • 13:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T318605)', diff saved to https://phabricator.wikimedia.org/P37348 and previous config saved to /var/cache/conftool/dbconfig/20221101-130856-ladsgroup.json
  • 13:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P37347 and previous config saved to /var/cache/conftool/dbconfig/20221101-130839-ladsgroup.json
  • 13:06 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1012.eqiad.wmnet
  • 13:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1010.eqiad.wmnet
  • 13:03 urbanecm@deploy1002: urbanecm and stang: Backport for viwiki: Increase autoconfirmed edit count to 10 (T322105) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 13:02 urbanecm@deploy1002: Started scap: Backport for viwiki: Increase autoconfirmed edit count to 10 (T322105)
  • 13:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P37346 and previous config saved to /var/cache/conftool/dbconfig/20221101-130056-ladsgroup.json
  • 12:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P37345 and previous config saved to /var/cache/conftool/dbconfig/20221101-125801-ladsgroup.json
  • 12:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1010.eqiad.wmnet
  • 12:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1008.eqiad.wmnet
  • 12:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37344 and previous config saved to /var/cache/conftool/dbconfig/20221101-125516-ladsgroup.json
  • 12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P37343 and previous config saved to /var/cache/conftool/dbconfig/20221101-125443-ladsgroup.json
  • 12:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P37342 and previous config saved to /var/cache/conftool/dbconfig/20221101-125348-ladsgroup.json
  • 12:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T318955)', diff saved to https://phabricator.wikimedia.org/P37341 and previous config saved to /var/cache/conftool/dbconfig/20221101-125331-ladsgroup.json
  • 12:48 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1008.eqiad.wmnet
  • 12:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37340 and previous config saved to /var/cache/conftool/dbconfig/20221101-124548-ladsgroup.json
  • 12:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37339 and previous config saved to /var/cache/conftool/dbconfig/20221101-124334-ladsgroup.json
  • 12:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 12:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 12:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 12:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 12:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T318950)', diff saved to https://phabricator.wikimedia.org/P37338 and previous config saved to /var/cache/conftool/dbconfig/20221101-124301-ladsgroup.json
  • 12:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37337 and previous config saved to /var/cache/conftool/dbconfig/20221101-124253-ladsgroup.json
  • 12:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37336 and previous config saved to /var/cache/conftool/dbconfig/20221101-124225-ladsgroup.json
  • 12:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 12:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 12:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37335 and previous config saved to /var/cache/conftool/dbconfig/20221101-124202-ladsgroup.json
  • 12:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2129 (T318955)', diff saved to https://phabricator.wikimedia.org/P37334 and previous config saved to /var/cache/conftool/dbconfig/20221101-124012-ladsgroup.json
  • 12:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 12:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T318955)', diff saved to https://phabricator.wikimedia.org/P37333 and previous config saved to /var/cache/conftool/dbconfig/20221101-123949-ladsgroup.json
  • 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T318950)', diff saved to https://phabricator.wikimedia.org/P37332 and previous config saved to /var/cache/conftool/dbconfig/20221101-123936-ladsgroup.json
  • 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P37331 and previous config saved to /var/cache/conftool/dbconfig/20221101-123839-ladsgroup.json
  • 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2126 (T318950)', diff saved to https://phabricator.wikimedia.org/P37330 and previous config saved to /var/cache/conftool/dbconfig/20221101-123714-ladsgroup.json
  • 12:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 12:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 12:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 12:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 12:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T318950)', diff saved to https://phabricator.wikimedia.org/P37329 and previous config saved to /var/cache/conftool/dbconfig/20221101-123646-ladsgroup.json
  • 12:28 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1006.eqiad.wmnet
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P37328 and previous config saved to /var/cache/conftool/dbconfig/20221101-122750-ladsgroup.json
  • 12:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P37327 and previous config saved to /var/cache/conftool/dbconfig/20221101-122654-ladsgroup.json
  • 12:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P37326 and previous config saved to /var/cache/conftool/dbconfig/20221101-122442-ladsgroup.json
  • 12:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T318605)', diff saved to https://phabricator.wikimedia.org/P37325 and previous config saved to /var/cache/conftool/dbconfig/20221101-122329-ladsgroup.json
  • 12:21 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1006.eqiad.wmnet
  • 12:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P37324 and previous config saved to /var/cache/conftool/dbconfig/20221101-122138-ladsgroup.json
  • 12:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P37323 and previous config saved to /var/cache/conftool/dbconfig/20221101-121242-ladsgroup.json
  • 12:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P37322 and previous config saved to /var/cache/conftool/dbconfig/20221101-121147-ladsgroup.json
  • 12:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P37321 and previous config saved to /var/cache/conftool/dbconfig/20221101-120934-ladsgroup.json
  • 12:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P37320 and previous config saved to /var/cache/conftool/dbconfig/20221101-120630-ladsgroup.json
  • 12:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37319 and previous config saved to /var/cache/conftool/dbconfig/20221101-120341-ladsgroup.json
  • 12:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 12:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 12:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37318 and previous config saved to /var/cache/conftool/dbconfig/20221101-120318-ladsgroup.json
  • 11:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T318950)', diff saved to https://phabricator.wikimedia.org/P37317 and previous config saved to /var/cache/conftool/dbconfig/20221101-115734-ladsgroup.json
  • 11:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37316 and previous config saved to /var/cache/conftool/dbconfig/20221101-115638-ladsgroup.json
  • 11:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T318955)', diff saved to https://phabricator.wikimedia.org/P37315 and previous config saved to /var/cache/conftool/dbconfig/20221101-115426-ladsgroup.json
  • 11:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T318950)', diff saved to https://phabricator.wikimedia.org/P37314 and previous config saved to /var/cache/conftool/dbconfig/20221101-115122-ladsgroup.json
  • 11:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2152 (T318605)', diff saved to https://phabricator.wikimedia.org/P37313 and previous config saved to /var/cache/conftool/dbconfig/20221101-114943-ladsgroup.json
  • 11:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance
  • 11:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance
  • 11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T318950)', diff saved to https://phabricator.wikimedia.org/P37312 and previous config saved to /var/cache/conftool/dbconfig/20221101-114858-ladsgroup.json
  • 11:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 11:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 11:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T318950)', diff saved to https://phabricator.wikimedia.org/P37311 and previous config saved to /var/cache/conftool/dbconfig/20221101-114835-ladsgroup.json
  • 11:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T318950)', diff saved to https://phabricator.wikimedia.org/P37310 and previous config saved to /var/cache/conftool/dbconfig/20221101-114820-ladsgroup.json
  • 11:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318', diff saved to https://phabricator.wikimedia.org/P37309 and previous config saved to /var/cache/conftool/dbconfig/20221101-114811-ladsgroup.json
  • 11:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 11:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 11:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37308 and previous config saved to /var/cache/conftool/dbconfig/20221101-114755-ladsgroup.json
  • 11:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37307 and previous config saved to /var/cache/conftool/dbconfig/20221101-114145-ladsgroup.json
  • 11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 11:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 11:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37306 and previous config saved to /var/cache/conftool/dbconfig/20221101-114123-ladsgroup.json
  • 11:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T318955)', diff saved to https://phabricator.wikimedia.org/P37305 and previous config saved to /var/cache/conftool/dbconfig/20221101-114121-ladsgroup.json
  • 11:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 11:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 11:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T318955)', diff saved to https://phabricator.wikimedia.org/P37304 and previous config saved to /var/cache/conftool/dbconfig/20221101-114057-ladsgroup.json
  • 11:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P37303 and previous config saved to /var/cache/conftool/dbconfig/20221101-113327-ladsgroup.json
  • 11:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318', diff saved to https://phabricator.wikimedia.org/P37302 and previous config saved to /var/cache/conftool/dbconfig/20221101-113301-ladsgroup.json
  • 11:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P37301 and previous config saved to /var/cache/conftool/dbconfig/20221101-113248-ladsgroup.json
  • 11:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P37300 and previous config saved to /var/cache/conftool/dbconfig/20221101-112612-ladsgroup.json
  • 11:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P37299 and previous config saved to /var/cache/conftool/dbconfig/20221101-112549-ladsgroup.json
  • 11:19 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host puppetdb-test2001.codfw.wmnet
  • 11:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P37298 and previous config saved to /var/cache/conftool/dbconfig/20221101-111819-ladsgroup.json
  • 11:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37297 and previous config saved to /var/cache/conftool/dbconfig/20221101-111753-ladsgroup.json
  • 11:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P37296 and previous config saved to /var/cache/conftool/dbconfig/20221101-111739-ladsgroup.json
  • 11:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 11:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 11:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P37295 and previous config saved to /var/cache/conftool/dbconfig/20221101-111106-ladsgroup.json
  • 11:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P37294 and previous config saved to /var/cache/conftool/dbconfig/20221101-111042-ladsgroup.json
  • 11:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetdb-test2001.codfw.wmnet
  • 11:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T318950)', diff saved to https://phabricator.wikimedia.org/P37293 and previous config saved to /var/cache/conftool/dbconfig/20221101-110311-ladsgroup.json
  • 11:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37292 and previous config saved to /var/cache/conftool/dbconfig/20221101-110232-ladsgroup.json
  • 11:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2104 (T318950)', diff saved to https://phabricator.wikimedia.org/P37291 and previous config saved to /var/cache/conftool/dbconfig/20221101-110045-ladsgroup.json
  • 11:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 11:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 11:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T318950)', diff saved to https://phabricator.wikimedia.org/P37290 and previous config saved to /var/cache/conftool/dbconfig/20221101-110019-ladsgroup.json
  • 11:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 11:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 11:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 11:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 10:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 10:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 10:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37289 and previous config saved to /var/cache/conftool/dbconfig/20221101-105557-ladsgroup.json
  • 10:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T318955)', diff saved to https://phabricator.wikimedia.org/P37288 and previous config saved to /var/cache/conftool/dbconfig/20221101-105534-ladsgroup.json
  • 10:48 moritzm: updating libdatetime-timezone-perl from latest Debian SUA update
  • 10:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T318955)', diff saved to https://phabricator.wikimedia.org/P37287 and previous config saved to /var/cache/conftool/dbconfig/20221101-104215-ladsgroup.json
  • 10:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 10:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T318955)', diff saved to https://phabricator.wikimedia.org/P37286 and previous config saved to /var/cache/conftool/dbconfig/20221101-104154-ladsgroup.json
  • 10:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 10:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 10:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3318 (T318605)', diff saved to https://phabricator.wikimedia.org/P37285 and previous config saved to /var/cache/conftool/dbconfig/20221101-103934-ladsgroup.json
  • 10:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 10:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 10:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 09:43 moritzm: imported quickstack 20161026-1+deb12u1 to apt.wikimedia.org/bookworm-wikimedia T321783
  • 08:32 moritzm: draining ganeti1028 for eventual reimage T311687
  • 08:29 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Faidon Liambotis out of all services on: 1203 hosts
  • 08:29 jmm@cumin2002: START - Cookbook sre.idm.logout Logging Faidon Liambotis out of all services on: 1203 hosts
  • 08:29 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Faidon Liambotis out of all services on: 802 hosts
  • 08:27 jmm@cumin2002: START - Cookbook sre.idm.logout Logging Faidon Liambotis out of all services on: 802 hosts
  • 03:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 03:45 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 03:45 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 03:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 03:36 mwpresync@deploy1002: Finished scap: testwikis wikis to 1.40.0-wmf.8 refs T320513 (duration: 33m 56s)
  • 03:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 03:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 03:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 03:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 03:02 mwpresync@deploy1002: Started scap: testwikis wikis to 1.40.0-wmf.8 refs T320513
  • 02:31 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 02:30 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 02:30 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 02:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 02:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 02:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 02:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 02:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debu

Other archives

2000s

2010s

2020s

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Server_Admin_Log/Archive_59&oldid=2037377"