Jump to content

Server Admin Log/Archive 59

From Wikitech

2022-11-15

  • 23:54 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudmetrics[1001-1002].eqiad.wmnet
  • 23:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 23:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 23:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T318605)', diff saved to https://phabricator.wikimedia.org/P39860 and previous config saved to /var/cache/conftool/dbconfig/20221115-234056-ladsgroup.json
  • 23:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 23:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 23:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T321130)', diff saved to https://phabricator.wikimedia.org/P39859 and previous config saved to /var/cache/conftool/dbconfig/20221115-233253-marostegui.json
  • 23:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1118 (T318605)', diff saved to https://phabricator.wikimedia.org/P39858 and previous config saved to /var/cache/conftool/dbconfig/20221115-232600-ladsgroup.json
  • 23:25 brennen@deploy1002: Finished scap: Backport for Feed: Use DerivativeContext and not clone main RequestContext (T323153) (duration: 06m 26s)
  • 23:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P39857 and previous config saved to /var/cache/conftool/dbconfig/20221115-232550-ladsgroup.json
  • 23:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 23:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 23:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T318605)', diff saved to https://phabricator.wikimedia.org/P39856 and previous config saved to /var/cache/conftool/dbconfig/20221115-232532-ladsgroup.json
  • 23:21 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@7762e35]: import_cirrus_indexes: snapshot partitioning should not use dashes (duration: 02m 16s)
  • 23:19 brennen@deploy1002: brennen and tstarling: Backport for Feed: Use DerivativeContext and not clone main RequestContext (T323153) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 23:19 brennen@deploy1002: Started scap: Backport for Feed: Use DerivativeContext and not clone main RequestContext (T323153)
  • 23:18 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@7762e35]: import_cirrus_indexes: snapshot partitioning should not use dashes
  • 23:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P39855 and previous config saved to /var/cache/conftool/dbconfig/20221115-231746-marostegui.json
  • 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P39854 and previous config saved to /var/cache/conftool/dbconfig/20221115-231043-ladsgroup.json
  • 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P39853 and previous config saved to /var/cache/conftool/dbconfig/20221115-231025-ladsgroup.json
  • 23:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P39852 and previous config saved to /var/cache/conftool/dbconfig/20221115-230240-marostegui.json
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T318605)', diff saved to https://phabricator.wikimedia.org/P39851 and previous config saved to /var/cache/conftool/dbconfig/20221115-225537-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P39850 and previous config saved to /var/cache/conftool/dbconfig/20221115-225518-ladsgroup.json
  • 22:52 ejegg: re-enabled civicrm dedupe jobs
  • 22:51 mutante: phab1004 - running public_task_dump.py
  • 22:51 ejegg: civicrm upgraded from d85589e8 to fa71f219
  • 22:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T321130)', diff saved to https://phabricator.wikimedia.org/P39849 and previous config saved to /var/cache/conftool/dbconfig/20221115-224733-marostegui.json
  • 22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T318605)', diff saved to https://phabricator.wikimedia.org/P39848 and previous config saved to /var/cache/conftool/dbconfig/20221115-224011-ladsgroup.json
  • 22:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T321130)', diff saved to https://phabricator.wikimedia.org/P39847 and previous config saved to /var/cache/conftool/dbconfig/20221115-221247-marostegui.json
  • 22:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 22:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 22:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T321130)', diff saved to https://phabricator.wikimedia.org/P39846 and previous config saved to /var/cache/conftool/dbconfig/20221115-221225-marostegui.json
  • 21:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P39845 and previous config saved to /var/cache/conftool/dbconfig/20221115-215719-marostegui.json
  • 21:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P39844 and previous config saved to /var/cache/conftool/dbconfig/20221115-214212-marostegui.json
  • 21:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T321126)', diff saved to https://phabricator.wikimedia.org/P39843 and previous config saved to /var/cache/conftool/dbconfig/20221115-214022-marostegui.json
  • 21:33 cjming: end of UTC late backport window
  • 21:27 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:27 cjming@deploy1002: Finished scap: Backport for tnwiki: Set timezone to Africa/Gaborone (UTC+2) (T318208) (duration: 06m 14s)
  • 21:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T321130)', diff saved to https://phabricator.wikimedia.org/P39842 and previous config saved to /var/cache/conftool/dbconfig/20221115-212706-marostegui.json
  • 21:25 cmooney@cumin1001: START - Cookbook sre.dns.netbox
  • 21:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39841 and previous config saved to /var/cache/conftool/dbconfig/20221115-212516-marostegui.json
  • 21:21 cjming@deploy1002: cjming and stang: Backport for tnwiki: Set timezone to Africa/Gaborone (UTC+2) (T318208) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 21:21 cjming@deploy1002: Started scap: Backport for tnwiki: Set timezone to Africa/Gaborone (UTC+2) (T318208)
  • 21:19 cjming@deploy1002: Finished scap: Backport for logos: Remove duplicated code (T307705) (duration: 04m 31s)
  • 21:15 cjming@deploy1002: cjming and stang: Backport for logos: Remove duplicated code (T307705) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 21:14 cjming@deploy1002: Started scap: Backport for logos: Remove duplicated code (T307705)
  • 21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2130 (T318605)', diff saved to https://phabricator.wikimedia.org/P39840 and previous config saved to /var/cache/conftool/dbconfig/20221115-211314-ladsgroup.json
  • 21:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 21:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 21:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T318605)', diff saved to https://phabricator.wikimedia.org/P39839 and previous config saved to /var/cache/conftool/dbconfig/20221115-211253-ladsgroup.json
  • 21:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39838 and previous config saved to /var/cache/conftool/dbconfig/20221115-211009-marostegui.json
  • 21:09 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:09 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:08 cjming@deploy1002: Finished scap: Backport for EditAttemptStep sampling rate to 1 everywhere (T312016) (duration: 04m 45s)
  • 21:07 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:07 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:04 cjming@deploy1002: cjming and phuedx: Backport for EditAttemptStep sampling rate to 1 everywhere (T312016) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 21:03 cjming@deploy1002: Started scap: Backport for EditAttemptStep sampling rate to 1 everywhere (T312016)
  • 21:02 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:01 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1107 (T318605)', diff saved to https://phabricator.wikimedia.org/P39837 and previous config saved to /var/cache/conftool/dbconfig/20221115-205929-ladsgroup.json
  • 20:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 20:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 20:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T318605)', diff saved to https://phabricator.wikimedia.org/P39836 and previous config saved to /var/cache/conftool/dbconfig/20221115-205907-ladsgroup.json
  • 20:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P39835 and previous config saved to /var/cache/conftool/dbconfig/20221115-205746-ladsgroup.json
  • 20:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T321126)', diff saved to https://phabricator.wikimedia.org/P39834 and previous config saved to /var/cache/conftool/dbconfig/20221115-205503-marostegui.json
  • 20:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2175 (T321126)', diff saved to https://phabricator.wikimedia.org/P39833 and previous config saved to /var/cache/conftool/dbconfig/20221115-205222-marostegui.json
  • 20:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 20:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T321130)', diff saved to https://phabricator.wikimedia.org/P39832 and previous config saved to /var/cache/conftool/dbconfig/20221115-205214-marostegui.json
  • 20:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 20:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 20:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39831 and previous config saved to /var/cache/conftool/dbconfig/20221115-205201-marostegui.json
  • 20:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 20:49 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dbprov2004.mgmt.codfw.wmnet with reboot policy GRACEFUL
  • 20:44 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 20:44 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P39830 and previous config saved to /var/cache/conftool/dbconfig/20221115-204401-ladsgroup.json
  • 20:42 eileen: civicrm upgraded from 16167e9a to d85589e8
  • 20:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P39829 and previous config saved to /var/cache/conftool/dbconfig/20221115-204239-ladsgroup.json
  • 20:39 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy GRACEFUL
  • 20:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39828 and previous config saved to /var/cache/conftool/dbconfig/20221115-203654-marostegui.json
  • 20:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 20:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 20:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T321130)', diff saved to https://phabricator.wikimedia.org/P39827 and previous config saved to /var/cache/conftool/dbconfig/20221115-203521-marostegui.json
  • 20:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P39826 and previous config saved to /var/cache/conftool/dbconfig/20221115-202854-ladsgroup.json
  • 20:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T318605)', diff saved to https://phabricator.wikimedia.org/P39825 and previous config saved to /var/cache/conftool/dbconfig/20221115-202733-ladsgroup.json
  • 20:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39824 and previous config saved to /var/cache/conftool/dbconfig/20221115-202148-marostegui.json
  • 20:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P39823 and previous config saved to /var/cache/conftool/dbconfig/20221115-202015-marostegui.json
  • 20:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T318605)', diff saved to https://phabricator.wikimedia.org/P39822 and previous config saved to /var/cache/conftool/dbconfig/20221115-201348-ladsgroup.json
  • 20:13 eileen: civicrm upgraded from 3eba6ad3 to 16167e9a
  • 20:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39821 and previous config saved to /var/cache/conftool/dbconfig/20221115-200641-marostegui.json
  • 20:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P39820 and previous config saved to /var/cache/conftool/dbconfig/20221115-200508-marostegui.json
  • 20:04 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39819 and previous config saved to /var/cache/conftool/dbconfig/20221115-200358-marostegui.json
  • 20:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 20:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 20:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T321126)', diff saved to https://phabricator.wikimedia.org/P39818 and previous config saved to /var/cache/conftool/dbconfig/20221115-200337-marostegui.json
  • 19:56 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:56 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:54 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:54 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:54 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:51 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2042.codfw.wmnet with OS bullseye
  • 19:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T321130)', diff saved to https://phabricator.wikimedia.org/P39817 and previous config saved to /var/cache/conftool/dbconfig/20221115-195002-marostegui.json
  • 19:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P39816 and previous config saved to /var/cache/conftool/dbconfig/20221115-194830-marostegui.json
  • 19:42 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:42 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T321130)', diff saved to https://phabricator.wikimedia.org/P39815 and previous config saved to /var/cache/conftool/dbconfig/20221115-194037-marostegui.json
  • 19:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T321130)', diff saved to https://phabricator.wikimedia.org/P39814 and previous config saved to /var/cache/conftool/dbconfig/20221115-194016-marostegui.json
  • 19:38 ejegg: turned off CiviCRM dedupe jobs for queue speed measurements
  • 19:38 ejegg: payments-wiki upgraded from a058fdbc to bba997aa
  • 19:34 jbond: updated pcc to 2.5.1
  • 19:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P39813 and previous config saved to /var/cache/conftool/dbconfig/20221115-193324-marostegui.json
  • 19:32 topranks: renumbering overlay vrf loopback interface lsw1-e3-eqiad
  • 19:28 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 19:28 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 19:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P39812 and previous config saved to /var/cache/conftool/dbconfig/20221115-192509-marostegui.json
  • 19:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T321126)', diff saved to https://phabricator.wikimedia.org/P39811 and previous config saved to /var/cache/conftool/dbconfig/20221115-191818-marostegui.json
  • 19:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2148 (T321126)', diff saved to https://phabricator.wikimedia.org/P39810 and previous config saved to /var/cache/conftool/dbconfig/20221115-191536-marostegui.json
  • 19:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 19:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 19:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39809 and previous config saved to /var/cache/conftool/dbconfig/20221115-191514-marostegui.json
  • 19:10 brennen@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.10 refs T320515
  • 19:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P39808 and previous config saved to /var/cache/conftool/dbconfig/20221115-191003-marostegui.json
  • 19:04 brennen: train 1.40.0-wmf.10 (T320515) - no current blockers, rolling to group0.
  • 19:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P39807 and previous config saved to /var/cache/conftool/dbconfig/20221115-190008-marostegui.json
  • 18:58 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp2042.codfw.wmnet with OS bullseye
  • 18:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2116 (T318605)', diff saved to https://phabricator.wikimedia.org/P39806 and previous config saved to /var/cache/conftool/dbconfig/20221115-185619-ladsgroup.json
  • 18:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 18:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 18:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T318605)', diff saved to https://phabricator.wikimedia.org/P39805 and previous config saved to /var/cache/conftool/dbconfig/20221115-185558-ladsgroup.json
  • 18:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T321130)', diff saved to https://phabricator.wikimedia.org/P39804 and previous config saved to /var/cache/conftool/dbconfig/20221115-185457-marostegui.json
  • 18:49 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5032.eqsin.wmnet with OS buster
  • 18:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P39803 and previous config saved to /var/cache/conftool/dbconfig/20221115-184501-marostegui.json
  • 18:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P39802 and previous config saved to /var/cache/conftool/dbconfig/20221115-184051-ladsgroup.json
  • 18:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T318605)', diff saved to https://phabricator.wikimedia.org/P39801 and previous config saved to /var/cache/conftool/dbconfig/20221115-183053-ladsgroup.json
  • 18:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 18:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 18:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 18:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 18:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39800 and previous config saved to /var/cache/conftool/dbconfig/20221115-183025-ladsgroup.json
  • 18:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39799 and previous config saved to /var/cache/conftool/dbconfig/20221115-182955-marostegui.json
  • 18:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39798 and previous config saved to /var/cache/conftool/dbconfig/20221115-182712-marostegui.json
  • 18:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T321126)', diff saved to https://phabricator.wikimedia.org/P39797 and previous config saved to /var/cache/conftool/dbconfig/20221115-182640-marostegui.json
  • 18:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P39796 and previous config saved to /var/cache/conftool/dbconfig/20221115-182545-ladsgroup.json
  • 18:16 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage
  • 18:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P39795 and previous config saved to /var/cache/conftool/dbconfig/20221115-181519-ladsgroup.json
  • 18:13 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage
  • 18:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P39794 and previous config saved to /var/cache/conftool/dbconfig/20221115-181133-marostegui.json
  • 18:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T318605)', diff saved to https://phabricator.wikimedia.org/P39793 and previous config saved to /var/cache/conftool/dbconfig/20221115-181037-ladsgroup.json
  • 18:06 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=idwiki` at mwmaint1002 (T318457)
  • 18:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P39792 and previous config saved to /var/cache/conftool/dbconfig/20221115-180012-ladsgroup.json
  • 17:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P39791 and previous config saved to /var/cache/conftool/dbconfig/20221115-175627-marostegui.json
  • 17:54 jbond: move pcc to 2.5.0
  • 17:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T321130)', diff saved to https://phabricator.wikimedia.org/P39790 and previous config saved to /var/cache/conftool/dbconfig/20221115-174827-marostegui.json
  • 17:48 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 17:48 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 17:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39789 and previous config saved to /var/cache/conftool/dbconfig/20221115-174805-marostegui.json
  • 17:46 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 17:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39788 and previous config saved to /var/cache/conftool/dbconfig/20221115-174506-ladsgroup.json
  • 17:42 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS buster
  • 17:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T321126)', diff saved to https://phabricator.wikimedia.org/P39787 and previous config saved to /var/cache/conftool/dbconfig/20221115-174120-marostegui.json
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2126 (T321126)', diff saved to https://phabricator.wikimedia.org/P39786 and previous config saved to /var/cache/conftool/dbconfig/20221115-173841-marostegui.json
  • 17:38 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T321126)', diff saved to https://phabricator.wikimedia.org/P39785 and previous config saved to /var/cache/conftool/dbconfig/20221115-173804-marostegui.json
  • 17:37 hnowlan@cumin1001: END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0)
  • 17:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P39784 and previous config saved to /var/cache/conftool/dbconfig/20221115-173258-marostegui.json
  • 17:29 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 17:29 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 17:28 mutante: deploy1002:~] $ sudo systemctl start wmf_auto_restart_apache2.service
  • 17:25 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 17:25 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 17:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P39783 and previous config saved to /var/cache/conftool/dbconfig/20221115-172257-marostegui.json
  • 17:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P39782 and previous config saved to /var/cache/conftool/dbconfig/20221115-171752-marostegui.json
  • 17:16 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=enwiki` at mwmaint1002 (T318457)
  • 17:10 godog: add 150G to prometheus/ops in eqiad
  • 17:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P39781 and previous config saved to /var/cache/conftool/dbconfig/20221115-170751-marostegui.json
  • 17:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39780 and previous config saved to /var/cache/conftool/dbconfig/20221115-170245-marostegui.json
  • 17:02 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=fawiki` at mwmaint1002 (T318457)
  • 16:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39779 and previous config saved to /var/cache/conftool/dbconfig/20221115-165323-marostegui.json
  • 16:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 16:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 16:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T321130)', diff saved to https://phabricator.wikimedia.org/P39778 and previous config saved to /var/cache/conftool/dbconfig/20221115-165302-marostegui.json
  • 16:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T321126)', diff saved to https://phabricator.wikimedia.org/P39777 and previous config saved to /var/cache/conftool/dbconfig/20221115-165244-marostegui.json
  • 16:50 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T321126)', diff saved to https://phabricator.wikimedia.org/P39776 and previous config saved to /var/cache/conftool/dbconfig/20221115-165001-marostegui.json
  • 16:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 16:49 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 16:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T321126)', diff saved to https://phabricator.wikimedia.org/P39775 and previous config saved to /var/cache/conftool/dbconfig/20221115-164939-marostegui.json
  • 16:49 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 16:43 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry
  • 16:41 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry
  • 16:40 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 16:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P39774 and previous config saved to /var/cache/conftool/dbconfig/20221115-163755-marostegui.json
  • 16:37 ladsgroup:: Deployed security patch for T320987
  • 16:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2103 (T318605)', diff saved to https://phabricator.wikimedia.org/P39773 and previous config saved to /var/cache/conftool/dbconfig/20221115-163721-ladsgroup.json
  • 16:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 16:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 16:36 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wcqs-public
  • 16:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P39772 and previous config saved to /var/cache/conftool/dbconfig/20221115-163432-marostegui.json
  • 16:34 ladsgroup:: Deployed security patch for T320987
  • 16:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1014.eqiad.wmnet
  • 16:22 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wcqs-public
  • 16:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P39770 and previous config saved to /var/cache/conftool/dbconfig/20221115-162249-marostegui.json
  • 16:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1022.eqiad.wmnet with OS bullseye
  • 16:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet
  • 16:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1014.eqiad.wmnet
  • 16:20 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=arwiki` at mwmaint1002 (T318457)
  • 16:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P39769 and previous config saved to /var/cache/conftool/dbconfig/20221115-161925-marostegui.json
  • 16:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS bullseye
  • 16:15 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wdqs-all
  • 16:15 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 16:14 urandom: initiating Cassandra bootstrap, aqs1019-b -- T307802
  • 16:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet
  • 16:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet
  • 16:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39768 and previous config saved to /var/cache/conftool/dbconfig/20221115-160804-ladsgroup.json
  • 16:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 16:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 16:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T321130)', diff saved to https://phabricator.wikimedia.org/P39767 and previous config saved to /var/cache/conftool/dbconfig/20221115-160742-marostegui.json
  • 16:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1012.eqiad.wmnet
  • 16:06 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wdqs-all
  • 16:05 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 16:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet
  • 16:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T321126)', diff saved to https://phabricator.wikimedia.org/P39766 and previous config saved to /var/cache/conftool/dbconfig/20221115-160419-marostegui.json
  • 16:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
  • 16:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1022.eqiad.wmnet with reason: host reimage
  • 16:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1012.eqiad.wmnet
  • 16:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2104 (T321126)', diff saved to https://phabricator.wikimedia.org/P39765 and previous config saved to /var/cache/conftool/dbconfig/20221115-160140-marostegui.json
  • 16:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 16:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T321126)', diff saved to https://phabricator.wikimedia.org/P39764 and previous config saved to /var/cache/conftool/dbconfig/20221115-160010-marostegui.json
  • 15:59 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1022.eqiad.wmnet with reason: host reimage
  • 15:58 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T321130)', diff saved to https://phabricator.wikimedia.org/P39763 and previous config saved to /var/cache/conftool/dbconfig/20221115-155821-marostegui.json
  • 15:58 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
  • 15:58 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 15:58 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 15:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321130)', diff saved to https://phabricator.wikimedia.org/P39762 and previous config saved to /var/cache/conftool/dbconfig/20221115-155800-marostegui.json
  • 15:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
  • 15:54 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P39761 and previous config saved to /var/cache/conftool/dbconfig/20221115-155237-ladsgroup.json
  • 15:49 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=ptwiki` at mwmaint1002 (T318457)
  • 15:45 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1022.eqiad.wmnet with OS bullseye
  • 15:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P39760 and previous config saved to /var/cache/conftool/dbconfig/20221115-154504-marostegui.json
  • 15:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1022.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 15:44 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1022.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 15:43 moritzm: uploaded cas 6.6.2 to apt.wikimedia.org T311235
  • 15:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P39759 and previous config saved to /var/cache/conftool/dbconfig/20221115-154253-marostegui.json
  • 15:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P39758 and previous config saved to /var/cache/conftool/dbconfig/20221115-153731-ladsgroup.json
  • 15:33 taavi@deploy1002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
  • 15:32 taavi@deploy1002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 15:32 taavi@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
  • 15:31 taavi@deploy1002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
  • 15:30 taavi@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 15:30 taavi@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 15:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P39757 and previous config saved to /var/cache/conftool/dbconfig/20221115-152957-marostegui.json
  • 15:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P39756 and previous config saved to /var/cache/conftool/dbconfig/20221115-152747-marostegui.json
  • 15:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39755 and previous config saved to /var/cache/conftool/dbconfig/20221115-152224-ladsgroup.json
  • 15:15 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=bnwiki` at mwmaint1002 (T318457)
  • 15:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T321126)', diff saved to https://phabricator.wikimedia.org/P39754 and previous config saved to /var/cache/conftool/dbconfig/20221115-151451-marostegui.json
  • 15:14 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS bullseye
  • 15:14 moritzm: installing expat security updates
  • 15:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321130)', diff saved to https://phabricator.wikimedia.org/P39753 and previous config saved to /var/cache/conftool/dbconfig/20221115-151241-marostegui.json
  • 15:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1197 (T321126)', diff saved to https://phabricator.wikimedia.org/P39752 and previous config saved to /var/cache/conftool/dbconfig/20221115-151232-marostegui.json
  • 15:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 15:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 15:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T321126)', diff saved to https://phabricator.wikimedia.org/P39751 and previous config saved to /var/cache/conftool/dbconfig/20221115-151211-marostegui.json
  • 15:10 hnowlan@cumin1001: START - Cookbook sre.postgresql.postgres-init
  • 15:09 hnowlan@cumin1001: END (FAIL) - Cookbook sre.postgresql.postgres-init (exit_code=99)
  • 15:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T318950)', diff saved to https://phabricator.wikimedia.org/P39750 and previous config saved to /var/cache/conftool/dbconfig/20221115-150901-ladsgroup.json
  • 15:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3003.esams.wmnet
  • 15:06 hnowlan@cumin1001: START - Cookbook sre.postgresql.postgres-init
  • 15:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T318955)', diff saved to https://phabricator.wikimedia.org/P39749 and previous config saved to /var/cache/conftool/dbconfig/20221115-150242-ladsgroup.json
  • 15:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 15:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti3003.esams.wmnet
  • 15:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 14:59 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=frwiki` at mwmaint1002 (T318457)
  • 14:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2032.codfw.wmnet
  • 14:57 urbanecm@deploy1002: Finished scap: Backport for updateIsActiveFlagForMentees: Process all mentees (T318457), MentorStore: Use $wgRCMaxAge instead of INACTIVITY_THRESHOLD (T318457) (duration: 05m 44s)
  • 14:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P39748 and previous config saved to /var/cache/conftool/dbconfig/20221115-145704-marostegui.json
  • 14:55 moritzm: installing tomcat security updates
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39747 and previous config saved to /var/cache/conftool/dbconfig/20221115-145355-ladsgroup.json
  • 14:53 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 14:52 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2032.codfw.wmnet
  • 14:52 urbanecm@deploy1002: urbanecm and urbanecm: Backport for updateIsActiveFlagForMentees: Process all mentees (T318457), MentorStore: Use $wgRCMaxAge instead of INACTIVITY_THRESHOLD (T318457) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 14:51 urbanecm@deploy1002: Started scap: Backport for updateIsActiveFlagForMentees: Process all mentees (T318457), MentorStore: Use $wgRCMaxAge instead of INACTIVITY_THRESHOLD (T318457)
  • 14:51 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp5032']
  • 14:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2031.codfw.wmnet
  • 14:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P39746 and previous config saved to /var/cache/conftool/dbconfig/20221115-144736-ladsgroup.json
  • 14:43 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2031.codfw.wmnet
  • 14:43 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5032']
  • 14:43 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cp5032']
  • 14:43 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 14:42 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2003-dev.codfw.wmnet with OS bullseye
  • 14:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P39745 and previous config saved to /var/cache/conftool/dbconfig/20221115-144158-marostegui.json
  • 14:40 moritzm: failover ganeti master in esams to ganeti3001
  • 14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39744 and previous config saved to /var/cache/conftool/dbconfig/20221115-143848-ladsgroup.json
  • 14:33 hnowlan@cumin1001: END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0)
  • 14:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P39743 and previous config saved to /var/cache/conftool/dbconfig/20221115-143229-ladsgroup.json
  • 14:31 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5032']
  • 14:27 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
  • 14:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3002.esams.wmnet
  • 14:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T321126)', diff saved to https://phabricator.wikimedia.org/P39742 and previous config saved to /var/cache/conftool/dbconfig/20221115-142652-marostegui.json
  • 14:25 aborrero@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
  • 14:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1188 (T321126)', diff saved to https://phabricator.wikimedia.org/P39741 and previous config saved to /var/cache/conftool/dbconfig/20221115-142432-marostegui.json
  • 14:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 14:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 14:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T321126)', diff saved to https://phabricator.wikimedia.org/P39740 and previous config saved to /var/cache/conftool/dbconfig/20221115-142411-marostegui.json
  • 14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T318950)', diff saved to https://phabricator.wikimedia.org/P39739 and previous config saved to /var/cache/conftool/dbconfig/20221115-142342-ladsgroup.json
  • 14:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T318950)', diff saved to https://phabricator.wikimedia.org/P39738 and previous config saved to /var/cache/conftool/dbconfig/20221115-142130-ladsgroup.json
  • 14:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 14:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 14:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti3002.esams.wmnet
  • 14:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T318955)', diff saved to https://phabricator.wikimedia.org/P39737 and previous config saved to /var/cache/conftool/dbconfig/20221115-141723-ladsgroup.json
  • 14:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T318955)', diff saved to https://phabricator.wikimedia.org/P39736 and previous config saved to /var/cache/conftool/dbconfig/20221115-141513-ladsgroup.json
  • 14:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 14:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 14:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T321130)', diff saved to https://phabricator.wikimedia.org/P39735 and previous config saved to /var/cache/conftool/dbconfig/20221115-141218-marostegui.json
  • 14:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 14:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 14:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39734 and previous config saved to /var/cache/conftool/dbconfig/20221115-141157-marostegui.json
  • 14:09 aborrero@cumin1001: START - Cookbook sre.hosts.reimage for host cloudgw2003-dev.codfw.wmnet with OS bullseye
  • 14:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P39733 and previous config saved to /var/cache/conftool/dbconfig/20221115-140905-marostegui.json
  • 14:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp5032.mgmt.eqsin.wmnet with reboot policy FORCED
  • 14:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3001.esams.wmnet
  • 13:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P39732 and previous config saved to /var/cache/conftool/dbconfig/20221115-135650-marostegui.json
  • 13:56 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cp5032.mgmt.eqsin.wmnet with reboot policy FORCED
  • 13:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P39731 and previous config saved to /var/cache/conftool/dbconfig/20221115-135358-marostegui.json
  • 13:52 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti3001.esams.wmnet
  • 13:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P39730 and previous config saved to /var/cache/conftool/dbconfig/20221115-134144-marostegui.json
  • 13:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 13:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 (T318605)', diff saved to https://phabricator.wikimedia.org/P39729 and previous config saved to /var/cache/conftool/dbconfig/20221115-134036-ladsgroup.json
  • 13:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 13:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 13:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 13:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T321126)', diff saved to https://phabricator.wikimedia.org/P39728 and previous config saved to /var/cache/conftool/dbconfig/20221115-133852-marostegui.json
  • 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T321126)', diff saved to https://phabricator.wikimedia.org/P39727 and previous config saved to /var/cache/conftool/dbconfig/20221115-133631-marostegui.json
  • 13:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 13:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39726 and previous config saved to /var/cache/conftool/dbconfig/20221115-133610-marostegui.json
  • 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4008.ulsfo.wmnet
  • 13:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet
  • 13:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39725 and previous config saved to /var/cache/conftool/dbconfig/20221115-132637-marostegui.json
  • 13:22 sukhe: running homer for Gerrit: 856946 in cr*-ulsfo*
  • 13:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P39724 and previous config saved to /var/cache/conftool/dbconfig/20221115-132103-marostegui.json
  • 13:20 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 13:20 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 13:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 13335
  • 13:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39723 and previous config saved to /var/cache/conftool/dbconfig/20221115-131710-marostegui.json
  • 13:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 13:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 13:08 moritzm: failover ganeti master in ulsfo to ganeti4005
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P39722 and previous config saved to /var/cache/conftool/dbconfig/20221115-130557-marostegui.json
  • 13:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321130)', diff saved to https://phabricator.wikimedia.org/P39721 and previous config saved to /var/cache/conftool/dbconfig/20221115-125950-marostegui.json
  • 12:59 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 13335
  • 12:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet
  • 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39720 and previous config saved to /var/cache/conftool/dbconfig/20221115-125050-marostegui.json
  • 12:49 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2003-dev.codfw.wmnet with OS bullseye
  • 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39719 and previous config saved to /var/cache/conftool/dbconfig/20221115-124830-marostegui.json
  • 12:48 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 12:48 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T321126)', diff saved to https://phabricator.wikimedia.org/P39718 and previous config saved to /var/cache/conftool/dbconfig/20221115-124808-marostegui.json
  • 12:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet
  • 12:45 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on idp-test1002.wikimedia.org with reason: experiment with CAS 6.6
  • 12:45 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on idp-test1002.wikimedia.org with reason: experiment with CAS 6.6
  • 12:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P39717 and previous config saved to /var/cache/conftool/dbconfig/20221115-124443-marostegui.json
  • 12:43 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=varnish-fe
  • 12:43 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=ats-be
  • 12:43 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=ats-tls
  • 12:37 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39716 and previous config saved to /var/cache/conftool/dbconfig/20221115-123735-root.json
  • 12:36 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39715 and previous config saved to /var/cache/conftool/dbconfig/20221115-123326-root.json
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39714 and previous config saved to /var/cache/conftool/dbconfig/20221115-123302-marostegui.json
  • 12:31 aborrero@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
  • 12:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P39713 and previous config saved to /var/cache/conftool/dbconfig/20221115-122937-marostegui.json
  • 12:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet
  • 12:22 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39712 and previous config saved to /var/cache/conftool/dbconfig/20221115-122230-root.json
  • 12:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet
  • 12:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39711 and previous config saved to /var/cache/conftool/dbconfig/20221115-121821-root.json
  • 12:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39710 and previous config saved to /var/cache/conftool/dbconfig/20221115-121755-marostegui.json
  • 12:16 aborrero@cumin1001: START - Cookbook sre.hosts.reimage for host cloudgw2003-dev.codfw.wmnet with OS bullseye
  • 12:14 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321130)', diff saved to https://phabricator.wikimedia.org/P39709 and previous config saved to /var/cache/conftool/dbconfig/20221115-121431-marostegui.json
  • 12:13 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:11 hnowlan: resyncing maps2005 replica
  • 12:11 hnowlan@cumin1001: START - Cookbook sre.postgresql.postgres-init
  • 12:07 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39708 and previous config saved to /var/cache/conftool/dbconfig/20221115-120725-root.json
  • 12:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T321130)', diff saved to https://phabricator.wikimedia.org/P39707 and previous config saved to /var/cache/conftool/dbconfig/20221115-120502-marostegui.json
  • 12:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:04 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 12:04 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 12:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39706 and previous config saved to /var/cache/conftool/dbconfig/20221115-120316-root.json
  • 12:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T321126)', diff saved to https://phabricator.wikimedia.org/P39705 and previous config saved to /var/cache/conftool/dbconfig/20221115-120249-marostegui.json
  • 12:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T321126)', diff saved to https://phabricator.wikimedia.org/P39704 and previous config saved to /var/cache/conftool/dbconfig/20221115-120030-marostegui.json
  • 12:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 12:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 12:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T321126)', diff saved to https://phabricator.wikimedia.org/P39703 and previous config saved to /var/cache/conftool/dbconfig/20221115-120009-marostegui.json
  • 11:52 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39702 and previous config saved to /var/cache/conftool/dbconfig/20221115-115220-root.json
  • 11:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39701 and previous config saved to /var/cache/conftool/dbconfig/20221115-114812-root.json
  • 11:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 11:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 11:45 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti6003.drmrs.wmnet
  • 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P39700 and previous config saved to /var/cache/conftool/dbconfig/20221115-114502-marostegui.json
  • 11:42 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2002-dev.codfw.wmnet with OS bullseye
  • 11:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39699 and previous config saved to /var/cache/conftool/dbconfig/20221115-114216-marostegui.json
  • 11:37 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39698 and previous config saved to /var/cache/conftool/dbconfig/20221115-113715-root.json
  • 11:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
  • 11:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39697 and previous config saved to /var/cache/conftool/dbconfig/20221115-113307-root.json
  • 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P39696 and previous config saved to /var/cache/conftool/dbconfig/20221115-112956-marostegui.json
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39695 and previous config saved to /var/cache/conftool/dbconfig/20221115-112709-marostegui.json
  • 11:22 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
  • 11:22 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39694 and previous config saved to /var/cache/conftool/dbconfig/20221115-112210-root.json
  • 11:20 moritzm: failover ganeti master in drmrs/B12 to ganeti6001
  • 11:20 aborrero@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
  • 11:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39693 and previous config saved to /var/cache/conftool/dbconfig/20221115-111802-root.json
  • 11:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T321126)', diff saved to https://phabricator.wikimedia.org/P39692 and previous config saved to /var/cache/conftool/dbconfig/20221115-111449-marostegui.json
  • 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T321126)', diff saved to https://phabricator.wikimedia.org/P39691 and previous config saved to /var/cache/conftool/dbconfig/20221115-111229-marostegui.json
  • 11:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39690 and previous config saved to /var/cache/conftool/dbconfig/20221115-111203-marostegui.json
  • 11:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 11:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39689 and previous config saved to /var/cache/conftool/dbconfig/20221115-111150-marostegui.json
  • 11:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet
  • 11:07 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39688 and previous config saved to /var/cache/conftool/dbconfig/20221115-110705-root.json
  • 11:05 aborrero@cumin1001: START - Cookbook sre.hosts.reimage for host cloudgw2002-dev.codfw.wmnet with OS bullseye
  • 11:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39687 and previous config saved to /var/cache/conftool/dbconfig/20221115-110257-root.json
  • 10:59 mfossati@deploy1002: Finished deploy [airflow-dags/platform_eng@3bb99c2]: (no justification provided) (duration: 00m 05s)
  • 10:59 mfossati@deploy1002: Started deploy [airflow-dags/platform_eng@3bb99c2]: (no justification provided)
  • 10:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39686 and previous config saved to /var/cache/conftool/dbconfig/20221115-105657-marostegui.json
  • 10:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P39685 and previous config saved to /var/cache/conftool/dbconfig/20221115-105644-marostegui.json
  • 10:52 marostegui@cumin1001: dbctl commit (dc=all): 'db2166 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39684 and previous config saved to /var/cache/conftool/dbconfig/20221115-105200-root.json
  • 10:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2162 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39683 and previous config saved to /var/cache/conftool/dbconfig/20221115-104752-root.json
  • 10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39682 and previous config saved to /var/cache/conftool/dbconfig/20221115-104435-marostegui.json
  • 10:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 10:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39681 and previous config saved to /var/cache/conftool/dbconfig/20221115-104402-marostegui.json
  • 10:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P39680 and previous config saved to /var/cache/conftool/dbconfig/20221115-104137-marostegui.json
  • 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39679 and previous config saved to /var/cache/conftool/dbconfig/20221115-102856-marostegui.json
  • 10:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39678 and previous config saved to /var/cache/conftool/dbconfig/20221115-102631-marostegui.json
  • 10:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39677 and previous config saved to /var/cache/conftool/dbconfig/20221115-102409-marostegui.json
  • 10:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39676 and previous config saved to /var/cache/conftool/dbconfig/20221115-102319-marostegui.json
  • 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39675 and previous config saved to /var/cache/conftool/dbconfig/20221115-101349-marostegui.json
  • 10:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P39674 and previous config saved to /var/cache/conftool/dbconfig/20221115-100812-marostegui.json
  • 09:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39673 and previous config saved to /var/cache/conftool/dbconfig/20221115-095843-marostegui.json
  • 09:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P39672 and previous config saved to /var/cache/conftool/dbconfig/20221115-095306-marostegui.json
  • 09:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39671 and previous config saved to /var/cache/conftool/dbconfig/20221115-094552-marostegui.json
  • 09:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39670 and previous config saved to /var/cache/conftool/dbconfig/20221115-093758-marostegui.json
  • 09:36 moritzm: draining ganeti1022 for eventual reimage T311687
  • 09:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39669 and previous config saved to /var/cache/conftool/dbconfig/20221115-093539-marostegui.json
  • 09:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 09:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 09:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39668 and previous config saved to /var/cache/conftool/dbconfig/20221115-093518-marostegui.json
  • 09:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P39667 and previous config saved to /var/cache/conftool/dbconfig/20221115-092011-marostegui.json
  • 09:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts bast5001.wikimedia.org
  • 09:14 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:12 jmm@cumin2002: START - Cookbook sre.dns.netbox
  • 09:07 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts bast5001.wikimedia.org
  • 09:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P39666 and previous config saved to /var/cache/conftool/dbconfig/20221115-090505-marostegui.json
  • 09:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2162', diff saved to https://phabricator.wikimedia.org/P39664 and previous config saved to /var/cache/conftool/dbconfig/20221115-090058-root.json
  • 08:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1138.eqiad.wmnet with reason: Maintenance
  • 08:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1138.eqiad.wmnet with reason: Maintenance
  • 08:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39663 and previous config saved to /var/cache/conftool/dbconfig/20221115-084959-marostegui.json
  • 08:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T321126)', diff saved to https://phabricator.wikimedia.org/P39662 and previous config saved to /var/cache/conftool/dbconfig/20221115-084637-marostegui.json
  • 08:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 08:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 08:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 08:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 08:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 08:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 08:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 08:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 08:06 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1021.eqiad.wmnet to cluster eqiad and group D
  • 08:05 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1021.eqiad.wmnet to cluster eqiad and group D
  • 08:05 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1021.eqiad.wmnet
  • 07:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1021.eqiad.wmnet
  • 07:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39661 and previous config saved to /var/cache/conftool/dbconfig/20221115-072202-marostegui.json
  • 07:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P39660 and previous config saved to /var/cache/conftool/dbconfig/20221115-070655-marostegui.json
  • 06:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P39659 and previous config saved to /var/cache/conftool/dbconfig/20221115-065149-marostegui.json
  • 06:43 robh: all in rack work in eqsin is complete for today
  • 06:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39658 and previous config saved to /var/cache/conftool/dbconfig/20221115-063642-marostegui.json
  • 06:30 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1008.eqiad.wmnet
  • 06:26 robh@cumin1001: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5032
  • 06:26 robh@cumin1001: START - Cookbook sre.network.configure-switch-interfaces for host cp5032
  • 06:25 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 06:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2179 (T321130)', diff saved to https://phabricator.wikimedia.org/P39657 and previous config saved to /var/cache/conftool/dbconfig/20221115-062400-marostegui.json
  • 06:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 06:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 06:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T321130)', diff saved to https://phabricator.wikimedia.org/P39656 and previous config saved to /var/cache/conftool/dbconfig/20221115-062339-marostegui.json
  • 06:23 robh@cumin1001: START - Cookbook sre.dns.netbox
  • 06:20 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1008.eqiad.wmnet
  • 06:19 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1007.eqiad.wmnet
  • 06:18 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1005.eqiad.wmnet
  • 06:15 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-tool1007.eqiad.wmnet
  • 06:13 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host karapace1001.eqiad.wmnet
  • 06:09 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host karapace1001.eqiad.wmnet
  • 06:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P39655 and previous config saved to /var/cache/conftool/dbconfig/20221115-060832-marostegui.json
  • 06:06 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1005.eqiad.wmnet
  • 06:05 stevemunene@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host stat1005.eqiad.wmnet
  • 06:05 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1005.eqiad.wmnet
  • 05:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P39654 and previous config saved to /var/cache/conftool/dbconfig/20221115-055326-marostegui.json
  • 05:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T321130)', diff saved to https://phabricator.wikimedia.org/P39653 and previous config saved to /var/cache/conftool/dbconfig/20221115-053819-marostegui.json
  • 05:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2172 (T321130)', diff saved to https://phabricator.wikimedia.org/P39652 and previous config saved to /var/cache/conftool/dbconfig/20221115-052543-marostegui.json
  • 05:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 05:25 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 05:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321130)', diff saved to https://phabricator.wikimedia.org/P39651 and previous config saved to /var/cache/conftool/dbconfig/20221115-052521-marostegui.json
  • 05:13 robh: ~5AM UTC when plugging a new host into asw1-ulsfo, the virtual chassis crashed and rebooted, causing loss of connectivity to hosts for a very short period
  • 05:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P39650 and previous config saved to /var/cache/conftool/dbconfig/20221115-051015-marostegui.json
  • 04:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P39649 and previous config saved to /var/cache/conftool/dbconfig/20221115-045508-marostegui.json
  • 04:48 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 04:40 mwpresync@deploy1002: Pruned MediaWiki: 1.40.0-wmf.7 (duration: 02m 01s)
  • 04:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321130)', diff saved to https://phabricator.wikimedia.org/P39648 and previous config saved to /var/cache/conftool/dbconfig/20221115-044002-marostegui.json
  • 04:38 mwpresync@deploy1002: Finished scap: testwikis wikis to 1.40.0-wmf.10 refs T320515 (duration: 36m 14s)
  • 04:37 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 04:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2155 (T321130)', diff saved to https://phabricator.wikimedia.org/P39647 and previous config saved to /var/cache/conftool/dbconfig/20221115-042713-marostegui.json
  • 04:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 04:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 04:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 04:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 04:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39646 and previous config saved to /var/cache/conftool/dbconfig/20221115-042647-marostegui.json
  • 04:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 04:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 04:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 04:15 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 04:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P39645 and previous config saved to /var/cache/conftool/dbconfig/20221115-041140-marostegui.json
  • 04:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 04:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 04:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 04:03 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 04:02 mwpresync@deploy1002: Started scap: testwikis wikis to 1.40.0-wmf.10 refs T320515
  • 03:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P39644 and previous config saved to /var/cache/conftool/dbconfig/20221115-035634-marostegui.json
  • 03:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39643 and previous config saved to /var/cache/conftool/dbconfig/20221115-034127-marostegui.json
  • 03:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 03:32 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 03:32 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 03:32 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 03:29 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39642 and previous config saved to /var/cache/conftool/dbconfig/20221115-032929-marostegui.json
  • 03:29 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 03:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 03:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 03:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 03:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39641 and previous config saved to /var/cache/conftool/dbconfig/20221115-031826-marostegui.json
  • 03:16 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 03:12 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 03:12 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 03:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 03:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P39640 and previous config saved to /var/cache/conftool/dbconfig/20221115-030320-marostegui.json
  • 02:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P39639 and previous config saved to /var/cache/conftool/dbconfig/20221115-024813-marostegui.json
  • 02:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39638 and previous config saved to /var/cache/conftool/dbconfig/20221115-023307-marostegui.json
  • 02:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39637 and previous config saved to /var/cache/conftool/dbconfig/20221115-022052-marostegui.json
  • 02:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 02:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 02:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39636 and previous config saved to /var/cache/conftool/dbconfig/20221115-022030-marostegui.json
  • 02:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P39635 and previous config saved to /var/cache/conftool/dbconfig/20221115-020523-marostegui.json
  • 01:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P39634 and previous config saved to /var/cache/conftool/dbconfig/20221115-015017-marostegui.json
  • 01:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39633 and previous config saved to /var/cache/conftool/dbconfig/20221115-013510-marostegui.json
  • 01:28 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS buster
  • 01:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39632 and previous config saved to /var/cache/conftool/dbconfig/20221115-012313-marostegui.json
  • 01:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 01:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 01:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39631 and previous config saved to /var/cache/conftool/dbconfig/20221115-012251-marostegui.json
  • 01:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P39630 and previous config saved to /var/cache/conftool/dbconfig/20221115-010745-marostegui.json
  • 01:05 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 01:02 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 00:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P39629 and previous config saved to /var/cache/conftool/dbconfig/20221115-005238-marostegui.json
  • 00:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39628 and previous config saved to /var/cache/conftool/dbconfig/20221115-003732-marostegui.json
  • 00:33 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS buster
  • 00:29 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host arclamp2001.codfw.wmnet with OS bullseye
  • 00:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39627 and previous config saved to /var/cache/conftool/dbconfig/20221115-002514-marostegui.json
  • 00:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 00:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 00:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T321130)', diff saved to https://phabricator.wikimedia.org/P39626 and previous config saved to /var/cache/conftool/dbconfig/20221115-002441-marostegui.json
  • 00:14 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on arclamp2001.codfw.wmnet with reason: host reimage
  • 00:13 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host puppetdb2003.codfw.wmnet with OS bullseye
  • 00:11 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on arclamp2001.codfw.wmnet with reason: host reimage
  • 00:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P39625 and previous config saved to /var/cache/conftool/dbconfig/20221115-000935-marostegui.json

2022-11-14

  • 23:58 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on puppetdb2003.codfw.wmnet with reason: host reimage
  • 23:55 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on puppetdb2003.codfw.wmnet with reason: host reimage
  • 23:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P39624 and previous config saved to /var/cache/conftool/dbconfig/20221114-235429-marostegui.json
  • 23:52 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host arclamp2001.codfw.wmnet with OS bullseye
  • 23:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T321130)', diff saved to https://phabricator.wikimedia.org/P39623 and previous config saved to /var/cache/conftool/dbconfig/20221114-233922-marostegui.json
  • 23:36 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host puppetdb2003.codfw.wmnet with OS bullseye
  • 23:32 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov2004.codfw.wmnet with OS bullseye
  • 23:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39622 and previous config saved to /var/cache/conftool/dbconfig/20221114-232744-marostegui.json
  • 23:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2119 (T321130)', diff saved to https://phabricator.wikimedia.org/P39621 and previous config saved to /var/cache/conftool/dbconfig/20221114-232714-marostegui.json
  • 23:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 23:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 23:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T321130)', diff saved to https://phabricator.wikimedia.org/P39620 and previous config saved to /var/cache/conftool/dbconfig/20221114-232653-marostegui.json
  • 23:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P39619 and previous config saved to /var/cache/conftool/dbconfig/20221114-231238-marostegui.json
  • 23:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P39618 and previous config saved to /var/cache/conftool/dbconfig/20221114-231146-marostegui.json
  • 23:10 eileen: civicrm upgraded from 93fa3f37 to 3eba6ad3
  • 22:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P39617 and previous config saved to /var/cache/conftool/dbconfig/20221114-225730-marostegui.json
  • 22:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P39616 and previous config saved to /var/cache/conftool/dbconfig/20221114-225638-marostegui.json
  • 22:56 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 22:55 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 22:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39614 and previous config saved to /var/cache/conftool/dbconfig/20221114-224224-marostegui.json
  • 22:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T321130)', diff saved to https://phabricator.wikimedia.org/P39613 and previous config saved to /var/cache/conftool/dbconfig/20221114-224132-marostegui.json
  • 22:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39612 and previous config saved to /var/cache/conftool/dbconfig/20221114-224006-marostegui.json
  • 22:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 22:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 22:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39611 and previous config saved to /var/cache/conftool/dbconfig/20221114-223945-marostegui.json
  • 22:31 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 22:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2110 (T321130)', diff saved to https://phabricator.wikimedia.org/P39610 and previous config saved to /var/cache/conftool/dbconfig/20221114-222706-marostegui.json
  • 22:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 22:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 22:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T321130)', diff saved to https://phabricator.wikimedia.org/P39609 and previous config saved to /var/cache/conftool/dbconfig/20221114-222644-marostegui.json
  • 22:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P39608 and previous config saved to /var/cache/conftool/dbconfig/20221114-222438-marostegui.json
  • 22:21 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 22:19 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 22:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P39607 and previous config saved to /var/cache/conftool/dbconfig/20221114-221138-marostegui.json
  • 22:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P39606 and previous config saved to /var/cache/conftool/dbconfig/20221114-220932-marostegui.json
  • 22:09 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 22:09 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 22:03 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 21:58 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P39605 and previous config saved to /var/cache/conftool/dbconfig/20221114-215631-marostegui.json
  • 21:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39604 and previous config saved to /var/cache/conftool/dbconfig/20221114-215425-marostegui.json
  • 21:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39603 and previous config saved to /var/cache/conftool/dbconfig/20221114-215204-marostegui.json
  • 21:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 21:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 21:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39602 and previous config saved to /var/cache/conftool/dbconfig/20221114-215143-marostegui.json
  • 21:48 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T321130)', diff saved to https://phabricator.wikimedia.org/P39601 and previous config saved to /var/cache/conftool/dbconfig/20221114-214125-marostegui.json
  • 21:38 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P39600 and previous config saved to /var/cache/conftool/dbconfig/20221114-213636-marostegui.json
  • 21:35 mutante: phab2002 - systemctl start phd, debug why it still fails
  • 21:35 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:29 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2106 (T321130)', diff saved to https://phabricator.wikimedia.org/P39599 and previous config saved to /var/cache/conftool/dbconfig/20221114-212934-marostegui.json
  • 21:29 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 21:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 21:25 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 21:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P39598 and previous config saved to /var/cache/conftool/dbconfig/20221114-212130-marostegui.json
  • 21:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 21:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 21:11 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 21:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39597 and previous config saved to /var/cache/conftool/dbconfig/20221114-210853-marostegui.json
  • 21:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39596 and previous config saved to /var/cache/conftool/dbconfig/20221114-210623-marostegui.json
  • 21:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39595 and previous config saved to /var/cache/conftool/dbconfig/20221114-210503-marostegui.json
  • 21:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 21:04 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 21:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T321126)', diff saved to https://phabricator.wikimedia.org/P39594 and previous config saved to /var/cache/conftool/dbconfig/20221114-210430-marostegui.json
  • 21:01 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39593 and previous config saved to /var/cache/conftool/dbconfig/20221114-205347-marostegui.json
  • 20:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P39592 and previous config saved to /var/cache/conftool/dbconfig/20221114-204924-marostegui.json
  • 20:47 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 20:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39591 and previous config saved to /var/cache/conftool/dbconfig/20221114-203841-marostegui.json
  • 20:37 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P39590 and previous config saved to /var/cache/conftool/dbconfig/20221114-203417-marostegui.json
  • 20:34 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 20:34 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 20:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39589 and previous config saved to /var/cache/conftool/dbconfig/20221114-202334-marostegui.json
  • 20:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T321126)', diff saved to https://phabricator.wikimedia.org/P39588 and previous config saved to /var/cache/conftool/dbconfig/20221114-201911-marostegui.json
  • 20:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T321126)', diff saved to https://phabricator.wikimedia.org/P39587 and previous config saved to /var/cache/conftool/dbconfig/20221114-201650-marostegui.json
  • 20:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 20:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 20:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39586 and previous config saved to /var/cache/conftool/dbconfig/20221114-201556-marostegui.json
  • 20:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P39585 and previous config saved to /var/cache/conftool/dbconfig/20221114-200050-marostegui.json
  • 19:55 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: Upgrade horizon to Z to prepare for Openstack upgrades past Wallaby -- T305828 (duration: 04m 41s)
  • 19:50 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: Upgrade horizon to Z to prepare for Openstack upgrades past Wallaby -- T305828
  • 19:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P39583 and previous config saved to /var/cache/conftool/dbconfig/20221114-194543-marostegui.json
  • 19:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39582 and previous config saved to /var/cache/conftool/dbconfig/20221114-193037-marostegui.json
  • 19:29 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS bullseye
  • 19:28 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2129 (T321126)', diff saved to https://phabricator.wikimedia.org/P39581 and previous config saved to /var/cache/conftool/dbconfig/20221114-192816-marostegui.json
  • 19:28 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 19:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 19:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321126)', diff saved to https://phabricator.wikimedia.org/P39580 and previous config saved to /var/cache/conftool/dbconfig/20221114-192754-marostegui.json
  • 19:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1199 (T321130)', diff saved to https://phabricator.wikimedia.org/P39579 and previous config saved to /var/cache/conftool/dbconfig/20221114-192318-marostegui.json
  • 19:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 19:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 19:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321130)', diff saved to https://phabricator.wikimedia.org/P39578 and previous config saved to /var/cache/conftool/dbconfig/20221114-192257-marostegui.json
  • 19:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P39577 and previous config saved to /var/cache/conftool/dbconfig/20221114-191247-marostegui.json
  • 19:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P39576 and previous config saved to /var/cache/conftool/dbconfig/20221114-190750-marostegui.json
  • 18:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P39575 and previous config saved to /var/cache/conftool/dbconfig/20221114-185741-marostegui.json
  • 18:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P39574 and previous config saved to /var/cache/conftool/dbconfig/20221114-185244-marostegui.json
  • 18:50 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:45 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321126)', diff saved to https://phabricator.wikimedia.org/P39573 and previous config saved to /var/cache/conftool/dbconfig/20221114-184235-marostegui.json
  • 18:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T321126)', diff saved to https://phabricator.wikimedia.org/P39572 and previous config saved to /var/cache/conftool/dbconfig/20221114-184014-marostegui.json
  • 18:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 18:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 18:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321126)', diff saved to https://phabricator.wikimedia.org/P39571 and previous config saved to /var/cache/conftool/dbconfig/20221114-183952-marostegui.json
  • 18:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321130)', diff saved to https://phabricator.wikimedia.org/P39570 and previous config saved to /var/cache/conftool/dbconfig/20221114-183738-marostegui.json
  • 18:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1190 (T321130)', diff saved to https://phabricator.wikimedia.org/P39569 and previous config saved to /var/cache/conftool/dbconfig/20221114-182506-marostegui.json
  • 18:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 18:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 18:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P39568 and previous config saved to /var/cache/conftool/dbconfig/20221114-182446-marostegui.json
  • 18:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39567 and previous config saved to /var/cache/conftool/dbconfig/20221114-182445-marostegui.json
  • 18:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 18:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 18:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T322618)', diff saved to https://phabricator.wikimedia.org/P39566 and previous config saved to /var/cache/conftool/dbconfig/20221114-181700-ladsgroup.json
  • 18:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39565 and previous config saved to /var/cache/conftool/dbconfig/20221114-180938-marostegui.json
  • 18:08 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['arclamp2001']
  • 18:07 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['arclamp2001']
  • 18:07 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['arclamp2001']
  • 18:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P39564 and previous config saved to /var/cache/conftool/dbconfig/20221114-180153-ladsgroup.json
  • 17:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39563 and previous config saved to /var/cache/conftool/dbconfig/20221114-175432-marostegui.json
  • 17:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T321126)', diff saved to https://phabricator.wikimedia.org/P39562 and previous config saved to /var/cache/conftool/dbconfig/20221114-175213-marostegui.json
  • 17:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 17:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 17:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 17:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 17:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T321126)', diff saved to https://phabricator.wikimedia.org/P39561 and previous config saved to /var/cache/conftool/dbconfig/20221114-175129-marostegui.json
  • 17:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P39560 and previous config saved to /var/cache/conftool/dbconfig/20221114-174647-ladsgroup.json
  • 17:42 hashar: Restored CI caching mechanism which has been serving stalled caches since March 29th 2022 :-\ T323051
  • 17:42 hashar: Restored CI caching mechanism which has been serving stalled caches since March 29th 2022 :-\ T307334
  • 17:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39559 and previous config saved to /var/cache/conftool/dbconfig/20221114-173925-marostegui.json
  • 17:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P39558 and previous config saved to /var/cache/conftool/dbconfig/20221114-173622-marostegui.json
  • 17:34 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['arclamp2001']
  • 17:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T322618)', diff saved to https://phabricator.wikimedia.org/P39557 and previous config saved to /var/cache/conftool/dbconfig/20221114-173140-ladsgroup.json
  • 17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1197 (T322618)', diff saved to https://phabricator.wikimedia.org/P39556 and previous config saved to /var/cache/conftool/dbconfig/20221114-172929-ladsgroup.json
  • 17:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 17:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 17:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T322618)', diff saved to https://phabricator.wikimedia.org/P39555 and previous config saved to /var/cache/conftool/dbconfig/20221114-172846-ladsgroup.json
  • 17:25 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:24 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 17:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P39554 and previous config saved to /var/cache/conftool/dbconfig/20221114-172116-marostegui.json
  • 17:13 dancy@deploy1002: Installation of scap version "4.28.1" completed for 559 hosts
  • 17:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P39553 and previous config saved to /var/cache/conftool/dbconfig/20221114-171340-ladsgroup.json
  • 17:13 dancy@deploy1002: Installing scap version "4.28.1" for 559 hosts
  • 17:10 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 2:00:00 on wcqs1002.eqiad.wmnet with reason: Reboot for kernel update
  • 17:10 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 2:00:00 on wcqs1002.eqiad.wmnet with reason: Reboot for kernel update
  • 17:09 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs4005.ulsfo.wmnet
  • 17:09 sukhe@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:08 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wcqs[2001-2003].codfw.wmnet with reason: Reboot for kernel update
  • 17:07 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wcqs[2001-2003].codfw.wmnet with reason: Reboot for kernel update
  • 17:07 ryankemper@cumin1001: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on 12 hosts with reason: Reboot for kernel update
  • 17:07 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 12 hosts with reason: Reboot for kernel update
  • 17:07 sukhe@cumin2002: START - Cookbook sre.dns.netbox
  • 17:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T321126)', diff saved to https://phabricator.wikimedia.org/P39552 and previous config saved to /var/cache/conftool/dbconfig/20221114-170609-marostegui.json
  • 17:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1201 (T321126)', diff saved to https://phabricator.wikimedia.org/P39551 and previous config saved to /var/cache/conftool/dbconfig/20221114-170357-marostegui.json
  • 17:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 17:03 jdrewniak@deploy1002: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 03m 45s)
  • 17:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 17:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T321126)', diff saved to https://phabricator.wikimedia.org/P39550 and previous config saved to /var/cache/conftool/dbconfig/20221114-170325-marostegui.json
  • 17:03 sukhe@cumin2002: START - Cookbook sre.hosts.decommission for hosts lvs4005.ulsfo.wmnet
  • 16:59 jdrewniak@deploy1002: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 03m 58s)
  • 16:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P39549 and previous config saved to /var/cache/conftool/dbconfig/20221114-165833-ladsgroup.json
  • 16:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P39548 and previous config saved to /var/cache/conftool/dbconfig/20221114-164818-marostegui.json
  • 16:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T322618)', diff saved to https://phabricator.wikimedia.org/P39547 and previous config saved to /var/cache/conftool/dbconfig/20221114-164327-ladsgroup.json
  • 16:41 sukhe: depooled lvs4005
  • 16:41 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4005.ulsfo.wmnet with reason: downtimed, in the process of decom
  • 16:41 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on lvs4005.ulsfo.wmnet with reason: downtimed, in the process of decom
  • 16:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1188 (T322618)', diff saved to https://phabricator.wikimedia.org/P39546 and previous config saved to /var/cache/conftool/dbconfig/20221114-164015-ladsgroup.json
  • 16:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 16:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39545 and previous config saved to /var/cache/conftool/dbconfig/20221114-163954-ladsgroup.json
  • 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T321130)', diff saved to https://phabricator.wikimedia.org/P39544 and previous config saved to /var/cache/conftool/dbconfig/20221114-163910-marostegui.json
  • 16:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 16:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 16:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2166.codfw.wmnet with reason: Host crashed T323040
  • 16:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2166.codfw.wmnet with reason: Host crashed T323040
  • 16:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P39543 and previous config saved to /var/cache/conftool/dbconfig/20221114-163312-marostegui.json
  • 16:30 sukhe: cr4-ulsfo: set routing-options static route 198.35.26.96/28 next-hop 10.128.0.18 [lvs4005 decomm]
  • 16:29 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 16:28 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 16:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T321130)', diff saved to https://phabricator.wikimedia.org/P39542 and previous config saved to /var/cache/conftool/dbconfig/20221114-162851-marostegui.json
  • 16:28 sukhe: cr3-ulsfo: set routing-options static route 198.35.26.96/28 next-hop 10.128.0.18 [lvs4005 decomm]
  • 16:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P39541 and previous config saved to /var/cache/conftool/dbconfig/20221114-162448-ladsgroup.json
  • 16:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T321126)', diff saved to https://phabricator.wikimedia.org/P39540 and previous config saved to /var/cache/conftool/dbconfig/20221114-161804-marostegui.json
  • 16:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T321126)', diff saved to https://phabricator.wikimedia.org/P39539 and previous config saved to /var/cache/conftool/dbconfig/20221114-161553-marostegui.json
  • 16:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 16:15 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['dbprov2004']
  • 16:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 16:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39537 and previous config saved to /var/cache/conftool/dbconfig/20221114-161520-marostegui.json
  • 16:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P39536 and previous config saved to /var/cache/conftool/dbconfig/20221114-161344-marostegui.json
  • 16:13 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['puppetdb2003']
  • 16:12 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['puppetdb2003']
  • 16:12 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['puppetdb2003']
  • 16:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P39535 and previous config saved to /var/cache/conftool/dbconfig/20221114-160941-ladsgroup.json
  • 16:08 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dbprov2004']
  • 16:08 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['dbprov2004']
  • 16:03 sukhe: reprepro -C main include bullseye-wikimedia varnish-modules_0.15.0-2_amd64.changes: T321309
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'MySQL issues', diff saved to https://phabricator.wikimedia.org/P39534 and previous config saved to /var/cache/conftool/dbconfig/20221114-160140-ladsgroup.json
  • 16:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P39533 and previous config saved to /var/cache/conftool/dbconfig/20221114-160014-marostegui.json
  • 15:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P39532 and previous config saved to /var/cache/conftool/dbconfig/20221114-155838-marostegui.json
  • 15:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39531 and previous config saved to /var/cache/conftool/dbconfig/20221114-155435-ladsgroup.json
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39530 and previous config saved to /var/cache/conftool/dbconfig/20221114-155222-ladsgroup.json
  • 15:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39529 and previous config saved to /var/cache/conftool/dbconfig/20221114-155201-ladsgroup.json
  • 15:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P39528 and previous config saved to /var/cache/conftool/dbconfig/20221114-154507-marostegui.json
  • 15:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T321130)', diff saved to https://phabricator.wikimedia.org/P39527 and previous config saved to /var/cache/conftool/dbconfig/20221114-154331-marostegui.json
  • 15:42 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['puppetdb2003']
  • 15:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T322618)', diff saved to https://phabricator.wikimedia.org/P39526 and previous config saved to /var/cache/conftool/dbconfig/20221114-153903-ladsgroup.json
  • 15:38 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dbprov2004']
  • 15:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P39525 and previous config saved to /var/cache/conftool/dbconfig/20221114-153654-ladsgroup.json
  • 15:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T321130)', diff saved to https://phabricator.wikimedia.org/P39524 and previous config saved to /var/cache/conftool/dbconfig/20221114-153030-marostegui.json
  • 15:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 15:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39523 and previous config saved to /var/cache/conftool/dbconfig/20221114-153001-marostegui.json
  • 15:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 15:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T321130)', diff saved to https://phabricator.wikimedia.org/P39522 and previous config saved to /var/cache/conftool/dbconfig/20221114-152936-marostegui.json
  • 15:28 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1021.eqiad.wmnet with OS bullseye
  • 15:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T321126)', diff saved to https://phabricator.wikimedia.org/P39521 and previous config saved to /var/cache/conftool/dbconfig/20221114-152749-marostegui.json
  • 15:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 15:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 15:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321126)', diff saved to https://phabricator.wikimedia.org/P39520 and previous config saved to /var/cache/conftool/dbconfig/20221114-152728-marostegui.json
  • 15:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39519 and previous config saved to /var/cache/conftool/dbconfig/20221114-152356-ladsgroup.json
  • 15:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P39518 and previous config saved to /var/cache/conftool/dbconfig/20221114-152148-ladsgroup.json
  • 15:15 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 15:15 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 15:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39516 and previous config saved to /var/cache/conftool/dbconfig/20221114-151428-marostegui.json
  • 15:13 urandom: initiating Cassandra bootstrap, aqs1019-a -- T307802
  • 15:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P39515 and previous config saved to /var/cache/conftool/dbconfig/20221114-151222-marostegui.json
  • 15:11 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 15:10 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 15:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1021.eqiad.wmnet with reason: host reimage
  • 15:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39514 and previous config saved to /var/cache/conftool/dbconfig/20221114-150850-ladsgroup.json
  • 15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39513 and previous config saved to /var/cache/conftool/dbconfig/20221114-150642-ladsgroup.json
  • 15:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39512 and previous config saved to /var/cache/conftool/dbconfig/20221114-150531-ladsgroup.json
  • 15:05 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1021.eqiad.wmnet with reason: host reimage
  • 15:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 15:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 15:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T322618)', diff saved to https://phabricator.wikimedia.org/P39511 and previous config saved to /var/cache/conftool/dbconfig/20221114-150509-ladsgroup.json
  • 14:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39510 and previous config saved to /var/cache/conftool/dbconfig/20221114-145921-marostegui.json
  • 14:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P39509 and previous config saved to /var/cache/conftool/dbconfig/20221114-145715-marostegui.json
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T322618)', diff saved to https://phabricator.wikimedia.org/P39508 and previous config saved to /var/cache/conftool/dbconfig/20221114-145343-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2175 (T322618)', diff saved to https://phabricator.wikimedia.org/P39507 and previous config saved to /var/cache/conftool/dbconfig/20221114-145122-ladsgroup.json
  • 14:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 14:51 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1021.eqiad.wmnet with OS bullseye
  • 14:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39506 and previous config saved to /var/cache/conftool/dbconfig/20221114-145101-ladsgroup.json
  • 14:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39505 and previous config saved to /var/cache/conftool/dbconfig/20221114-145003-ladsgroup.json
  • 14:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1021.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 14:49 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1021.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 14:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T321130)', diff saved to https://phabricator.wikimedia.org/P39504 and previous config saved to /var/cache/conftool/dbconfig/20221114-144415-marostegui.json
  • 14:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321126)', diff saved to https://phabricator.wikimedia.org/P39503 and previous config saved to /var/cache/conftool/dbconfig/20221114-144209-marostegui.json
  • 14:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T321126)', diff saved to https://phabricator.wikimedia.org/P39502 and previous config saved to /var/cache/conftool/dbconfig/20221114-143957-marostegui.json
  • 14:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 14:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 14:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T321126)', diff saved to https://phabricator.wikimedia.org/P39501 and previous config saved to /var/cache/conftool/dbconfig/20221114-143936-marostegui.json
  • 14:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39500 and previous config saved to /var/cache/conftool/dbconfig/20221114-143554-ladsgroup.json
  • 14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39499 and previous config saved to /var/cache/conftool/dbconfig/20221114-143456-ladsgroup.json
  • 14:33 taavi: ^ correction, starting it on mwmaint1002, not deploy1002
  • 14:32 taavi: START taavi@deploy1002:~$ foreachwikiindblist group1 extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all | tee T315510.log # T315510
  • 14:30 taavi@deploy1002: Finished scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353) (duration: 07m 05s)
  • 14:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P39498 and previous config saved to /var/cache/conftool/dbconfig/20221114-142429-marostegui.json
  • 14:23 taavi@deploy1002: taavi and matmarex: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 14:23 taavi@deploy1002: Started scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353)
  • 14:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39497 and previous config saved to /var/cache/conftool/dbconfig/20221114-142048-ladsgroup.json
  • 14:20 taavi@deploy1002: Finished scap: Backport for Use legacy DiscussionTools heading markup except on beta cluster (T314714), ThreadItemStore: Handle race conditions when finding/inserting outside of transaction (T322701), persistRevisionThreadItems: Print time taken (duration: 06m 14s)
  • 14:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T322618)', diff saved to https://phabricator.wikimedia.org/P39496 and previous config saved to /var/cache/conftool/dbconfig/20221114-141950-ladsgroup.json
  • 14:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T322618)', diff saved to https://phabricator.wikimedia.org/P39495 and previous config saved to /var/cache/conftool/dbconfig/20221114-141738-ladsgroup.json
  • 14:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 14:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T321130)', diff saved to https://phabricator.wikimedia.org/P39494 and previous config saved to /var/cache/conftool/dbconfig/20221114-141731-marostegui.json
  • 14:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 14:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 14:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T322618)', diff saved to https://phabricator.wikimedia.org/P39493 and previous config saved to /var/cache/conftool/dbconfig/20221114-141717-ladsgroup.json
  • 14:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 14:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39492 and previous config saved to /var/cache/conftool/dbconfig/20221114-141710-marostegui.json
  • 14:14 taavi@deploy1002: taavi and matmarex: Backport for Use legacy DiscussionTools heading markup except on beta cluster (T314714), ThreadItemStore: Handle race conditions when finding/inserting outside of transaction (T322701), persistRevisionThreadItems: Print time taken synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmn
  • 14:14 taavi@deploy1002: Started scap: Backport for Use legacy DiscussionTools heading markup except on beta cluster (T314714), ThreadItemStore: Handle race conditions when finding/inserting outside of transaction (T322701), persistRevisionThreadItems: Print time taken
  • 14:13 taavi@deploy1002: backport aborted: (duration: 01m 23s)
  • 14:13 taavi@deploy1002: prep aborted: (duration: 00m 06s)
  • 14:11 taavi@deploy1002: Finished scap: Backport for Separate identifiers from other statements for Lexemes (T318310) (duration: 06m 27s)
  • 14:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P39491 and previous config saved to /var/cache/conftool/dbconfig/20221114-140923-marostegui.json
  • 14:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39490 and previous config saved to /var/cache/conftool/dbconfig/20221114-140541-ladsgroup.json
  • 14:05 taavi@deploy1002: taavi and migr: Backport for Separate identifiers from other statements for Lexemes (T318310) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 14:05 taavi@deploy1002: Started scap: Backport for Separate identifiers from other statements for Lexemes (T318310)
  • 14:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39489 and previous config saved to /var/cache/conftool/dbconfig/20221114-140320-ladsgroup.json
  • 14:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 14:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 14:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T322618)', diff saved to https://phabricator.wikimedia.org/P39488 and previous config saved to /var/cache/conftool/dbconfig/20221114-140259-ladsgroup.json
  • 14:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P39487 and previous config saved to /var/cache/conftool/dbconfig/20221114-140210-ladsgroup.json
  • 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P39486 and previous config saved to /var/cache/conftool/dbconfig/20221114-140203-marostegui.json
  • 13:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T321126)', diff saved to https://phabricator.wikimedia.org/P39485 and previous config saved to /var/cache/conftool/dbconfig/20221114-135416-marostegui.json
  • 13:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T321126)', diff saved to https://phabricator.wikimedia.org/P39484 and previous config saved to /var/cache/conftool/dbconfig/20221114-135204-marostegui.json
  • 13:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T321126)', diff saved to https://phabricator.wikimedia.org/P39483 and previous config saved to /var/cache/conftool/dbconfig/20221114-135114-marostegui.json
  • 13:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P39482 and previous config saved to /var/cache/conftool/dbconfig/20221114-134752-ladsgroup.json
  • 13:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P39481 and previous config saved to /var/cache/conftool/dbconfig/20221114-134704-ladsgroup.json
  • 13:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P39480 and previous config saved to /var/cache/conftool/dbconfig/20221114-134657-marostegui.json
  • 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P39479 and previous config saved to /var/cache/conftool/dbconfig/20221114-133608-marostegui.json
  • 13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P39478 and previous config saved to /var/cache/conftool/dbconfig/20221114-133246-ladsgroup.json
  • 13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T322618)', diff saved to https://phabricator.wikimedia.org/P39477 and previous config saved to /var/cache/conftool/dbconfig/20221114-133157-ladsgroup.json
  • 13:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39476 and previous config saved to /var/cache/conftool/dbconfig/20221114-133150-marostegui.json
  • 13:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P39475 and previous config saved to /var/cache/conftool/dbconfig/20221114-132101-marostegui.json
  • 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T321130)', diff saved to https://phabricator.wikimedia.org/P39474 and previous config saved to /var/cache/conftool/dbconfig/20221114-132008-marostegui.json
  • 13:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 13:19 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 13:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39473 and previous config saved to /var/cache/conftool/dbconfig/20221114-131946-marostegui.json
  • 13:19 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
  • 13:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T322618)', diff saved to https://phabricator.wikimedia.org/P39472 and previous config saved to /var/cache/conftool/dbconfig/20221114-131740-ladsgroup.json
  • 13:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2148 (T322618)', diff saved to https://phabricator.wikimedia.org/P39471 and previous config saved to /var/cache/conftool/dbconfig/20221114-131519-ladsgroup.json
  • 13:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 13:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 13:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39470 and previous config saved to /var/cache/conftool/dbconfig/20221114-131457-ladsgroup.json
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T321126)', diff saved to https://phabricator.wikimedia.org/P39469 and previous config saved to /var/cache/conftool/dbconfig/20221114-130555-marostegui.json
  • 13:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P39468 and previous config saved to /var/cache/conftool/dbconfig/20221114-130440-marostegui.json
  • 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T321126)', diff saved to https://phabricator.wikimedia.org/P39467 and previous config saved to /var/cache/conftool/dbconfig/20221114-130343-marostegui.json
  • 13:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 13:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39466 and previous config saved to /var/cache/conftool/dbconfig/20221114-130322-marostegui.json
  • 12:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P39465 and previous config saved to /var/cache/conftool/dbconfig/20221114-125951-ladsgroup.json
  • 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P39464 and previous config saved to /var/cache/conftool/dbconfig/20221114-124934-marostegui.json
  • 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P39463 and previous config saved to /var/cache/conftool/dbconfig/20221114-124815-marostegui.json
  • 12:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P39462 and previous config saved to /var/cache/conftool/dbconfig/20221114-124444-ladsgroup.json
  • 12:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39461 and previous config saved to /var/cache/conftool/dbconfig/20221114-123427-marostegui.json
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P39460 and previous config saved to /var/cache/conftool/dbconfig/20221114-123309-marostegui.json
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T322618)', diff saved to https://phabricator.wikimedia.org/P39459 and previous config saved to /var/cache/conftool/dbconfig/20221114-123141-ladsgroup.json
  • 12:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:31 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 12:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 12:31 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 12:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39458 and previous config saved to /var/cache/conftool/dbconfig/20221114-123103-ladsgroup.json
  • 12:30 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 12:30 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 12:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39457 and previous config saved to /var/cache/conftool/dbconfig/20221114-122938-ladsgroup.json
  • 12:28 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39456 and previous config saved to /var/cache/conftool/dbconfig/20221114-122717-ladsgroup.json
  • 12:27 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 12:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T322618)', diff saved to https://phabricator.wikimedia.org/P39455 and previous config saved to /var/cache/conftool/dbconfig/20221114-122655-ladsgroup.json
  • 12:26 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:25 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39454 and previous config saved to /var/cache/conftool/dbconfig/20221114-122214-marostegui.json
  • 12:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 12:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 12:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39453 and previous config saved to /var/cache/conftool/dbconfig/20221114-121802-marostegui.json
  • 12:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P39452 and previous config saved to /var/cache/conftool/dbconfig/20221114-121556-ladsgroup.json
  • 12:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39451 and previous config saved to /var/cache/conftool/dbconfig/20221114-121547-marostegui.json
  • 12:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39450 and previous config saved to /var/cache/conftool/dbconfig/20221114-121525-marostegui.json
  • 12:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 12:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 12:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39449 and previous config saved to /var/cache/conftool/dbconfig/20221114-121202-marostegui.json
  • 12:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P39448 and previous config saved to /var/cache/conftool/dbconfig/20221114-121149-ladsgroup.json
  • 12:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P39447 and previous config saved to /var/cache/conftool/dbconfig/20221114-120043-ladsgroup.json
  • 12:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P39446 and previous config saved to /var/cache/conftool/dbconfig/20221114-120019-marostegui.json
  • 11:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39445 and previous config saved to /var/cache/conftool/dbconfig/20221114-115656-marostegui.json
  • 11:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P39444 and previous config saved to /var/cache/conftool/dbconfig/20221114-115641-ladsgroup.json
  • 11:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39443 and previous config saved to /var/cache/conftool/dbconfig/20221114-114537-ladsgroup.json
  • 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P39442 and previous config saved to /var/cache/conftool/dbconfig/20221114-114512-marostegui.json
  • 11:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39441 and previous config saved to /var/cache/conftool/dbconfig/20221114-114326-ladsgroup.json
  • 11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 11:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 11:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 11:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T322618)', diff saved to https://phabricator.wikimedia.org/P39440 and previous config saved to /var/cache/conftool/dbconfig/20221114-114244-ladsgroup.json
  • 11:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39439 and previous config saved to /var/cache/conftool/dbconfig/20221114-114150-marostegui.json
  • 11:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T322618)', diff saved to https://phabricator.wikimedia.org/P39438 and previous config saved to /var/cache/conftool/dbconfig/20221114-114134-ladsgroup.json
  • 11:40 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 9231
  • 11:40 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 9231
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2126 (T322618)', diff saved to https://phabricator.wikimedia.org/P39437 and previous config saved to /var/cache/conftool/dbconfig/20221114-113913-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T322618)', diff saved to https://phabricator.wikimedia.org/P39436 and previous config saved to /var/cache/conftool/dbconfig/20221114-113837-ladsgroup.json
  • 11:34 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 37271
  • 11:32 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 37271
  • 11:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39435 and previous config saved to /var/cache/conftool/dbconfig/20221114-113006-marostegui.json
  • 11:29 ladsgroup@deploy1002: Finished scap: Backport for Re-add s11 in db config reload callback (T322598) (duration: 05m 01s)
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39434 and previous config saved to /var/cache/conftool/dbconfig/20221114-112750-marostegui.json
  • 11:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 11:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P39433 and previous config saved to /var/cache/conftool/dbconfig/20221114-112736-ladsgroup.json
  • 11:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39432 and previous config saved to /var/cache/conftool/dbconfig/20221114-112729-marostegui.json
  • 11:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39431 and previous config saved to /var/cache/conftool/dbconfig/20221114-112643-marostegui.json
  • 11:24 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for Re-add s11 in db config reload callback (T322598) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 11:24 ladsgroup@deploy1002: Started scap: Backport for Re-add s11 in db config reload callback (T322598)
  • 11:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P39430 and previous config saved to /var/cache/conftool/dbconfig/20221114-112330-ladsgroup.json
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T321130)', diff saved to https://phabricator.wikimedia.org/P39429 and previous config saved to /var/cache/conftool/dbconfig/20221114-111434-marostegui.json
  • 11:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 11:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T321130)', diff saved to https://phabricator.wikimedia.org/P39428 and previous config saved to /var/cache/conftool/dbconfig/20221114-111412-marostegui.json
  • 11:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P39427 and previous config saved to /var/cache/conftool/dbconfig/20221114-111229-ladsgroup.json
  • 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P39426 and previous config saved to /var/cache/conftool/dbconfig/20221114-111222-marostegui.json
  • 11:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P39425 and previous config saved to /var/cache/conftool/dbconfig/20221114-110824-ladsgroup.json
  • 10:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P39424 and previous config saved to /var/cache/conftool/dbconfig/20221114-105906-marostegui.json
  • 10:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T322618)', diff saved to https://phabricator.wikimedia.org/P39423 and previous config saved to /var/cache/conftool/dbconfig/20221114-105723-ladsgroup.json
  • 10:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P39422 and previous config saved to /var/cache/conftool/dbconfig/20221114-105716-marostegui.json
  • 10:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T322618)', diff saved to https://phabricator.wikimedia.org/P39421 and previous config saved to /var/cache/conftool/dbconfig/20221114-105512-ladsgroup.json
  • 10:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 10:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 10:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39420 and previous config saved to /var/cache/conftool/dbconfig/20221114-105450-ladsgroup.json
  • 10:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T322618)', diff saved to https://phabricator.wikimedia.org/P39419 and previous config saved to /var/cache/conftool/dbconfig/20221114-105317-ladsgroup.json
  • 10:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T322618)', diff saved to https://phabricator.wikimedia.org/P39418 and previous config saved to /var/cache/conftool/dbconfig/20221114-105056-ladsgroup.json
  • 10:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 10:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 10:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T322618)', diff saved to https://phabricator.wikimedia.org/P39417 and previous config saved to /var/cache/conftool/dbconfig/20221114-105034-ladsgroup.json
  • 10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P39416 and previous config saved to /var/cache/conftool/dbconfig/20221114-104400-marostegui.json
  • 10:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39415 and previous config saved to /var/cache/conftool/dbconfig/20221114-104209-marostegui.json
  • 10:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T321126)', diff saved to https://phabricator.wikimedia.org/P39414 and previous config saved to /var/cache/conftool/dbconfig/20221114-103953-marostegui.json
  • 10:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P39413 and previous config saved to /var/cache/conftool/dbconfig/20221114-103944-ladsgroup.json
  • 10:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 10:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P39412 and previous config saved to /var/cache/conftool/dbconfig/20221114-103528-ladsgroup.json
  • 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T321130)', diff saved to https://phabricator.wikimedia.org/P39411 and previous config saved to /var/cache/conftool/dbconfig/20221114-102853-marostegui.json
  • 10:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P39410 and previous config saved to /var/cache/conftool/dbconfig/20221114-102437-ladsgroup.json
  • 10:21 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39409 and previous config saved to /var/cache/conftool/dbconfig/20221114-102155-root.json
  • 10:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P39408 and previous config saved to /var/cache/conftool/dbconfig/20221114-102021-ladsgroup.json
  • 10:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T321130)', diff saved to https://phabricator.wikimedia.org/P39407 and previous config saved to /var/cache/conftool/dbconfig/20221114-101659-marostegui.json
  • 10:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 10:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 10:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39406 and previous config saved to /var/cache/conftool/dbconfig/20221114-101637-marostegui.json
  • 10:12 vgutierrez: upgrading acme-chief on acmechief1001 to version 0.35 (requires disabling puppet on R:acme_chief::cert)
  • 10:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39405 and previous config saved to /var/cache/conftool/dbconfig/20221114-100931-ladsgroup.json
  • 10:09 ladsgroup@deploy1002: Finished scap: Backport for Rework SpecialPagesWithoutScans query (T322849) (duration: 11m 17s)
  • 10:07 vgutierrez: upload acme-chief 0.35 to apt.wm.o (buster-wikimedia) - T244232 T262251
  • 10:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39404 and previous config saved to /var/cache/conftool/dbconfig/20221114-100720-ladsgroup.json
  • 10:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 10:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 10:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 10:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39403 and previous config saved to /var/cache/conftool/dbconfig/20221114-100650-root.json
  • 10:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 10:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T322618)', diff saved to https://phabricator.wikimedia.org/P39402 and previous config saved to /var/cache/conftool/dbconfig/20221114-100515-ladsgroup.json
  • 10:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2104 (T322618)', diff saved to https://phabricator.wikimedia.org/P39401 and previous config saved to /var/cache/conftool/dbconfig/20221114-100254-ladsgroup.json
  • 10:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 10:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 10:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 10:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 10:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P39400 and previous config saved to /var/cache/conftool/dbconfig/20221114-100131-marostegui.json
  • 09:58 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for Rework SpecialPagesWithoutScans query (T322849) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 09:57 ladsgroup@deploy1002: Started scap: Backport for Rework SpecialPagesWithoutScans query (T322849)
  • 09:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 09:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 09:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39399 and previous config saved to /var/cache/conftool/dbconfig/20221114-095145-root.json
  • 09:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P39398 and previous config saved to /var/cache/conftool/dbconfig/20221114-094624-marostegui.json
  • 09:44 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
  • 09:39 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
  • 09:38 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 4788
  • 09:38 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 50083
  • 09:38 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 50083
  • 09:37 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 32934
  • 09:36 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39397 and previous config saved to /var/cache/conftool/dbconfig/20221114-093640-root.json
  • 09:35 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 32934
  • 09:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39396 and previous config saved to /var/cache/conftool/dbconfig/20221114-093118-marostegui.json
  • 09:21 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39395 and previous config saved to /var/cache/conftool/dbconfig/20221114-092135-root.json
  • 09:19 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39394 and previous config saved to /var/cache/conftool/dbconfig/20221114-091934-marostegui.json
  • 09:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 09:19 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 09:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321130)', diff saved to https://phabricator.wikimedia.org/P39393 and previous config saved to /var/cache/conftool/dbconfig/20221114-091912-marostegui.json
  • 09:18 ayounsi@cumin1001: END (ERROR) - Cookbook sre.network.peering (exit_code=97) with action 'configure' for AS: 13335
  • 09:18 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 13335
  • 09:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39392 and previous config saved to /var/cache/conftool/dbconfig/20221114-090630-root.json
  • 09:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P39391 and previous config saved to /var/cache/conftool/dbconfig/20221114-090406-marostegui.json
  • 08:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39389 and previous config saved to /var/cache/conftool/dbconfig/20221114-085125-root.json
  • 08:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P39388 and previous config saved to /var/cache/conftool/dbconfig/20221114-084859-marostegui.json
  • 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39387 and previous config saved to /var/cache/conftool/dbconfig/20221114-083620-root.json
  • 08:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321130)', diff saved to https://phabricator.wikimedia.org/P39386 and previous config saved to /var/cache/conftool/dbconfig/20221114-083352-marostegui.json
  • 08:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2145 T322620', diff saved to https://phabricator.wikimedia.org/P39385 and previous config saved to /var/cache/conftool/dbconfig/20221114-082458-root.json
  • 08:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T321130)', diff saved to https://phabricator.wikimedia.org/P39384 and previous config saved to /var/cache/conftool/dbconfig/20221114-082205-marostegui.json
  • 08:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 08:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 08:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39383 and previous config saved to /var/cache/conftool/dbconfig/20221114-082144-marostegui.json
  • 08:20 moritzm: installing php7.4 security updates (as packaged in Debian)
  • 08:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P39381 and previous config saved to /var/cache/conftool/dbconfig/20221114-080637-marostegui.json
  • 08:02 marostegui@deploy1002: Finished scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc1 master" (duration: 04m 34s)
  • 07:57 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "ProductionServices.php: Promote pc2014 to pc1 master" synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 07:57 marostegui@deploy1002: Started scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc1 master"
  • 07:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P39380 and previous config saved to /var/cache/conftool/dbconfig/20221114-075131-marostegui.json
  • 07:50 moritzm: draining ganeti1021 for eventual reimage T311687
  • 07:47 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
  • 07:47 marostegui@deploy1002: Finished scap: Backport for ProductionServices.php: Promote pc2014 to pc1 master (T322295) (duration: 05m 14s)
  • 07:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
  • 07:42 marostegui@deploy1002: marostegui and marostegui: Backport for ProductionServices.php: Promote pc2014 to pc1 master (T322295) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 07:42 marostegui@deploy1002: Started scap: Backport for ProductionServices.php: Promote pc2014 to pc1 master (T322295)
  • 07:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39379 and previous config saved to /var/cache/conftool/dbconfig/20221114-073624-marostegui.json
  • 07:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39378 and previous config saved to /var/cache/conftool/dbconfig/20221114-072203-marostegui.json
  • 07:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 07:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 07:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 07:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 07:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1181.eqiad.wmnet with reason: Maintenance
  • 07:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1181.eqiad.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2118.codfw.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2118.codfw.wmnet with reason: Maintenance
  • 06:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39377 and previous config saved to /var/cache/conftool/dbconfig/20221114-065620-marostegui.json
  • 06:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39376 and previous config saved to /var/cache/conftool/dbconfig/20221114-064113-marostegui.json
  • 06:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39375 and previous config saved to /var/cache/conftool/dbconfig/20221114-062607-marostegui.json
  • 06:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39374 and previous config saved to /var/cache/conftool/dbconfig/20221114-061100-marostegui.json
  • 06:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39373 and previous config saved to /var/cache/conftool/dbconfig/20221114-060847-marostegui.json
  • 06:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 06:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 06:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2173', diff saved to https://phabricator.wikimedia.org/P39372 and previous config saved to /var/cache/conftool/dbconfig/20221114-060207-root.json

2022-11-12

  • 23:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T318605)', diff saved to https://phabricator.wikimedia.org/P39371 and previous config saved to /var/cache/conftool/dbconfig/20221112-233420-ladsgroup.json
  • 23:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P39370 and previous config saved to /var/cache/conftool/dbconfig/20221112-231914-ladsgroup.json
  • 23:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P39369 and previous config saved to /var/cache/conftool/dbconfig/20221112-230407-ladsgroup.json
  • 22:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T318605)', diff saved to https://phabricator.wikimedia.org/P39368 and previous config saved to /var/cache/conftool/dbconfig/20221112-224900-ladsgroup.json
  • 22:46 urandom: initiating bootstrap, aqs1016-b -- T307802
  • 21:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 21:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T318605)', diff saved to https://phabricator.wikimedia.org/P39367 and previous config saved to /var/cache/conftool/dbconfig/20221112-210527-ladsgroup.json
  • 20:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39366 and previous config saved to /var/cache/conftool/dbconfig/20221112-205020-ladsgroup.json
  • 20:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P39365 and previous config saved to /var/cache/conftool/dbconfig/20221112-203514-ladsgroup.json
  • 20:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T318605)', diff saved to https://phabricator.wikimedia.org/P39364 and previous config saved to /var/cache/conftool/dbconfig/20221112-202007-ladsgroup.json
  • off: uploaded python3-gjson_0.4.0 to apt.wikimedia.org bullseye-wikimedia
  • 17:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2179 (T318605)', diff saved to https://phabricator.wikimedia.org/P39363 and previous config saved to /var/cache/conftool/dbconfig/20221112-171705-ladsgroup.json
  • 17:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 17:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 17:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T318605)', diff saved to https://phabricator.wikimedia.org/P39362 and previous config saved to /var/cache/conftool/dbconfig/20221112-171643-ladsgroup.json
  • 17:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P39361 and previous config saved to /var/cache/conftool/dbconfig/20221112-170137-ladsgroup.json
  • 16:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P39360 and previous config saved to /var/cache/conftool/dbconfig/20221112-164630-ladsgroup.json
  • 16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T318605)', diff saved to https://phabricator.wikimedia.org/P39359 and previous config saved to /var/cache/conftool/dbconfig/20221112-163124-ladsgroup.json
  • 14:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1199 (T318605)', diff saved to https://phabricator.wikimedia.org/P39358 and previous config saved to /var/cache/conftool/dbconfig/20221112-144302-ladsgroup.json
  • 14:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 14:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 14:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T318605)', diff saved to https://phabricator.wikimedia.org/P39357 and previous config saved to /var/cache/conftool/dbconfig/20221112-144240-ladsgroup.json
  • 14:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P39356 and previous config saved to /var/cache/conftool/dbconfig/20221112-142734-ladsgroup.json
  • 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P39355 and previous config saved to /var/cache/conftool/dbconfig/20221112-141227-ladsgroup.json
  • 13:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T318605)', diff saved to https://phabricator.wikimedia.org/P39354 and previous config saved to /var/cache/conftool/dbconfig/20221112-135721-ladsgroup.json
  • 10:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2172 (T318605)', diff saved to https://phabricator.wikimedia.org/P39353 and previous config saved to /var/cache/conftool/dbconfig/20221112-105847-ladsgroup.json
  • 10:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 10:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 10:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T318605)', diff saved to https://phabricator.wikimedia.org/P39352 and previous config saved to /var/cache/conftool/dbconfig/20221112-105825-ladsgroup.json
  • 10:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P39351 and previous config saved to /var/cache/conftool/dbconfig/20221112-104319-ladsgroup.json
  • 10:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P39350 and previous config saved to /var/cache/conftool/dbconfig/20221112-102812-ladsgroup.json
  • 10:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T318605)', diff saved to https://phabricator.wikimedia.org/P39349 and previous config saved to /var/cache/conftool/dbconfig/20221112-101306-ladsgroup.json
  • 08:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1190 (T318605)', diff saved to https://phabricator.wikimedia.org/P39348 and previous config saved to /var/cache/conftool/dbconfig/20221112-082623-ladsgroup.json
  • 08:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 08:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 08:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T318605)', diff saved to https://phabricator.wikimedia.org/P39347 and previous config saved to /var/cache/conftool/dbconfig/20221112-082601-ladsgroup.json
  • 08:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39346 and previous config saved to /var/cache/conftool/dbconfig/20221112-081055-ladsgroup.json
  • 07:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P39345 and previous config saved to /var/cache/conftool/dbconfig/20221112-075548-ladsgroup.json
  • 07:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T318605)', diff saved to https://phabricator.wikimedia.org/P39344 and previous config saved to /var/cache/conftool/dbconfig/20221112-074042-ladsgroup.json
  • 04:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2155 (T318605)', diff saved to https://phabricator.wikimedia.org/P39343 and previous config saved to /var/cache/conftool/dbconfig/20221112-043203-ladsgroup.json
  • 04:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 04:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 04:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 04:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 04:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39342 and previous config saved to /var/cache/conftool/dbconfig/20221112-043137-ladsgroup.json
  • 04:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P39341 and previous config saved to /var/cache/conftool/dbconfig/20221112-041631-ladsgroup.json
  • 04:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P39340 and previous config saved to /var/cache/conftool/dbconfig/20221112-040124-ladsgroup.json
  • 03:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39339 and previous config saved to /var/cache/conftool/dbconfig/20221112-034618-ladsgroup.json
  • 02:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T321130)', diff saved to https://phabricator.wikimedia.org/P39338 and previous config saved to /var/cache/conftool/dbconfig/20221112-022827-marostegui.json
  • 02:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T318605)', diff saved to https://phabricator.wikimedia.org/P39337 and previous config saved to /var/cache/conftool/dbconfig/20221112-022535-ladsgroup.json
  • 02:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 02:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 02:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P39336 and previous config saved to /var/cache/conftool/dbconfig/20221112-021321-marostegui.json
  • 01:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P39335 and previous config saved to /var/cache/conftool/dbconfig/20221112-015814-marostegui.json
  • 01:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T321130)', diff saved to https://phabricator.wikimedia.org/P39334 and previous config saved to /var/cache/conftool/dbconfig/20221112-014308-marostegui.json
  • 01:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2182 (T321130)', diff saved to https://phabricator.wikimedia.org/P39333 and previous config saved to /var/cache/conftool/dbconfig/20221112-013650-marostegui.json
  • 01:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 01:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 01:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39332 and previous config saved to /var/cache/conftool/dbconfig/20221112-013628-marostegui.json
  • 01:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P39331 and previous config saved to /var/cache/conftool/dbconfig/20221112-012122-marostegui.json
  • 01:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P39330 and previous config saved to /var/cache/conftool/dbconfig/20221112-010615-marostegui.json
  • 00:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39329 and previous config saved to /var/cache/conftool/dbconfig/20221112-005107-marostegui.json
  • 00:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39328 and previous config saved to /var/cache/conftool/dbconfig/20221112-004443-marostegui.json
  • 00:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 00:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 00:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39327 and previous config saved to /var/cache/conftool/dbconfig/20221112-004422-marostegui.json
  • 00:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P39326 and previous config saved to /var/cache/conftool/dbconfig/20221112-002915-marostegui.json
  • 00:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P39325 and previous config saved to /var/cache/conftool/dbconfig/20221112-001408-marostegui.json

2022-11-11

  • 23:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39324 and previous config saved to /var/cache/conftool/dbconfig/20221111-235902-marostegui.json
  • 23:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2168:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39323 and previous config saved to /var/cache/conftool/dbconfig/20221111-235235-marostegui.json
  • 23:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 23:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 23:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T321130)', diff saved to https://phabricator.wikimedia.org/P39322 and previous config saved to /var/cache/conftool/dbconfig/20221111-235214-marostegui.json
  • 23:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P39321 and previous config saved to /var/cache/conftool/dbconfig/20221111-233707-marostegui.json
  • 23:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P39320 and previous config saved to /var/cache/conftool/dbconfig/20221111-232201-marostegui.json
  • 23:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T321130)', diff saved to https://phabricator.wikimedia.org/P39319 and previous config saved to /var/cache/conftool/dbconfig/20221111-230654-marostegui.json
  • 23:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2159 (T321130)', diff saved to https://phabricator.wikimedia.org/P39318 and previous config saved to /var/cache/conftool/dbconfig/20221111-230037-marostegui.json
  • 23:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 23:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 23:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 23:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 23:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T321130)', diff saved to https://phabricator.wikimedia.org/P39317 and previous config saved to /var/cache/conftool/dbconfig/20221111-230000-marostegui.json
  • 22:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P39316 and previous config saved to /var/cache/conftool/dbconfig/20221111-224454-marostegui.json
  • 22:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P39315 and previous config saved to /var/cache/conftool/dbconfig/20221111-222948-marostegui.json
  • 22:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T321130)', diff saved to https://phabricator.wikimedia.org/P39314 and previous config saved to /var/cache/conftool/dbconfig/20221111-221441-marostegui.json
  • 22:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39313 and previous config saved to /var/cache/conftool/dbconfig/20221111-220939-ladsgroup.json
  • 22:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 22:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 22:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2150 (T321130)', diff saved to https://phabricator.wikimedia.org/P39312 and previous config saved to /var/cache/conftool/dbconfig/20221111-220820-marostegui.json
  • 22:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 22:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 22:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T321130)', diff saved to https://phabricator.wikimedia.org/P39311 and previous config saved to /var/cache/conftool/dbconfig/20221111-220758-marostegui.json
  • 21:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P39310 and previous config saved to /var/cache/conftool/dbconfig/20221111-215252-marostegui.json
  • 21:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P39309 and previous config saved to /var/cache/conftool/dbconfig/20221111-213745-marostegui.json
  • 21:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T321130)', diff saved to https://phabricator.wikimedia.org/P39308 and previous config saved to /var/cache/conftool/dbconfig/20221111-212239-marostegui.json
  • 21:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2122 (T321130)', diff saved to https://phabricator.wikimedia.org/P39307 and previous config saved to /var/cache/conftool/dbconfig/20221111-211611-marostegui.json
  • 21:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 21:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 21:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39306 and previous config saved to /var/cache/conftool/dbconfig/20221111-211550-marostegui.json
  • 21:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P39305 and previous config saved to /var/cache/conftool/dbconfig/20221111-210043-marostegui.json
  • 20:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 20:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 20:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T318605)', diff saved to https://phabricator.wikimedia.org/P39304 and previous config saved to /var/cache/conftool/dbconfig/20221111-205919-ladsgroup.json
  • 20:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P39303 and previous config saved to /var/cache/conftool/dbconfig/20221111-204536-marostegui.json
  • 20:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P39302 and previous config saved to /var/cache/conftool/dbconfig/20221111-204413-ladsgroup.json
  • 20:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39301 and previous config saved to /var/cache/conftool/dbconfig/20221111-203030-marostegui.json
  • 20:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P39300 and previous config saved to /var/cache/conftool/dbconfig/20221111-202906-ladsgroup.json
  • 20:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39299 and previous config saved to /var/cache/conftool/dbconfig/20221111-202413-marostegui.json
  • 20:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 20:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 20:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T321130)', diff saved to https://phabricator.wikimedia.org/P39298 and previous config saved to /var/cache/conftool/dbconfig/20221111-202351-marostegui.json
  • 20:21 mutante: phab1001,phab1004,phab2002 - systemctl reset-failed
  • 20:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T318605)', diff saved to https://phabricator.wikimedia.org/P39297 and previous config saved to /var/cache/conftool/dbconfig/20221111-201400-ladsgroup.json
  • 20:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P39296 and previous config saved to /var/cache/conftool/dbconfig/20221111-200845-marostegui.json
  • 19:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P39295 and previous config saved to /var/cache/conftool/dbconfig/20221111-195338-marostegui.json
  • 19:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T321130)', diff saved to https://phabricator.wikimedia.org/P39294 and previous config saved to /var/cache/conftool/dbconfig/20221111-193832-marostegui.json
  • 19:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2120 (T321130)', diff saved to https://phabricator.wikimedia.org/P39293 and previous config saved to /var/cache/conftool/dbconfig/20221111-193214-marostegui.json
  • 19:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 19:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 19:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T321130)', diff saved to https://phabricator.wikimedia.org/P39292 and previous config saved to /var/cache/conftool/dbconfig/20221111-193152-marostegui.json
  • 19:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P39291 and previous config saved to /var/cache/conftool/dbconfig/20221111-191646-marostegui.json
  • 19:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P39290 and previous config saved to /var/cache/conftool/dbconfig/20221111-190139-marostegui.json
  • 18:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T321130)', diff saved to https://phabricator.wikimedia.org/P39289 and previous config saved to /var/cache/conftool/dbconfig/20221111-184633-marostegui.json
  • 18:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2108 (T321130)', diff saved to https://phabricator.wikimedia.org/P39288 and previous config saved to /var/cache/conftool/dbconfig/20221111-184017-marostegui.json
  • 18:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 18:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 18:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 18:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 18:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T321130)', diff saved to https://phabricator.wikimedia.org/P39287 and previous config saved to /var/cache/conftool/dbconfig/20221111-182640-marostegui.json
  • 18:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39286 and previous config saved to /var/cache/conftool/dbconfig/20221111-181134-marostegui.json
  • 17:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39285 and previous config saved to /var/cache/conftool/dbconfig/20221111-175627-marostegui.json
  • 17:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T321130)', diff saved to https://phabricator.wikimedia.org/P39284 and previous config saved to /var/cache/conftool/dbconfig/20221111-174121-marostegui.json
  • 17:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T321130)', diff saved to https://phabricator.wikimedia.org/P39283 and previous config saved to /var/cache/conftool/dbconfig/20221111-173907-marostegui.json
  • 17:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T321130)', diff saved to https://phabricator.wikimedia.org/P39282 and previous config saved to /var/cache/conftool/dbconfig/20221111-173846-marostegui.json
  • 17:34 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=varnish-fe
  • 17:34 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-be
  • 17:34 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-tls
  • 17:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39281 and previous config saved to /var/cache/conftool/dbconfig/20221111-172339-marostegui.json
  • 17:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39280 and previous config saved to /var/cache/conftool/dbconfig/20221111-170833-marostegui.json
  • 16:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T321130)', diff saved to https://phabricator.wikimedia.org/P39279 and previous config saved to /var/cache/conftool/dbconfig/20221111-165326-marostegui.json
  • 16:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1194 (T321130)', diff saved to https://phabricator.wikimedia.org/P39278 and previous config saved to /var/cache/conftool/dbconfig/20221111-165113-marostegui.json
  • 16:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 16:50 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 16:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39277 and previous config saved to /var/cache/conftool/dbconfig/20221111-165051-marostegui.json
  • 16:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39275 and previous config saved to /var/cache/conftool/dbconfig/20221111-163545-marostegui.json
  • 16:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39274 and previous config saved to /var/cache/conftool/dbconfig/20221111-162038-marostegui.json
  • 16:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 16:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 16:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39273 and previous config saved to /var/cache/conftool/dbconfig/20221111-161528-ladsgroup.json
  • 16:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39272 and previous config saved to /var/cache/conftool/dbconfig/20221111-160532-marostegui.json
  • 16:05 vgutierrez: restart varnish in cp2042
  • 16:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P39271 and previous config saved to /var/cache/conftool/dbconfig/20221111-160022-ladsgroup.json
  • 15:58 vgutierrez: rolling restart of varnish in cp4045 - cp4050 - T322903
  • 15:57 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:56 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS buster
  • 15:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P39270 and previous config saved to /var/cache/conftool/dbconfig/20221111-154515-ladsgroup.json
  • 15:43 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS buster
  • 15:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39269 and previous config saved to /var/cache/conftool/dbconfig/20221111-153009-ladsgroup.json
  • 15:21 moritzm: installing node-end-of-stream security updates
  • 15:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39268 and previous config saved to /var/cache/conftool/dbconfig/20221111-150516-marostegui.json
  • 15:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 15:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 15:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T321130)', diff saved to https://phabricator.wikimedia.org/P39267 and previous config saved to /var/cache/conftool/dbconfig/20221111-150454-marostegui.json
  • 14:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P39266 and previous config saved to /var/cache/conftool/dbconfig/20221111-144948-marostegui.json
  • 14:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T318605)', diff saved to https://phabricator.wikimedia.org/P39265 and previous config saved to /var/cache/conftool/dbconfig/20221111-144047-ladsgroup.json
  • 14:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 14:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 14:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T318605)', diff saved to https://phabricator.wikimedia.org/P39264 and previous config saved to /var/cache/conftool/dbconfig/20221111-144025-ladsgroup.json
  • 14:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P39263 and previous config saved to /var/cache/conftool/dbconfig/20221111-143441-marostegui.json
  • 14:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39262 and previous config saved to /var/cache/conftool/dbconfig/20221111-142519-ladsgroup.json
  • 14:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T321130)', diff saved to https://phabricator.wikimedia.org/P39261 and previous config saved to /var/cache/conftool/dbconfig/20221111-141935-marostegui.json
  • 14:17 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T321130)', diff saved to https://phabricator.wikimedia.org/P39260 and previous config saved to /var/cache/conftool/dbconfig/20221111-141721-marostegui.json
  • 14:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 14:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 14:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 14:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 14:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39259 and previous config saved to /var/cache/conftool/dbconfig/20221111-141233-marostegui.json
  • 14:12 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2003-dev.codfw.wmnet with OS bullseye
  • 14:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39258 and previous config saved to /var/cache/conftool/dbconfig/20221111-141012-ladsgroup.json
  • 13:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P39257 and previous config saved to /var/cache/conftool/dbconfig/20221111-135727-marostegui.json
  • 13:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T318605)', diff saved to https://phabricator.wikimedia.org/P39256 and previous config saved to /var/cache/conftool/dbconfig/20221111-135506-ladsgroup.json
  • 13:51 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 13:50 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 13:49 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2003-dev.codfw.wmnet with reason: host reimage
  • 13:47 aborrero@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2003-dev.codfw.wmnet with reason: host reimage
  • 13:45 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 13:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P39255 and previous config saved to /var/cache/conftool/dbconfig/20221111-134221-marostegui.json
  • 13:42 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:42 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:30 moritzm: installing procmail security updates
  • 13:30 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirt2003-dev.codfw.wmnet with OS bullseye
  • 13:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39254 and previous config saved to /var/cache/conftool/dbconfig/20221111-132714-marostegui.json
  • 13:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39253 and previous config saved to /var/cache/conftool/dbconfig/20221111-132105-marostegui.json
  • 13:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T321130)', diff saved to https://phabricator.wikimedia.org/P39252 and previous config saved to /var/cache/conftool/dbconfig/20221111-132043-marostegui.json
  • 13:20 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:13 jnuche@deploy1002: sync-world aborted: (no justification provided) (duration: 17m 49s)
  • 13:13 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
  • 13:13 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 13:13 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 13:13 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 13:13 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 13:13 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 13:13 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:12 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 13:10 jnuche@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 13:10 jnuche@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
  • 13:08 jnuche@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 13:08 jnuche@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
  • 13:08 jnuche@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 13:07 jnuche@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
  • 13:06 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 13:06 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
  • 13:06 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 13:06 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 13:06 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 13:06 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 13:06 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 13:06 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 13:05 jnuche@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P39251 and previous config saved to /var/cache/conftool/dbconfig/20221111-130537-marostegui.json
  • 13:05 jnuche@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 13:01 jnuche@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 13:01 jnuche@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 12:55 jnuche@deploy1002: Started scap: (no justification provided)
  • 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P39249 and previous config saved to /var/cache/conftool/dbconfig/20221111-125030-marostegui.json
  • 12:42 moritzm: installing debootstrap bugfix updates from buster point release
  • 12:37 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 12:35 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T321130)', diff saved to https://phabricator.wikimedia.org/P39248 and previous config saved to /var/cache/conftool/dbconfig/20221111-123524-marostegui.json
  • 12:35 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:34 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host ganeti1033.eqiad.wmnet
  • 12:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T321130)', diff saved to https://phabricator.wikimedia.org/P39247 and previous config saved to /var/cache/conftool/dbconfig/20221111-123310-marostegui.json
  • 12:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 12:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 12:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39246 and previous config saved to /var/cache/conftool/dbconfig/20221111-123232-marostegui.json
  • 12:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P39245 and previous config saved to /var/cache/conftool/dbconfig/20221111-121725-marostegui.json
  • 12:14 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 12:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet
  • 12:10 aborrero@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 12:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P39244 and previous config saved to /var/cache/conftool/dbconfig/20221111-120219-marostegui.json
  • 11:53 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 11:51 aborrero@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 11:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39243 and previous config saved to /var/cache/conftool/dbconfig/20221111-114712-marostegui.json
  • 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T321130)', diff saved to https://phabricator.wikimedia.org/P39242 and previous config saved to /var/cache/conftool/dbconfig/20221111-114458-marostegui.json
  • 11:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 11:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 11:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T321130)', diff saved to https://phabricator.wikimedia.org/P39241 and previous config saved to /var/cache/conftool/dbconfig/20221111-114437-marostegui.json
  • 11:42 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P39240 and previous config saved to /var/cache/conftool/dbconfig/20221111-112931-marostegui.json
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P39239 and previous config saved to /var/cache/conftool/dbconfig/20221111-111424-marostegui.json
  • 11:03 moritzm: installing wireshark security updates
  • 10:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T321130)', diff saved to https://phabricator.wikimedia.org/P39238 and previous config saved to /var/cache/conftool/dbconfig/20221111-105918-marostegui.json
  • 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T321130)', diff saved to https://phabricator.wikimedia.org/P39237 and previous config saved to /var/cache/conftool/dbconfig/20221111-105305-marostegui.json
  • 10:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 10:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 10:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39236 and previous config saved to /var/cache/conftool/dbconfig/20221111-105244-marostegui.json
  • 10:52 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 10:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P39235 and previous config saved to /var/cache/conftool/dbconfig/20221111-103738-marostegui.json
  • 10:22 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 10:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P39234 and previous config saved to /var/cache/conftool/dbconfig/20221111-102231-marostegui.json
  • 10:18 aborrero@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 10:15 elukey@cumin1001: END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES eqiad cluster: Roll restart of ORES's daemons.
  • 10:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39233 and previous config saved to /var/cache/conftool/dbconfig/20221111-100725-marostegui.json
  • 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39232 and previous config saved to /var/cache/conftool/dbconfig/20221111-100054-marostegui.json
  • 10:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 10:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39231 and previous config saved to /var/cache/conftool/dbconfig/20221111-100033-marostegui.json
  • 09:55 elukey@cumin1001: START - Cookbook sre.ores.roll-restart-workers for ORES eqiad cluster: Roll restart of ORES's daemons.
  • 09:54 elukey@cumin1001: END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES codfw cluster: Roll restart of ORES's daemons.
  • 09:45 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirt2002-dev.codfw.wmnet with OS bullseye
  • 09:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P39230 and previous config saved to /var/cache/conftool/dbconfig/20221111-094526-marostegui.json
  • 09:35 elukey@cumin1001: START - Cookbook sre.ores.roll-restart-workers for ORES codfw cluster: Roll restart of ORES's daemons.
  • 09:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P39229 and previous config saved to /var/cache/conftool/dbconfig/20221111-093020-marostegui.json
  • 09:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39228 and previous config saved to /var/cache/conftool/dbconfig/20221111-092503-ladsgroup.json
  • 09:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 09:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 09:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39227 and previous config saved to /var/cache/conftool/dbconfig/20221111-092441-ladsgroup.json
  • 09:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39226 and previous config saved to /var/cache/conftool/dbconfig/20221111-091514-marostegui.json
  • 09:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P39225 and previous config saved to /var/cache/conftool/dbconfig/20221111-090935-ladsgroup.json
  • 09:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T321130)', diff saved to https://phabricator.wikimedia.org/P39224 and previous config saved to /var/cache/conftool/dbconfig/20221111-090846-marostegui.json
  • 09:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:07 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1020.eqiad.wmnet to cluster eqiad and group D
  • 09:06 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1130.eqiad.wmnet with reason: Maintenance
  • 09:06 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1130.eqiad.wmnet with reason: Maintenance
  • 09:06 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1020.eqiad.wmnet to cluster eqiad and group D
  • 09:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2113.codfw.wmnet with reason: Maintenance
  • 09:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2113.codfw.wmnet with reason: Maintenance
  • 09:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1020.eqiad.wmnet
  • 09:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 09:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 09:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2112.codfw.wmnet with reason: Maintenance
  • 09:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2112.codfw.wmnet with reason: Maintenance
  • 08:55 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1020.eqiad.wmnet
  • 08:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P39223 and previous config saved to /var/cache/conftool/dbconfig/20221111-085428-ladsgroup.json
  • 08:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1020.eqiad.wmnet with OS bullseye
  • 08:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39222 and previous config saved to /var/cache/conftool/dbconfig/20221111-083922-ladsgroup.json
  • 08:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T318605)', diff saved to https://phabricator.wikimedia.org/P39221 and previous config saved to /var/cache/conftool/dbconfig/20221111-083611-ladsgroup.json
  • 08:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 08:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 08:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39220 and previous config saved to /var/cache/conftool/dbconfig/20221111-083549-ladsgroup.json
  • 08:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1020.eqiad.wmnet with reason: host reimage
  • 08:28 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1020.eqiad.wmnet with reason: host reimage
  • 08:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P39219 and previous config saved to /var/cache/conftool/dbconfig/20221111-082042-ladsgroup.json
  • 08:14 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1020.eqiad.wmnet with OS bullseye
  • 08:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1020.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 08:09 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1020.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 08:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P39218 and previous config saved to /var/cache/conftool/dbconfig/20221111-080536-ladsgroup.json
  • 07:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39217 and previous config saved to /var/cache/conftool/dbconfig/20221111-075028-ladsgroup.json
  • 06:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T321123)', diff saved to https://phabricator.wikimedia.org/P39216 and previous config saved to /var/cache/conftool/dbconfig/20221111-063240-marostegui.json
  • 06:22 vgutierrez: restart varnish on cp4047 to clear VarnishChildRestarted alert - T322903
  • 06:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P39215 and previous config saved to /var/cache/conftool/dbconfig/20221111-061733-marostegui.json
  • 06:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P39214 and previous config saved to /var/cache/conftool/dbconfig/20221111-060227-marostegui.json
  • 05:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T321123)', diff saved to https://phabricator.wikimedia.org/P39213 and previous config saved to /var/cache/conftool/dbconfig/20221111-054720-marostegui.json
  • 05:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2176 (T321123)', diff saved to https://phabricator.wikimedia.org/P39212 and previous config saved to /var/cache/conftool/dbconfig/20221111-054511-marostegui.json
  • 05:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 05:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 05:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T321123)', diff saved to https://phabricator.wikimedia.org/P39211 and previous config saved to /var/cache/conftool/dbconfig/20221111-054449-marostegui.json
  • 05:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P39210 and previous config saved to /var/cache/conftool/dbconfig/20221111-052943-marostegui.json
  • 05:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P39209 and previous config saved to /var/cache/conftool/dbconfig/20221111-051436-marostegui.json
  • 04:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T321123)', diff saved to https://phabricator.wikimedia.org/P39208 and previous config saved to /var/cache/conftool/dbconfig/20221111-045930-marostegui.json
  • 04:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2174 (T321123)', diff saved to https://phabricator.wikimedia.org/P39207 and previous config saved to /var/cache/conftool/dbconfig/20221111-045720-marostegui.json
  • 04:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 04:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 04:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T321123)', diff saved to https://phabricator.wikimedia.org/P39206 and previous config saved to /var/cache/conftool/dbconfig/20221111-045659-marostegui.json
  • 04:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P39205 and previous config saved to /var/cache/conftool/dbconfig/20221111-044152-marostegui.json
  • 04:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P39204 and previous config saved to /var/cache/conftool/dbconfig/20221111-042646-marostegui.json
  • 04:15 ejegg: civicrm upgraded from fd60273a to 93fa3f37
  • 04:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T321123)', diff saved to https://phabricator.wikimedia.org/P39203 and previous config saved to /var/cache/conftool/dbconfig/20221111-041139-marostegui.json
  • 04:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2173 (T321123)', diff saved to https://phabricator.wikimedia.org/P39202 and previous config saved to /var/cache/conftool/dbconfig/20221111-041030-marostegui.json
  • 04:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 04:10 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 04:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 04:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 04:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39201 and previous config saved to /var/cache/conftool/dbconfig/20221111-040953-marostegui.json
  • 03:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P39200 and previous config saved to /var/cache/conftool/dbconfig/20221111-035447-marostegui.json
  • 03:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P39199 and previous config saved to /var/cache/conftool/dbconfig/20221111-033940-marostegui.json
  • 03:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39198 and previous config saved to /var/cache/conftool/dbconfig/20221111-032434-marostegui.json
  • 03:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39197 and previous config saved to /var/cache/conftool/dbconfig/20221111-032224-marostegui.json
  • 03:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 03:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 03:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39196 and previous config saved to /var/cache/conftool/dbconfig/20221111-032203-marostegui.json
  • 03:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39195 and previous config saved to /var/cache/conftool/dbconfig/20221111-031358-ladsgroup.json
  • 03:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P39194 and previous config saved to /var/cache/conftool/dbconfig/20221111-030656-marostegui.json
  • 02:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P39193 and previous config saved to /var/cache/conftool/dbconfig/20221111-025851-ladsgroup.json
  • 02:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P39192 and previous config saved to /var/cache/conftool/dbconfig/20221111-025150-marostegui.json
  • 02:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P39191 and previous config saved to /var/cache/conftool/dbconfig/20221111-024345-ladsgroup.json
  • 02:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39190 and previous config saved to /var/cache/conftool/dbconfig/20221111-023643-marostegui.json
  • 02:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2167:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P39189 and previous config saved to /var/cache/conftool/dbconfig/20221111-023534-marostegui.json
  • 02:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 02:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 02:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T321123)', diff saved to https://phabricator.wikimedia.org/P39188 and previous config saved to /var/cache/conftool/dbconfig/20221111-023513-marostegui.json
  • 02:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39187 and previous config saved to /var/cache/conftool/dbconfig/20221111-023252-ladsgroup.json
  • 02:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 02:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 02:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T318605)', diff saved to https://phabricator.wikimedia.org/P39186 and previous config saved to /var/cache/conftool/dbconfig/20221111-023231-ladsgroup.json
  • 02:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39185 and previous config saved to /var/cache/conftool/dbconfig/20221111-022838-ladsgroup.json
  • 02:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2182 (T322618)', diff saved to https://phabricator.wikimedia.org/P39184 and previous config saved to /var/cache/conftool/dbconfig/20221111-022619-ladsgroup.json
  • 02:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 02:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 02:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39183 and previous config saved to /var/cache/conftool/dbconfig/20221111-022557-ladsgroup.json
  • 02:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P39182 and previous config saved to /var/cache/conftool/dbconfig/20221111-022006-marostegui.json
  • 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T318605)', diff saved to https://phabricator.wikimedia.org/P39181 and previous config saved to /var/cache/conftool/dbconfig/20221111-021738-ladsgroup.json
  • 02:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P39180 and previous config saved to /var/cache/conftool/dbconfig/20221111-021725-ladsgroup.json
  • 02:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39179 and previous config saved to /var/cache/conftool/dbconfig/20221111-021717-ladsgroup.json
  • 02:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P39178 and previous config saved to /var/cache/conftool/dbconfig/20221111-021051-ladsgroup.json
  • 02:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P39177 and previous config saved to /var/cache/conftool/dbconfig/20221111-020500-marostegui.json
  • 02:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P39176 and previous config saved to /var/cache/conftool/dbconfig/20221111-020218-ladsgroup.json
  • 02:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P39175 and previous config saved to /var/cache/conftool/dbconfig/20221111-020211-ladsgroup.json
  • 01:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P39174 and previous config saved to /var/cache/conftool/dbconfig/20221111-015544-ladsgroup.json
  • 01:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T321123)', diff saved to https://phabricator.wikimedia.org/P39173 and previous config saved to /var/cache/conftool/dbconfig/20221111-014953-marostegui.json
  • 01:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2153 (T321123)', diff saved to https://phabricator.wikimedia.org/P39172 and previous config saved to /var/cache/conftool/dbconfig/20221111-014744-marostegui.json
  • 01:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 01:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 01:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T321123)', diff saved to https://phabricator.wikimedia.org/P39171 and previous config saved to /var/cache/conftool/dbconfig/20221111-014722-marostegui.json
  • 01:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T318605)', diff saved to https://phabricator.wikimedia.org/P39170 and previous config saved to /var/cache/conftool/dbconfig/20221111-014712-ladsgroup.json
  • 01:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P39169 and previous config saved to /var/cache/conftool/dbconfig/20221111-014704-ladsgroup.json
  • 01:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39168 and previous config saved to /var/cache/conftool/dbconfig/20221111-014037-ladsgroup.json
  • 01:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39167 and previous config saved to /var/cache/conftool/dbconfig/20221111-013818-ladsgroup.json
  • 01:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 01:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 01:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39166 and previous config saved to /var/cache/conftool/dbconfig/20221111-013756-ladsgroup.json
  • 01:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P39165 and previous config saved to /var/cache/conftool/dbconfig/20221111-013209-marostegui.json
  • 01:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39164 and previous config saved to /var/cache/conftool/dbconfig/20221111-013157-ladsgroup.json
  • 01:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P39163 and previous config saved to /var/cache/conftool/dbconfig/20221111-012250-ladsgroup.json
  • 01:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P39162 and previous config saved to /var/cache/conftool/dbconfig/20221111-011703-marostegui.json
  • 01:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P39161 and previous config saved to /var/cache/conftool/dbconfig/20221111-010743-ladsgroup.json
  • 01:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T321123)', diff saved to https://phabricator.wikimedia.org/P39160 and previous config saved to /var/cache/conftool/dbconfig/20221111-010156-marostegui.json
  • 00:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2146 (T321123)', diff saved to https://phabricator.wikimedia.org/P39159 and previous config saved to /var/cache/conftool/dbconfig/20221111-005947-marostegui.json
  • 00:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 00:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 00:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T321123)', diff saved to https://phabricator.wikimedia.org/P39158 and previous config saved to /var/cache/conftool/dbconfig/20221111-005925-marostegui.json
  • 00:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39157 and previous config saved to /var/cache/conftool/dbconfig/20221111-005237-ladsgroup.json
  • 00:50 jclark@cumin1001: START - Cookbook sre.hosts.provision for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2168:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39156 and previous config saved to /var/cache/conftool/dbconfig/20221111-005017-ladsgroup.json
  • 00:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 00:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 00:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T322618)', diff saved to https://phabricator.wikimedia.org/P39155 and previous config saved to /var/cache/conftool/dbconfig/20221111-004945-ladsgroup.json
  • 00:47 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 00:45 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 00:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P39154 and previous config saved to /var/cache/conftool/dbconfig/20221111-004419-marostegui.json
  • 00:43 jclark@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:43 jclark@cumin1001: START - Cookbook sre.hosts.provision for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:42 jclark@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:38 jclark@cumin1001: START - Cookbook sre.hosts.provision for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P39153 and previous config saved to /var/cache/conftool/dbconfig/20221111-003438-ladsgroup.json
  • 00:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39152 and previous config saved to /var/cache/conftool/dbconfig/20221111-003141-ladsgroup.json
  • 00:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 00:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 00:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P39151 and previous config saved to /var/cache/conftool/dbconfig/20221111-002913-marostegui.json
  • 00:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P39150 and previous config saved to /var/cache/conftool/dbconfig/20221111-001932-ladsgroup.json
  • 00:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T321123)', diff saved to https://phabricator.wikimedia.org/P39149 and previous config saved to /var/cache/conftool/dbconfig/20221111-001406-marostegui.json
  • 00:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2145 (T321123)', diff saved to https://phabricator.wikimedia.org/P39148 and previous config saved to /var/cache/conftool/dbconfig/20221111-001156-marostegui.json
  • 00:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2145.codfw.wmnet with reason: Maintenance
  • 00:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2145.codfw.wmnet with reason: Maintenance
  • 00:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 00:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 00:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T321123)', diff saved to https://phabricator.wikimedia.org/P39147 and previous config saved to /var/cache/conftool/dbconfig/20221111-001056-marostegui.json
  • 00:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T322618)', diff saved to https://phabricator.wikimedia.org/P39146 and previous config saved to /var/cache/conftool/dbconfig/20221111-000425-ladsgroup.json
  • 00:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2159 (T322618)', diff saved to https://phabricator.wikimedia.org/P39145 and previous config saved to /var/cache/conftool/dbconfig/20221111-000206-ladsgroup.json
  • 00:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 00:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 00:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 00:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 00:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T322618)', diff saved to https://phabricator.wikimedia.org/P39144 and previous config saved to /var/cache/conftool/dbconfig/20221111-000118-ladsgroup.json

2022-11-10

  • 23:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P39143 and previous config saved to /var/cache/conftool/dbconfig/20221110-235549-marostegui.json
  • 23:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P39142 and previous config saved to /var/cache/conftool/dbconfig/20221110-234612-ladsgroup.json
  • 23:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P39141 and previous config saved to /var/cache/conftool/dbconfig/20221110-234043-marostegui.json
  • 23:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P39140 and previous config saved to /var/cache/conftool/dbconfig/20221110-233105-ladsgroup.json
  • 23:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T321123)', diff saved to https://phabricator.wikimedia.org/P39139 and previous config saved to /var/cache/conftool/dbconfig/20221110-232536-marostegui.json
  • 23:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2130 (T321123)', diff saved to https://phabricator.wikimedia.org/P39138 and previous config saved to /var/cache/conftool/dbconfig/20221110-232327-marostegui.json
  • 23:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 23:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 23:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T321123)', diff saved to https://phabricator.wikimedia.org/P39137 and previous config saved to /var/cache/conftool/dbconfig/20221110-232305-marostegui.json
  • 23:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T322618)', diff saved to https://phabricator.wikimedia.org/P39136 and previous config saved to /var/cache/conftool/dbconfig/20221110-231558-ladsgroup.json
  • 23:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2150 (T322618)', diff saved to https://phabricator.wikimedia.org/P39135 and previous config saved to /var/cache/conftool/dbconfig/20221110-231339-ladsgroup.json
  • 23:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 23:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 23:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T322618)', diff saved to https://phabricator.wikimedia.org/P39134 and previous config saved to /var/cache/conftool/dbconfig/20221110-231306-ladsgroup.json
  • 23:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P39133 and previous config saved to /var/cache/conftool/dbconfig/20221110-230759-marostegui.json
  • 22:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P39132 and previous config saved to /var/cache/conftool/dbconfig/20221110-225759-ladsgroup.json
  • 22:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P39131 and previous config saved to /var/cache/conftool/dbconfig/20221110-225253-marostegui.json
  • 22:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P39130 and previous config saved to /var/cache/conftool/dbconfig/20221110-224253-ladsgroup.json
  • 22:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T321123)', diff saved to https://phabricator.wikimedia.org/P39129 and previous config saved to /var/cache/conftool/dbconfig/20221110-223746-marostegui.json
  • 22:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2116 (T321123)', diff saved to https://phabricator.wikimedia.org/P39128 and previous config saved to /var/cache/conftool/dbconfig/20221110-223537-marostegui.json
  • 22:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 22:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 22:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T321123)', diff saved to https://phabricator.wikimedia.org/P39127 and previous config saved to /var/cache/conftool/dbconfig/20221110-223515-marostegui.json
  • 22:27 eileen: thank you back on config revision changed from bbdd4315 to 2bb73bb1
  • 22:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T322618)', diff saved to https://phabricator.wikimedia.org/P39126 and previous config saved to /var/cache/conftool/dbconfig/20221110-222746-ladsgroup.json
  • 22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2122 (T322618)', diff saved to https://phabricator.wikimedia.org/P39125 and previous config saved to /var/cache/conftool/dbconfig/20221110-222526-ladsgroup.json
  • 22:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 22:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T322618)', diff saved to https://phabricator.wikimedia.org/P39124 and previous config saved to /var/cache/conftool/dbconfig/20221110-222505-ladsgroup.json
  • 22:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P39123 and previous config saved to /var/cache/conftool/dbconfig/20221110-222009-marostegui.json
  • 22:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P39122 and previous config saved to /var/cache/conftool/dbconfig/20221110-220958-ladsgroup.json
  • 22:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P39121 and previous config saved to /var/cache/conftool/dbconfig/20221110-220502-marostegui.json
  • 21:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P39120 and previous config saved to /var/cache/conftool/dbconfig/20221110-215452-ladsgroup.json
  • 21:53 jgleeson: payments updated from 17cd1956 to a058fdbc
  • 21:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 (T321123)', diff saved to https://phabricator.wikimedia.org/P39119 and previous config saved to /var/cache/conftool/dbconfig/20221110-214956-marostegui.json
  • 21:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2103 (T321123)', diff saved to https://phabricator.wikimedia.org/P39118 and previous config saved to /var/cache/conftool/dbconfig/20221110-214746-marostegui.json
  • 21:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 21:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 21:47 greg-g: 3:43:33 <eileen> !civicrm upgraded from 6c2e07e0 to fd60273a
  • 21:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 21:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 21:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 21:42 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 21:42 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 21:42 eileen: process-control config revision changed from 4e438cf5 to bbdd4315 (disable thank you)
  • 21:42 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 21:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T321123)', diff saved to https://phabricator.wikimedia.org/P39117 and previous config saved to /var/cache/conftool/dbconfig/20221110-214240-marostegui.json
  • 21:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T322618)', diff saved to https://phabricator.wikimedia.org/P39116 and previous config saved to /var/cache/conftool/dbconfig/20221110-213945-ladsgroup.json
  • 21:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2121 (T322618)', diff saved to https://phabricator.wikimedia.org/P39115 and previous config saved to /var/cache/conftool/dbconfig/20221110-213726-ladsgroup.json
  • 21:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 21:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 21:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T322618)', diff saved to https://phabricator.wikimedia.org/P39114 and previous config saved to /var/cache/conftool/dbconfig/20221110-213704-ladsgroup.json
  • 21:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P39113 and previous config saved to /var/cache/conftool/dbconfig/20221110-212734-marostegui.json
  • 21:27 dancy@deploy1002: Installation of scap version "4.28.0" completed for 559 hosts
  • 21:26 dancy@deploy1002: Installing scap version "4.28.0" for 559 hosts
  • 21:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P39112 and previous config saved to /var/cache/conftool/dbconfig/20221110-212158-ladsgroup.json
  • 21:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P39111 and previous config saved to /var/cache/conftool/dbconfig/20221110-211227-marostegui.json
  • 21:10 mutante: deploy1002 - armed the keyholder with deployment keys - 2 hours ago alerts started that it was not armed (does it notify people?) - got pinged that deployers got scap problems - unknown why it was disarmed - now it is armed again
  • 21:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P39110 and previous config saved to /var/cache/conftool/dbconfig/20221110-210651-ladsgroup.json
  • 21:04 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:03 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 21:03 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕞🍵 sudo cumin -b 8 A:cp 'run-puppet-agent --enable T306580'
  • 20:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T321123)', diff saved to https://phabricator.wikimedia.org/P39109 and previous config saved to /var/cache/conftool/dbconfig/20221110-205720-marostegui.json
  • 20:56 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1196 (T321123)', diff saved to https://phabricator.wikimedia.org/P39108 and previous config saved to /var/cache/conftool/dbconfig/20221110-205613-marostegui.json
  • 20:56 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 20:55 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 20:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T321123)', diff saved to https://phabricator.wikimedia.org/P39107 and previous config saved to /var/cache/conftool/dbconfig/20221110-205552-marostegui.json
  • 20:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T322618)', diff saved to https://phabricator.wikimedia.org/P39106 and previous config saved to /var/cache/conftool/dbconfig/20221110-205145-ladsgroup.json
  • 20:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2120 (T322618)', diff saved to https://phabricator.wikimedia.org/P39105 and previous config saved to /var/cache/conftool/dbconfig/20221110-204925-ladsgroup.json
  • 20:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 20:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 20:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T322618)', diff saved to https://phabricator.wikimedia.org/P39104 and previous config saved to /var/cache/conftool/dbconfig/20221110-204904-ladsgroup.json
  • 20:43 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:42 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 20:41 dancy@deploy1002: Installing scap version "4.28.0" for 559 hosts
  • 20:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P39103 and previous config saved to /var/cache/conftool/dbconfig/20221110-204045-marostegui.json
  • 20:40 jclark@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:39 jclark@cumin1001: START - Cookbook sre.dns.netbox
  • 20:36 cdanis: ✔️ cdanis@cp3053.esams.wmnet ~ 🕞🍵 sudo run-puppet-agent --enable T306580
  • 20:36 cdanis: ✔️ cdanis@cp3052.esams.wmnet ~ 🕞🍵 sudo run-puppet-agent --enable T306580
  • 20:35 cdanis: ✔️ cdanis@cp2027.codfw.wmnet ~ 🕞🍵 sudo run-puppet-agent --enable T306580
  • 20:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P39102 and previous config saved to /var/cache/conftool/dbconfig/20221110-203357-ladsgroup.json
  • 20:33 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕞🍵 sudo cumin A:cp 'disable-puppet T306580'
  • 20:32 dancy@deploy1002: Installing scap version "4.28.0" for 559 hosts
  • 20:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T321130)', diff saved to https://phabricator.wikimedia.org/P39101 and previous config saved to /var/cache/conftool/dbconfig/20221110-202744-marostegui.json
  • 20:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P39100 and previous config saved to /var/cache/conftool/dbconfig/20221110-202539-marostegui.json
  • 20:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P39099 and previous config saved to /var/cache/conftool/dbconfig/20221110-201851-ladsgroup.json
  • 20:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P39098 and previous config saved to /var/cache/conftool/dbconfig/20221110-201237-marostegui.json
  • 20:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T321123)', diff saved to https://phabricator.wikimedia.org/P39097 and previous config saved to /var/cache/conftool/dbconfig/20221110-201032-marostegui.json
  • 20:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1186 (T321123)', diff saved to https://phabricator.wikimedia.org/P39096 and previous config saved to /var/cache/conftool/dbconfig/20221110-200924-marostegui.json
  • 20:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 20:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 20:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T321123)', diff saved to https://phabricator.wikimedia.org/P39095 and previous config saved to /var/cache/conftool/dbconfig/20221110-200903-marostegui.json
  • 20:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T322618)', diff saved to https://phabricator.wikimedia.org/P39094 and previous config saved to /var/cache/conftool/dbconfig/20221110-200344-ladsgroup.json
  • 20:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2108 (T322618)', diff saved to https://phabricator.wikimedia.org/P39093 and previous config saved to /var/cache/conftool/dbconfig/20221110-200125-ladsgroup.json
  • 20:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 20:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 20:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 20:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 20:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 20:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 19:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 19:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 19:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T322618)', diff saved to https://phabricator.wikimedia.org/P39092 and previous config saved to /var/cache/conftool/dbconfig/20221110-195938-ladsgroup.json
  • 19:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P39091 and previous config saved to /var/cache/conftool/dbconfig/20221110-195731-marostegui.json
  • 19:57 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:55 dzahn@cumin2002: START - Cookbook sre.dns.netbox
  • 19:54 mutante: netbox - deleting special case phab2001-vcs.codfw.wmnet IPv4 (10.192.32.149) and IPv6 (2620:0:860:103:10:192:32:149) - T296022 - T322250
  • 19:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P39090 and previous config saved to /var/cache/conftool/dbconfig/20221110-195357-marostegui.json
  • 19:52 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:51 robh@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1033.eqiad.wmnet with OS bullseye
  • 19:51 dzahn@cumin2002: START - Cookbook sre.dns.netbox
  • 19:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39089 and previous config saved to /var/cache/conftool/dbconfig/20221110-194431-ladsgroup.json
  • 19:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T321130)', diff saved to https://phabricator.wikimedia.org/P39088 and previous config saved to /var/cache/conftool/dbconfig/20221110-194224-marostegui.json
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2178 (T321130)', diff saved to https://phabricator.wikimedia.org/P39087 and previous config saved to /var/cache/conftool/dbconfig/20221110-194031-marostegui.json
  • 19:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39086 and previous config saved to /var/cache/conftool/dbconfig/20221110-194009-marostegui.json
  • 19:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P39085 and previous config saved to /var/cache/conftool/dbconfig/20221110-193850-marostegui.json
  • 19:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2136 (T318605)', diff saved to https://phabricator.wikimedia.org/P39084 and previous config saved to /var/cache/conftool/dbconfig/20221110-193459-ladsgroup.json
  • 19:34 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1033.eqiad.wmnet with reason: host reimage
  • 19:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 19:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 19:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T318605)', diff saved to https://phabricator.wikimedia.org/P39083 and previous config saved to /var/cache/conftool/dbconfig/20221110-193437-ladsgroup.json
  • 19:32 damilare: civicrm upgraded from 07fdeed5 to 6c2e07e0
  • 19:31 robh@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1033.eqiad.wmnet with reason: host reimage
  • 19:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P39082 and previous config saved to /var/cache/conftool/dbconfig/20221110-192925-ladsgroup.json
  • 19:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P39081 and previous config saved to /var/cache/conftool/dbconfig/20221110-192503-marostegui.json
  • 19:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T321123)', diff saved to https://phabricator.wikimedia.org/P39080 and previous config saved to /var/cache/conftool/dbconfig/20221110-192343-marostegui.json
  • 19:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1184 (T321123)', diff saved to https://phabricator.wikimedia.org/P39079 and previous config saved to /var/cache/conftool/dbconfig/20221110-192236-marostegui.json
  • 19:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 19:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 19:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T321123)', diff saved to https://phabricator.wikimedia.org/P39078 and previous config saved to /var/cache/conftool/dbconfig/20221110-192215-marostegui.json
  • 19:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P39077 and previous config saved to /var/cache/conftool/dbconfig/20221110-191930-ladsgroup.json
  • 19:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 19:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 19:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39076 and previous config saved to /var/cache/conftool/dbconfig/20221110-191900-ladsgroup.json
  • 19:18 robh@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1033.eqiad.wmnet with OS bullseye
  • 19:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T322618)', diff saved to https://phabricator.wikimedia.org/P39075 and previous config saved to /var/cache/conftool/dbconfig/20221110-191418-ladsgroup.json
  • 19:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T322618)', diff saved to https://phabricator.wikimedia.org/P39074 and previous config saved to /var/cache/conftool/dbconfig/20221110-191208-ladsgroup.json
  • 19:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 19:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 19:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P39073 and previous config saved to /var/cache/conftool/dbconfig/20221110-191146-ladsgroup.json
  • 19:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P39072 and previous config saved to /var/cache/conftool/dbconfig/20221110-190957-marostegui.json
  • 19:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P39071 and previous config saved to /var/cache/conftool/dbconfig/20221110-190708-marostegui.json
  • 19:05 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts phab2001.codfw.wmnet
  • 19:05 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P39070 and previous config saved to /var/cache/conftool/dbconfig/20221110-190424-ladsgroup.json
  • 19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39069 and previous config saved to /var/cache/conftool/dbconfig/20221110-190353-ladsgroup.json
  • 19:03 dzahn@cumin2002: START - Cookbook sre.dns.netbox
  • 18:58 mutante: phabricator - running decom cookbook on phab2001 - T322250
  • 18:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39068 and previous config saved to /var/cache/conftool/dbconfig/20221110-185640-ladsgroup.json
  • 18:55 dzahn@cumin2002: START - Cookbook sre.hosts.decommission for hosts phab2001.codfw.wmnet
  • 18:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39067 and previous config saved to /var/cache/conftool/dbconfig/20221110-185450-marostegui.json
  • 18:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P39066 and previous config saved to /var/cache/conftool/dbconfig/20221110-185202-marostegui.json
  • 18:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39065 and previous config saved to /var/cache/conftool/dbconfig/20221110-185135-marostegui.json
  • 18:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 18:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 18:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39064 and previous config saved to /var/cache/conftool/dbconfig/20221110-185103-marostegui.json
  • 18:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T318605)', diff saved to https://phabricator.wikimedia.org/P39063 and previous config saved to /var/cache/conftool/dbconfig/20221110-184917-ladsgroup.json
  • 18:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39062 and previous config saved to /var/cache/conftool/dbconfig/20221110-184847-ladsgroup.json
  • 18:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39061 and previous config saved to /var/cache/conftool/dbconfig/20221110-184133-ladsgroup.json
  • 18:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T321123)', diff saved to https://phabricator.wikimedia.org/P39060 and previous config saved to /var/cache/conftool/dbconfig/20221110-183655-marostegui.json
  • 18:36 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4008.ulsfo.wmnet with reason: downtimed as we are resolving issues with LVS configuration
  • 18:36 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on lvs4008.ulsfo.wmnet with reason: downtimed as we are resolving issues with LVS configuration
  • 18:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P39059 and previous config saved to /var/cache/conftool/dbconfig/20221110-183556-marostegui.json
  • 18:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1169 (T321123)', diff saved to https://phabricator.wikimedia.org/P39058 and previous config saved to /var/cache/conftool/dbconfig/20221110-183548-marostegui.json
  • 18:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 18:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 18:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 18:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T321123)', diff saved to https://phabricator.wikimedia.org/P39057 and previous config saved to /var/cache/conftool/dbconfig/20221110-183455-marostegui.json
  • 18:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39056 and previous config saved to /var/cache/conftool/dbconfig/20221110-183340-ladsgroup.json
  • 18:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P39055 and previous config saved to /var/cache/conftool/dbconfig/20221110-182627-ladsgroup.json
  • 18:26 ladsgroup@deploy1002: Synchronized portals: (no justification provided) (duration: 03m 38s)
  • 18:22 ladsgroup@deploy1002: Synchronized portals/wikipedia.org/assets: (no justification provided) (duration: 03m 46s)
  • 18:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P39054 and previous config saved to /var/cache/conftool/dbconfig/20221110-182049-marostegui.json
  • 18:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P39053 and previous config saved to /var/cache/conftool/dbconfig/20221110-181948-marostegui.json
  • 18:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 18:18 ladsgroup@deploy1002: Finished scap: Backport for Bump portals to HEAD (T273179) (duration: 05m 14s)
  • 18:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 18:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 18:17 dcausse@deploy1002: Finished deploy [wikimedia/discovery/analytics@a030f5f]: T320656: convert_to_esbulk: fix typo in config (duration: 02m 22s)
  • 18:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 18:15 dcausse@deploy1002: Started deploy [wikimedia/discovery/analytics@a030f5f]: T320656: convert_to_esbulk: fix typo in config
  • 18:13 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for Bump portals to HEAD (T273179) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 18:12 ladsgroup@deploy1002: Started scap: Backport for Bump portals to HEAD (T273179)
  • 18:09 volans: upgrading spicerack to 5.0.0 on cumin hosts
  • 18:05 volans: uploaded spicerack_5.0.0 to apt.wikimedia.org bullseye-wikimedia
  • 18:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39052 and previous config saved to /var/cache/conftool/dbconfig/20221110-180543-marostegui.json
  • 18:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P39051 and previous config saved to /var/cache/conftool/dbconfig/20221110-180442-marostegui.json
  • 18:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39050 and previous config saved to /var/cache/conftool/dbconfig/20221110-180228-marostegui.json
  • 18:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 18:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 18:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39049 and previous config saved to /var/cache/conftool/dbconfig/20221110-180206-marostegui.json
  • 18:01 volans: uploaded python3-gjson_0.3.0 to apt.wikimedia.org bullseye-wikimedia,unstable-wikimedia
  • 17:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T321123)', diff saved to https://phabricator.wikimedia.org/P39048 and previous config saved to /var/cache/conftool/dbconfig/20221110-174935-marostegui.json
  • 17:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1135 (T321123)', diff saved to https://phabricator.wikimedia.org/P39047 and previous config saved to /var/cache/conftool/dbconfig/20221110-174828-marostegui.json
  • 17:48 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 17:48 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 17:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T321123)', diff saved to https://phabricator.wikimedia.org/P39046 and previous config saved to /var/cache/conftool/dbconfig/20221110-174806-marostegui.json
  • 17:47 dcausse@deploy1002: Finished deploy [wikimedia/discovery/analytics@84dd7b5]: T320656: image_suggestions: schedule ad hoc dataset to fix improper suggestions (duration: 02m 18s)
  • 17:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P39045 and previous config saved to /var/cache/conftool/dbconfig/20221110-174659-marostegui.json
  • 17:44 dcausse@deploy1002: Started deploy [wikimedia/discovery/analytics@84dd7b5]: T320656: image_suggestions: schedule ad hoc dataset to fix improper suggestions
  • 17:37 sukhe: [done] running sukhe@cumin2002:~$ homer "cr*-ulsfo*" commit "Gerrit 855583: sites.yaml: add lvs4008 (ulsfo hardware refresh)"
  • 17:36 sukhe: running sukhe@cumin2002:~$ homer "cr*-ulsfo*" commit "Gerrit 855583: sites.yaml: add lvs4008 (ulsfo hardware refresh)"
  • 17:35 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4008.ulsfo.wmnet with OS buster
  • 17:34 rzl: rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_growthexperiments-userImpactDelete.service # test run for T322706 T322541
  • 17:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P39044 and previous config saved to /var/cache/conftool/dbconfig/20221110-173300-marostegui.json
  • 17:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P39043 and previous config saved to /var/cache/conftool/dbconfig/20221110-173153-marostegui.json
  • 17:28 urandom: restarting bootstrap of aqs1016-a -- T307802
  • 17:26 urandom: increasing stream throughput to 400mbit, aqs1011-{a,b} & aqs1013-{a,b} -- T307802
  • 17:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P39042 and previous config saved to /var/cache/conftool/dbconfig/20221110-172611-ladsgroup.json
  • 17:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 17:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 17:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T322618)', diff saved to https://phabricator.wikimedia.org/P39041 and previous config saved to /var/cache/conftool/dbconfig/20221110-172549-ladsgroup.json
  • 17:23 rzl: rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_growthexperiments-userImpactUpdateRecentlyEdited.service # test run for T322706 T322541
  • 17:18 robh@cumin1001: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ganeti1033']
  • 17:18 rzl: rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_growthexperiments-userImpactUpdateRecentlyRegistered.service # test run for T322706 T322541
  • 17:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P39040 and previous config saved to /var/cache/conftool/dbconfig/20221110-171753-marostegui.json
  • 17:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39039 and previous config saved to /var/cache/conftool/dbconfig/20221110-171646-marostegui.json
  • 17:13 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 17:13 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39038 and previous config saved to /var/cache/conftool/dbconfig/20221110-171329-marostegui.json
  • 17:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 17:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 17:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T321130)', diff saved to https://phabricator.wikimedia.org/P39037 and previous config saved to /var/cache/conftool/dbconfig/20221110-171308-marostegui.json
  • 17:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39036 and previous config saved to /var/cache/conftool/dbconfig/20221110-171043-ladsgroup.json
  • 17:10 robh@cumin1001: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1033']
  • 17:09 robh@cumin1001: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ganeti1033']
  • 17:09 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 17:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T321123)', diff saved to https://phabricator.wikimedia.org/P39035 and previous config saved to /var/cache/conftool/dbconfig/20221110-170247-marostegui.json
  • 17:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1134 (T321123)', diff saved to https://phabricator.wikimedia.org/P39034 and previous config saved to /var/cache/conftool/dbconfig/20221110-170139-marostegui.json
  • 17:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1134.eqiad.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1134.eqiad.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1133.eqiad.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1133.eqiad.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 (T321123)', diff saved to https://phabricator.wikimedia.org/P39033 and previous config saved to /var/cache/conftool/dbconfig/20221110-170102-marostegui.json
  • 16:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P39032 and previous config saved to /var/cache/conftool/dbconfig/20221110-165802-marostegui.json
  • 16:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39031 and previous config saved to /var/cache/conftool/dbconfig/20221110-165536-ladsgroup.json
  • 16:53 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host lvs4008.ulsfo.wmnet with OS buster
  • 16:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P39030 and previous config saved to /var/cache/conftool/dbconfig/20221110-164556-marostegui.json
  • 16:44 sukhe@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs4008.ulsfo.wmnet with OS buster
  • 16:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P39029 and previous config saved to /var/cache/conftool/dbconfig/20221110-164255-marostegui.json
  • 16:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T322618)', diff saved to https://phabricator.wikimedia.org/P39028 and previous config saved to /var/cache/conftool/dbconfig/20221110-164030-ladsgroup.json
  • 16:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T322618)', diff saved to https://phabricator.wikimedia.org/P39027 and previous config saved to /var/cache/conftool/dbconfig/20221110-163819-ladsgroup.json
  • 16:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 16:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 16:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T322618)', diff saved to https://phabricator.wikimedia.org/P39026 and previous config saved to /var/cache/conftool/dbconfig/20221110-163758-ladsgroup.json
  • 16:37 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 16:34 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
  • 16:33 robh@cumin1001: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1033']
  • 16:33 robh@cumin1001: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti1033']
  • 16:32 robh@cumin1001: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1033']
  • 16:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P39025 and previous config saved to /var/cache/conftool/dbconfig/20221110-163049-marostegui.json
  • 16:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T321130)', diff saved to https://phabricator.wikimedia.org/P39024 and previous config saved to /var/cache/conftool/dbconfig/20221110-162749-marostegui.json
  • 16:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2128 (T321130)', diff saved to https://phabricator.wikimedia.org/P39023 and previous config saved to /var/cache/conftool/dbconfig/20221110-162453-marostegui.json
  • 16:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 16:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 16:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 16:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 16:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T321130)', diff saved to https://phabricator.wikimedia.org/P39022 and previous config saved to /var/cache/conftool/dbconfig/20221110-162416-marostegui.json
  • 16:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P39021 and previous config saved to /var/cache/conftool/dbconfig/20221110-162251-ladsgroup.json
  • 16:16 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host lvs4008.ulsfo.wmnet with OS buster
  • 16:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 (T321123)', diff saved to https://phabricator.wikimedia.org/P39020 and previous config saved to /var/cache/conftool/dbconfig/20221110-161543-marostegui.json
  • 16:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1132 (T321123)', diff saved to https://phabricator.wikimedia.org/P39019 and previous config saved to /var/cache/conftool/dbconfig/20221110-161435-marostegui.json
  • 16:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1132.eqiad.wmnet with reason: Maintenance
  • 16:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1132.eqiad.wmnet with reason: Maintenance
  • 16:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128 (T321123)', diff saved to https://phabricator.wikimedia.org/P39018 and previous config saved to /var/cache/conftool/dbconfig/20221110-161413-marostegui.json
  • 16:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P39017 and previous config saved to /var/cache/conftool/dbconfig/20221110-160906-marostegui.json
  • 16:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P39016 and previous config saved to /var/cache/conftool/dbconfig/20221110-160745-ladsgroup.json
  • 16:03 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2002-dev.codfw.wmnet with OS bullseye
  • 15:59 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 15:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P39015 and previous config saved to /var/cache/conftool/dbconfig/20221110-155907-marostegui.json
  • 15:56 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-eqiad
  • 15:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P39014 and previous config saved to /var/cache/conftool/dbconfig/20221110-155400-marostegui.json
  • 15:53 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T322618)', diff saved to https://phabricator.wikimedia.org/P39013 and previous config saved to /var/cache/conftool/dbconfig/20221110-155238-ladsgroup.json
  • 15:50 aborrero@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
  • 15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T322618)', diff saved to https://phabricator.wikimedia.org/P39012 and previous config saved to /var/cache/conftool/dbconfig/20221110-154827-ladsgroup.json
  • 15:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 15:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 15:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 15:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 15:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P39011 and previous config saved to /var/cache/conftool/dbconfig/20221110-154746-ladsgroup.json
  • 15:47 aborrero@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
  • 15:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P39010 and previous config saved to /var/cache/conftool/dbconfig/20221110-154400-marostegui.json
  • 15:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T321130)', diff saved to https://phabricator.wikimedia.org/P39009 and previous config saved to /var/cache/conftool/dbconfig/20221110-153853-marostegui.json
  • 15:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2123 (T321130)', diff saved to https://phabricator.wikimedia.org/P39008 and previous config saved to /var/cache/conftool/dbconfig/20221110-153559-marostegui.json
  • 15:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 15:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 15:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T321130)', diff saved to https://phabricator.wikimedia.org/P39007 and previous config saved to /var/cache/conftool/dbconfig/20221110-153537-marostegui.json
  • 15:33 aborrero@cumin2002: START - Cookbook sre.hosts.reimage for host cloudgw2002-dev.codfw.wmnet with OS bullseye
  • 15:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P39006 and previous config saved to /var/cache/conftool/dbconfig/20221110-153240-ladsgroup.json
  • 15:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1128 (T321123)', diff saved to https://phabricator.wikimedia.org/P39005 and previous config saved to /var/cache/conftool/dbconfig/20221110-152854-marostegui.json
  • 15:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P39004 and previous config saved to /var/cache/conftool/dbconfig/20221110-152031-marostegui.json
  • 15:19 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1128 (T321123)', diff saved to https://phabricator.wikimedia.org/P39003 and previous config saved to /var/cache/conftool/dbconfig/20221110-151945-marostegui.json
  • 15:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1128.eqiad.wmnet with reason: Maintenance
  • 15:19 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1128.eqiad.wmnet with reason: Maintenance
  • 15:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T321123)', diff saved to https://phabricator.wikimedia.org/P39002 and previous config saved to /var/cache/conftool/dbconfig/20221110-151924-marostegui.json
  • 15:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P39001 and previous config saved to /var/cache/conftool/dbconfig/20221110-151733-ladsgroup.json
  • 15:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 15:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 15:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 15:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 15:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P39000 and previous config saved to /var/cache/conftool/dbconfig/20221110-150524-marostegui.json
  • 15:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P38999 and previous config saved to /var/cache/conftool/dbconfig/20221110-150417-marostegui.json
  • 15:03 kharlan@deploy1002: Finished scap: Backport for refreshUserImpactData: Add option to use job queue (T322706), refreshUserImpactData: Add feature flag (T313395) (duration: 04m 47s)
  • 15:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38998 and previous config saved to /var/cache/conftool/dbconfig/20221110-150226-ladsgroup.json
  • 15:02 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 15:01 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 15:01 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 15:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38997 and previous config saved to /var/cache/conftool/dbconfig/20221110-150015-ladsgroup.json
  • 15:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 15:00 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 14:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38996 and previous config saved to /var/cache/conftool/dbconfig/20221110-145953-ladsgroup.json
  • 14:58 kharlan@deploy1002: kharlan and kharlan: Backport for refreshUserImpactData: Add option to use job queue (T322706), refreshUserImpactData: Add feature flag (T313395) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 14:58 kharlan@deploy1002: Started scap: Backport for refreshUserImpactData: Add option to use job queue (T322706), refreshUserImpactData: Add feature flag (T313395)
  • 14:51 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-codfw
  • 14:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111 (T321130)', diff saved to https://phabricator.wikimedia.org/P38995 and previous config saved to /var/cache/conftool/dbconfig/20221110-145018-marostegui.json
  • 14:49 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp6016.drmrs.wmnet
  • 14:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P38994 and previous config saved to /var/cache/conftool/dbconfig/20221110-144911-marostegui.json
  • 14:46 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet
  • 14:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2111 (T321130)', diff saved to https://phabricator.wikimedia.org/P38993 and previous config saved to /var/cache/conftool/dbconfig/20221110-144602-marostegui.json
  • 14:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 14:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 14:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P38992 and previous config saved to /var/cache/conftool/dbconfig/20221110-144447-ladsgroup.json
  • 14:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 14:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 14:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 14:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 14:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T321130)', diff saved to https://phabricator.wikimedia.org/P38991 and previous config saved to /var/cache/conftool/dbconfig/20221110-144121-marostegui.json
  • 14:39 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:38 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:38 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:38 kharlan@deploy1002: backport aborted: (duration: 02m 16s)
  • 14:37 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:35 kharlan@deploy1002: Finished scap: Backport for GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled (T313395) (duration: 04m 57s)
  • 14:34 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
  • 14:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38990 and previous config saved to /var/cache/conftool/dbconfig/20221110-143404-marostegui.json
  • 14:33 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 14:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38989 and previous config saved to /var/cache/conftool/dbconfig/20221110-143256-marostegui.json
  • 14:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1119.eqiad.wmnet with reason: Maintenance
  • 14:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1119.eqiad.wmnet with reason: Maintenance
  • 14:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 (T321123)', diff saved to https://phabricator.wikimedia.org/P38988 and previous config saved to /var/cache/conftool/dbconfig/20221110-143235-marostegui.json
  • 14:31 kharlan@deploy1002: kharlan and kharlan: Backport for GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled (T313395) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 14:30 kharlan@deploy1002: Started scap: Backport for GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled (T313395)
  • 14:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P38987 and previous config saved to /var/cache/conftool/dbconfig/20221110-142938-ladsgroup.json
  • 14:29 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 14:28 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 14:28 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
  • 14:27 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
  • 14:27 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
  • 14:26 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 14:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P38986 and previous config saved to /var/cache/conftool/dbconfig/20221110-142614-marostegui.json
  • 14:25 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
  • 14:24 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 14:24 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 14:22 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 14:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:21 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:21 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:21 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 14:19 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 14:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P38985 and previous config saved to /var/cache/conftool/dbconfig/20221110-141728-marostegui.json
  • 14:14 daniel@deploy1002: Finished scap: Backport for mediawiki.org: set VE to new direct mode (duration: 08m 17s)
  • 14:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38984 and previous config saved to /var/cache/conftool/dbconfig/20221110-141431-ladsgroup.json
  • 14:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38983 and previous config saved to /var/cache/conftool/dbconfig/20221110-141220-ladsgroup.json
  • 14:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 14:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 14:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 14:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 14:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T322618)', diff saved to https://phabricator.wikimedia.org/P38982 and previous config saved to /var/cache/conftool/dbconfig/20221110-141141-ladsgroup.json
  • 14:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P38981 and previous config saved to /var/cache/conftool/dbconfig/20221110-141106-marostegui.json
  • 14:06 daniel@deploy1002: daniel and daniel: Backport for mediawiki.org: set VE to new direct mode synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 14:06 daniel@deploy1002: Started scap: Backport for mediawiki.org: set VE to new direct mode
  • 14:04 moritzm: rolling restart of FPM and Apache on mw canaries to pick up expat security update
  • 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P38980 and previous config saved to /var/cache/conftool/dbconfig/20221110-140222-marostegui.json
  • 14:00 moritzm: drain ganeti1020 for eventual reimage to bullseye T311687
  • 13:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P38979 and previous config saved to /var/cache/conftool/dbconfig/20221110-135635-ladsgroup.json
  • 13:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T321130)', diff saved to https://phabricator.wikimedia.org/P38978 and previous config saved to /var/cache/conftool/dbconfig/20221110-135600-marostegui.json
  • 13:53 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D
  • 13:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1200 (T321130)', diff saved to https://phabricator.wikimedia.org/P38977 and previous config saved to /var/cache/conftool/dbconfig/20221110-135334-marostegui.json
  • 13:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 13:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 13:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T321130)', diff saved to https://phabricator.wikimedia.org/P38976 and previous config saved to /var/cache/conftool/dbconfig/20221110-135313-marostegui.json
  • 13:53 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
  • 13:52 moritzm: installing expat securiy updates
  • 13:50 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
  • 13:48 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 (T321123)', diff saved to https://phabricator.wikimedia.org/P38975 and previous config saved to /var/cache/conftool/dbconfig/20221110-134715-marostegui.json
  • 13:46 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 13:46 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 13:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1118 (T321123)', diff saved to https://phabricator.wikimedia.org/P38974 and previous config saved to /var/cache/conftool/dbconfig/20221110-134608-marostegui.json
  • 13:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 13:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1118.eqiad.wmnet with reason: Maintenance
  • 13:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T321123)', diff saved to https://phabricator.wikimedia.org/P38973 and previous config saved to /var/cache/conftool/dbconfig/20221110-134546-marostegui.json
  • 13:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P38972 and previous config saved to /var/cache/conftool/dbconfig/20221110-134128-ladsgroup.json
  • 13:40 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D
  • 13:39 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 13:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38971 and previous config saved to /var/cache/conftool/dbconfig/20221110-133806-marostegui.json
  • 13:37 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 13:37 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 13:36 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 13:36 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 13:35 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 13:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P38970 and previous config saved to /var/cache/conftool/dbconfig/20221110-133040-marostegui.json
  • 13:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T322618)', diff saved to https://phabricator.wikimedia.org/P38969 and previous config saved to /var/cache/conftool/dbconfig/20221110-132622-ladsgroup.json
  • 13:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38968 and previous config saved to /var/cache/conftool/dbconfig/20221110-132300-marostegui.json
  • 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T322618)', diff saved to https://phabricator.wikimedia.org/P38967 and previous config saved to /var/cache/conftool/dbconfig/20221110-132010-ladsgroup.json
  • 13:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 13:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T322618)', diff saved to https://phabricator.wikimedia.org/P38966 and previous config saved to /var/cache/conftool/dbconfig/20221110-131949-ladsgroup.json
  • 13:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P38965 and previous config saved to /var/cache/conftool/dbconfig/20221110-131533-marostegui.json
  • 13:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T321130)', diff saved to https://phabricator.wikimedia.org/P38964 and previous config saved to /var/cache/conftool/dbconfig/20221110-130753-marostegui.json
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1185 (T321130)', diff saved to https://phabricator.wikimedia.org/P38963 and previous config saved to /var/cache/conftool/dbconfig/20221110-130527-marostegui.json
  • 13:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 13:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T321130)', diff saved to https://phabricator.wikimedia.org/P38962 and previous config saved to /var/cache/conftool/dbconfig/20221110-130506-marostegui.json
  • 13:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38961 and previous config saved to /var/cache/conftool/dbconfig/20221110-130443-ladsgroup.json
  • 13:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D
  • 13:04 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D
  • 13:01 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet
  • 13:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1107 (T321123)', diff saved to https://phabricator.wikimedia.org/P38960 and previous config saved to /var/cache/conftool/dbconfig/20221110-130027-marostegui.json
  • 12:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1107 (T321123)', diff saved to https://phabricator.wikimedia.org/P38959 and previous config saved to /var/cache/conftool/dbconfig/20221110-125919-marostegui.json
  • 12:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1107.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38958 and previous config saved to /var/cache/conftool/dbconfig/20221110-125858-marostegui.json
  • 12:53 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet
  • 12:51 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 12:51 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 12:51 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:51 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38957 and previous config saved to /var/cache/conftool/dbconfig/20221110-124959-marostegui.json
  • 12:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38956 and previous config saved to /var/cache/conftool/dbconfig/20221110-124936-ladsgroup.json
  • 12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2119 (T318605)', diff saved to https://phabricator.wikimedia.org/P38955 and previous config saved to /var/cache/conftool/dbconfig/20221110-124805-ladsgroup.json
  • 12:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 12:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T318605)', diff saved to https://phabricator.wikimedia.org/P38954 and previous config saved to /var/cache/conftool/dbconfig/20221110-124743-ladsgroup.json
  • 12:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P38953 and previous config saved to /var/cache/conftool/dbconfig/20221110-124352-marostegui.json
  • 12:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38952 and previous config saved to /var/cache/conftool/dbconfig/20221110-123527-ladsgroup.json
  • 12:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38951 and previous config saved to /var/cache/conftool/dbconfig/20221110-123453-marostegui.json
  • 12:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T322618)', diff saved to https://phabricator.wikimedia.org/P38950 and previous config saved to /var/cache/conftool/dbconfig/20221110-123428-ladsgroup.json
  • 12:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P38949 and previous config saved to /var/cache/conftool/dbconfig/20221110-123237-ladsgroup.json
  • 12:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T322618)', diff saved to https://phabricator.wikimedia.org/P38948 and previous config saved to /var/cache/conftool/dbconfig/20221110-123215-ladsgroup.json
  • 12:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 12:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38947 and previous config saved to /var/cache/conftool/dbconfig/20221110-123153-ladsgroup.json
  • 12:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P38946 and previous config saved to /var/cache/conftool/dbconfig/20221110-122845-marostegui.json
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P38945 and previous config saved to /var/cache/conftool/dbconfig/20221110-122720-ladsgroup.json
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T318605)', diff saved to https://phabricator.wikimedia.org/P38944 and previous config saved to /var/cache/conftool/dbconfig/20221110-122708-ladsgroup.json
  • 12:26 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:26 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:26 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 12:26 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 12:25 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 12:25 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 12:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38943 and previous config saved to /var/cache/conftool/dbconfig/20221110-122020-ladsgroup.json
  • 12:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T321130)', diff saved to https://phabricator.wikimedia.org/P38942 and previous config saved to /var/cache/conftool/dbconfig/20221110-121946-marostegui.json
  • 12:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P38941 and previous config saved to /var/cache/conftool/dbconfig/20221110-121730-ladsgroup.json
  • 12:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P38940 and previous config saved to /var/cache/conftool/dbconfig/20221110-121647-ladsgroup.json
  • 12:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T321130)', diff saved to https://phabricator.wikimedia.org/P38939 and previous config saved to /var/cache/conftool/dbconfig/20221110-121601-marostegui.json
  • 12:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 12:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 12:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38938 and previous config saved to /var/cache/conftool/dbconfig/20221110-121339-marostegui.json
  • 12:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 12:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 12:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38937 and previous config saved to /var/cache/conftool/dbconfig/20221110-121313-marostegui.json
  • 12:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P38936 and previous config saved to /var/cache/conftool/dbconfig/20221110-121202-ladsgroup.json
  • 12:08 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry
  • 12:06 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry
  • 12:06 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:06 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 100%: Index rebuilt', diff saved to https://phabricator.wikimedia.org/P38935 and previous config saved to /var/cache/conftool/dbconfig/20221110-120537-ladsgroup.json
  • 12:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38934 and previous config saved to /var/cache/conftool/dbconfig/20221110-120513-ladsgroup.json
  • 12:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T318605)', diff saved to https://phabricator.wikimedia.org/P38933 and previous config saved to /var/cache/conftool/dbconfig/20221110-120224-ladsgroup.json
  • 12:02 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:02 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:01 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 12:01 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 12:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P38932 and previous config saved to /var/cache/conftool/dbconfig/20221110-120140-ladsgroup.json
  • 11:59 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 11:59 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 11:58 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 11:58 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 11:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P38931 and previous config saved to /var/cache/conftool/dbconfig/20221110-115807-marostegui.json
  • 11:57 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 11:57 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 11:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P38930 and previous config saved to /var/cache/conftool/dbconfig/20221110-115655-ladsgroup.json
  • 11:55 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 11:52 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 75%: Index rebuilt', diff saved to https://phabricator.wikimedia.org/P38929 and previous config saved to /var/cache/conftool/dbconfig/20221110-115032-ladsgroup.json
  • 11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38928 and previous config saved to /var/cache/conftool/dbconfig/20221110-115007-ladsgroup.json
  • 11:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38927 and previous config saved to /var/cache/conftool/dbconfig/20221110-114634-ladsgroup.json
  • 11:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38926 and previous config saved to /var/cache/conftool/dbconfig/20221110-114422-ladsgroup.json
  • 11:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 11:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 11:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38925 and previous config saved to /var/cache/conftool/dbconfig/20221110-114400-ladsgroup.json
  • 11:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P38924 and previous config saved to /var/cache/conftool/dbconfig/20221110-114300-marostegui.json
  • 11:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T318605)', diff saved to https://phabricator.wikimedia.org/P38923 and previous config saved to /var/cache/conftool/dbconfig/20221110-114149-ladsgroup.json
  • 11:41 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wdqs-all
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38922 and previous config saved to /var/cache/conftool/dbconfig/20221110-113958-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38921 and previous config saved to /var/cache/conftool/dbconfig/20221110-113948-ladsgroup.json
  • 11:38 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38920 and previous config saved to /var/cache/conftool/dbconfig/20221110-113853-root.json
  • 11:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 25%: Index rebuilt', diff saved to https://phabricator.wikimedia.org/P38919 and previous config saved to /var/cache/conftool/dbconfig/20221110-113526-ladsgroup.json
  • 11:31 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wdqs-all
  • 11:31 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wcqs-public
  • 11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P38918 and previous config saved to /var/cache/conftool/dbconfig/20221110-112854-ladsgroup.json
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38917 and previous config saved to /var/cache/conftool/dbconfig/20221110-112753-marostegui.json
  • 11:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38916 and previous config saved to /var/cache/conftool/dbconfig/20221110-112441-ladsgroup.json
  • 11:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38915 and previous config saved to /var/cache/conftool/dbconfig/20221110-112403-marostegui.json
  • 11:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 11:23 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38914 and previous config saved to /var/cache/conftool/dbconfig/20221110-112348-root.json
  • 11:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 11:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38913 and previous config saved to /var/cache/conftool/dbconfig/20221110-112342-marostegui.json
  • 11:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 10%: Index rebuilt', diff saved to https://phabricator.wikimedia.org/P38912 and previous config saved to /var/cache/conftool/dbconfig/20221110-112022-ladsgroup.json
  • 11:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P38911 and previous config saved to /var/cache/conftool/dbconfig/20221110-111347-ladsgroup.json
  • 11:13 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wcqs-public
  • 11:13 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38910 and previous config saved to /var/cache/conftool/dbconfig/20221110-111323-marostegui.json
  • 11:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 11:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 11:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1106.eqiad.wmnet with reason: Maintenance
  • 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38909 and previous config saved to /var/cache/conftool/dbconfig/20221110-111244-marostegui.json
  • 11:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38908 and previous config saved to /var/cache/conftool/dbconfig/20221110-110935-ladsgroup.json
  • 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38907 and previous config saved to /var/cache/conftool/dbconfig/20221110-110843-root.json
  • 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P38906 and previous config saved to /var/cache/conftool/dbconfig/20221110-110835-marostegui.json
  • 11:00 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 10:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38905 and previous config saved to /var/cache/conftool/dbconfig/20221110-105841-ladsgroup.json
  • 10:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P38904 and previous config saved to /var/cache/conftool/dbconfig/20221110-105738-marostegui.json
  • 10:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38903 and previous config saved to /var/cache/conftool/dbconfig/20221110-105628-ladsgroup.json
  • 10:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 10:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 10:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38902 and previous config saved to /var/cache/conftool/dbconfig/20221110-105428-ladsgroup.json
  • 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38901 and previous config saved to /var/cache/conftool/dbconfig/20221110-105338-root.json
  • 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P38900 and previous config saved to /var/cache/conftool/dbconfig/20221110-105329-marostegui.json
  • 10:53 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 10:50 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38899 and previous config saved to /var/cache/conftool/dbconfig/20221110-104919-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 10:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 10:47 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 10:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P38898 and previous config saved to /var/cache/conftool/dbconfig/20221110-104231-marostegui.json
  • 10:38 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38897 and previous config saved to /var/cache/conftool/dbconfig/20221110-103827-root.json
  • 10:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38896 and previous config saved to /var/cache/conftool/dbconfig/20221110-103822-marostegui.json
  • 10:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38895 and previous config saved to /var/cache/conftool/dbconfig/20221110-103533-marostegui.json
  • 10:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 10:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T321130)', diff saved to https://phabricator.wikimedia.org/P38894 and previous config saved to /var/cache/conftool/dbconfig/20221110-103512-marostegui.json
  • 10:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38893 and previous config saved to /var/cache/conftool/dbconfig/20221110-102725-marostegui.json
  • 10:26 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38892 and previous config saved to /var/cache/conftool/dbconfig/20221110-102617-marostegui.json
  • 10:26 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 10:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 10:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38891 and previous config saved to /var/cache/conftool/dbconfig/20221110-102556-marostegui.json
  • 10:25 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1013.eqiad.wmnet to cluster eqiad and group B
  • 10:24 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1013.eqiad.wmnet to cluster eqiad and group B
  • 10:23 moritzm: installing libxml2 security updates
  • 10:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1013.eqiad.wmnet
  • 10:23 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 10:22 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 10:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P38890 and previous config saved to /var/cache/conftool/dbconfig/20221110-102005-marostegui.json
  • 10:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1013.eqiad.wmnet
  • 10:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P38889 and previous config saved to /var/cache/conftool/dbconfig/20221110-101050-marostegui.json
  • 10:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P38888 and previous config saved to /var/cache/conftool/dbconfig/20221110-100459-marostegui.json
  • 09:57 marostegui@cumin1001: dbctl commit (dc=all): 'Reduce es4 master weight', diff saved to https://phabricator.wikimedia.org/P38887 and previous config saved to /var/cache/conftool/dbconfig/20221110-095724-marostegui.json
  • 09:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P38886 and previous config saved to /var/cache/conftool/dbconfig/20221110-095543-marostegui.json
  • 09:53 marostegui@cumin1001: dbctl commit (dc=all): 'es1023 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38885 and previous config saved to /var/cache/conftool/dbconfig/20221110-095313-root.json
  • 09:52 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 09:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T321130)', diff saved to https://phabricator.wikimedia.org/P38884 and previous config saved to /var/cache/conftool/dbconfig/20221110-094952-marostegui.json
  • 09:49 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 09:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T321130)', diff saved to https://phabricator.wikimedia.org/P38883 and previous config saved to /var/cache/conftool/dbconfig/20221110-094604-marostegui.json
  • 09:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1110.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1110.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100 (T321130)', diff saved to https://phabricator.wikimedia.org/P38882 and previous config saved to /var/cache/conftool/dbconfig/20221110-094542-marostegui.json
  • 09:44 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 09:43 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 09:43 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 09:42 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 09:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38881 and previous config saved to /var/cache/conftool/dbconfig/20221110-094037-marostegui.json
  • 09:39 marostegui@deploy1002: Finished scap: Backport for Revert "db-production.php: Disable es5 writes" (duration: 04m 20s)
  • 09:39 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 09:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 (T321123)', diff saved to https://phabricator.wikimedia.org/P38880 and previous config saved to /var/cache/conftool/dbconfig/20221110-093929-marostegui.json
  • 09:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 09:36 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-eqiad
  • 09:35 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "db-production.php: Disable es5 writes" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 09:35 marostegui@deploy1002: Started scap: Backport for Revert "db-production.php: Disable es5 writes"
  • 09:34 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
  • 09:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es1023 T322187', diff saved to https://phabricator.wikimedia.org/P38879 and previous config saved to /var/cache/conftool/dbconfig/20221110-093354-root.json
  • 09:32 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es1024 to es5 primary T322187', diff saved to https://phabricator.wikimedia.org/P38878 and previous config saved to /var/cache/conftool/dbconfig/20221110-093243-root.json
  • 09:32 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 09:32 marostegui: Starting es5 eqiad failover from es1023 to es1024 T322187
  • 09:31 marostegui@deploy1002: Finished scap: Backport for db-production.php: Disable es5 writes (T322187) (duration: 04m 39s)
  • 09:31 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 09:31 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 09:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P38877 and previous config saved to /var/cache/conftool/dbconfig/20221110-093036-marostegui.json
  • 09:30 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 09:27 marostegui@deploy1002: marostegui and marostegui: Backport for db-production.php: Disable es5 writes (T322187) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 09:27 marostegui@deploy1002: Started scap: Backport for db-production.php: Disable es5 writes (T322187)
  • 09:23 marostegui@cumin1001: dbctl commit (dc=all): 'Set es1024 with weight 0 T322187', diff saved to https://phabricator.wikimedia.org/P38876 and previous config saved to /var/cache/conftool/dbconfig/20221110-092336-root.json
  • 09:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322187
  • 09:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322187
  • 09:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38875 and previous config saved to /var/cache/conftool/dbconfig/20221110-092215-marostegui.json
  • 09:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1013.eqiad.wmnet with OS bullseye
  • 09:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P38874 and previous config saved to /var/cache/conftool/dbconfig/20221110-091530-marostegui.json
  • 09:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38873 and previous config saved to /var/cache/conftool/dbconfig/20221110-090708-marostegui.json
  • 09:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1013.eqiad.wmnet with reason: host reimage
  • 09:00 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1013.eqiad.wmnet with reason: host reimage
  • 09:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1100 (T321130)', diff saved to https://phabricator.wikimedia.org/P38872 and previous config saved to /var/cache/conftool/dbconfig/20221110-090023-marostegui.json
  • 08:58 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-codfw
  • 08:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1100 (T321130)', diff saved to https://phabricator.wikimedia.org/P38871 and previous config saved to /var/cache/conftool/dbconfig/20221110-085756-marostegui.json
  • 08:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 08:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1100.eqiad.wmnet with reason: Maintenance
  • 08:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38870 and previous config saved to /var/cache/conftool/dbconfig/20221110-085735-marostegui.json
  • 08:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38869 and previous config saved to /var/cache/conftool/dbconfig/20221110-085201-marostegui.json
  • 08:46 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1013.eqiad.wmnet with OS bullseye
  • 08:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P38868 and previous config saved to /var/cache/conftool/dbconfig/20221110-084229-marostegui.json
  • 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38867 and previous config saved to /var/cache/conftool/dbconfig/20221110-083655-marostegui.json
  • 08:30 hashar@deploy1002: Finished deploy [gerrit/gerrit@84648b3]: Gerrit to 3.4.8 on gerrit1001 # T322724 (duration: 00m 08s)
  • 08:30 hashar@deploy1002: Started deploy [gerrit/gerrit@84648b3]: Gerrit to 3.4.8 on gerrit1001 # T322724
  • 08:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P38866 and previous config saved to /var/cache/conftool/dbconfig/20221110-082722-marostegui.json
  • 08:26 hashar@deploy1002: Finished deploy [gerrit/gerrit@84648b3]: Gerrit to 3.4.8 on gerrit2002 # T322724 (duration: 00m 10s)
  • 08:26 hashar@deploy1002: Started deploy [gerrit/gerrit@84648b3]: Gerrit to 3.4.8 on gerrit2002 # T322724
  • 08:17 moritzm: installing pixman security updates on buster
  • 08:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38865 and previous config saved to /var/cache/conftool/dbconfig/20221110-081216-marostegui.json
  • 08:09 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 136933
  • 08:08 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 136933
  • 08:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P38864 and previous config saved to /var/cache/conftool/dbconfig/20221110-080823-marostegui.json
  • 08:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 08:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 08:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38863 and previous config saved to /var/cache/conftool/dbconfig/20221110-080746-marostegui.json
  • 08:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 08:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 07:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1123.eqiad.wmnet with reason: Maintenance
  • 07:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1122.eqiad.wmnet with reason: Maintenance
  • 07:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1123.eqiad.wmnet with reason: Maintenance
  • 07:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1122.eqiad.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2107.codfw.wmnet with reason: Maintenance
  • 07:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 07:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2107.codfw.wmnet with reason: Maintenance
  • 07:00 ayounsi@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 06:45 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1011.eqiad.wmnet
  • 06:41 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-tool1011.eqiad.wmnet
  • 06:40 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1096.eqiad.wmnet
  • 06:32 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1096.eqiad.wmnet
  • 06:30 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1097.eqiad.wmnet
  • 06:22 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1097.eqiad.wmnet
  • 06:21 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1098.eqiad.wmnet
  • 06:13 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1098.eqiad.wmnet
  • 04:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T318605)', diff saved to https://phabricator.wikimedia.org/P38862 and previous config saved to /var/cache/conftool/dbconfig/20221110-044050-ladsgroup.json
  • 04:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 04:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 04:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T318605)', diff saved to https://phabricator.wikimedia.org/P38861 and previous config saved to /var/cache/conftool/dbconfig/20221110-044028-ladsgroup.json
  • 04:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38860 and previous config saved to /var/cache/conftool/dbconfig/20221110-042522-ladsgroup.json
  • 04:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38859 and previous config saved to /var/cache/conftool/dbconfig/20221110-041016-ladsgroup.json
  • 03:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T318605)', diff saved to https://phabricator.wikimedia.org/P38858 and previous config saved to /var/cache/conftool/dbconfig/20221110-035509-ladsgroup.json
  • 00:03 tzatziki: removing 1 file for legal compliance

2022-11-09

  • 23:57 tzatziki: removing 1 file for legal compliance
  • 23:44 tzatziki: removing 2 files for legal compliance
  • 23:22 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1034.eqiad.wmnet with OS bullseye
  • 23:17 tzatziki: removing 1 file for legal compliance
  • 23:07 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1034.eqiad.wmnet with reason: host reimage
  • 23:04 robh@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1034.eqiad.wmnet with reason: host reimage
  • 23:03 aikochou@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 23:00 aikochou@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 22:51 robh@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1034.eqiad.wmnet with OS bullseye
  • 22:34 damilare: civicrm upgraded from f2017495 to 07fdeed5
  • 22:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T318605)', diff saved to https://phabricator.wikimedia.org/P38857 and previous config saved to /var/cache/conftool/dbconfig/20221109-221551-ladsgroup.json
  • 22:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 22:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 22:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T318605)', diff saved to https://phabricator.wikimedia.org/P38856 and previous config saved to /var/cache/conftool/dbconfig/20221109-221529-ladsgroup.json
  • 22:06 robh@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti1034.eqiad.wmnet with OS bullseye
  • 22:01 robh@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1034.eqiad.wmnet with OS bullseye
  • 22:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38855 and previous config saved to /var/cache/conftool/dbconfig/20221109-220023-ladsgroup.json
  • 21:55 robh@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti1034.eqiad.wmnet with OS bullseye
  • 21:48 robh@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1034.eqiad.wmnet with OS bullseye
  • 21:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38854 and previous config saved to /var/cache/conftool/dbconfig/20221109-214516-ladsgroup.json
  • 21:37 TheresNoTime: closing UTC late backport window
  • 21:35 samtar@deploy1002: Finished scap: Backport for Fix TOC misaligned when max width option is disable (T322162) (duration: 08m 48s)
  • 21:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T318605)', diff saved to https://phabricator.wikimedia.org/P38853 and previous config saved to /var/cache/conftool/dbconfig/20221109-213010-ladsgroup.json
  • 21:30 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:29 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:29 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:26 samtar@deploy1002: samtar and nray: Backport for Fix TOC misaligned when max width option is disable (T322162) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 21:26 samtar@deploy1002: Started scap: Backport for Fix TOC misaligned when max width option is disable (T322162)
  • 21:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2110 (T318605)', diff saved to https://phabricator.wikimedia.org/P38852 and previous config saved to /var/cache/conftool/dbconfig/20221109-211613-ladsgroup.json
  • 21:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 21:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 21:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T318605)', diff saved to https://phabricator.wikimedia.org/P38851 and previous config saved to /var/cache/conftool/dbconfig/20221109-211551-ladsgroup.json
  • 21:15 samtar@deploy1002: Finished scap: Backport for Only Enable LBFactory config callback in CLI in production (T298485) (duration: 05m 41s)
  • 21:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:10 samtar@deploy1002: samtar and dancy: Backport for Only Enable LBFactory config callback in CLI in production (T298485) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 21:09 samtar@deploy1002: Started scap: Backport for Only Enable LBFactory config callback in CLI in production (T298485)
  • 21:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:09 root@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 21:08 samtar@deploy1002: Finished scap: Backport for Add no=>nb to wgInterlanguageLinkCodeMap for some multilingual wikis (T322696) (duration: 06m 06s)
  • 21:02 samtar@deploy1002: samtar and jhsoby: Backport for Add no=>nb to wgInterlanguageLinkCodeMap for some multilingual wikis (T322696) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 21:02 samtar@deploy1002: Started scap: Backport for Add no=>nb to wgInterlanguageLinkCodeMap for some multilingual wikis (T322696)
  • 21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P38850 and previous config saved to /var/cache/conftool/dbconfig/20221109-210044-ladsgroup.json
  • 20:45 root@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1023.eqiad.wmnet with reason: host reimage
  • 20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P38849 and previous config saved to /var/cache/conftool/dbconfig/20221109-204538-ladsgroup.json
  • 20:42 root@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1023.eqiad.wmnet with reason: host reimage
  • 20:41 robh@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:35 mutante: gerrit1001 (gerrit) - restarting gerrit service to disable aggressive garbage collection. gerrit:854514 - T237807
  • 20:30 mutante: gerrit2002 (gerrit-replica) - restarting gerrit service
  • 20:30 root@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 20:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T318605)', diff saved to https://phabricator.wikimedia.org/P38848 and previous config saved to /var/cache/conftool/dbconfig/20221109-203031-ladsgroup.json
  • 20:26 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:26 robh@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1034.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:23 root@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 20:12 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1034.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:05 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:04 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 20:01 root@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 20:01 root@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 20:01 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1034.mgmt.eqiad.wmnet with reboot policy FORCED
  • 19:59 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1034.mgmt.eqiad.wmnet with reboot policy FORCED
  • 19:52 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 19:50 robh@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 19:47 robh@cumin1001: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1034
  • 19:47 robh@cumin1001: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1034
  • 19:47 robh@cumin1001: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1033
  • 19:47 robh@cumin1001: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1033
  • 19:37 root@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:35 root@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:35 root@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1013.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 19:13 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1013.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 19:07 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:05 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:01 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 19:00 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:49 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:49 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:41 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:40 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:35 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:33 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:20 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 18:19 ayounsi@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 17:57 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wcqs1003.eqiad.wmnet with reason: data reload
  • 17:55 bking@cumin2002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on wcqs1003.eqiad.wmnet with reason: data reload
  • 17:45 ayounsi@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye
  • 17:40 volans@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Converted existing STAGED hosts to ACTIVE - volans@cumin1001 - T320696"
  • 17:37 volans@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Converted existing STAGED hosts to ACTIVE - volans@cumin1001 - T320696"
  • 17:29 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 17:28 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 17:17 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 17:16 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 17:09 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 17:09 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:55 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 8218
  • 16:52 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 8218
  • 16:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T322618)', diff saved to https://phabricator.wikimedia.org/P38843 and previous config saved to /var/cache/conftool/dbconfig/20221109-162849-ladsgroup.json
  • 16:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38842 and previous config saved to /var/cache/conftool/dbconfig/20221109-161343-ladsgroup.json
  • 15:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38841 and previous config saved to /var/cache/conftool/dbconfig/20221109-155836-ladsgroup.json
  • 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T318605)', diff saved to https://phabricator.wikimedia.org/P38840 and previous config saved to /var/cache/conftool/dbconfig/20221109-154933-ladsgroup.json
  • 15:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 15:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 15:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38839 and previous config saved to /var/cache/conftool/dbconfig/20221109-154911-ladsgroup.json
  • 15:47 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet
  • 15:47 aikochou@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T322618)', diff saved to https://phabricator.wikimedia.org/P38838 and previous config saved to /var/cache/conftool/dbconfig/20221109-154330-ladsgroup.json
  • 15:42 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet
  • 15:42 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1099.eqiad.wmnet
  • 15:42 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-stretch2002.codfw.wmnet
  • 15:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T322618)', diff saved to https://phabricator.wikimedia.org/P38837 and previous config saved to /var/cache/conftool/dbconfig/20221109-153922-ladsgroup.json
  • 15:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 15:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 15:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T322618)', diff saved to https://phabricator.wikimedia.org/P38836 and previous config saved to /var/cache/conftool/dbconfig/20221109-153901-ladsgroup.json
  • 15:34 aikochou@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38835 and previous config saved to /var/cache/conftool/dbconfig/20221109-153405-ladsgroup.json
  • 15:33 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host kafka-stretch2002.codfw.wmnet
  • 15:32 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1099.eqiad.wmnet
  • 15:31 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1100.eqiad.wmnet
  • 15:24 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1100.eqiad.wmnet
  • 15:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38834 and previous config saved to /var/cache/conftool/dbconfig/20221109-152354-ladsgroup.json
  • 15:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38833 and previous config saved to /var/cache/conftool/dbconfig/20221109-151858-ladsgroup.json
  • 15:18 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-stretch2001.codfw.wmnet
  • 15:12 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1101.eqiad.wmnet
  • 15:11 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host kafka-stretch2001.codfw.wmnet
  • 15:10 stevemunene@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host kafka-stretch2001.codfw.wmnet
  • 15:10 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host kafka-stretch2001.codfw.wmnet
  • 15:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38832 and previous config saved to /var/cache/conftool/dbconfig/20221109-150848-ladsgroup.json
  • 15:04 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet
  • 15:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38831 and previous config saved to /var/cache/conftool/dbconfig/20221109-150351-ladsgroup.json
  • 15:03 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1101.eqiad.wmnet
  • 15:02 stevemunene@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host an-worker1101.eqiad.wmnet
  • 15:02 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1101.eqiad.wmnet
  • 15:02 moritzm: installing pixman security updates on buster
  • 14:57 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet
  • 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T322618)', diff saved to https://phabricator.wikimedia.org/P38830 and previous config saved to /var/cache/conftool/dbconfig/20221109-145341-ladsgroup.json
  • 14:50 moritzm: rolling restart of mw canaries to pick up libxml security update
  • 14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T322618)', diff saved to https://phabricator.wikimedia.org/P38829 and previous config saved to /var/cache/conftool/dbconfig/20221109-144933-ladsgroup.json
  • 14:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 14:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T322618)', diff saved to https://phabricator.wikimedia.org/P38828 and previous config saved to /var/cache/conftool/dbconfig/20221109-144912-ladsgroup.json
  • 14:46 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=frwiki` in a tmux at mwmaint1002 (locally applied shorter MentorStore::INACTIVITY_THRESHOLD; T318457)
  • 14:43 sukhe: reprepro remove bullseye-wikimedia trafficserver: T321309
  • 14:40 moritzm: installing libxml2 security updates
  • 14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38827 and previous config saved to /var/cache/conftool/dbconfig/20221109-143404-ladsgroup.json
  • 14:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2106 (T318605)', diff saved to https://phabricator.wikimedia.org/P38826 and previous config saved to /var/cache/conftool/dbconfig/20221109-143050-ladsgroup.json
  • 14:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 14:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 14:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38825 and previous config saved to /var/cache/conftool/dbconfig/20221109-141858-ladsgroup.json
  • 14:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:12 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:12 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:11 Lucas_WMDE: UTC afternoon backport+config window done
  • 14:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:09 lucaswerkmeister-wmde@deploy1002: Finished scap: Backport for Enable show nearby feature for ruwiki (T321548) (duration: 05m 42s)
  • 14:04 lucaswerkmeister-wmde@deploy1002: lucaswerkmeister-wmde and lilients: Backport for Enable show nearby feature for ruwiki (T321548) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 14:04 lucaswerkmeister-wmde@deploy1002: Started scap: Backport for Enable show nearby feature for ruwiki (T321548)
  • 14:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T322618)', diff saved to https://phabricator.wikimedia.org/P38824 and previous config saved to /var/cache/conftool/dbconfig/20221109-140351-ladsgroup.json
  • 13:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T322618)', diff saved to https://phabricator.wikimedia.org/P38823 and previous config saved to /var/cache/conftool/dbconfig/20221109-135943-ladsgroup.json
  • 13:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 13:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 13:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T322618)', diff saved to https://phabricator.wikimedia.org/P38822 and previous config saved to /var/cache/conftool/dbconfig/20221109-135911-ladsgroup.json
  • 13:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38821 and previous config saved to /var/cache/conftool/dbconfig/20221109-134404-ladsgroup.json
  • 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38820 and previous config saved to /var/cache/conftool/dbconfig/20221109-132858-ladsgroup.json
  • 13:24 moritzm: drain ganeti1013 for eventual reimage to bullseye T311687
  • 13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T322618)', diff saved to https://phabricator.wikimedia.org/P38818 and previous config saved to /var/cache/conftool/dbconfig/20221109-131903-ladsgroup.json
  • 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf1004.eqiad.wmnet
  • 13:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T322618)', diff saved to https://phabricator.wikimedia.org/P38817 and previous config saved to /var/cache/conftool/dbconfig/20221109-131351-ladsgroup.json
  • 13:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf1004.eqiad.wmnet
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T322618)', diff saved to https://phabricator.wikimedia.org/P38816 and previous config saved to /var/cache/conftool/dbconfig/20221109-130944-ladsgroup.json
  • 13:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 13:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38815 and previous config saved to /var/cache/conftool/dbconfig/20221109-130923-ladsgroup.json
  • 13:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38814 and previous config saved to /var/cache/conftool/dbconfig/20221109-130357-ladsgroup.json
  • 12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38813 and previous config saved to /var/cache/conftool/dbconfig/20221109-125416-ladsgroup.json
  • 12:54 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 63199
  • 12:51 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply
  • 12:50 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 63199
  • 12:50 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-main: apply
  • 12:49 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 29169
  • 12:48 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 29169
  • 12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38812 and previous config saved to /var/cache/conftool/dbconfig/20221109-124850-ladsgroup.json
  • 12:43 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply
  • 12:42 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-main: apply
  • 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38811 and previous config saved to /var/cache/conftool/dbconfig/20221109-123910-ladsgroup.json
  • 12:33 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
  • 12:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T322618)', diff saved to https://phabricator.wikimedia.org/P38810 and previous config saved to /var/cache/conftool/dbconfig/20221109-123344-ladsgroup.json
  • 12:33 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
  • 12:31 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:30 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:28 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:28 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:26 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
  • 12:26 hashar@deploy1002: Finished deploy [gerrit/gerrit@b83625a]: gerrit1001: Gerrit JavaScript plugins as standalone files # T319378 (duration: 00m 09s)
  • 12:26 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
  • 12:25 hashar@deploy1002: Started deploy [gerrit/gerrit@b83625a]: gerrit1001: Gerrit JavaScript plugins as standalone files # T319378
  • 12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T322618)', diff saved to https://phabricator.wikimedia.org/P38809 and previous config saved to /var/cache/conftool/dbconfig/20221109-122528-ladsgroup.json
  • 12:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 12:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T322618)', diff saved to https://phabricator.wikimedia.org/P38808 and previous config saved to /var/cache/conftool/dbconfig/20221109-122507-ladsgroup.json
  • 12:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38807 and previous config saved to /var/cache/conftool/dbconfig/20221109-122403-ladsgroup.json
  • 12:18 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply
  • 12:17 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply
  • 12:16 hashar@deploy1002: Finished deploy [gerrit/gerrit@b83625a]: gerrit2002: Gerrit JavaScript plugins as standalone files # T319378 (duration: 00m 10s)
  • 12:16 hashar@deploy1002: Started deploy [gerrit/gerrit@b83625a]: gerrit2002: Gerrit JavaScript plugins as standalone files # T319378
  • 12:11 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-logging-external: apply
  • 12:10 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-logging-external: apply
  • 12:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38806 and previous config saved to /var/cache/conftool/dbconfig/20221109-121001-ladsgroup.json
  • 12:03 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply
  • 12:03 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply
  • 11:56 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply
  • 11:56 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply
  • 11:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38805 and previous config saved to /var/cache/conftool/dbconfig/20221109-115454-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T322618)', diff saved to https://phabricator.wikimedia.org/P38804 and previous config saved to /var/cache/conftool/dbconfig/20221109-113948-ladsgroup.json
  • 11:38 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host datahubsearch1003.eqiad.wmnet
  • 11:36 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 11:34 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host datahubsearch1003.eqiad.wmnet
  • 11:33 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-eqiad
  • 11:32 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd1001.eqiad.wmnet
  • 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T322618)', diff saved to https://phabricator.wikimedia.org/P38803 and previous config saved to /var/cache/conftool/dbconfig/20221109-113144-ladsgroup.json
  • 11:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 11:31 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
  • 11:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 11:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 11:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P38802 and previous config saved to /var/cache/conftool/dbconfig/20221109-113108-ladsgroup.json
  • 11:28 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1001.eqiad.wmnet
  • 11:28 stevemunene@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host dse-k8s-etcd1001.eqiad.wmnet
  • 11:28 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1001.eqiad.wmnet
  • 11:25 stevemunene@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host datahubsearch1002.eqiad.wmnet
  • 11:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38801 and previous config saved to /var/cache/conftool/dbconfig/20221109-112347-ladsgroup.json
  • 11:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38800 and previous config saved to /var/cache/conftool/dbconfig/20221109-112326-ladsgroup.json
  • 11:21 stevemunene@cumin1001: START - Cookbook sre.hosts.reboot-single for host datahubsearch1002.eqiad.wmnet
  • 11:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38799 and previous config saved to /var/cache/conftool/dbconfig/20221109-111601-ladsgroup.json
  • 11:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38798 and previous config saved to /var/cache/conftool/dbconfig/20221109-110819-ladsgroup.json
  • 11:05 jmm@cumin2002: START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-codfw
  • 11:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38797 and previous config saved to /var/cache/conftool/dbconfig/20221109-110055-ladsgroup.json
  • 10:59 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1024.eqiad.wmnet to cluster eqiad and group C
  • 10:58 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1024.eqiad.wmnet to cluster eqiad and group C
  • 10:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38796 and previous config saved to /var/cache/conftool/dbconfig/20221109-105313-ladsgroup.json
  • 10:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P38794 and previous config saved to /var/cache/conftool/dbconfig/20221109-104548-ladsgroup.json
  • 10:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38793 and previous config saved to /var/cache/conftool/dbconfig/20221109-103806-ladsgroup.json
  • 10:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P38792 and previous config saved to /var/cache/conftool/dbconfig/20221109-103722-ladsgroup.json
  • 10:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 10:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 10:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 10:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 10:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T322618)', diff saved to https://phabricator.wikimedia.org/P38791 and previous config saved to /var/cache/conftool/dbconfig/20221109-103026-ladsgroup.json
  • 10:22 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host datahubsearch1001.eqiad.wmnet
  • 10:17 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1018.eqiad.wmnet to cluster eqiad and group B
  • 10:17 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host datahubsearch1001.eqiad.wmnet
  • 10:16 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1018.eqiad.wmnet to cluster eqiad and group B
  • 10:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38788 and previous config saved to /var/cache/conftool/dbconfig/20221109-101519-ladsgroup.json
  • 10:02 volans: set Netbox status to Active for 299 devices with role=server, tenant=none, status=staged - T320696
  • 10:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38787 and previous config saved to /var/cache/conftool/dbconfig/20221109-100013-ladsgroup.json
  • 09:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1024.eqiad.wmnet
  • 09:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1024.eqiad.wmnet
  • 09:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T322618)', diff saved to https://phabricator.wikimedia.org/P38786 and previous config saved to /var/cache/conftool/dbconfig/20221109-094506-ladsgroup.json
  • 09:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1018.eqiad.wmnet
  • 09:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T322618)', diff saved to https://phabricator.wikimedia.org/P38785 and previous config saved to /var/cache/conftool/dbconfig/20221109-093751-ladsgroup.json
  • 09:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 09:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T322618)', diff saved to https://phabricator.wikimedia.org/P38784 and previous config saved to /var/cache/conftool/dbconfig/20221109-093650-ladsgroup.json
  • 09:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T322618)', diff saved to https://phabricator.wikimedia.org/P38783 and previous config saved to /var/cache/conftool/dbconfig/20221109-093629-ladsgroup.json
  • 09:35 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1018.eqiad.wmnet
  • 09:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 09:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 09:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T322618)', diff saved to https://phabricator.wikimedia.org/P38782 and previous config saved to /var/cache/conftool/dbconfig/20221109-093454-ladsgroup.json
  • 09:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38781 and previous config saved to /var/cache/conftool/dbconfig/20221109-092122-ladsgroup.json
  • 09:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38780 and previous config saved to /var/cache/conftool/dbconfig/20221109-091947-ladsgroup.json
  • 09:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1024.eqiad.wmnet with OS bullseye
  • 09:07 moritzm: installing nodejs security updates
  • 09:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38779 and previous config saved to /var/cache/conftool/dbconfig/20221109-090616-ladsgroup.json
  • 09:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38778 and previous config saved to /var/cache/conftool/dbconfig/20221109-090441-ladsgroup.json
  • 09:03 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 54994
  • 08:59 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 54994
  • 08:57 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1024.eqiad.wmnet with reason: host reimage
  • 08:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38777 and previous config saved to /var/cache/conftool/dbconfig/20221109-085542-ladsgroup.json
  • 08:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 08:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 08:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 08:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 08:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 08:54 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1024.eqiad.wmnet with reason: host reimage
  • 08:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 08:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T322618)', diff saved to https://phabricator.wikimedia.org/P38776 and previous config saved to /var/cache/conftool/dbconfig/20221109-085109-ladsgroup.json
  • 08:51 kartik@deploy1002: Finished scap: Backport for Add channel for MessageBundle feature of Translate extension (T322430) (duration: 11m 19s)
  • 08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T322618)', diff saved to https://phabricator.wikimedia.org/P38775 and previous config saved to /var/cache/conftool/dbconfig/20221109-084934-ladsgroup.json
  • 08:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T322618)', diff saved to https://phabricator.wikimedia.org/P38774 and previous config saved to /var/cache/conftool/dbconfig/20221109-084525-ladsgroup.json
  • 08:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 08:45 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 08:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 08:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 08:44 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:44 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:43 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T322618)', diff saved to https://phabricator.wikimedia.org/P38773 and previous config saved to /var/cache/conftool/dbconfig/20221109-084254-ladsgroup.json
  • 08:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 08:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 08:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 08:40 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1024.eqiad.wmnet with OS bullseye
  • 08:40 kartik@deploy1002: kartik and abi: Backport for Add channel for MessageBundle feature of Translate extension (T322430) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
  • 08:39 kartik@deploy1002: Started scap: Backport for Add channel for MessageBundle feature of Translate extension (T322430)
  • 08:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1018.eqiad.wmnet with OS bullseye
  • 08:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:33 kartik@deploy1002: Finished scap: Backport for Update Metrics Platform streams (T322277) (duration: 08m 17s)
  • 08:32 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:32 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:31 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:30 ayounsi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1182.eqiad.wmnet with reason: paged then depooled
  • 08:30 ayounsi@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1182.eqiad.wmnet with reason: paged then depooled
  • 08:25 kartik@deploy1002: kartik and phuedx: Backport for Update Metrics Platform streams (T322277) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 08:24 kartik@deploy1002: Started scap: Backport for Update Metrics Platform streams (T322277)
  • 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1018.eqiad.wmnet with reason: host reimage
  • 08:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:21 kartik@deploy1002: Finished scap: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016) (duration: 08m 10s)
  • 08:20 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1018.eqiad.wmnet with reason: host reimage
  • 08:20 ayounsi@cumin1001: dbctl commit (dc=all): 'Depool db1182', diff saved to https://phabricator.wikimedia.org/P38772 and previous config saved to /var/cache/conftool/dbconfig/20221109-082045-ayounsi.json
  • 08:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:19 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:13 kartik@deploy1002: kartik and phuedx: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 08:12 kartik@deploy1002: Started scap: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016)
  • 08:06 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1018.eqiad.wmnet with OS bullseye
  • 07:24 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 6461
  • 07:22 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 6461
  • 07:22 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 8218
  • 07:22 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 8218
  • 07:21 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37271
  • 07:21 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 37271
  • 07:20 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37662
  • 07:20 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 37662
  • 07:20 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 29608
  • 07:19 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 29608
  • 07:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 8309
  • 07:18 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 8309
  • 07:17 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 54994
  • 07:16 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 54994
  • 07:16 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37693
  • 07:15 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 37693
  • 07:14 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 23889
  • 07:14 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 23889
  • 07:11 ayounsi@cumin1001: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 23889
  • 07:11 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 23889
  • 07:11 ayounsi@cumin1001: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 23889
  • 07:11 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 23889
  • 07:10 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 3225
  • 07:10 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 3225
  • 07:10 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 15412
  • 07:09 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 15412
  • 07:09 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 29169
  • 07:08 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 29169
  • 07:08 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 37613
  • 07:08 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 37613
  • 07:08 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30990
  • 07:07 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 30990
  • 07:06 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 61955
  • 07:06 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 61955
  • 07:05 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 8220
  • 07:04 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 8220
  • 07:04 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 6774
  • 07:02 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 6774
  • 06:42 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 11404
  • 06:41 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 11404

2022-11-08

  • 22:00 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:59 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:59 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:59 urbanecm: UTC late evening B&C window done
  • 21:58 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:58 urbanecm@deploy1002: Finished scap: Backport for Revert "Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis" (duration: 05m 04s)
  • 21:53 urbanecm@deploy1002: urbanecm and urbanecm: Backport for Revert "Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis" synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 21:53 urbanecm@deploy1002: Started scap: Backport for Revert "Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis"
  • 21:43 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:42 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:42 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:41 urbanecm@deploy1002: Finished scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353) (duration: 06m 36s)
  • 21:41 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:35 urbanecm@deploy1002: urbanecm and matmarex: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 21:35 urbanecm@deploy1002: Started scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353)
  • 21:32 urbanecm@deploy1002: Finished scap: Backport for Bump sampling rate to 0.2 for various editing schemas on a/b test wikis (T321734), ThreadItemStore: Fix setting parent IDs when parent already existed (T322599) (duration: 05m 45s)
  • 21:31 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:30 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:30 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:26 urbanecm@deploy1002: urbanecm and kemayo and matmarex: Backport for Bump sampling rate to 0.2 for various editing schemas on a/b test wikis (T321734), ThreadItemStore: Fix setting parent IDs when parent already existed (T322599) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 21:26 urbanecm@deploy1002: Started scap: Backport for Bump sampling rate to 0.2 for various editing schemas on a/b test wikis (T321734), ThreadItemStore: Fix setting parent IDs when parent already existed (T322599)
  • 21:25 urbanecm@deploy1002: backport aborted: (duration: 01m 16s)
  • 21:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:23 urbanecm@deploy1002: Finished scap: Backport for Keep DiscussionTools "Share feedback..." links on WMF wikis for now (T322494) (duration: 04m 14s)
  • 21:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:19 urbanecm@deploy1002: Started scap: Backport for Keep DiscussionTools "Share feedback..." links on WMF wikis for now (T322494)
  • 21:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:17 urbanecm@deploy1002: Finished scap: Backport for ABtest for mobile, logged in users (T320993), ABtest for mobile, logged out users (T320993) (duration: 04m 10s)
  • 21:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:15 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:13 urbanecm@deploy1002: Started scap: Backport for ABtest for mobile, logged in users (T320993), ABtest for mobile, logged out users (T320993)
  • 21:13 urbanecm@deploy1002: backport aborted: (duration: 00m 01s)
  • 21:13 urbanecm@deploy1002: Finished scap: Backport for Enable history page visual diffs on beta cluster (T314588), Update wgSpecialContributeSkinsDisabled → wgSpecialContributeSkinsEnabled (T319327) (duration: 04m 33s)
  • 21:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:08 urbanecm@deploy1002: Started scap: Backport for Enable history page visual diffs on beta cluster (T314588), Update wgSpecialContributeSkinsDisabled → wgSpecialContributeSkinsEnabled (T319327)
  • 20:58 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 20:58 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 20:57 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:57 reedy@deploy1002: Synchronized wmf-config/LabsServices.php: T322667 (duration: 04m 02s)
  • 20:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T321130)', diff saved to https://phabricator.wikimedia.org/P38770 and previous config saved to /var/cache/conftool/dbconfig/20221108-203111-marostegui.json
  • 20:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P38769 and previous config saved to /var/cache/conftool/dbconfig/20221108-201604-marostegui.json
  • 20:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P38768 and previous config saved to /var/cache/conftool/dbconfig/20221108-200058-marostegui.json
  • 19:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T321130)', diff saved to https://phabricator.wikimedia.org/P38767 and previous config saved to /var/cache/conftool/dbconfig/20221108-194551-marostegui.json
  • 19:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 19:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 19:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T322618)', diff saved to https://phabricator.wikimedia.org/P38766 and previous config saved to /var/cache/conftool/dbconfig/20221108-194206-ladsgroup.json
  • 19:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2175 (T321130)', diff saved to https://phabricator.wikimedia.org/P38765 and previous config saved to /var/cache/conftool/dbconfig/20221108-193907-marostegui.json
  • 19:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 19:38 cstone: civicrm upgraded from b95f46bb to f2017495
  • 19:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 19:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38764 and previous config saved to /var/cache/conftool/dbconfig/20221108-193845-marostegui.json
  • 19:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P38763 and previous config saved to /var/cache/conftool/dbconfig/20221108-192659-ladsgroup.json
  • 19:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P38762 and previous config saved to /var/cache/conftool/dbconfig/20221108-192339-marostegui.json
  • 19:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P38761 and previous config saved to /var/cache/conftool/dbconfig/20221108-191152-ladsgroup.json
  • 19:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P38760 and previous config saved to /var/cache/conftool/dbconfig/20221108-190832-marostegui.json
  • 19:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T321123)', diff saved to https://phabricator.wikimedia.org/P38759 and previous config saved to /var/cache/conftool/dbconfig/20221108-190827-marostegui.json
  • 18:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1024.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 18:58 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1024.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 18:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T322618)', diff saved to https://phabricator.wikimedia.org/P38758 and previous config saved to /var/cache/conftool/dbconfig/20221108-185646-ladsgroup.json
  • 18:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1201 (T322618)', diff saved to https://phabricator.wikimedia.org/P38757 and previous config saved to /var/cache/conftool/dbconfig/20221108-185437-ladsgroup.json
  • 18:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 18:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 18:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T322618)', diff saved to https://phabricator.wikimedia.org/P38756 and previous config saved to /var/cache/conftool/dbconfig/20221108-185416-ladsgroup.json
  • 18:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38755 and previous config saved to /var/cache/conftool/dbconfig/20221108-185326-marostegui.json
  • 18:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38754 and previous config saved to /var/cache/conftool/dbconfig/20221108-185320-marostegui.json
  • 18:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38753 and previous config saved to /var/cache/conftool/dbconfig/20221108-184642-marostegui.json
  • 18:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 18:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 18:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T321130)', diff saved to https://phabricator.wikimedia.org/P38752 and previous config saved to /var/cache/conftool/dbconfig/20221108-184620-marostegui.json
  • 18:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P38751 and previous config saved to /var/cache/conftool/dbconfig/20221108-183909-ladsgroup.json
  • 18:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38750 and previous config saved to /var/cache/conftool/dbconfig/20221108-183814-marostegui.json
  • 18:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P38749 and previous config saved to /var/cache/conftool/dbconfig/20221108-183114-marostegui.json
  • 18:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P38748 and previous config saved to /var/cache/conftool/dbconfig/20221108-182403-ladsgroup.json
  • 18:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T321123)', diff saved to https://phabricator.wikimedia.org/P38747 and previous config saved to /var/cache/conftool/dbconfig/20221108-182307-marostegui.json
  • 18:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P38746 and previous config saved to /var/cache/conftool/dbconfig/20221108-181607-marostegui.json
  • 18:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T322618)', diff saved to https://phabricator.wikimedia.org/P38745 and previous config saved to /var/cache/conftool/dbconfig/20221108-180856-ladsgroup.json
  • 18:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T321123)', diff saved to https://phabricator.wikimedia.org/P38744 and previous config saved to /var/cache/conftool/dbconfig/20221108-180808-marostegui.json
  • 18:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 18:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 18:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T321123)', diff saved to https://phabricator.wikimedia.org/P38743 and previous config saved to /var/cache/conftool/dbconfig/20221108-180747-marostegui.json
  • 18:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T322618)', diff saved to https://phabricator.wikimedia.org/P38742 and previous config saved to /var/cache/conftool/dbconfig/20221108-180648-ladsgroup.json
  • 18:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 18:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 18:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38741 and previous config saved to /var/cache/conftool/dbconfig/20221108-180626-ladsgroup.json
  • 17:58 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2148 (T321130)', diff saved to https://phabricator.wikimedia.org/P38739 and previous config saved to /var/cache/conftool/dbconfig/20221108-175425-marostegui.json
  • 17:58 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 17:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38738 and previous config saved to /var/cache/conftool/dbconfig/20221108-175404-marostegui.json
  • 17:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38737 and previous config saved to /var/cache/conftool/dbconfig/20221108-175240-marostegui.json
  • 17:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P38736 and previous config saved to /var/cache/conftool/dbconfig/20221108-175120-ladsgroup.json
  • 17:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1015.eqiad.wmnet
  • 17:40 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1015.eqiad.wmnet
  • 17:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1014.eqiad.wmnet
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P38735 and previous config saved to /var/cache/conftool/dbconfig/20221108-173857-marostegui.json
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38734 and previous config saved to /var/cache/conftool/dbconfig/20221108-173734-marostegui.json
  • 17:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P38733 and previous config saved to /var/cache/conftool/dbconfig/20221108-173613-ladsgroup.json
  • 17:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1014.eqiad.wmnet
  • 17:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-presto1013.eqiad.wmnet
  • 17:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P38732 and previous config saved to /var/cache/conftool/dbconfig/20221108-172351-marostegui.json
  • 17:23 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-presto1013.eqiad.wmnet
  • 17:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T321123)', diff saved to https://phabricator.wikimedia.org/P38731 and previous config saved to /var/cache/conftool/dbconfig/20221108-172227-marostegui.json
  • 17:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38730 and previous config saved to /var/cache/conftool/dbconfig/20221108-172107-ladsgroup.json
  • 17:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38729 and previous config saved to /var/cache/conftool/dbconfig/20221108-171857-ladsgroup.json
  • 17:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 17:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 17:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T322618)', diff saved to https://phabricator.wikimedia.org/P38728 and previous config saved to /var/cache/conftool/dbconfig/20221108-171835-ladsgroup.json
  • 17:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38727 and previous config saved to /var/cache/conftool/dbconfig/20221108-170844-marostegui.json
  • 17:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T321123)', diff saved to https://phabricator.wikimedia.org/P38726 and previous config saved to /var/cache/conftool/dbconfig/20221108-170752-marostegui.json
  • 17:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 17:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 17:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 17:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 17:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38725 and previous config saved to /var/cache/conftool/dbconfig/20221108-170715-marostegui.json
  • 17:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P38724 and previous config saved to /var/cache/conftool/dbconfig/20221108-170329-ladsgroup.json
  • 17:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38723 and previous config saved to /var/cache/conftool/dbconfig/20221108-170157-marostegui.json
  • 17:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 17:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T321130)', diff saved to https://phabricator.wikimedia.org/P38722 and previous config saved to /var/cache/conftool/dbconfig/20221108-170136-marostegui.json
  • 17:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet
  • 16:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet
  • 16:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2002.codfw.wmnet
  • 16:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38721 and previous config saved to /var/cache/conftool/dbconfig/20221108-165208-marostegui.json
  • 16:49 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2002.codfw.wmnet
  • 16:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P38720 and previous config saved to /var/cache/conftool/dbconfig/20221108-164822-ladsgroup.json
  • 16:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2002.codfw.wmnet
  • 16:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P38719 and previous config saved to /var/cache/conftool/dbconfig/20221108-164629-marostegui.json
  • 16:41 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow2002.codfw.wmnet
  • 16:39 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 16:38 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:37 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 16:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38718 and previous config saved to /var/cache/conftool/dbconfig/20221108-163702-marostegui.json
  • 16:36 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 16:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow5002.eqsin.wmnet
  • 16:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T322618)', diff saved to https://phabricator.wikimedia.org/P38717 and previous config saved to /var/cache/conftool/dbconfig/20221108-163316-ladsgroup.json
  • 16:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P38716 and previous config saved to /var/cache/conftool/dbconfig/20221108-163122-marostegui.json
  • 16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T322618)', diff saved to https://phabricator.wikimedia.org/P38715 and previous config saved to /var/cache/conftool/dbconfig/20221108-163107-ladsgroup.json
  • 16:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 16:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 16:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T322618)', diff saved to https://phabricator.wikimedia.org/P38714 and previous config saved to /var/cache/conftool/dbconfig/20221108-163045-ladsgroup.json
  • 16:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow5002.eqsin.wmnet
  • 16:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow6001.drmrs.wmnet
  • 16:24 moritzm: drain ganeti1024 for eventual reimage to bullseye T311687
  • 16:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1018.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 16:22 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1018.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 16:22 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow6001.drmrs.wmnet
  • 16:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38713 and previous config saved to /var/cache/conftool/dbconfig/20221108-162155-marostegui.json
  • 16:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T321130)', diff saved to https://phabricator.wikimedia.org/P38712 and previous config saved to /var/cache/conftool/dbconfig/20221108-161616-marostegui.json
  • 16:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P38711 and previous config saved to /var/cache/conftool/dbconfig/20221108-161538-ladsgroup.json
  • 16:13 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2126 (T321130)', diff saved to https://phabricator.wikimedia.org/P38710 and previous config saved to /var/cache/conftool/dbconfig/20221108-161338-marostegui.json
  • 16:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 16:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T321130)', diff saved to https://phabricator.wikimedia.org/P38709 and previous config saved to /var/cache/conftool/dbconfig/20221108-161312-marostegui.json
  • 16:10 Emperor: upload wmf-beamer-style version 0.4 to apt
  • 16:06 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38708 and previous config saved to /var/cache/conftool/dbconfig/20221108-160632-marostegui.json
  • 16:06 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:06 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P38707 and previous config saved to /var/cache/conftool/dbconfig/20221108-160032-ladsgroup.json
  • 15:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P38706 and previous config saved to /var/cache/conftool/dbconfig/20221108-155805-marostegui.json
  • 15:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 15:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 15:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T321123)', diff saved to https://phabricator.wikimedia.org/P38705 and previous config saved to /var/cache/conftool/dbconfig/20221108-155229-marostegui.json
  • 15:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T322618)', diff saved to https://phabricator.wikimedia.org/P38704 and previous config saved to /var/cache/conftool/dbconfig/20221108-154525-ladsgroup.json
  • 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T322618)', diff saved to https://phabricator.wikimedia.org/P38703 and previous config saved to /var/cache/conftool/dbconfig/20221108-154317-ladsgroup.json
  • 15:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 15:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P38702 and previous config saved to /var/cache/conftool/dbconfig/20221108-154259-marostegui.json
  • 15:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 15:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T322618)', diff saved to https://phabricator.wikimedia.org/P38701 and previous config saved to /var/cache/conftool/dbconfig/20221108-154221-ladsgroup.json
  • 15:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38700 and previous config saved to /var/cache/conftool/dbconfig/20221108-153722-marostegui.json
  • 15:35 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:35 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:34 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:34 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T321130)', diff saved to https://phabricator.wikimedia.org/P38699 and previous config saved to /var/cache/conftool/dbconfig/20221108-152752-marostegui.json
  • 15:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P38698 and previous config saved to /var/cache/conftool/dbconfig/20221108-152715-ladsgroup.json
  • 15:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38697 and previous config saved to /var/cache/conftool/dbconfig/20221108-152548-ladsgroup.json
  • 15:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38696 and previous config saved to /var/cache/conftool/dbconfig/20221108-152216-marostegui.json
  • 15:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T321130)', diff saved to https://phabricator.wikimedia.org/P38695 and previous config saved to /var/cache/conftool/dbconfig/20221108-152037-marostegui.json
  • 15:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 15:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 15:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T321130)', diff saved to https://phabricator.wikimedia.org/P38694 and previous config saved to /var/cache/conftool/dbconfig/20221108-152016-marostegui.json
  • 15:18 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:18 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:17 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:16 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P38693 and previous config saved to /var/cache/conftool/dbconfig/20221108-151208-ladsgroup.json
  • 15:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P38692 and previous config saved to /var/cache/conftool/dbconfig/20221108-151041-ladsgroup.json
  • 15:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T321123)', diff saved to https://phabricator.wikimedia.org/P38691 and previous config saved to /var/cache/conftool/dbconfig/20221108-150709-marostegui.json
  • 15:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P38690 and previous config saved to /var/cache/conftool/dbconfig/20221108-150509-marostegui.json
  • 15:04 daniel@deploy1002: Finished scap: Backport for Stash original wikitext when rendering unsaved content. (T321862) (duration: 07m 25s)
  • 15:03 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:03 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:01 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 15:01 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:01 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:01 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 15:01 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 15:00 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:57 daniel@deploy1002: daniel and daniel: Backport for Stash original wikitext when rendering unsaved content. (T321862) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 14:57 daniel@deploy1002: Started scap: Backport for Stash original wikitext when rendering unsaved content. (T321862)
  • 14:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T322618)', diff saved to https://phabricator.wikimedia.org/P38689 and previous config saved to /var/cache/conftool/dbconfig/20221108-145702-ladsgroup.json
  • 14:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P38688 and previous config saved to /var/cache/conftool/dbconfig/20221108-145535-ladsgroup.json
  • 14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T322618)', diff saved to https://phabricator.wikimedia.org/P38687 and previous config saved to /var/cache/conftool/dbconfig/20221108-145453-ladsgroup.json
  • 14:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 14:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 14:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38686 and previous config saved to /var/cache/conftool/dbconfig/20221108-145432-ladsgroup.json
  • 14:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T321123)', diff saved to https://phabricator.wikimedia.org/P38685 and previous config saved to /var/cache/conftool/dbconfig/20221108-145210-marostegui.json
  • 14:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 14:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 14:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T321123)', diff saved to https://phabricator.wikimedia.org/P38684 and previous config saved to /var/cache/conftool/dbconfig/20221108-145148-marostegui.json
  • 14:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104', diff saved to https://phabricator.wikimedia.org/P38683 and previous config saved to /var/cache/conftool/dbconfig/20221108-145003-marostegui.json
  • 14:43 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 14:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38682 and previous config saved to /var/cache/conftool/dbconfig/20221108-144028-ladsgroup.json
  • 14:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P38681 and previous config saved to /var/cache/conftool/dbconfig/20221108-143925-ladsgroup.json
  • 14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T322618)', diff saved to https://phabricator.wikimedia.org/P38680 and previous config saved to /var/cache/conftool/dbconfig/20221108-143815-ladsgroup.json
  • 14:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 14:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 14:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38679 and previous config saved to /var/cache/conftool/dbconfig/20221108-143743-ladsgroup.json
  • 14:37 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 14:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38678 and previous config saved to /var/cache/conftool/dbconfig/20221108-143642-marostegui.json
  • 14:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2104 (T321130)', diff saved to https://phabricator.wikimedia.org/P38677 and previous config saved to /var/cache/conftool/dbconfig/20221108-143457-marostegui.json
  • 14:33 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 14:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2104 (T321130)', diff saved to https://phabricator.wikimedia.org/P38676 and previous config saved to /var/cache/conftool/dbconfig/20221108-143220-marostegui.json
  • 14:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 14:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2104.codfw.wmnet with reason: Maintenance
  • 14:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 14:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 14:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P38675 and previous config saved to /var/cache/conftool/dbconfig/20221108-142419-ladsgroup.json
  • 14:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 14:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 14:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T321130)', diff saved to https://phabricator.wikimedia.org/P38674 and previous config saved to /var/cache/conftool/dbconfig/20221108-142247-marostegui.json
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P38673 and previous config saved to /var/cache/conftool/dbconfig/20221108-142236-ladsgroup.json
  • 14:22 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 14:22 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 14:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38672 and previous config saved to /var/cache/conftool/dbconfig/20221108-142135-marostegui.json
  • 14:21 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 14:21 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 14:11 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 14:11 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 14:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38671 and previous config saved to /var/cache/conftool/dbconfig/20221108-140912-ladsgroup.json
  • 14:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38670 and previous config saved to /var/cache/conftool/dbconfig/20221108-140803-ladsgroup.json
  • 14:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 14:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 14:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P38669 and previous config saved to /var/cache/conftool/dbconfig/20221108-140741-marostegui.json
  • 14:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P38668 and previous config saved to /var/cache/conftool/dbconfig/20221108-140730-ladsgroup.json
  • 14:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T321123)', diff saved to https://phabricator.wikimedia.org/P38667 and previous config saved to /var/cache/conftool/dbconfig/20221108-140628-marostegui.json
  • 13:56 dcausse@deploy1002: Finished deploy [wikimedia/discovery/analytics@248d897]: import_cirrus_indexes: increase driver mem (duration: 02m 23s)
  • 13:54 dcausse@deploy1002: Started deploy [wikimedia/discovery/analytics@248d897]: import_cirrus_indexes: increase driver mem
  • 13:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P38666 and previous config saved to /var/cache/conftool/dbconfig/20221108-135234-marostegui.json
  • 13:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38665 and previous config saved to /var/cache/conftool/dbconfig/20221108-135224-ladsgroup.json
  • 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T321123)', diff saved to https://phabricator.wikimedia.org/P38664 and previous config saved to /var/cache/conftool/dbconfig/20221108-135129-marostegui.json
  • 13:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 13:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38663 and previous config saved to /var/cache/conftool/dbconfig/20221108-135011-ladsgroup.json
  • 13:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 13:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 13:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38662 and previous config saved to /var/cache/conftool/dbconfig/20221108-134949-ladsgroup.json
  • 13:49 btullis@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 13:49 btullis@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 13:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 13:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 13:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T321123)', diff saved to https://phabricator.wikimedia.org/P38661 and previous config saved to /var/cache/conftool/dbconfig/20221108-134656-marostegui.json
  • 13:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P38660 and previous config saved to /var/cache/conftool/dbconfig/20221108-133730-ladsgroup.json
  • 13:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T321130)', diff saved to https://phabricator.wikimedia.org/P38659 and previous config saved to /var/cache/conftool/dbconfig/20221108-133721-marostegui.json
  • 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1197 (T321130)', diff saved to https://phabricator.wikimedia.org/P38658 and previous config saved to /var/cache/conftool/dbconfig/20221108-133505-marostegui.json
  • 13:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 13:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P38657 and previous config saved to /var/cache/conftool/dbconfig/20221108-133442-ladsgroup.json
  • 13:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 13:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T321130)', diff saved to https://phabricator.wikimedia.org/P38656 and previous config saved to /var/cache/conftool/dbconfig/20221108-133433-marostegui.json
  • 13:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38655 and previous config saved to /var/cache/conftool/dbconfig/20221108-133149-marostegui.json
  • 13:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow3002.esams.wmnet
  • 13:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38654 and previous config saved to /var/cache/conftool/dbconfig/20221108-132223-ladsgroup.json
  • 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38653 and previous config saved to /var/cache/conftool/dbconfig/20221108-132014-ladsgroup.json
  • 13:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 13:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38652 and previous config saved to /var/cache/conftool/dbconfig/20221108-131952-ladsgroup.json
  • 13:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P38651 and previous config saved to /var/cache/conftool/dbconfig/20221108-131936-ladsgroup.json
  • 13:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P38650 and previous config saved to /var/cache/conftool/dbconfig/20221108-131927-marostegui.json
  • 13:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow3002.esams.wmnet
  • 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4002.ulsfo.wmnet
  • 13:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38649 and previous config saved to /var/cache/conftool/dbconfig/20221108-131643-marostegui.json
  • 13:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netflow4002.ulsfo.wmnet
  • 13:07 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 15557
  • 13:05 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 15557
  • 13:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38648 and previous config saved to /var/cache/conftool/dbconfig/20221108-130446-ladsgroup.json
  • 13:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38647 and previous config saved to /var/cache/conftool/dbconfig/20221108-130429-ladsgroup.json
  • 13:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P38646 and previous config saved to /var/cache/conftool/dbconfig/20221108-130420-marostegui.json
  • 13:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38645 and previous config saved to /var/cache/conftool/dbconfig/20221108-130216-ladsgroup.json
  • 13:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 13:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 13:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38644 and previous config saved to /var/cache/conftool/dbconfig/20221108-130205-ladsgroup.json
  • 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T321123)', diff saved to https://phabricator.wikimedia.org/P38643 and previous config saved to /var/cache/conftool/dbconfig/20221108-130136-marostegui.json
  • 12:55 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T321123)', diff saved to https://phabricator.wikimedia.org/P38642 and previous config saved to /var/cache/conftool/dbconfig/20221108-125529-marostegui.json
  • 12:55 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T321123)', diff saved to https://phabricator.wikimedia.org/P38641 and previous config saved to /var/cache/conftool/dbconfig/20221108-125508-marostegui.json
  • 12:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38640 and previous config saved to /var/cache/conftool/dbconfig/20221108-124939-ladsgroup.json
  • 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T321130)', diff saved to https://phabricator.wikimedia.org/P38639 and previous config saved to /var/cache/conftool/dbconfig/20221108-124914-marostegui.json
  • 12:47 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P38638 and previous config saved to /var/cache/conftool/dbconfig/20221108-124659-ladsgroup.json
  • 12:47 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1188 (T321130)', diff saved to https://phabricator.wikimedia.org/P38637 and previous config saved to /var/cache/conftool/dbconfig/20221108-124658-marostegui.json
  • 12:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 12:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 12:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T321130)', diff saved to https://phabricator.wikimedia.org/P38636 and previous config saved to /var/cache/conftool/dbconfig/20221108-124636-marostegui.json
  • 12:42 ladsgroup@deploy1002: Finished scap: Backport for Include core PSR-4 classes in the generated classmap (T274041) (duration: 05m 29s)
  • 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki2002.codfw.wmnet
  • 12:40 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:40 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38635 and previous config saved to /var/cache/conftool/dbconfig/20221108-124001-marostegui.json
  • 12:38 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:38 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 12:37 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for Include core PSR-4 classes in the generated classmap (T274041) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 12:37 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 12:37 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 12:37 ladsgroup@deploy1002: Started scap: Backport for Include core PSR-4 classes in the generated classmap (T274041)
  • 12:36 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 12:36 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host pki2002.codfw.wmnet
  • 12:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38634 and previous config saved to /var/cache/conftool/dbconfig/20221108-123433-ladsgroup.json
  • 12:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P38633 and previous config saved to /var/cache/conftool/dbconfig/20221108-123152-ladsgroup.json
  • 12:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P38632 and previous config saved to /var/cache/conftool/dbconfig/20221108-123130-marostegui.json
  • 12:29 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp1002.wikimedia.org
  • 12:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P38631 and previous config saved to /var/cache/conftool/dbconfig/20221108-122923-ladsgroup.json
  • 12:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 12:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 12:28 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:27 sukhe: reprepro -C main include bullseye-wikimedia libvmod-re2_1.5.3-3_amd64.changes: T321309
  • 12:25 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host idp1002.wikimedia.org
  • 12:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38630 and previous config saved to /var/cache/conftool/dbconfig/20221108-122455-marostegui.json
  • 12:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38629 and previous config saved to /var/cache/conftool/dbconfig/20221108-121646-ladsgroup.json
  • 12:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P38628 and previous config saved to /var/cache/conftool/dbconfig/20221108-121623-marostegui.json
  • 12:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T322618)', diff saved to https://phabricator.wikimedia.org/P38627 and previous config saved to /var/cache/conftool/dbconfig/20221108-121433-ladsgroup.json
  • 12:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 12:14 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 12:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 12:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 12:14 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 12:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 12:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T322618)', diff saved to https://phabricator.wikimedia.org/P38626 and previous config saved to /var/cache/conftool/dbconfig/20221108-121347-ladsgroup.json
  • 12:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T321123)', diff saved to https://phabricator.wikimedia.org/P38625 and previous config saved to /var/cache/conftool/dbconfig/20221108-120949-marostegui.json
  • 12:09 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:08 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:02 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T321123)', diff saved to https://phabricator.wikimedia.org/P38624 and previous config saved to /var/cache/conftool/dbconfig/20221108-120143-marostegui.json
  • 12:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 12:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38623 and previous config saved to /var/cache/conftool/dbconfig/20221108-120122-marostegui.json
  • 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T321130)', diff saved to https://phabricator.wikimedia.org/P38622 and previous config saved to /var/cache/conftool/dbconfig/20221108-120117-marostegui.json
  • 11:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P38621 and previous config saved to /var/cache/conftool/dbconfig/20221108-115840-ladsgroup.json
  • 11:54 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T321130)', diff saved to https://phabricator.wikimedia.org/P38620 and previous config saved to /var/cache/conftool/dbconfig/20221108-115452-marostegui.json
  • 11:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 11:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 11:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38619 and previous config saved to /var/cache/conftool/dbconfig/20221108-115430-marostegui.json
  • 11:52 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 11:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38618 and previous config saved to /var/cache/conftool/dbconfig/20221108-114615-marostegui.json
  • 11:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P38617 and previous config saved to /var/cache/conftool/dbconfig/20221108-114333-ladsgroup.json
  • 11:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P38616 and previous config saved to /var/cache/conftool/dbconfig/20221108-113924-marostegui.json
  • 11:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38615 and previous config saved to /var/cache/conftool/dbconfig/20221108-113109-marostegui.json
  • 11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T322618)', diff saved to https://phabricator.wikimedia.org/P38614 and previous config saved to /var/cache/conftool/dbconfig/20221108-112825-ladsgroup.json
  • 11:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2129 (T322618)', diff saved to https://phabricator.wikimedia.org/P38613 and previous config saved to /var/cache/conftool/dbconfig/20221108-112612-ladsgroup.json
  • 11:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 11:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 11:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T322618)', diff saved to https://phabricator.wikimedia.org/P38612 and previous config saved to /var/cache/conftool/dbconfig/20221108-112551-ladsgroup.json
  • 11:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P38611 and previous config saved to /var/cache/conftool/dbconfig/20221108-112417-marostegui.json
  • 11:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38610 and previous config saved to /var/cache/conftool/dbconfig/20221108-111602-marostegui.json
  • 11:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P38609 and previous config saved to /var/cache/conftool/dbconfig/20221108-111044-ladsgroup.json
  • 11:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38608 and previous config saved to /var/cache/conftool/dbconfig/20221108-110956-marostegui.json
  • 11:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 11:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 11:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T321123)', diff saved to https://phabricator.wikimedia.org/P38607 and previous config saved to /var/cache/conftool/dbconfig/20221108-110934-marostegui.json
  • 11:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38606 and previous config saved to /var/cache/conftool/dbconfig/20221108-110911-marostegui.json
  • 11:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38605 and previous config saved to /var/cache/conftool/dbconfig/20221108-110243-marostegui.json
  • 11:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 11:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 11:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T321130)', diff saved to https://phabricator.wikimedia.org/P38604 and previous config saved to /var/cache/conftool/dbconfig/20221108-110232-marostegui.json
  • 11:00 moritzm: drain ganeti1024 for eventual reimage to bullseye T311687
  • 11:00 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 10:58 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 10:57 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 10:55 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 10:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P38603 and previous config saved to /var/cache/conftool/dbconfig/20221108-105538-ladsgroup.json
  • 10:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38602 and previous config saved to /var/cache/conftool/dbconfig/20221108-105428-marostegui.json
  • 10:52 btullis: added stevemunene to wmf and ops LDAP groups T322339
  • 10:51 moritzm: installing batik security updates
  • 10:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P38601 and previous config saved to /var/cache/conftool/dbconfig/20221108-104726-marostegui.json
  • 10:42 moritzm: installing ntfs-3g security updates
  • 10:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T322618)', diff saved to https://phabricator.wikimedia.org/P38600 and previous config saved to /var/cache/conftool/dbconfig/20221108-104031-ladsgroup.json
  • 10:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38599 and previous config saved to /var/cache/conftool/dbconfig/20221108-103921-marostegui.json
  • 10:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T322618)', diff saved to https://phabricator.wikimedia.org/P38598 and previous config saved to /var/cache/conftool/dbconfig/20221108-103817-ladsgroup.json
  • 10:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 10:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 10:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P38597 and previous config saved to /var/cache/conftool/dbconfig/20221108-103756-ladsgroup.json
  • 10:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P38596 and previous config saved to /var/cache/conftool/dbconfig/20221108-103219-marostegui.json
  • 10:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T321123)', diff saved to https://phabricator.wikimedia.org/P38595 and previous config saved to /var/cache/conftool/dbconfig/20221108-102415-marostegui.json
  • 10:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P38594 and previous config saved to /var/cache/conftool/dbconfig/20221108-102249-ladsgroup.json
  • 10:18 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T321123)', diff saved to https://phabricator.wikimedia.org/P38593 and previous config saved to /var/cache/conftool/dbconfig/20221108-101806-marostegui.json
  • 10:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 10:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 10:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38592 and previous config saved to /var/cache/conftool/dbconfig/20221108-101745-marostegui.json
  • 10:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T321130)', diff saved to https://phabricator.wikimedia.org/P38591 and previous config saved to /var/cache/conftool/dbconfig/20221108-101713-marostegui.json
  • 10:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1162 (T321130)', diff saved to https://phabricator.wikimedia.org/P38590 and previous config saved to /var/cache/conftool/dbconfig/20221108-101457-marostegui.json
  • 10:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 10:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 10:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T321130)', diff saved to https://phabricator.wikimedia.org/P38589 and previous config saved to /var/cache/conftool/dbconfig/20221108-101435-marostegui.json
  • 10:10 moritzm: installing glibc security updates on buster
  • 10:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P38588 and previous config saved to /var/cache/conftool/dbconfig/20221108-100743-ladsgroup.json
  • 10:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38587 and previous config saved to /var/cache/conftool/dbconfig/20221108-100239-marostegui.json
  • 09:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P38586 and previous config saved to /var/cache/conftool/dbconfig/20221108-095928-marostegui.json
  • 09:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P38585 and previous config saved to /var/cache/conftool/dbconfig/20221108-095236-ladsgroup.json
  • 09:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P38584 and previous config saved to /var/cache/conftool/dbconfig/20221108-095026-ladsgroup.json
  • 09:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 09:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 09:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Partially done', diff saved to https://phabricator.wikimedia.org/P38583 and previous config saved to /var/cache/conftool/dbconfig/20221108-094950-ladsgroup.json
  • 09:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38582 and previous config saved to /var/cache/conftool/dbconfig/20221108-094732-marostegui.json
  • 09:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P38581 and previous config saved to /var/cache/conftool/dbconfig/20221108-094655-ladsgroup.json
  • 09:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 09:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 09:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P38580 and previous config saved to /var/cache/conftool/dbconfig/20221108-094422-marostegui.json
  • 09:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38579 and previous config saved to /var/cache/conftool/dbconfig/20221108-093226-marostegui.json
  • 09:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T321130)', diff saved to https://phabricator.wikimedia.org/P38578 and previous config saved to /var/cache/conftool/dbconfig/20221108-092915-marostegui.json
  • 09:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T321130)', diff saved to https://phabricator.wikimedia.org/P38577 and previous config saved to /var/cache/conftool/dbconfig/20221108-092256-marostegui.json
  • 09:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 09:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 09:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38576 and previous config saved to /var/cache/conftool/dbconfig/20221108-092229-marostegui.json
  • 09:22 moritzm: installing ffmpeg security updates on buster
  • 09:17 moritzm: drain ganeti1018 for eventual reimage to bullseye T311687
  • 09:09 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1004.eqiad.wmnet to plain
  • 09:08 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1004.eqiad.wmnet to plain
  • 09:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P38575 and previous config saved to /var/cache/conftool/dbconfig/20221108-090722-marostegui.json
  • 08:54 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1004.eqiad.wmnet to drbd
  • 08:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P38574 and previous config saved to /var/cache/conftool/dbconfig/20221108-085216-marostegui.json
  • 08:48 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:47 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:47 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:44 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1004.eqiad.wmnet to drbd
  • 08:44 kartik@deploy1002: Finished scap: Backport for Revert "EditAttemptStep sampling rate to 1 for group1 wikis" (duration: 04m 26s)
  • 08:40 kartik@deploy1002: kartik and trainbranchbot: Backport for Revert "EditAttemptStep sampling rate to 1 for group1 wikis" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 08:39 kartik@deploy1002: Started scap: Backport for Revert "EditAttemptStep sampling rate to 1 for group1 wikis"
  • 08:37 kartik@deploy1002: Sync cancelled.
  • 08:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38573 and previous config saved to /var/cache/conftool/dbconfig/20221108-083709-marostegui.json
  • 08:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:35 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:35 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:34 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1006.eqiad.wmnet to plain
  • 08:33 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1006.eqiad.wmnet to plain
  • 08:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T321123)', diff saved to https://phabricator.wikimedia.org/P38572 and previous config saved to /var/cache/conftool/dbconfig/20221108-083210-marostegui.json
  • 08:32 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 08:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 08:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321123)', diff saved to https://phabricator.wikimedia.org/P38571 and previous config saved to /var/cache/conftool/dbconfig/20221108-083148-marostegui.json
  • 08:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38570 and previous config saved to /var/cache/conftool/dbconfig/20221108-083037-marostegui.json
  • 08:30 kartik@deploy1002: kartik and phuedx: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 08:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 08:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 08:30 kartik@deploy1002: Started scap: Backport for EditAttemptStep sampling rate to 1 for group1 wikis (T312016)
  • 08:26 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 08:25 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 08:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38569 and previous config saved to /var/cache/conftool/dbconfig/20221108-082546-marostegui.json
  • 08:24 kartik@deploy1002: Finished scap: Backport for Enable Content and Section translation on 6 Wikipedias (T319175) (duration: 08m 20s)
  • 08:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1006.eqiad.wmnet to drbd
  • 08:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:16 kartik@deploy1002: kartik and kartik: Backport for Enable Content and Section translation on 6 Wikipedias (T319175) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 08:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38568 and previous config saved to /var/cache/conftool/dbconfig/20221108-081641-marostegui.json
  • 08:16 kartik@deploy1002: Started scap: Backport for Enable Content and Section translation on 6 Wikipedias (T319175)
  • 08:14 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1006.eqiad.wmnet to drbd
  • 08:12 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P38567 and previous config saved to /var/cache/conftool/dbconfig/20221108-081040-marostegui.json
  • 08:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:09 kartik@deploy1002: Finished scap: Backport for Enable Content and Section translation in Bambara and Goan Konkani Wikipedias (T314557) (duration: 06m 44s)
  • 08:03 kartik@deploy1002: kartik and kartik: Backport for Enable Content and Section translation in Bambara and Goan Konkani Wikipedias (T314557) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 08:03 kartik@deploy1002: Started scap: Backport for Enable Content and Section translation in Bambara and Goan Konkani Wikipedias (T314557)
  • 08:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38566 and previous config saved to /var/cache/conftool/dbconfig/20221108-080135-marostegui.json
  • 07:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P38565 and previous config saved to /var/cache/conftool/dbconfig/20221108-075533-marostegui.json
  • 07:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321123)', diff saved to https://phabricator.wikimedia.org/P38564 and previous config saved to /var/cache/conftool/dbconfig/20221108-074628-marostegui.json
  • 07:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38563 and previous config saved to /var/cache/conftool/dbconfig/20221108-074027-marostegui.json
  • 07:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T321123)', diff saved to https://phabricator.wikimedia.org/P38562 and previous config saved to /var/cache/conftool/dbconfig/20221108-074022-marostegui.json
  • 07:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 07:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 07:37 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38561 and previous config saved to /var/cache/conftool/dbconfig/20221108-073711-marostegui.json
  • 07:37 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 07:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 07:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38560 and previous config saved to /var/cache/conftool/dbconfig/20221108-073649-marostegui.json
  • 07:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 07:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 07:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321123)', diff saved to https://phabricator.wikimedia.org/P38559 and previous config saved to /var/cache/conftool/dbconfig/20221108-073549-marostegui.json
  • 07:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T318605)', diff saved to https://phabricator.wikimedia.org/P38558 and previous config saved to /var/cache/conftool/dbconfig/20221108-072648-ladsgroup.json
  • 07:22 XioNoX: push pfw policies - T322613
  • 07:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P38557 and previous config saved to /var/cache/conftool/dbconfig/20221108-072143-marostegui.json
  • 07:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38556 and previous config saved to /var/cache/conftool/dbconfig/20221108-072042-marostegui.json
  • 07:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P38555 and previous config saved to /var/cache/conftool/dbconfig/20221108-071142-ladsgroup.json
  • 07:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P38554 and previous config saved to /var/cache/conftool/dbconfig/20221108-070636-marostegui.json
  • 07:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38553 and previous config saved to /var/cache/conftool/dbconfig/20221108-070536-marostegui.json
  • 06:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P38552 and previous config saved to /var/cache/conftool/dbconfig/20221108-065635-ladsgroup.json
  • 06:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38551 and previous config saved to /var/cache/conftool/dbconfig/20221108-065130-marostegui.json
  • 06:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321123)', diff saved to https://phabricator.wikimedia.org/P38550 and previous config saved to /var/cache/conftool/dbconfig/20221108-065029-marostegui.json
  • 06:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T321130)', diff saved to https://phabricator.wikimedia.org/P38549 and previous config saved to /var/cache/conftool/dbconfig/20221108-064447-marostegui.json
  • 06:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 06:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 06:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T321123)', diff saved to https://phabricator.wikimedia.org/P38548 and previous config saved to /var/cache/conftool/dbconfig/20221108-064422-marostegui.json
  • 06:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 06:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 06:43 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 06:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 06:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T318605)', diff saved to https://phabricator.wikimedia.org/P38547 and previous config saved to /var/cache/conftool/dbconfig/20221108-064129-ladsgroup.json
  • 06:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 06:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 06:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 06:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 06:38 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 63199
  • 06:36 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 63199
  • 06:35 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 22381
  • 06:35 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 22381
  • 06:34 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 4817
  • 06:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 06:34 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 06:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 06:33 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 4817
  • 06:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1138.eqiad.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1138.eqiad.wmnet with reason: Maintenance
  • 06:32 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 13150
  • 06:31 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 13150
  • 06:30 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 30058
  • 06:29 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 30058
  • 06:27 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 46416
  • 06:27 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 06:27 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 46416
  • 05:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2182 (T318605)', diff saved to https://phabricator.wikimedia.org/P38546 and previous config saved to /var/cache/conftool/dbconfig/20221108-052025-ladsgroup.json
  • 05:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 05:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 05:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38545 and previous config saved to /var/cache/conftool/dbconfig/20221108-052004-ladsgroup.json
  • 05:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P38544 and previous config saved to /var/cache/conftool/dbconfig/20221108-050457-ladsgroup.json
  • 04:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P38543 and previous config saved to /var/cache/conftool/dbconfig/20221108-044951-ladsgroup.json
  • 04:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38542 and previous config saved to /var/cache/conftool/dbconfig/20221108-043444-ladsgroup.json
  • 03:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 03:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 03:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T318605)', diff saved to https://phabricator.wikimedia.org/P38541 and previous config saved to /var/cache/conftool/dbconfig/20221108-034607-ladsgroup.json
  • 03:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P38540 and previous config saved to /var/cache/conftool/dbconfig/20221108-033101-ladsgroup.json
  • 03:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20221108-031550-ladsgroup.json
  • 03:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38539 and previous config saved to /var/cache/conftool/dbconfig/20221108-031102-ladsgroup.json
  • 03:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 03:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 03:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38538 and previous config saved to /var/cache/conftool/dbconfig/20221108-031041-ladsgroup.json
  • 03:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 03:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 03:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 03:03 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 03:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T318605)', diff saved to https://phabricator.wikimedia.org/P38537 and previous config saved to /var/cache/conftool/dbconfig/20221108-030043-ladsgroup.json
  • 02:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P38536 and previous config saved to /var/cache/conftool/dbconfig/20221108-025533-ladsgroup.json
  • 02:47 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host contint1002.mgmt.eqiad.wmnet with reboot policy FORCED
  • 02:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1202 (T318605)', diff saved to https://phabricator.wikimedia.org/P38535 and previous config saved to /var/cache/conftool/dbconfig/20221108-024600-ladsgroup.json
  • 02:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 02:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 02:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T318605)', diff saved to https://phabricator.wikimedia.org/P38534 and previous config saved to /var/cache/conftool/dbconfig/20221108-024539-ladsgroup.json
  • 02:44 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host contint1002.mgmt.eqiad.wmnet with reboot policy FORCED
  • 02:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P38533 and previous config saved to /var/cache/conftool/dbconfig/20221108-024027-ladsgroup.json
  • 02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P38532 and previous config saved to /var/cache/conftool/dbconfig/20221108-023032-ladsgroup.json
  • 02:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38531 and previous config saved to /var/cache/conftool/dbconfig/20221108-022520-ladsgroup.json
  • 02:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P38530 and previous config saved to /var/cache/conftool/dbconfig/20221108-021525-ladsgroup.json
  • 02:14 eileen: civicrm upgraded from 72fccce1 to b95f46bb
  • 02:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T318605)', diff saved to https://phabricator.wikimedia.org/P38529 and previous config saved to /var/cache/conftool/dbconfig/20221108-020019-ladsgroup.json
  • 01:47 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 01:47 tgr@deploy1002: Finished scap: Backport for Add UserRegistrationLookupHelper (duration: 04m 36s)
  • 01:46 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 01:46 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 01:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 01:43 tgr@deploy1002: tgr and tgr: Backport for Add UserRegistrationLookupHelper synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
  • 01:42 tgr@deploy1002: Started scap: Backport for Add UserRegistrationLookupHelper
  • 01:39 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 01:38 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 01:37 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
  • 01:25 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 01:22 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
  • 01:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2168:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38528 and previous config saved to /var/cache/conftool/dbconfig/20221108-010245-ladsgroup.json
  • 01:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 01:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 01:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T318605)', diff saved to https://phabricator.wikimedia.org/P38527 and previous config saved to /var/cache/conftool/dbconfig/20221108-010224-ladsgroup.json
  • 00:55 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1194 (T318605)', diff saved to https://phabricator.wikimedia.org/P38526 and previous config saved to /var/cache/conftool/dbconfig/20221108-005338-ladsgroup.json
  • 00:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 00:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 00:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T318605)', diff saved to https://phabricator.wikimedia.org/P38525 and previous config saved to /var/cache/conftool/dbconfig/20221108-005317-ladsgroup.json
  • 00:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P38524 and previous config saved to /var/cache/conftool/dbconfig/20221108-004717-ladsgroup.json
  • 00:47 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED
  • 00:40 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 00:39 tgr@deploy1002: Finished scap: Backport for createExtensionTables.php: Remove closeConnection() (duration: 04m 43s)
  • 00:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38523 and previous config saved to /var/cache/conftool/dbconfig/20221108-003934-marostegui.json
  • 00:39 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 00:39 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 00:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 00:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P38522 and previous config saved to /var/cache/conftool/dbconfig/20221108-003810-ladsgroup.json
  • 00:36 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 00:35 tgr@deploy1002: tgr and tgr: Backport for createExtensionTables.php: Remove closeConnection() synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 00:35 tgr@deploy1002: Started scap: Backport for createExtensionTables.php: Remove closeConnection()
  • 00:34 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
  • 00:33 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 00:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P38521 and previous config saved to /var/cache/conftool/dbconfig/20221108-003210-ladsgroup.json
  • 00:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P38520 and previous config saved to /var/cache/conftool/dbconfig/20221108-002428-marostegui.json
  • 00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P38519 and previous config saved to /var/cache/conftool/dbconfig/20221108-002304-ladsgroup.json
  • 00:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T318605)', diff saved to https://phabricator.wikimedia.org/P38518 and previous config saved to /var/cache/conftool/dbconfig/20221108-001704-ladsgroup.json
  • 00:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P38517 and previous config saved to /var/cache/conftool/dbconfig/20221108-000922-marostegui.json
  • 00:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T318605)', diff saved to https://phabricator.wikimedia.org/P38516 and previous config saved to /var/cache/conftool/dbconfig/20221108-000757-ladsgroup.json
  • 00:02 tgr: running foreachwikiindblist growthexperiments.dblist extensions/WikimediaMaintenance/createExtensionTables.php growthexperiments

2022-11-07

  • 23:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1191 (T318605)', diff saved to https://phabricator.wikimedia.org/P38515 and previous config saved to /var/cache/conftool/dbconfig/20221107-235526-ladsgroup.json
  • 23:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 23:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 23:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T318605)', diff saved to https://phabricator.wikimedia.org/P38514 and previous config saved to /var/cache/conftool/dbconfig/20221107-235505-ladsgroup.json
  • 23:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38513 and previous config saved to /var/cache/conftool/dbconfig/20221107-235415-marostegui.json
  • 23:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38512 and previous config saved to /var/cache/conftool/dbconfig/20221107-235206-marostegui.json
  • 23:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 23:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 23:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T321123)', diff saved to https://phabricator.wikimedia.org/P38511 and previous config saved to /var/cache/conftool/dbconfig/20221107-235144-marostegui.json
  • 23:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P38510 and previous config saved to /var/cache/conftool/dbconfig/20221107-233637-marostegui.json
  • 23:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P38509 and previous config saved to /var/cache/conftool/dbconfig/20221107-232447-ladsgroup.json
  • 23:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P38508 and previous config saved to /var/cache/conftool/dbconfig/20221107-232131-marostegui.json
  • 23:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T318605)', diff saved to https://phabricator.wikimedia.org/P38507 and previous config saved to /var/cache/conftool/dbconfig/20221107-230940-ladsgroup.json
  • 23:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T321123)', diff saved to https://phabricator.wikimedia.org/P38506 and previous config saved to /var/cache/conftool/dbconfig/20221107-230624-marostegui.json
  • 23:04 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2172 (T321123)', diff saved to https://phabricator.wikimedia.org/P38505 and previous config saved to /var/cache/conftool/dbconfig/20221107-230414-marostegui.json
  • 23:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 23:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 23:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321123)', diff saved to https://phabricator.wikimedia.org/P38504 and previous config saved to /var/cache/conftool/dbconfig/20221107-230353-marostegui.json
  • 22:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38503 and previous config saved to /var/cache/conftool/dbconfig/20221107-225943-marostegui.json
  • 22:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2159 (T318605)', diff saved to https://phabricator.wikimedia.org/P38502 and previous config saved to /var/cache/conftool/dbconfig/20221107-225602-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T318605)', diff saved to https://phabricator.wikimedia.org/P38501 and previous config saved to /var/cache/conftool/dbconfig/20221107-225536-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T318605)', diff saved to https://phabricator.wikimedia.org/P38500 and previous config saved to /var/cache/conftool/dbconfig/20221107-225525-ladsgroup.json
  • 22:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 22:53 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on phab2001.codfw.wmnet with reason: T322250
  • 22:53 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on phab2001.codfw.wmnet with reason: T322250
  • 22:51 mutante: phab2001 - removing from production puppet role - removes ssh access, ferm rules, exim config and more T322250
  • 22:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P38499 and previous config saved to /var/cache/conftool/dbconfig/20221107-224847-marostegui.json
  • 22:44 maryum: Deployed patches for T316414 and T315123
  • 22:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P38498 and previous config saved to /var/cache/conftool/dbconfig/20221107-224437-marostegui.json
  • 22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P38497 and previous config saved to /var/cache/conftool/dbconfig/20221107-224029-ladsgroup.json
  • 22:36 ejegg: fundraising CiviCRM upgraded from c0db8f34 to 72fccce1
  • 22:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P38496 and previous config saved to /var/cache/conftool/dbconfig/20221107-223340-marostegui.json
  • 22:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P38495 and previous config saved to /var/cache/conftool/dbconfig/20221107-222930-marostegui.json
  • 22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P38494 and previous config saved to /var/cache/conftool/dbconfig/20221107-222523-ladsgroup.json
  • 22:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321123)', diff saved to https://phabricator.wikimedia.org/P38493 and previous config saved to /var/cache/conftool/dbconfig/20221107-221834-marostegui.json
  • 22:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 22:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 22:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 22:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2155 (T321123)', diff saved to https://phabricator.wikimedia.org/P38492 and previous config saved to /var/cache/conftool/dbconfig/20221107-221624-marostegui.json
  • 22:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 22:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 22:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 22:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 22:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38491 and previous config saved to /var/cache/conftool/dbconfig/20221107-221557-marostegui.json
  • 22:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38490 and previous config saved to /var/cache/conftool/dbconfig/20221107-221423-marostegui.json
  • 22:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 22:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38489 and previous config saved to /var/cache/conftool/dbconfig/20221107-221209-marostegui.json
  • 22:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 22:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 22:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38488 and previous config saved to /var/cache/conftool/dbconfig/20221107-221148-marostegui.json
  • 22:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T318605)', diff saved to https://phabricator.wikimedia.org/P38487 and previous config saved to /var/cache/conftool/dbconfig/20221107-221016-ladsgroup.json
  • 22:07 mutante: [apt1001:~] $ sudo -E reprepro --verbose --component thirdparty/terraform update bullseye-wikimedia - T322344
  • 22:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P38486 and previous config saved to /var/cache/conftool/dbconfig/20221107-220051-marostegui.json
  • 21:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P38485 and previous config saved to /var/cache/conftool/dbconfig/20221107-215641-marostegui.json
  • 21:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P38484 and previous config saved to /var/cache/conftool/dbconfig/20221107-214545-marostegui.json
  • 21:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 21:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 21:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38483 and previous config saved to /var/cache/conftool/dbconfig/20221107-214254-ladsgroup.json
  • 21:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316', diff saved to https://phabricator.wikimedia.org/P38482 and previous config saved to /var/cache/conftool/dbconfig/20221107-214135-marostegui.json
  • 21:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38481 and previous config saved to /var/cache/conftool/dbconfig/20221107-213038-marostegui.json
  • 21:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318955)', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20221107-212900-ladsgroup.json
  • 21:28 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38480 and previous config saved to /var/cache/conftool/dbconfig/20221107-212828-marostegui.json
  • 21:28 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38479 and previous config saved to /var/cache/conftool/dbconfig/20221107-212800-marostegui.json
  • 21:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P38478 and previous config saved to /var/cache/conftool/dbconfig/20221107-212748-ladsgroup.json
  • 21:26 mutante: DNS - removing phab1001-aphlict.eqiad.wmnet - should have no effect because we use aphlict.discovery.wmnet - but if it does, then it's Phabricator realtime notifications - T280597
  • 21:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2171:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38477 and previous config saved to /var/cache/conftool/dbconfig/20221107-212628-marostegui.json
  • 21:26 urbanecm: Start [urbanecm@mwmaint1002 /srv/mediawiki]$ foreachwikiindblist group0 extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all # T315510, running in mwmaint1002 at a tmux session under my name
  • 21:25 mutante: DNS - removing phab1001-aphlict.eqiad.wmnet - should have no effect because we use aphlict.discovery.wmnet - but if it does, then it's Phabricator realtime notifications
  • 21:23 urbanecm@deploy1002: Finished scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353) (duration: 05m 47s)
  • 21:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2171:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38476 and previous config saved to /var/cache/conftool/dbconfig/20221107-212156-marostegui.json
  • 21:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 21:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 21:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38475 and previous config saved to /var/cache/conftool/dbconfig/20221107-212135-marostegui.json
  • 21:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:19 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:17 urbanecm@deploy1002: urbanecm and matmarex: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 21:17 urbanecm@deploy1002: Started scap: Backport for Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353)
  • 21:16 urbanecm@deploy1002: Finished scap: Backport for ThreadItemStore: Update existing rows if possible rather than insert+delete (T321121) (duration: 07m 30s)
  • 21:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38474 and previous config saved to /var/cache/conftool/dbconfig/20221107-211353-ladsgroup.json
  • 21:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P38473 and previous config saved to /var/cache/conftool/dbconfig/20221107-211253-marostegui.json
  • 21:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P38472 and previous config saved to /var/cache/conftool/dbconfig/20221107-211241-ladsgroup.json
  • 21:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 21:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:09 urbanecm@deploy1002: urbanecm and matmarex: Backport for ThreadItemStore: Update existing rows if possible rather than insert+delete (T321121) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 21:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 21:09 urbanecm@deploy1002: Started scap: Backport for ThreadItemStore: Update existing rows if possible rather than insert+delete (T321121)
  • 21:08 urbanecm@deploy1002: Finished scap: Backport for Simplify some redundant settings, Clean up wgDiscussionToolsABTest config for beta cluster (duration: 04m 40s)
  • 21:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P38471 and previous config saved to /var/cache/conftool/dbconfig/20221107-210628-marostegui.json
  • 21:03 urbanecm@deploy1002: Started scap: Backport for Simplify some redundant settings, Clean up wgDiscussionToolsABTest config for beta cluster
  • 21:02 urbanecm@deploy1002: backport aborted: (duration: 00m 02s)
  • 20:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38470 and previous config saved to /var/cache/conftool/dbconfig/20221107-205847-ladsgroup.json
  • 20:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P38469 and previous config saved to /var/cache/conftool/dbconfig/20221107-205747-marostegui.json
  • 20:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38468 and previous config saved to /var/cache/conftool/dbconfig/20221107-205735-ladsgroup.json
  • 20:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P38467 and previous config saved to /var/cache/conftool/dbconfig/20221107-205122-marostegui.json
  • 20:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2150 (T318605)', diff saved to https://phabricator.wikimedia.org/P38466 and previous config saved to /var/cache/conftool/dbconfig/20221107-204827-ladsgroup.json
  • 20:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 20:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 20:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T318605)', diff saved to https://phabricator.wikimedia.org/P38465 and previous config saved to /var/cache/conftool/dbconfig/20221107-204805-ladsgroup.json
  • 20:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38464 and previous config saved to /var/cache/conftool/dbconfig/20221107-204340-ladsgroup.json
  • 20:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38463 and previous config saved to /var/cache/conftool/dbconfig/20221107-204240-marostegui.json
  • 20:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38462 and previous config saved to /var/cache/conftool/dbconfig/20221107-204131-marostegui.json
  • 20:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 20:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 20:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38461 and previous config saved to /var/cache/conftool/dbconfig/20221107-204110-marostegui.json
  • 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38460 and previous config saved to /var/cache/conftool/dbconfig/20221107-203626-ladsgroup.json
  • 20:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38459 and previous config saved to /var/cache/conftool/dbconfig/20221107-203615-marostegui.json
  • 20:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 20:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38458 and previous config saved to /var/cache/conftool/dbconfig/20221107-203609-ladsgroup.json
  • 20:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P38457 and previous config saved to /var/cache/conftool/dbconfig/20221107-203258-ladsgroup.json
  • 20:31 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2169:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38456 and previous config saved to /var/cache/conftool/dbconfig/20221107-203138-marostegui.json
  • 20:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 20:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 20:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T321130)', diff saved to https://phabricator.wikimedia.org/P38455 and previous config saved to /var/cache/conftool/dbconfig/20221107-203116-marostegui.json
  • 20:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P38454 and previous config saved to /var/cache/conftool/dbconfig/20221107-202603-marostegui.json
  • 20:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38453 and previous config saved to /var/cache/conftool/dbconfig/20221107-202102-ladsgroup.json
  • 20:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P38452 and previous config saved to /var/cache/conftool/dbconfig/20221107-201752-ladsgroup.json
  • 20:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P38451 and previous config saved to /var/cache/conftool/dbconfig/20221107-201610-marostegui.json
  • 20:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P38450 and previous config saved to /var/cache/conftool/dbconfig/20221107-201057-marostegui.json
  • 20:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38449 and previous config saved to /var/cache/conftool/dbconfig/20221107-200556-ladsgroup.json
  • 20:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T318605)', diff saved to https://phabricator.wikimedia.org/P38448 and previous config saved to /var/cache/conftool/dbconfig/20221107-200245-ladsgroup.json
  • 20:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P38447 and previous config saved to /var/cache/conftool/dbconfig/20221107-200103-marostegui.json
  • 19:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38446 and previous config saved to /var/cache/conftool/dbconfig/20221107-195550-marostegui.json
  • 19:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38445 and previous config saved to /var/cache/conftool/dbconfig/20221107-195340-marostegui.json
  • 19:53 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 19:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 19:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T321123)', diff saved to https://phabricator.wikimedia.org/P38444 and previous config saved to /var/cache/conftool/dbconfig/20221107-195319-marostegui.json
  • 19:51 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 19:51 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host sretest2002
  • 19:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38443 and previous config saved to /var/cache/conftool/dbconfig/20221107-195049-ladsgroup.json
  • 19:50 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2002
  • 19:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T321130)', diff saved to https://phabricator.wikimedia.org/P38442 and previous config saved to /var/cache/conftool/dbconfig/20221107-194557-marostegui.json
  • 19:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38441 and previous config saved to /var/cache/conftool/dbconfig/20221107-194335-ladsgroup.json
  • 19:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 19:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 19:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 19:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 19:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38440 and previous config saved to /var/cache/conftool/dbconfig/20221107-194319-ladsgroup.json
  • 19:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2158 (T321130)', diff saved to https://phabricator.wikimedia.org/P38439 and previous config saved to /var/cache/conftool/dbconfig/20221107-194026-marostegui.json
  • 19:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2095.codfw.wmnet with reason: Maintenance
  • 19:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 19:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 19:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P38438 and previous config saved to /var/cache/conftool/dbconfig/20221107-193813-marostegui.json
  • 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38437 and previous config saved to /var/cache/conftool/dbconfig/20221107-193646-ladsgroup.json
  • 19:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T318605)', diff saved to https://phabricator.wikimedia.org/P38436 and previous config saved to /var/cache/conftool/dbconfig/20221107-193625-ladsgroup.json
  • 19:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 19:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 19:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38435 and previous config saved to /var/cache/conftool/dbconfig/20221107-193604-marostegui.json
  • 19:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38434 and previous config saved to /var/cache/conftool/dbconfig/20221107-192813-ladsgroup.json
  • 19:25 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P38433 and previous config saved to /var/cache/conftool/dbconfig/20221107-192306-marostegui.json
  • 19:24 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:24 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P38432 and previous config saved to /var/cache/conftool/dbconfig/20221107-192119-ladsgroup.json
  • 19:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P38431 and previous config saved to /var/cache/conftool/dbconfig/20221107-192058-marostegui.json
  • 19:16 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38430 and previous config saved to /var/cache/conftool/dbconfig/20221107-191306-ladsgroup.json
  • 19:10 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 19:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T321123)', diff saved to https://phabricator.wikimedia.org/P38429 and previous config saved to /var/cache/conftool/dbconfig/20221107-190800-marostegui.json
  • 19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P38428 and previous config saved to /var/cache/conftool/dbconfig/20221107-190612-ladsgroup.json
  • 19:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P38427 and previous config saved to /var/cache/conftool/dbconfig/20221107-190551-marostegui.json
  • 19:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2136 (T321123)', diff saved to https://phabricator.wikimedia.org/P38426 and previous config saved to /var/cache/conftool/dbconfig/20221107-190550-marostegui.json
  • 19:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 19:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 19:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38425 and previous config saved to /var/cache/conftool/dbconfig/20221107-190528-marostegui.json
  • 18:58 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38424 and previous config saved to /var/cache/conftool/dbconfig/20221107-185800-ladsgroup.json
  • 18:57 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T318605)', diff saved to https://phabricator.wikimedia.org/P38423 and previous config saved to /var/cache/conftool/dbconfig/20221107-185105-ladsgroup.json
  • 18:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38422 and previous config saved to /var/cache/conftool/dbconfig/20221107-185044-marostegui.json
  • 18:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38421 and previous config saved to /var/cache/conftool/dbconfig/20221107-185035-ladsgroup.json
  • 18:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 18:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 18:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P38420 and previous config saved to /var/cache/conftool/dbconfig/20221107-185022-marostegui.json
  • 18:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2129 (T321130)', diff saved to https://phabricator.wikimedia.org/P38419 and previous config saved to /var/cache/conftool/dbconfig/20221107-184510-marostegui.json
  • 18:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 18:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 18:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38418 and previous config saved to /var/cache/conftool/dbconfig/20221107-184502-ladsgroup.json
  • 18:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 18:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321130)', diff saved to https://phabricator.wikimedia.org/P38417 and previous config saved to /var/cache/conftool/dbconfig/20221107-184448-marostegui.json
  • 18:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T318605)', diff saved to https://phabricator.wikimedia.org/P38416 and previous config saved to /var/cache/conftool/dbconfig/20221107-183722-ladsgroup.json
  • 18:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 18:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 18:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 18:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 18:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T318605)', diff saved to https://phabricator.wikimedia.org/P38415 and previous config saved to /var/cache/conftool/dbconfig/20221107-183643-ladsgroup.json
  • 18:36 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P38414 and previous config saved to /var/cache/conftool/dbconfig/20221107-183515-marostegui.json
  • 18:33 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host arclamp2001.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38413 and previous config saved to /var/cache/conftool/dbconfig/20221107-182956-ladsgroup.json
  • 18:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P38412 and previous config saved to /var/cache/conftool/dbconfig/20221107-182941-marostegui.json
  • 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2122 (T318605)', diff saved to https://phabricator.wikimedia.org/P38411 and previous config saved to /var/cache/conftool/dbconfig/20221107-182704-ladsgroup.json
  • 18:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 18:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 18:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38410 and previous config saved to /var/cache/conftool/dbconfig/20221107-182642-ladsgroup.json
  • 18:25 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host arclamp2001.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:24 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host puppetdb2003.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P38409 and previous config saved to /var/cache/conftool/dbconfig/20221107-182137-ladsgroup.json
  • 18:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38408 and previous config saved to /var/cache/conftool/dbconfig/20221107-182009-marostegui.json
  • 18:18 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host puppetdb2003.mgmt.codfw.wmnet with reboot policy FORCED
  • 18:18 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2119 (T321123)', diff saved to https://phabricator.wikimedia.org/P38407 and previous config saved to /var/cache/conftool/dbconfig/20221107-181759-marostegui.json
  • 18:17 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 18:17 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 18:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T321123)', diff saved to https://phabricator.wikimedia.org/P38406 and previous config saved to /var/cache/conftool/dbconfig/20221107-181737-marostegui.json
  • 18:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38405 and previous config saved to /var/cache/conftool/dbconfig/20221107-181449-ladsgroup.json
  • 18:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P38404 and previous config saved to /var/cache/conftool/dbconfig/20221107-181435-marostegui.json
  • 18:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P38403 and previous config saved to /var/cache/conftool/dbconfig/20221107-181135-ladsgroup.json
  • 18:11 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host arclamp2001
  • 18:10 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host arclamp2001
  • 18:09 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host puppetdb2003
  • 18:08 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host puppetdb2003
  • 18:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P38402 and previous config saved to /var/cache/conftool/dbconfig/20221107-180630-ladsgroup.json
  • 18:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P38401 and previous config saved to /var/cache/conftool/dbconfig/20221107-180230-marostegui.json
  • 18:00 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38400 and previous config saved to /var/cache/conftool/dbconfig/20221107-175943-ladsgroup.json
  • 17:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321130)', diff saved to https://phabricator.wikimedia.org/P38399 and previous config saved to /var/cache/conftool/dbconfig/20221107-175928-marostegui.json
  • 17:58 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P38398 and previous config saved to /var/cache/conftool/dbconfig/20221107-175629-ladsgroup.json
  • 17:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2124 (T321130)', diff saved to https://phabricator.wikimedia.org/P38397 and previous config saved to /var/cache/conftool/dbconfig/20221107-175357-marostegui.json
  • 17:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 17:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 17:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321130)', diff saved to https://phabricator.wikimedia.org/P38396 and previous config saved to /var/cache/conftool/dbconfig/20221107-175335-marostegui.json
  • 17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38395 and previous config saved to /var/cache/conftool/dbconfig/20221107-175228-ladsgroup.json
  • 17:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 17:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38394 and previous config saved to /var/cache/conftool/dbconfig/20221107-175217-ladsgroup.json
  • 17:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 (T318605)', diff saved to https://phabricator.wikimedia.org/P38393 and previous config saved to /var/cache/conftool/dbconfig/20221107-175124-ladsgroup.json
  • 17:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P38392 and previous config saved to /var/cache/conftool/dbconfig/20221107-174724-marostegui.json
  • 17:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38391 and previous config saved to /var/cache/conftool/dbconfig/20221107-174123-ladsgroup.json
  • 17:41 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 17:38 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 17:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P38390 and previous config saved to /var/cache/conftool/dbconfig/20221107-173829-marostegui.json
  • 17:37 filippo@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 17:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38389 and previous config saved to /var/cache/conftool/dbconfig/20221107-173711-ladsgroup.json
  • 17:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2110 (T321123)', diff saved to https://phabricator.wikimedia.org/P38388 and previous config saved to /var/cache/conftool/dbconfig/20221107-173217-marostegui.json
  • 17:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2110 (T321123)', diff saved to https://phabricator.wikimedia.org/P38387 and previous config saved to /var/cache/conftool/dbconfig/20221107-173007-marostegui.json
  • 17:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 17:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 17:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38386 and previous config saved to /var/cache/conftool/dbconfig/20221107-172946-marostegui.json
  • 17:24 krinkle@deploy1002: Finished deploy [performance/arc-lamp@e1ac118]: https://gerrit.wikimedia.org/r/c/825870 - T322561, T315056 (duration: 00m 07s)
  • 17:24 krinkle@deploy1002: Started deploy [performance/arc-lamp@e1ac118]: https://gerrit.wikimedia.org/r/c/825870 - T322561, T315056
  • 17:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P38385 and previous config saved to /var/cache/conftool/dbconfig/20221107-172322-marostegui.json
  • 17:22 sukhe: reprepro -C main include bullseye-wikimedia purged_0.19_amd64.changes: T321309
  • 17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38384 and previous config saved to /var/cache/conftool/dbconfig/20221107-172204-ladsgroup.json
  • 17:22 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 17:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P38383 and previous config saved to /var/cache/conftool/dbconfig/20221107-171439-marostegui.json
  • 17:13 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 17:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321130)', diff saved to https://phabricator.wikimedia.org/P38382 and previous config saved to /var/cache/conftool/dbconfig/20221107-170816-marostegui.json
  • 17:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38381 and previous config saved to /var/cache/conftool/dbconfig/20221107-170658-ladsgroup.json
  • 17:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2117 (T321130)', diff saved to https://phabricator.wikimedia.org/P38380 and previous config saved to /var/cache/conftool/dbconfig/20221107-170247-marostegui.json
  • 17:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 17:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 16:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38379 and previous config saved to /var/cache/conftool/dbconfig/20221107-165943-ladsgroup.json
  • 16:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P38378 and previous config saved to /var/cache/conftool/dbconfig/20221107-165933-marostegui.json
  • 16:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 16:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 16:59 filippo@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host dispatch-be2001.codfw.wmnet
  • 16:59 filippo@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync-mgmt - filippo@cumin1001"
  • 16:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 16:58 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
  • 16:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T321130)', diff saved to https://phabricator.wikimedia.org/P38377 and previous config saved to /var/cache/conftool/dbconfig/20221107-165847-marostegui.json
  • 16:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 (T318605)', diff saved to https://phabricator.wikimedia.org/P38376 and previous config saved to /var/cache/conftool/dbconfig/20221107-165108-ladsgroup.json
  • 16:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T318605)', diff saved to https://phabricator.wikimedia.org/P38375 and previous config saved to /var/cache/conftool/dbconfig/20221107-165046-ladsgroup.json
  • 16:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T318955)', diff saved to https://phabricator.wikimedia.org/P38374 and previous config saved to /var/cache/conftool/dbconfig/20221107-165036-ladsgroup.json
  • 16:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38373 and previous config saved to /var/cache/conftool/dbconfig/20221107-164427-marostegui.json
  • 16:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P38372 and previous config saved to /var/cache/conftool/dbconfig/20221107-164340-marostegui.json
  • 16:42 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db2106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38371 and previous config saved to /var/cache/conftool/dbconfig/20221107-164217-marostegui.json
  • 16:42 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 16:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321123)', diff saved to https://phabricator.wikimedia.org/P38370 and previous config saved to /var/cache/conftool/dbconfig/20221107-164122-marostegui.json
  • 16:38 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:35 filippo@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dispatch-be2001.codfw.wmnet on all recursors
  • 16:35 filippo@cumin1001: START - Cookbook sre.dns.wipe-cache dispatch-be2001.codfw.wmnet on all recursors
  • 16:35 filippo@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38369 and previous config saved to /var/cache/conftool/dbconfig/20221107-163540-ladsgroup.json
  • 16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38368 and previous config saved to /var/cache/conftool/dbconfig/20221107-163529-ladsgroup.json
  • 16:33 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P38367 and previous config saved to /var/cache/conftool/dbconfig/20221107-162834-marostegui.json
  • 16:26 filippo@cumin1001: START - Cookbook sre.dns.netbox
  • 16:26 filippo@cumin1001: START - Cookbook sre.ganeti.makevm for new host dispatch-be2001.codfw.wmnet
  • 16:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P38366 and previous config saved to /var/cache/conftool/dbconfig/20221107-162616-marostegui.json
  • 16:23 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:21 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:21 volans@cumin1001: START - Cookbook sre.dns.netbox
  • 16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38365 and previous config saved to /var/cache/conftool/dbconfig/20221107-162033-ladsgroup.json
  • 16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P38364 and previous config saved to /var/cache/conftool/dbconfig/20221107-162023-ladsgroup.json
  • 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2121 (T318605)', diff saved to https://phabricator.wikimedia.org/P38363 and previous config saved to /var/cache/conftool/dbconfig/20221107-161837-ladsgroup.json
  • 16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 16:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T318605)', diff saved to https://phabricator.wikimedia.org/P38362 and previous config saved to /var/cache/conftool/dbconfig/20221107-161816-ladsgroup.json
  • 16:14 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T321130)', diff saved to https://phabricator.wikimedia.org/P38361 and previous config saved to /var/cache/conftool/dbconfig/20221107-161327-marostegui.json
  • 16:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1201 (T321130)', diff saved to https://phabricator.wikimedia.org/P38360 and previous config saved to /var/cache/conftool/dbconfig/20221107-161118-marostegui.json
  • 16:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P38359 and previous config saved to /var/cache/conftool/dbconfig/20221107-161109-marostegui.json
  • 16:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 16:10 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 16:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T321130)', diff saved to https://phabricator.wikimedia.org/P38358 and previous config saved to /var/cache/conftool/dbconfig/20221107-161050-marostegui.json
  • 16:06 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 16:06 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@e51ff67]: import_cirrus_indexes: set executor cores to 1 (duration: 02m 19s)
  • 16:05 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 16:05 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T318605)', diff saved to https://phabricator.wikimedia.org/P38357 and previous config saved to /var/cache/conftool/dbconfig/20221107-160527-ladsgroup.json
  • 16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T318955)', diff saved to https://phabricator.wikimedia.org/P38356 and previous config saved to /var/cache/conftool/dbconfig/20221107-160516-ladsgroup.json
  • 16:04 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 16:03 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1010.eqiad.wmnet to cluster eqiad and group C
  • 16:03 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@e51ff67]: import_cirrus_indexes: set executor cores to 1
  • 16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P38355 and previous config saved to /var/cache/conftool/dbconfig/20221107-160310-ladsgroup.json
  • 16:02 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1010.eqiad.wmnet to cluster eqiad and group C
  • 16:02 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:02 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 16:02 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1198 (T318955)', diff saved to https://phabricator.wikimedia.org/P38354 and previous config saved to /var/cache/conftool/dbconfig/20221107-160124-ladsgroup.json
  • 16:01 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 16:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 16:01 claime: cleaning up stale mwdebug kubernetes config
  • 16:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T318955)', diff saved to https://phabricator.wikimedia.org/P38353 and previous config saved to /var/cache/conftool/dbconfig/20221107-160102-ladsgroup.json
  • 16:00 cgoubert@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 15:59 cgoubert@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 15:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321123)', diff saved to https://phabricator.wikimedia.org/P38352 and previous config saved to /var/cache/conftool/dbconfig/20221107-155603-marostegui.json
  • 15:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P38351 and previous config saved to /var/cache/conftool/dbconfig/20221107-155544-marostegui.json
  • 15:55 elukey: upgrade istioctl to 1.15.3 on apt1001 for {buster,bullseye}-wikimedia - T322193
  • 15:54 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1199 (T321123)', diff saved to https://phabricator.wikimedia.org/P38350 and previous config saved to /var/cache/conftool/dbconfig/20221107-155455-marostegui.json
  • 15:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 15:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 15:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321123)', diff saved to https://phabricator.wikimedia.org/P38349 and previous config saved to /var/cache/conftool/dbconfig/20221107-155434-marostegui.json
  • 15:51 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:50 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:50 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:49 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P38348 and previous config saved to /var/cache/conftool/dbconfig/20221107-154803-ladsgroup.json
  • 15:46 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:46 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38347 and previous config saved to /var/cache/conftool/dbconfig/20221107-154556-ladsgroup.json
  • 15:44 sukhe: reprepro -C main include bullseye-wikimedia libvmod-querysort_0.3_amd64.changes: T321309
  • 15:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P38346 and previous config saved to /var/cache/conftool/dbconfig/20221107-154037-marostegui.json
  • 15:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P38345 and previous config saved to /var/cache/conftool/dbconfig/20221107-153927-marostegui.json
  • 15:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T318605)', diff saved to https://phabricator.wikimedia.org/P38344 and previous config saved to /var/cache/conftool/dbconfig/20221107-153257-ladsgroup.json
  • 15:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P38343 and previous config saved to /var/cache/conftool/dbconfig/20221107-153049-ladsgroup.json
  • 15:27 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 15:27 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 15:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T321130)', diff saved to https://phabricator.wikimedia.org/P38342 and previous config saved to /var/cache/conftool/dbconfig/20221107-152531-marostegui.json
  • 15:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P38341 and previous config saved to /var/cache/conftool/dbconfig/20221107-152421-marostegui.json
  • 15:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1187 (T321130)', diff saved to https://phabricator.wikimedia.org/P38340 and previous config saved to /var/cache/conftool/dbconfig/20221107-152322-marostegui.json
  • 15:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 15:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 15:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38339 and previous config saved to /var/cache/conftool/dbconfig/20221107-152301-marostegui.json
  • 15:18 sukhe: reprepro -C main include bullseye-wikimedia libvmod-netmapper_1.9-2_amd64.changes: T321309
  • 15:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T318955)', diff saved to https://phabricator.wikimedia.org/P38338 and previous config saved to /var/cache/conftool/dbconfig/20221107-151543-ladsgroup.json
  • 15:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1189 (T318955)', diff saved to https://phabricator.wikimedia.org/P38337 and previous config saved to /var/cache/conftool/dbconfig/20221107-151151-ladsgroup.json
  • 15:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 15:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 15:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T318955)', diff saved to https://phabricator.wikimedia.org/P38336 and previous config saved to /var/cache/conftool/dbconfig/20221107-151130-ladsgroup.json
  • 15:09 sukhe: reprepro -C main include bullseye-wikimedia varnishkafka_1.1.0-2_amd64.changes: T321309
  • 15:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321123)', diff saved to https://phabricator.wikimedia.org/P38335 and previous config saved to /var/cache/conftool/dbconfig/20221107-150914-marostegui.json
  • 15:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1190 (T321123)', diff saved to https://phabricator.wikimedia.org/P38334 and previous config saved to /var/cache/conftool/dbconfig/20221107-150807-marostegui.json
  • 15:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 15:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P38333 and previous config saved to /var/cache/conftool/dbconfig/20221107-150754-marostegui.json
  • 15:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 15:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321123)', diff saved to https://phabricator.wikimedia.org/P38332 and previous config saved to /var/cache/conftool/dbconfig/20221107-150745-marostegui.json
  • 14:57 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1010.eqiad.wmnet
  • 14:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38331 and previous config saved to /var/cache/conftool/dbconfig/20221107-145623-ladsgroup.json
  • 14:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P38330 and previous config saved to /var/cache/conftool/dbconfig/20221107-145248-marostegui.json
  • 14:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P38329 and previous config saved to /var/cache/conftool/dbconfig/20221107-145239-marostegui.json
  • 14:48 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1010.eqiad.wmnet
  • 14:43 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P38328 and previous config saved to /var/cache/conftool/dbconfig/20221107-144117-ladsgroup.json
  • 14:40 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:40 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38327 and previous config saved to /var/cache/conftool/dbconfig/20221107-143741-marostegui.json
  • 14:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P38326 and previous config saved to /var/cache/conftool/dbconfig/20221107-143732-marostegui.json
  • 14:37 urbanecm: UTC afternoon B&C window done
  • 14:36 urbanecm@deploy1002: Finished scap: Backport for MentorHooks: Add missing check for GEMentorshipUseIsActiveFlag (T322538), Rename QuitMentorship to ReassignMentees (T321382), ReassignMentees: Pass the actual performer to ChangeMentor (T321382), ManageMentorsRemoveMentor: Reassign mentees to a different mentor (T321382) (duration: 14m 56s)
  • 14:35 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1180 (T321130)', diff saved to https://phabricator.wikimedia.org/P38324 and previous config saved to /var/cache/conftool/dbconfig/20221107-143526-marostegui.json
  • 14:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 14:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 14:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321130)', diff saved to https://phabricator.wikimedia.org/P38323 and previous config saved to /var/cache/conftool/dbconfig/20221107-143504-marostegui.json
  • 14:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T318605)', diff saved to https://phabricator.wikimedia.org/P38322 and previous config saved to /var/cache/conftool/dbconfig/20221107-142904-ladsgroup.json
  • 14:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 14:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 14:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38321 and previous config saved to /var/cache/conftool/dbconfig/20221107-142832-ladsgroup.json
  • 14:26 urbanecm@deploy1002: urbanecm and urbanecm: Backport for MentorHooks: Add missing check for GEMentorshipUseIsActiveFlag (T322538), Rename QuitMentorship to ReassignMentees (T321382), ReassignMentees: Pass the actual performer to ChangeMentor (T321382), ManageMentorsRemoveMentor: Reassign mentees to a different mentor (T321382) synced to the testse
  • 14:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T318955)', diff saved to https://phabricator.wikimedia.org/P38320 and previous config saved to /var/cache/conftool/dbconfig/20221107-142610-ladsgroup.json
  • 14:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321123)', diff saved to https://phabricator.wikimedia.org/P38319 and previous config saved to /var/cache/conftool/dbconfig/20221107-142226-marostegui.json
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T318955)', diff saved to https://phabricator.wikimedia.org/P38318 and previous config saved to /var/cache/conftool/dbconfig/20221107-142219-ladsgroup.json
  • 14:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 14:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1179.eqiad.wmnet with reason: Maintenance
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T318955)', diff saved to https://phabricator.wikimedia.org/P38317 and previous config saved to /var/cache/conftool/dbconfig/20221107-142157-ladsgroup.json
  • 14:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1010.eqiad.wmnet with OS bullseye
  • 14:21 urbanecm@deploy1002: Started scap: Backport for MentorHooks: Add missing check for GEMentorshipUseIsActiveFlag (T322538), Rename QuitMentorship to ReassignMentees (T321382), ReassignMentees: Pass the actual performer to ChangeMentor (T321382), ManageMentorsRemoveMentor: Reassign mentees to a different mentor (T321382)
  • 14:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T321123)', diff saved to https://phabricator.wikimedia.org/P38316 and previous config saved to /var/cache/conftool/dbconfig/20221107-142118-marostegui.json
  • 14:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 14:21 urbanecm@deploy1002: Backport cancelled.
  • 14:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 14:20 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 14:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 14:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38315 and previous config saved to /var/cache/conftool/dbconfig/20221107-142041-marostegui.json
  • 14:20 urbanecm@deploy1002: Finished scap: Backport for Set timezone for knwiki , knwiktionary , knwikiquote and knwikisource (T322471) (duration: 06m 01s)
  • 14:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P38314 and previous config saved to /var/cache/conftool/dbconfig/20221107-141958-marostegui.json
  • 14:19 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:19 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:18 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:14 urbanecm@deploy1002: urbanecm and anzx: Backport for Set timezone for knwiki , knwiktionary , knwikiquote and knwikisource (T322471) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 14:14 urbanecm@deploy1002: Started scap: Backport for Set timezone for knwiki , knwiktionary , knwikiquote and knwikisource (T322471)
  • 14:13 urbanecm@deploy1002: Finished scap: Backport for Enable flood flag on knwiki (T322472) (duration: 05m 10s)
  • 14:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38313 and previous config saved to /var/cache/conftool/dbconfig/20221107-141344-ladsgroup.json
  • 14:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P38312 and previous config saved to /var/cache/conftool/dbconfig/20221107-141326-ladsgroup.json
  • 14:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2120 (T318605)', diff saved to https://phabricator.wikimedia.org/P38311 and previous config saved to /var/cache/conftool/dbconfig/20221107-141224-ladsgroup.json
  • 14:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 14:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T318605)', diff saved to https://phabricator.wikimedia.org/P38310 and previous config saved to /var/cache/conftool/dbconfig/20221107-141203-ladsgroup.json
  • 14:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:09 xcollazo@deploy1002: Finished deploy [airflow-dags/platform_eng@3bb99c2]: Deploying to Airflow platform_eng instance (duration: 00m 20s)
  • 14:09 urbanecm@deploy1002: urbanecm and anzx: Backport for Enable flood flag on knwiki (T322472) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
  • 14:09 xcollazo@deploy1002: Started deploy [airflow-dags/platform_eng@3bb99c2]: Deploying to Airflow platform_eng instance
  • 14:08 urbanecm@deploy1002: Started scap: Backport for Enable flood flag on knwiki (T322472)
  • 14:08 urbanecm@deploy1002: Finished scap: Backport for testwiki: Add config for Visual Editor Feature Use instrument (T309602) (duration: 06m 43s)
  • 14:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 14:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38309 and previous config saved to /var/cache/conftool/dbconfig/20221107-140651-ladsgroup.json
  • 14:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 14:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P38308 and previous config saved to /var/cache/conftool/dbconfig/20221107-140535-marostegui.json
  • 14:05 btullis@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:05 btullis@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 14:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 14:05 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1010.eqiad.wmnet with reason: host reimage
  • 14:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P38307 and previous config saved to /var/cache/conftool/dbconfig/20221107-140451-marostegui.json
  • 14:02 urbanecm@deploy1002: urbanecm and cjming: Backport for testwiki: Add config for Visual Editor Feature Use instrument (T309602) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 14:01 urbanecm@deploy1002: Started scap: Backport for testwiki: Add config for Visual Editor Feature Use instrument (T309602)
  • 14:01 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1010.eqiad.wmnet with reason: host reimage
  • 13:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38306 and previous config saved to /var/cache/conftool/dbconfig/20221107-135837-ladsgroup.json
  • 13:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P38305 and previous config saved to /var/cache/conftool/dbconfig/20221107-135819-ladsgroup.json
  • 13:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P38304 and previous config saved to /var/cache/conftool/dbconfig/20221107-135656-ladsgroup.json
  • 13:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P38303 and previous config saved to /var/cache/conftool/dbconfig/20221107-135144-ladsgroup.json
  • 13:50 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P38302 and previous config saved to /var/cache/conftool/dbconfig/20221107-135028-marostegui.json
  • 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321130)', diff saved to https://phabricator.wikimedia.org/P38301 and previous config saved to /var/cache/conftool/dbconfig/20221107-134944-marostegui.json
  • 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1168 (T321130)', diff saved to https://phabricator.wikimedia.org/P38300 and previous config saved to /var/cache/conftool/dbconfig/20221107-134735-marostegui.json
  • 13:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 13:47 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti1010.eqiad.wmnet with OS bullseye
  • 13:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T321130)', diff saved to https://phabricator.wikimedia.org/P38299 and previous config saved to /var/cache/conftool/dbconfig/20221107-134714-marostegui.json
  • 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P38298 and previous config saved to /var/cache/conftool/dbconfig/20221107-134331-ladsgroup.json
  • 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38297 and previous config saved to /var/cache/conftool/dbconfig/20221107-134313-ladsgroup.json
  • 13:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P38296 and previous config saved to /var/cache/conftool/dbconfig/20221107-134150-ladsgroup.json
  • 13:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T318955)', diff saved to https://phabricator.wikimedia.org/P38295 and previous config saved to /var/cache/conftool/dbconfig/20221107-133638-ladsgroup.json
  • 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38294 and previous config saved to /var/cache/conftool/dbconfig/20221107-133522-marostegui.json
  • 13:34 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T321123)', diff saved to https://phabricator.wikimedia.org/P38293 and previous config saved to /var/cache/conftool/dbconfig/20221107-133414-marostegui.json
  • 13:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 13:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T321123)', diff saved to https://phabricator.wikimedia.org/P38292 and previous config saved to /var/cache/conftool/dbconfig/20221107-133353-marostegui.json
  • 13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T318955)', diff saved to https://phabricator.wikimedia.org/P38291 and previous config saved to /var/cache/conftool/dbconfig/20221107-133246-ladsgroup.json
  • 13:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 13:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T318955)', diff saved to https://phabricator.wikimedia.org/P38290 and previous config saved to /var/cache/conftool/dbconfig/20221107-133225-ladsgroup.json
  • 13:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P38289 and previous config saved to /var/cache/conftool/dbconfig/20221107-133208-marostegui.json
  • 13:29 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1010.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 13:29 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1010.eqiad.wmnet with reason: Remove from cluster for eventual reimage
  • 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38288 and previous config saved to /var/cache/conftool/dbconfig/20221107-132824-ladsgroup.json
  • 13:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T318605)', diff saved to https://phabricator.wikimedia.org/P38287 and previous config saved to /var/cache/conftool/dbconfig/20221107-132643-ladsgroup.json
  • 13:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2177 (T318955)', diff saved to https://phabricator.wikimedia.org/P38286 and previous config saved to /var/cache/conftool/dbconfig/20221107-132109-ladsgroup.json
  • 13:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 13:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38285 and previous config saved to /var/cache/conftool/dbconfig/20221107-132048-ladsgroup.json
  • 13:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P38284 and previous config saved to /var/cache/conftool/dbconfig/20221107-131846-marostegui.json
  • 13:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38283 and previous config saved to /var/cache/conftool/dbconfig/20221107-131718-ladsgroup.json
  • 13:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P38282 and previous config saved to /var/cache/conftool/dbconfig/20221107-131701-marostegui.json
  • 13:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38281 and previous config saved to /var/cache/conftool/dbconfig/20221107-130541-ladsgroup.json
  • 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P38280 and previous config saved to /var/cache/conftool/dbconfig/20221107-130340-marostegui.json
  • 13:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P38279 and previous config saved to /var/cache/conftool/dbconfig/20221107-130212-ladsgroup.json
  • 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T321130)', diff saved to https://phabricator.wikimedia.org/P38278 and previous config saved to /var/cache/conftool/dbconfig/20221107-130155-marostegui.json
  • 12:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1165 (T321130)', diff saved to https://phabricator.wikimedia.org/P38277 and previous config saved to /var/cache/conftool/dbconfig/20221107-125946-marostegui.json
  • 12:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 12:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T321130)', diff saved to https://phabricator.wikimedia.org/P38276 and previous config saved to /var/cache/conftool/dbconfig/20221107-125529-marostegui.json
  • 12:51 XioNoX: Add NAT for frmon2001 - T321735
  • 12:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P38275 and previous config saved to /var/cache/conftool/dbconfig/20221107-125035-ladsgroup.json
  • 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T321123)', diff saved to https://phabricator.wikimedia.org/P38274 and previous config saved to /var/cache/conftool/dbconfig/20221107-124833-marostegui.json
  • 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T318955)', diff saved to https://phabricator.wikimedia.org/P38273 and previous config saved to /var/cache/conftool/dbconfig/20221107-124706-ladsgroup.json
  • 12:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T321123)', diff saved to https://phabricator.wikimedia.org/P38272 and previous config saved to /var/cache/conftool/dbconfig/20221107-124526-marostegui.json
  • 12:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 12:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 12:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38271 and previous config saved to /var/cache/conftool/dbconfig/20221107-124504-marostegui.json
  • 12:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P38270 and previous config saved to /var/cache/conftool/dbconfig/20221107-124022-marostegui.json
  • 12:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38269 and previous config saved to /var/cache/conftool/dbconfig/20221107-123528-ladsgroup.json
  • 12:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P38268 and previous config saved to /var/cache/conftool/dbconfig/20221107-122957-marostegui.json
  • 12:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2156 (T318955)', diff saved to https://phabricator.wikimedia.org/P38267 and previous config saved to /var/cache/conftool/dbconfig/20221107-122814-ladsgroup.json
  • 12:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38266 and previous config saved to /var/cache/conftool/dbconfig/20221107-122737-ladsgroup.json
  • 12:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P38265 and previous config saved to /var/cache/conftool/dbconfig/20221107-122516-marostegui.json
  • 12:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P38264 and previous config saved to /var/cache/conftool/dbconfig/20221107-121451-marostegui.json
  • 12:13 sukhe: reprepro -C main include bullseye-wikimedia prometheus-varnishkafka-exporter_0.1-2_amd64.changes: T321309
  • 12:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38263 and previous config saved to /var/cache/conftool/dbconfig/20221107-121230-ladsgroup.json
  • 12:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T321130)', diff saved to https://phabricator.wikimedia.org/P38262 and previous config saved to /var/cache/conftool/dbconfig/20221107-121009-marostegui.json
  • 12:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38261 and previous config saved to /var/cache/conftool/dbconfig/20221107-121004-ladsgroup.json
  • 12:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 12:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 12:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38260 and previous config saved to /var/cache/conftool/dbconfig/20221107-120942-ladsgroup.json
  • 12:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T321130)', diff saved to https://phabricator.wikimedia.org/P38259 and previous config saved to /var/cache/conftool/dbconfig/20221107-120800-marostegui.json
  • 12:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 12:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1131.eqiad.wmnet with reason: Maintenance
  • 12:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38258 and previous config saved to /var/cache/conftool/dbconfig/20221107-120739-marostegui.json
  • 12:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2108 (T318605)', diff saved to https://phabricator.wikimedia.org/P38257 and previous config saved to /var/cache/conftool/dbconfig/20221107-120614-ladsgroup.json
  • 12:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 12:05 vgutierrez: testing acme-chief 0.35 in acmechief-test1001
  • 12:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 12:05 volans@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts ganeti4003.ulsfo.wmnet
  • 12:05 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:04 volans@cumin1001: START - Cookbook sre.dns.netbox
  • 12:00 volans@cumin1001: START - Cookbook sre.hosts.decommission for hosts ganeti4003.ulsfo.wmnet
  • 11:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38256 and previous config saved to /var/cache/conftool/dbconfig/20221107-115944-marostegui.json
  • 11:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T321123)', diff saved to https://phabricator.wikimedia.org/P38255 and previous config saved to /var/cache/conftool/dbconfig/20221107-115737-marostegui.json
  • 11:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 11:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P38254 and previous config saved to /var/cache/conftool/dbconfig/20221107-115723-ladsgroup.json
  • 11:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 11:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38253 and previous config saved to /var/cache/conftool/dbconfig/20221107-115715-marostegui.json
  • 11:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P38252 and previous config saved to /var/cache/conftool/dbconfig/20221107-115436-ladsgroup.json
  • 11:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P38251 and previous config saved to /var/cache/conftool/dbconfig/20221107-115232-marostegui.json
  • 11:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T318955)', diff saved to https://phabricator.wikimedia.org/P38250 and previous config saved to /var/cache/conftool/dbconfig/20221107-114649-ladsgroup.json
  • 11:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 11:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T318955)', diff saved to https://phabricator.wikimedia.org/P38249 and previous config saved to /var/cache/conftool/dbconfig/20221107-114628-ladsgroup.json
  • 11:44 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 112
  • 11:44 ayounsi@cumin1001: START - Cookbook sre.network.debug for Netbox circuit ID 112
  • 11:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38248 and previous config saved to /var/cache/conftool/dbconfig/20221107-114217-ladsgroup.json
  • 11:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P38247 and previous config saved to /var/cache/conftool/dbconfig/20221107-114209-marostegui.json
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P38246 and previous config saved to /var/cache/conftool/dbconfig/20221107-113929-ladsgroup.json
  • 11:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P38245 and previous config saved to /var/cache/conftool/dbconfig/20221107-113726-marostegui.json
  • 11:36 arturo: running homer on cr-eqiad/cr-codfw for https://gerrit.wikimedia.org/r/853947 (T321220, T309407)
  • 11:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2149 (T318955)', diff saved to https://phabricator.wikimedia.org/P38244 and previous config saved to /var/cache/conftool/dbconfig/20221107-113452-ladsgroup.json
  • 11:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 11:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 11:32 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38243 and previous config saved to /var/cache/conftool/dbconfig/20221107-113224-root.json
  • 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38242 and previous config saved to /var/cache/conftool/dbconfig/20221107-113122-ladsgroup.json
  • 11:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 11:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38241 and previous config saved to /var/cache/conftool/dbconfig/20221107-112857-ladsgroup.json
  • 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P38240 and previous config saved to /var/cache/conftool/dbconfig/20221107-112702-marostegui.json
  • 11:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38239 and previous config saved to /var/cache/conftool/dbconfig/20221107-112423-ladsgroup.json
  • 11:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38238 and previous config saved to /var/cache/conftool/dbconfig/20221107-112219-marostegui.json
  • 11:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd1002.eqiad.wmnet to plain
  • 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd1002.eqiad.wmnet to plain
  • 11:17 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38237 and previous config saved to /var/cache/conftool/dbconfig/20221107-111719-root.json
  • 11:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38236 and previous config saved to /var/cache/conftool/dbconfig/20221107-111637-marostegui.json
  • 11:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 11:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance
  • 11:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38235 and previous config saved to /var/cache/conftool/dbconfig/20221107-111616-marostegui.json
  • 11:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38234 and previous config saved to /var/cache/conftool/dbconfig/20221107-111615-ladsgroup.json
  • 11:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38233 and previous config saved to /var/cache/conftool/dbconfig/20221107-111351-ladsgroup.json
  • 11:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38232 and previous config saved to /var/cache/conftool/dbconfig/20221107-111156-marostegui.json
  • 11:06 arturo: running homer on cr-eqiad/cr-codfw for https://gerrit.wikimedia.org/r/c/operations/homer/public/+/853374 (T321220, T309407)
  • 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd1002.eqiad.wmnet to drbd
  • 11:04 _joe_: manually started dump_cloud_ip_ranges.service
  • 11:02 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38231 and previous config saved to /var/cache/conftool/dbconfig/20221107-110215-root.json
  • 11:01 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 11:01 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 11:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P38230 and previous config saved to /var/cache/conftool/dbconfig/20221107-110110-marostegui.json
  • 11:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T318955)', diff saved to https://phabricator.wikimedia.org/P38229 and previous config saved to /var/cache/conftool/dbconfig/20221107-110109-ladsgroup.json
  • 10:59 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: apply
  • 10:59 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: apply
  • 10:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38228 and previous config saved to /var/cache/conftool/dbconfig/20221107-105844-ladsgroup.json
  • 10:57 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 10:57 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 10:56 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd1002.eqiad.wmnet to drbd
  • 10:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T318955)', diff saved to https://phabricator.wikimedia.org/P38227 and previous config saved to /var/cache/conftool/dbconfig/20221107-105015-ladsgroup.json
  • 10:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance
  • 10:47 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38226 and previous config saved to /var/cache/conftool/dbconfig/20221107-104710-root.json
  • 10:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P38225 and previous config saved to /var/cache/conftool/dbconfig/20221107-104603-marostegui.json
  • 10:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38224 and previous config saved to /var/cache/conftool/dbconfig/20221107-104338-ladsgroup.json
  • 10:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 10:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 10:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T318955)', diff saved to https://phabricator.wikimedia.org/P38223 and previous config saved to /var/cache/conftool/dbconfig/20221107-104101-ladsgroup.json
  • 10:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38222 and previous config saved to /var/cache/conftool/dbconfig/20221107-103622-ladsgroup.json
  • 10:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 10:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 10:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38221 and previous config saved to /var/cache/conftool/dbconfig/20221107-103549-ladsgroup.json
  • 10:32 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 5398
  • 10:32 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38220 and previous config saved to /var/cache/conftool/dbconfig/20221107-103205-root.json
  • 10:31 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 5398
  • 10:31 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30103
  • 10:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38219 and previous config saved to /var/cache/conftool/dbconfig/20221107-103056-marostegui.json
  • 10:30 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 30103
  • 10:29 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 399338
  • 10:29 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 399338
  • 10:28 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 59796
  • 10:28 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 59796
  • 10:27 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4817
  • 10:26 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 4817
  • 10:26 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 35598
  • 10:26 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 35598
  • 10:26 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30058
  • 10:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38218 and previous config saved to /var/cache/conftool/dbconfig/20221107-102555-ladsgroup.json
  • 10:25 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 30058
  • 10:25 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 46416
  • 10:25 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 46416
  • 10:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38217 and previous config saved to /var/cache/conftool/dbconfig/20221107-102509-marostegui.json
  • 10:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 10:25 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 10:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38216 and previous config saved to /var/cache/conftool/dbconfig/20221107-102458-marostegui.json
  • 10:24 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 3214
  • 10:24 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 3214
  • 10:21 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 17511
  • 10:21 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 17511
  • 10:21 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 12400
  • 10:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38215 and previous config saved to /var/cache/conftool/dbconfig/20221107-102043-ladsgroup.json
  • 10:20 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 12400
  • 10:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 7459
  • 10:19 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 7459
  • 10:17 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 6661
  • 10:17 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 6661
  • 10:17 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 5398
  • 10:17 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'email' for AS: 5398
  • 10:17 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38214 and previous config saved to /var/cache/conftool/dbconfig/20221107-101700-root.json
  • 10:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38213 and previous config saved to /var/cache/conftool/dbconfig/20221107-101140-marostegui.json
  • 10:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38212 and previous config saved to /var/cache/conftool/dbconfig/20221107-101102-marostegui.json
  • 10:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38211 and previous config saved to /var/cache/conftool/dbconfig/20221107-101048-ladsgroup.json
  • 10:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38210 and previous config saved to /var/cache/conftool/dbconfig/20221107-100952-marostegui.json
  • 10:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38209 and previous config saved to /var/cache/conftool/dbconfig/20221107-100536-ladsgroup.json
  • 10:01 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38208 and previous config saved to /var/cache/conftool/dbconfig/20221107-100155-root.json
  • 09:55 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P38207 and previous config saved to /var/cache/conftool/dbconfig/20221107-095556-marostegui.json
  • 09:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T318955)', diff saved to https://phabricator.wikimedia.org/P38206 and previous config saved to /var/cache/conftool/dbconfig/20221107-095542-ladsgroup.json
  • 09:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38205 and previous config saved to /var/cache/conftool/dbconfig/20221107-095445-marostegui.json
  • 09:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T318955)', diff saved to https://phabricator.wikimedia.org/P38204 and previous config saved to /var/cache/conftool/dbconfig/20221107-095149-ladsgroup.json
  • 09:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 09:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance
  • 09:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38203 and previous config saved to /var/cache/conftool/dbconfig/20221107-095030-ladsgroup.json
  • 09:49 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 09:48 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 09:48 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 09:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudmetrics1004.eqiad.wmnet
  • 09:46 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38202 and previous config saved to /var/cache/conftool/dbconfig/20221107-094650-root.json
  • 09:46 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 09:46 cgoubert@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 09:45 cgoubert@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 09:44 cgoubert@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 09:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38201 and previous config saved to /var/cache/conftool/dbconfig/20221107-094315-ladsgroup.json
  • 09:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 09:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 09:42 cgoubert@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 09:42 ladsgroup@deploy1002: Finished scap: Backport for pruneRevData: Make it reload config (duration: 08m 10s)
  • 09:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 09:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 09:41 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 09:40 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 09:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P38200 and previous config saved to /var/cache/conftool/dbconfig/20221107-094050-marostegui.json
  • 09:40 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 09:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 09:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38199 and previous config saved to /var/cache/conftool/dbconfig/20221107-093939-marostegui.json
  • 09:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudmetrics1004.eqiad.wmnet
  • 09:38 elukey: restart rsyslog on ml-serve2001
  • 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudmetrics1003.eqiad.wmnet
  • 09:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38198 and previous config saved to /var/cache/conftool/dbconfig/20221107-093629-ladsgroup.json
  • 09:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 09:34 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for pruneRevData: Make it reload config synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 09:34 ladsgroup@deploy1002: Started scap: Backport for pruneRevData: Make it reload config
  • 09:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38197 and previous config saved to /var/cache/conftool/dbconfig/20221107-093352-marostegui.json
  • 09:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 09:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 5:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 09:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudmetrics1003.eqiad.wmnet
  • 09:29 moritzm: installing Django security updates
  • 09:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38196 and previous config saved to /var/cache/conftool/dbconfig/20221107-092543-marostegui.json
  • 09:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38195 and previous config saved to /var/cache/conftool/dbconfig/20221107-092436-marostegui.json
  • 09:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 09:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 09:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T321123)', diff saved to https://phabricator.wikimedia.org/P38194 and previous config saved to /var/cache/conftool/dbconfig/20221107-092414-marostegui.json
  • 09:18 moritzm: draining ganeti1010 for eventual reimage T311687
  • 09:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P38193 and previous config saved to /var/cache/conftool/dbconfig/20221107-090908-marostegui.json
  • 09:08 Emperor: set thanos ring replicas to 3.30 T311690
  • 09:06 urbanecm@deploy1002: Finished scap: Backport for Set wmgVisualEditorAccessRestbaseDirectly = false for testwiki and dewiki.beta (T320531) (duration: 09m 07s)
  • 09:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 09:03 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 09:03 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 09:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:57 urbanecm@deploy1002: urbanecm and daniel: Backport for Set wmgVisualEditorAccessRestbaseDirectly = false for testwiki and dewiki.beta (T320531) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 08:57 urbanecm@deploy1002: Started scap: Backport for Set wmgVisualEditorAccessRestbaseDirectly = false for testwiki and dewiki.beta (T320531)
  • 08:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P38192 and previous config saved to /var/cache/conftool/dbconfig/20221107-085402-marostegui.json
  • 08:52 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:51 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:51 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:50 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:49 marostegui@deploy1002: Finished scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc3 master" (duration: 04m 22s)
  • 08:45 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "ProductionServices.php: Promote pc2014 to pc3 master" synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 08:45 marostegui@deploy1002: Started scap: Backport for Revert "ProductionServices.php: Promote pc2014 to pc3 master"
  • 08:40 urbanecm: UTC morning B&C window done
  • 08:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T321123)', diff saved to https://phabricator.wikimedia.org/P38191 and previous config saved to /var/cache/conftool/dbconfig/20221107-083855-marostegui.json
  • 08:38 urbanecm@deploy1002: Finished scap: Backport for ContentTranslation: Move haw, ps and xh Wikipedias out of Beta (duration: 06m 56s)
  • 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T321123)', diff saved to https://phabricator.wikimedia.org/P38190 and previous config saved to /var/cache/conftool/dbconfig/20221107-083648-marostegui.json
  • 08:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 08:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321123)', diff saved to https://phabricator.wikimedia.org/P38189 and previous config saved to /var/cache/conftool/dbconfig/20221107-083626-marostegui.json
  • 08:35 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:34 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:33 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:31 urbanecm@deploy1002: urbanecm and kartik: Backport for ContentTranslation: Move haw, ps and xh Wikipedias out of Beta synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
  • 08:31 urbanecm@deploy1002: Started scap: Backport for ContentTranslation: Move haw, ps and xh Wikipedias out of Beta
  • 08:30 urbanecm@deploy1002: Finished scap: Backport for Set ContentTranslation MT threshold to 75 in Japanese WP (T321819) (duration: 06m 24s)
  • 08:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:24 urbanecm@deploy1002: urbanecm and kartik: Backport for Set ContentTranslation MT threshold to 75 in Japanese WP (T321819) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 08:24 urbanecm@deploy1002: Started scap: Backport for Set ContentTranslation MT threshold to 75 in Japanese WP (T321819)
  • 08:23 urbanecm@deploy1002: Finished scap: Backport for Set VisualEditorDefaultParsoidClient for dewiki-beta mad testwiki (T320531) (duration: 18m 54s)
  • 08:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38188 and previous config saved to /var/cache/conftool/dbconfig/20221107-082120-marostegui.json
  • 08:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38185 and previous config saved to /var/cache/conftool/dbconfig/20221107-080613-marostegui.json
  • 08:05 urbanecm@deploy1002: urbanecm and daniel: Backport for Set VisualEditorDefaultParsoidClient for dewiki-beta mad testwiki (T320531) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 08:04 urbanecm@deploy1002: Started scap: Backport for Set VisualEditorDefaultParsoidClient for dewiki-beta mad testwiki (T320531)
  • 08:03 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 08:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 08:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 08:01 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 08:00 marostegui@deploy1002: Finished scap: Backport for ProductionServices.php: Promote pc2014 to pc3 master (duration: 04m 18s)
  • 07:56 marostegui@deploy1002: marostegui and marostegui: Backport for ProductionServices.php: Promote pc2014 to pc3 master synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 07:55 marostegui@deploy1002: Started scap: Backport for ProductionServices.php: Promote pc2014 to pc3 master
  • 07:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 07:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321123)', diff saved to https://phabricator.wikimedia.org/P38184 and previous config saved to /var/cache/conftool/dbconfig/20221107-075106-marostegui.json
  • 07:50 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 07:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 07:50 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T321123)', diff saved to https://phabricator.wikimedia.org/P38183 and previous config saved to /var/cache/conftool/dbconfig/20221107-074959-marostegui.json
  • 07:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 07:49 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 07:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321123)', diff saved to https://phabricator.wikimedia.org/P38182 and previous config saved to /var/cache/conftool/dbconfig/20221107-074938-marostegui.json
  • 07:47 marostegui@deploy1002: Finished scap: Backport for Revert "ProductionServices.php: Promote pc1014 to pc1 master" (duration: 04m 53s)
  • 07:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 07:44 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=frwiki` in a tmux at mwmaint1002 (T318457)
  • 07:42 marostegui@deploy1002: marostegui and marostegui: Backport for Revert "ProductionServices.php: Promote pc1014 to pc1 master" synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
  • 07:42 marostegui@deploy1002: Started scap: Backport for Revert "ProductionServices.php: Promote pc1014 to pc1 master"
  • 07:37 elukey: `elukey@aux-k8s-worker1002:~$ sudo systemctl reset-failed ifup@ens13.service`
  • 07:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38181 and previous config saved to /var/cache/conftool/dbconfig/20221107-073431-marostegui.json
  • 07:31 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 07:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 07:28 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 07:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 07:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 07:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38180 and previous config saved to /var/cache/conftool/dbconfig/20221107-071925-marostegui.json
  • 07:17 marostegui@deploy1002: Finished scap: Backport for ProductionServices.php: Promote pc1014 to pc1 master (T322295) (duration: 04m 29s)
  • 07:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 07:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 07:13 marostegui@deploy1002: marostegui and marostegui: Backport for ProductionServices.php: Promote pc1014 to pc1 master (T322295) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
  • 07:13 marostegui@deploy1002: Started scap: Backport for ProductionServices.php: Promote pc1014 to pc1 master (T322295)
  • 07:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 07:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2011.codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover
  • 07:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on pc2011.codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover
  • 07:05 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=bnwiki` in a tmux at mwmaint1002 (T318457)
  • 07:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321123)', diff saved to https://phabricator.wikimedia.org/P38179 and previous config saved to /var/cache/conftool/dbconfig/20221107-070418-marostegui.json
  • 07:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T321123)', diff saved to https://phabricator.wikimedia.org/P38178 and previous config saved to /var/cache/conftool/dbconfig/20221107-070311-marostegui.json
  • 07:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 07:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 07:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38177 and previous config saved to /var/cache/conftool/dbconfig/20221107-070249-marostegui.json
  • 07:02 urbanecm: Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=cswiki` in a tmux at mwmaint1002 (T318457)
  • 07:01 urbanecm@deploy1002: Finished scap: Backport for Add support for gemm_mentee_is_active (T318457), Calculate mentorship-related metrics (T318684) (duration: 06m 27s)
  • 06:55 urbanecm@deploy1002: urbanecm and urbanecm: Backport for Add support for gemm_mentee_is_active (T318457), Calculate mentorship-related metrics (T318684) synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 06:55 urbanecm@deploy1002: Started scap: Backport for Add support for gemm_mentee_is_active (T318457), Calculate mentorship-related metrics (T318684)
  • 06:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es2024 T322406', diff saved to https://phabricator.wikimedia.org/P38176 and previous config saved to /var/cache/conftool/dbconfig/20221107-065251-root.json
  • 06:50 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es2023 to es5 primary and set section read-write T322406', diff saved to https://phabricator.wikimedia.org/P38175 and previous config saved to /var/cache/conftool/dbconfig/20221107-065048-root.json
  • 06:49 marostegui: Starting es5 codfw failover from es2024 to es2023 - T322406
  • 06:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38174 and previous config saved to /var/cache/conftool/dbconfig/20221107-064743-marostegui.json
  • 06:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322406
  • 06:46 marostegui@cumin1001: dbctl commit (dc=all): 'Set es2023 with weight 0 T322406', diff saved to https://phabricator.wikimedia.org/P38173 and previous config saved to /var/cache/conftool/dbconfig/20221107-064608-root.json
  • 06:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322406
  • 06:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38172 and previous config saved to /var/cache/conftool/dbconfig/20221107-063236-marostegui.json
  • 06:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38171 and previous config saved to /var/cache/conftool/dbconfig/20221107-061730-marostegui.json
  • 06:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38170 and previous config saved to /var/cache/conftool/dbconfig/20221107-061019-marostegui.json
  • 06:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 06:10 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 3292
  • 06:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 06:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 06:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1121.eqiad.wmnet with reason: Maintenance
  • 06:09 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 3292
  • 06:06 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 61461
  • 06:05 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 61461
  • 06:01 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 25091
  • 05:59 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 25091
  • 05:54 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 20115
  • 05:54 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 20115
  • 05:53 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 7843
  • 05:53 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 7843

2022-11-06

  • 08:23 elukey: restart rsyslog on centralog2002
  • 08:19 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
  • 08:19 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
  • 08:17 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
  • 08:17 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
  • 07:50 elukey: restart kube-apiserver on ml-serve-ctrl1001
  • 07:48 elukey: restart kube-apiserver on ml-serve-ctrl1002 - high HTTP 409 registered since days ago

2022-11-05

  • 12:56 mfossati@deploy1002: Finished deploy [airflow-dags/platform_eng@c849762]: (no justification provided) (duration: 00m 49s)
  • 12:55 mfossati@deploy1002: Started deploy [airflow-dags/platform_eng@c849762]: (no justification provided)
  • 09:39 elukey: reinstall kubernetes-node on ml-staging200[12] to allow puppet to run (cleanup after yesterday issue, worker nodes had master role applied)
  • 09:32 elukey: restart kube-apiserver on ml-staging-ctrl2001
  • 09:31 elukey: restart kube-apiserver on ml-staging-ctrl2002

2022-11-04

  • 18:31 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS buster
  • 18:12 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 18:09 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 17:44 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS buster
  • 17:25 fnegri@cumin1001: conftool action : set/pooled=yes; selector: name=dbproxy1019.eqiad.wmnet,service=wikireplicas-a
  • 17:19 fnegri@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host dbproxy1018.eqiad.wmnet
  • 17:08 fnegri@cumin1001: START - Cookbook sre.hosts.reboot-single for host dbproxy1018.eqiad.wmnet
  • 17:06 fnegri@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host dbproxy1018.eqiad.wmnet
  • 17:06 fnegri@cumin1001: START - Cookbook sre.hosts.reboot-single for host dbproxy1018.eqiad.wmnet
  • 17:04 fnegri@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host dbproxy1018.eqiad.wmnet
  • 17:04 fnegri@cumin1001: START - Cookbook sre.hosts.reboot-single for host dbproxy1018.eqiad.wmnet
  • 17:01 mvernon@cumin2002: conftool action : set/weight=40; selector: service=nginx,name=moss-fe2001.codfw.wmnet
  • 17:01 mvernon@cumin2002: conftool action : set/weight=40; selector: service=swift-fe,name=moss-fe2001.codfw.wmnet
  • 17:00 mvernon@cumin2002: conftool action : set/weight=40; selector: service=nginx,name=moss-fe1001.eqiad.wmnet
  • 17:00 mvernon@cumin2002: conftool action : set/weight=40; selector: service=swift-fe,name=moss-fe1001.eqiad.wmnet
  • 16:58 Emperor: rolling restart of swift-proxies to bring moss-fe{1,2}001 into service T322424
  • 16:55 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe2001.codfw.wmnet
  • 16:53 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe1001.eqiad.wmnet
  • 16:48 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host moss-fe2001.codfw.wmnet
  • 16:48 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host moss-fe1001.eqiad.wmnet
  • 16:41 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe2001.codfw.wmnet
  • 16:41 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-fe1001.eqiad.wmnet
  • 16:35 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host moss-fe2001.codfw.wmnet
  • 16:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp4052']
  • 16:34 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host moss-fe1001.eqiad.wmnet
  • 16:34 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 16:33 pt1979@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cp4052']
  • 16:29 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe2001.codfw.wmnet with OS bullseye
  • 16:26 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe1001.eqiad.wmnet with OS bullseye
  • 16:13 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe2001.codfw.wmnet with reason: host reimage
  • 16:11 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: host reimage
  • 16:10 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe2001.codfw.wmnet with reason: host reimage
  • 16:07 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: host reimage
  • 16:06 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 16:06 jhathaway@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 15:57 Emperor: repool ms-fe{1,2}009
  • 15:55 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host moss-fe2001.codfw.wmnet with OS bullseye
  • 15:54 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host moss-fe1001.eqiad.wmnet with OS bullseye
  • 15:48 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 15:43 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:41 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 15:00 elukey: `elukey@cumin1001:~$ sudo cumin 'ms-fe2*' 'systemctl restart swift-proxy' -b 1 -s 20`
  • 14:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 14:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 14:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T318955)', diff saved to https://phabricator.wikimedia.org/P38159 and previous config saved to /var/cache/conftool/dbconfig/20221104-145225-ladsgroup.json
  • 14:52 vgutierrez@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=swift,name=eqiad
  • 14:51 Emperor: restart swift-proxy on ms-fe1012
  • 14:48 elukey: restart swift-proxy on ms-fe1011
  • 14:44 Emperor: restart swift-proxy on ms-fe1010
  • 14:41 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 14:37 vgutierrez@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=swift,name=eqiad
  • 14:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P38158 and previous config saved to /var/cache/conftool/dbconfig/20221104-143718-ladsgroup.json
  • 14:28 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 14:26 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp4052']
  • 14:25 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 14:24 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 14:23 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp4052']
  • 14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P38157 and previous config saved to /var/cache/conftool/dbconfig/20221104-142212-ladsgroup.json
  • 14:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T318955)', diff saved to https://phabricator.wikimedia.org/P38156 and previous config saved to /var/cache/conftool/dbconfig/20221104-140705-ladsgroup.json
  • 14:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1196 (T318955)', diff saved to https://phabricator.wikimedia.org/P38155 and previous config saved to /var/cache/conftool/dbconfig/20221104-140427-ladsgroup.json
  • 14:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 14:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 14:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T318955)', diff saved to https://phabricator.wikimedia.org/P38154 and previous config saved to /var/cache/conftool/dbconfig/20221104-140405-ladsgroup.json
  • 13:58 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED
  • 13:58 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dbprov2004
  • 13:57 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dbprov2004
  • 13:56 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:54 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 13:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P38153 and previous config saved to /var/cache/conftool/dbconfig/20221104-134859-ladsgroup.json
  • 13:45 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp4052']
  • 13:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P38152 and previous config saved to /var/cache/conftool/dbconfig/20221104-133353-ladsgroup.json
  • 13:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T318955)', diff saved to https://phabricator.wikimedia.org/P38151 and previous config saved to /var/cache/conftool/dbconfig/20221104-131846-ladsgroup.json
  • 13:17 sukhe: reprepro -C main include bullseye-wikimedia python-logstash_0.4.6-3_amd64.changes: T321309
  • 13:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1186 (T318955)', diff saved to https://phabricator.wikimedia.org/P38150 and previous config saved to /var/cache/conftool/dbconfig/20221104-131607-ladsgroup.json
  • 13:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 13:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 13:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T318955)', diff saved to https://phabricator.wikimedia.org/P38149 and previous config saved to /var/cache/conftool/dbconfig/20221104-131546-ladsgroup.json
  • 13:11 sukhe: reprepro -C main include bullseye-wikimedia prometheus-rdkafka-exporter_0.3_amd64.changes: T321309
  • 13:10 sukhe: reprepro -C main include bullseye-wikimedia file-read-backwards_2.0.0-3_amd64.changes: T321309
  • 13:09 sukhe: reprepro -C main include bullseye-wikimedia fifo-log-demux_0.6.3_amd64.changes: T321309
  • 13:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P38148 and previous config saved to /var/cache/conftool/dbconfig/20221104-130039-ladsgroup.json
  • 12:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P38147 and previous config saved to /var/cache/conftool/dbconfig/20221104-124533-ladsgroup.json
  • 12:36 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 100%: After reboot', diff saved to https://phabricator.wikimedia.org/P38146 and previous config saved to /var/cache/conftool/dbconfig/20221104-123606-root.json
  • 12:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T318955)', diff saved to https://phabricator.wikimedia.org/P38145 and previous config saved to /var/cache/conftool/dbconfig/20221104-123026-ladsgroup.json
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1184 (T318955)', diff saved to https://phabricator.wikimedia.org/P38144 and previous config saved to /var/cache/conftool/dbconfig/20221104-122747-ladsgroup.json
  • 12:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 12:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T318955)', diff saved to https://phabricator.wikimedia.org/P38143 and previous config saved to /var/cache/conftool/dbconfig/20221104-122726-ladsgroup.json
  • 12:21 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 75%: After reboot', diff saved to https://phabricator.wikimedia.org/P38142 and previous config saved to /var/cache/conftool/dbconfig/20221104-122101-root.json
  • 12:19 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/thumbor: sync
  • 12:18 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/thumbor: sync
  • 12:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T318955)', diff saved to https://phabricator.wikimedia.org/P38141 and previous config saved to /var/cache/conftool/dbconfig/20221104-121848-ladsgroup.json
  • 12:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P38140 and previous config saved to /var/cache/conftool/dbconfig/20221104-121219-ladsgroup.json
  • 12:05 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 50%: After reboot', diff saved to https://phabricator.wikimedia.org/P38139 and previous config saved to /var/cache/conftool/dbconfig/20221104-120556-root.json
  • 12:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P38138 and previous config saved to /var/cache/conftool/dbconfig/20221104-120342-ladsgroup.json
  • 11:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P38137 and previous config saved to /var/cache/conftool/dbconfig/20221104-115713-ladsgroup.json
  • 11:50 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 25%: After reboot', diff saved to https://phabricator.wikimedia.org/P38136 and previous config saved to /var/cache/conftool/dbconfig/20221104-115051-root.json
  • 11:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P38135 and previous config saved to /var/cache/conftool/dbconfig/20221104-114835-ladsgroup.json
  • 11:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T318955)', diff saved to https://phabricator.wikimedia.org/P38134 and previous config saved to /var/cache/conftool/dbconfig/20221104-114207-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1169 (T318955)', diff saved to https://phabricator.wikimedia.org/P38133 and previous config saved to /var/cache/conftool/dbconfig/20221104-113929-ladsgroup.json
  • 11:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 11:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 11:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 11:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T318955)', diff saved to https://phabricator.wikimedia.org/P38132 and previous config saved to /var/cache/conftool/dbconfig/20221104-113725-ladsgroup.json
  • 11:35 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 10%: After reboot', diff saved to https://phabricator.wikimedia.org/P38131 and previous config saved to /var/cache/conftool/dbconfig/20221104-113546-root.json
  • 11:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T318955)', diff saved to https://phabricator.wikimedia.org/P38130 and previous config saved to /var/cache/conftool/dbconfig/20221104-113329-ladsgroup.json
  • 11:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2176 (T318955)', diff saved to https://phabricator.wikimedia.org/P38129 and previous config saved to /var/cache/conftool/dbconfig/20221104-113048-ladsgroup.json
  • 11:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 11:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 11:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T318955)', diff saved to https://phabricator.wikimedia.org/P38128 and previous config saved to /var/cache/conftool/dbconfig/20221104-113027-ladsgroup.json
  • 11:27 elukey: restart kube-apiserver on ml-serve-ctrl2002 - high latencies for LIST (knative resources)
  • 11:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P38127 and previous config saved to /var/cache/conftool/dbconfig/20221104-112218-ladsgroup.json
  • 11:20 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 5%: After reboot', diff saved to https://phabricator.wikimedia.org/P38125 and previous config saved to /var/cache/conftool/dbconfig/20221104-112041-root.json
  • 11:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P38124 and previous config saved to /var/cache/conftool/dbconfig/20221104-111521-ladsgroup.json
  • 11:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P38123 and previous config saved to /var/cache/conftool/dbconfig/20221104-110712-ladsgroup.json
  • 11:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P38122 and previous config saved to /var/cache/conftool/dbconfig/20221104-110014-ladsgroup.json
  • 10:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T318955)', diff saved to https://phabricator.wikimedia.org/P38121 and previous config saved to /var/cache/conftool/dbconfig/20221104-105205-ladsgroup.json
  • 10:50 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 1%: After reboot', diff saved to https://phabricator.wikimedia.org/P38120 and previous config saved to /var/cache/conftool/dbconfig/20221104-105031-root.json
  • 10:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1135 (T318955)', diff saved to https://phabricator.wikimedia.org/P38119 and previous config saved to /var/cache/conftool/dbconfig/20221104-104927-ladsgroup.json
  • 10:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 10:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 10:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T318955)', diff saved to https://phabricator.wikimedia.org/P38117 and previous config saved to /var/cache/conftool/dbconfig/20221104-104508-ladsgroup.json
  • 10:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2174 (T318955)', diff saved to https://phabricator.wikimedia.org/P38116 and previous config saved to /var/cache/conftool/dbconfig/20221104-104227-ladsgroup.json
  • 10:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 10:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 8:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 07:27 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P38115 and previous config saved to /var/cache/conftool/dbconfig/20221104-072722-root.json
  • 07:12 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P38114 and previous config saved to /var/cache/conftool/dbconfig/20221104-071217-root.json
  • 06:57 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P38113 and previous config saved to /var/cache/conftool/dbconfig/20221104-065712-root.json
  • 06:42 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P38112 and previous config saved to /var/cache/conftool/dbconfig/20221104-064207-root.json
  • 06:32 marostegui@cumin1001: dbctl commit (dc=all): 'Give weight to es2021', diff saved to https://phabricator.wikimedia.org/P38111 and previous config saved to /var/cache/conftool/dbconfig/20221104-063250-root.json
  • 06:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es2020 T322389', diff saved to https://phabricator.wikimedia.org/P38110 and previous config saved to /var/cache/conftool/dbconfig/20221104-063224-root.json
  • 06:31 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es2021 to es4 primary and set section read-write T322389', diff saved to https://phabricator.wikimedia.org/P38109 and previous config saved to /var/cache/conftool/dbconfig/20221104-063128-root.json
  • 06:30 marostegui: Starting es4 codfw failover from es2020 to es2021 - T322389
  • 06:27 marostegui@cumin1001: dbctl commit (dc=all): 'Set es2021 with weight 0 T322389', diff saved to https://phabricator.wikimedia.org/P38108 and previous config saved to /var/cache/conftool/dbconfig/20221104-062740-root.json
  • 06:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es4 T322389
  • 06:27 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 10%: After schema change', diff saved to https://phabricator.wikimedia.org/P38107 and previous config saved to /var/cache/conftool/dbconfig/20221104-062702-root.json
  • 06:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es4 T322389
  • 06:11 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 5%: After schema change', diff saved to https://phabricator.wikimedia.org/P38106 and previous config saved to /var/cache/conftool/dbconfig/20221104-061157-root.json
  • 05:56 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 3%: After schema change', diff saved to https://phabricator.wikimedia.org/P38105 and previous config saved to /var/cache/conftool/dbconfig/20221104-055652-root.json
  • 05:41 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 1%: After schema change', diff saved to https://phabricator.wikimedia.org/P38104 and previous config saved to /var/cache/conftool/dbconfig/20221104-054147-root.json
  • 03:24 ejegg: payments-wiki upgraded from 1c8f522f to 8baa6bb5
  • 01:29 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 46s)
  • 01:28 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: (no justification provided)