Talk:Deployments/Emergencies

De-draftifying

Looks good to me, I'd say it's ready to remove the "draft" template --20after4 (talk) 22:07, 19 February 2020 (UTC)

Over to @Thcipriani:, then. :-) Jforrester (talk) 23:25, 19 February 2020 (UTC)

+1 I read through today. I think in its current state this is worth calling a v1.0 of this doc Thcipriani (talk) 23:41, 19 February 2020 (UTC)

Other examples

Other examples of situations that have come up where we've allowed deployments:

• loss of functionality for the majority of users
• "looks broken" i.e., CSS breakage for the majority of users

I might be fine saying that's no longer allowed. If it is still allowed, let's codify it.

-- Thcipriani (talk) 23:41, 19 February 2020 (UTC)

@Thcipriani: How is this change? Jforrester (talk) 20:35, 20 February 2020 (UTC)

LGTM! Thcipriani (talk) 20:53, 20 February 2020 (UTC)

Contact info?

Would it make sense to either give contact info for releng and SRE directly on the page, or, better, link to somewhere where one can find them? I imagine anyone who's looking at this page in an urgent situation is going to be a little stressed and would benefit finding that information easily and unambiguously and quickly. Lars Wirzenius (talk)

Maybe. Partially, this is a "if you have to ask, it's not you" kind of situation — anyone with deployment rights should already have all of that with them as part of their duties, in the same way that implicitly you can ssh into production. Jforrester (talk) 20:30, 20 February 2020 (UTC)

Fine, I’ll come out and say it – I have deployment rights and am not sure how to contact those teams. I’m in the wikimedia-operations and wikimedia-releng channels; more recently I’ve seen wikimedia-sre mentioned a few times – should I be in that one as well? --Lucas Werkmeister (WMDE) (talk) 11:01, 27 February 2020 (UTC)

Even better, there should be a documented way of getting the relevant team's attention. It's a little embarrassing IMO that you can just say !stewards if you want to contact stewards, but there are no equivalents for WMF teams. --tgr (talk) 23:52, 26 February 2020 (UTC)

+1, for example during Incident_documentation/20200225-mediawiki_interface_language there was nobody from releng around on the releng IRC channel to give a green (or red) light to deployment. --Nikerabbit (talk) 08:20, 27 February 2020 (UTC)

+1, I was about to make the same suggestion (that an official mechanism of contact be specified). Especially if there are now two teams being hard blockers (my guess previously would have been to treat those teams as soft blockers: try to contact them, but if they were unresponsive to use best judgment as to whether to go ahead anyway), they should ensure someone is available around the clock. I think SRE already does this on #wikimedia-operations (is that correct?), but if RelEng does similarly I don't know where. Anomie (talk) 14:36, 27 February 2020 (UTC)

On-wiki edits

"Make on-wiki edits rarely, and only when you really have to; each wiki's editing community expects autonomy."

This seems a bit disconnected from the rest of the document. Why would you make on-wiki edits as part of an emergency deployment? I presume this refers to site JS in some way but I have a hard time of thinking of examples. --tgr (talk) 23:55, 26 February 2020 (UTC)

Also things like when I had to emergency-change Commons's POTD, which was an 8MiB GIF which was breaking the page cache for the front page and causing an outage. Jforrester (talk) 00:41, 27 February 2020 (UTC)

Where to IRC, how to know who's who?

"you should check in IRC and get positive confirmation from both Release Engineering and SRE" - should this explicitly mention IRC channel names on Freenode? (And how would people know who is who? Or do we just assume that's somehow common knowledge?) --aklapper (talk) 18:34, 29 February 2020 (UTC)

Got a point here - a lot of people have different account names and/or nicknames than their on-wiki or LDAP usernames; and even different on-wiki names on wikitech vs. the rest of the projects for that matter. Should look into having an IRC/Wiki user index of some sort. DSquirrelGM (talk) 03:51, 1 March 2020 (UTC)