writing to kafka for mediawiki).
** These should be causing alerts in #wikimedia-operations
* Job runner are having issues
** Review JobQueue Job dashboard...
80 KB (10,416 words) - 14:29, 2 April 2024
one of the backport windows on this page. You must be online in #wikimedia-operations on IRC during your deployment and install [[WikimediaDebug]] ahead...
34 KB (1,144 words) - 17:52, 18 April 2024
updates on jessie
* 09:37 moritzm: added jbond as chanserv ops for #wikimedia-operations
* 08:08 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime...
849 KB (119,958 words) - 18:25, 5 July 2020
that you've verified that the new release works great, announce to #wikimedia-operations your intention to upgrade a prod node. For example:
<code>
!log...
26 KB (3,696 words) - 22:53, 29 February 2024
-s</kbd>
* Be present on IRC. <tt>#wikimedia-tech</tt> and <tt>#wikimedia-operations</tt> are two places where people will come to yell at you if something...
47 KB (6,884 words) - 21:54, 11 April 2024
your deploy. Ideally, stay online and in IRC channels like <code>#wikimedia-operations</code> for an hour or two.
If you must go offline, let people...
12 KB (1,591 words) - 15:13, 14 April 2023
main include bullseye-wikimedia $PWD/*.changes</code>
#In <code>#wikimedia-operations</code> on irc: <code>!log uploaded orchestrator $VERSION packages...
11 KB (1,438 words) - 10:26, 21 March 2024
actions to SAL, please remember to announce to #wikimedia-sre and to #wikimedia-operations too, that you will be running this test. Unless something goes really...
36 KB (3,911 words) - 13:22, 1 April 2024
WMF_BUILD_DIR/packagename_version_amd64.changes
2. Log your uploads in #wikimedia-operations on IRC using <code>!log</code>
=== Upload to a component directory...
7 KB (973 words) - 09:40, 22 January 2024
scap ===
* Tell the <code>#wikimedia-analytics</code> and <code>#wikimedia-operations</code> IRC channels that you are deploying (use <code>!log</code>...
24 KB (3,454 words) - 20:24, 13 October 2023
can, respond even if you’re not at your desk.
* '''Speak up''' in #wikimedia-operations to say you got the page and you’re looking at it. Read up in that...
9 KB (1,253 words) - 18:58, 7 September 2023
duties is required between weeks. The topic in IRC channel <code>#wikimedia-operations</code>, section ''SRE Clinic Duty:'' is updated automatically and...
12 KB (1,825 words) - 01:03, 3 April 2024
default). Do the diffs looks reasonable?
* Keep a close eye on <code>#wikimedia-operations</code>, if someone spots a problem, they're likely to raise the...
7 KB (843 words) - 19:32, 31 January 2024
that a member of the SRE team has received the message.
* <code>#wikimedia-operations</code> IRC channel (in an emergency, consider using the hotword...
13 KB (1,728 words) - 19:00, 2 June 2023
[[mw:MediaWiki on IRC|IRC]]: <tt>volans</tt> (on Libera.Chat in the <tt>#wikimedia-operations</tt> channel)
}}
{{user info
| full name = Chris Danis
| image name...
5 KB (682 words) - 10:07, 26 January 2024
"Victory! ORES deploy looks good" (or something equally effusive) in #wikimedia-operations.
=== In case of a production accident ===
The ORES extension has...
17 KB (2,592 words) - 11:07, 17 November 2023
is happening; (!log it in #wikimedia-releng and announce it in #wikimedia-operations a few minutes beforehand).
# Monitor https://integration.wikimedia...
15 KB (2,063 words) - 19:52, 22 October 2023
intermittently unable to edit the SAL. Every !log command from #wikimedia-operations is included here for completeness.
All times in UTC.
=== May...
29 KB (4,388 words) - 15:14, 19 August 2022
alerts on traffic availability) plus multiple staff/user reports in #wikimedia-operations.
== Timeline ==
'''All times in UTC.'''
* 18:24: HTTP 503 error...
3 KB (327 words) - 17:47, 8 April 2022
(graph)].
=== Detection ===
The first Icinga CRITICAL alert in #wikimedia-operations was at 16:03 (for "phpfpm_up reduced availability"). The first page...
10 KB (1,135 words) - 17:48, 8 April 2022
