Incident status

No incidents reported in the last 14 days.

Create a new incident report

1. Deal with the issue by following Incident response/Runbook.
2. Use this form to create a report based on the report template:
   
3. Write up what happened, filling out the template. Use past incident reports for inspiration.

Past incidents

This page tracks incident reports about Wikimedia Foundation properties, such as Wikipedia.

2024

• 2024-02-28 etcd-mirror
• 2024-03-27 Elasticsearch Omega Cluster Failure
• 2024-04-17 mw-on-k8s eqiad outage
• 2024-04-26 Search unavailable for some eqiad users
• 2024-04-28 WMCS Toolforge Redis refusing connections
• 2024-05-28 wikikube-api
• 2024-05-31 Image hotlinking
• 2024-06-10 puppet volatile data broken sync
• 2024-06-11 WMCS Ceph
• 2024-06-12 WMCS toolforge k8s control plane
• 2024-06-21 - 2024-06-24 Outage Incidents - Dumps Generation Issues
• 2024-07-21 s4 and x1 write overload
• 2024-09-16 logstash unavailability
• 2024-09-18 replication
• 2024-09-23 pc1013 hardware issue
• 2024-09-28 cr2-eqsin down
• 2024-10-09 calico-codfw
• 2024-10-24 codfw pc and kafka mirrormaker issues
• 2024-11-25 WMCS proxy nginx failure
• 2024-11-26 WMCS network problems
• 2024-12-03 Port saturation from cached Varnish HEAD-GET upgrades

Incident history

Check Category:Incident documentation for older incident reports.

See also

• https://www.wikimediastatus.net (docs).
• Incident graphs by Krinkle.
• #wikimedia-tech ^connect on Libera Chat