Incident status

• CloudVPS vms losing network connectivity

• Kafka-jumbo mirror-makers

• Elasticsearch unavailable
• mw-page-content-change-enrich

No incidents reported in the last 14 days.

Create a new incident report

1. Deal with the issue by following Incident response/Runbook.
2. Use this form to create a report based on the report template:
   
3. Write up what happened, filling out the template. Use past incident reports for inspiration.

Past incidents

This page tracks incident reports about Wikimedia Foundation properties, such as Wikipedia.

2023

• 2023-01-09 API errors - db1143
• 2023-01-10 eqsin network outage
• 2023-01-14 asw-b2-codfw failure
• 2023-01-17 MediaWiki
• 2023-01-17 asw-b2-codfw failure redux
• 2023-01-24 sessionstore quorum issues
• 2023-01-30 kartotherian
• 2023-02-07 mediawiki.page-undelete event stream
• 2023-02-11 logstash latency
• 2023-02-22 read only
• 2023-02-22 wiki outage
• 2023-02-23 PHP worker threads exhaustion
• 2023-02-28 GitLab data loss
• 2023-03-09 mailman
• 2023-04-17 eqiad/LVS
• 2023-05-02 Phabricator outage during GitLab maintenance
• 2023-05-05 prometheus down in ulsfo and eqsin
• 2023-05-05 wdqs not updating in codfw
• 2023-05-19 videoscaler/jobrunner
• 2023-05-23 wdqs CODFW 5xx errors
• 2023-05-25 eqiad/LVS
• 2023-05-28 wikireplicas lag
• 2023-05-30 Unintentional +2 on a config patch without deployment
• 2023-06-06 dawiktionary/dawiki/svwiktionary partial outage
• 2023-06-18 search broken on wikidata and commons
• 2023-08-09 mw-on-k8s outage due to wrong tls cert
• 2023-08-30 hadoop-yarn
• 2023-09-20 Elasticsearch unavailable
• 2023-09-20 mw-page-content-change-enrich
• 2023-09-27 Kafka-jumbo mirror-makers
• 2023-09-29 CloudVPS vms losing network connectivity

Incident history

Check Category:Incident documentation for older incident reports.

See also

• https://www.wikimediastatus.net (docs).
• Incident graphs by Krinkle.
• #wikimedia-tech ^connect on Libera Chat