Incident documentation
Jump to navigation
Jump to search
This page tracks postmortems of past site outages.
- How to create a new post-mortem
- Deal with the issue (See also Incident response)
- Write up what happened on Wikitech (this wiki, using the form below)
- Send to the ops@ mailing list
- File tasks on Phabricator for any actionable conclusions.
Post-mortem review
Post-mortem reviews are conducted on a case-by-case basis. Reviews will be initiated and conducted by the Release Engineering team.
Review triggers include:
- Holding the Train (see: Deployments/Holding the train)
Review meeting minutes are available as a sub-page of the incident documentation.
Incident reports
2019
- 0110-WDQS
- 0115-PDU-fuses-blown
- 0117-RESTBase-Parsoid
- 0122-maps
- 0208-logstash-mediawiki
- 0211-varnish
- 0213-PAWS
- 0213-cloudvps
- 0214-labsdb1005
- 0220-irc-outage
- 0228-eventbus extension and logstash
- 0301-proton
- 0308-maps
- 0313-MW API App Servers and EventGate
- 0320-ORES
- 0321-acmechief
- 0327-elasticsearch
- 0327-gerrit
- 0328-commons
- 0402-0401KafkaJumbo
- 0407-Zotero
- 0413-elasticsearch
- 0415-maps
- 0416-varnish
- 0417-Jobqueue
- 0423-varnish
- 0425-prometheus
- 0503-varnish
- 0506-zuul
- 0507-opcache
- 0509-puppet
2018
- 0110-swift
- 0111-LegacyEncoding
- 0111-LegacyEncoding/minutes
- 0129-MediaWiki
- 0206-Phabricator
- 0214-labvirt1008-failure
- 0220-thumbor
- 0222-wdqs
- 0226-WikibaseQualityConstraints
- 0228-puppetmaster
- 0229-Train-1.31.0-wmf.27
- 0302-Train
- 0306-MasterMergedInWMFBranch
- 0312-Cache-text
- 0314-ORES
- 0404-mathoid
- 0410-Deleting a page on enwiki
- 0410-Routing
- 0423-wdqs
- 0522-MediaWiki
- 0524-wikidata
- 0606-mx
- 0615-phabricator-vandalism
- 0625-wdqs
- 0626-LoadBalancers
- 0628-dumps
- 0704-mathoid
- 0705-Eventlogging-in-Hive
- 0710-MediaWiki
- 0711-fundraising undeliverable mail
- 0711-kafka-eqiad
- 0717-Train
- 0724-Train
- 0728-eventlogging
- 0731-Phabricator
- 0808-Network
- 0821-Train
- 0918-train
- 1001-DeleteLocalPasswords
- 1002-maps
- 1015-eqsin-power
- 1016-Replication gap on s8 eqiad hosts
- 1016-eqiad parsercache empty post-switchover
- 1024-WDQS
- 1025-train-1.33.0-wmf.1
- 1104-s8-replication
- 1106-maps
- 1107-TwoColConflict
- 1129-ores
- 1204-Toolforge-Kubernetes
- 1212-Train-1.33.0-wmf.8
- 1212-icinga-down
- 1218-train-1.33.0-wmf.9
2017
- 0104-MonologSpi
- 0106-Cache-upload
- 0111-multiversion
- 0118-Labs
- 0119-Labstore
- 0124-WikibaseClient-InterwikiSorting
- 0126-API Slowdown
- 0201-Phabricator
- 0222-www-portals
- 0223-AQS
- 0306-maps
- 0308-Labstore
- 0320-Special AllPages
- 0322-AuthDNS
- 0323-wikibase
- 0419-ContentTranslation
- 0419-ToolsProxy
- 0419-restbase
- 0426-ORES
- 0427-archiva
- 0427-ocg
- 0427-redis-ipaddress
- 0503-missing index
- 0503-x1
- 0507-labs-dns
- 0512-API
- 0525-MediaWiki
- 0601-Maps
- 0607-WikiScrape
- 0613-ORES
- 0616-PHAB
- 0617-Phabricator-spam
- 0623-ORES
- 0629-Labstore Kernel Upgrade
- 0629-ganeti
- 0711-EventLogging
- 0718-JobQueue
- 0719-ldap
- 0720-novaadmin LDAP password
- 0721-Train-Wikidata
- 0728-s5 (WikiData and dewiki) read-only
- 0814-Train
- 0829-EventStreams
- 0831-Zookeeper
- 0917-MediaWiki-API
- 0920-Elasticsearch
- 0920-Logstash
- 0922-REST-API-alert-pageimage-update
- 1001-labvirt1015
- 1017-webm-audio-mediatype
- 1018-wdqs
- 1116-s5-dewiki-wikidata
- 1120-Ext:ORES
- 1120-Ext:ORES/minutes
- 1125-statsd
- 1128-ORES
- 1130-wdqs
2016
- 0112-20160111-toollabs-SGE
- 0123-SessionManagerRolloutFailure
- 0125-EventLogging
- 0126-WikimediaDomainRedirection
- 0128-MediaWiki-API
- 0202-deployment-server-loss
- 0204-Phabricator
- 0209-PlannedMaintenance
- 0212-AllWikisOutage
- 0212-LabsSudoVulnerability
- 0214-labsdb1002
- 0319-Ores
- 0407-Mediawiki
- 0503-Wikidata-Query-Service
- 0505-ChangeProp RESTBase Parsoid
- 0509-CirrusSearch
- 0512-LabsLdapOutage
- 0525-Toollabs Webservices
- 0531-RESTBase
- 0531-wikilabels
- 0601-CentralNotice
- 0601-MediaWiki
- 0606-otrsmail
- 0608-SCB
- 0608-gallium-disk-failure
- 0610-ORES
- 0620-ores
- 0623-etherpad
- 0628-wdqs
- 0705-commons-replication
- 0706-CI-Outage
- 0707-Echo
- 0712-EchoCentralAuth
- 0712-EchoCentralAuth/Retrospective
- 0713-ContentTranslation
- 0713-MediaWiki
- 0721-LocalRenameUserJob
- 0801-ORES
- 0805-LabsSecurityGroups
- 0807-CI
- 0809-MediaWiki
- 0810-CI
- 0810-MediaWiki
- 0818-Elasticsearch
- 0823-ToolsProxy
- 0829-ToolLabs-Outage
- 0831-ulsfo
- 0907-Android
- 0915-MediaWiki
- 0918-Wikitech2fa
- 0924-ORES
- 0925-ores
- 1013-GlobalSign
- 1021-Maps
- 1112-OurMine
- 1127-API
- 1202-20161201-PageImages
- 1204-labservices1001
- 1221-ToollabsKubernetesAPI
- 1227-ores
2015
- 0103-Parsoid
- 0106-Zuul
- 0107-EventLogging
- 0205-EventLogging
- 0205-SiteOutage
- 0206-EventLogging
- 0217-LabsOutage
- 0224-LabsOutage
- 0227-LabsOutage
- 0320-EventLogging
- 0324-Eqiad-Rack-C4
- 0331-LabsNFS-Filesystem-Switch
- 0331-LabsNFS-Overload
- 0401-LabsNFS-Overload
- 0403-API
- 0406-EventLogging
- 0406-Flow
- 0408-s2-SiteStatsInit
- 0409-EventLogging
- 0413-LabsNFS
- 0422-LabsOutage
- 0423-Commons
- 0424-Parsoid
- 0428-EventLogging
- 0429-Parsoid
- 0503-JobQueueRedis
- 0506-EventLogging
- 0515-LabsOutage
- 0518-LabsOutage
- 0519-LabsOutage
- 0519-Mailman
- 0519-RESTBase
- 0527-Cookie
- 0527-GridEngine
- 0529-Tagging
- 0602-gridengine-dns-failure
- 0605-redis
- 0607-LabsNFS-Outage
- 0613-Esams-Text-80
- 0615-Elasticsearch
- 0616-Elasticsearch
- 0617-LabsNFSOutage
- 0617-LabsNFSOutage/Updates
- 0617-LabsNFSOutage/Updates/details
- 0622-EventLogging
- 0625-LabsNFSOutage
- 0702-RESTBase
- 0709-poolcounter
- 0729-HttpsRedirects
- 0803-Kafka
- 0806-poolcounter
- 0810-Kafka
- 0812-LabsOutage
- 0814-MediaWiki
- 0817-ToolLabs-WebgridOutage
- 0820-OCSP
- 0820-PageCategorization (Privacy)
- 0825-Redis
- 0830-labs-nfs
- 0901-Elasticsearch
- 0908-ores
- 0914-extension1
- 0925-LabsOutage
- 0930-enwiki-readonly
- 1003-Parsoid
- 1005-MediaWiki
- 1014-MediaWiki
- 1015-EventLogging
- 1022-EventLogging
- 1023-graphite-grafana
- 1024-LabsNFS-Lag
- 1026-MediaWiki
- 1105-LabsOutage
- 1118-LVS-PyBal
- 1123-LabsOutage
- 1126-graphite-grafana
- 1127-EventLogging
- 1127-PartialNetworkOutage
- 1203-SiliconOutage
- 1214-Math-and-Mathoid
- 1215-swift-syslog
- 1216-Labs-NFS
- 1216-ores
- 1217-cxserver
- 1218-payments wiki
- 1229-SiliconOutage
2014
- 0113-Poolcounter
- 0121-BitsApplicationServers
- 0203-LVS
- 0206-Math
- 0211-Parsoid
- 0228-Cirrus
- 0313-API-Parsoid
- 0313-Deploy
- 0318-EventLogging
- 0328-DB-Queries
- 0403-Deploy
- 0503-Thumbnails
- 0509-EventLogging
- 0517-bits
- 0526-m1
- 0529-appservers
- 0607-Elasticsearch
- 0608-Kafka
- 0612-Math
- 0613-Videoscalers
- 0616-LuceneSearch
- 0618-Wikitech
- 0619-parsercache
- 0619-parsercache/IRC log
- 0622-es1006
- 0622-imagescaler
- 0625-CirrusSearch
- 0714-Etherpad
- 0714-Lists
- 0715 CirrusSearch
- 0716-DNS
- 0730-Icinga
- 0806-Upload
- 0820-SwiftMemcache
- 0905-BitsVarnishMobileApp
- 0910-BetaCluster
- 0910-swift-syslog
- 0919-s1
- 1027-CirrusSearch
- 1106-Labstore
- 1113-EventLogging
- 1114-EventLogging
- 1118-EventLogging
- 1118-m2
- 1122-m2
- 1125-EventLogging
- 1126-oauth
- 1126-ocg
- 1130-Eqiad-Rack-C4
