Incident documentation/20140608-Kafka

From Wikitech
Jump to: navigation, search

Summary

A single disk on one of the two analytics Kafka brokers failed. The second broker took over as leader for all partitions. Since we currently send more data to Kafka than a single broker can handle (bits, upload, text), messages were dropped.

Timeline

At 2014-06-07T23:37:04, kafka on analytics1021 shut down with a storage exception due to 'Read-only file system'. syslog at this time shows

 Jun  7 23:37:04 analytics1021 kernel: [10635272.839586] lost page write due to I/O error on sdf1

Sean was the first to respond and notice. The disk was not fixable remotely, so he disabled puppet and shut down Kafka. As far as I know, no pages for this were sent. Otto or Gage couldn't have fixed the problem remotely anyway, but others should not have been left wondering how to react.

Monday morning on June 9th, Otto checked email and learned of the problem. Analytics team then met and deliberated on how to proceed.

It has been known that a single broker was not enough to handle the data we are currently sending through it. Ever since we added text back in May, a single broker has not been enough. There are plans in the work to provision more Kafka brokers, but these have not yet come through. We plan to use some of the existing Hadoop DataNodes as additional Kafka brokers, and to order more new replacement DataNodes. We had planned on waiting for these new DataNodes to come in before repurposing existent DataNodes as brokers, but this past weekend's downtime has pushed that schedule forward.

We plan to take DataNodes from the existing Hadoop cluster ASAP to provision as new Kafka Brokers. This will give us some time to do more failover and load testing with more brokers and more traffic, before we start relying on this service more heavily.

Actionables