Incident documentation/20130617-AccountCreation

From Wikitech
Jump to: navigation, search

Summary

Due to code in VisualEditor's A/B testing sorting on new accounts, newly created user accounts were not having their email and password set. Thus, many thousands of new accounts were only partially created.

Reported as bugzilla:49727.

What Happened

Basically, this commit 68845 caused the whole thing. This one 69277 fixed it.

Lessons Learned

Our logging works, but we don't monitor those logs and report spikes/errors.

Action Items

  • Errors and exceptions are currently broadcast to fluorine and vanadium via UDP. Ori has code that parses the stream and generates the Ganglia graphs, but it isn't hooked up to Icinga or any other form of monitoring. We need someone from Ops to pair with Ori to set this up. This is bugzilla:49757. Yes check.svg Done