Incidents/2017-07-20 novaadmin LDAP password

Summary

While on a hangout and trying to reset Coren's LDAP password, Bryan instead changed the password of the uid=novaadmin,ou=people,dc=wikimedia,dc=org account. This broke things like Wikitech, OpenStack Keystone, and Striker that use the novaadmin account to do privileged things in LDAP. The bad password stayed in place long enough that when Bryan tried to change it back LDAP had locked the account and would not let it be changed by the user directly. Andrew was able to use the uid=admin account and credentials to change uid=novadmin's password back to the expected value.

Timeline

• 2017-07-20T01:16:53Z ldappasswd -v -D 'uid=novaadmin,ou=people,dc=wikimedia,dc=org' -W -s "uid=marc,ou=people,dc=wikimedia,dc=org"
  • "Hmm... that's not right, it didn't prompt me for a new password"
  • ldappasswd -v -D 'uid=novaadmin,ou=people,dc=wikimedia,dc=org' -W -S "uid=marc,ou=people,dc=wikimedia,dc=org"
  • "Huh. It says that auth failed when I try to change your password Coren."
  • ldapsearch -v -D 'uid=novaadmin,ou=people,dc=wikimedia,dc=org' -W uid=marc
  • "Now I can't even do a search? I must have the wrong password for novadamin."
• 2017-07-20T01:13 < bd808> keystone may be down all together
• 2017-07-20T01:14 < bd808> https://tools.wmflabs.org/openstack-browser/ is not working too
• 2017-07-20T01:15 < bd808> [REACTED]. I think I broke novaadmin's LDAP password
• Bryan tries and fails to set the password back after realizing it is set to Coren's dn due to the in initial typo of -s instead of -S.
• 2017-07-20T01:28:11Z Andrew resets password for uid=novaadmin back to expected value.

Conclusions

• ldappasswd has horrible CLI arguments.
• We should have better tools for regular LDAP maintenance operations. There has been discussion of building a web portal for managing LDAP accounts too.

Actionables

• …