Open main menu


Keyholder is a set of scripts that allow a group of users to use an SSH key without sharing the private key with the members of the group. This is accomplished by running a locked-down instance of ssh-agent, and running ssh-agent-proxy in front of it. The proxy allows trusted users to list the ssh-agent's identities and to send the agent signing requests.

Administration of Keyholder is done via a shell script, keyholder:

 keyholder -- Manage shared SSH agent
   keyholder status
     Lists service status and the fingerprints of all identities
     currently represented by the agent
   keyholder add KEY
     Add a private key identity to the agent
   keyholder arm
     Add all keys in /etc/keyholder.d
   keyholder disarm
     Deletes all identities from the agent
   keyholder start/stop/restart
     Start / stop / restart the keyholder service

We use Keyholder for SCAP deploys. Keyholder starts automatically, but a user with root has to load both the mwdeploy and deploy-service identities. This is by design. If the agent is unarmed, an Icinga check will issue the following alert:

PROBLEM - Keyholder SSH agent on tin is CRITICAL: Keyholder is not armed. Run `keyholder arm` to arm it.

To arm the agent, log in to tin, and run keyholder arm. The agent will automatically attempt to load the secret deployment key, which is protected with a passphrase. The passphrase is stored in:

Identity Realm Location
mwdeploy production pwstore/pw.git/deployment-key-passphrase
deploy-service production pwstore/pw.git/deployment-key-passphrase
eventlogging production pwstore/pw.git/deployment-key-passphrase
phab-deploy production pwstore/pw.git/deployment-key-passphrase
deploy-debmonitor production pwstore/pw.git/deployment-key-passphrase
deploy-librenms production pwstore/pw.git/deployment-key-passphrase
deploy-puppetboard production pwstore/pw.git/deployment-key-passphrase
dumpsdeploy production pwstore/pw.git/deployment-key-passphrase
deploy-rancid production pwstore/pw.git/rancid-key-passphrase
analytics-deploy production pwstore/pw.git/deployment-key-passphrase
rancid production pwstore/pw.git/network-monitoring-keys-passphrase
netbox production pwstore/pw.git/network-monitoring-keys-passphrase
cumin_master production pwstore/pw.git/cumin-master-key-passphrase
authdns_certcentral production pwstore/pw.git/authdns-certcentral-key-passphrase
cumin_openstack_master production for WMCS pwstore/pw.git/cumin-openstack-master-key-passphrase
cumin integration integration-puppetmaster01.integration.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/cumin.passphrase
cumin beta cluster deployment-puppetmaster.deployment-prep.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
mwdeploy beta cluster deployment-puppetmaster.deployment-prep.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/tin/mwdeploy_rsa.passphrase
deploy-service beta cluster deployment-puppetmaster.deployment-prep.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/tin/servicedeploy_rsa.passphrase
eventlogging beta cluster passphrase is: 'eventlogging'
dumpsdeploy beta cluster passphrase is (without the quote marks): 'some boring passphrase'
analytics-deploy beta cluster passphrase is: 'analytics'

You can verify that the agent is armed by running keyholder status.

SCAP deployers do not have access to the private key or the passphrase, so in case tin is rebooted, SCAP deployments will be blocked until a root arms the agent.

A deployer can then use the proxy to connect to a host: SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l mwdeploy@somehost

See also