From Wikitech
Jump to navigation Jump to search

Keyholder is a set of scripts that allow a group of users to use an SSH key without sharing the private key with the members of the group. This is accomplished by running a locked-down instance of ssh-agent, and running ssh-agent-proxy in front of it. The proxy allows trusted users to list the ssh-agent's identities and to send the agent signing requests. Access to a given key is restricted to members of a given Unix group and will be rejected when the user is not a member.


Administration of Keyholder is done via a shell script, keyholder located under /usr/local/sbin which is typically not in a normal user PATH.

 keyholder -- Manage shared SSH agent
   keyholder status
     Lists service status and the fingerprints of all identities
     currently represented by the agent
   keyholder add KEY
     Add a private key identity to the agent
   keyholder arm
     Add all keys in /etc/keyholder.d
   keyholder disarm
     Deletes all identities from the agent
   keyholder start/stop/restart
     Start / stop / restart the keyholder service

Arming keys

Keys are stored under /srv/private/modules/secret/secrets/keyholder/ in the Puppet#Private puppet repo.

We use Keyholder for SCAP deploys. Keyholder starts automatically, but a user with root has to load both the mwdeploy and deploy-service identities. This is by design. If the agent is unarmed, an Icinga check will issue the following alert:

PROBLEM - Keyholder SSH agent on tin is CRITICAL: Keyholder is not armed. Run `keyholder arm` to arm it.

To arm the agent, log in to deploy1002, and run keyholder arm. The agent will automatically attempt to load the secret deployment key, which is protected with a passphrase. The passphrases for production are stored in Pwstore, for WMCS OpenStack projects they are usually stored on the project-specific puppetmaster.

Production passphrases

Identity Realm Location
mwdeploy production pwstore/pw.git/deployment-key-passphrase
deploy-service production pwstore/pw.git/deployment-key-passphrase
eventlogging production pwstore/pw.git/deployment-key-passphrase
deploy-ci-docroot production pwstore/pw.git/deployment-key-passphrase
phab-deploy production pwstore/pw.git/deployment-key-passphrase
deploy-debmonitor production pwstore/pw.git/deployment-key-passphrase
deploy-librenms production pwstore/pw.git/deployment-key-passphrase
deploy-puppetboard production pwstore/pw.git/deployment-key-passphrase
dumpsdeploy production pwstore/pw.git/deployment-key-passphrase
deploy-rancid production pwstore/pw.git/rancid-key-passphrase
analytics-deploy production pwstore/pw.git/deployment-key-passphrase
deploy_airflow production pwstore/pw.git/deployment-key-passphrase
zuul production pwstore/pw.git/deployment-key-passphrase
rancid production pwstore/pw.git/network-monitoring-keys-passphrase
netbox production pwstore/pw.git/deployment-key-passphrase
cumin_master production pwstore/pw.git/cumin-master-key-passphrase
authdns_acmechief production pwstore/pw.git/authdns-acmechief-key-passphrase
metamonitor production pwstore/pw.git/metamonitor-key-passphrase
homer production pwstore/pw.git/homer-key-passphrase
deploy-homer production pwstore/pw.git/deployment-key-passphrase
deploy-ci-docroot production pwstore/pw.git/deployment-key-passphrase
apache2modsec production pwstore/pw.git/apache2modsec-key-passphrase
cumin_openstack_master production for WMCS pwstore/pw.git/cumin-openstack-master-key-passphrase
cloud_cumin_master production for WMCS pwstore/pw.git/cloud-cumin-master-key-passphrase

WMCS projects passphrases

Identity Realm Location
cumin integration integration-puppetmaster01.integration.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/cumin.passphrase
cumin beta cluster deployment-puppetmaster04.deployment-prep.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
mwdeploy beta cluster deployment-puppetmaster04.deployment-prep.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/tin/mwdeploy_rsa.passphrase
deploy-service beta cluster deployment-puppetmaster04.deployment-prep.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/tin/servicedeploy_rsa.passphrase
eventlogging beta cluster passphrase is: 'eventlogging'
dumpsdeploy beta cluster passphrase is (without the quote marks): 'some boring passphrase'
analytics-deploy beta cluster deployment-puppetmaster04.deployment-prep.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/tin/analytics_deploy.passphrase
scap beta cluster deployment-puppetmaster04:/var/lib/git/labs/private/files/ssh/tin/scap.passphrase
wmcs_openstack_instance_puppet_user cloudinfra cloudinfra-internal-puppetmaster-02:/var/lib/git/labs/private/passwords.txt


You can verify that the agent is armed by running /usr/local/sbin/keyholder status.

SCAP deployers do not have access to the private key or the passphrase, so in case the deployment server is rebooted, SCAP deployments will be blocked until a root arms the agent.

A deployer can then use the proxy to connect to a host: SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -oIdentitiesOnly=yes -oIdentityFile=/etc/keyholder.d/<somekey> <somehost>

Generating a key for a new identity or update an existing one

Use the following commands on puppetmaster1001 (so it never touches the network or other host before being setup into the private repo):

 puppetmaster1001# cd /root
 puppetmaster1001# ssh-keygen -t ed25519 -b 256 -C "/etc/keyholder.d/<identity>" -f <identity> # check keys look fine
 puppetmaster1001# mv <identity> <identity>.pub /srv/private/modules/secret/secrets/keyholder/
 puppetmaster1001# cd /srv/private/modules/secret/secrets/keyholder/; git add <identity> <identity>.pub; git commit
 application_host# run-puppet-agent
 deployment_host# run-puppet-agent  # note there are 2 deployment hosts, one for each dc as of this writing
 deployment_host# keyholder restart  # you can just add, I did a restart because I was updating one
 deployment_host# keyholder arm

If using ED25519 keys is not possible because incompatibilities (e.g. network equipment or access to PSUs), it is possible to failback to generate RSA keys:

 puppetmaster1001# ssh-keygen -t rsa -b 4096 -C "/etc/keyholder.d/<identity>" -f <identity>

See also