Keyholder

Source code: operations/puppet.git:modules/keyholder operations/software/keyholder.git

Keyholder is a set of scripts that allow a group of users to use an SSH key without sharing the private key with the members of the group. This is accomplished by running a locked-down instance of ssh-agent, and running ssh-agent-proxy in front of it. The proxy allows trusted users to list the ssh-agent's identities and to send the agent signing requests. Access to a given key is restricted to members of a given Unix group and will be rejected when the user is not a member.

Overview

Administration of Keyholder is done via a shell script, keyholder located under /usr/local/sbin which is typically not in a normal user PATH.

 keyholder -- Manage shared SSH agent
 
   keyholder status
     Lists service status and the fingerprints of all identities
     currently represented by the agent
 
   keyholder add KEY
     Add a private key identity to the agent
 
   keyholder arm
     Add all keys in /etc/keyholder.d
 
   keyholder disarm
     Deletes all identities from the agent
 
   keyholder start/stop/restart
     Start / stop / restart the keyholder service

Arming keys

Keys are stored under /srv/private/modules/secret/secrets/keyholder/ in the Puppet#Private puppet repo.

We use Keyholder for SCAP deploys. Keyholder starts automatically, but a user with root has to load both the mwdeploy and deploy-service identities. This is by design. If the agent is unarmed, an Icinga check will issue the following alert:

PROBLEM - Keyholder SSH agent on tin is CRITICAL: Keyholder is not armed. Run `keyholder arm` to arm it.

To arm the agent, log in to deploy1003, and run keyholder arm. The agent will automatically attempt to load the secret deployment key, which is protected with a passphrase. The passphrases for production are stored in Pwstore, for WMCS OpenStack projects they are usually stored on the project-specific puppetmaster.

Production passphrases

Identity	Realm	Location
mwdeploy	production	`pwstore/pw.git/deployment-key-passphrase`
deploy-service	production	`pwstore/pw.git/deployment-key-passphrase`
eventlogging	production	`pwstore/pw.git/deployment-key-passphrase`
deploy-ci-docroot	production	`pwstore/pw.git/deployment-key-passphrase`
phab-deploy	production	`pwstore/pw.git/deployment-key-passphrase`
deploy-debmonitor	production	`pwstore/pw.git/deployment-key-passphrase`
deploy-librenms	production	`pwstore/pw.git/deployment-key-passphrase`
deploy-puppetboard	production	`pwstore/pw.git/deployment-key-passphrase`
dumpsdeploy	production	`pwstore/pw.git/deployment-key-passphrase`
deploy-rancid	production	`pwstore/pw.git/network-monitoring-keys-passphrase`
analytics-deploy	production	`pwstore/pw.git/deployment-key-passphrase`
deploy_airflow	production	`pwstore/pw.git/deployment-key-passphrase`
zuul	production	`pwstore/pw.git/deployment-key-passphrase`
rancid	production	`pwstore/pw.git/network-monitoring-keys-passphrase`
netbox	production	`pwstore/pw.git/deployment-key-passphrase`
cumin_master	production	`pwstore/pw.git/cumin-master-key-passphrase`
authdns_acmechief	production	`pwstore/pw.git/authdns-acmechief-key-passphrase`
metamonitor	production	`pwstore/pw.git/metamonitor-key-passphrase`
deploy-hiddenparma	production	`pwstore/pw.git/deployment-key-passphrase`
homer	production	`pwstore/pw.git/homer-key-passphrase`
deploy-homer	production	`pwstore/pw.git/deployment-key-passphrase`
deploy-ci-docroot	production	`pwstore/pw.git/deployment-key-passphrase`
apache2modsec	production	`pwstore/pw.git/apache2modsec-key-passphrase`
cumin_openstack_master	production for WMCS	`pwstore/pw.git/cumin-openstack-master-key-passphrase`
cloud_cumin_master	production for WMCS	`pwstore/pw.git/cloud-cumin-master-key-passphrase`

WMCS projects passphrases

Identity	Realm	Location
cumin	integration	`integration-puppetmaster01.integration.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/cumin.passphrase`
cumin	beta cluster	`deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud:/srv/git/labs/private/files/ssh/tin/cumin_rsa.passphrase`
mwdeploy	beta cluster	`deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud:/srv/git/labs/private/files/ssh/tin/mwdeploy_rsa.passphrase`
deploy-service	beta cluster	`deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud:/srv/git/labs/private/files/ssh/tin/servicedeploy_rsa.passphrase`
eventlogging	beta cluster	passphrase is: 'eventlogging'
dumpsdeploy	beta cluster	passphrase is (without the quote marks): 'some boring passphrase'
analytics-deploy	beta cluster	`deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud:/srv/git/labs/private/files/ssh/tin/analytics_deploy.passphrase`
scap	beta cluster	`deployment-puppetserver-1:/srv/git/labs/private/files/ssh/tin/scap.passphrase`
wmcs_openstack_instance_puppet_user	cloudinfra	`cloudinfra-internal-puppetmaster-02:/var/lib/git/labs/private/passwords.txt`

Hints

You can verify that the agent is armed by running /usr/local/sbin/keyholder status.

SCAP deployers do not have access to the private key or the passphrase, so in case the deployment server is rebooted, SCAP deployments will be blocked until a root arms the agent.

A deployer can then use the proxy to connect to a host: SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -oIdentitiesOnly=yes -oIdentityFile=/etc/keyholder.d/<somekey> <somehost>

Generating a key for a new identity or update an existing one

Use the following commands on puppetmaster1001 (so it never touches the network or other host before being setup into the private repo):

 puppetmaster1001# cd /root
 puppetmaster1001# ssh-keygen -t ed25519 -b 256 -C "/etc/keyholder.d/<identity>" -f <identity> # check keys look fine
 puppetmaster1001# mv <identity> <identity>.pub /srv/private/modules/secret/secrets/keyholder/
 puppetmaster1001# cd /srv/private/modules/secret/secrets/keyholder/; git add <identity> <identity>.pub; git commit
 
 application_host# run-puppet-agent
 
 deployment_host# run-puppet-agent  # note there are 2 deployment hosts, one for each dc as of this writing
 deployment_host# keyholder restart  # you can just add, I did a restart because I was updating one
 deployment_host# keyholder arm

If using ED25519 keys is not possible because incompatibilities (e.g. network equipment or access to PSUs), it is possible to failback to generate RSA keys:

 puppetmaster1001# ssh-keygen -t rsa -b 4096 -C "/etc/keyholder.d/<identity>" -f <identity>