Help:Standalone puppetmaster

From Wikitech
Jump to: navigation, search

A standalone puppetmaster can be used to deliver custom or experimental Puppet configuration to one or more hosts in a Labs project. All Labs instances run puppet via cron about every 30 minutes. The default configuration pulls from the labs puppetmaster which has a pristine checkout of operations/puppet.git and labs/private.git. Sometimes you want to have a puppetmaster that you control as a labs user. You can use role::puppetmaster::standalone to accomplish this.

Why would I want a local Puppetmaster?

These are the popular reasons for wanting to use one:

  1. Testing puppet patches before they have been merged in ops/puppet.git
  2. Keep project local secrets in a labs/private.git repo

How can I use a local Puppetmaster?

Step 1: Setup a standalone puppetmaster

  1. Create a Debian Jessie instance (a small instance is enough for most projects)
  2. Apply the role role::puppetmaster::standalone
  3. Force two consecutive puppet runs on the instance using sudo -i puppet agent -t

You will now find a checkout of operations/puppet.git in /var/lib/git/operations/puppet and a checkout of labs/private.git in /var/lib/git/labs/private.


If you are not planning on keeping any secrets in your puppetmaster, you can turn on autosigning. This will automatically accept new clients as they contact the puppetmaster, and save you a step later on when adding clients. You can do that by adding the following to the hiera configuration for the puppetmaster instance:

role::puppetmaster::standalone::autosign: true

If you are keeping secrets, you must not turn this on! If you do, anyone with network access to the puppetmaster can access all your secrets!

Step 2: Setup a puppet client

This works for any number of instances, including the puppetmaster itself.

  1. Using the Horizon interface:
    1. Set the hiera variable puppetmaster: <your puppetmaster's FQDN>. Use the fully qualified domain name of the puppetmaster (i.e. hostname.projectname.eqiad.wmflabs which can be found by running hostname -f on the puppetmaster host)
      • This variable can be set either per-instance or project wide.
  2. On the client instance:
    1. Run puppet to apply the modified hiera configuration: sudo -i puppet agent --test --verbose
    2. Remove existing SSL certificates created with the prior puppetmaster: sudo rm -rf /var/lib/puppet/ssl
    3. Only if the client is the puppetmaster itself, run also: sudo rm /var/lib/puppet/server/ssl/ca/signed/$(hostname -f).pem
    4. Force a puppet run to generate a new local SSL certificate: sudo -i puppet agent --test --verbose
  3. On the puppetmaster (If you enabled autosigning in the master, you can skip this step):
    1. Check that the fingerprint of the client's key matches the one shown when it was generated running: sudo -i puppet cert list
    2. Sign the key of the client: sudo -i puppet cert sign <client-fqdn>.
  4. On the client:
    1. Run puppet twice to check that is connecting correctly to it's new master and that the second run does not produce any new configuration changes: sudo -i puppet agent --test --verbose


Forgetting to run puppet agent with sudo

Failing to run puppet agent as root can result in the surprising error:

Debug: Creating new connection for https://puppet:8140
Error: Could not request certificate: getaddrinfo: Name or service not known
Exiting; failed to retrieve certificate and waitforcert is disabled

Advanced use cases

Git push from workstation to puppetmaster

Typically development is done on non-Labs machines, this applies to Puppet as well, so a common use case is to develop locally and then push to your Labs project's puppetmaster for testing.

Push using multiple branches

If you want your Labs instance to act as yet another remote Git repository, you have to configure Git to use sudo as well. To do that, in your local computer's git clone of the Puppet repository

$ git remote add my-labs-instance
$ git config "sudo git-receive-pack"
$ git config "sudo git-upload-pack"

After that, you can push and pull branches from and to your Labs instance. If you pull commits from your Labs instance, remember to reset authorship information with git commit --amend --reset-author before pushing to Gerrit.

You cannot push to the branch that is currently checked out. To avoid having to log into your Labs puppet master and checking out branches, etc., you can create a post-receive hook by copying:

to /var/lib/git/operations/puppet/.git/hooks/post-receive and making it executable (chmod +x /var/lib/git/operations/puppet/.git/hooks/post-receive).

With this hook in place, when you push commits to a branch with a name that starts with development/:

  • if a "local hack" (a commit that is in the production branch, but not in origin/production) with the same Change-Id already exists, it will replace that commit with the one in your pushed branch, or
  • if no local hack with the same Change-Id exists, it will apply it on top of other local hacks.

If you delete a branch (git push -f my-labs-instance :development/branch), the local hacks in that branch are removed.

You can use the script git-puppet-test to combine pushing your local development branch to the Labs puppet master with running Puppet on several hosts and executing a test script:

This is especially useful if you work on a patch over a longer period of time, or the tests that you need to run are complex or boring. The shell command to run before and after pushing a branch and the hosts to run Puppet on are specified by the entries puppet-test.$changeid.pre-command, puppet-test.$ and puppet-test.$changeid.hosts in .git/config with $changeid being the Change-Id of the commit at HEAD. You can set those also by git puppet-test --set-post-command $command, etc.

When a patch is merged that you previously pushed as a local hack, if it is identical, i. e. the changes are the same, it will just "disappear" from the list of local hacks. If it has been amended, the automatic pull with git-sync-upstream will fail, and you will have to log in and remove the local hack with git rebase -i origin/production production and deleting the line of the local hack in question. You need to apply the same steps if you want to abandon a local hack that you no longer want to work on.

NOTE: If the puppet master is shared among more than one person, pushing will overwrite or mess in other ways with each other's changes!

Push using a single branch

In case a puppetmaster is used to test a single branch/patchset at a time, a simple solution is to to push to an intermediate bare repo and then update the non-bare authoritative repo.

The git hook added in 159720 achieves this. So assuming you have a role::puppetmaster::standalone configured, you can setup a local bare repo e.g. in your labs home (as your user):

$ cd ~
$ [ -d puppet.git/.git ] || git clone --bare --no-hardlinks /var/lib/git/operations/puppet/ puppet.git
$ install -m755 /var/lib/git/operations/puppet/modules/puppet/files/self-master-post-receive puppet.git/hooks/post-receive

and on the development host add the relevant remote:

$ git remote add labs_project_puppetmaster ssh://PUPPET_MASTER_HOSTNAME/~/puppet.git

then to test a puppet change in your labs puppet master from your local host you can force-push to that remote in the "production" branch, e.g.

$ git push -f labs_project_puppetmaster HEAD:production