Help:Standalone puppetmaster
This page contains information about standalone Puppetmaster, helps you decide why you would use one, and provides basic trouble shooting information.
About standalone Puppetmaster
A standalone Puppetmaster can be used to deliver either a custom or experimental Puppet configuration to one or more hosts in a Cloud VPS project. All Cloud VPS instances run puppet via cron about every 30 minutes. The default configuration pulls from the shared Cloud VPS puppetmaster which has a pristine checkout of operations/puppet.git
and labs/private.git
. Sometimes you want to have a puppetmaster that you control as a Cloud VPS user. You can use role::puppetmaster::standalone
to accomplish this.
Why would I want a local Puppetmaster?
These are the popular reasons for wanting to use one:
- Testing puppet patches before they have been merged in
operations/puppet.git
- Keep project local secrets in a
labs/private.git
repo
Help! Hiera values from local commits don't seem to apply!
Hiera in Cloud VPS has different settings than in Wikimedia Production puppet, and it can be tricky to get your values into the right place in labs/private.git
or puppet in general. First, check /etc/puppet/hiera.yaml
on your standalone puppetmaster for guidance. You may just need to add things to labs/private/hieradata/labs/<VPS project name>/common.yaml
.
You can verify that a node can "see" your hiera settings with the command sudo puppet lookup --node <fqdn of puppet client> --compile --explain <your::hiera::key>
on the standalone puppetmaster. The compile is needed to set all the right facts for hiera in Cloud VPS, and there will be a lot of warnings you can ignore. The answer will be at the end of the command's output.
How can I use a local Puppetmaster?
Step 1: Setup a standalone puppetmaster
- Create a Debian Buster instance (an m1.small instance is enough for most projects)
- Apply the role
role::puppetmaster::standalone
- Force two consecutive puppet runs on the instance using
sudo -i puppet agent -t
- Generate the server certs:
sudo puppet cert generate $(hostname -f)
- Make sure puppetmaster can restart:
sudo systemctl restart apache2
You will now find a checkout of operations/puppet.git in /var/lib/git/operations/puppet
and a checkout of labs/private.git in /var/lib/git/labs/private
.
A standalone puppetmaster is not its own client by default; to configure this, see below. |
Autosigning
If you are not planning on keeping any secrets in your puppetmaster, you can turn on autosigning. This will automatically accept new clients as they contact the puppetmaster, and save you a step later on when adding clients. You can do that by adding the following to the hiera configuration for the puppetmaster instance:
role::puppetmaster::standalone::autosign: true
If you are keeping secrets, you must not turn this on! If you do, anyone with network access to the puppetmaster can access all your secrets! |
Step 2: Setup a puppet client
This works for any number of instances, including the puppetmaster itself.
- Using the Horizon interface:
- Set the hiera variable
puppetmaster: <your puppetmaster's FQDN>
. (Scroll all the way to the bottom of the Puppet Configuration tab, to the Hiera Config button, and add the variable name/value in YAML format.) Use the fully qualified domain name of the puppetmaster (i.e.hostname.projectname.eqiad1.wikimedia.cloud
which can be found by runninghostname -f
on the puppetmaster host).- This variable can be set either per-instance or project wide.
- Set the hiera variable
- On the client instance:
- Run puppet to apply the modified hiera configuration:
sudo -i run-puppet-agent
- Remove existing SSL certificates created with the prior puppetmaster:
sudo rm -rf /var/lib/puppet/ssl
(NOTE: you will seecertificate verify failed: [self signed certificate in certificate chain ...]
when you run puppet if you don't do this. - Only if the client is the puppetmaster itself, run also:
sudo rm /var/lib/puppet/server/ssl/ca/signed/$(hostname -f).pem sudo mkdir -p /var/lib/puppet/ssl/private_keys sudo cp -v /var/lib/puppet/{server/,}ssl/private_keys/$(hostname -f).pem sudo mkdir /var/lib/puppet/ssl/certs/ sudo cp -v /var/lib/puppet/{server/,}ssl/certs/$(hostname -f).pem
- Force a puppet run to generate a new local SSL certificate:
sudo -i run-puppet-agent
- Run puppet to apply the modified hiera configuration:
- On the puppetmaster (If you enabled autosigning in the master, you can skip this step):
- Check that the fingerprint of the client's key matches the one shown when it was generated running:
sudo -i puppet cert list
- Sign the key of the client:
sudo -i puppet cert sign <client-fqdn>
.
- Check that the fingerprint of the client's key matches the one shown when it was generated running:
- On the client:
- Run puppet twice to check that is connecting correctly to it's new master and that the second run does not produce any new configuration changes:
sudo -i run-puppet-agent
- Run puppet twice to check that is connecting correctly to it's new master and that the second run does not produce any new configuration changes:
Troubleshooting
Forgetting to run puppet agent
with sudo
Failing to run puppet agent
as root can result in the surprising error:
Debug: Creating new connection for https://puppet:8140 Error: Could not request certificate: getaddrinfo: Name or service not known Exiting; failed to retrieve certificate and waitforcert is disabled
Advanced use cases
Using puppetdb
WARNING: This is kind of experimental at this time. Proceed with caution.
Puppetdb can be enabled on standalone puppetmasters on Cloud VPS by designating a puppetdb server with the role::puppetmaster::puppetdb
role and a lot of hiera values on both the DB server and the puppetmaster. This requires significant effort and is not a recommended configuration unless you have significant experience with puppet, need it and are able to maintain the setup, including your own postgresql database. It is not likely to succeed unless puppetmaster runs on Debian Stretch/Buster (you may have problems if your puppetmaster OS and puppetdb OS do not match due to PuppetDB termini packaging), and it is guaranteed to break your puppet setup if you just enable it without following a particular order. Notes on that can be found on the standalone puppetdb notes page.
Git push from workstation to puppetmaster
Typically development is done on non-Cloud VPS machines, this applies to Puppet as well, so a common use case is to develop locally and then push to your Cloud VPS project's puppetmaster for testing.
Push using multiple branches
If you want your Cloud VPS instance to act as yet another remote Git repository, you have to configure Git to use sudo
as well. To do that, in your local computer's git clone of the Puppet repository
$ git remote add my-instance my-instance.my-project.eqiad1.wikimedia.cloud:/var/lib/git/operations/puppet
$ git config remote.my-instance.receivepack "sudo git-receive-pack"
$ git config remote.my-instance.uploadpack "sudo git-upload-pack"
After that, you can push and pull branches from and to your Cloud VPS instance. If you pull commits from your Cloud VPS instance, remember to reset authorship information with git commit --amend --reset-author
before pushing to Gerrit.
You cannot push to the branch that is currently checked out. To avoid having to log into your Cloud VPS puppet master and checking out branches, etc., you can create a post-receive
hook by copying:
to /var/lib/git/operations/puppet/.git/hooks/post-receive
and making it executable (chmod +x /var/lib/git/operations/puppet/.git/hooks/post-receive
).
With this hook in place, when you push commits to a branch with a name that starts with development/
:
- if a "local hack" (a commit that is in the
production
branch, but not inorigin/production
) with the sameChange-Id
already exists, it will replace that commit with the one in your pushed branch, or - if no local hack with the same
Change-Id
exists, it will apply it on top of other local hacks.
If you delete a branch (git push -f my-instance :development/branch
), the local hacks in that branch are removed.
You can use the script git-puppet-test
to combine pushing your local development branch to the Cloud VPS puppet master with running Puppet on several hosts and executing a test script:
This is especially useful if you work on a patch over a longer period of time, or the tests that you need to run are complex or boring. The shell command to run before and after pushing a branch and the hosts to run Puppet on are specified by the entries puppet-test.$changeid.pre-command
, puppet-test.$changeid.post-command
and puppet-test.$changeid.hosts
in .git/config
with $changeid
being the Change-Id
of the commit at HEAD
. You can set those also by git puppet-test --set-post-command $command
, etc.
When a patch is merged that you previously pushed as a local hack, if it is identical, i. e. the changes are the same, it will just "disappear" from the list of local hacks. If it has been amended, the automatic pull with git-sync-upstream
will fail, and you will have to log in and remove the local hack with git rebase -i origin/production production
and deleting the line of the local hack in question. You need to apply the same steps if you want to abandon a local hack that you no longer want to work on.
NOTE: If the puppet master is shared among more than one person, pushing will overwrite or mess in other ways with each other's changes!
Push using a single branch
In case a puppetmaster is used to test a single branch/patchset at a time, a simple solution is to to push to an intermediate bare repo and then update the non-bare authoritative repo.
The git hook added in 159720 achieves this. So assuming you have a role::puppetmaster::standalone
configured, you can setup a local bare repo e.g. in your Cloud VPS home (as your user):
cd ~
[ -d ~/puppet.git ] || git clone --bare --no-hardlinks --branch production /var/lib/git/operations/puppet/ ~/puppet.git
install -m755 /var/lib/git/operations/puppet/modules/puppetmaster/files/self-master-post-receive ~/puppet.git/hooks/post-receive
and on the development host add the relevant remote:
git remote add project_puppetmaster ssh://PUPPET_MASTER_HOSTNAME/~/puppet.git
then to test a puppet change in your Cloud VPS puppet master from your local host you can force-push to that remote in the "production" branch, e.g.
git push -f project_puppetmaster HEAD:production
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia Movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect, the bridged Telegram group, or the bridged Mattermost channel
- Discuss via email after you subscribed to the cloud@ mailing list
See also
- Testing catalog with puppet-compiler