Obsolete:Standalone puppetmaster/PuppetDB

If you got here without reading it yet, please go back to Help:Standalone puppetmaster first and read that. The process described here is only for special cases that need it and for those who are very confident with PuppetDB already. With those warnings, these are the things to know so far.

Why would I want a local PuppetDB to go with my local Puppetmaster?

In general, this would mean you already know you need a local Puppetmaster, are probably already using it and you have discovered that you must also be able to use exported resources

What should I know before I try this?

First, this isn't a foolproof process. Expect frustration. Stick to the basic puppetmaster if you can.

How do I do it?

This order of operations seems to work well: For this documentation i will use the following host names in the examples

• puppetmaster: puppetprimary.sso.eqiad1.wikimedia.cloud
• puppetdb: sso-pdb.sso.eqiad1.wikimedia.cloud

Further i will assign roles using the classes array in hiera however you can also assign theses via horizon

Create a new puppetmaster

Create a node in horizon using Debian Buster and assign it the following role and explicitly set the puppetmaster hiera value set for the new puppet master to the following this ensure it is not accidentally overridden at some other point in hiera e.g. the project puppet settings

classes:
 - role::puppetmaster::standalone
puppetmaster: puppetmaster.cloudinfra.wmflabs.org

.

Do **not** configure this node to use storeconfigs or puppetdb. This is very important, especially if you have an existing standalone puppetmaster that matters to you.

NOTE: Do not attempt to install puppetdb on your puppetmaster at this time. There are conflicts between nginx (installed by the puppetdb role) and apache (installed by puppetmaster) and some odd potential issues around the firewalling that currently get installed as well.

• Copy any local commits from /var/lib/git on your current puppetmaster to the new one.
• Add the following secrets to hiera you can either add theses as local commits to the private repo e.g. /var/lib/git/labs/private/hieradata/cloud/eqiad1/toolsbeta/common.yaml. Or add them to the secrets part of hiera e.g. /etc/puppet/secret/hieradata/$project.yaml

puppetdb::password::replication: ''
puppetdb::password::rw: 'your-very-secret-password'

Create the puppetdb server

Use same distribution as used for the master (buster is recommended at the time of writing). You will need to point this server to the new puppet master and assign it the puppetdb role. We also specify the hostname of the puppet master and add a block for puppetmasters::servers which is required to configure various ACL's.

When setting up puppetdb version 7 there's an issue with getting the locale properly set on the postgres backing db. To prevent that, run

sudo dpkg-reconfigure locales

before applying the puppetdb class. You will be prompted to select a locale; choose en_US.UTF-8, make it the system default, and log out and in again before applying puppet changes.

classes:
 - role::puppetdb  # you can also add this via horizon
profile::puppetdb::master: "%{facts.networking.fqdn}"  # This is essential for the module to work.
puppetmaster: puppetprimary.sso.eqiad1.wikimedia.cloud  # Without this, it will not work.

After setting it up the PuppetDB host as a client of the new puppetmaster as per the Standalone Puppetmaster Guide, rerun puppet until the configurations converge and no more changes are preformed

$ sudo -i puppet agent -t

Once things have converged restart postgresql, Until you do this, the puppetization may have changed the location of the database files, and you may very well not have the DB set up anymore, despite puppet's best efforts.

$ sudo systemctl restart postgresql.service

Unfortunately at this point you probably still do not have a working PuppetDB server yet, but that's sort of expected. Even if you do have a working PuppetDB server you should still continue with the steps to avoid common issues.

If everything went well you should be able to run the following command from the puppetdb server, using the password you configured earlier when promoted

$ psql -h $your_puppetdb_ip -d puppetdb -U puppetdb -W

If it doesn't work, you will need to do consult the docs.

Once this is working restart the puppetdb service

$ sudo systemctl restart puppetdb.service

If the PupppetDB server is now complete and sudo -i puppet agent -t runs without any problems, proceed to the next step. If it has issues, proceeding will break your new puppet setup so continue troubleshooting.

Reconfigure the puppetmaster server

On the puppetmaster you now need to configure storeconfigs to use the new puppetdb server

profile::puppetmaster::common::puppetdb_host:
 - sso-pdb.sso.eqiad1.wikimedia.cloud
profile::puppetmaster::common::storeconfigs: puppetdb
# The default is to use the global puppet master.
# You can omit the following line to keep your puppet master attached to itself
puppetmaster: puppetmaster.cloudinfra.wmflabs.org

After puppet has run and applied theses settings you will need to restart apache on the puppet master to make sure the settings are active

$ sudo systemctl restart apache2.service

Obviously, if you've rigged up this puppetmaster to totally bootstrap itself somehow, you should update the puppetmaster parameter appropriately, but that is now how the author of these notes did it.

Once this configuration is in place run sudo -i puppet agent -t on the new puppetmaster. Run it until it doesn't do anything new. Then run sudo -i puppet agent -t on the puppetdb server to make sure it hasn't broken somehow.

If you get 500s, you may have to start all over :(.

If it is working, before it settles into doing nothing new, you should see puppet add puppetdb server's ssh key to known hosts via exported resources. This is the surest way to know you have everything working.

1. Once everything is working, change the puppetmaster for all your clients, repeating the ssl changes and key signings as needed, ensuring they all work again on the new puppetmaster.
2. Shut down and delete the old puppetmaster, if you had one.

Trouble shooting

/etc/puppetdb not being a link from puppet

If you encounter errors about /etc/puppetdb not being a link from puppet, move that dir aside (sudo mv /etc/puppetdb /etc/puppetdb.bak) and check the latest version available of puppetdb with apt-cache policy puppetdb. At the time of writing it is 6.2.0-3 which is the stock package from debian. If you do not have the stock debian version you should switch to that version

PuppetDB's service cannot run if openjdk-9 is installed

This was an issue present on stretch boxes and no longer observed on buster. however if you bump into this the easiest fix for was sudo apt-get remove openjdk-9-jre-headless

systemctl restart puppetdb.service hangs

Puppetdb is not a fast service to restart, but it should timeout and not hang forever. However if If it does, the java process is likely not connecting to the DB. Test again with the following and generally descend into troubleshooting postgresql.

$ psql -h $your_puppetdb_ip -d puppetdb -U puppetdb -W

postgresql password change on every run

I noticed that when i used a password with using a complex chcracter set postgress reset the password on every run and you see a message like the following

Notice: /Stage[main]/Puppetmaster::Puppetdb::Database/Postgresql::User[puppetdb@localhost]/Exec[pass_set-puppetdb@localhost]/returns: executed successfully (corrective)

to fix this add a new password to puppetdb::password::rw: using a simpler password set e.g. pwgen 20 1

Changing IP address

If you suffer a migration that changes your puppetdb vm's IP address, puppetdb will stop working. To fix this, you must change the IP in your postgresql pg_hba.conf file (to connect to your DB successfully), and you also need to change /etc/default/puppetdb where you'll find the old IP address provided as a java argument. With those updated, reload postgres and start puppetdb!