Help:Project puppetserver/PuppetDB
More information and discussion about changes to this draft on the talk page.
If you got here without reading it yet, please go back to Help:project puppetserver first and read that. The process described here is only for special cases that need it and for those who are very confident with PuppetDB already. With those warnings, these are the things to know so far.
Why would I want a local PuppetDB to go with my local puppetserver?
In general, this would mean you already know you need a local puppetserver, are probably already using it and you have discovered that you must also be able to use exported resources
What should I know before I try this?
First, this isn't a foolproof process. Expect frustration. Stick to the basic puppetserver if you can.
How do I do it?
This order of operations seems to work well: For this documentation i will use the following host names in the examples
- puppetserver: puppetprimary.sso.eqiad1.wikimedia.cloud
- puppetdb: sso-pdb.sso.eqiad1.wikimedia.cloud
Further i will assign roles using the classes array in hiera however you can also assign theses via horizon
Create a new puppetserver
Create a node in horizon using Debian Buster and assign it the following role and explicitly set the puppetserver hiera value set for the new puppetserver to the following this ensure it is not accidentally overridden at some other point in hiera e.g. the project puppet settings
classes:
- role::puppetserver::cloud_vps_project
puppetmaster: puppetmaster.cloudinfra.wmflabs.org
.
Do **not** configure this node to use storeconfigs or puppetdb. This is very important, especially if you have an existing standalone puppetserver that matters to you.
NOTE: Do not attempt to install puppetdb on your puppetserver at this time. This is untested and there are some odd potential issues around the firewalling that currently gets installed.
- Copy any local commits from /var/lib/git on your current puppetserver to the new one.
- Add the following secrets to hiera you can either add theses as local commits to the private repo e.g.
/var/lib/git/labs/private/hieradata/cloud/eqiad1/toolsbeta/common.yaml. Or add them to the secrets part of hiera e.g./etc/puppet/secret/hieradata/$project.yaml
puppetdb::password::replication: ''
puppetdb::password::rw: 'your-very-secret-password'
Create the puppetdb server
Use same distribution as used for the server (bookworm is recommended at the time of writing). You will need to point this server to the new puppetserver and assign it the puppetdb role. We also specify the hostname of the puppetserver and add a block for puppetmasters::servers which is required to configure various ACL's.
When setting up puppetdb version 7 there's an issue with getting the locale properly set on the postgres backing db. To prevent that, run
sudo dpkg-reconfigure locales
before applying the puppetdb class. You will be prompted to select a locale; choose en_US.UTF-8, make it the system default, and log out and in again before applying puppet changes.
classes:
- role::puppetdb # you can also add this via horizon
profile::puppetdb::master: "%{facts.networking.fqdn}" # This is essential for the module to work.
puppetmaster: puppetprimary.sso.eqiad1.wikimedia.cloud # Without this, it will not work.
After setting it up the PuppetDB host as a client of the new puppetserver as per the project puppetserver guide, rerun puppet until the configurations converge and no more changes are preformed
$ sudo -i puppet agent -t
Once things have converged restart postgresql, Until you do this, the puppetization may have changed the location of the database files, and you may very well not have the DB set up anymore, despite puppet's best efforts.
$ sudo systemctl restart postgresql.service
Unfortunately at this point you probably still do not have a working PuppetDB server yet, but that's sort of expected. Even if you do have a working PuppetDB server you should still continue with the steps to avoid common issues.
If everything went well you should be able to run the following command from the puppetdb server, using the password you configured earlier when promoted
$ psql -h $your_puppetdb_ip -d puppetdb -U puppetdb -W
If it doesn't work, you will need to do consult the docs.
Once this is working restart the puppetdb service
$ sudo systemctl restart puppetdb.service
If the PupppetDB server is now complete and sudo -i puppet agent -t runs without any problems, proceed to the next step. If it has issues, proceeding will break your new puppet setup so continue troubleshooting.
Reconfigure the puppetserver
On the puppetserver you now need to configure storeconfigs to use the new puppetdb server
profile::puppetserver::common::puppetdb_host:
- sso-pdb.sso.eqiad1.wikimedia.cloud
profile::puppetserver::common::storeconfigs: puppetdb
# The default is to use the global puppet master.
# You can omit the following line to keep your puppet master attached to itself
puppetmaster: puppetmaster.cloudinfra.wmflabs.org
After puppet has run and applied theses settings you will need to restart the puppetserver to make sure the settings are active
$ sudo systemctl restart puppetserver
Obviously, if you've rigged up this puppetserver to totally bootstrap itself somehow, you should update the puppetserver parameter appropriately, but that is now how the author of these notes did it.
Once this configuration is in place run sudo -i puppet agent -t on the new puppetserver. Run it until it doesn't do anything new.
Then run sudo -i puppet agent -t on the puppetdb server to make sure it hasn't broken somehow.
If you get 500s, you may have to start all over :(.
If it is working, before it settles into doing nothing new, you should see puppet add puppetdb server's ssh key to known hosts via exported resources. This is the surest way to know you have everything working.
- Once everything is working, change the puppetserver for all your clients, repeating the ssl changes and key signings as needed, ensuring they all work again on the new puppetserver.
- Shut down and delete the old puppetserver, if you had one.
Trouble shooting
/etc/puppetdb not being a link from puppet
If you encounter errors about /etc/puppetdb not being a link from puppet, move that dir aside (sudo mv /etc/puppetdb /etc/puppetdb.bak) and check the latest version available of puppetdb with apt-cache policy puppetdb. At the time of writing it is 6.2.0-3 which is the stock package from debian. If you do not have the stock debian version you should switch to that version
PuppetDB's service cannot run if openjdk-9 is installed
This was an issue present on stretch boxes and no longer observed on buster. however if you bump into this the easiest fix for was sudo apt-get remove openjdk-9-jre-headless
systemctl restart puppetdb.service hangs
Puppetdb is not a fast service to restart, but it should timeout and not hang forever. However if If it does, the java process is likely not connecting to the DB. Test again with the following and generally descend into troubleshooting postgresql.
$ psql -h $your_puppetdb_ip -d puppetdb -U puppetdb -W
postgresql password change on every run
I noticed that when i used a password with using a complex chcracter set postgress reset the password on every run and you see a message like the following
Notice: /Stage[main]/Puppetmaster::Puppetdb::Database/Postgresql::User[puppetdb@localhost]/Exec[pass_set-puppetdb@localhost]/returns: executed successfully (corrective)
to fix this add a new password to puppetdb::password::rw: using a simpler password set e.g. pwgen 20 1
Changing IP address
If you suffer a migration that changes your puppetdb vm's IP address, puppetdb will stop working. To fix this, you must change the IP in your postgresql pg_hba.conf file (to connect to your DB successfully), and you also need to change /etc/default/puppetdb where you'll find the old IP address provided as a java argument. With those updated, reload postgres and start puppetdb!