Help:Standalone puppetmaster with puppetdb notes

From Wikitech
Jump to navigation Jump to search


If you got here without reading it yet, please go back to Help:Standalone puppetmaster first and read that. The process described here is only for special cases that need it and for those who are very confident with PuppetDB already. With those warnings, these are the things to know so far.

Why would I want a local PuppetDB to go with my local Puppetmaster?

In general, this would mean you already know you need a local Puppetmaster, are probably already using it and you have discovered that you must also be able to use exported resources

What should I know before I try this?

First, this isn't a foolproof process. Expect frustration. Stick to the basic puppetmaster if you can.

How do I do it?

This order of operations seems to work well:

  1. Create a new puppetmaster running Debian Stretch, using the role::puppetmaster::standalone role in Horizon. You will probably do best to have its puppetmaster hiera value set to labs-puppetmaster.wikimedia.org. Do **not** set this up to use "storeconfigs" or puppetdb. This is very helpful, especially if you have an existing standalone puppetmaster that matters to you. NOTE: Do not attempt to install puppetdb on your puppetmaster at this time. There are conflicts between nginx (installed by the puppetdb role) and apache (installed by puppetmaster) and some odd potential issues around the firewalling that currently get installed as well.
  2. Copy any local commits from /var/lib/git on your current puppetmaster to the new one.
  3. Add two values to your /var/lib/git/labs/private/hieradata set for puppetdb (changing /var/lib/git/labs/private/hieradata/common.yaml to include these values works:
    puppetdb::password::replication: ''
    puppetdb::password::rw: 'your-very-secret-password'
    
    Make sure you make a local commit after you add them in case you ever want to rebase this again.
  4. Create the puppetdb server (again on stretch), pointed at the new puppetmaster using the role role::puppetmaster::puppetdb with the hiera values you need already in place. The hiera values are:
    1. profile::puppetdb::database::shared_buffers: 768MB The default is for a huge server, you need a smaller-than huge number on VPS.
    2. profile::puppetdb::master: $this_servers_fqdn This is essential for the module to work.
    3. puppetdb_major_version: 4 This helps things along.
    4. puppet_major_version: 4 This also helps.
    5. puppetmaster: $puppetmaster_you_just_made Without this, it will not work.
    6. You need a little block to set up the firewall right. In this example below, the new puppetmaster is named toolsbeta-puppetmaster-02.toolsbeta.eqiad.wmflabs. Obviously, this module is intended for a more complex set of puppetmasters. If you are running such a set, I encourage you to figure out how this should correctly be laid out.
      puppetmaster::servers:
        toolsbeta-puppetmaster-02.toolsbeta.eqiad.wmflabs:
        - worker: toolsbeta-puppetmaster-02.toolsbeta.eqiad.wmflabs
      
  5. On the PuppetDB server (after setting it up as a client of the new server per this, rerun sudo -i puppet agent -t until most of the bleeding stops. You probably do not have a working PuppetDB yet, but that's sort of expected. Also, the following sub-steps are probably needed to stop some of the errors. If you do all this, don't be so sure until after the next few steps.
    1. In this phase if you encounter errors about /etc/puppetdb not being a link from puppet, move that dir aside (sudo mv /etc/puppetdb /etc/puppetdb.bak) and check the latest version available of puppetdb with apt-cache policy puppetdb. At this time it is 4.4.0-1~wmf1. If it were that version for you, run sudo apt-get install puppetdb=4.4.0-1~wmf1, which will allow puppet runs to work a lot better.
    2. PuppetDB's service cannot run if openjdk-9 is installed (or you haven't fiddled with alternatives to suppress it. The easiest fix for me was sudo apt-get remove openjdk-9-jre-headless
  6. Restart postgresql. Until you do this, the puppetization may have changed the location of the database files, and you may very well not have the DB set up anymore, despite puppet's best efforts.
  7. If you don't need to do further DB setup, you should be able to psql -h $your_puppetdb_ip -d puppetdb -U puppetdb -W followed by whatever your password was in step 3 when prompted on the puppetdb server. If it doesn't work, you need to do the steps here.
  8. Now, run sudo systemctl restart puppetdb.service. It should actually work this time. It's not a fast service, but it shouldn't hang forever and timeout. If it does, the java process is likely not connecting to the DB. Test with psql -h $your_puppetdb_ip -d puppetdb -U puppetdb -W again, and generally descend into troubleshooting postgresql.
  9. If the PupppetDB server can now finish a sudo -i puppet agent -t run without any problems, proceed to the next step. If it has issues, proceeding will break your new puppet setup so continue troubleshooting.
  10. Remove the role::puppetmaster::standalone role from your new puppetmaster and re-add it, adding the parameters storeconfigs: puppetdb and puppetdb_host: $your_puppetdb_server to the role. It won't work if you just add them as hiera values at the end. That said, also add these hiera values at the bottom of the page on Horizon:
    profile::puppetmaster::common::puppetdb_host: $your_puppetdb_server
    profile::puppetmaster::common::storeconfigs: puppetdb
    puppet_major_version: 4
    puppetdb_major_version: 4
    puppetmaster: labs-puppetmaster.wikimedia.org
    
    Obviously, if you've rigged up this puppetmaster to totally bootstrap itself somehow, it can call itself the puppetmaster there, but that is now how the author of these notes did it.
  11. Try sudo -i puppet agent -t on the new puppetmaster. Run it until it doesn't do anything new.
  12. Try sudo -i puppet agent -t on the puppetdb server to make sure it isn't now broken. If you get 500s, you may have to start all over. If it is working, before it settles into doing nothing new, it should add your puppetdb server's ssh key to known hosts via exported resources. This is the surest way to know you have everything working.
  13. Once everything is working, change the puppetmaster for all your clients, repeating the ssl changes and key signings as needed, ensuring they all work again on the new puppetmaster.
  14. Shut down and delete the old puppetmaster, if you had one.