Nova Resource:Puppet-diffs

From Wikitech
Jump to navigation Jump to search

Project Name puppet-diffs



Puppet-compiler simulates puppet changes and determine the effective difference before/after a given change to puppet files.

This project contains execution hosts for the Jenkins job operations-puppet-catalog-compiler. It computes the effective difference before/after a proposed puppet change for a given set of nodes. Also known as puppet compiler.

Build output is accessible via web service

For documentation on the service, see Puppet Testing.


Who has access?

Results of completed compiler jobs are published for all to see. The UI for launching custom jobs can be operated by people in the 'wmf' ldap group and also probably by members of the 'nda' and 'wmde' groups.

Regular users can also schedule puppet compiler jobs for a patch by specifying hostnames in the git patch description, e.g.:

Phabricator: Fix aphlict to not try and start service if ensure == absent

Hosts: phab1003.eqiad.wmnet, phab2001.codfw.wmnet

Change-Id: Id899bdc35e203fb620d4bce6b426b2c2b93dd9ff

How to update the compiler's facts? (e.g. INFO: Unable to find facts for host conf2001.codfw.wmnet, skipping)

You'll need: 1)access to all the puppet master workers (puppetmaster::servers in hieradata) 2) access to the compiler hosts (membership to the project) 3) have ruby and ruby-safe-yaml packages installed 4) a local checkout of operations/puppet git tree. Then launch this script from your local checkout of the puppet repository for each compiler hosts. The list of compiler hosts is available in Jenkis. As of Dec. 20th 2018 there are only two compiler hosts and one is the default for the script, so you just have to run:

# Run this for all the compiler hosts (see for the current list of compilers)
PUPPET_COMPILER=<fqdn-of-compiler-host> ./modules/puppet_compiler/files/compiler-update-facts

# Example bash one-liner to be run from the root of operations/puppet git on a laptop/workstation with root access to the puppet masters
COMPILERS="compiler1001.puppet-diffs.eqiad.wmflabs compiler1002.puppet-diffs.eqiad.wmflabs"; for COMPILER in $COMPILERS; do PUPPET_COMPILER="$COMPILER" ./modules/puppet_compiler/files/compiler-update-facts; done

It will cycle through all the puppet master workers and sync the facts from all of them

Only the most recent fact for each host will be kept on the compiler host.

FYI: Jcrespo got a warning after running the above:

/usr/lib/python3/dist-packages/urllib3/ SubjectAltNameWarning: Certificate for puppetdb1001.eqiad.wmnet has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See for details.)


How to update the facts for cloud VMs? (e.g. INFO: Unable to find facts for host util-abogott-stretch.testlabs.eqiad.wmflabs, skipping)

Different Cloud VPS VMs use different puppet masters, and you can specify which master to use with the PUPPET_MASTER environment variable. To refresh the facts for a given VM, first determine which puppet master it uses. For example, refreshing for the default cloud puppetmaster looks like this:

COMPILERS="compiler1001.puppet-diffs.eqiad.wmflabs compiler1002.puppet-diffs.eqiad.wmflabs"; for COMPILER in $COMPILERS; do PUPPET_MASTER=cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs PUPPET_COMPILER="$COMPILER" ./modules/puppet_compiler/files/compiler-update-facts; done

To update facts for Toolforge VMs:

COMPILERS="compiler1001.puppet-diffs.eqiad.wmflabs compiler1002.puppet-diffs.eqiad.wmflabs"; for COMPILER in $COMPILERS; do PUPPET_COMPILER="$COMPILER" ./modules/puppet_compiler/files/compiler-update-facts; done

The puppet compiler stores facts for each puppet master separately. If there are multiple fact files for a given FQDN (for example due to a change in puppetmaster) it will use which ever fact file was most-recently.

How do you run the puppet-compiler locally on a compiler host?

From time to time it may be necessary to attempt compilation from a shell on one of the compilers. In order to do so, first ensure you have shell access to the puppet-diffs openstack instances. Ask a horizon project admin to add you to the project if you are unable to log in. After you've logged in to a compiler host via ssh:

# Become the jenkins-deploy user
sudo su - jenkins-deploy

# Run the puppet-compiler (optionally you may add --debug to the end of the command for additional debugging output)
CHANGE=<gerrit change number> NODES=<comma separated list of fqdns to compile> BUILD_NUMBER=<unique build number> puppet-compiler
Edit documentation

Server admin log


  • 20:14 andrewbogott: moved project-wide hiera settings into a 'compiler' prefix so I can test the project without implicit hiera settings for T228056


  • 22:18 andrewbogott: upgrade puppet-compiler version to 0.5.0 (via hiera setting on Horizon) for T219430


  • 03:14 andrewbogott: migrating VMs to eqiad1-r, then hopefully figuring out how to make them work in Jenkins again