Puppet-compiler simulates puppet changes and determine the effective difference before/after a given change to puppet files.
This project contains execution hosts for the Jenkins job operations-puppet-catalog-compiler. It computes the effective difference before/after a proposed puppet change for a given set of nodes. Also known as puppet compiler.
Build output is accessible via web service http://puppet-compiler.wmflabs.org/
For documentation on the service, see Puppet Testing.
Who has access?
Results of completed compiler jobs are published for all to see. The UI for launching custom jobs can be operated by people in the 'wmf' ldap group and also probably by members of the 'nda' and 'wmde' groups.
Regular users can also schedule puppet compiler jobs for a patch by specifying hostnames in the git patch description, e.g.:
Phabricator: Fix aphlict to not try and start service if ensure == absent Hosts: phab1003.eqiad.wmnet, phab2001.codfw.wmnet Change-Id: Id899bdc35e203fb620d4bce6b426b2c2b93dd9ff
How to update the compiler's facts? (e.g. INFO:
Unable to find facts for host conf2001.codfw.wmnet, skipping)
You'll need: 1)access to all the puppet master workers (puppetmaster::servers in hieradata) 2) access to the compiler hosts (membership to the project) 3) have ruby and ruby-safe-yaml packages installed 4) a local checkout of operations/puppet git tree. Then launch this script from your local checkout of the puppet repository for each compiler hosts. The list of compiler hosts is available in Jenkis. As of Dec. 20th 2018 there are only two compiler hosts and one is the default for the script, so you just have to run:
# Run this for all the compiler hosts (see https://integration.wikimedia.org/ci/label/puppet-compiler-node/ for the current list of compilers) PUPPET_COMPILER=<fqdn-of-compiler-host> ./modules/puppet_compiler/files/compiler-update-facts # Example bash one-liner to be run from the root of operations/puppet git on a laptop/workstation with root access to the puppet masters COMPILERS="compiler1001.puppet-diffs.eqiad.wmflabs compiler1002.puppet-diffs.eqiad.wmflabs compiler1003.puppet-diffs.eqiad.wmflabs"; for COMPILER in $COMPILERS; do PUPPET_COMPILER="$COMPILER" ./modules/puppet_compiler/files/compiler-update-facts; done
It will cycle through all the puppet master workers and sync the facts from all of them
Only the most recent fact for each host will be kept on the compiler host.
FYI: Jcrespo got a warning after running the above:
/usr/lib/python3/dist-packages/urllib3/connection.py:337: SubjectAltNameWarning: Certificate for puppetdb1001.eqiad.wmnet has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning
How to update the facts for cloud VMs? (e.g. INFO:
Unable to find facts for host util-abogott-stretch.testlabs.eqiad.wmflabs, skipping)
Different Cloud VPS VMs use different puppet masters, and you can specify which master to use with the PUPPET_MASTER environment variable. To refresh the facts for a given VM, first determine which puppet master it uses. For example, refreshing for the default cloud puppetmaster looks like this:
COMPILERS="compiler1001.puppet-diffs.eqiad.wmflabs compiler1002.puppet-diffs.eqiad.wmflabs compiler1003.puppet-diffs.eqiad.wmflabs"; for COMPILER in $COMPILERS; do PUPPET_MASTER=cloud-puppetmaster-03.cloudinfra.eqiad.wmflabs PUPPET_COMPILER="$COMPILER" ./modules/puppet_compiler/files/compiler-update-facts; done
To update facts for Toolforge VMs:
COMPILERS="compiler1001.puppet-diffs.eqiad.wmflabs compiler1002.puppet-diffs.eqiad.wmflabs compiler1003.puppet-diffs.eqiad.wmflabs"; for COMPILER in $COMPILERS; do PUPPET_MASTER=tools-puppetmaster-02.tools.eqiad.wmflabs PUPPET_COMPILER="$COMPILER" ./modules/puppet_compiler/files/compiler-update-facts; done
The puppet compiler stores facts for each puppet master separately. If there are multiple fact files for a given FQDN (for example due to a change in puppetmaster) it will use which ever fact file was most-recently.
How do you run the puppet-compiler locally on a compiler host?
From time to time it may be necessary to attempt compilation from a shell on one of the compilers. In order to do so, first ensure you have shell access to the puppet-diffs openstack instances. Ask a horizon project admin to add you to the project if you are unable to log in. After you've logged in to a compiler host via ssh:
# Become the jenkins-deploy user sudo su - jenkins-deploy # Run the puppet-compiler (optionally you may add --debug to the end of the command for additional debugging output) CHANGE=<gerrit change number> NODES=<comma separated list of fqdns to compile> BUILD_NUMBER=<unique build number> puppet-compiler
Update secrets / labs/private
The compiler uses fake secrets from the public repository
labs/private.git. To update it, on each of the compilers:
sudo -u jenkins-deploy git -C /var/lib/catalog-differ/private pull
Server admin log
- 10:42 arturo: updated facts from the tools project: `PUPPET_MASTER="tools-puppetmaster-02.eqiad.wmflabs" modules/puppet_compiler/files/compiler-update-facts`
- 23:15 mutante: syncing facts from production masters
- 20:40 mutante: added new member razzi
- 11:27 arturo: syncing facts from puppetmaster1001
- 10:42 arturo: updating facts from puppetmaster2001.codfw.wmnet
- 10:39 arturo: updating facts from puppetmaster1001.eqiad.wmnet
- 21:47 andrewbogott: updating facts for production hosts
=== 2... (more)