Jump to content

Nova Resource:Puppet-diffs

From Wikitech

Project Name puppet-diffs



Puppet-compiler simulates puppet changes and determine the effective difference before/after a given change to puppet files.

This project contains execution hosts for the Jenkins job operations-puppet-catalog-compiler. It computes the effective difference before/after a proposed puppet change for a given set of nodes. Also known as puppet compiler.

Build output is accessible via web service http://puppet-compiler.wmflabs.org/

For documentation on the service, see Puppet Testing.


Who has access?

Results of completed compiler jobs are published for all to see. The UI for launching custom jobs can be operated by people in the 'wmf' ldap group and also probably by members of the 'nda' and 'wmde' groups.

Regular users can also schedule puppet compiler jobs for a patch by specifying hostnames in the git patch description, e.g.:

Phabricator: Fix aphlict to not try and start service if ensure == absent

Hosts: phab1003.eqiad.wmnet, phab2001.codfw.wmnet

Change-Id: Id899bdc35e203fb620d4bce6b426b2c2b93dd9ff

How to update the compiler's facts? (e.g. INFO: Unable to find facts for host conf2001.codfw.wmnet, skipping)

See: https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Updating_nodes

How do you run the puppet-compiler locally on a compiler host?

From time to time it may be necessary to attempt compilation from a shell on one of the compilers. In order to do so, first ensure you have shell access to the puppet-diffs openstack instances. Ask a horizon project admin to add you to the project if you are unable to log in. After you've logged in to a compiler host via ssh:

# Become the jenkins-deploy user
sudo su - jenkins-deploy

# Run the puppet-compiler (optionally you may add --debug to the end of the command for additional debugging output)
CHANGE=<gerrit change number> NODES=<comma separated list of fqdns to compile> BUILD_NUMBER=<unique build number> puppet-compiler

Update secrets / labs/private

The compiler uses fake secrets from the public repository labs/private.git. To update it, on each of the compilers: sudo -u jenkins-deploy git -C /var/lib/catalog-differ/private pull

Out of disk space

Sometimes, especially when multiple users compiled a change on all hosts (" / leaving the host form field empty), the compiler VMs can run out of disk space. Then nobody can compile until some old data is deleted. Find the right compiler instance name from the error message and then, for example:

ssh pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud
cd /srv/jenkins/puppet-compiler/output
du -hs *  
identify largest builds, for example "du -hs * | grep G" to find only those that are over 1G. A common run uses maybe 50MB, a run on * uses 5GB!
on https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ find the matching number and click "delete build" in the web UI
rm -rf <directory name> in the file system

Server admin log


  • 17:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T368669)
  • 16:59 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase (T368669)


  • 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0)
  • 11:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs


  • 18:43 bd808: Added dduvall as reader+member so he can help with instance maintenance chores