Nova Resource:Puppet-diffs
| Project Name | puppet-diffs |
|---|---|
| Details, admins/members |
openstack-browser |
| Monitoring |
Puppet-diffs
Description
Puppet-compiler simulates puppet changes and determine the effective difference before/after a given change to puppet files.
This project contains execution hosts for the Jenkins job operations-puppet-catalog-compiler. It computes the effective difference before/after a proposed puppet change for a given set of nodes. Also known as puppet compiler.
Build output is accessible via web service http://puppet-compiler.wmflabs.org/
For documentation on the service, see Puppet Testing.
FAQ
Who has access?
Results of completed compiler jobs are published for all to see. The UI for launching custom jobs can be operated by people in the 'wmf' ldap group and also probably by members of the 'nda' and 'wmde' groups.
Regular users can also schedule puppet compiler jobs for a patch by specifying hostnames in the git patch description, e.g.:
Phabricator: Fix aphlict to not try and start service if ensure == absent Hosts: phab1003.eqiad.wmnet, phab2001.codfw.wmnet Change-Id: Id899bdc35e203fb620d4bce6b426b2c2b93dd9ff
How to update the compiler's facts? (e.g. INFO: Unable to find facts for host conf2001.codfw.wmnet, skipping)
See: https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Updating_nodes
How do you run the puppet-compiler locally on a compiler host?
From time to time it may be necessary to attempt compilation from a shell on one of the compilers. In order to do so, first ensure you have shell access to the puppet-diffs openstack instances. Ask a horizon project admin to add you to the project if you are unable to log in. After you've logged in to a compiler host via ssh:
# Become the jenkins-deploy user
sudo su - jenkins-deploy
# Run the puppet-compiler (optionally you may add --debug to the end of the command for additional debugging output)
CHANGE=<gerrit change number> NODES=<comma separated list of fqdns to compile> BUILD_NUMBER=<unique build number> puppet-compiler
Update secrets / labs/private
The compiler uses fake secrets from the public repository labs/private.git. To update it, on each of the compilers:
sudo -u jenkins-deploy git -C /var/lib/catalog-differ/private pull
Out of disk space
Sometimes, especially when multiple users compiled a change on all hosts (" / leaving the host form field empty), the compiler VMs can run out of disk space. Then nobody can compile until some old data is deleted. Find the right compiler instance name from the error message and then, for example:
ssh pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud cd /srv/jenkins/puppet-compiler/output du -hs * identify largest builds, for example "du -hs * | grep G" to find only those that are over 1G. A common run uses maybe 50MB, a run on * uses 5GB! on https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ find the matching number and click "delete build" in the web UI rm -rf <directory name> in the file system
Server admin log
2024-07-02
- 17:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T368669)
- 16:59 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase (T368669)
2024-06-20
- 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0)
- 11:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs
2023-11-15
- 18:43 bd808: Added dduvall as reader+member so he can help with instance maintenance chores
2023-10-17
- 12:39 w... (more)