Puppet/Testing
We have a set of helpers to lint, check style and even test the Puppet code we write. This part cover how to run the utilities, how to write your own tests and even how to debug!
Configuring the environment - Bundler
The puppet repo uses bundler to create a static environment for all dependencies required for puppet CI. bundler looks at the Gemfile in the root directory and installs the necessary gems in the specified location. The default location is to install in the same place as gem install i.e. into the system path. this is often not desired as such its better to specify a default directory first, which causes bundler to act a bit more like virtualenv.
run the following commands to configure the local repo directory and install the various gem files locally. Alternatively you can use the setup_rake.sh script in the puppet repo
$ git clone ssh://gerrit.wikimedia.org:29418/operations/puppet
Cloning into 'puppet'...
remote: Counting objects: 102, done
remote: Finding sources: 100% (102/102)
remote: Getting sizes: 100% (40/40)
remote: Compressing objects: 100% (88490/88490)
remote: Total 669923 (delta 51), reused 669899 (delta 47)
Receiving objects: 100% (669923/669923), 139.63 MiB | 12.58 MiB/s, done.
Resolving deltas: 100% (531219/531219), done.
$ bundle config set --local path '.bundle/vendor'
$ cat .bundle/config
---
BUNDLE_PATH: ".bundle/vendor"
$ bundle install
$ bundle install
Fetching gem metadata from https://rubygems.org/........
Resolving dependencies.......
Fetching rake 12.0.0
Installing rake 12.0.0
---SNIP---
Fetching rbnacl 4.0.2
Installing rbnacl 4.0.2
Bundle complete! 29 Gemfile dependencies, 112 gems now installed.
Bundled gems are installed into `./.bundle/vendor`
Post-install message from minitar:
The `minitar` executable is no longer bundled with `minitar`. If you are
expecting this executable, make sure you also install `minitar-cli`.
Running tests
Some puppet modules have test suites using the ruby test runner rspec[1] with rspec-puppet[2] and a set of rake tasks to run linting check (to validate manifests, puppet-lint, hiera yaml files, erb templates). The ruby dependencies are listed in Gemfile at the root of each git repositories (operations/puppet, operations/puppet/nginx ...). The dependencies are to be installed using bundler (the ruby world package manager).
To install all required dependencies: bundle install and to run a command in that environment: bundle exec <some command>.
Assuming a puppet module has a Rakefile and tests defined in a ./spec sub directory, one can run syntax checks, style and tests via the three commands:
bundle exec rake syntax
bundle exec rake puppet-lint
bundle exec rake spec
Ignoring lint warnings and errors
If you want you can always ignore specific warnings/errors by surrounding the relevant lines with a special "lint::ignore"-comment. For example, this would ignore "WARNING: top-scope variable being used without an explicit namespace" on line 54.
53 # lint:ignore:variable_scope 54 default => $fqdn 55 # lint:endignore
You can get all the names of the separate checks with puppet-lint --help. More info is on http://puppet-lint.com/controlcomments/
Find out which warnings/errors are currently ignored
There are 2 parts to this. The first is to check the global .puppet-lint.rc file in the root of the puppet repo. You will see that the following 4 checks are currently ignored globally:
- --no-80-chars-check (This is about the lines over 80 chars, we are not planning to remove this exception)
- --no-autoloader_layout-check (This is about moving all remaining class out of manifests/role/ and https://phabricator.wikimedia.org/T119042 we want this fixed)
- --no-puppet-url_without_modules-check (Yes, we want this fixed)
- --no-documentation-check (https://phabricator.wikimedia.org/T127797)
The second part is checking for individual ignores, find these with a grep -r "lint:ignore" * in the root of the puppet repo. You can help by fixing and removing any of these remaining issues.
The tracking task for getting this to perfection is https://phabricator.wikimedia.org/T93645.
Rake explained
Global tasks
The Gemfile we use at the repo root asks for the ruby gem puppetlabs_spec_helper (doc) which contains several predefined rake tasks. We have also added a lot of our own tasks which are mostly defined either directly in the Rakefile or the taskgen ruby file. The following is a list of global tasks which may be useful to users
$ bundle exec rake -T [13:25:40]
Cloning into '/home/jbond/tmp/puppet/spec/fixtures/private'...
remote: Total 11739 (delta 0), reused 11739 (delta 0)
Receiving objects: 100% (11739/11739), 2.67 MiB | 1.41 MiB/s, done.
Resolving deltas: 100% (7210/7210), done.
rake global:doc # Build documentation
rake global:parallel_spec # run Global rspec using parralel_spec (this is experiment...
rake global:spec # Run all spec tests found in modules
rake global:wmf_style # Run the wmf style guide check on all files, or on a sing...
rake help # Display the list of available rake tasks / Show the help
rake lint_fix # Run puppet-lint
rake parallel_spec # Run spec tests in parallel and clean the fixtures direct...
rake puppet_lint # Run puppet-lint
rake rubocop # Run RuboCop
rake rubocop:auto_correct # Auto-correct RuboCop offenses
rake spdx:check:all # Check all files
rake spdx:check:changed # Check changed files
rake spdx:check:missing_permission # Check all files
rake spdx:convert:module[module] # Convert a module to SPDX
rake spdx:convert:profile[profile] # Convert a profile to SPDX
rake spdx:convert:role[role] # Convert a profile to SPDX
rake spec # Run spec tests and clean the fixtures directory if succe...
rake strings:generate[patterns,debug,backtrace,markup,json,yard_args] # Generate Puppet documentation with YARD
rake syntax # Syntax check Puppet manifests and templates
rake syntax:hiera # Syntax check Hiera config files
rake syntax:manifests # Syntax check Puppet manifests
rake syntax:templates # Syntax check Puppet templates
rake test # Run all actual tests in parallel for changes in HEAD
rake tox # Run all the tox-related tasks
rake tox:commit_message # Check commit message
rake typos # Check common typos from /typos
rake validate # Check syntax of Ruby files and call :syntax and :metadat...
rake wmf_styleguide # Check wmf styleguide violations in the current commit
rake wmf_styleguide_delta # Check regressions for the wmf style guide
out of the above list i think the following are the most usefull to point out
bundle exec rake test: this is the task used by CI. it essentially runs all tests defined in taskgen.rbbundle exec rake syntax: run basic syntax checks, ideally your editor will catch anything mentioned herebundle exec rake static: Run WMF specific linting like tasks e.g. json schem checksbundle exec rake test: Run WMF specific unit style test e.g. tox jobsbundle exec tox: run the CI test handled by toxbundle exec wmf_styleguide_delta: run the puppet-lint tests with the wmf_styleguide checks enabled. the delta variant ensures we only test committed/changed filesbundle exec rake rubocop:auto_correctfix easily resolved ruby violations
We also have the ability to run rspec globally with bundle exec rake global:parallel_spec or bundle exec rake global:spec however this can take quite a bit of time and its often better to run theses test on just the files module you have changed
At the global level we also have the SPDX tasks, theses tasks are designed to make it easier to add SPDX headers to modules, roles and profiles in our puppet repo. Please see the original task T308013 for more contex
Module task
For module level Rake files we often just add include the puppetlabs tasks as well as a small script to make some config changes
require 'puppetlabs_spec_helper/rake_tasks'
require_relative '../../rake_modules/module_rake_tasks.rb'
In the module, rake -T gives the list of all available tasks (and rake -P list the dependency tree), though most would do nothing:
$ cd modules/mymodule
$ bundle exec rake -T
rake beaker # Run beaker acceptance tests
rake beaker:sets # List available beaker nodesets
rake beaker:ssh[set,node] # Try to use vagrant to login to the Beaker node
rake build # Build puppet module package
rake check:dot_underscore # Fails if any ._ files are present in directory
rake check:git_ignore # Fails if directories contain the files specified in .gitignore
rake check:symlinks # Fails if symlinks are present in directory
rake check:test_file # Fails if .pp files present in tests folder
rake clean # Clean a built module package
rake compute_dev_version # Print development version of module
rake help # Display the list of available rake tasks
rake lint # Run puppet-lint
rake parallel_spec # Parallel spec tests
rake release_checks # Runs all necessary checks on a module in preparation for a release
rake rubocop # Run RuboCop
rake rubocop:auto_correct # Auto-correct RuboCop offenses
rake spec # Run spec tests and clean the fixtures directory if successful
rake spec_clean # Clean up the fixtures directory
rake spec_prep # Create the fixtures directory
rake spec_standalone # Run spec tests on an existing fixtures directory
rake syntax # Syntax check Puppet manifests and templates
rake syntax:hiera # Syntax check Hiera config files
rake syntax:manifests # Syntax check Puppet manifests
rake syntax:templates # Syntax check Puppet templates
rake validate # Check syntax of Ruby files and call :syntax and :metadata_lint
$
The syntax* tasks come from the rubygem puppet-syntax.
The spec* tasks are helpers to prepare a puppet environment to run rspec into:
specsetup a test environment and run the testsspec_prepadds fixtures and module dependencies for the test environmentspec_standalonerun tests, assuming the test environment has been previously setup (withspec_preorspec).spec_cleantears down that environment
Rspec
To test puppet resources, we rely on rspec-puppet an helper on top of the ruby test runner rspec. rspec-puppet provides utilities to setup puppet, to compile a catalog and it provides built-in assert methods to run against the generated catalog.
A minimal case requires:
- a
Rakefile - a spec defining the tests to conduct
At first the Rakefile reuses the puppetlabs_spec_helper rake tasks described in the previous section:
Rakefile:
require 'puppetlabs_spec_helper/rake_tasks'
require_relative '../../rake_modules/module_rake_tasks.rb'
The tests are placed in sub directories of spec/ based on the type of Puppet resource being tested. That convention lets rspec-puppet properly setup the rspec helpers for the type of puppet resource being tested. rspec finds tests by crawling the hierachy under spec looking for files with the suffix _spec.rb. The hierarchy is:
spec/ ├── applications/ ├── classes/ ├──── someclass_spec.rb ├── defines/ ├── functions/ ├──── some_function_spec.rb ├── hosts/ ├── types/ └── types_aliases/
We have created a global shared_spec file which is responsible for configurring a fixtures directory as well as configuring the private repo and injecting some default facts and global values. in order to use this you need to add the following to any spec file you create
require_relative '../../../../rake_modules/spec_helper'
Given a puppet module mymodule consisting of a single class in manifests/init.pp:
class mymodule {
}
Since we will test a class, we create our test file under spec/classes/ as mymodule_spec.rb:
require_relative '../../../../rake_modules/spec_helper'
describe 'mymodule' do
on_supported_os(WMFConfig.test_on).each do |os, os_facts|
context "on #{os}" do
let(:facts) { os_facts }
describe 'test compilation with default parameters' do
it { is_expected.to compile.with_all_deps }
end
end
end
end
The above file along with a commented version with more explanation is avalible in the puppet repo
And we can finally get the test environment prepared and run the spec:
$ /usr/bin/ruby2.7 -I/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/lib:/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-support-3.11.0/lib /home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb
...
Finished in 0.12021 seconds (files took 2.52 seconds to load)
3 examples, 0 failures
Had we had an error in the manifest, for example a missing curly brace:
$ bundle exec rake spec
/usr/bin/ruby2.7 -I/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/lib:/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-support-3.11.0/lib /home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb
FFF
Failures:
1) mymodule on debian-9-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
Failure/Error: it { is_expected.to compile.with_all_deps }
error during compilation: Syntax error at end of input (file: /home/jbond/tmp/puppet/modules/mymodule/manifests/init.pp) on node storage.home.arpa
# ./spec/classes/init_spec.rb:8:in `block (5 levels) in <top (required)>'
2) mymodule on debian-10-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
Failure/Error: it { is_expected.to compile.with_all_deps }
error during compilation: Syntax error at end of input (file: /home/jbond/tmp/puppet/modules/mymodule/manifests/init.pp) on node storage.home.arpa
# ./spec/classes/init_spec.rb:8:in `block (5 levels) in <top (required)>'
3) mymodule on debian-11-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
Failure/Error: it { is_expected.to compile.with_all_deps }
error during compilation: Syntax error at end of input (file: /home/jbond/tmp/puppet/modules/mymodule/manifests/init.pp) on node storage.home.arpa
# ./spec/classes/init_spec.rb:8:in `block (5 levels) in <top (required)>'
Finished in 0.10432 seconds (files took 2.29 seconds to load)
3 examples, 3 failures
Failed examples:
rspec './spec/classes/init_spec.rb[1:1:1:1]' # mymodule on debian-9-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
rspec './spec/classes/init_spec.rb[1:2:1:1]' # mymodule on debian-10-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
rspec './spec/classes/init_spec.rb[1:3:1:1]' # mymodule on debian-11-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
/usr/bin/ruby2.7 -I/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/lib:/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-support-3.11.0/lib /home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb failed
If you see the following error when running rspecs, it means that you have a ruby version issue:
ArgumentError:
wrong number of arguments (given 1, expected 0)
# ./.bundle/vendor/ruby/3.1.0/gems/rspec-puppet-2.9.0/lib/rspec-puppet/support.rb:429:in `build_catalog_without_cache_v2'
# ...
The way I (brouberol) have been going around it is to run tests in a docker image including ruby 2.7.0. Runningrbenv install 2.7.0 om Debian Bookworm failed due to openSSL issues, so Docker proved easier to get right.
$ cd ~/wmf/puppet
$ cat << EOF > ./Dockerfile.test-runner
FROM ruby:2.7.0-slim-buster
WORKDIR /tmp
RUN gem install bundler:2.3.15 && apt-get update && apt-get install -y --no-install-recommends make
COPY Gemfile Gemfile
COPY utils/setup_rake.sh utils/setup_rake.sh
RUN ./utils/setup_rake.sh && gem install
RUN mkdir /puppet
WORKDIR /puppet
EOF
$ docker build -t puppet-test-runner -f Dockerfile.test-runner .
Once the image is built, we run a container with the puppet codebase mounted inside it, and run the tests
$ docker run -it -v $(pwd):/puppet puppet-test-runner bash
root@ef53174e939e:/puppet % bundle exec rspec modules/install_server/spec/classes/install_server_preseed_server_spec.rb
.
Finished in 0.19117 seconds (files took 1.16 seconds to load)
4 examples, 0 failures
Debugging
A collection of tips to debug spec failures.
Dump resources
If an example fail for now obvious reason, it is sometime helpful to dump the catalog resources just before the example. One can just print it before the failing rspec expectation:
it {
pp catalogue.resources
should compile
}
Puppet debug log
Enable puppet debug log to the console. In the spec_helper.rb add:
RSpec.configure do |conf|
if ENV['PUPPET_DEBUG']
conf.before(:each) do
Puppet::Util::Log.level = :debug
Puppet::Util::Log.newdestination(:console)
end
end
end
Then run tests with PUPPET_DEBUG=1 bundle exec rake spec
Credits: maxlinc@github gist).
Run a single spec / example
Run in the bundle environment, run rspec on a specific spec:
bundle exec rspec spec/classes/someclass_spec.rb
Or you can filter based on the spec name:
bundle exec rspec --example mymodule::someclass
See rspec help for more details.
Pass options to rspec from env
You can pass extra options to rspec via SPEC_OPTS environment variable. Useful when you want to invoke your tests from rake but want to refine what rspec does:
SPEC_OPTS="--example mymodule::someclass" bundle exec rake
Which would be the equivalent of:
bundle exec rake spec_prep
bundle exec rspec --example my module::someclass
ruby debugger
You can use the gem pry to break on error and get shown a console in the context of the failure. To your Gemfile add gem 'pry' and install it with bundle install then to break inside a spec:
require 'spec_helper'
require 'pry'
describe 'mymodule::someclass' do
it {
# enable debugger
binding.pry
# compilation that fails:
should compile
}
end
You will then be in a console before the breakage that let you inspect the environment (ls) or print the compiled catalogue (p catalogue). See https://github.com/pry/pry for details.
Cloud VPS testing
Nontrivial puppet changes should be applied to a Cloud VPS instance before being merged into production. This can uncover some behaviors and code interactions that don't appear during individual file tests -- for example, puppet runs frequently fail due to duplicate resource definitions that aren't obvious to the naked eye.
To test a puppet patch:
1. Create a standalone puppetmaster instance.
2. Configure that instance so that it defines the class you're working on. You can do this either via the 'configure instance' page or by editing /var/lib/git/operations/puppet/manifests/site.pp to contain something like this:
node this-is-my-hostname {
include class::I::am::working::on
}
3. Run puppet a couple of times ('$ sudo puppetd -tv') until each subsequent puppet run is clean doesn't modify anything
4. Apply your patch to /var/lib/git/operations/puppet. Do this by cherry-picking from gerrit or by rsyncing from a local working directory.
5. Run puppet again, and note the changes that this puppet run makes. Does puppet succeed? Are the changes what you expected?
catalog compilation
You can compile the puppet catalog and see a diff and have a preview of warnings/errors.
There are several ways of doing this (Jekins, local, etc). See main article: puppet-compiler.
Manual module testing
A relatively simple and crude testing way is
$ puppet apply --noop --modulepath /path/to/modules <manifest>.pp
Do note however that this might not work if you reference stuff outside of the module hierarchy
You can get around the missing module hierarchy problem by cloning a local copy of the puppet repo and symlinking in your new module directory.
eg.
$ git clone --branch production https://gerrit.wikimedia.org/r/operations/puppet.git
$ cd puppet/modules
$ ln -s /path/to/mymodule .
$ puppet apply --verbose --noop --modulepath=/home/${USER}/puppet/modules /path/to/mymodule/manifest/init.pp
Users should also check out the Bolt page which allows for using a noop run to test changes
Integration with Jenkins
Jenkins job simply runs rake test (CI entry point) from the root of the operations/puppet.git. The checks we want to run automatically are marked as prerequisites of the test task, for example:
task test: [:parallel, :wmf_styleguide_delta]
The tasks suffixed with _head are optimized to have the utility to only run on files changed in the proposed patch. Typically puppet-lint takes minutes to run against all the puppet manifests, when for CI we only are interested in the manifests that are actually being changed.
The rakefile add a task for each module having a spec directory. The task is named after the module and put under the namespace spec. Hence as soon as you create a basic structure for a module mymodule, you can run it from the root of the repository with:
bundle exec rake spec:mymodule
And it is dynamically made a prerequisite of the spec task which is run by CI. To say it otherwise, once a spec directory is created, Jenkins will try to run the spec.
Common errors
- Tab character found on line number
- Do not use tabs, use 4-space soft tabs! See the Puppet style guidelines suggestions for Vim or Emacs.
- Double quoted string containing no variables
- Use single quotes (
') for all strings unless there are variables to parse in it
- Unquoted file mode
- Always quote file modes with single quotes, like:
mode => '0750'
- Line has more than 80 characters
- Wrap your lines to be less than 80 chars, if you have to, there is \<newline>. See the Puppet style guidelines suggestions for Vim or Emacs for suggestions on setting this up automatically.
- Not in autoload module layout
- Turn your code into a puppet module (See Module Fundamentals)
- Ensure found on line but it's not the first attribute
- Move your
ensure =>to the top of the resource section. (Don't forget to turn a;into a,if it was the last attribute before).
- unquoted resource title
- Quote all resource titles with single quotes.
- Top-scope variable being used without an explicit namespace
- Use an explicit namespace in variable names (See Scope and Puppet)
- Class defined inside a class
- Don't define classes inside classes
- Quoted boolean value
- Do NOT quote boolean values
- Case statement without a default case
- Add a default case to your case statement
Resources
- http://rspec-puppet.com/ (might not be up-to-date)
- https://github.com/rodjek/rspec-puppet
- https://github.com/puppetlabs/puppetlabs_spec_helper#puppet-labs-spec-helper (among others: doc about fixtures).
- ↑ https://rspec.info/, Behaviour Driven Development for Ruby
- ↑ http://rspec-puppet.com/, RSpec test framework for your Puppet manifests