Jump to content

Puppet/Testing

From Wikitech

We have a set of helpers to lint, check style and even test the Puppet code we write. This part cover how to run the utilities, how to write your own tests and even how to debug!

Configuring the environment - Bundler

The puppet repo uses bundler to create a static environment for all dependencies required for puppet CI. bundler looks at the Gemfile in the root directory and installs the necessary gems in the specified location. The default location is to install in the same place as gem install i.e. into the system path. this is often not desired as such its better to specify a default directory first, which causes bundler to act a bit more like virtualenv.

run the following commands to configure the local repo directory and install the various gem files locally. Alternatively you can use the setup_rake.sh script in the puppet repo

$ git clone ssh://gerrit.wikimedia.org:29418/operations/puppet                                                  
Cloning into 'puppet'...
remote: Counting objects: 102, done
remote: Finding sources: 100% (102/102)
remote: Getting sizes: 100% (40/40)
remote: Compressing objects: 100% (88490/88490)
remote: Total 669923 (delta 51), reused 669899 (delta 47)
Receiving objects: 100% (669923/669923), 139.63 MiB | 12.58 MiB/s, done.
Resolving deltas: 100% (531219/531219), done.
$ bundle config set --local path '.bundle/vendor'
$ cat .bundle/config                                                                                         
---
BUNDLE_PATH: ".bundle/vendor"
$ bundle install
$ bundle install                                                                                             
Fetching gem metadata from https://rubygems.org/........
Resolving dependencies.......
Fetching rake 12.0.0
Installing rake 12.0.0
---SNIP---
Fetching rbnacl 4.0.2
Installing rbnacl 4.0.2
Bundle complete! 29 Gemfile dependencies, 112 gems now installed.
Bundled gems are installed into `./.bundle/vendor`
Post-install message from minitar:
The `minitar` executable is no longer bundled with `minitar`. If you are
expecting this executable, make sure you also install `minitar-cli`.

Running tests

Some puppet modules have test suites using the ruby test runner rspec[1] with rspec-puppet[2] and a set of rake tasks to run linting check (to validate manifests, puppet-lint, hiera yaml files, erb templates). The ruby dependencies are listed in Gemfile at the root of each git repositories (operations/puppet, operations/puppet/nginx ...). The dependencies are to be installed using bundler (the ruby world package manager).

To install all required dependencies: bundle install and to run a command in that environment: bundle exec <some command>.

Assuming a puppet module has a Rakefile and tests defined in a ./spec sub directory, one can run syntax checks, style and tests via the three commands:

bundle exec rake syntax
bundle exec rake puppet-lint
bundle exec rake spec

Ignoring lint warnings and errors

If you want you can always ignore specific warnings/errors by surrounding the relevant lines with a special "lint::ignore"-comment. For example, this would ignore "WARNING: top-scope variable being used without an explicit namespace" on line 54.

 53         # lint:ignore:variable_scope
 54         default     => $fqdn
 55         # lint:endignore

You can get all the names of the separate checks with puppet-lint --help. More info is on http://puppet-lint.com/controlcomments/

Find out which warnings/errors are currently ignored

There are 2 parts to this. The first is to check the global .puppet-lint.rc file in the root of the puppet repo. You will see that the following 4 checks are currently ignored globally:

The second part is checking for individual ignores, find these with a grep -r "lint:ignore" * in the root of the puppet repo. You can help by fixing and removing any of these remaining issues.

The tracking task for getting this to perfection is https://phabricator.wikimedia.org/T93645.

Rake explained

Global tasks

The Gemfile we use at the repo root asks for the ruby gem puppetlabs_spec_helper (doc) which contains several predefined rake tasks. We have also added a lot of our own tasks which are mostly defined either directly in the Rakefile or the taskgen ruby file. The following is a list of global tasks which may be useful to users

$ bundle exec rake -T                                                                                         [13:25:40]
Cloning into '/home/jbond/tmp/puppet/spec/fixtures/private'...
remote: Total 11739 (delta 0), reused 11739 (delta 0)
Receiving objects: 100% (11739/11739), 2.67 MiB | 1.41 MiB/s, done.
Resolving deltas: 100% (7210/7210), done.
rake global:doc                                                        # Build documentation
rake global:parallel_spec                                              # run Global rspec using parralel_spec (this is experiment...
rake global:spec                                                       # Run all spec tests found in modules
rake global:wmf_style                                                  # Run the wmf style guide check on all files, or on a sing...
rake help                                                              # Display the list of available rake tasks / Show the help
rake lint_fix                                                          # Run puppet-lint
rake parallel_spec                                                     # Run spec tests in parallel and clean the fixtures direct...
rake puppet_lint                                                       # Run puppet-lint
rake rubocop                                                           # Run RuboCop
rake rubocop:auto_correct                                              # Auto-correct RuboCop offenses
rake spdx:check:all                                                    # Check all files
rake spdx:check:changed                                                # Check changed files
rake spdx:check:missing_permission                                     # Check all files
rake spdx:convert:module[module]                                       # Convert a module to SPDX
rake spdx:convert:profile[profile]                                     # Convert a profile to SPDX
rake spdx:convert:role[role]                                           # Convert a profile to SPDX
rake spec                                                              # Run spec tests and clean the fixtures directory if succe...
rake strings:generate[patterns,debug,backtrace,markup,json,yard_args]  # Generate Puppet documentation with YARD
rake syntax                                                            # Syntax check Puppet manifests and templates
rake syntax:hiera                                                      # Syntax check Hiera config files
rake syntax:manifests                                                  # Syntax check Puppet manifests
rake syntax:templates                                                  # Syntax check Puppet templates
rake test                                                              # Run all actual tests in parallel for changes in HEAD
rake tox                                                               # Run all the tox-related tasks
rake tox:commit_message                                                # Check commit message
rake typos                                                             # Check common typos from /typos
rake validate                                                          # Check syntax of Ruby files and call :syntax and :metadat...
rake wmf_styleguide                                                    # Check wmf styleguide violations in the current commit
rake wmf_styleguide_delta                                              # Check regressions for the wmf style guide

out of the above list i think the following are the most usefull to point out

  • bundle exec rake test: this is the task used by CI. it essentially runs all tests defined in taskgen.rb
  • bundle exec rake syntax: run basic syntax checks, ideally your editor will catch anything mentioned here
  • bundle exec rake static: Run WMF specific linting like tasks e.g. json schem checks
  • bundle exec rake test: Run WMF specific unit style test e.g. tox jobs
  • bundle exec tox: run the CI test handled by tox
  • bundle exec wmf_styleguide_delta: run the puppet-lint tests with the wmf_styleguide checks enabled. the delta variant ensures we only test committed/changed files
  • bundle exec rake rubocop:auto_correct fix easily resolved ruby violations

We also have the ability to run rspec globally with bundle exec rake global:parallel_spec or bundle exec rake global:spec however this can take quite a bit of time and its often better to run theses test on just the files module you have changed

At the global level we also have the SPDX tasks, theses tasks are designed to make it easier to add SPDX headers to modules, roles and profiles in our puppet repo. Please see the original task T308013 for more contex

Module task

For module level Rake files we often just add include the puppetlabs tasks as well as a small script to make some config changes

require 'puppetlabs_spec_helper/rake_tasks'
require_relative '../../rake_modules/module_rake_tasks.rb'

In the module, rake -T gives the list of all available tasks (and rake -P list the dependency tree), though most would do nothing:

$ cd modules/mymodule
$ bundle exec rake -T
rake beaker                # Run beaker acceptance tests
rake beaker:sets           # List available beaker nodesets
rake beaker:ssh[set,node]  # Try to use vagrant to login to the Beaker node
rake build                 # Build puppet module package
rake check:dot_underscore  # Fails if any ._ files are present in directory
rake check:git_ignore      # Fails if directories contain the files specified in .gitignore
rake check:symlinks        # Fails if symlinks are present in directory
rake check:test_file       # Fails if .pp files present in tests folder
rake clean                 # Clean a built module package
rake compute_dev_version   # Print development version of module
rake help                  # Display the list of available rake tasks
rake lint                  # Run puppet-lint
rake parallel_spec         # Parallel spec tests
rake release_checks        # Runs all necessary checks on a module in preparation for a release
rake rubocop               # Run RuboCop
rake rubocop:auto_correct  # Auto-correct RuboCop offenses
rake spec                  # Run spec tests and clean the fixtures directory if successful
rake spec_clean            # Clean up the fixtures directory
rake spec_prep             # Create the fixtures directory
rake spec_standalone       # Run spec tests on an existing fixtures directory
rake syntax                # Syntax check Puppet manifests and templates
rake syntax:hiera          # Syntax check Hiera config files
rake syntax:manifests      # Syntax check Puppet manifests
rake syntax:templates      # Syntax check Puppet templates
rake validate              # Check syntax of Ruby files and call :syntax and :metadata_lint
$

The syntax* tasks come from the rubygem puppet-syntax.

The spec* tasks are helpers to prepare a puppet environment to run rspec into:

  • spec setup a test environment and run the tests
  • spec_prep adds fixtures and module dependencies for the test environment
  • spec_standalone run tests, assuming the test environment has been previously setup (with spec_pre or spec).
  • spec_clean tears down that environment

Rspec

To test puppet resources, we rely on rspec-puppet an helper on top of the ruby test runner rspec. rspec-puppet provides utilities to setup puppet, to compile a catalog and it provides built-in assert methods to run against the generated catalog.

A minimal case requires:

  • a Rakefile
  • a spec defining the tests to conduct

At first the Rakefile reuses the puppetlabs_spec_helper rake tasks described in the previous section:

Rakefile:

require 'puppetlabs_spec_helper/rake_tasks'
require_relative '../../rake_modules/module_rake_tasks.rb'

The tests are placed in sub directories of spec/ based on the type of Puppet resource being tested. That convention lets rspec-puppet properly setup the rspec helpers for the type of puppet resource being tested. rspec finds tests by crawling the hierachy under spec looking for files with the suffix _spec.rb. The hierarchy is:

spec/
  ├── applications/
  ├── classes/
  ├──── someclass_spec.rb
  ├── defines/
  ├── functions/
  ├──── some_function_spec.rb
  ├── hosts/
  ├── types/
  └── types_aliases/


We have created a global shared_spec file which is responsible for configurring a fixtures directory as well as configuring the private repo and injecting some default facts and global values. in order to use this you need to add the following to any spec file you create

require_relative '../../../../rake_modules/spec_helper'


Given a puppet module mymodule consisting of a single class in manifests/init.pp:

class mymodule {
}


Since we will test a class, we create our test file under spec/classes/ as mymodule_spec.rb:

require_relative '../../../../rake_modules/spec_helper'

describe 'mymodule' do
  on_supported_os(WMFConfig.test_on).each do |os, os_facts|
    context "on #{os}" do
      let(:facts) { os_facts }
      describe 'test compilation with default parameters' do
        it { is_expected.to compile.with_all_deps }
      end
    end
  end
end

The above file along with a commented version with more explanation is avalible in the puppet repo


And we can finally get the test environment prepared and run the spec:

$ /usr/bin/ruby2.7 -I/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/lib:/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-support-3.11.0/lib /home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb
...

Finished in 0.12021 seconds (files took 2.52 seconds to load)
3 examples, 0 failures

Had we had an error in the manifest, for example a missing curly brace:

$ bundle exec rake spec
/usr/bin/ruby2.7 -I/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/lib:/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-support-3.11.0/lib /home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb
FFF

Failures:

  1) mymodule on debian-9-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
     Failure/Error: it { is_expected.to compile.with_all_deps }
       error during compilation: Syntax error at end of input (file: /home/jbond/tmp/puppet/modules/mymodule/manifests/init.pp) on node storage.home.arpa
     # ./spec/classes/init_spec.rb:8:in `block (5 levels) in <top (required)>'

  2) mymodule on debian-10-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
     Failure/Error: it { is_expected.to compile.with_all_deps }
       error during compilation: Syntax error at end of input (file: /home/jbond/tmp/puppet/modules/mymodule/manifests/init.pp) on node storage.home.arpa
     # ./spec/classes/init_spec.rb:8:in `block (5 levels) in <top (required)>'

  3) mymodule on debian-11-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
     Failure/Error: it { is_expected.to compile.with_all_deps }
       error during compilation: Syntax error at end of input (file: /home/jbond/tmp/puppet/modules/mymodule/manifests/init.pp) on node storage.home.arpa
     # ./spec/classes/init_spec.rb:8:in `block (5 levels) in <top (required)>'

Finished in 0.10432 seconds (files took 2.29 seconds to load)
3 examples, 3 failures

Failed examples:

rspec './spec/classes/init_spec.rb[1:1:1:1]' # mymodule on debian-9-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
rspec './spec/classes/init_spec.rb[1:2:1:1]' # mymodule on debian-10-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles
rspec './spec/classes/init_spec.rb[1:3:1:1]' # mymodule on debian-11-x86_64 test compilation with default parameters is expected to compile into a catalogue without dependency cycles

/usr/bin/ruby2.7 -I/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/lib:/home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-support-3.11.0/lib /home/jbond/tmp/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb failed

If you see the following error when running rspecs, it means that you have a ruby version issue:

ArgumentError:
    wrong number of arguments (given 1, expected 0)
# ./.bundle/vendor/ruby/3.1.0/gems/rspec-puppet-2.9.0/lib/rspec-puppet/support.rb:429:in `build_catalog_without_cache_v2'
# ...

The way I (brouberol) have been going around it is to run tests in a docker image including ruby 2.7.0. Runningrbenv install 2.7.0 om Debian Bookworm failed due to openSSL issues, so Docker proved easier to get right.

$ cd ~/wmf/puppet
$ cat << EOF > ./Dockerfile.test-runner
FROM ruby:2.7.0-slim-buster
WORKDIR /tmp
RUN gem install bundler:2.3.15 && apt-get update && apt-get install -y --no-install-recommends make
COPY Gemfile Gemfile
COPY utils/setup_rake.sh utils/setup_rake.sh
RUN ./utils/setup_rake.sh && gem install
RUN mkdir /puppet
WORKDIR /puppet
EOF
$ docker build -t puppet-test-runner -f Dockerfile.test-runner .

Once the image is built, we run a container with the puppet codebase mounted inside it, and run the tests

$ docker run -it -v $(pwd):/puppet puppet-test-runner bash
root@ef53174e939e:/puppet % bundle exec rspec modules/install_server/spec/classes/install_server_preseed_server_spec.rb 
.

Finished in 0.19117 seconds (files took 1.16 seconds to load)
4 examples, 0 failures

Debugging

A collection of tips to debug spec failures.

Dump resources

If an example fail for now obvious reason, it is sometime helpful to dump the catalog resources just before the example. One can just print it before the failing rspec expectation:

it {
  pp catalogue.resources
  should compile
}

Puppet debug log

Enable puppet debug log to the console. In the spec_helper.rb add:

RSpec.configure do |conf|
  if ENV['PUPPET_DEBUG']
    conf.before(:each) do
      Puppet::Util::Log.level = :debug
      Puppet::Util::Log.newdestination(:console)
    end
  end
end

Then run tests with PUPPET_DEBUG=1 bundle exec rake spec

Credits: maxlinc@github gist).

Run a single spec / example

Run in the bundle environment, run rspec on a specific spec:

bundle exec rspec spec/classes/someclass_spec.rb

Or you can filter based on the spec name:

bundle exec rspec --example mymodule::someclass

See rspec help for more details.

Pass options to rspec from env

You can pass extra options to rspec via SPEC_OPTS environment variable. Useful when you want to invoke your tests from rake but want to refine what rspec does:

SPEC_OPTS="--example mymodule::someclass" bundle exec rake

Which would be the equivalent of:

bundle exec rake spec_prep
bundle exec rspec --example my module::someclass

ruby debugger

You can use the gem pry to break on error and get shown a console in the context of the failure. To your Gemfile add gem 'pry' and install it with bundle install then to break inside a spec:

require 'spec_helper'
require 'pry'

describe 'mymodule::someclass' do

  it {
     # enable debugger
     binding.pry
     # compilation that fails: 
     should compile
  }
end

You will then be in a console before the breakage that let you inspect the environment (ls) or print the compiled catalogue (p catalogue). See https://github.com/pry/pry for details.

reference


Cloud VPS testing

Nontrivial puppet changes should be applied to a Cloud VPS instance before being merged into production. This can uncover some behaviors and code interactions that don't appear during individual file tests -- for example, puppet runs frequently fail due to duplicate resource definitions that aren't obvious to the naked eye.

To test a puppet patch:

1. Create a standalone puppetmaster instance.

2. Configure that instance so that it defines the class you're working on. You can do this either via the 'configure instance' page or by editing /var/lib/git/operations/puppet/manifests/site.pp to contain something like this:

    node this-is-my-hostname {
        include class::I::am::working::on
    }

3. Run puppet a couple of times ('$ sudo puppetd -tv') until each subsequent puppet run is clean doesn't modify anything

4. Apply your patch to /var/lib/git/operations/puppet. Do this by cherry-picking from gerrit or by rsyncing from a local working directory.

5. Run puppet again, and note the changes that this puppet run makes. Does puppet succeed? Are the changes what you expected?


catalog compilation

You can compile the puppet catalog and see a diff and have a preview of warnings/errors.

There are several ways of doing this (Jekins, local, etc). See main article: puppet-compiler.

Manual module testing

A relatively simple and crude testing way is

$ puppet apply --noop --modulepath /path/to/modules <manifest>.pp

Do note however that this might not work if you reference stuff outside of the module hierarchy

You can get around the missing module hierarchy problem by cloning a local copy of the puppet repo and symlinking in your new module directory.

eg.

$ git clone --branch production https://gerrit.wikimedia.org/r/operations/puppet.git
$ cd puppet/modules
$ ln -s /path/to/mymodule .
$ puppet apply --verbose --noop --modulepath=/home/${USER}/puppet/modules /path/to/mymodule/manifest/init.pp

Users should also check out the Bolt page which allows for using a noop run to test changes

Integration with Jenkins

Jenkins job simply runs rake test (CI entry point) from the root of the operations/puppet.git. The checks we want to run automatically are marked as prerequisites of the test task, for example:

task test: [:parallel, :wmf_styleguide_delta]

The tasks suffixed with _head are optimized to have the utility to only run on files changed in the proposed patch. Typically puppet-lint takes minutes to run against all the puppet manifests, when for CI we only are interested in the manifests that are actually being changed.

The rakefile add a task for each module having a spec directory. The task is named after the module and put under the namespace spec. Hence as soon as you create a basic structure for a module mymodule, you can run it from the root of the repository with:

bundle exec rake spec:mymodule

And it is dynamically made a prerequisite of the spec task which is run by CI. To say it otherwise, once a spec directory is created, Jenkins will try to run the spec.

Common errors

Tab character found on line number
Do not use tabs, use 4-space soft tabs! See the Puppet style guidelines suggestions for Vim or Emacs.
Double quoted string containing no variables
Use single quotes (') for all strings unless there are variables to parse in it
Unquoted file mode
Always quote file modes with single quotes, like:
mode => '0750'
Line has more than 80 characters
Wrap your lines to be less than 80 chars, if you have to, there is \<newline>. See the Puppet style guidelines suggestions for Vim or Emacs for suggestions on setting this up automatically.
Not in autoload module layout
Turn your code into a puppet module (See Module Fundamentals)
Ensure found on line but it's not the first attribute
Move your ensure => to the top of the resource section. (Don't forget to turn a ; into a , if it was the last attribute before).
unquoted resource title
Quote all resource titles with single quotes.
Top-scope variable being used without an explicit namespace
Use an explicit namespace in variable names (See Scope and Puppet)
Class defined inside a class
Don't define classes inside classes
Quoted boolean value
Do NOT quote boolean values
Case statement without a default case
Add a default case to your case statement

Resources

  1. https://rspec.info/, Behaviour Driven Development for Ruby
  2. http://rspec-puppet.com/, RSpec test framework for your Puppet manifests