Puppet/Beaker
Beaker is a project to allow users to perform application testing on Puppet repositories. It allows you to define a host or set of hosts, apply a Puppet policy and then uses serverspec to test the state of the built VM or container. The Puppet repo contains some tooling to allow users to try and spin up and build a Debian container using the Beaker framework. The current Beaker configuration is set to use the Buster image from the production Docker register (docker-registry.wikimedia.org/buster:latest) and apply the spare role. This allows us to test that the base Puppet policy applies cleanly to a machine with no errors or dependency cycles.
Running beaker
Beaker is ran via bundler as such you should ensure that it is configured. Once that is in place we simply need to tell Beaker which nodeset to use by setting BEAKER_set=debian10 e.g.
As we apply the full policy this can take 2-5 minutes to run depending on system resources.
$ BEAKER_set="debian10" bundle exec rake beaker
TEST_TIERS env variable not defined. Defaulting to run all tests.
/usr/bin/ruby2.7 -I/home/jbond/git/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/lib:/home/jbond/git/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-support-3.11.0/lib /home/jbond/git/puppet/.bundle/vendor/ruby/2.7.0/gems/rspec-core-3.11.0/exe/rspec spec/acceptance
/home/jbond/git/puppet/.bundle/vendor/ruby/2.7.0/gems/beaker-rspec-7.1.0/lib/beaker-rspec/helpers/serverspec.rb:43: warning: already initialized constant Module::VALID_OPTIONS_KEYS
/home/jbond/git/puppet/.bundle/vendor/ruby/2.7.0/gems/specinfra-2.83.2/lib/specinfra/configuration.rb:4: warning: previous definition of VALID_OPTIONS_KEYS was here
Beaker::Hypervisor, found some docker boxes to create
Provisioning docker
provisioning buster
Using container connection at 127.0.0.1:301
Disabling updates.puppetlabs.com by modifying hosts file to resolve updates to 127.0.0.1 on buster
sretest role
running puppet code
localhost $ scp /tmp/beaker20220804-3616255-4nuehe buster:/tmp/apply_manifest_161941092.pp.aJDWKi {:ignore => }
localhost $ scp /tmp/beaker20220804-3616255-tieng6 buster:/tmp/apply_manifest_162453939.pp.sNoc59 {:ignore => }
localhost $ scp /tmp/beaker20220804-3616255-uzdr4u buster:/tmp/apply_manifest_162509935.pp.zcok92 {:ignore => }
work with no errors
Cleaning up docker
Finished in 5 minutes 46 seconds (files took 35.2 seconds to load)
1 example, 0 failures
Debug Mode
The output from the default run allows us to see if the Puppet catalogue compiles and applies with no errors. We also ensure that Puppet performs no changes after applying the same catalogue three times. i.e. we reach convergence. however if there are errors we often need to see the output of the Puppet run. For this we need to run Beaker with debug mode enabled. this is as simple as setting BEAKER_debug=yes
$ BEAKER_debug=yes BEAKER_set="debian10" bundle exec rake beaker
The debug output is quite verbose but very helpful for debugging.
Preserve image
By default Beaker will destroy the container that it creates. However if there are errors it's often useful to leave the container online so that we can login, look around and test changes. For this we need to pass BEAKER_destroy=no
$ BEAKER_destroy=no BEAKER_debug=yes BEAKER_set="debian10" bundle exec rake beaker
once completed you can just use Docker to find the host and exec bash
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f7aa5ce5c72d 9be40b9a35d0 "/sbin/init" 3 minutes ago Up 3 minutes 0.0.0.0:4956->22/tcp beaker-buster-b2bb52fea28f
8e9d3c0adc1b victronenergy/venus-docker-grafana:latest "/entrypoint.sh" 3 weeks ago Up 3 days 0.0.0.0:3000->3000/tcp venus-docker-grafana_graphing_1
06f48f1dfe8f influxdb:1.7 "/entrypoint.sh infl…" 3 weeks ago Up 3 days 0.0.0.0:8086->8086/tcp venus-docker-grafana_influxdb_1
4e819fff9a7d victronenergy/venus-docker-upnp:latest "docker-entrypoint.s…" 3 weeks ago Up 3 days venus-docker-grafana_upnp_1
117837fc6418 victronenergy/venus-docker-server:latest "docker-entrypoint.s…" 3 weeks ago Up 3 days 0.0.0.0:8088->8088/tcp venus-docker-grafana_server_1
The container will be named beaker-${distro}-$randomehex, so from above we can see mine is named beaker-buster-b2bb52fea28f with container id f7aa5ce5c72d. As such we can access it with.
$ docker exec -i -t f7aa5ce5c72d /bin/bash
root@insetup1001:/#
One logged in one of the most common things to do is to run the policy that just failed. Beaker copies the manifests it plans to apply to /tmp/apply_manifest_$timestamp.pp.$randomstr. You should be able to see this in the output from Buster but you can also just ls /tmp/apply_manifest* and run one of those files
root@insetup1001:/$ ls -la /tmp/apply_manifest_*
-rw------- 1 root root 272 Aug 4 14:29 /tmp/apply_manifest_162912952.pp.nUup3y
-rw------- 1 root root 272 Aug 4 14:34 /tmp/apply_manifest_163416638.pp.1EOGYv
-rw------- 1 root root 272 Aug 4 14:34 /tmp/apply_manifest_163432892.pp.HqbjsU
root@insetup1001:/# puppet apply /tmp/apply_manifest_162912952.pp.nUup3y
Warning: Undefined variable '::_role'; \n (file & line not available)
Warning: Scope(Class[Profile::Netbox::Host]): insetup1001.eqiad.wmnet is unknown in netbox
Warning: Scope(Class[Profile::Netbox::Host]): insetup1001.eqiad.wmnet: no netbox location found
Warning: Scope(Class[main]): puppetdb_query function not usable, returning an empty array
Warning: Unknown variable: '::use_puppetdb'. (file: /etc/puppet/code/modules/ssh/manifests/client.pp, line: 21, column: 29)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/ssh/manifests/server.pp, line: 80, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_host.pp, line: 21, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Warning: You cannot collect exported resources without storeconfigs being set; the export is ignored (file: /etc/puppet/code/modules/monitoring/manifests/exported_nagios_service.pp, line: 26, column: 5)
Notice: Compiled catalog for insetup1001.eqiad.wmnet in environment production in 6.44 seconds
Notice: Applied catalog in 4.75 seconds
Testing random WMF puppet code
Keeping the Docker image around also gives us a Puppet environment configured with the production Puppet environment, this allows us to test specific functions or individual classes/defines e.g.
root@insetup1001:/# puppet apply -e "notice(ipresolve('www.wikimedia.org'))"
Notice: Scope(Class[main]): 185.15.58.224
Notice: Compiled catalog for insetup1001.eqiad.wmnet in environment production in 0.09 seconds
Notice: Applied catalog in 0.14 seconds
root@insetup1001:/# puppet apply -e 'wmflib::dir::mkdir_p("/tmp/foo/bar")'
Notice: Compiled catalog for insetup1001.eqiad.wmnet in environment production in 0.05 seconds
Notice: /Stage[main]/Main/File[/tmp/foo]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/foo/bar]/ensure: created
Notice: Applied catalog in 0.13 seconds
root@insetup1001:/# ls -la /tmp/foo/bar/
total 1
drwxr-xr-x 2 root root 2 Aug 4 14:43 .
drwxr-xr-x 3 root root 3 Aug 4 14:43 ..
root@insetup1001:/# puppet apply -e 'admin::user { "foobar": ensure => present}'
Warning: Undefined variable '::site'; \n (file & line not available)
Warning: Undefined variable '::_role'; \n (file & line not available)
Notice: Compiled catalog for insetup1001.eqiad.wmnet in environment production in 3.17 seconds
Notice: /Stage[main]/Main/Admin::User[foobar]/User[foobar]/ensure: created
Notice: /Stage[main]/Main/Admin::User[foobar]/File[/home/foobar]/ensure: created
Notice: /Stage[main]/Main/Admin::User[foobar]/File[/home/foobar/.gitignore]/ensure: defined content as '{md5}fcc0a1c2f12fcf1d56262612d8cb291b'
Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: /usr/local/sbin/enforce-users-groups removing user/id: foobar/40199
Notice: /Stage[main]/Admin/Exec[enforce-users-groups-cleanup]/returns: executed successfully
Notice: Applied catalog in 2.98 seconds
root@insetup1001:/# puppet apply -e 'class {"debian":}'
Notice: Compiled catalog for insetup1001.eqiad.wmnet in environment production in 0.02 seconds
Notice: Applied catalog in 0.13 seconds
Changing Roles
Before getting too excited its worth pointing out at this point that the majority of roles will not compile successfully. This is due to a number of reasons the main ones being:
- No ability to resolve wmnet domain names (Linux users can resolve this with the wmf-laptop-dnsmasq package)
- Lack of shared services like acme_chief, cfssl, dbs etc
However you may still have some success and for simple role you may find that you have total success :)
To override the default role you simply set PUPPET_ROLE, this will also set the host name of the container to $role1001. This catches the default case but it's also common for the role to not match the hostname as such one can also set PUPPET_HOSTNAME to override this behaviour e.g.
$ PUPPET_ROLE=sretest BEAKER_set="debian10" bundle exec rake beaker
$ PUPPET_ROLE=dnsbox PUPPET_hostname=dns1001.codfw.wmnet BEAKER_set="debian10" bundle exec rake beaker
Other environment variables
Beaker also configures the Puppet realm to production and the site to eqiad. These can also be changed using PUPPET_REALM and PUPPET_SITE. However please not that only production realm has been tested and the scaffolding has been built with production in mind so YMMC with WMCS. putting that all together we have
$ PUPPET_SITE=codfw PUPPET_REALM=production PUPPET_ROLE=dnsbox PUPPET_hostname=dns2001.codfw.wmnet BEAKER_destroy=no BEAKER_debug=yes BEAKER_set="debian10" bundle exec rake beaker