Nova Resource:Puppet-diffs/Development
(Redirected from Nova Resource:Puppet-diffs/Devlopment)
Puppet diffs are provided by the puppet_compiler tool. In order to develop puppet compiler it is useful to create a development environment.
Configure a Development environment
Create the configuration files
base: "/srv/jenkins-workspace/puppet-compiler"
puppet_src: "/var/lib/catalog-differ/production"
puppet_private: "/var/lib/catalog-differ/private"
http_url: "http://localhost/pcc"
puppet_var: "/var/lib/catalog-differ/puppet"
Create the necessary directories
# sudo mkdir -p /mnt/jenkins-workspace /var/lib/catalog-differ/{public,private}
# sudo chown ${USER} /mnt/jenkins-workspace /var/lib/catalog-differ/{public,private}
Clone the Repos
# git clone ssh://gerrit.wikimedia.org:29418/operations/puppet /var/lib/catalog-differ/production
# git clone "https://gerrit.wikimedia.org/r/labs/private" /var/lib/catalog-differ/private
Create the puppet directories
# puppet master --compile test --vardir /var/lib/catalog-differ/puppet
Generate SSL material
# puppet cert --ssldir /var/lib/catalog-differ/puppet/ssl --vardir /var/lib/catalog-differ/puppet generate $(hostname -f)
Notice: work-laptop.home.arpa has a waiting certificate request
Notice: Signed certificate request for work-laptop.home.arpa
Notice: Removing file Puppet::SSL::CertificateRequest work-laptop.home.arpa at '/var/lib/catalog-differ/puppet/ssl/ca/requests/work-laptop.home.arpa.pem'
Notice: Removing file Puppet::SSL::CertificateRequest work-laptop.home.arpa at '/var/lib/catalog-differ/puppet/ssl/certificate_requests/work-laptop.home.arpa.pem'
Populate facts
# ssh puppetmaster1001.eqiad.wmnet 'sudo /usr/local/bin/puppet-facts-export'
# ssh puppetmaster1001.eqiad.wmnet 'xzcat /tmp/puppet-facts-export.tar.xz' | tar -C /var/lib/catalog-differ/puppet -xf -
# ssh puppetmaster1001.eqiad.wmnet 'sudo rm -f /tmp/puppet-facts-export.tar.xz'
Test
At this point you should be able to run basic checks, however keep in mind that there is still no puppetdb so anything which relies on a puppetdb, e.g. external resources, won't work
# CHANGE=544943 NODES=puppetmaster1001.eqiad.wmnet BUILD_NUMBER=42 python -m puppet_compiler.cli --debug
[ 2019-12-13T14:14:30 ] INFO: Working on change 544943
[ 2019-12-13T14:14:31 ] INFO: Refreshing the common repos from upstream if needed
[ 2019-12-13T14:14:31 ] DEBUG: refreshing /var/lib/catalog-differ/production
[ 2019-12-13T14:14:42 ] DEBUG: refreshing /var/lib/catalog-differ/private
[ 2019-12-13T14:14:43 ] INFO: Creating directories under /srv/jenkins-workspace/puppet-compiler
[ 2019-12-13T14:14:43 ] DEBUG: Creating directories under /srv/jenkins-workspace/puppet-compiler/42
[ 2019-12-13T14:14:43 ] DEBUG: Cloning directories...
[ 2019-12-13T14:14:43 ] DEBUG: Adding symlinks
[ 2019-12-13T14:14:43 ] DEBUG: Realm is production, skipping writing puppet.conf
[ 2019-12-13T14:14:43 ] DEBUG: Cloning directories...
[ 2019-12-13T14:14:44 ] DEBUG: Adding symlinks
[ 2019-12-13T14:14:44 ] DEBUG: Starting new HTTPS connection (1): gerrit.wikimedia.org:443
[ 2019-12-13T14:14:45 ] DEBUG: https://gerrit.wikimedia.org:443 "GET /r/changes/544943?o=CURRENT_REVISION HTTP/1.1" 200 521
[ 2019-12-13T14:14:45 ] DEBUG: Downloading patch for project operations/puppet, change 544943, revision 10
Note: checking out 'FETCH_HEAD'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at 99f4115838 puppet-merge: add Repository class
From /var/lib/catalog-differ/production
* branch production -> FETCH_HEAD
First, rewinding head to replay your work on top of it...
Applying: puppet-merge: add Repository class
[ 2019-12-13T14:14:52 ] DEBUG: Running submodule init
[ 2019-12-13T14:14:52 ] DEBUG: Realm is production, skipping writing puppet.conf
[ 2019-12-13T14:14:52 ] DEBUG: Spawning a Thread executor
[ 2019-12-13T14:14:52 ] DEBUG: Spawning a Thread executor
[ 2019-12-13T14:14:52 ] INFO: Starting run in mode change
[ 2019-12-13T14:14:52 ] DEBUG: Executing payload <bound method HostWorker.run_host of <puppet_compiler.worker.HostWorker object at 0x7f88842df990>>
[ 2019-12-13T14:14:52 ] DEBUG: Patching /var/lib/catalog-differ/puppet/yaml/facts/puppetmaster1001.eqiad.wmnet.yaml
[ 2019-12-13T14:14:52 ] INFO: Compiling host puppetmaster1001.eqiad.wmnet (prod)
[ 2019-12-13T14:15:14 ] INFO: Compiling host puppetmaster1001.eqiad.wmnet (change)
[ 2019-12-13T14:15:33 ] INFO: Calculating diffs for puppetmaster1001.eqiad.wmnet
[ 2019-12-13T14:15:34 ] DEBUG: Rendering index page for puppetmaster1001.eqiad.wmnet
[ 2019-12-13T14:15:34 ] DEBUG: Msg(is_error=False, value=(0, 0, None), args=(), kwargs={'classes': (<class 'puppet_compiler.state.ChangeState'>, <class 'puppet_compiler.presentation.html.Index'>), 'hostname': 'puppetmaster1001.eqiad.wmnet', 'mode': 'change'})
[ 2019-12-13T14:15:34 ] DEBUG: Execution terminated
[ 2019-12-13T14:15:34 ] INFO: [change] Nodes: 1 NOOP
[ 2019-12-13T14:15:34 ] DEBUG: Rendering the main index page
[ 2019-12-13T14:15:34 ] DEBUG: Stopping Thread
[ 2019-12-13T14:15:34 ] DEBUG: Stopping Thread
[ 2019-12-13T14:15:34 ] INFO: Run finished for mode change; see your results at https://puppet-compiler.wmflabs.org/compiler1001/42/
Webserver
Adding the following nginx config allows you to view the reports at http://localhost/pcc/
location /pcc {
alias /srv/jenkins-workspace/puppet-compiler/output/;
autoindex on;
types {
application/json pson;
text/plain err;
text/html html;
}
}