Spicerack/Cookbooks

	Cookbooks
	automation made delicious
	The Plain Sailing Cook Book - Cover
gerrit	operations/cookbooks
installed on	cumin1002.eqiad.wmnet, cumin2002.codfw.wmnet

Cookbook Operations

To list or run a Cookbook you must be logged into one of the cumin hosts (cumin1002.eqiad.wmnet, cumin2002.codfw.wmnet).

List available Cookbooks

To show a tree of the available cookbooks run:

 $ cookbook -l

To include a brief one-line description of each cookbook in the tree use:

 $ cookbook -lv

Run a single Cookbook

The generic way of running a cookbook is:

 $ sudo cookbook [-d] [-v] COOKBOOK [COOKBOOK_ARGS]

The full dotted name as shown in the tree of the available cookbooks (e.g. sre.hosts.downtime) must be used. To see the global options of the cookbook executable:

 $ cookbook -h

To see the specific options of a given cookbook:

 $ sudo cookbook sre.hosts.downtime -h

To run the cookbook in dry-run mode (it will not !log and will not execute any RW operation in the infrastructure):

 $ sudo cookbook -d sre.hosts.downtime [COOKBOOK_ARGS]

And finally, to actually run the cookbook:

 $ sudo cookbook sre.hosts.downtime [COOKBOOK_ARGS]

Logs

How the cookbook store their logs is described in the Spicerack documentation.

Development

Documentation

See the Cookbooks API section of the Spicerack documentation.

Test before merging

Gerrit patches can be tested on the cumin hosts (cumin1002.eqiad.wmnet, cumin2002.codfw.wmnet) before actually merging them using the test-cookbook binary. It allows to checkout the patch from Gerrit and run spicerack using that checkout of the cookbooks. It manages automatically the checkout and allows to clean it up too. Here's the current (June 2023) help message for the script:

$ test-cookbook -h
usage: test-cookbook [-c CHANGE] [--ps PS] [--delete] [--no-sal-logging] [-h]

Cookbook testing script

It allows testing a Gerrit patch on the cookbook repository checking out the change locally
on the cumin host where the command is run and running the cookbook with the change code instead
of the officially deployed one. It also allows to specify a given patch set (PS) to use.
The cookbook binary will be invoked with the -c/--config argument already set to point to the
custom configuration pointing to the testing checkout.

After a first checkout it is possible to modify the files in place and test it again. The script
will automatically detect that there are local changes and ask the operator what to do, as long
as the local modifications are on the same branch (hence the same PS) requested.

It should be used with caution but at least provides a standard way to test cookbook changes.
All the extra parameters are passed to the cookbook binary.

Example usage:

    # Use the latest PS of Gerrit change 12345 to make a DRY-RUN of the downtime cookbook
    test-cookbook -c 12345 --dry-run sre.hosts.downtime -h

    # Use a specific PS of Gerrit change 12345 to make a REAL run of the downtime cookbook
    test-cookbook -c 12345 --ps 3 sre.hosts.downtime -h

    # Cleanup a previously tested change
    test-cookbook --delete -c 12345

The generated files structure is as follows:

    ~/cookbooks_testing/  # Parent directory of all modified files
    ~/cookbooks_testing/config.yaml  # The configuration file that is passed to the cookbook binary
    ~/cookbooks_testing/cookbooks  # Symlink that points to the currently tested change
    ~/cookbooks_testing/cookbooks-$CHANGE_ID  # The git checkout of a given the change ID
    ~/cookbook_testing/logs  # The log directory where all cookbooks will log into

optional arguments:
  -c CHANGE, --change CHANGE
                        The Gerrit change ID to fetch.
  --ps PS               The optional patch set to fetch. If not specified the last one will be fetched.
  --delete              Delete any existing environment for the given change.
  --no-sal-logging      Do not log to SAL. Use with CAUTION!
  -h, --help            show this help message and exit

Deployment

Once a patch is +2 on Gerrit, CI will take care of the actual merge and at the next Puppet run the cookbook will be updated automatically on the cluster management hosts (cumin1002.eqiad.wmnet, cumin2002.codfw.wmnet). Forcing a Puppet run after the merge will update the cookbooks immediately.

Renaming/Deleting a cookbook

Due to the current way of deploying cookbooks, if a cookbook gets renamed or deleted, its compiled files (.pyc), if present, are not automatically deleted and need a manual deletion.

Find the existing files with this command, assuming the cookbook being renamed/deleted was called sre.somedir.example-action, and then proceed to delete them, for example by appending -delete to the find command.

sudo cumin 'A:cumin' 'find /srv/deployment/spicerack/cookbooks/sre/somedir/ -name "example-action*.pyc"'

Creating your local environment

Pre-requisites

tox

Installation

Clone the required repositories from Gerrit under ~/wmf/gerrit for simplicity

$ git clone ssh://$GERRIT_USER@gerrit.wikimedia.org:29418/operations/software/spicerack
$ git clone ssh://$GERRIT_USER@gerrit.wikimedia.org:29418/operations/cookbooks

Change to the spicerack directory and run tox. It will create the proper virtualenv for us with all the dependencies required:

$ tox -e py3N-unit  # where N is picked from any of environment listed with `tox -av | grep unit` for which the related Python version is available on the system

Note: We could just run 'tox' at this point, it will be just a bit slower because it will run all the tests with all the compatible python versions that are installed in the system and configured in the tox.ini file to run the tests
There should be a .tox/py3N-tests/ directory with the virtualenv created, we can source the activate script

$ source .tox/py3N-tests/bin/activate

Create a configuration file for spicerack called config.yaml:

cookbooks_base_dirs:
  - ~/wmf/gerrit/cookbooks
logs_base_dir: ~/wmf/gerrit/spicerack/logs

The cookbook entry point script should be available, and you should be able to see the list of available cookbooks

$ cookbook -c config.yaml -l

Additional steps for WMCS cookbooks

Clone the additional repository for WMCS-specific cookbooks:

$ git clone ssh://$GERRIT_USER@gerrit.wikimedia.org:29418/cloud/wmcs-cookbooks

Add the new checked out repository to the list of available cookbooks in the config.yaml file:

cookbooks_base_dirs:
  - ~/wmf/gerrit/cookbooks
  - ~/wmf/gerrit/wmcs-cookbooks
logs_base_dir: ~/wmf/gerrit/spicerack/logs

Also see

Runbook