Spicerack

Spicerack is a Python library to orchestrate tasks in the Wikimedia Foundation production environment. It comes with an easy API and a cookbook entry point script that allows to write simple Cookbooks to automate and orchestrate tasks.

Documentation

See the automatically generated Spicerack documentation.

How to contribute

Bug fixes and small feature updates

For any trivial changes simply send a Gerrit request for review. Context is always in demand, so we'll greatly appreciate a Phabricator task with the SRE-Tools tag and include some background that would help us understand your request better.

In case of any uncertainty or questions, feel free to contact the SRE Infrastructure Foundations team.

Adding new module or change in core behaviour

For any bigger changes like adding a new module or changing core behaviour, please create a Phabricator task with Spicerack tag documenting your proposal.

The ideal proposal would include

A problem statement

Here please detail the benefits that the proposed changes will bring or the issues they will assist with.
In case of a new or refactored module, describe the high level API definition including:
- Any new data structures
- All affected public methods signatures with type hints to clearly describe parameters and return types
- The spicerack accessor(s) to be added/modified to spicerack.Spicerack (how the cookbook will access the module features)

Third party dependencies

Any additional third party dependency that would be required with version constraints and if they are available as Debian packages.

Additional configuration

Any additional configuration file and its structure.

Alternatively

At an early stage of a project, you might not be sure about API structure, dependencies and configuration needed - that’s fine, in such a case just create a task with a problem statement and we can backfill the proposal template as we work through the solution together.

This will allow the SRE Infrastructure Foundations team to provide early feedback and guidance on where best in the Spicerack source tree the code should live, or if the code fits more logically in a different library such as wmflib.

We can also provide feedback on any current Spicerack interfaces which may assist with the implementation, either in their current form or with minor refactoring. Finally it allows the SRE Infrastructure Foundations team to provide guidance on how best to organise the module to ensure its interface is consistent with the current Spicerack modules.

Automation and Tooling office hours

Further to this all are welcome to attend the Automation and Tooling office hour to discuss new ideas or specific proposals. Please ask us on IRC for more details and an invite to the office hours meeting.

Test newly released Spicerack features

In order to test changes included in a new Spicerack release, upgrade the spicerack package on a single host (usually cumin2002) and then within a Python shell (sudo -i python3) run:

import logging

logging.basicConfig(level=logging.DEBUG)

from spicerack import Spicerack

spicerack_dry_run = Spicerack(verbose=True, dry_run=True)  # An instance of Spicerack with dry-run set to True
# ATTENTION: spicerack will make read-write changes to the infrastructure!!!
spicerack = Spicerack(verbose=True, dry_run=False)  # An instance of Spicerack with dry-run set to False

At this point spicerack_dry_run and spicerack are two instances of Spicerack like the ones passed to the Spicerack/Cookbooks with the dry-run mode set to True and False respectively.

Alternatively create a script with the above content and a shebang of #!/usr/bin/python3 -i to be run with sudo.