Overview

The procedures in this runbook require project admin permissions to complete.

Error / Incident

There's some stale certificates on the puppetmaster after the removal of some VMs.

Common issues

This usually happens after manually removing a VM in a project that has it's own puppetmaster.

If this is the case, you can follow the guideline here: https://wikitech.wikimedia.org/wiki/Puppet#node_cleanup

But running on the puppetmaster that triggered the alert, for the list of expired certs+puppetmasters you can check:

https://grafana-labs.wikimedia.org/d/MaLO-EpZk/puppet-failures?orgId=1

To avoid, you can use the dedicated cookbook to remove the instances:

dcaro@vulcanus$ cookbook wmcs.vps.remove_instance --help
usage: cookbooks.wmcs.vps.remove_instance [-h] [--project PROJECT] [--task-id TASK_ID] [--no-dologmsg] [--revoke-puppet-certs] --server-name SERVER_NAME

WMCS Toolforge - Remove an instance from a project.

Usage example:
    cookbook wmcs.vps.remove_instance \
        --project toolsbeta \
        --server-name toolsbeta-k8s-test-etcd-08

options:
  -h, --help            show this help message and exit
  --project PROJECT     Relevant Cloud VPS openstack project (for operations, dologmsg, etc). If this cookbook is for hardware, this only affects dologmsg calls. Default
                        is 'admin'.
  --task-id TASK_ID     Id of the task related to this operation (ex. T123456). (default: None)
  --no-dologmsg         To disable dologmsg calls (no SAL messages on IRC). (default: False)
  --revoke-puppet-certs
                        If set, the Puppet certificates of this server will be revoked on a custom Puppetmaster (default: False)
  --server-name SERVER_NAME
                        Name of the server to remove (without domain, ex. toolsbeta-test-k8s-etcd-9). (default: None)

Related information

Puppet: SRE puppet docs.
Help:Standalone_puppetmaster: Cloud VPS standalone puppetmaster docs

Support contacts

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia Movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support

Chat in real time in the IRC channel #wikimedia-cloud ^connect, the bridged Telegram group, or the bridged Mattermost channel
Discuss via email after you subscribed to the cloud@ mailing list

Receive mail announcements about critical changes

Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)

Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Learn about major near-term plans

Read the News wiki page

Read news and stories about Wikimedia Cloud Services

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

Old incidents

Add your incident here:

Portal:Cloud VPS/Admin/Runbooks/PuppetStaleCertificates

Contents