Portal:Cloud VPS/Admin/Runbooks/PuppetStaleCertificates
Overview
Error / Incident
There's some stale certificates on the puppetmaster after the removal of some VMs.
Common issues
This usually happens after manually removing a VM in a project that has it's own puppetmaster.
If this is the case, you can follow the guideline here: https://wikitech.wikimedia.org/wiki/Puppet#node_cleanup
But running on the puppetmaster that triggered the alert, for the list of expired certs+puppetmasters you can check:
https://grafana-labs.wikimedia.org/d/MaLO-EpZk/puppet-failures?orgId=1
To avoid, you can use the dedicated cookbook to remove the instances:
dcaro@vulcanus$ cookbook wmcs.vps.remove_instance --help
usage: cookbooks.wmcs.vps.remove_instance [-h] [--project PROJECT] [--task-id TASK_ID] [--no-dologmsg] [--revoke-puppet-certs] --server-name SERVER_NAME
WMCS Toolforge - Remove an instance from a project.
Usage example:
cookbook wmcs.vps.remove_instance \
--project toolsbeta \
--server-name toolsbeta-k8s-test-etcd-08
options:
-h, --help show this help message and exit
--project PROJECT Relevant Cloud VPS openstack project (for operations, dologmsg, etc). If this cookbook is for hardware, this only affects dologmsg calls. Default
is 'admin'.
--task-id TASK_ID Id of the task related to this operation (ex. T123456). (default: None)
--no-dologmsg To disable dologmsg calls (no SAL messages on IRC). (default: False)
--revoke-puppet-certs
If set, the Puppet certificates of this server will be revoked on a custom Puppetmaster (default: False)
--server-name SERVER_NAME
Name of the server to remove (without domain, ex. toolsbeta-test-k8s-etcd-9). (default: None)
Related information
- Puppet: SRE puppet docs.
- Help:Standalone_puppetmaster: Cloud VPS standalone puppetmaster docs
Support contacts
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia Movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect, the bridged Telegram group, or the bridged Mattermost channel
- Discuss via email after you subscribed to the cloud@ mailing list
Old incidents
Add your incident here: