The procedures in this runbook require admin permissions to complete.

Error / Incident

This alert means that our tofu tests failed to destroy all the resources after creating them.

We run the tofu tests from the machine tf-bastion.tf-infra-test.eqiad1.wikimedia.cloud, that regularly try to apply (create) and then destroy a collection of resources on cloud vps.

Debugging

Ssh to the tf-bastion.tf-infra-test.eqiad1.wikimedia.cloud and try running the creation manually, that will give you more logs.

NOTE: ssh might not work (see phab:T341764, in that case try using the console cookbook (wmcs.openstack.cloudvirt.vm_console).

dcaro@urcuchillay$ ssh tf-bastion.tf-infra-test.eqiad1.wikimedia.cloud
dcaro@tf-bastion$ sudo -i
root@tf-bastion:~# cd tf-infra-test/                                                           
root@tf-bastion:~/tf-infra-test# tofu destroy -var datacenter=eqiad1
data.openstack_images_image_v2.debian: Reading...
...
Do you really want to destroy all resources?
  OpenTofu will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes
...

Common issues

Add here any new common issues you find.

Related information

tf-infra-test repo

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support

Chat in real time in the IRC channel #wikimedia-cloud ^connect or the bridged Telegram group
Discuss via email after you have subscribed to the cloud@ mailing list

Stay aware of critical changes and plans

Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
Read the News wiki page

Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

Old incidents

Add here any new tasks for incidents you might encounter.