User:Arturo Borrero Gonzalez

From Wikitech
Jump to navigation Jump to search

I am part of the Cloud Services team.


Some things I would like to get done eventually.

  • Puppet cleanup/refactor/rewrite for WMCS.
  • Doc write: operationals procedurs and best practices.
  • Doc write: WMCS naming conventions.


Some random tricks I would like to persist somewhere.


Some interesting openstack tricks that I use from time to time.

  • Create a VM in a given project, and schedule it directly in a given hypervisor:
root@cloudcontrol1004:~# openstack --os-project-id openstack server create --flavor 3 --image 10783e59-b30e-4426-b509-2fbef7d3103c --nic net-id=7425e328-560c-4f00-8e99-706f3fb90bb4 --availability-zone nova:cloudvirt1024:cloudvirt1024.eqiad.wmnet moritz-mds-stretch-test
  • Generate a list of running VMs (with project) in a given hypervisor:
aborrero@cloudcontrol1004:~$ for i in $(sudo wmcs-openstack server list --all-projects --host cloudvirt1030 -c ID -f value) ; do VM=$(sudo wmcs-openstack server show $i -c name -f value) ; PROJ=$(sudo wmcs-openstack server show $i -c project_id -f value) ; echo "VM: $VM PROJECT: $PROJ" ; done
VM: taxonbota-b PROJECT: dwl
VM: irc-buster PROJECT: dwl
VM: janus1-1 PROJECT: analytics
VM: integration-agent-docker-1012 PROJECT: integration
VM: integration-agent-docker-1011 PROJECT: integration

Compare number of scheduled VMs to number stored in the nova database for the hypervisor stats:

aborrero@cloudcontrol1004:~$ ./ 
cloudvirt1001 scheduled VMs: 20 DB stats: 20 
cloudvirt1002 scheduled VMs: 21 DB stats: 24 <---
cloudvirt1003 scheduled VMs: 23 DB stats: 23 
cloudvirt1004 scheduled VMs: 20 DB stats: 20 
cloudvirt1005 scheduled VMs: 24 DB stats: 24 
cloudvirt1006 scheduled VMs: 23 DB stats: 16 <---
cloudvirt1007 scheduled VMs: 19 DB stats: 19 
cloudvirt1008 scheduled VMs: 21 DB stats: 21 
cloudvirt1009 scheduled VMs: 28 DB stats: 15 <---
cloudvirt1012 scheduled VMs: 32 DB stats: 32 
cloudvirt1013 scheduled VMs: 1 DB stats: 1 
cloudvirt1014 scheduled VMs: 52 DB stats: 52 
cloudvirt1015 scheduled VMs: 0 DB stats: 0
cloudvirt1016 scheduled VMs: 42 DB stats: 42 
cloudvirt1017 scheduled VMs: 66 DB stats: 66 
cloudvirt1018 scheduled VMs: 57 DB stats: 57 
cloudvirt1019 scheduled VMs: 3 DB stats: 3 
cloudvirt1020 scheduled VMs: 3 DB stats: 3 
cloudvirt1021 scheduled VMs: 45 DB stats: 45 
cloudvirt1022 scheduled VMs: 4 DB stats: 4 
cloudvirt1023 scheduled VMs: 1 DB stats: 1 
cloudvirt1024 scheduled VMs: 1 DB stats: 1 
cloudvirt1025 scheduled VMs: 42 DB stats: 39 <---
cloudvirt1026 scheduled VMs: 55 DB stats: 46 <---
cloudvirt1027 scheduled VMs: 48 DB stats: 45 <---
cloudvirt1028 scheduled VMs: 51 DB stats: 50 <---
cloudvirt1029 scheduled VMs: 36 DB stats: 36 
cloudvirt1030 scheduled VMs: 38 DB stats: 38

Source code:

This script is used to delete and re-create all the canary VMs we have in each hypervisor.

You can use it like this:

aborrero@cloudcontrol1004:~$ ./
[.. dry run ..]
aborrero@cloudcontrol1004:~$ ./ --doit
[.. actual run ..]

Source code:


Some interesting cumin commands I used.

Select servers using an alias and a fact:

aborrero@cumin1001:~ $ sudo cumin -x 'A:cloud-eqiad1 and P{F:lsbdistcodename = jessie}' "dpkg -l python-pysaml2"
IGNORE EXIT CODES mode enabled, all commands executed will be considered successful
15 hosts will be targeted:

Select servers using a puppet role and a fact:

aborrero@cumin1001:~ $ sudo cumin 'P{O:wmcs::openstack::eqiad1::virt} and P{F:lsbdistcodename = stretch}'
7 hosts will be targeted:
DRY-RUN mode enabled, aborting

Running puppet in installs servers before installing a server:

aborrero@cumin1001:~ $ sudo cumin A:installserver run-puppet-agent
2 hosts will be targeted:
Confirm to continue [y/n]? y

Finding cloudvirts of a given vendor:

aborrero@cumin1001:~ $ sudo cumin 'P{cloudvirt1* and F:manufacturer = "Dell Inc."}'
14 hosts will be targeted:
DRY-RUN mode enabled, aborting

Using a regexp match in server name in CloudVPS:

aborrero@labpuppetmaster1001:~ $ sudo cumin "project:tools name:^tools-static*" uname
2 hosts will be targeted:
Confirm to continue [y/n]? y

Matching 2 different server names in CloudVPS:

aborrero@labpuppetmaster1001:~ $ sudo cumin "O{project:tools name:^tools-k8s-master*} OR O{project:tools name:^tools-docker-registry*}" ":"
3 hosts will be targeted:
Confirm to continue [y/n]?

Very quick & basic healthcheck for VM instances in CloudVPS after draining a hypervisor:

aborrero@labpuppetmaster1002:~ $ sudo cumin -m sync F{file.txt} 'cat /etc/debian_version' 'touch /tmp/cumintest && rm -f /tmp/cumintest'

You can generate the list of hosts to check with a command like this, and the copy-paste the list to a file in the cumin server:

root@cloudcontrol1004:~# nova list --all-tenants --host cloudvirt1024 | grep ACTIVE | awk -F' ' '{print $4}'

local scripts

Some local scripts I run in my laptop when working in the WMF/WMCS environment.

I copy/pasted here so I don't lost them. And to allow others to reuse them.
I don't track them in git, and the code here may be outdated. Ping me if you need a refresh or need any help!

I use this script to cleanup the puppet git repository in a CloudVPS project-local puppetmaster.

I use this script to export a local puppet patch to live-hack a puppetmaster (commonly CloudVPS local puppetmasters):

I use this script to rebase a patch before submitting to WMF's gerrit:

I use this script to get an idea of the puppet classes tree for a given puppet class.

Usefull when working with complex roles/profiles which interact between them (for example, CloudVPS/Openstack):

I use this script to fetch & list WMCS server info from Netbox and generate a CSV (to later import into a spreadsheet):

other scripts

Other random scripts.

I use this script to help in jumping into the console of a given VM instance.

Example usage:

aborrero@cloudcontrol1004:~ $ python3 toolsbeta-test-k8s-etcd-2 toolsbeta
VM nova ID:    59922260-bd39-4bdc-b233-feb78ddb2665
VM Hypervisor: cloudvirt1014.eqiad.wmnet
VM libvirt id: i-0000ddf4

Try this in your laptop:

	ssh -t cloudvirt1014.eqiad.wmnet "sudo virsh console --devname serial1 i-0000ddf4"

(I would really love to do that myself, but SSH creds, etc...)

I use this script in cloudcontrol servers to hard-reboot a given Cloud VPS instance.

Example usage:

aborrero@cloudcontrol1004:~$ sudo su
root@cloudcontrol1004:/home/aborrero# cd
root@cloudcontrol1004:~# source 
root@cloudcontrol1004:~# cd -
root@cloudcontrol1004:/home/aborrero# bash tools-sgeexec-0924
I: UUID is 4087f96d-7cd8-445c-9e9c-7aabe6d564d7
I: stopping instance
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is SHUTOFF
I: starting instance

I used this script to rollout the sssd stack to Toolforge in a controlled fashion.

This is meant to be executed in the clushmaster node, with only one input paramenter pointing to a file with a list of nodes:

This script is used to generate the list of tool webservices allowed in the legacy redirector system. The output is a LUA map text that can be copy/pasted into the legacy redirector LUA script.

To simplify access/credentials, execute this in a tools-k8s-control node, example:

aborrero@tools-k8s-control-3:~$ sudo -i python3 /home/aborrero/
    local allowed = Set {

1014 webservices detected


Some personal puppet stuff.

catalog compilation in jenkins

Jenkins puppet-catalog-compier friendly list of canary servers from different Cloud VPS deployments.
Those are meant to quickly copy-paste into the compiler job.

  • all the cloud hardware!
  • all Toolforge!
  • all the cloudXXX-dev hardware!

switch puppetmaster for CloudVPS VMs

I used this quick and dirty bash script to test puppetmaster enrollment for several CloudVPS VMs. It should be run from the WMCS cumin server (which is the main puppetmaster too).


See also

Disclaimer: I work for or provide services to the Wikimedia Foundation, and this is the account I try to use for edits or statements I make in that role. However, the Foundation does not vet all my activity, so edits, statements, or other contributions made by this account may not reflect the views of the Foundation.