User:Arturo Borrero Gonzalez

From Wikitech
Jump to navigation Jump to search

I am part of the Cloud Services team.


TODO

Some things I would like to get done eventually.

  • Puppet cleanup/refactor/rewrite for WMCS.
  • Doc write: operationals procedurs and best practices.
  • Doc write: WMCS naming conventions.

Kung-fu

Some random tricks I would like to persist somewhere.

openstack

Some interesting openstack tricks that I use from time to time.

  • Create a VM in a given project, and schedule it directly in a given hypervisor:
root@cloudcontrol1004:~# openstack --os-project-id openstack server create --flavor 3 --image 10783e59-b30e-4426-b509-2fbef7d3103c --nic net-id=7425e328-560c-4f00-8e99-706f3fb90bb4 --availability-zone nova:cloudvirt1024:cloudvirt1024.eqiad.wmnet moritz-mds-stretch-test
[..]
  • Generate a list of running VMs (with project) in a given hypervisor:
aborrero@cloudcontrol1004:~$ for i in $(sudo wmcs-openstack server list --all-projects --host cloudvirt1030 -c ID -f value) ; do VM=$(sudo wmcs-openstack server show $i -c name -f value) ; PROJ=$(sudo wmcs-openstack server show $i -c project_id -f value) ; echo "VM: $VM PROJECT: $PROJ" ; done
VM: taxonbota-b PROJECT: dwl
VM: irc-buster PROJECT: dwl
VM: janus1-1 PROJECT: analytics
VM: integration-agent-docker-1012 PROJECT: integration
VM: integration-agent-docker-1011 PROJECT: integration
[..]

cumin

Some interesting cumin commands I used.

Select servers using an alias and a fact:

aborrero@cumin1001:~ $ sudo cumin -x 'A:cloud-eqiad1 and P{F:lsbdistcodename = jessie}' "dpkg -l python-pysaml2"
IGNORE EXIT CODES mode enabled, all commands executed will be considered successful
15 hosts will be targeted:
cloudcontrol[1003-1004].wikimedia.org,cloudservices[1003-1004].wikimedia.org,cloudvirt[1014,1016-1017,1021-1023].eqiad.wmnet,cloudvirtan[1001-1005].eqiad.wmnet

Select servers using a puppet role and a fact:

aborrero@cumin1001:~ $ sudo cumin 'P{O:wmcs::openstack::eqiad1::virt} and P{F:lsbdistcodename = stretch}'
7 hosts will be targeted:
cloudvirt[1013,1024,1026-1030].eqiad.wmnet
DRY-RUN mode enabled, aborting

Running puppet in installs servers before installing a server:

aborrero@cumin1001:~ $ sudo cumin A:installserver run-puppet-agent
2 hosts will be targeted:
install[1002,2002].wikimedia.org
Confirm to continue [y/n]? y

Finding cloudvirts of a given vendor:

aborrero@cumin1001:~ $ sudo cumin 'P{cloudvirt1* and F:manufacturer = "Dell Inc."}'
14 hosts will be targeted:
cloudvirt[1015-1018,1021-1030].eqiad.wmnet
DRY-RUN mode enabled, aborting

Using a regexp match in server name in CloudVPS:

aborrero@labpuppetmaster1001:~ $ sudo cumin "project:tools name:^tools-static*" uname
2 hosts will be targeted:
tools-static-[12-13].tools.eqiad.wmflabs
Confirm to continue [y/n]? y

Matching 2 different server names in CloudVPS:

aborrero@labpuppetmaster1001:~ $ sudo cumin "O{project:tools name:^tools-k8s-master*} OR O{project:tools name:^tools-docker-registry*}" ":"
3 hosts will be targeted:
tools-docker-registry-[03-04].tools.eqiad.wmflabs,tools-k8s-master-01.tools.eqiad.wmflabs
Confirm to continue [y/n]?

Very quick & basic healthcheck for VM instances in CloudVPS after draining a hypervisor:

aborrero@labpuppetmaster1002:~ $ sudo cumin -m sync F{file.txt} 'cat /etc/debian_version' 'touch /tmp/cumintest && rm -f /tmp/cumintest'

You can generate the list of hosts to check with a command like this, and the copy-paste the list to a file in the cumin server:

root@cloudcontrol1004:~# nova list --all-tenants --host cloudvirt1024 | grep ACTIVE | awk -F' ' '{print $4}'
accounts-appserver5
canary1024-01
[...]

local scripts

Some local scripts I run in my laptop when working in the WMF/WMCS environment.

I copy/pasted here so I don't lost them. And to allow others to reuse them.
I don't track them in git, and the code here may be outdated. Ping me if you need a refresh or need any help!

wmf-export-puppet-patch.sh

I use this script to export a local puppet patch to live-hack a puppetmaster (comonly CloudVPS local puppetmasters):

wmf-git-review.sh

I use this script to rebase a patch before submitting to WMF's gerrit:

wmf-puppet-class-tree.sh

I use this script to get an idea of the puppet classes tree for a given puppet class.

Usefull when working with complex roles/profiles which interact between them (for example, CloudVPS/Openstack):

wmcs-netbox-list.py

I use this script to fetch & list WMCS server info from Netbox and generate a CSV (to later import into a spreadsheet):

other scripts

Other random scripts.

wmcs-instance-hard-reboot.sh

I use this script in cloudcontrol servers to hard-reboot a given Cloud VPS instance.

Example usage:

aborrero@cloudcontrol1004:~$ sudo su
root@cloudcontrol1004:/home/aborrero# cd
root@cloudcontrol1004:~# source novaenv.sh 
root@cloudcontrol1004:~# cd -
/home/aborrero
root@cloudcontrol1004:/home/aborrero# bash wmcs-instance-hard-reboot.sh tools-sgeexec-0924
I: UUID is 4087f96d-7cd8-445c-9e9c-7aabe6d564d7
I: stopping instance
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is ACTIVE
I: status is SHUTOFF
I: starting instance

sssd_rollout.sh

I used this script to rollout the sssd stack to Toolforge in a controlled fashion.

This is meant to be executed in the clushmaster node, with only one input paramenter pointing to a file with a list of nodes:

tools-sgeexec-0908.tools.eqiad.wmflabs
tools-sgeexec-0932.tools.eqiad.wmflabs
tools-sgeexec-0918.tools.eqiad.wmflabs
tools-sgeexec-0927.tools.eqiad.wmflabs
tools-sgeexec-0941.tools.eqiad.wmflabs
tools-sgeexec-0907.tools.eqiad.wmflabs
tools-sgeexec-0913.tools.eqiad.wmflabs
tools-sgeexec-0933.tools.eqiad.wmflabs
tools-sgeexec-0914.tools.eqiad.wmflabs
[...]

puppet

Some personal puppet stuff.

catalog compilation in jenkins

Jenkins puppet-catalog-compier friendly list of canary servers from different Cloud VPS deployments.
Those are meant to quickly copy-paste into the compiler job.

  • all the cloud hardware!
re:^cloud.*(wmnet|org)
  • all Toolforge!
re:.*\.tools\.eqiad\.wmflabs
  • all the cloudXXX-dev hardware!
re:^cloud.*-dev\..*(wmnet|org)

switch puppetmaster for CloudVPS VMs

I used this quick and dirty bash script to test puppetmaster enrollment for several CloudVPS VMs. It should be run from the WMCS cumin server (which is the main puppetmaster too).

Notes

See also


Disclaimer: I work for or provide services to the Wikimedia Foundation, and this is the account I try to use for edits or statements I make in that role. However, the Foundation does not vet all my activity, so edits, statements, or other contributions made by this account may not reflect the views of the Foundation.