Jump to content

Traffic/Cookbooks

From Wikitech

CDN

roll-restart-varnish.py

Purpose

Roll restart Varnish frontend based on parameters

How to Use

Roll restart Varnish frontend based on parameters. 

   Example usage:
       cookbook sre.cdn.roll-restart-varnish --alias cp-text_codfw --reason 'Emergency restart' \
           --grace-sleep 30 restart_daemons
       cookbook sre.cdn.roll-restart-varnish --query 'A:cp-eqiad and not P{cp1001*}' --reason 'Emergency restart' \
           --batchsize 2 restart_daemons --threads-limited 100000


Flags

--threads-limited
Restart Varnish only if the varnish_main_threads_limited metric variation in the last 10 minutes metric is above the given threshold.


transfer-purged-positions.py

Purpose

Transfer kafka consumer offsets for purged topics.

This cookbook is useful when moving purged from one DC to another

Those are the steps provided by the cookbook:

- Depool the host (automatic) - Disable icinga notification (automatic) - Check that puppet-agent is disabled (pre_scripts) - Transfer kafka consumer offset (_custom_action) - Enable and run Puppet (puppet agent **must** be disabled first with cumin) (_custom_action()) - Repool the host (automatic)


How to Use

Roll apply configuration using puppet-agent for purged 

   This cookbook is useful when moving purged consumers from one kafka DC to another
   and syncing positions across consumers groups is required

   Example usage(s):

       cookbook sre.cdn.transfer-purged-positions \
           --alias cp-text_codfw \
           --dc-to codfw \
           --reason 'move purged back to codfw' \
           --puppet-reason 'TXXXXXX'

Flags

--puppet-reason
Puppet reason. Must match puppet disable reason used by Cumin
--dc-to
Name of the datacenter switch to. One of %(choices)s.


roll-upgrade-ats.py

Purpose

Upgrade ATS on CDN nodes

  • Depool
  • Install specified trafficserver version
  • Restart trafficserver
  • Repool


How to Use

Roll upgrade Apache Traffic Server based on parameters. 

   Example usage:

       cookbook sre.cdn.roll-upgrade-ats \
           --alias cp-text_codfw \
           --reason '9.2.0 upgrade' \
           --version '9.2.0-1wm1'

Flags

--version
Specific version to install.


roll-restart-reboot-ncredir.py

Purpose

NCRedir roll operations cookbook.

How to Use

Cookbook to perform a rolling reboot/restart of NCRedir 

   Usage example:
       cookbook sre.cdn.roll-restart-reboot-ncredir            --reason "Rolling reboot to pick up new kernel" reboot

       cookbook sre.cdn.roll-restart-reboot-ncredir         --reason "Rolling restart to pick new OpenSSL" restart_daemons



run-puppet-restart-varnish.py

Purpose

Swap port 80 from Varnish to HAProxy, stopping Varnish first and then running Puppet.

This cookbook is useful when managing both Varnish and HAProxy configuration and a ordered restart/reload is needed. Has been created to swap the daemon listening on port 80 from Varnish to HAProxy but can be used as template for other tasks too.

Those are the steps provided by the cookbook:

- Depool the host (automatic) - Disable icinga notification (automatic) - Check that puppet-agent is disabled (pre_scripts) - Stop Varnish service (_custom_action) - Enable and run Puppet (puppet agent **must** be disabled first with cumin) (_custom_action()) - Start Varnish service (_custom_action()) - Test that ports (80 and 443) are open (post_action) - Repool the host (automatic)


How to Use

Roll apply configuration using puppet-agent for both HAProxy and Varnish 

   This cookbook is useful when managing both Varnish and HAProxy configuration
   and a ordered restart/reload is needed.
   Has been created to swap the daemon listening on port 80 from Varnish to HAProxy
   but can be used as template for other tasks too.

   Example usage(s):

       cookbook.sre.cdn.run-puppet-restart-varnish \
           --alias cp-text_codfw \
           --reason 'Let HAProxy manage port 80' \
           --puppet-reason 'TXXXXXX' \
           --grace-sleep 1200

Flags

--puppet-reason
Puppet reason. Must match puppet disable reason used by Cumin


roll-reboot.py

Purpose

Depool, unmonitor, and reboot instances one-by-one.

How to Use

Reboot CP nodes in the CDN 

   Example usage:
       cookbook sre.cdn.roll-reboot             --alias 'cp-text_ulsfo'             --reason 'Kernel update'             --task-id T123456

       cookbook sre.cdn.roll-reboot             --alias 'cp-text_ulsfo'             --reason 'Kernel update'             --task-id T123456             --grace-sleep 1200


roll-upgrade-haproxy.py

Purpose

Upgrade HAProxy on CDN nodes

How to Use

Roll upgrade HAProxy based on parameters. 

   Example usage:
       cookbook sre.cdn.roll-upgrade-haproxy --alias cp-text_codfw --reason '2.6.9 upgrade' \
           --grace-sleep 30 restart_daemons
       cookbook sre.cdn.roll-upgrade-haproxy --query 'A:cp-eqiad and not P{cp1001*}' --reason '2.6.9 upgrade' \
           --batchsize 2 restart_daemons



upgrade-varnish.py

Purpose

Upgrade/downgrade Varnish on the given cache host between major releases.

- Set Icinga/Alertmanager downtime - Depool - Disable puppet (unless invoked with --hiera-merged) - Wait for admin to merge hiera puppet change (unless invoked with --hiera-merged) - Remove packages - Re-enable puppet and run it to upgrade/downgrade - Run a test request - Repool - Remove Icinga/Alertmanager downtime

Usage example: 

   cookbook sre.hosts.upgrade-varnish --hiera-merged "Upgrading varnish -- TXXXXXX" cp3030.esams.wmnet


How to Use

Flags

host
FQDN of the host to act upon.
--downgrade
Downgrade varnish instead of upgrading it
--hiera-merged
Pass this flag if hiera is already updated and puppet is disabled on the host with this message


DNS

roll-restart-reboot-wikimedia-dns.py

Purpose

Rolling restart of Wikimedia DNS services or full reboot.

Whether restarting the services or rebooting the entire host, typical decommissioning logic is preserved. systemd unit ordering should safeguard these units as well.


How to Use

Rolling restart of Wikimedia DNS services 

   Example usage:
       cookbook sre.dns.roll-restart-reboot-wikimedia-dns             --alias wikidough-codfw             --reason "Scheduled maintenance"             restart_daemons

       cookbook sre.dns.roll-restart-reboot-wikimedia-dns             --query 'A:wikidough-eqiad and not P{doh1001*}'             --reason "Scheduled maintenance"             --task-id "T12345"             --ignore-restart-errors             --batchsize 2             --grace-sleep 90             restart_daemons

       cookbook sre.dns.roll-restart-reboot-wikimedia-dns             --query 'A:wikidough-eqiad and not P{doh1001*}'             --reason "Scheduled maintenance"             --task-id "T12345"             --batchsize 2             reboot


roll-restart-haproxy.py

Purpose

Rolling service restart of haproxy, part of A:dnsbox and providing internal recursor services.

How to Use

Rolling restart of HAProxy on <TODO>. 

   Example usage:
       cookbook sre.dns.roll-restart-haproxy                 --query '<TODO> and not P{dns1004*}'            --reason "Scheduled maintenance"            reboot


roll-restart-reboot-durum.py

Purpose

Rolling service restart or reboot of durum, the Wikimedia DNS check service.

This is based on the roll-restart-reboot-wikimedia-dns cookbook, with relevant changes to durum.


How to Use

Rolling restart of durum. 

   Example usage:
       cookbook sre.dns.roll-restart-reboot-durum             --query 'A:durum-eqiad and not P{durum1001*}'             --reason "Scheduled maintenance"             reboot

       cookbook sre.dns.roll-restart-reboot-durum             --query 'A:durum-eqiad and not P{durum1001*}'             --reason "Scheduled maintenance"             --task-id "T12345"             --ignore-restart-errors             --batchsize 2             --grace-sleep 90             restart_daemons


roll-restart-ntp.py

Purpose

Rolling restart of ntpsec.service on the DNS hosts identified by A:dnsbox.

How to Use

Rolling restart of ntpsec.service on the DNS hosts. 

   This cookbook is for the rolling restarts of ntpsec.service on the DNS
   hosts.  Since Puppet no longer manages the restarts for us (intentionally),
   this cookbook helps us do that and sets sane automatic defaults for the
   batches and sleep intervals.

   Note that there is an alert in place for ntp.conf: if the file is modified
   and ntpsec.service is not restarted to pick up the changes, we are alerted
   about that. The fix for that is to restart ntpsec.service and now it should
   be done through this cookbook.

   Example usage:
       cookbook sre.dns.roll-restart-ntp                 --alias 'A:dnsbox'                 --task-id T12345                 --reason 'Restarting ntp service' 
       cookbook sre.dns.roll-restart-ntp                 --alias 'A:dnsbox'                 --task-id T12345                 --reason 'Restarting ntp host'                 --grace-sleep 900


admin.py

Purpose

Cookbook for GeoDNS pool/depool of a site.

How to Use

Cookbook for GeoDNS pool/depool of a site. 

   Pool or depool a site for GeoDNS. By default, it will act on a given site
   for all services (text-addrs, upload-addrs, etc.) unless a service is
   manually specified via --service.

   Usage examples:
       cookbook sre.dns.admin depool eqiad # [depools eqiad for everything]
       cookbook sre.dns.admin pool magru   # [pools magru for everything]
       cookbook sre.dns.admin --service upload-addrs -- depool codfw      # [depool codfw for upload-addrs]
       cookbook sre.dns.admin depool esams --service text-addrs text-next # [depool esams for text*]

Flags

action
The kind of action to perform (pool, depool, or show).
valid arguments are: pool, depool, show
site
The site/DC on which to perform the action on.
-s, --service
The service in the site/DC on which the action should be performed.
-r, --reason
An optional reason for the action.
-t, --task-id
An optional Phabricator task ID to log the action.
-f, --force
If passed, do not prompt for any actions (default: prompt)
--emergency-depool-policy
If passed, override the depool threshold and ignore all depool safety checks


roll-reboot.py

Purpose

Rolling reboot of DNS hosts identified by the cumin alias A:dnsbox.

How to Use

Rolling reboot of DNS hosts. 

   This cookbook is for the rolling reboots of the DNS hosts, referred to by
   the cumin alias A:dnsbox. This covers both the DNS rec and auth hosts since
   a given DNS box serves both roles.

   Example usage:
       cookbook sre.dns.roll-reboot                 --alias 'A:dnsbox'                 --task-id T12345                 --reason 'Restarting DNS host' 
       cookbook sre.dns.roll-reboot                 --alias 'A:dnsbox'                 --task-id T12345                 --reason 'Restarting DNS host'                 --grace-sleep 900


roll-restart.py

Purpose

Rolling service restart of pdns-recursor and HAProxy on A:dnsbox

How to Use

Rolling restart of pdns-recursor and HAProxy on A:dnsbox. 

   Example usage:
       cookbook sre.dns.roll-restart             --query 'A:dnsbox and not P{dns1004*}'             --reason "Scheduled maintenance"             restart_daemons

Flags

service
The service to restart
valid arguments are: authdns-.*


Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Traffic/Cookbooks&oldid=2250500"