Requestctl/Overview

This page explains the basic concepts behind how the requestctl tool manages configuration to control access and routing of web requests.

Overview of http request control configuration

The requestctl tool modifies configuration data that resides in the main Etcd cluster. The configs are:

• Written/modified by conftool: in production, Conftool2git synchronizes requestctl objects to a git repository on the puppet servers under /srv/git/conftool/auditlog.
• Read from confd;
• Translated to VCL, which gets loaded by Varnish.

The configuration stored in etcd loads the following into the Varnish configuration:

• The list of pooled ats backends to connect to in the same datacenter
• The list of IP ranges for every public cloud, in the form of a netmapper file
• A list of rate-limiting/ban rules for incoming traffic

How requestctl modifies configuration

requestctl modifies data that resides under /conftool/v1/request-{ipblock,action,pattern}s/. requestctl enables you to manage the configuration for:

• IP ranges for every public cloud
• Rate-limiting/ban rules for incoming traffic

When you run the requestctl commit command, the tool automatically generates a derived data type for each type of action, called vcl and haproxy_dsl respectively. This derived object is then injected into Varnish and haproxy.

A given Varnish host has:

• One VCL condition per enabled action defined for the cluster the host is in.
  • The action is either enabled for all datacenters, or it’s enabled for the specific datacenter the host is in.
• A netmapper file containing all the ipblock entries defined under the cloud scope
  • Possibly more netmapper files for things like crawlers.
• A VCL list of ACLs, one for each ipblock entry under the abuse scope

haproxy has:

• One or multiple ACL definitions per pattern/ipblock used in an enabled action
• A map file for each ipblock scope.
• One http-request condition per haproxy_action that is enabled

On all cache proxy servers, a Confd instance watches these keyspaces and generates the following files:

• /var/netmapper/public_clouds.json from the data at /conftool/v1/request-ipblocks/cloud/, using this template. This netmap is then used to add an X-Public-Cloud: <name> header to requests coming from any IP address in those ranges. This data is updated daily using a script that runs on the puppetservers.
• /etc/varnish/blocked-nets.inc.vcl from the data at /conftool/v1/request-ipblocks/abuse/, using another template. This generates a list of varnish acls that can be referenced later in the VCL. This data is currently duplicated from the private puppet hiera, and will need to be kept in sync somehow.
• /etc/varnish/requestctl-filters.inc.vcl from the data at /conftool/v1/request-vcl/cache-$CLUSTER/{global,$datacenter}, where vcl gets stored when we run requestctl commit. This code gets injected directly in the cluster_fe_ratelimit VCL subroutine, so it only applies to cache misses at the moment (see: phab:T317794).

Related documentation

• requestctl user guide
• requestctl tutorials