Requestctl/Overview
This page explains the basic concepts behind how the requestctl tool manages configuration to control access and routing of web requests.
Overview of http request control configuration
The requestctl tool modifies configuration data that resides in the main Etcd cluster. The configs are:
- Written/modified by conftool: in production, Conftool2git synchronizes requestctl objects to a git repository on the puppet servers under
/srv/git/conftool/auditlog
. - Read from confd;
- Translated to VCL, which gets loaded by Varnish.
The configuration stored in etcd loads the following into the Varnish configuration:
- The list of pooled ats backends to connect to in the same datacenter
- The list of IP ranges for every public cloud, in the form of a netmapper file
- A list of rate-limiting/ban rules for incoming traffic
How requestctl modifies configuration
requestctl modifies data that resides under /conftool/v1/request-{ipblock,action,pattern}s/
. requestctl enables you to manage the configuration for:
- IP ranges for every public cloud
- Rate-limiting/ban rules for incoming traffic
When you run the requestctl commit
command, the tool automatically generates a derived data type for each type of action, called vcl
and haproxy_dsl
respectively. This derived object is then injected into Varnish and haproxy.
A given Varnish host has:
- One VCL condition per enabled
action
defined for the cluster the host is in.- The action is either enabled for all datacenters, or it’s enabled for the specific datacenter the host is in.
- A netmapper file containing all the
ipblock
entries defined under the cloud scope- Possibly more netmapper files for things like crawlers.
- A VCL list of ACLs, one for each
ipblock
entry under the abuse scope
haproxy has:
- One or multiple ACL definitions per pattern/ipblock used in an enabled action
- A map file for each
ipblock
scope. - One
http-request
condition perhaproxy_action
that is enabled
On all cache proxy servers, a Confd instance watches these keyspaces and generates the following files:
- /var/netmapper/public_clouds.json from the data at /conftool/v1/request-ipblocks/cloud/, using this template. This netmap is then used to add an
X-Public-Cloud: <name>
header to requests coming from any IP address in those ranges. This data is updated daily using a script that runs on the puppetservers. - /etc/varnish/blocked-nets.inc.vcl from the data at /conftool/v1/request-ipblocks/abuse/, using another template. This generates a list of varnish acls that can be referenced later in the VCL. This data is currently duplicated from the private puppet hiera, and will need to be kept in sync somehow.
- /etc/varnish/requestctl-filters.inc.vcl from the data at
/conftool/v1/request-vcl/cache-$CLUSTER/{global,$datacenter}
, where vcl gets stored when we runrequestctl commit
. This code gets injected directly in thecluster_fe_ratelimit
VCL subroutine, so it only applies to cache misses at the moment (see: phab:T317794).