Bits.wikimedia.org/Varnish testing

From Wikitech
This page contains historical information. It may be outdated or unreliable. 2010

Few notes:

  • We hit a bug where all threads are writing to acceptor pipe, but acceptor thread doesn't seem to pick that up
  • Originally thought as 2.6.24 kernel problem, a 2.6.32.3 was deployed, but still got same problem
  • This happens with both poll and epoll acceptors (managed to hit it much earlier with poll acceptor, may be coincidence)
  • Currently we are running:
    • varnish 2.0-branch, Feb15 build (standard configure options)
    • Following sysctl.conf additional changes loaded:
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.ipv4.tcp_fin_timeout = 3
net.core.netdev_max_backlog = 30000
net.ipv4.tcp_no_metrics_save=1
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
    • ulimit -s 128
    • ulimit -n 500000
    • varnishd -n /dev/shm -smalloc,1G -f /usr/local/etc/varnish/bits.vcl -T 127.0.0.1:6000 -w 2000 -a 0.0.0.0:80 -p thread_pool_add_delay=1 -p send_timeout=30 -p listen_depth=4096

Apparently when varnish reaches 100% cpu usage, accept thread gets into a state where it leaks worker threads. poll() reaches 100% cpu sooner than epoll, so effect visible way earlier. As sq1 is single-cpu/single-core, high ratio of context switches consumes way more CPU resources, than it would in multithreaded environment - this is why we saw much better scalability on multi-core machines.

Currently bits.pmtpa is being handled by db19, which is handling 14000 requests/s with 180% cpu load and ~100MByte/s traffic.

db19 currently is connected via bond0 over eth0/eth1 directly to core switch.

VCL:

backend default {
.host = "10.2.1.1";
.port = "80";
}

sub vcl_recv {
    if (req.request != "GET" && req.request != "HEAD") {
        /* We only deal with GET and HEAD by default */
        error 403 "this is readonly domain";
    }
    if (req.http.host != "bits.wikimedia.org") { 
	error 403 "bad bad, very bad request"; 
    }
    return (lookup);
}
sub vcl_error {
    set obj.http.Content-Type = "text/html; charset=utf-8";
    synthetic {"
<!DOCTYPE html>
<html>
  <head>
    <title>"} obj.status " " obj.response {"</title>
  </head>
  <body>
    <h1>Error "} obj.status " " obj.response {"</h1>
    <p>We didn't feel like serving it</p>
  </body>
</html>
"};
    return (deliver);
}