User:FFurnari-WMF/HaproxyAWSLC

Procedure [draft]

This procedure is meant to be a checklist to review before start testing the differences between OpenSSL 3.5, OpenSSL 1.1.1 (current baseline) and AWS-LC in terms of performance and eventual unexpected issues. The procedure focuses on using the different TLS libraries with HAProxy 3.2.

Hosts preparation

Preliminary steps

• Depool and silence 2 hosts to use as server and client
  • cp7001.magru.wmnet and cp7009.magru.wmnet are on the same rack and can be used for this task
  • From now on the two hosts will be labeled as [client] and [server]
  • Puppet agent must be disabled on [server] to avoid overriding local modifications.

Packages and libraries

Copy the packages and libraries on [server] (see below for package building)

Packages and binaries listed below are also present on apt1002.wikimedia.org:/home/fabfur/tls-tests/

Nomenclature for HAProxy package is <haproxy_deb_version>_identifier_<...> where identifier can be os111 for packages built against OpenSSL 1.1.1, os35 for packages built against OpenSSL 3.5 and awslc for packages built against AWS-LC libraries

• OpenSSL 3.5 packages

libssl3t64_3.5.1-1_amd64.deb
libssl3t64-dbgsym_3.5.1-1_amd64.deb
libssl-dev_3.5.1-1_amd64.deb
openssl_3.5.1-1_amd64.deb
openssl-dbgsym_3.5.1-1_amd64.deb
openssl-provider-legacy_3.5.1-1_amd64.deb
openssl-provider-legacy-dbgsym_3.5.1-1_amd64.deb

• Haproxy 3.2 compiled against OpenSSL 1.1.1:

haproxy-dbgsym_3.2.4-1.1os111_amd64.deb
haproxy_3.2.4-1.1os111_amd64.deb

• Haproxy 3.2 compiled against OpenSSL 3.5

haproxy_3.2.4-1.1os35_amd64.deb
haproxy-dbgsym_3.2.4-1.1os35_amd64.deb

• Haproxy 3.2 compiled against AWS-LC

haproxy_3.2.4-1awslc_amd64.deb
haproxy-dbgsym_3.2.4-1awslc_amd64.deb

• AWS-LC libraries aren't packaged (yet) but must be copied in /opt/awslc/{bin,include,lib} path

AWS-LC is distributed as sources and must be compiled in advance (see instructions below). If the test results will indicate superior performance compared to the other libaries we will create a specific debian package for this (not really useful for the test context)

|-- bin
|   |-- bssl
|   |-- c_rehash
|   `-- openssl
|-- include
|   `-- openssl
`-- lib
    |-- crypto
    |-- libcrypto.so
    |-- libssl.so
    |-- pkgconfig
    `-- ssl

HAProxy configuration

• On [server] HAProxy can reuse the same certificates and tls-ticket keys as usual, but configuration can be vastly simplified for the testing. Also, logs shouldn't be sent to HaproxyKafka socket or we'll pollute analytics data. A minimal configuration file like this can be copied over /etc/haproxy/haproxy.cfg

The following configuration is vastly simplified, keeping only the TLS parameters in common with the production one, as that's the only thing we're interested in testing in this case

global
    user haproxy
    group haproxy
    stats socket /run/haproxy/haproxy.sock mode 600 expose-fd listeners level admin
    log /var/lib/haproxy/dev/log local0 info
    #log /var/run/haproxykafka/haproxykafka.sock len 8192 format rfc5424 local0 info
    tune.http.logurilen 2048
    # do not keep old processes longer than 5m after a reload
    hard-stop-after 5m
    set-dumpable
    nbthread 48
    cpu-map 1/1- 0 48 2 50 4 52 6 54 8 56 10 58 12 60 14 62 16 64 18 66 20 68 22 70 24 72 26 74 28 76 30 78 32 80 34 82 36 84 38 86 40 88 42 90 44 92 46 94

    #lua-load-per-thread /etc/haproxy/lua/maxmind-lookup.lua

    ssl-default-bind-options ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3
    ssl-default-bind-ciphers -ALL:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384
    ssl-dh-param-file /etc/ssl/dhparam.pem
    tune.ssl.cachesize 512000
    tune.ssl.lifetime 86400
    maxconn 200000


    tune.h2.header-table-size 4096
    tune.h2.initial-window-size 65535
    tune.h2.max-concurrent-streams 100

defaults
    mode       http

    log-format "%rt %Tr %Tw %Tc %ST {%[capture.req.hdr(0)]} {%[capture.res.hdr(0)]} %ts"
    #log-format-sd %{+E}o\ [haproxykafka@0\ server_pid=\"%pid\"\ ip=\"%ci\"\ sequence=\"%ID\"\ dt=\"%tr\"\ time_backend_response=\"%Tr\"\ http_status=\"%ST\"\ response_size=\"%B\"\ termination_state=\"%ts\"\ uri_host=\"%[capture.req.hdr(0)]\"\ referer=\"%[capture.req.hdr(1)]\"\ user_agent=\"%[capture.req.hdr(2)]\"\ accept_language=\"%[capture.req.hdr(3)]\"\ range=\"%[capture.req.hdr(4)]\"\ accept=\"%[capture.req.hdr(5)]\"\ tls=\"%[capture.req.hdr(6)]\"\ cache_status=\"%[var(txn.x_cache_status)]\"\ content_type=\"%[var(txn.content_type)]\"\ x_analytics=\"%[var(txn.x_analytics)]\"\ x_cache=\"%[var(txn.x_cache)]\"\ backend=\"%[var(txn.server)]\"\ http_method=\"%HM\"\ uri_path=\"%HPO\"\ uri_query=\"%HQ\"]

    option     httplog
    option     dontlognull
    option     accept-invalid-http-request
    option     accept-invalid-http-response
    option     http-ignore-probes
    retries    1
    timeout    connect 50000
    timeout    client 500000
    timeout    server 500000

frontend http
    # Just for test
    bind :80
    mode http
    default_backend ok

listen https
    log global
    maxconn 199000
    bind :443 tfo ssl crt-list /etc/haproxy/crt-list.cfg tls-ticket-keys /run/haproxy-secrets/stek.keys
    bind :::443 tfo v6only ssl crt-list /etc/haproxy/crt-list.cfg tls-ticket-keys /run/haproxy-secrets/stek.keys
    timeout http-request 3600s
    timeout http-keep-alive 120s
    timeout client 120s
    timeout client-fin 120s
    timeout connect 3s
    timeout server 180s
    timeout tunnel 3600s

    default_backend ok

backend ok
    http-request return status 200 content-type "text/plain" string "OK"

Metrics and logs

• Stop and disable haproxykafka.service before starting test to avoid polluting analytics data
• Edit mtail configuration to avoid polluting SLO metrics:

@@ -10,15 +10,11 @@
 histogram haproxy_client_ttfb by cache_status, http_status_family buckets -1, 0.001, 0.005, 0.01, 0.02, 0.045, 0.07, 0.1, 0.15, 0.25, 0.35, 0.5, 0.75, 1.2, 3.0, 10.0, 30.0, 60.0
 histogram haproxy_client_healthcheck_ttfb by cache_status, http_status_family buckets -1, 0.001, 0.005, 0.01, 0.02, 0.045, 0.07, 0.1, 0.15, 0.25, 0.35, 0.5, 0.75, 1.2, 3.0, 10.0, 30.0, 60.0
 counter haproxy_termination_states_total by termination_state
-counter haproxy_sli_total
-counter haproxy_sli_good
-counter haproxy_sli_bad
 hidden text cstatus
 hidden gauge process_time
 hidden text http_status_family
 
 / \d+ (?P<client_ttfb>\-?\d+) (?P<queue_time>\-?\d+) (?P<server_connection_time>\-?\d+) (?P<http_status_family>[1-5|\-])(1|\d\d) {(?P<host>[0-9A-Za-z\-\.:]+)} {(?P<cache_status>[a-z-]*)} (?P<termination_state>[A-Za-z-]{2})$/ {
-  haproxy_sli_total++
   process_time = 0
 
   $http_status_family =~ /^\-/ {
@@ -46,16 +42,4 @@
   $server_connection_time > 0 {
     process_time += $server_connection_time
   }
-  # We are excluding the following states:
-  # More details on http://docs.haproxy.org/2.6/configuration.html#8.5
-  # R --> Resource on the proxy has been exhausted
-  # I --> Internal error
-  # D --> Session killed by HAProxy
-  # U --> Session killed by HAProxy (this shouldn't happen here)
-  # K --> Session actively killed by an admin operating on HAProxy (HAProxy config/TLS material reloads would trigger this one)
-  $termination_state =~ /^[\-CSPLcs]/ && process_time < 50 {
-    haproxy_sli_good++
-  } else {
-    haproxy_sli_bad++
-  }
 }

• Restart the haproxy-mtail@tls.service

Client preparation

• The only needed setup on the [client] should be depooling and copying the required scripts for benchmarking
• Benchmarking tools can be found at https://gitlab.wikimedia.org/fabfur/benchmark-scripts

Running the benchmarks

• Overall procedure:
  • OpenSSL 1.1.1
  • Install HAProxy 3.2 os111 (compiled against OpenSSL 1.1.1) on [server] using the debian packages (haproxy-dbgsym_3.2.4-1.1os111_amd64.deb haproxy_3.2.4-1.1os111_amd64.deb)

This will use OpenSSL 1.1.1 version that is already installed on Bullseye, so no need to install other packages

• • Ensure the HAProxy version and compilation options are the expected one (haproxy -vv, openssl version)
  • Ensure the HAProxy configuration is the "custom" one
  • Start HAProxy with the systemd unit shipped (already present on Debian Bullseye cache hosts)
  • Check HAProxy's journal log for eventual errors (leave it open)
  • Gather and save metrics on [server]
    • Start perf top on [server] to gather system metrics for later inspection
    • Start perf record -F 99 -p <HAPROXY_PID> -g on [server] and stop it after each benchmark
  • Start benchmarks on [client]
    • Run curl benchmark script:
      • To run the script on the same NUMA node as the NIC:
        • (example) cat /sys/class/net/eno12399np0/device/numa_node
        • (example) numactl --cpunodebind=0 --membind=0 ./benchmark-curl.py --target 10.140.0.7 --num-requests 1000000 --concurrency 1000
        • Repeat for concurrency = 10_000 and 100_000
    • Run TLS benchmark script:
      • Same procedure as above to run the script on the same NUMA node as the nic
      • numactl --cpunodebind=0 --membind=0 ./tls-tester --endpoint 10.140.0.7 --requests 1000000 --goroutines 1000
      • Repeat for concurrency = 10_000 and 100_000
  • After benchmark
    • Copy perf result files locally
  • Remove HAProxy 3.2 os111 package from [server]
  • OpenSSL 3.5
  • Install OpenSSL 3.5 packages over existing ones (upgrade)
  • Install HAProxy 3.2 os3.5 (compiled against OpenSSL 3.5) packages
  • Ensure HAProxy and OpenSSL version are the correct one, restart haproxy systemd unit
  • Start benchmarks on the client as above and gather metrics
  • Remove HAProxy 3.2 os3.5 packages from [server]
  • ** AWS-LC
  • Copy (if not alreay) awslc binaries on [server]/opt (as described above)
  • Install HAProxy 3.2 awslc (compiled against AWS-LC) packages
  • Ensure HAProxy and OpenSSL version are the correct one, restart haproxy systemd unit
  • Start benchmarks on the client as above and gather metrics

Cleanup

• Repool [client]
• Reimage [server] and repool it

The following instructions could be outdated. Use them as reference to build the various packages if needed

Build HAProxy 3.2, AWSLC and OpenSSL 3.5 on Debian Bullseye

The following has been performed on a Debian Bullseye container created with this Dockerfile:

FROM docker-registry.wikimedia.org/bullseye:latest

ENV container=docker
ENV LC_ALL=C
ENV DEBIAN_FRONTEND=noninteractive

WORKDIR /opt


RUN apt-get update && apt-get install -y libpcre2-dev libjemalloc-dev python3-sphinx zlib1g-dev build-essential devscripts libssl-dev liblua5.4-dev python3-mako cmake libsystemd-dev pkgconf debhelper libsystemd-dev
RUN apt-get install -y curl wget vim
# RUN apt-get install -y systemd-dev libopentracing-c-wrapper-dev
RUN echo "deb http://apt.wikimedia.org/wikimedia bullseye-wikimedia component/golang" > /etc/apt/sources.list.d/wikimedia.list && apt-get update && apt-get install -y golang-1.23
RUN update-alternatives --install /usr/bin/go go /usr/lib/go-1.23/bin/go 3 --slave /usr/bin/gofmt gofmt /usr/lib/go-1.23/bin/gofmt

WORKDIR /opt
VOLUME /opt

Build AWSLC

Use the following script to fetch, build and install AWSLC on /opt/awslc (standard path used by others). Run it on the Debian Bullseye container (mount /opt locally for easy debugging and copying).

#!/bin/bash

set -e

CODE_PATH=/tmp/aws-lc
BUILD_PATH=/tmp/aws-lc-build
INSTALL_PATH=/opt/awslc

echo "[*] fetching dependencies..."
apt-get update
apt-get install -y git cmake ninja-build perl clang perl tree
echo

echo "[*] cloning awslc in ${CODE_PATH}"
if [[ -d ${CODE_PATH} ]]; then
    echo "[!] removing awslc code in ${CODE_PATH}"
    rm -fr "${CODE_PATH}"
fi
git clone --depth=1 https://github.com/aws/aws-lc.git "${CODE_PATH}"
echo

echo "[*] building awslc (build path: ${BUILD_PATH}, destination: ${INSTALL_PATH})"
if [[ -d ${BUILD_PATH} ]]; then
    echo "[!] removing awslc build dir ${BUILD_PATH}"
    rm -fr "${BUILD_PATH}"
fi
mkdir -p "${BUILD_PATH}"
cd "${BUILD_PATH}"
cmake -GNinja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_INSTALL_PREFIX="${INSTALL_PATH}" -DBUILD_SHARED_LIBS=1 "${CODE_PATH}"
echo

# echo "[*] running tests"
# ninja run_tests
# echo

echo "[*] installing into ${INSTALL_PATH}"
ninja install
tree -L 2 "${INSTALL_PATH}"
echo

echo "[*] awslc installed in ${INSTALL_PATH}"

Build HAProxy 3.2 with OpenSSL 1.1.1

Run it on the Debian Bullseye container (mount /opt locally for easy debugging and copying).

#!/bin/bash

set -e

HAPROXY_SRC_PATH=/opt/haproxy-openssl-1.1.1
HAPROXY_DEB_PATH="$HAPROXY_SRC_PATH/haproxy-3.2.3"

if [[ ! -d $HAPROXY_DEB_PATH ]]; then
    cd $HAPROXY_DEB_PATH
    dget -u http://deb.debian.org/debian/pool/main/h/haproxy/haproxy_3.2.3-2.dsc
    cd "$HAPROXY_DEB_PATH"
    dpkg-buildpackage -b
else
    echo "[*] $HAPROXY_DEB_PATH already exists"
    cd "$HAPROXY_DEB_PATH"
    fakeroot debian/rules clean
    dpkg-buildpackage -b
    echo
fi

cd "$HAPROXY_DEB_PATH"/

Build HAProxy 3.2 with AWSLC

The patched code for HAProxy to use AWSLC can be found at https://gitlab.wikimedia.org/fabfur/haproxy-awslc/-/tree/awslc-3.2/debian?ref_type=heads

The following script (to be run on the Debian Bullseye container) just clones it and build the package.

#!/bin/bash

set -e

HAPROXY_DEB_PATH=/tmp/haproxy-deb
HAPROXY_REPO=https://gitlab.wikimedia.org/fabfur/haproxy-awslc.git
HAPROXY_AWSLC_BRANCH=awslc-3.2
# Standard path for aws-lc
AWSLC_PATH=/opt/awslc

echo "[*] fetching dependencies ..."
apt-get update
apt-get install -y devscripts git
echo

echo "[*] downloading haproxy source package in ${HAPROXY_DEB_PATH}"
if [[ -d "${HAPROXY_DEB_PATH}" ]]; then
    echo "[!] removing ${HAPROXY_DEB_PATH}"
    rm -fr "${HAPROXY_DEB_PATH}"
fi
mkdir "${HAPROXY_DEB_PATH}"
cd "${HAPROXY_DEB_PATH}"
git clone --depth=1 --branch "$HAPROXY_AWSLC_BRANCH" "${HAPROXY_REPO}" "${HAPROXY_DEB_PATH}/haproxy-awslc"
echo

echo "[*] Building haproxy binary package against awslc (${AWSLC_PATH})"
cd "${HAPROXY_DEB_PATH}/haproxy-awslc"
dpkg-buildpackage -b
cd ..
echo "[*] Debian packages in ${HAPROXY_DEB_PATH}"
echo

echo "[*] Verify binaries dependencies"
ldd "${HAPROXY_DEB_PATH}/haproxy-awslc/debian/haproxy/usr/sbin/haproxy"
echo

Build OpenSSL 3.5 on Debian Bullseye

The following script can be used as guideline to build OpenSSL and related libraries/debian packages on a Debian Bullseye.

Note that the patched repository for this lives in https://gitlab.wikimedia.org/fabfur/openssl#

#!/bin/bash

set -e

OPENSSL_DEB_PATH=/tmp/openssl-deb
OPENSSL_REPO=https://gitlab.wikimedia.org/fabfur/openssl.git

echo "[*] fetching dependencies ..."
apt-get update
apt-get install quilt
echo

echo "[*] downloading openssl repo in ${OPENSSL_DEB_PATH}"
if [[ -d "{OPENSSL_DEB_PATH}" ]]; then
    echo "[!] removing ${OPENSSL_DEB_PATH}"
    rm -fr "${OPENSSL_DEB_PATH}"
fi
mkdir "${OPENSSL_DEB_PATH}"
cd "${OPENSSL_DEB_PATH}"
git clone --depth=1 $OPENSSL_REPO "${OPENSSL_DEB_PATH}/openssl-3.5"
echo

echo "[*] building openssl 3.5 ..."
cd "${OPENSSL_DEB_PATH}/openssl-3.5"
quilt push -a
dpkg-buildpackage -b
cd ..
echo "[*] Debian packages in ${OPENSSL_DEB_PATH}"
echo