User:FFurnari-WMF/HaproxyAWSLC

Performance comparison between HAProxy 3.2 + OpenSSL 3.5 vs HAProxy 3.2 + AWS-LC (Debian Trixie)

Host setup

Using cp2041 and cp2042 to host, respectively, HAProxy 3.2 + OpenSSL 3.5 and HAProxy 3.2 + AWS-LC. These hosts are depooled and waiting to be decommissioned.
On both hosts:
- Puppet has been disabled
- HaproxyKafka service has been stopped and disabled to avoid polluting metrics
- haproxy-mtail service has been stopped and disabled to avoid polluting metrics
- Installed (manually) python3.13-venv to run benchmark script
- Fetched benchmark script from GitLab repo
- Created a Python virtualenv and installed requirements for the script above
- Compiled a curl version against OpenSSL 1.1.1 to test the impact of OpenSSL version on the client side and not be bound on the (slowness?) of the OpenSSL version to pollute the results.
On cp2041:
- Before disabling puppet, installed HAProxy 3.2 (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1254195)

On cp2042:

Installed haproxy-awslc (official HAProxy performance packages), haproxy-awslc, libssl-awslc

Changed HAProxy configuration to adapt to AWS-LC:

@@ -26,7 +26,7 @@
     ssl-default-bind-options ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3
     ssl-default-bind-ciphers -ALL:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256
     ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384
-    #ssl-dh-param-file /etc/ssl/dhparam.pem
+    ssl-dh-param-file /etc/ssl/dhparam.pem
     tune.ssl.cachesize 512000
     tune.ssl.lifetime 86400
     maxconn 200000

Set /etc/default/haproxy back to using EXTRAOPTS (overridden by haproxy-awslc package?): EXTRAOPTS="-f /etc/haproxy/conf.d"
Created /etc/systemd/system/haproxy.service (copied from other hosts' /lib/systemd/system/haproxy.service)

Run the benchmark script

On both hosts: remember to unset the envvars: HTTP_PROXY, HTTPS_PROXY, http_proxy, https_proxy, NO_PROXY, no_proxy
Number of requests issued is always 10M, with fixed concurrency at 10k concurrent requests.
Run the test from the "other" host:
- From cp2041 test HAProxy + AWSLC with both curl + OpenSSL 3.5 and curl + OpenSSL 1.1.1:
  - benchmark-curl.py --concurrency 10000 --target https://upload.wikimedia.org/nonexistant --resolve upload.wikipedia.org:443:10.192.48.157 --num-requests 10000000 --http-version 2 --header nonexistant --curl /home/fabfur/curl-1.1.1 --label "client: curl (openssl 1.1.1), server: haproxy (awslc)"
  - benchmark-curl.py --concurrency 10000 --target https://upload.wikimedia.org/nonexistant --resolve upload.wikipedia.org:443:10.192.48.157 --num-requests 10000000 --http-version 2 --header nonexistant --label "client: curl (openssl 3.5), server: haproxy (awslc)"
- From cp2042 test HAProxy + OpenSSL 3.5 with both curl + OpenSSL 3.5 and curl + OpenSSL 1.1.1:
  - benchmark-curl.py --concurrency 10000 --target https://en.wikipedia.org/nonexistent --resolve en.wikipedia.org:443:10.192.48.156 --num-requests 10000000 --http-version 2 --header 'nonexistant' --curl /home/fabfur/curl-1.1.1 --label "client: curl (openssl 1.1.1), server: haproxy (openssl 3.5)"
  - benchmark-curl.py --concurrency 10000 --target https://en.wikipedia.org/nonexistent --resolve en.wikipedia.org:443:10.192.48.156 --num-requests 10000000 --http-version 2 --header 'nonexistant'--label "client: curl (openssl 3.5), server: haproxy (openssl 3.5)"

Results:

Old procedure

OLD

Procedure [draft]

This procedure is meant to be a checklist to review before start testing the differences between OpenSSL 3.5, OpenSSL 1.1.1 (current baseline) and AWS-LC in terms of performance and eventual unexpected issues. The procedure focuses on using the different TLS libraries with HAProxy 3.2.

Hosts preparation

Preliminary steps

Depool and silence 2 hosts to use as server and client
- cp7001.magru.wmnet and cp7009.magru.wmnet are on the same rack and can be used for this task
- From now on the two hosts will be labeled as [client] and [server]
- Puppet agent must be disabled on [server] to avoid overriding local modifications.

Packages and libraries

Copy the packages and libraries on [server] (see below for package building)

Packages and binaries listed below are also present on apt1002.wikimedia.org:/home/fabfur/tls-tests/

Nomenclature for HAProxy package is <haproxy_deb_version>_identifier_<...> where identifier can be os111 for packages built against OpenSSL 1.1.1, os35 for packages built against OpenSSL 3.5 and awslc for packages built against AWS-LC libraries

OpenSSL 3.5 packages

libssl3t64_3.5.1-1_amd64.deb
libssl3t64-dbgsym_3.5.1-1_amd64.deb
libssl-dev_3.5.1-1_amd64.deb
openssl_3.5.1-1_amd64.deb
openssl-dbgsym_3.5.1-1_amd64.deb
openssl-provider-legacy_3.5.1-1_amd64.deb
openssl-provider-legacy-dbgsym_3.5.1-1_amd64.deb

Haproxy 3.2 compiled against OpenSSL 1.1.1:

haproxy-dbgsym_3.2.4-1.1os111_amd64.deb
haproxy_3.2.4-1.1os111_amd64.deb

Haproxy 3.2 compiled against OpenSSL 3.5

haproxy_3.2.4-1.1os35_amd64.deb
haproxy-dbgsym_3.2.4-1.1os35_amd64.deb

Haproxy 3.2 compiled against AWS-LC

haproxy_3.2.4-1awslc_amd64.deb
haproxy-dbgsym_3.2.4-1awslc_amd64.deb

AWS-LC libraries aren't packaged (yet) but must be copied in /opt/awslc/{bin,include,lib} path

AWS-LC is distributed as sources and must be compiled in advance (see instructions below). If the test results will indicate superior performance compared to the other libaries we will create a specific debian package for this (not really useful for the test context)

|-- bin
|   |-- bssl
|   |-- c_rehash
|   `-- openssl
|-- include
|   `-- openssl
`-- lib
    |-- crypto
    |-- libcrypto.so
    |-- libssl.so
    |-- pkgconfig
    `-- ssl

HAProxy configuration

On [server] HAProxy can reuse the same certificates and tls-ticket keys as usual, but configuration can be vastly simplified for the testing. Also, logs shouldn't be sent to HaproxyKafka socket or we'll pollute analytics data. A minimal configuration file like this can be copied over /etc/haproxy/haproxy.cfg

The following configuration is vastly simplified, keeping only the TLS parameters in common with the production one, as that's the only thing we're interested in testing in this case

global
    user haproxy
    group haproxy
    stats socket /run/haproxy/haproxy.sock mode 600 expose-fd listeners level admin
    log /var/lib/haproxy/dev/log local0 info
    #log /var/run/haproxykafka/haproxykafka.sock len 8192 format rfc5424 local0 info
    tune.http.logurilen 2048
    # do not keep old processes longer than 5m after a reload
    hard-stop-after 5m
    set-dumpable
    nbthread 48
    cpu-map 1/1- 0 48 2 50 4 52 6 54 8 56 10 58 12 60 14 62 16 64 18 66 20 68 22 70 24 72 26 74 28 76 30 78 32 80 34 82 36 84 38 86 40 88 42 90 44 92 46 94

    #lua-load-per-thread /etc/haproxy/lua/maxmind-lookup.lua

    ssl-default-bind-options ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3
    ssl-default-bind-ciphers -ALL:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384
    ssl-dh-param-file /etc/ssl/dhparam.pem
    tune.ssl.cachesize 512000
    tune.ssl.lifetime 86400
    maxconn 200000


    tune.h2.header-table-size 4096
    tune.h2.initial-window-size 65535
    tune.h2.max-concurrent-streams 100

defaults
    mode       http

    log-format "%rt %Tr %Tw %Tc %ST {%[capture.req.hdr(0)]} {%[capture.res.hdr(0)]} %ts"
    #log-format-sd %{+E}o\ [haproxykafka@0\ server_pid=\"%pid\"\ ip=\"%ci\"\ sequence=\"%ID\"\ dt=\"%tr\"\ time_backend_response=\"%Tr\"\ http_status=\"%ST\"\ response_size=\"%B\"\ termination_state=\"%ts\"\ uri_host=\"%[capture.req.hdr(0)]\"\ referer=\"%[capture.req.hdr(1)]\"\ user_agent=\"%[capture.req.hdr(2)]\"\ accept_language=\"%[capture.req.hdr(3)]\"\ range=\"%[capture.req.hdr(4)]\"\ accept=\"%[capture.req.hdr(5)]\"\ tls=\"%[capture.req.hdr(6)]\"\ cache_status=\"%[var(txn.x_cache_status)]\"\ content_type=\"%[var(txn.content_type)]\"\ x_analytics=\"%[var(txn.x_analytics)]\"\ x_cache=\"%[var(txn.x_cache)]\"\ backend=\"%[var(txn.server)]\"\ http_method=\"%HM\"\ uri_path=\"%HPO\"\ uri_query=\"%HQ\"]

    option     httplog
    option     dontlognull
    option     accept-invalid-http-request
    option     accept-invalid-http-response
    option     http-ignore-probes
    retries    1
    timeout    connect 50000
    timeout    client 500000
    timeout    server 500000

frontend http
    # Just for test
    bind :80
    mode http
    default_backend ok

listen https
    log global
    maxconn 199000
    bind :443 tfo ssl crt-list /etc/haproxy/crt-list.cfg tls-ticket-keys /run/haproxy-secrets/stek.keys
    bind :::443 tfo v6only ssl crt-list /etc/haproxy/crt-list.cfg tls-ticket-keys /run/haproxy-secrets/stek.keys
    timeout http-request 3600s
    timeout http-keep-alive 120s
    timeout client 120s
    timeout client-fin 120s
    timeout connect 3s
    timeout server 180s
    timeout tunnel 3600s

    default_backend ok

backend ok
    http-request return status 200 content-type "text/plain" string "OK"

Metrics and logs

Stop and disable haproxykafka.service before starting test to avoid polluting analytics data
Edit mtail configuration to avoid polluting SLO metrics:

@@ -10,15 +10,11 @@
 histogram haproxy_client_ttfb by cache_status, http_status_family buckets -1, 0.001, 0.005, 0.01, 0.02, 0.045, 0.07, 0.1, 0.15, 0.25, 0.35, 0.5, 0.75, 1.2, 3.0, 10.0, 30.0, 60.0
 histogram haproxy_client_healthcheck_ttfb by cache_status, http_status_family buckets -1, 0.001, 0.005, 0.01, 0.02, 0.045, 0.07, 0.1, 0.15, 0.25, 0.35, 0.5, 0.75, 1.2, 3.0, 10.0, 30.0, 60.0
 counter haproxy_termination_states_total by termination_state
-counter haproxy_sli_total
-counter haproxy_sli_good
-counter haproxy_sli_bad
 hidden text cstatus
 hidden gauge process_time
 hidden text http_status_family
 
 / \d+ (?P<client_ttfb>\-?\d+) (?P<queue_time>\-?\d+) (?P<server_connection_time>\-?\d+) (?P<http_status_family>[1-5|\-])(1|\d\d) {(?P<host>[0-9A-Za-z\-\.:]+)} {(?P<cache_status>[a-z-]*)} (?P<termination_state>[A-Za-z-]{2})$/ {
-  haproxy_sli_total++
   process_time = 0
 
   $http_status_family =~ /^\-/ {
@@ -46,16 +42,4 @@
   $server_connection_time > 0 {
     process_time += $server_connection_time
   }
-  # We are excluding the following states:
-  # More details on http://docs.haproxy.org/2.6/configuration.html#8.5
-  # R --> Resource on the proxy has been exhausted
-  # I --> Internal error
-  # D --> Session killed by HAProxy
-  # U --> Session killed by HAProxy (this shouldn't happen here)
-  # K --> Session actively killed by an admin operating on HAProxy (HAProxy config/TLS material reloads would trigger this one)
-  $termination_state =~ /^[\-CSPLcs]/ && process_time < 50 {
-    haproxy_sli_good++
-  } else {
-    haproxy_sli_bad++
-  }
 }

Restart the haproxy-mtail@tls.service

Client preparation

The only needed setup on the [client] should be depooling and copying the required scripts for benchmarking
Benchmarking tools can be found at https://gitlab.wikimedia.org/fabfur/benchmark-scripts

Running the benchmarks

Overall procedure:
- OpenSSL 1.1.1
- Install HAProxy 3.2 os111 (compiled against OpenSSL 1.1.1) on [server] using the debian packages (haproxy-dbgsym_3.2.4-1.1os111_amd64.deb haproxy_3.2.4-1.1os111_amd64.deb)

This will use OpenSSL 1.1.1 version that is already installed on Bullseye, so no need to install other packages

- Ensure the HAProxy version and compilation options are the expected one (haproxy -vv, openssl version)
- Ensure the HAProxy configuration is the "custom" one
- Start HAProxy with the systemd unit shipped (already present on Debian Bullseye cache hosts)
- Check HAProxy's journal log for eventual errors (leave it open)
- Gather and save metrics on [server]
  - Start perf top on [server] to gather system metrics for later inspection
  - Start perf record -F 99 -p <HAPROXY_PID> -g on [server] and stop it after each benchmark
- Start benchmarks on [client]
  - Run curl benchmark script:
    - To run the script on the same NUMA node as the NIC:
      - (example) cat /sys/class/net/eno12399np0/device/numa_node
      - (example) numactl --cpunodebind=0 --membind=0 ./benchmark-curl.py --target 10.140.0.7 --num-requests 1000000 --concurrency 1000
      - Repeat for concurrency = 10_000 and 100_000
  - Run TLS benchmark script:
    - Same procedure as above to run the script on the same NUMA node as the nic
    - numactl --cpunodebind=0 --membind=0 ./tls-tester --endpoint 10.140.0.7 --requests 1000000 --goroutines 1000
    - Repeat for concurrency = 10_000 and 100_000
- After benchmark
  - Copy perf result files locally
- Remove HAProxy 3.2 os111 package from [server]
- OpenSSL 3.5
- Install OpenSSL 3.5 packages over existing ones (upgrade)
- Install HAProxy 3.2 os3.5 (compiled against OpenSSL 3.5) packages
- Ensure HAProxy and OpenSSL version are the correct one, restart haproxy systemd unit
- Start benchmarks on the client as above and gather metrics
- Remove HAProxy 3.2 os3.5 packages from [server]
- ** AWS-LC
- Copy (if not alreay) awslc binaries on [server]/opt (as described above)
- Install HAProxy 3.2 awslc (compiled against AWS-LC) packages
- Ensure HAProxy and OpenSSL version are the correct one, restart haproxy systemd unit
- Start benchmarks on the client as above and gather metrics

Cleanup

Repool [client]
Reimage [server] and repool it

The following instructions could be outdated. Use them as reference to build the various packages if needed

Build HAProxy 3.2, AWSLC and OpenSSL 3.5 on Debian Bullseye

The following has been performed on a Debian Bullseye container created with this Dockerfile:

FROM docker-registry.wikimedia.org/bullseye:latest

ENV container=docker
ENV LC_ALL=C
ENV DEBIAN_FRONTEND=noninteractive

WORKDIR /opt


RUN apt-get update && apt-get install -y libpcre2-dev libjemalloc-dev python3-sphinx zlib1g-dev build-essential devscripts libssl-dev liblua5.4-dev python3-mako cmake libsystemd-dev pkgconf debhelper libsystemd-dev
RUN apt-get install -y curl wget vim
# RUN apt-get install -y systemd-dev libopentracing-c-wrapper-dev
RUN echo "deb http://apt.wikimedia.org/wikimedia bullseye-wikimedia component/golang" > /etc/apt/sources.list.d/wikimedia.list && apt-get update && apt-get install -y golang-1.23
RUN update-alternatives --install /usr/bin/go go /usr/lib/go-1.23/bin/go 3 --slave /usr/bin/gofmt gofmt /usr/lib/go-1.23/bin/gofmt

WORKDIR /opt
VOLUME /opt

Build AWSLC

Use the following script to fetch, build and install AWSLC on /opt/awslc (standard path used by others). Run it on the Debian Bullseye container (mount /opt locally for easy debugging and copying).

#!/bin/bash

set -e

CODE_PATH=/tmp/aws-lc
BUILD_PATH=/tmp/aws-lc-build
INSTALL_PATH=/opt/awslc

echo "[*] fetching dependencies..."
apt-get update
apt-get install -y git cmake ninja-build perl clang perl tree
echo

echo "[*] cloning awslc in ${CODE_PATH}"
if [[ -d ${CODE_PATH} ]]; then
    echo "[!] removing awslc code in ${CODE_PATH}"
    rm -fr "${CODE_PATH}"
fi
git clone --depth=1 https://github.com/aws/aws-lc.git "${CODE_PATH}"
echo

echo "[*] building awslc (build path: ${BUILD_PATH}, destination: ${INSTALL_PATH})"
if [[ -d ${BUILD_PATH} ]]; then
    echo "[!] removing awslc build dir ${BUILD_PATH}"
    rm -fr "${BUILD_PATH}"
fi
mkdir -p "${BUILD_PATH}"
cd "${BUILD_PATH}"
cmake -GNinja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_INSTALL_PREFIX="${INSTALL_PATH}" -DBUILD_SHARED_LIBS=1 "${CODE_PATH}"
echo

# echo "[*] running tests"
# ninja run_tests
# echo

echo "[*] installing into ${INSTALL_PATH}"
ninja install
tree -L 2 "${INSTALL_PATH}"
echo

echo "[*] awslc installed in ${INSTALL_PATH}"

Build HAProxy 3.2 with OpenSSL 1.1.1

Run it on the Debian Bullseye container (mount /opt locally for easy debugging and copying).

#!/bin/bash

set -e

HAPROXY_SRC_PATH=/opt/haproxy-openssl-1.1.1
HAPROXY_DEB_PATH="$HAPROXY_SRC_PATH/haproxy-3.2.3"

if [[ ! -d $HAPROXY_DEB_PATH ]]; then
    cd $HAPROXY_DEB_PATH
    dget -u http://deb.debian.org/debian/pool/main/h/haproxy/haproxy_3.2.3-2.dsc
    cd "$HAPROXY_DEB_PATH"
    dpkg-buildpackage -b
else
    echo "[*] $HAPROXY_DEB_PATH already exists"
    cd "$HAPROXY_DEB_PATH"
    fakeroot debian/rules clean
    dpkg-buildpackage -b
    echo
fi

cd "$HAPROXY_DEB_PATH"/

Build HAProxy 3.2 with AWSLC

The patched code for HAProxy to use AWSLC can be found at https://gitlab.wikimedia.org/fabfur/haproxy-awslc/-/tree/awslc-3.2/debian?ref_type=heads

The following script (to be run on the Debian Bullseye container) just clones it and build the package.

#!/bin/bash

set -e

HAPROXY_DEB_PATH=/tmp/haproxy-deb
HAPROXY_REPO=https://gitlab.wikimedia.org/fabfur/haproxy-awslc.git
HAPROXY_AWSLC_BRANCH=awslc-3.2
# Standard path for aws-lc
AWSLC_PATH=/opt/awslc

echo "[*] fetching dependencies ..."
apt-get update
apt-get install -y devscripts git
echo

echo "[*] downloading haproxy source package in ${HAPROXY_DEB_PATH}"
if [[ -d "${HAPROXY_DEB_PATH}" ]]; then
    echo "[!] removing ${HAPROXY_DEB_PATH}"
    rm -fr "${HAPROXY_DEB_PATH}"
fi
mkdir "${HAPROXY_DEB_PATH}"
cd "${HAPROXY_DEB_PATH}"
git clone --depth=1 --branch "$HAPROXY_AWSLC_BRANCH" "${HAPROXY_REPO}" "${HAPROXY_DEB_PATH}/haproxy-awslc"
echo

echo "[*] Building haproxy binary package against awslc (${AWSLC_PATH})"
cd "${HAPROXY_DEB_PATH}/haproxy-awslc"
dpkg-buildpackage -b
cd ..
echo "[*] Debian packages in ${HAPROXY_DEB_PATH}"
echo

echo "[*] Verify binaries dependencies"
ldd "${HAPROXY_DEB_PATH}/haproxy-awslc/debian/haproxy/usr/sbin/haproxy"
echo

Build OpenSSL 3.5 on Debian Bullseye

The following script can be used as guideline to build OpenSSL and related libraries/debian packages on a Debian Bullseye.

Note that the patched repository for this lives in https://gitlab.wikimedia.org/fabfur/openssl#

#!/bin/bash

set -e

OPENSSL_DEB_PATH=/tmp/openssl-deb
OPENSSL_REPO=https://gitlab.wikimedia.org/fabfur/openssl.git

echo "[*] fetching dependencies ..."
apt-get update
apt-get install quilt
echo

echo "[*] downloading openssl repo in ${OPENSSL_DEB_PATH}"
if [[ -d "{OPENSSL_DEB_PATH}" ]]; then
    echo "[!] removing ${OPENSSL_DEB_PATH}"
    rm -fr "${OPENSSL_DEB_PATH}"
fi
mkdir "${OPENSSL_DEB_PATH}"
cd "${OPENSSL_DEB_PATH}"
git clone --depth=1 $OPENSSL_REPO "${OPENSSL_DEB_PATH}/openssl-3.5"
echo

echo "[*] building openssl 3.5 ..."
cd "${OPENSSL_DEB_PATH}/openssl-3.5"
quilt push -a
dpkg-buildpackage -b
cd ..
echo "[*] Debian packages in ${OPENSSL_DEB_PATH}"
echo

The following instructions and benchmarks were based on Debian Trixie, so I keep them around for reference only. The distribution used to build and test the various pieces is Bullseye instead. Please refer to above paragraphs.

All steps must be performed in a trixie environment

Step 1 - script to compile and run awslc from source

#!/bin/bash

set -e

CODE_PATH=/tmp/aws-lc
BUILD_PATH=/tmp/aws-lc-build
INSTALL_PATH=/opt/awslc

echo "[*] Checking debian version"
. /etc/os-release
if [[ ! $VERSION_ID -eq 13 ]]; then
    echo "[!] This must be run on Debian Trixie"
    exit 1
fi
echo

echo "[*] fetching dependencies..."
apt-get update
apt-get install -y git cmake ninja-build perl clang perl golang tree
echo

echo "[*] cloning awslc in ${CODE_PATH}"
if [[ -d ${CODE_PATH} ]]; then
    echo "[!] removing awslc code in ${CODE_PATH}"
    rm -fr "${CODE_PATH}"
fi
git clone --depth=1 https://github.com/aws/aws-lc.git "${CODE_PATH}"
echo

echo "[*] building awslc (build path: ${BUILD_PATH}, destination: ${INSTALL_PATH})"
if [[ -d ${BUILD_PATH} ]]; then
    echo "[!] removing awslc build dir ${BUILD_PATH}"
    rm -fr "${BUILD_PATH}"
fi
mkdir -p "${BUILD_PATH}"
cd "${BUILD_PATH}"
cmake -GNinja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_INSTALL_PREFIX="${INSTALL_PATH}" -DBUILD_SHARED_LIBS=1 "${CODE_PATH}"
echo

# echo "[*] running tests"
# ninja run_tests
# echo

echo "[*] installing into ${INSTALL_PATH}"
ninja install
tree -L 2 "${INSTALL_PATH}"
echo

echo "[*] awslc installed in ${INSTALL_PATH}"

Step 2 - script to fetch and compile haproxy against awslc

#!/bin/bash

set -e

HAPROXY_DEB_PATH=/tmp/haproxy-deb
HAPROXY_REPO=https://gitlab.wikimedia.org/fabfur/haproxy-awslc.git
# Standard path for aws-lc
AWSLC_PATH=/opt/awslc

echo "[*] Checking debian version"
. /etc/os-release
if [[ ! $VERSION_ID -eq 13 ]]; then
    echo "[!] This must be run on Debian Trixie"
    exit 1
fi
echo

echo "[*] fetching dependencies ..."
apt-get update
apt-get install -y devscripts git
echo

echo "[*] downloading haproxy source package in ${HAPROXY_DEB_PATH}"
if [[ -d "${HAPROXY_DEB_PATH}" ]]; then
    echo "[!] removing ${HAPROXY_DEB_PATH}"
    rm -fr "${HAPROXY_DEB_PATH}"
fi
mkdir "${HAPROXY_DEB_PATH}"
cd "${HAPROXY_DEB_PATH}"
git clone --depth=1 "${HAPROXY_REPO}" "${HAPROXY_DEB_PATH}/haproxy-awslc"
echo

echo "[*] Building haproxy binary package against awslc (${AWSLC_PATH})"
cd "${HAPROXY_DEB_PATH}/haproxy-awslc"
dpkg-buildpackage -b
cd ..
echo "[*] Debian packages in ${HAPROXY_DEB_PATH}"
echo

echo "[*] Verify binaries dependencies"
ldd "${HAPROXY_DEB_PATH}/haproxy-awslc/debian/haproxy/usr/sbin/haproxy"
echo

Step 3

Test with sample configuration and self-signed certs:

global
        log stdout local0 notice
        #log /dev/log   local1 notice
        #chroot /var/lib/haproxy
        #stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000


frontend https
    mode http
    bind :443 ssl crt /opt/test-configuration/snakeoil.pem
    http-request return status 200 content-type "text/plain" string "OK!"

Different haproxy version (binaries) run w/ haproxy -V -db -f /opt/test-configuration/haproxy.cfg

Preliminary results

On local container, using the compiled binary haproxy with the above configuration and the benchmark-curl.py script (roughly curl -s -Z --parallel-max <concurrency> -w "%{time_starttransfer}\\n" --insecure --no-sessionid -o /dev/null <URL>)

Debian Trixie, Haproxy 2.9.9, OpenSSL 3.5

1000 requests, concurrency 100:

  Max time: 5.718 ms
  p50   p75     p95     p99     p99.9   p99.99
  0.023 0.024   4.133   5.159   5.338   5.680

10k requests, concurrency 1000:

  Max time: 37.359 ms
  p50   p75     p95     p99     p99.9   p99.99
  0.023 0.024   0.051   9.658   11.786  15.090

Debian trixie, Haproxy 2.9.9, AwsLC

1000 requests, concurrency 100:

  Max time: 5.666 ms
  p50   p75     p95     p99     p99.9   p99.99
  0.029 0.047   3.878   4.859   5.066   5.606

10k requests, concurrency 1000:

  Max time: 36.490 ms
  p50   p75     p95     p99     p99.9   p99.99
  0.023 0.024   1.424   11.396  15.303  15.772

Debian Bullseye, Haproxy 2.9.9. OpenSSL 1.1.1

1000 requests, concurrency 100

  Max time: 5.217 ms
  p50   p75     p95     p99     p99.9   p99.99
  1.240 3.028   3.673   3.953   4.043   5.100

  Max time: 4.364 ms
  p50   p75     p95     p99     p99.9   p99.99
  1.781 1.900   2.167   2.530   2.885   3.012