Clarification on HAProxy Connection Pooling

Help!
1

Hi,

I was doing some experiments with HAProxy 2.0 with pool-max-conn and pool-purge-delay options for connection pooling.

Usage:

I have an application making external calls to some backend. I am evaluating using HAProxy between the application and the vendor so that HAProxy would manage connection pool with the backend.

The setup would be: HAProxy would perform SSL connection verification on the backend as part of connection establishment and maintain a connection pool with the backend. The backend does not have TLS SNI extension enabled.

Config:


# Removed log related config for brevity and actual backend addresses. 

global

  stats socket /var/run/haproxy.sock
  ca-base /etc/ssl/certs

defaults
  mode  http
  timeout connect 5s
  timeout client  10s
  timeout server 10s
  timeout http-keep-alive 100s
  option redispatch
  option prefer-last-server
  option http-keep-alive
  retries 3
  compression algo gzip
  timeout http-request 10s

frontend http_internal
  bind *:80 name http_internal

  # stats
  stats enable
  stats uri  /stats
  stats refresh 5s

  acl haproxy_stats url_beg /stats
  use_backend bk_haproxy_stats if haproxy_stats

  acl host_example hdr(host) -i example.com
  use_backend bk_test if host_example

 default_backend bk_test


backend bk_test
  mode http
  http-reuse aggressive # tried different values {safe, aggressive, always}
  server server-1 example.com:443 ssl verify required ca-file ca-certificates.crt pool-max-conn -1 pool-purge-delay 200s

Observations:

As described in https://www.haproxy.com/blog/haproxy-1-9-has-arrived/#connection-management, if connection between application and HAProxy closes, it should not affect connection between HAProxy and backend server.

I tested by giving a series of requests (20,30,40 etc) via curl to the HAProxy with a gap between each request. I noticed that it does not always reuse the backend connection and creates a new connection again. Each request is from a new curl process.

One such run’s output (total of 40 requests):


Got this data by running packet capture and Lua script:

The "Connection" below is each unique connection between HAProxy and Backend server (identified by the 4-tuple of {IP, port}) and number of times it got used for requests. 

First 2 connections got used 7 times (**not for consecutive requests** though)

Connection -> Number of times used.

Connection 1 -> 7  # used 7 times. 
Connection 2 -> 7
Connection 3 -> 1  # used 1 time and closed. 
Connection 4 -> 1
Connection 5 -> 1
Connection 6 -> 1
Connection 7 -> 1
Connection 8 -> 1
Connection 9 -> 1
Connection 10 -> 1
Connection 11 -> 1
Connection 12 -> 1
Connection 13 -> 1
Connection 14 -> 1
Connection 15 -> 1
Connection 16 -> 1
Connection 17 -> 1
Connection 18 -> 1
Connection 19 -> 1
Connection 20 -> 1
Connection 21 -> 1
Connection 22 -> 1
Connection 23 -> 1
Connection 24 -> 1
Connection 25 -> 1
Connection 26 -> 1
Connection 27 -> 1
Connection 28 -> 1

My understanding is it should not create new connection for every new request to HAProxy but instead use the connections in pool. The backend server’s response has “Connection: Keep-alive” too.

After the tests, when I ran ss command, it showed only the first 2 connections in ESTABLISHED mode. Also in packet capture, I notice that HAProxy is sending RST to the backend after the response for the other connections except the initial 2 connections.

Is this expected or am I missing anything?

Thanks.