HAProxy is creating new TCP connection for each http request in spite of http-keep-alive

Help!
1

Hi,
I configured HAProxy for ACL based distribution. Say, there is a header ‘msisdn’ in http requests whose value is of int type and HAProxy would distribute requests between 2 backend servers based on value of msisdn.
I created a Jmeter Test plan for testing this scenario – Contains 5 http requests, out of which, 3 requests should go to server1 and 2 to server2 based on value of msisdn header and ran 2 loops of it with single thread. Set Connection: Keep-Alive in http requests. Distribution works perfectly fine but tcpdump between HAproxy and backend server shows that for every http request, there is new tcp connection. I made sure to set option http-keep-alive and http-reuse always in defaults, frontend and backend sections of haproxy.cfg .
Attachments:

  1. haproxy.cfg :

#---------------------------------------------------------------------

Example configuration for a possible web application. See the

full configuration options online.

http://haproxy.1wt.eu/download/1.4/doc/configuration.txt

#---------------------------------------------------------------------

#---------------------------------------------------------------------

Global settings

#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the ‘-r’ option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2

chroot      /var/lib/haproxy
pidfile     /var/run/haproxy.pid
maxconn     4000
user        haproxy
group       haproxy
daemon

# turn on stats unix socket
stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------

common defaults that all the ‘listen’ and ‘backend’ sections will

use if not designated in their block

#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option http-keep-alive
option dontlognull
option redispatch
retries 3
timeout http-request 3600000
timeout queue 3600000
timeout connect 3600000
timeout client 3600000
timeout server 3600000
timeout http-keep-alive 3600000
timeout check 3600000
maxconn 3000
http-reuse always

listen stats
bind 10.200.208.96:9001
mode http
log global
stats enable
stats realm Haproxy\ Statistics
stats uri /stats
stats hide-version
stats auth haproxy:redhat
http-reuse always

#---------------------------------------------------------------------

main frontend which proxys to the backends

#---------------------------------------------------------------------
frontend LB
bind 10.200.208.96:9005
reqadd X-Forwarded-Proto:\ http
option http-keep-alive
acl msisdn_range_1 hdr(msisdn) -m int 0:4
acl msisdn_range_2 hdr(msisdn) -m int 5:9
use_backend LB if msisdn_range_1
use_backend CR if msisdn_range_2

#---------------------------------------------------------------------

round robin balancing between the various backends

#---------------------------------------------------------------------

backend LB
mode http
balance roundrobin
option http-keep-alive
option forwardfor
option tcp-check
http-reuse always
server 10.200.208.97 10.200.208.97:8585 check # active node

backend CR
mode http
option http-keep-alive
option forwardfor
option tcp-check
http-reuse always
server 10.200.208.96 10.200.208.96:8585 check # active node

  1. tcpdump screen shots:
    dump_1st_request
    dump_1st_request.jpeg1603×648 309 KB

Problem looks similar to topic https://discourse.haproxy.org/t/problems-enabling-keep-alive-over-to-high-latency-backend/2852 but in our case, client doesn’t close connection, HAProxy is creating new connection to server.
Any help would be highly appreciated.

Thanks!

2

Are you sure your backend doesn’t set a Connection: close header after the POST?

3

Yes, backend server doesn’t set Connection: close. Here is screenshot of server’s response:

http_response
http_response.jpeg1591×664 175 KB

4

Version of HAProxy used is 1.7.11 .

5

Can share a capture with frontend and backend data?

6

Attaching tcpdump between client and proxy, proxy and backend server. 172.16.11.219 is client ip. Haproxy is running on 10.200.208.96 and backend server is running on 10.200.208.97.

  1. tcpdump b/w client and proxy: https://packettotal.com/app/analysis?id=58680e155b1e91f15775c5ad8e1f2a61

  2. tcpdump b/w proxy and server: https://packettotal.com/app/analysis?id=656e894e0a31001440acb6a2d48debce

7

Really your client closes the connection after each requests:

image
image.png995×314 36.3 KB

1 Like
8

Thanks @lukastribus, seems some timeout in client (Jmeter). Now I sent 3 http requests for whom 404 response was expected. This time response was very fast and all 3 requests used same tcp connection. Thanks again!