Multiple requests received on backend server : hidden retry?

Help!
1

Hello guys !

I’m having some trouble with configuration in haproxy.
I have a form which is sending a big file in upload to the server.
It is passing through haproxy, and arriving onto the destination server.
Then, the server is making a quite long processing (several minutes).

The problem is, if I set ‘timeout server’ to something like 30sec, the destination server is receiving the whole request twice. Is there some retry setting by default that I missed ?

I would prefer to have an error, in case of timeouts.

I’m on HaProxy 1.5.18, on CentOs 7.
I’m using TCP, and choosing backend based on SNI indication.

Configuration 
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
	# to have these messages end up in /var/log/haproxy.log you will
	# need to:
	#
	# 1) configure syslog to accept network log events.  This is done
	#    by adding the '-r' option to the SYSLOGD_OPTIONS in
	#    /etc/sysconfig/syslog
	#
	# 2) configure local2 events to go to the /var/log/haproxy.log
	#   file. A line like the following can be added to
	#   /etc/sysconfig/syslog
	#
	#    local2.*                       /var/log/haproxy.log
	#
	log         127.0.0.1 local2

	chroot      /var/lib/haproxy
	pidfile     /var/run/haproxy.pid
	maxconn     4000
	user        haproxy
	group       haproxy
	daemon

	# turn on stats unix socket
	stats socket /var/lib/haproxy/stats

	#SSL Tunning
	tune.ssl.default-dh-param 2048

	tune.maxrewrite 16384
	tune.bufsize 32768
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
	timeout connect         10s
	timeout client          1m
	timeout server          1m


listen stats 0.0.0.0:9000       #Listen on all IP's on port 9000
	mode http
	balance
	timeout client 5000
	timeout connect 4000
	timeout server 30000

	#This is the virtual URL to access the stats page
	stats uri /haproxy_stats        

	#Authentication realm. This can be set to anything. Escape space characters with a backslash.
	stats realm HAProxy\ Statistics 

	#The user/pass you want to use. Change this password!
	stats auth admin:password

	#This allows you to take down and bring up back end servers.
	#This will produce an error on older versions of HAProxy.
	stats admin if TRUE

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend VIP_ALL
	bind *:443
	mode tcp

	tcp-request inspect-delay 5s
	tcp-request content accept if { req_ssl_hello_type 1 }

	acl acl_host_frontal req.ssl_sni -i <hostname>

	use_backend bk_ssl_frontal if acl_host_frontal

	default_backend             default

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend default


backend bk_ssl_frontal
  mode tcp
  balance roundrobin
  # maximum SSL session ID length is 32 bytes.
  stick-table type binary len 32 size 30k expire 30m
  acl clienthello req_ssl_hello_type 1
  acl serverhello rep_ssl_hello_type 2
  # use tcp content accepts to detects ssl client and server hello.
  tcp-request inspect-delay 5s
  tcp-request content accept if clienthello
  # no timeout on response inspect delay by default.
  tcp-response content accept if serverhello
  stick on payload_lv(43,1) if clienthello
  # Learn on response if server hello.
  stick store-response payload_lv(43,1) if serverhello
  option tcp-check
  server <ServerName> <ip>:443 check port 443
2

No, there is absolutely no such retry and remember, you are not even in http mode; in fact - haproxy only passes encrypted TCP traffic from the client to the server without any decryption happening in haproxy - so in this configuration it’s not possible to retry even if it would be implemented in such an erroneous way.

I assume the client retries when the TCP connection is dropped by haproxy. Checking your logs will help you understand if/how/when the client retries.

3

Thank you for the quick answer. I found it weird, but I wanted to be sure.

Is there any possibility to add some informations to avoid the client to retry ? Some options or headers ?
According to the devtools of my web navigator (Chrome), there is only one request sent (I know this is a bit out of subject, but I would like a way to avoid this kind of retry).

4

Not in TCP mode, no.

If you install the SSL certificate on haproxy and use HTTP mode, the behavior will be completely different and haproxy will return a 500 server error, but with TCP mode, haproxy can only close the connection.

5

OK, thanks for your answer.

One last question : could you explain to me, why would the navigator retry the request (http verb POST), in case of tcp connection closed ?
In my (poor) understanding, I thought some dataloss inside a TCP connection would be acceptable, but a whole retry (and resend of data) because to a tcp connection closing seems weird to me.

Am I missing something ?

6

Right, non-idempotent methods like POST should not be retried on a TCP failure. Not sure why a browser would behave this way.

Like I said, check haproxy logs to confirm this or take a look at the tcpdump on the client side.