HAProxy http-reuse never option not working for HAProxy 1.9.8 and 2.0.1

haproxy -vv output

HA-Proxy version 2.0.1 2019/06/26 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.1.0j 20 Nov 2018
Running on OpenSSL version : OpenSSL 1.1.0j 20 Nov 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as cannot be specified using ‘proto’ keyword)
h2 : mode=HTX side=FE|BE mux=H2
h2 : mode=HTTP side=FE mux=H2
: mode=HTX side=FE|BE mux=H1
: mode=TCP|HTTP side=FE|BE mux=PASS

Available services : none

Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace

Config

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
#maxconn 65536

defaults
log global
mode http
http-reuse never
no log
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend test
bind :80
mode http
maxconn 65536
default_backend nginx

backend nginx
mode http
balance leastconn
server 10.32.95.110 10.32.95.110:81 weight 255
server 10.32.110.236 10.32.110.236:81 weight 255
server 10.33.238.57 10.33.238.57:81 weight 255

Here, connection to the backends are not getting closed after response

That’s not what http-reuse is about.

Forget about that, and set option httpclose or better yet option http-server-close as appropriate.

But with the same configuration, connections to the backends are getting closed after response in HAProxy 1.8

That has nothing to do with the http-reuse option though.

If you want to troubleshoot why something closes in 1.8, but not in 1.9 and 2.0, then you need to provide additional informations about that. Like the request/response headers.

If you want to close after each request, configure haproxy as mentioned earlier.

@lukastribus got it. Thanks.

But in this case, I am sending same requests (without giving any specific headers) using wrk tool to HAProxy 2.0 and HAProxy 1.8 with the same configuration.

As given in the documentation:

KAL : keep alive (“option http-keep-alive”) which is the default mode : all requests and responses are processed, and connections remain open but idle between responses and new requests.

And under option http-keep-alive

If the client request has to go to another backend or another server due to
content switching or the load balancing algorithm, the idle connection will
immediately be closed and a new one re-opened.

According to my understanding, in HAProxy 1.8, connections are getting closed when new request comes as another server is selected for next request. Which is in-line with that given in the documentation.

But contrary to the documentation, in HAProxy 2.0, connection to server is not getting closed even when second server is selected for next request. Instead, requests are sent over the same connection to the server when its turn comes.

Please let me know if I am missing something.

To my knowledge – i.e. without looking in the source code, but based on what I’ve read through the mailing lists – the following is how things worked in 1.8 (and previous versions) regarding HTTP keep-alive and HTTP reuse:

  • first of all, a connection’s lifetime, between the HAProxy backend and the selected server, is tied to the lifetime of the connection between the client and the HAProxy frontend;
  • i.e. HAProxy didn’t have any sort of “connection pool”; (in fact if one used a round-robin load-balancing algorithm, and there were multiple servers, no server connection would have been reused regardless if it was for the same client…)
  • thus the emerging behavior you’ve seen by using http-keep-alive, namely if you had multiple servers, no connection would have been reused;

(In fact http-reuse refers to the fact that another client connection can (or shouldn’t) reuse an existing server connection opened by the original client connection. This is due to some badly implemented web-servers would assume that an inbound connection always belongs to the same “user”.)

Now in HAProxy 2.0, I think they’ve started to implement a sort of “connection pool” on the server side, thus even if you set http-reuse never it means that the same “client connection” could reuse its own initiated connections to a bunch of servers.

As Lukas said, if you really need to close the server connection after each request (as I do when the server is on 127.0.0.1), then you need to use http-close.

@ciprian.craciun then the above statement (mentioned in the documentation) will be wrong for HAProxy 2.0?

@tt100 OK, so I’ve looked through the 2.0 configuration manual, especially at the following sections:

, and they seem to imply that there isn’t any sort of connection pooling for servers.

However looking at the following two options:

, it seems that there is now such a pooling behaviour…

Looking through the changelog (and grep-ing for connection) I didn’t find any clear message that connection pooling was implemented, however multiple such messages do suggest in that direction.


The bottom line: it is highly likely that the documentation is out-of-sync with latest features.


But in any case, the option that you should be using is option http-server-close if you don’t want to reuse server-side connections for multiple requests.

The behaviour you’ve observed in 1.8 with option http-reuse never only worked “by mistake”, because you had multiple servers. Had you had only one server, then the connection would have been reused for the same client, even with http-reuse never

I don’t think the behaviour that I observed only worked “by mistake”, it is exactly as the statement given below:

If the client request has to go to another backend or another server due to
content switching or the load balancing algorithm, the idle connection will
immediately be closed and a new one re-opened.

I don’t want to close connection after each response, I want the same behaviour as I get for HAProxy 1.8. I agree it has nothing to do with http-reuse though. But if the above statement is true, behaviour should have been the same.

I think this statement would be wrong as HAProxy introduced connection pooling in HAProxy 1.6 itself with introduction of http-reuse. (Given in: Announcing HAProxy 1.6). According to my understanding, earlier (in HAProxy 1.6) connection sharing could have been done only till owner of the session died. Now, it can be configured for connections that can be shared according to the same principles as those applying to “http-reuse” . (Source: HAProxy version 1.9.16 - Configuration Manual)

pool-max-conn
This only applies to connections that can be shared according to the same principles as those applying to “http-reuse”.

Therefore, if I use setting http-reuse none, I am in-effect disabling connection pooling. (As given in Announcing HAProxy 1.6)

According to the announcement for HAProxy 1.9 or 2.0, It doesn’t seem like the behaviour to backend should change if I add http-reuse none. (As The http-reuse directive now defaults to safe if not set. is the change made in HAProxy 1.9). Please let me know if I am missing something.

Please describe in concrete terms what you “think” the behaviour is in 1.8, given the following circumstances:

(In all cases we have only one frontend and one backend, and all clients use keep-alive; also we assume that “client” is the same as “connection”, and if we say “different clients” we mean “different connections, perhaps from different user-agents”. Also we assume that requests are completely “serialized”, i.e. they come one-after-another, without any parallelism.)

(A) There is a single (client) connection, and a single server; should two separate requests use the same server connection?

(B) There is a single (client) connection, but two servers; should two separate requests use the same server connection (for the first, or the second one), or should one request go to the first server, and the second request to the second server? (In the second case, should the connection to the server be closed when it switches from one server to another?)

(C ) And there is where http-reuse starts to come into play
There are two client connections, and one server; the first client makes a request that goes through the server; after that is finished the second client makes a request; should the second request reuse the existing connection to the server?

For each of these cases, ask yourself what you think http-reuse never should imply? Does it change anything in the behaviour?

For the last time, :slight_smile: , http-reuse none enters into action only when requests from different connections could reuse a server connection, but this never option states that each client connection should be “paired” to its “server” connections, and that no requests should be delivered “cross clients”.

I.e.

  • client-connection-1 spawn server-connection-1a and server-connection-1b (in the behaviour you’ve observed in 2.0);
  • client-connection-2 spawns server-connection-2a and server-connection-2b;

With http-reuse never none of the requests from client-connection-2 would use server-connection-1* (and vice-versa). However requests from client-connection-1 are free to reuse any of the server-connection-1*.

With http-reuse always any request (from any client connection) could reuse any server connection regardless of who initiated it.

@ciprian.craciun I understood the use-case of http-reuse. Thanks :slight_smile:

I agree that the behaviour which I am observing is not because of http-reuse none

But the behaviour that you mentioned should have been shown in HAProxy 1.8 as well, right? As http-reuse never is set by default in that.

I have the “suspicion” that in HAProxy 2.0 (or perhaps 1.9) we now have server side connection pooling, which would imply that now server connections are kept idle, and only in the case when you have multiple servers you could end-up with one client connection (in keep-alive mode), that if it makes two requests, that are routed through two different servers, would keep two separate server side connections open (Also in keep-alive mode).

To summarize:

  • in HAProxy 1.8, if you have two servers and you use round-robin load-balancing, the http-reuse does absolutely nothing, but instead it behaves just like option http-server-close would have been set;
  • in HAProxy 1.8, if you have only one server, then http-reuse never (the default) would basically imply that there is an open “pipeline” between the client and the server;
  • in HAProxy 2.0, if you have only one server and you set http-reuse never it would have the same behaviour as in case of 1.8;
  • if however in case of HAProxy 2.0, you have two servers, based on your observation and my hunch, even if you set http-reuse never you might end-up with tho “pipelines” between the client and each of the servers;

(By “pipeline” I mean a server connection tied to a particular client connection, in a 1-to-N relation, i.e. one client connection could have many server connections.)

Just a clarification, I’ve asked the HAProxy Slack channel, and someone confirmed that in HAProxy 2.0 yes it does (have server side connection pooling).

@ciprian.craciun the behaviour seems like that. Thanks for the help!

I’m gonna take a look at this, in the mean time stop opening new threads for the same exact issue. Thanks.

Sure. Thanks

@tt100 please try the current 2.0 git tree (git clone http://git.haproxy.org/git/haproxy-2.0.git/) or a current snapshot (wget http://www.haproxy.org/download/2.0/src/snapshot/haproxy-2.0.1-patches-20190704.tar.gz).

I believe that with HTX enabled (which is default in 2.0), commit 6c7e96a3e (“BUG/MEDIUM: connections: Always call shutdown, with no linger.”) fixes the issue.

@capflam on the other hand, in 1.9 with HTX disabled, the issue looks still present in the current 1.9 tree and caused by commit 729b5b3 (“BUG/MINOR: channel: Set CF_WROTE_DATA when outgoing data are skipped”) instead.

Enabling htx in 1.9 post 6c7e96a3e also works around the issue.

So it appears we only need to troubleshoot the 1.9 HTX disabled case.

The behaviour is still the same when I used current master (commit 6c7e96a3e is merged in master). Also, I observed that SYN-SYN,ACK-RST,ACK (establish new connection to server, then sending reset just after creation of connection) to servers from HAProxy quite frequently. Is this because of some other bug?