I have a simple goal: I’d like to load balance a couple servers using the
uri hash balance algorithm and I’d like to support persistent connections to and from haproxy with load balancing for each request. Unfortunately, I haven’t been able to achieve this behavior. I’ve tried a variety of configurations, namely
option http-keep-alive, but nothing is working. The configuration file is below.
The behavior I’m witnessing is the hash is used on the first request to bind to a server for the duration of the persistent client connection. I was able to achieve load balancing per request with
option httpclose or
option http-server-close, but those prevent persistent connections. Is there a way to have persistent client and server connections with load balancing on each request?
I’m using HA-Proxy version 1.8.14-52e4d43 2018/09/20
timeout client 60s
timeout connect 5s
timeout server 60s
option httpchk GET /health
http-check expect status 200
server server1 <ip>:<port> check
server server2 <ip>:<port> check
That is strange, are you sure you didn’t specify option prefer-last-server somewhere in this configuration?
Can you provide the output of
Yes, I’m certain. That is my entire config, verbatim. I have a larger config with logging and stats, but those aren’t required to reproduce the issue. With logging enabled, I can confirm the behavior via logs. With the simple config above I can confirm the behavior with backend server logs. I’m conducting tests using a simple netcat command and repeated
GET requests to different uris. I’m using the DockerHub haproxy:1.8.14 (https://hub.docker.com/_/haproxy/). Dockerfile is below and
haproxy -vv output below.
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
HA-Proxy version 1.8.14-52e4d43 2018/09/20
Copyright 2000-2018 Willy Tarreau <email@example.com>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-null-dereference -Wno-unused-label
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.1.0f 25 May 2017
Running on OpenSSL version : OpenSSL 1.1.0f 25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
In your tests, does the backend server answer with a 401 or 407 response? Because that would make haproxy stick to that particular server (basically doing what
option prefer-last-server) does:
Note, haproxy already automatically tries to stick to a server which sends a 401 or
to a proxy which sends a 407 (authentication required). This is mandatory for
use with the broken NTLM authentication challenge, and significantly helps in
troubleshooting some faulty applications.
Yes, it does. I didn’t bother dealing with authentication during testing. Wow.
Is there any way to disable that behavior? This effectively blocks me from achieving the desired behavior. If the first request happens to result in a 401, the load balancing is broken for the duration of the persistent connection.
Not via configuration, no.
You can change the behavior by commenting the relevant code with the following patch if you are willing to recompile:
diff --git a/src/proto_http.c b/src/proto_http.c
index 8f86422..f5b6033 100644
@@ -4387,9 +4387,9 @@ void http_end_txn_clean_session(struct stream *s)
* an opportunity for sending the challenge to the proper place,
* it's better to do it (at least it helps with debugging).
- s->txn->flags |= TX_PREFER_LAST;
- if (srv_conn)
- srv_conn->flags |= CO_FL_PRIVATE;
+ //s->txn->flags |= TX_PREFER_LAST;
+ //if (srv_conn)
+ // srv_conn->flags |= CO_FL_PRIVATE;
/* Never ever allow to reuse a connection from a non-reuse backend */
The behavior was introduced in commit 068621e4 (MINOR: http: try to stick to same server after status 401/407) mainly to help buggy backends like NTLM. Most often preferring an already open connection to the backend server is actually a good idea, but certainly not in the case of URI balancing …
However I could also see how this would fail when digest authentication is used and different backends are hit, while the nonce/opaque are not shared across backend servers.
For the record: are you using basic authentication, digest or something else? Are you guaranteeing that when server1 sends back a 401 Unauthorized, the client can resend the request with authentication data to server2 or could this fail, for example due to missing cross-server nonce/opaque synchronization (when using digest authentication)?
I’m trying to understand if this setup could actually work, if haproxy would not behave this way, or if it just fails at some later point because the load-balancing breaks the authentication anyway (which is why this was introduced in the first place).
In this case, haproxy is load balancing requests to backend REST API servers. Requests carry OAuth access tokens in the
Authorization header. A
401 Unauthorized response indicates that a request didn’t carry a token at all or a request carried an invalid or expired token. The client could certainly send another request with a valid access token subsequently.
Our backend servers respond with
403 Forbidden on authorization failures. If I add an haproxy rule to change 401 to 403, would the prefer-last persistence still apply?
No, I don’t think rewriting the response code in haproxy helps, the detection of the 401 code is most likely happening before that.
You’d probably also brake the application, I think the 401 code is required to solicit the actual Authorization header from the client.
@willy in this case we are breaking a valid use-case (uri-balancing), when the backend is responding with 401/407, because of a workaround for broken setups (NTLM or broken digest auth I assume).
I think we should address this use-case, but I’m not sure what the approach should look like:
- the simplest and safest option is yet another config knob. But, it’s yet another config knob … also the knob would probably have to default to the existing behavior, so the naming would probably be something cryptic like option dont-stick-on-auth-scheme
- only do it with ntlm and digest authentication? This assumption may be wrong, and this heuristic may fail, a bad choice
- don’t do it if the balancing alg is
hdr balancing (I don’t know about this, it fixes the problem, but it makes the behavior inconsistent across load-balancing algorithms)
Hi Lukas! Good catch on this one!
Well, I’d be tempted to do both #2 and #3. First, I thought that we were doing this only for non-deterministic algorithms, but I was wrong, that’s only what option prefer-last-server does. But I think it is totally reasonable to tell NTLM users that load balancing has higher priority than their crappy protocol, and that under no circumstance a request will be sent to the wrong server, meaning that if they use hashing, it’s likely that connections will regularly be broken and re-established and the client will have to authenticate a lot. But with leastconn/roundrobin/random we don’t care and we simply fall back to prefer-last-server when seeing a 401/407.
Second, Digest auth is not broken like NTLM, the client is supposed to send its credentials with each request, and the protocol works on a per-request basis, not a per-connection one. With hash-based algorithms, there is no issue during the digest operation because the request will be replayed and sent to the same URL hence the same server. Thus we only need the equivalent of prefer-last-server during the digest operation for non-hash algos.
The problem with NTLM is that we must absolutely not put the connection back into the http-reuse connection pool because another request sent there would be understood by the server as being set by the authenticated client.
Thus I think the following solution would address the problem :
- automatically enable prefer-last on the transaction when seeing 401/407 to make most auth schemes work, like digest, when non-hash algos are used ;
- mark the connection as private only when its auth scheme shows NTLM (“Negotiate”), because we don’t want this connection to be used by another client. RFC4559 suggests it’s only sent with 401, but I’ve seen proxies rely on NTLM so I suspect they also use 407. Then better stay safe and continue to match the two and look for “Negotiate” as well in proxy-authenticate.
I’m not sure I can do something for this this week-end since I’d like to issue dev5 first. But it looks clear enough to me now.
@willy agreed, that sounds like the correct course of action. Seems like the value could be both Negotiate and NTLM?
I would like to take a look at this, unless you are already working on it. I will let you know if I’m stuck or gave up.
Thank you both for the responses! What timeline should I expect for the fix? Would you apply it to the 1.8.x version or exclusively to 1.9.x? I have a modified Dockerfile that applies the patch above to
proto_http.c. It’s working just fine, for now.
Two changes have been applied to 1.9 to fix this behavior. Both are marked for a 1.8 backport, which will happen before the next release, so
1.8.15 will fix this issue.
I don’t know when 1.8.15 will be released exactly, but I don’t think it will be a long wait, as others have requested a new 1.8 release already.
Is there a target date for 1.8.15? I’m anxiously waiting to get rid of my custom Dockerfile with code patch.
There is no date set in stone, it depends when they can interrupt their work on the 1.9 development:
Now I’m waiting for the Docker Hub haproxy repo to be updated.
Make sure you use
1.8.16 instead of
1.8.15, the latter has a DNS bug that is triggered by the Docker internal resolution.