I have a simple goal: I’d like to load balance a couple servers using the uri hash balance algorithm and I’d like to support persistent connections to and from haproxy with load balancing for each request. Unfortunately, I haven’t been able to achieve this behavior. I’ve tried a variety of configurations, namely option http-keep-alive, but nothing is working. The configuration file is below.
The behavior I’m witnessing is the hash is used on the first request to bind to a server for the duration of the persistent client connection. I was able to achieve load balancing per request with option httpclose or option http-server-close, but those prevent persistent connections. Is there a way to have persistent client and server connections with load balancing on each request?
I’m using HA-Proxy version 1.8.14-52e4d43 2018/09/20
global
daemon
frontend http-in
mode http
timeout client 60s
option http-keep-alive
bind *:8080
default_backend servers
backend servers
mode http
timeout connect 5s
timeout server 60s
option http-keep-alive
balance uri
option httpchk GET /health
http-check expect status 200
server server1 <ip>:<port> check
server server2 <ip>:<port> check
Yes, I’m certain. That is my entire config, verbatim. I have a larger config with logging and stats, but those aren’t required to reproduce the issue. With logging enabled, I can confirm the behavior via logs. With the simple config above I can confirm the behavior with backend server logs. I’m conducting tests using a simple netcat command and repeated GET requests to different uris. I’m using the DockerHub haproxy:1.8.14 (https://hub.docker.com/_/haproxy/). Dockerfile is below and haproxy -vv output below.
Dockerfile
FROM haproxy:1.8
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
haproxy
#haproxy -vv
HA-Proxy version 1.8.14-52e4d43 2018/09/20
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-null-dereference -Wno-unused-label
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.1.0f 25 May 2017
Running on OpenSSL version : OpenSSL 1.1.0f 25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
In your tests, does the backend server answer with a 401 or 407 response? Because that would make haproxy stick to that particular server (basically doing what option prefer-last-server) does:
Note, haproxy already automatically tries to stick to a server which sends a 401 or
to a proxy which sends a 407 (authentication required). This is mandatory for
use with the broken NTLM authentication challenge, and significantly helps in
troubleshooting some faulty applications.
Is there any way to disable that behavior? This effectively blocks me from achieving the desired behavior. If the first request happens to result in a 401, the load balancing is broken for the duration of the persistent connection.
You can change the behavior by commenting the relevant code with the following patch if you are willing to recompile:
diff --git a/src/proto_http.c b/src/proto_http.c
index 8f86422..f5b6033 100644
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -4387,9 +4387,9 @@ void http_end_txn_clean_session(struct stream *s)
* an opportunity for sending the challenge to the proper place,
* it's better to do it (at least it helps with debugging).
*/
- s->txn->flags |= TX_PREFER_LAST;
- if (srv_conn)
- srv_conn->flags |= CO_FL_PRIVATE;
+ //s->txn->flags |= TX_PREFER_LAST;
+ //if (srv_conn)
+ // srv_conn->flags |= CO_FL_PRIVATE;
}
/* Never ever allow to reuse a connection from a non-reuse backend */
However I could also see how this would fail when digest authentication is used and different backends are hit, while the nonce/opaque are not shared across backend servers.
For the record: are you using basic authentication, digest or something else? Are you guaranteeing that when server1 sends back a 401 Unauthorized, the client can resend the request with authentication data to server2 or could this fail, for example due to missing cross-server nonce/opaque synchronization (when using digest authentication)?
I’m trying to understand if this setup could actually work, if haproxy would not behave this way, or if it just fails at some later point because the load-balancing breaks the authentication anyway (which is why this was introduced in the first place).
In this case, haproxy is load balancing requests to backend REST API servers. Requests carry OAuth access tokens in the Authorization header. A 401 Unauthorized response indicates that a request didn’t carry a token at all or a request carried an invalid or expired token. The client could certainly send another request with a valid access token subsequently.
Our backend servers respond with 403 Forbidden on authorization failures. If I add an haproxy rule to change 401 to 403, would the prefer-last persistence still apply?
I think we should address this use-case, but I’m not sure what the approach should look like:
the simplest and safest option is yet another config knob. But, it’s yet another config knob … also the knob would probably have to default to the existing behavior, so the naming would probably be something cryptic like option dont-stick-on-auth-scheme
only do it with ntlm and digest authentication? This assumption may be wrong, and this heuristic may fail, a bad choice
don’t do it if the balancing alg is uri, url_param or hdr balancing (I don’t know about this, it fixes the problem, but it makes the behavior inconsistent across load-balancing algorithms)
Well, I’d be tempted to do both #2 and #3. First, I thought that we were doing this only for non-deterministic algorithms, but I was wrong, that’s only what option prefer-last-server does. But I think it is totally reasonable to tell NTLM users that load balancing has higher priority than their crappy protocol, and that under no circumstance a request will be sent to the wrong server, meaning that if they use hashing, it’s likely that connections will regularly be broken and re-established and the client will have to authenticate a lot. But with leastconn/roundrobin/random we don’t care and we simply fall back to prefer-last-server when seeing a 401/407.
Second, Digest auth is not broken like NTLM, the client is supposed to send its credentials with each request, and the protocol works on a per-request basis, not a per-connection one. With hash-based algorithms, there is no issue during the digest operation because the request will be replayed and sent to the same URL hence the same server. Thus we only need the equivalent of prefer-last-server during the digest operation for non-hash algos.
The problem with NTLM is that we must absolutely not put the connection back into the http-reuse connection pool because another request sent there would be understood by the server as being set by the authenticated client.
Thus I think the following solution would address the problem :
automatically enable prefer-last on the transaction when seeing 401/407 to make most auth schemes work, like digest, when non-hash algos are used ;
mark the connection as private only when its auth scheme shows NTLM (“Negotiate”), because we don’t want this connection to be used by another client. RFC4559 suggests it’s only sent with 401, but I’ve seen proxies rely on NTLM so I suspect they also use 407. Then better stay safe and continue to match the two and look for “Negotiate” as well in proxy-authenticate.
I’m not sure I can do something for this this week-end since I’d like to issue dev5 first. But it looks clear enough to me now.
Thank you both for the responses! What timeline should I expect for the fix? Would you apply it to the 1.8.x version or exclusively to 1.9.x? I have a modified Dockerfile that applies the patch above to proto_http.c. It’s working just fine, for now.
Two changes have been applied to 1.9 to fix this behavior. Both are marked for a 1.8 backport, which will happen before the next release, so 1.8.15 will fix this issue.
I don’t know when 1.8.15 will be released exactly, but I don’t think it will be a long wait, as others have requested a new 1.8 release already.
