Haproxy 1.8 vs 2.7.3

Help!
1

Hello,
I am new to haproxy so learning as I go, so please excuse any obvious mistakes.
I have a working haproxy setup that forwards http requests to load balance between two servers. This is running on a CentOS Linux release 7.9.2009 VM
It is running haproxy18, installed via rpm

haproxy18 -vvv
HA-Proxy version 1.8.27-493ce0b 2020/11/06
Copyright 2000-2020 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label
  OPTIONS = USE_LINUX_TPROXY=1 USE_CRYPT_H=1 USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.1k  FIPS 25 Mar 2021
Running on OpenSSL version : OpenSSL 1.1.1k  FIPS 25 Mar 2021
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
        [SPOE] spoe
        [COMP] compression
        [TRACE] trace

My config is as follows

global
    log /dev/log local0 debug
    maxconn 8000
    user haproxy
    group haproxy
    daemon

defaults
    balance roundrobin
    log global
    option httplog
    timeout client 30s
    timeout connect 4s
    timeout server 30s
    timeout check 5s
    option allbackups

listen admin_page
    bind *:9600
    mode http
    stats enable
    stats refresh 60s
    stats uri /

frontend fe_connector_443
    bind *:443 ssl crt /etc/ssl/private/cert.pem
    mode http
    capture cookie JSESSIONID= len 32
    option forwardfor
    default_backend be_connector_443

backend be_connector_443
    mode http
    balance roundrobin
    cookie JSESSIONID prefix indirect nocache
    option forwardfor
    server cac01 192.168.30.11:443 check cookie s1 verify none ssl
    server cac02 192.168.30.12:443 check cookie s2 verify none ssl

This works as I would expect, I use the client application to connect to the proxy, the application presents me with a cert warning which I accept, the proxy then forwards the http traffic to the backend servers and then I am able to log in to my application and use it as expected.
In the log, I get these messages when, using my client application, connecting to the proxy:

Feb 21 13:07:46 anyware-proxy01 haproxy18[9950]: IP:38390 [21/Feb/2023:13:07:46.681] fe_connector_443~ fe_connector_443/<NOSRV> -1/-1/-1/-1/22 400 187 - - CR-- 1/1/0/0/0 0/0 "<BADREQ>"
Feb 21 13:07:51 anyware-proxy01 haproxy18[9950]: IP:38391 [21/Feb/2023:13:07:51.414] fe_connector_443~ be_connector_443/cac01 0/0/1/251/253 200 1280 JSESSIONID=s1~8bec635c-9f38-42f8 JSESSIONID=8bec635c-9f38-42f8-a4 --VD 1/1/0/0/0 0/0 "POST /pcoip-broker/xml HTTP/1.1"
Feb 21 13:07:51 anyware-proxy01 haproxy18[9950]: IP:40326 [21/Feb/2023:13:07:51.677] fe_connector_443~ fe_connector_443/<NOSRV> -1/-1/-1/-1/31 400 187 - - CR-- 1/1/0/0/0 0/0 "<BADREQ>"
Feb 21 13:07:53 anyware-proxy01 haproxy18[9950]: IP:40327 [21/Feb/2023:13:07:51.768] fe_connector_443~ be_connector_443/cac01 0/0/1/1865/1867 200 2229 JSESSIONID=s1~8bec635c-9f38-42f8 JSESSIONID=8bec635c-9f38-42f8-a4 --VD 1/1/0/0/0 0/0 "POST /pcoip-broker/xml HTTP/1.1"

However as this is an old version of haproxy, I would like to upgrade to the latest version, which I have built from source:

/opt/haproxy-2.7.3/sbin/haproxy -vvv
HAProxy version 2.7.3-1065b10 2023/02/14 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2024.
Known bugs: http://www.haproxy.org/bugs/bugs-2.7.3.html
Running on: Linux 3.10.0-1160.81.1.el7.x86_64 #1 SMP Fri Dec 16 17:29:43 UTC 2022 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_PCRE=1 USE_THREAD=1 USE_LIBCRYPT=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_TFO=1 USE_NS=1 USE_SYSTEMD=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT +PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC +RT +SHM_OPEN -SLZ -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=4).
Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.4.4
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 4.8.5 20150623 (Red Hat 4.8.5-44)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

When using the same config as above, I am no longer able to connect via my client application, it just times out, and I get these messages in the haproxy log

Feb 21 13:18:11 anyware-proxy01 haproxy[10121]: IP:38980 [21/Feb/2023:13:18:11.239] fe_connector_443~ be_connector_443/cac01 0/0/4/74/78 200 1855 - - ---- 1/1/0/0/0 0/0 "POST /pcoip-broker/xml HTTP/1.1"
Feb 21 13:18:12 anyware-proxy01 haproxy[10121]: IP:38990 [21/Feb/2023:13:18:12.017] fe_connector_443~ be_connector_443/cac01 0/0/3/1/4 200 519 - - ---- 1/1/0/0/0 0/0 "POST /broker/xml HTTP/1.1"
Feb 21 13:18:12 anyware-proxy01 haproxy[10121]: IP:39000 [21/Feb/2023:13:18:12.716] fe_connector_443~ be_connector_443/cac01 0/0/3/35/38 200 1855 - - ---- 1/1/0/0/0 0/0 "POST /pcoip-broker/xml HTTP/1.1"
Feb 21 13:18:15 anyware-proxy01 haproxy[10121]: IP:39011 [21/Feb/2023:13:18:15.035] fe_connector_443~ be_connector_443/cac01 0/0/4/1/5 200 519 - - ---- 1/1/0/0/0 0/0 "POST /broker/xml HTTP/1.1"

I can’t see where the issue might be. All I can tell is using 2.7.3 behaves differently from 1.8.27
Would anyone be able to help, or provide any suggestions as to what might be happening?

Thank you
Matt

2

I’m no expert (quite the novice, actually) but according to your logs… the traffic is hitting the backend and returning a 200 (successful) response.
I find it interesting on your “working” config you’re getting NOSRV errrors followed by 200 responses.

I’m following because I’m curious.

3

Hello,

Thanks for your reply.
I agree, it does look like, in the second set of logs, it it returning a successful response. However, I am expecting (on the client application) get a certificate warning to accept, then a login window. The second set of logs come in quick succession, one after the other before the client application times out. However, with the first set of logs, each messages comes after an interaction with the client application; i.e. connect and accept cert (log), login (log).

Yeah I am not sure what those NOSRV messages refer to;

Feb 21 13:07:46 anyware-proxy01 haproxy18[9950]: IP:38390 [21/Feb/2023:13:07:46.681] fe_connector_443~ fe_connector_443/<NOSRV> -1/-1/-1/-1/22 400 187 - - CR-- 1/1/0/0/0 0/0 "<BADREQ>"

They seem to suggest a front end to front end connection, but I don’t know why that would be?
Thanks

4

I have gone back and compiled the latest version of 1.8 and 1.9. Both versions work as expected for me.
I then tried version 2.0 and that did not work, presenting the same behaviour as my initial post for 2.7.3
There is likely something fundamentally different to the way haproxy works as of v2. I suspect it is to do with HTTP/2 but I am unsure how or or why.
The responses in the log all seem to be HTTP/1.1 but I don’t know how or if I can force this behaviour in haproxy v2 or newer?

5

NOSRV means the frontend can’t find a suitable backend (usually no backend up) to pass the traffic to, is my understanding. Again, I’m a novice.

Is this all on the same server between different versions? Or are you using a separate server to test? Just curious if any firewall rules or routing issues could be taking place.

I recently updated all my haproxy loadbalancers from 1.8.x to 2.6.x and haven’t had any major issues, serving many different websites behind them. Maybe try a lower version of 2.y.x to see if it’s a specific issue to 2.7.3?

6

I am testing the different haproxy versions all on the same server. There should be no firewall rules stopping communication between haproxy and the backend server

I have tried 1.5.18, 1.8.27, 1.8.31, 1.9.16 all of which work.
I have then tried: 2.0.31 and 2.7.3 neither of which work

Unfortunately the backend servers are products and essentially black boxes to me. But what I need is forward the 443 requests to them for authentication. Client -443-> proxy -443-> be_server then once authenticated the backend server talks directly to the client

7

Hopefully the expert comes along soon enough to resolve this for you… following because I’m curious. Good luck mate.

8

It’s really not obvious what happens here, especially since the logs don’t confirm any issues.

Are the clients browsers? Can you use the developer console (F12) in chrome and firefox and check the network tab before tryting the request, to see what happens? Also check the console.

Also, in haproxy 2.0, try setting no option http-use-htx in the default section to see if this restores the old behaviour; this will help narrow it down further.

9

Hello, thanks for your reply.

The clients are applications called pcoip client, part of the hp anyware product family so I am unsure how to check that traffic?

I have added the option: no option http-use-htx to the defaults section of the config file:

defaults
    balance roundrobin
    log global
    option httplog
    timeout client 30s
    timeout connect 4s
    timeout server 30s
    timeout check 5s
    option allbackups
    no option http-use-htx

And now the service fails to start with this error:

Feb 24 09:59:18 anyware-proxy01 systemd[1]: Starting HAProxy 2.7.3...
-- Subject: Unit haproxy-2.7.3.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit haproxy-2.7.3.service has begun starting up.
Feb 24 09:59:18 anyware-proxy01 haproxy[18415]: [NOTICE]   (18415) : haproxy version is 2.7.3-1065b10
Feb 24 09:59:18 anyware-proxy01 haproxy[18415]: [NOTICE]   (18415) : path to executable is /opt/haproxy-2.7.3/sbin/haproxy
Feb 24 09:59:18 anyware-proxy01 haproxy[18415]: [ALERT]    (18415) : config : parsing [/etc/haproxy/haproxy-2.7.3.conf:26]: negation/default is not supported for option 'http-use-htx'.
Feb 24 09:59:18 anyware-proxy01 haproxy[18415]: [ALERT]    (18415) : config : Error(s) found in configuration file : /etc/haproxy/haproxy-2.7.3.conf
Feb 24 09:59:18 anyware-proxy01 haproxy[18415]: [ALERT]    (18415) : config : Fatal errors found in configuration.
Feb 24 09:59:18 anyware-proxy01 systemd[1]: haproxy-2.7.3.service: main process exited, code=exited, status=1/FAILURE
Feb 24 09:59:18 anyware-proxy01 systemd[1]: Failed to start HAProxy 2.7.3.
-- Subject: Unit haproxy-2.7.3.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit haproxy-2.7.3.service has failed.
--
-- The result is failed.
Feb 24 09:59:18 anyware-proxy01 systemd[1]: Unit haproxy-2.7.3.service entered failed state.
Feb 24 09:59:18 anyware-proxy01 systemd[1]: haproxy-2.7.3.service failed.
Feb 24 09:59:18 anyware-proxy01 polkitd[830]: Unregistered Authentication Agent for unix-process:18409:23835718 (system bus name :1.1028, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_GB.UTF-8) (disconnected from bus)

Do I need to add something else to the config?

Thank you

10

@lukastribus apologies I misread your comment and tried with 2.7.3 not 2.0.
I added the suggested option to the defaults section of the config no option http-use-htx and that seems to have restored the behaviour I was getting with versions pre 2.0 previously.

â—Ź haproxy-2.0.31.service - HAProxy 2.0.31
   Loaded: loaded (/etc/systemd/system/haproxy-2.0.31.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-02-24 10:28:16 GMT; 4min 44s ago
 Main PID: 18949 (haproxy)
   CGroup: /system.slice/haproxy-2.0.31.service
           ├─18949 /opt/haproxy-2.0.31/sbin/haproxy -Ws -f /etc/haproxy/haproxy-2.7.3.conf -f /etc/haproxy/conf.d -p /var/run/haproxy-2.0.31.pid
           └─18951 /opt/haproxy-2.0.31/sbin/haproxy -Ws -f /etc/haproxy/haproxy-2.7.3.conf -f /etc/haproxy/conf.d -p /var/run/haproxy-2.0.31.pid

Feb 24 10:28:16 anyware-proxy01 haproxy[18949]: [NOTICE] 054/102816 (18949) : New worker #1 (18951) forked
Feb 24 10:28:16 anyware-proxy01 haproxy[18949]: Proxy be_connector_443 started.
Feb 24 10:28:16 anyware-proxy01 systemd[1]: Started HAProxy 2.0.31.
Feb 24 10:28:59 anyware-proxy01 haproxy[18951]: IP:62228 [24/Feb/2023:10:28:59.565] fe_connector_443~ be_connector_443/cac01 0/0/3/75/78 200 1923 - JSESSIONID=82538ab9-996e-4556-a0 --NR 1/1/0/0/0 0/0 "POST /pcoip-broker/xml HTTP/1.1"
Feb 24 10:28:59 anyware-proxy01 haproxy[18951]: IP:62231 [24/Feb/2023:10:28:59.776] fe_connector_443~ fe_connector_443/<NOSRV> -1/-1/-1/-1/30 400 207 - - CR-- 1/1/0/0/0 0/0 "<BADREQ>"
Feb 24 10:29:02 anyware-proxy01 haproxy[18951]: IP:62232 [24/Feb/2023:10:29:02.250] fe_connector_443~ be_connector_443/cac01 0/0/3/31/34 200 1927 - JSESSIONID=5c35af2f-7a55-41d5-b9 --NR 1/1/0/0/0 0/0 "POST /pcoip-broker/xml HTTP/1.1"
Feb 24 10:29:02 anyware-proxy01 haproxy[18951]: IP:62233 [24/Feb/2023:10:29:02.295] fe_connector_443~ fe_connector_443/<NOSRV> -1/-1/-1/-1/31 400 207 - - CR-- 1/1/0/0/0 0/0 "<BADREQ>"
Feb 24 10:29:09 anyware-proxy01 haproxy[18951]: IP:62234 [24/Feb/2023:10:29:09.031] fe_connector_443~ be_connector_443/cac01 0/0/2/239/241 200 1280 JSESSIONID=s1~5c35af2f-7a55-41d5 JSESSIONID=5c35af2f-7a55-41d5-b9 --VD 1/1/0/0/0 0/0 "POST /pcoip-broker/xml HTTP/1.1"
Feb 24 10:29:09 anyware-proxy01 haproxy[18951]: IP:33502 [24/Feb/2023:10:29:09.292] fe_connector_443~ fe_connector_443/<NOSRV> -1/-1/-1/-1/25 400 207 - - CR-- 1/1/0/0/0 0/0 "<BADREQ>"
Feb 24 10:29:10 anyware-proxy01 haproxy[18951]: IP:33503 [24/Feb/2023:10:29:09.378] fe_connector_443~ be_connector_443/cac01 0/0/2/1269/1271 200 2229 JSESSIONID=s1~5c35af2f-7a55-41d5 JSESSIONID=5c35af2f-7a55-41d5-b9 --VD 1/1/0/0/0 0/0 "POST /pcoip-broker/xml HTTP/1.1"

What does that suggest?
Thank you

11

Ok, so the client is a blackbox too, thats very important.

Downgrading to pre-HTX times fixes the issue (but this is possible only in 2.0, as those old code paths have been subsequently removed from haproxy).

Most likely your client doesn’t like the lowercased header names that haproxy in HTX mode will send by default. This means the client is not RFC compliant.

Let’s try to work around this by using h1-case-adjust-bogus-client. You may need to use different header names and you probably only need a few of those header name adjustments:

global
 h1-case-adjust connection Connection
 h1-case-adjust content-length Content-Length
 h1-case-adjust content-encoding Content-Encoding 
 h1-case-adjust content-type Content-Type
 h1-case-adjust cache-control Cache-Control
 h1-case-adjust date Date
 h1-case-adjust expires Expires
 h1-case-adjust etag ETag
 h1-case-adjust last-modified Last-Modified
 h1-case-adjust server Server
 h1-case-adjust via Via
 h1-case-adjust age Age
 h1-case-adjust accept-ranges Accept-Ranges
 h1-case-adjust access-control-allow-origin Access-Control-Allow-Origin

frontend fe_connector_443
 option h1-case-adjust-bogus-client
12

@lukastribus thank you for that.

Adding those header name adjustments in my config for 2.7.3 seems to have resolved the issue. My logs are now looking the same as they did using pre v2 haproxy

Feb 28 17:42:15 anyware-proxy01 haproxy[3205]: IP:55091 [28/Feb/2023:17:42:15.117] fe_connector_443~ be_connector_443/cac01 0/0/2/30/32 200 1855 - JSESSIONID=e912a635-9d81-47e0-a2 --NR 1/1/0/0/0 0/0 "POST /pcoip-broker/xml HTTP/1.1"
Feb 28 17:42:15 anyware-proxy01 haproxy[3205]: IP:55094 [28/Feb/2023:17:42:15.263] fe_connector_443~ fe_connector_443/<NOSRV> -1/-1/-1/-1/1 400 0 - - CR-- 1/1/0/0/0 0/0 "<BADREQ>"
Feb 28 17:42:17 anyware-proxy01 haproxy[3205]: IP:55095 [28/Feb/2023:17:42:17.093] fe_connector_443~ be_connector_443/cac01 0/0/1/32/33 200 1859 - JSESSIONID=e0522474-098a-4655-ab --NR 1/1/0/0/0 0/0 "POST /pcoip-broker/xml HTTP/1.1"
Feb 28 17:42:17 anyware-proxy01 haproxy[3205]: IP:55096 [28/Feb/2023:17:42:17.176] fe_connector_443~ fe_connector_443/<NOSRV> -1/-1/-1/-1/3 400 0 - - CR-- 1/1/0/0/0 0/0 "<BADREQ>"
Feb 28 17:42:22 anyware-proxy01 haproxy[3205]: IP:55098 [28/Feb/2023:17:42:22.403] fe_connector_443~ be_connector_443/cac01 0/0/1/234/235 200 1212 JSESSIONID=s1~e0522474-098a-4655 JSESSIONID=e0522474-098a-4655-ab --VD 1/1/0/0/0 0/0 "POST /pcoip-broker/xml HTTP/1.1"
Feb 28 17:42:22 anyware-proxy01 haproxy[3205]: IP:55099 [28/Feb/2023:17:42:22.759] fe_connector_443~ fe_connector_443/<NOSRV> -1/-1/-1/-1/2 400 0 - - CR-- 1/1/0/0/0 0/0 "<BADREQ>"
Feb 28 17:42:24 anyware-proxy01 haproxy[3205]: IP:55100 [28/Feb/2023:17:42:22.845] fe_connector_443~ be_connector_443/cac01 0/0/2/1365/1367 200 2161 JSESSIONID=s1~e0522474-098a-4655 JSESSIONID=e0522474-098a-4655-ab --VD 1/1/0/0/0 0/0 "POST /pcoip-broker/xml HTTP/1.1"

Thank you for your help

1 Like