Issue with HAProxy not retrying on retry-on

Help!
1

We were having issues with Apache mod_proxy getting random 502/503 errors from our backend server (that we don’t control), so we decided to give HAProxy a shot in testing. We setup HAProxy and got the same errors, so decided to try the retry-on all-retryable-errors but keep getting the same errors. We would have thought that HAProxy would have attempted retries on this but it doesn’t seem to be happening.

For testing, we do a WGET every half a second for 10000 tries. out of the 10000 tries, we get about 10 errors.

Didn’t know if someone could look at our setup and logs to help us determine why the retry isn’t occuring.

haproxy.cfg:

global
    log         127.0.0.1 local2 debug
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
frontend  main
    bind 127.0.0.1:5000
    default_backend             app
    mode http
backend app
    balance roundrobin
    http-send-name-header Host
    retry-on all-retryable-errors
    retries 10
    http-request disable-l7-retry if METH_POST
    server srv1 backend-server:443 ssl verify none
    server srv2 backend-server:443 ssl verify none
    server srv3 backend-server:443 ssl verify none

haproxy.log (you see the 502 error in the middle of the log)

Apr 27 19:01:29 localhost haproxy[26058]: 127.0.0.1:58028 [27/Apr/2022:19:01:29.769] main app/srv1 0/0/5/111/116 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:30 localhost haproxy[12306]: 127.0.0.1:58032 [27/Apr/2022:19:01:30.430] main app/srv2 0/0/2/119/121 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:31 localhost haproxy[8726]: 127.0.0.1:58036 [27/Apr/2022:19:01:31.099] main app/srv2 0/0/6/114/120 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:33 localhost haproxy[26058]: 127.0.0.1:58040 [27/Apr/2022:19:01:31.764] main app/srv2 0/0/6/-1/1385 502 209 - - SH-- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:33 localhost haproxy[8726]: 127.0.0.1:58044 [27/Apr/2022:19:01:33.695] main app/srv3 0/0/10/112/122 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:34 localhost haproxy[26058]: 127.0.0.1:58048 [27/Apr/2022:19:01:34.362] main app/srv3 0/0/3/113/116 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:45 localhost haproxy[8726]: 127.0.0.1:58052 [27/Apr/2022:19:01:35.023] main app/srv1 0/0/16/10552/10568 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"

output from the wget:

--2022-04-27 19:01:31--  http://localhost:5000/PBI_PBI1151/Login/RemoteInitialize/053103585
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:5000... connected.
HTTP request sent, awaiting response... 502 Bad Gateway
2022-04-27 19:01:33 ERROR 502: Bad Gateway.