502 Bad Gateway with SH flag on the log

Help!
1

hello,

I am struggling with a problem for about a week. looked up everything here and everywhere on the internet.

I am using:
[root@srv-lba04 haproxy]# haproxy -v
HAProxy version 2.4.22-f8e3218 2023/02/14 - https://haproxy.org/
Running on: Linux 5.14.0-503.21.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Dec 19 09:37:00 EST 2024 x86_64

config:

global
log 127.0.0.1 local2 debug
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 5000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
ssl-default-bind-ciphers PROFILE=SYSTEM
ssl-default-server-ciphers PROFILE=SYSTEM

ssl-server-verify none
tune.ssl.cachesize 1000000
tune.ssl.lifetime 1800
tune.ssl.default-dh-param 2048

tune.http.maxhdr 8192

tune.maxrewrite 4096
tune.bufsize 65535
nbthread 4
ssl-default-bind-options no-sslv3     # Disable SSLv3 to mitigate POODLE attack

defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 4m
timeout connect 10s
timeout client 4m
timeout server 4m
timeout http-keep-alive 10s
timeout check 10s
maxconn 5000

frontend cbsapiexternal
log global

bind 10.190.18.124:443 ssl ca-file /etc/haproxy/root.pem crt /etc/haproxy/wildcard.pem no-tls-tickets ciphers ECDHE-RSA-AES256-SHA:RC4-SHA:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM

bind 10.190.18.124:80
option forwardfor
option httplog
http-request add-header X-Forwarded-For %[src]
default_backend cbsapiexternal

backend cbsapiexternal
log global

retry-on conn-failure empty-response

retry-on all-retryable-errors
http-reuse never
option redispatch 2
retries 4
option forwardfor
option http-server-close
server disp02 srv-disp02:11223 check

http://srv-disp02:11223 work from my desktop, also works if I do a curl from the server.
behind that port there are some API services (using GET, not POST), something called zato.

it is to be noted that the exact same configuration works perfectly on Haproxy 1.5.18 (ergo the need for upgrade).
no matter how many options I tried in the backend or global… same result.

I tried every option I could find - same result when accessing the frontend. 502 Bad Gateway.

the log:
Jan 20 22:34:16 localhost haproxy[262438]: 172.16.132.15:50784 [20/Jan/2025:22:34:16.588] cbsapiexternal cbsapiexternal/disp02 0/0/0/-1/1 502 209 - - SH-- 1/1/0/0/0 0/0 “GET / HTTP/1.1”

from the stats, nothing.

[root@srv-lba04 haproxy]# echo “show errors” | socat stdio /var/lib/haproxy/stats

Total events captured on [20/Jan/2025:22:41:18.597] : 0

any ideas?

2

Here’s the meaning of SH in the failed 502 request (according to the manual):

     SH   The server aborted before sending its full HTTP response headers, or
          it crashed while processing the request. Since a server aborting at
          this moment is very rare, it would be wise to inspect its logs to
          control whether it crashed and why. The logged request may indicate a
          small set of faulty requests, demonstrating bugs in the application.
          Sometimes this might also be caused by an IDS killing the connection
          between HAProxy and the server.

Perhaps the application is unhappy with something from the request. Did you get a chance to check application’s logs?
I would also recommend to capture the request when made from your browser directly vs when it goes through haproxy and see what change in the request could cause the server to fail on the request.

Comparing the request with haproxy 1.8 vs 2.4 could help also to figure what changed between the 2 versions that the server is not happy with

3

Hello,

Thank you for the answer.

The issue was - The firewall was dropping the requests.
Although we looked at the firewall 10 times, the issue was how the firewall works.

We are using a Palo Alto Networks firewall that has enabled the use of X-Forwarded-For option, that basically obliges you to have rules from the originating IPs (as source) to the backends.
It keeps the IP of the load balancer as source, but the rules use the X-Forwarded-For as true source for the firewall rules.

Although during investigation we were looking at traffic from the load balancer to the backend - and it was allowed , we didn’t think to check other traffic attempting to access the backends.

Obviously this was discovered with a traffic dump :slight_smile:

Thank you

1 Like