Randomly respond 502 error on only some backends

Help!
1

Haproxy randomly respond 502 error on only some host. There is multiple host working on our backend server with kubernetes node. If I call request directly to kubernetes node it works okay, during the 502 error. The error persist only few minutes to hours.

For example:

Our admin panel nextjs service working on kubernetes responding randomly 502.
admin.example.com - 502

The other services such as backend API service working fine that time.
api.example.com - 200 OK

global
        log 127.0.0.1:514 local0
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        tune.maxrewrite 4096
        tune.bufsize 32768
        tune.http.maxhdr 200
        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-
ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
frontend fe
    bind *:443 ssl crt /root/cert-bundle-2023.pem
    bind *:80
    http-request redirect scheme https code 301 if !{ ssl_fc }
    default_backend be-http
    log 127.0.0.1 local0 info
backend be-http
    server static 192.168.1.1:80 check

HAProxy version 2.4.24-0ubuntu0.22.04.1 2023/10/31 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2026.
Known bugs: http://www.haproxy.org/bugs/bugs-2.4.24.html
Running on: Linux 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64
1 Like