Different/wrong number of sessions on backend?

Help!
1

I have a strange problem with HTTP load balancing. The issue is that number of sessions on the www-https frontend is drastically different compared to its backend. Here is a screenshot from the stats page how it looks like:

Screenshot-20190404-145332
Screenshot-20190404-145332.png2709×1323 296 KB

Check the number of sessions on www-https (4934) and on events-backend-https (157). What is even stranger is that if I check the number of established connections on that HAProxy host to the backend servers (which are all listening on 8080), I get this:

[root@events-client-haproxy-01 ~]# ss -nta state established 'dport = 8080' | wc -l
1981

which is a lot more than 157 that HAProxy is reporting.

Other thing that is strange is that on that same frontend session rate is 222, while on the backends that session rate is 1009?

I pasted below the whole config, but the most relevant part is that keep-alive should be used, because it’s not disabled anywhere. Also because option prefer-last-server is set, those keep-alive connections should be reused on the servers. On the servers keep-alive timeout is set to be 30s, which is the same as on HAProxy.

global
    log         /dev/log local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    nbthread 6
    maxconn     300000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats expose-fd listeners mode 0777 level admin
    server-state-file /var/lib/haproxy/server-state

defaults
    mode http
    log global
    maxconn 2000
    backlog     4000
    retries 2
    load-server-state-from-file global
    option httplog
    option dontlognull
    option dontlog-normal
    option forwardfor except 127.0.0.0/8
    option redispatch
    option prefer-last-server
    timeout http-request 5s
    timeout queue 5s
    timeout connect 5s
    timeout client 15m
    timeout server 15m
    timeout http-keep-alive 30s
    timeout check 5s

frontend tcp
    bind :7705
    mode tcp
    maxconn 200000
    default_backend events-backend-tcp

frontend www-http
    bind :80 alpn h2,http/1.1
    maxconn 1000
    acl is_trace_track method TRACE TRACK
    http-request deny if is_trace_track
    http-response set-header Strict-Transport-Security max-age=15768000
    default_backend events-backend-https

frontend www-https
    bind :443 ssl crt /etc/pki/tls/private/events.example.com.pem alpn h2,http/1.1
    maxconn 200000
    acl is_trace_track method TRACE TRACK
    http-request deny if is_trace_track
    http-response set-header Strict-Transport-Security max-age=15768000
    default_backend events-backend-https

backend events-backend-tcp
    mode tcp
    balance static-rr
    option tcp-check
    default-server inter 4s rise 2 fall 4 maxconn 50000 check agent-check agent-port 8081
    server events-client-app-01 events-client-app-01:7705
    server events-client-app-02 events-client-app-02:7705
    server events-client-app-03 events-client-app-03:7705
    server events-client-app-04 events-client-app-04:7705
    server events-client-app-05 events-client-app-05:7705

backend events-backend-https
    redirect scheme https if !{ ssl_fc }
    balance static-rr
    option httpchk GET /api/v1/health-check/simple-check
    default-server inter 4s rise 2 fall 4 maxconn 50000 check agent-check agent-port 8081
    server events-client-app-01 events-client-app-01:8080
    server events-client-app-02 events-client-app-02:8080
    server events-client-app-03 events-client-app-03:8080
    server events-client-app-04 events-client-app-04:8080
    server events-client-app-05 events-client-app-05:8080

listen stats
    bind :9000
    mode http
    stats enable
    stats hide-version
    stats uri /
    stats refresh 10s

Any ideas why is this happening?