Error GET Request with chucked payload

We have deployed in Openshift an HAProxy to forward the traffic to upstream APIs.

The HAProxy is working fine and it can address thousands of request but, sometimes, it seems that there is an issue with a specific request. This request is a GET request where the payload is up to 10MB. Directly querying the upstream server works every time. The issue is never reproducible. if we try to perform the GET request going through the HAProxy it does not work all the times.

We created a bash script that is calling the same GET request with curl every 2 seconds, the script went ahead for ours and all the request where successful but suddenly curl started “hanging” Here the logs of a successful GET

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed

0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying XXX.XXX.XXX.XXX:5004...
* Connected to myhaproxy (XXX.XXX.XXX.XXX) port 5004 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
........
* Server certificate:
......
} [5 bytes data]
> GET /my/endpoint HTTP/1.1
> Host: myhost:5004
> User-Agent: curl/7.74.0
> Accept-Encoding: gzip,deflate
> Accept: application/json, application/*+json
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
....
* old SSL session ID is stale, removing

0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< max-forwards: 20
< transfer-encoding: chunked
< server: Jetty(9.4.48.v20220622)
< content-type: application/json
< vary: Accept-Encoding
< content-encoding: gzip
< 
{ [765 bytes data]

100   758    0   758    0     0    651      0 --:--:--  0:00:01 --:--:--   651
100  953k    0  953k    0     0   790k      0 --:--:--  0:00:01 --:--:--  790k

While this is the log when the issue happens for the very same endpoint

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed

0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying XXX.XXX.XXX.XXX:5004...
* Connected to myhaproxy (XXX.XXX.XXX.XXX) port 5004 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
......
* Server certificate:
......
} [5 bytes data]
> GET /my/endpoint HTTP/1.1
> Host: myhost:5004
> User-Agent: curl/7.74.0
> Accept-Encoding: gzip,deflate
> Accept: application/json, application/*+json
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
....
* old SSL session ID is stale, removing

0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< max-forwards: 20
< transfer-encoding: chunked
< server: Jetty(9.4.48.v20220622)
< content-type: application/json
< vary: Accept-Encoding
< content-encoding: gzip
< 
{ [765 bytes data]

100  198k    0  198k    0     0  94419      0 --:--:--  0:00:02 --:--:-- 94375
100  357k    0  357k    0     0   112k      0 --:--:--  0:00:03 --:--:--  112k
100  357k    0  357k    0     0  87817      0 --:--:--  0:00:04 --:--:-- 87796
100  357k    0  357k    0     0  70804      0 --:--:--  0:00:05 --:--:-- 73744
100  357k    0  357k    0     0  59313      0 --:--:--  0:00:06 --:--:-- 73744
100  357k    0  357k    0     0  51031      0 --:--:--  0:00:07 --:--:-- 32359
100  357k    0  357k    0     0  44779      0 --:--:--  0:00:08 --:--:--     0
100  357k    0  357k    0     0  39891      0 --:--:--  0:00:09 --:--:--     0
100  357k    0  357k    0     0  35965      0 --:--:--  0:00:10 --:--:--     0
....
100  357k    0  357k    0     0  11728      0 --:--:--  0:00:31 --:--:--     0
100  357k    0  357k    0     0  11363      0 --:--:--  0:00:32 --:--:--     0
100  357k    0  357k    0     0  11021      0 --:--:--  0:00:33 --:--:--     0
100  357k    0  357k    0     0  10107      0 --:--:--  0:00:36 --:--:--     0
100  357k    0  357k    0     0   9835      0 --:--:--  0:00:37 --:--:--     0
....
100  357k    0  357k    0     0     44      0 --:--:--  2:17:47 --:--:--     0
100  357k    0  357k    0     0     44      0 --:--:--  2:17:48 --:--:--     0
100  357k    0  357k    0     0     44      0 --:--:--  2:17:49 --:--:--     0
100  357k    0  357k    0     0     44      0 --:--:--  2:17:50 --:--:--     0
100  357k    0  357k    0     0     44      0 --:--:--  2:17:54 --:--:--     0
100  357k    0  357k    0     0     44      0 --:--:--  2:17:56 --:--:--     0
100  357k    0  357k    0     0     44      0 --:--:--  2:17:57 --:--:--     0

As you can see it’s hanging forever.

This is the haproxy config

global
  maxconn 4096
  master-worker
  log stdout len 8192 format raw local0 info
  stats timeout 2m
  default-path origin /opt/app-root/haproxy/
  hard-stop-after 15s

  tune.ssl.maxrecord 0
  tune.ssl.lifetime 275
  tune.ssl.cachesize 25000
  tune.ssl.default-dh-param 2048
  
  tune.bufsize 32768
  tune.maxrewrite 10240

resolvers resolvcnf
  parse-resolv-conf
  hold valid        50s
  hold nx           20s
  hold other        20s
  hold refused      20s
  hold timeout      20s
  hold obsolete     20s
  resolve_retries   5
  timeout resolve   3s
  timeout retry     3s

defaults httpdefaults
  mode http
  log global
  option httplog
  timeout check 5s
  timeout connect 5s
  timeout client 30s
  timeout server 90s
  timeout client-fin 3s
  timeout server-fin 90s
  timeout http-request 120s
  timeout http-keep-alive 60s
  timeout tunnel 1s
  compression algo gzip
  compression type text/html text/plain application/json
  log-format '.....'

In HAProxy I do not see any error logs. Any Idea what it could be or how to better investigate this issue?