Requests time out when uploading a zip file

Help!
1

Hello,

I configured Haproxy to forward requests to an internal dashboard as following:

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    #option http-server-close
    #option http-keep-alive
    #http-reuse always

#    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    #retries                 3
    retries                 10
    timeout http-request    10m
    timeout queue           1m
    #timeout connect         10s
    timeout connect         10m
    timeout client          10m
    timeout server          10m
    timeout http-keep-alive 10m
    timeout check           10m
    #timeout http-keep-alive 10s
    #timeout check           10s
    maxconn                 10000

frontend ray-dashboard
    bind *:32002
    default_backend ray-dashboard
    mode tcp
    option tcplog

backend ray-dashboard
    balance source
    mode tcp
    server director 10.1.1.208:8265 check

The current large timeouts are the result of a various attempt to overcome the current issue. In any case, this is the current setup.

Issue: When I try issue a PUT request to upload a zip or tar file to the ray-dashboard endpoint I get the error Recv failure: Operation timed out

Following an example:

─[mgazz@local❯❯❯❯
└❯time curl -v -X PUT -T _ray_pkg_0b527194125c16d4.zip http://9.2.252.61:32002/api/packages/gcs/_ray_pkg_0b527194125c16d4.zip
Note: Unnecessary use of -X or --request, PUT is already inferred.
*   Trying 9.2.252.61:32002...
* Connected to 9.2.252.61 (9.2.252.61) port 32002
> PUT /api/packages/gcs/_ray_pkg_0b527194125c16d4.zip HTTP/1.1
> Host: 9.2.252.61:32002
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 6875599
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
<
* Recv failure: Operation timed out
* Closing connection
curl: (56) Recv failure: Operation timed out

real    0m28.685s
user    0m0.007s
sys     0m0.027s

If I create a random file with the command dd if=/dev/urandom of=randomfile bs=7M count=1 iflag=fullblock I can successfully upload the file.

┌─[mgazz@local❯❯❯❯
└❯time curl -v -X PUT -T randomfile http://9.2.252.61:32002/api/packages/gcs/randomfile
Note: Unnecessary use of -X or --request, PUT is already inferred.
*   Trying 9.2.252.61:32002...
* Connected to 9.2.252.61 (9.2.252.61) port 32002
> PUT /api/packages/gcs/randomfile HTTP/1.1
> Host: 9.2.252.61:32002
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 7340032
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
<
* upload completely sent off: 7340032 bytes
< HTTP/1.1 200 OK
< Content-Length: 0
< Date: Wed, 02 Oct 2024 12:56:58 GMT
< Server: Python/3.10 aiohttp/3.10.6
<
* Connection #0 to host 9.2.252.61 left intact

real    0m14.033s
user    0m0.019s
sys     0m0.067s

The files are around the same size:

┌─[mgazz@local❯❯❯❯
└❯ls -lah
total 41208
drwxr-xr-x    7 mgazz  staff   224B  2 Oct 13:54 .
drwxr-xr-x  121 mgazz  staff   3.8K  1 Oct 10:20 ..
-rw-r--r--    1 mgazz  staff   6.6M  2 Oct 13:48 _ray_pkg_0b527194125c16d4.zip
drwxr-xr-x   31 mgazz  staff   992B 26 Aug 11:24 ad-orchestrator
-rw-r--r--    1 mgazz  staff   7.0M  2 Oct 13:54 randomfile

Note: Changing the mode to http did not solve the problem.

#haproxy config using http mode
frontend ray-dashboard-http
    bind *:32003
    default_backend ray-dashboard-http
    mode http

backend ray-dashboard-http
    balance source
    mode http
    server director 10.1.1.208:8265 check

Result:

❯ time curl -v -X PUT -T _ray_pkg_0b527194125c16d4.zip http://9.2.252.61:32003/api/packages/gcs/_ray_pkg_0b527194125c16d4.zip
Note: Unnecessary use of -X or --request, PUT is already inferred.
*   Trying 9.2.252.61:32003...
* Connected to 9.2.252.61 (9.2.252.61) port 32003
> PUT /api/packages/gcs/_ray_pkg_0b527194125c16d4.zip HTTP/1.1
> Host: 9.2.252.61:32003
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 6875599
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
<
* Recv failure: Operation timed out
* Closing connection
curl: (56) Recv failure: Operation timed out

Any idea why the upload of zip files return this error?

2

Did you check if bypassing haproxy helps or not? I’m wondering if that could be the server being unhappy with the PUT payload. Because as HAProxy is in TCP mode, I don’t see what would prevent the request from passing through

3

Thank you for the message and apologies for the late reply. Reaching directly the server from the internal networks without issues. We are now in contact with the network team to verify that package is not dropped by external components like a potential firewall.