Hello,
I configured Haproxy to forward requests to an internal dashboard as following:
defaults
mode http
log global
option httplog
option dontlognull
#option http-server-close
#option http-keep-alive
#http-reuse always
# option forwardfor except 127.0.0.0/8
option redispatch
#retries 3
retries 10
timeout http-request 10m
timeout queue 1m
#timeout connect 10s
timeout connect 10m
timeout client 10m
timeout server 10m
timeout http-keep-alive 10m
timeout check 10m
#timeout http-keep-alive 10s
#timeout check 10s
maxconn 10000
frontend ray-dashboard
bind *:32002
default_backend ray-dashboard
mode tcp
option tcplog
backend ray-dashboard
balance source
mode tcp
server director 10.1.1.208:8265 check
The current large timeouts are the result of a various attempt to overcome the current issue. In any case, this is the current setup.
Issue: When I try issue a PUT request to upload a zip or tar file to the ray-dashboard endpoint I get the error Recv failure: Operation timed out
Following an example:
─[mgazz@local❯❯❯❯
└❯time curl -v -X PUT -T _ray_pkg_0b527194125c16d4.zip http://9.2.252.61:32002/api/packages/gcs/_ray_pkg_0b527194125c16d4.zip
Note: Unnecessary use of -X or --request, PUT is already inferred.
* Trying 9.2.252.61:32002...
* Connected to 9.2.252.61 (9.2.252.61) port 32002
> PUT /api/packages/gcs/_ray_pkg_0b527194125c16d4.zip HTTP/1.1
> Host: 9.2.252.61:32002
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 6875599
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
<
* Recv failure: Operation timed out
* Closing connection
curl: (56) Recv failure: Operation timed out
real 0m28.685s
user 0m0.007s
sys 0m0.027s
If I create a random file with the command dd if=/dev/urandom of=randomfile bs=7M count=1 iflag=fullblock I can successfully upload the file.
┌─[mgazz@local❯❯❯❯
└❯time curl -v -X PUT -T randomfile http://9.2.252.61:32002/api/packages/gcs/randomfile
Note: Unnecessary use of -X or --request, PUT is already inferred.
* Trying 9.2.252.61:32002...
* Connected to 9.2.252.61 (9.2.252.61) port 32002
> PUT /api/packages/gcs/randomfile HTTP/1.1
> Host: 9.2.252.61:32002
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 7340032
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
<
* upload completely sent off: 7340032 bytes
< HTTP/1.1 200 OK
< Content-Length: 0
< Date: Wed, 02 Oct 2024 12:56:58 GMT
< Server: Python/3.10 aiohttp/3.10.6
<
* Connection #0 to host 9.2.252.61 left intact
real 0m14.033s
user 0m0.019s
sys 0m0.067s
The files are around the same size:
┌─[mgazz@local❯❯❯❯
└❯ls -lah
total 41208
drwxr-xr-x 7 mgazz staff 224B 2 Oct 13:54 .
drwxr-xr-x 121 mgazz staff 3.8K 1 Oct 10:20 ..
-rw-r--r-- 1 mgazz staff 6.6M 2 Oct 13:48 _ray_pkg_0b527194125c16d4.zip
drwxr-xr-x 31 mgazz staff 992B 26 Aug 11:24 ad-orchestrator
-rw-r--r-- 1 mgazz staff 7.0M 2 Oct 13:54 randomfile
Note: Changing the mode to http did not solve the problem.
#haproxy config using http mode
frontend ray-dashboard-http
bind *:32003
default_backend ray-dashboard-http
mode http
backend ray-dashboard-http
balance source
mode http
server director 10.1.1.208:8265 check
Result:
❯ time curl -v -X PUT -T _ray_pkg_0b527194125c16d4.zip http://9.2.252.61:32003/api/packages/gcs/_ray_pkg_0b527194125c16d4.zip
Note: Unnecessary use of -X or --request, PUT is already inferred.
* Trying 9.2.252.61:32003...
* Connected to 9.2.252.61 (9.2.252.61) port 32003
> PUT /api/packages/gcs/_ray_pkg_0b527194125c16d4.zip HTTP/1.1
> Host: 9.2.252.61:32003
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 6875599
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
<
* Recv failure: Operation timed out
* Closing connection
curl: (56) Recv failure: Operation timed out
Any idea why the upload of zip files return this error?