Backend server timeout config for slow reponse server

Help!
1

I am using HAProxy 2.9.1 and I am trying to setup 2 backends as follows:

  1. Backend named svelte

This backend is fast to reponse i.e. it could take 0.1s to reponse.

Example: time to make http request

❯ time curl -I http://localhost:5173/
HTTP/1.1 404 Not Found
Access-Control-Allow-Origin: *
content-length: 928
content-type: text/html
etag: "qrysmh"
x-sveltekit-page: true
Date: Wed, 10 Jan 2024 07:20:20 GMT
Connection: keep-alive
Keep-Alive: timeout=5

curl -I http://localhost:5173/  0.00s user 0.00s system 38% cpu 0.022 total
  1. Backend named magento

This backends is slow to reponse, i.e it could take 5s or more to reponse.

Example time to make http request:

❯ time curl -I http://localhost:8090
HTTP/1.1 200 OK
Server: nginx/1.25.3
Date: Wed, 10 Jan 2024 07:20:35 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Powered-By: PHP/7.0.33
Set-Cookie: created_timestamp=1704871230; expires=Thu, 09-Jan-2025 07:20:30 GMT; Max-Age=31536000; path=/
Set-Cookie: frontend=2c9ccb6743227aed1ddaf2955986a516; expires=Wed, 17-Jan-2024 07:20:31 GMT; Max-Age=604800; path=/; domain=localhost; HttpOnly
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
X-Frame-Options: SAMEORIGIN

curl -I http://localhost:8090  0.00s user 0.01s system 0% cpu 4.861 total

And, the haproxy.cfg config file looks as follows:

global
    log 127.0.0.1   local0
    log 127.0.0.1   local1 debug
    maxconn 4096

  defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    retries 3
    option redispatch
    maxconn 2000
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 500000ms

  frontend localnodes
    bind *:8000
    mode http
    default_backend magento
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST 

    use_backend svelte if { path /a }
    use_backend magento if { path /b } || { path_beg /b/ }

  backend svelte
    mode http
    balance roundrobin
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    server server1 localhost:5173 check    

  backend magento
    mode http
    balance roundrobin
    server server1 localhost:8090 check

When I check in http://localhost:8000/stats as shown in this picture:

haproxy-stats
haproxy-stats2860×1134 418 KB

The backend named magento is always down, no matter how I change timeout connect, timeout client, timeout server.

So, I’m wondering if there is any way to setup haproxy to make it work with slow response backend ?
or in the other word, it’s kind of known server that slow reponse but we want HAProxy to mark as UP (not DOWN)? What specific config to adjust to work with particular server or backend.

Any idea or suggestion would be really appreciated.

Thanks,

2

Read about timeout check and inter keywords:

https://docs.haproxy.org/2.8/configuration.html#4.2-timeout%20check

3

Thanks for pointing out, appreciated and helpful.
I have just checked the timeout check and inter out.

And trying to test it (I am trying to understand) by making the haproxy.cfg config file to be simpler and have 1 server like so:

global
    maxconn 100
defaults
    log global
    mode tcp
    retries 2
    timeout client 30m
    timeout connect 4s
    timeout server 30m
    timeout check 5s
 
listen prd_stats
    mode http
    bind *:8001
    stats enable
    stats uri /stats

listen magento
    bind *:8000
    option httpchk
    http-check expect status 200
    default-server inter 5s fall 3 rise 2 on-marked-down shutdown-sessions
    server server1 localhost:8090 check

And, the details time of the server I’m trying to add is like so:

❯ curl -w @- -o /dev/null -s http://localhost:8090 <<'EOF'
    time_namelookup:  %{time_namelookup}\n
       time_connect:  %{time_connect}\n
    time_appconnect:  %{time_appconnect}\n
   time_pretransfer:  %{time_pretransfer}\n
      time_redirect:  %{time_redirect}\n
 time_starttransfer:  %{time_starttransfer}\n
                    ----------\n
         time_total:  %{time_total}\n
EOF
    time_namelookup:  0.002603
       time_connect:  0.002872
    time_appconnect:  0.000000
   time_pretransfer:  0.002888
      time_redirect:  0.000000
 time_starttransfer:  4.695344
                    ----------
         time_total:  4.712296

But, still HAProxy still mark the status as DOWN.

8090
80901794×844 164 KB

I think I am reading HAProxy version 2.9.1-33 - Configuration Manual quite carefully now. But, for this test, it gives the expected result.

Any other suggestions to try out ?
or, in worst case, any possible way to just make the haproxy always forward to the server without check ?

4

My bad, I think I was confused by myself.

To debug, I just try turning the log on:

global
    log stdout local0 debug

And, then I can see the following message:

 Server magento/server1 is DOWN, reason: Layer4 connection problem, info: "Connection refused"

it shows the actual root cause and it is not related to timeout at all.

Why so, because I run the server:8090 in docker-compose and I think the resolver is somehow not be able to resolve localhost:8090 in haproxy but other programs like curl or chrome is fine (which causes my cufusion at the first place).

So, I just simply use 127.0.0.1 for this testing.

Note for someone who might stumble across to this thread, might be useful to check out DNS resolution | HAProxy config tutorials
But, I haven’t configured this yet.

1 Like