1.6 - Combining HTTP Health Check and Agent Check

Help!
1

Hi,

If the HTTP check fails and the agent returns a feedback of “up 50%”, would HA Proxy mark that host as DOWN?

There was an incident where requests would continue to be routed to the host failed the HTTP health check and I’m wondering if it’s possible that the agent is returning “up XX%” which might mark it as UP instead of DOWN.

Here’s an example config:

backend apiservers
balance roundrobin
option httpchk GET /check
http-check expect status 200
server server1 192.168.50.3:80 check agent-check agent-inter 5s agent-addr 192.168.50.3 agent-port 5555

Thanks

2

If the agent-check returns “up”, the server will be marked up overwriting health checks:

  • The word “up” sets back the server’s operating state as UP if health checks
    also report that the service is accessible.

https://cbonte.github.io/haproxy-dconv/1.6/configuration.html#5.2-agent-check

3

Thanks - I’ve removed the agent-check troubleshoot the issue.

It turns out that the only requests that are going to the server marked as DOWN are the ones that have the session id that is stored in the stick table with the matching server id. Does anyone know if requests that have a match in the stick table will continue to be routed to the server marked as DOWN? I was under the impression that it should only route to UP servers.

In my backend configuration, I have defined the server ID for each server.

4

Yes, use option redispatch to break persistence in this case.

5

Thanks @lukastribus !

Along with the stick table, I actually have both option persist and option redispatch. In this case, should it have redistributed the request in case of failure? It seems like if there’s a match in the stick table, it sends the request to a DOWN server and doesn’t redispatch.

If I remove option persist and have only option redispatch, then it works as I expected, which is to route requests only to healthy servers.

Thanks

6

Yes, this is expected:

When an HTTP request reaches a backend with a cookie which references a dead server, by default it is redispatched to another server. It is possible to force the request to be sent to the dead server first using “option persist” if absolutely needed. A common use case is when servers are under extreme load and spend their time flapping. In this case, the users would still be directed to the server they opened the session on, in the hope they would be correctly served. It is recommended to use “option redispatch” in conjunction with this option so that in the event it would not be possible to connect to the server at all (server definitely dead), the client would finally be redirected to another valid server.