Best practices for upgrading application server

Help!
1

When upgrading the applications behind haproxy I’ve been putting the specific backend into maintenance mode in haproxy, then upgrading the application and finally bringing up the backend in haproxy.

This is a bit cumbersome in our configuration management environment and it would be easier to just take the application server down and allow haproxy to fail over that backend. If haproxy checks backends every 15 seconds, will:

  1. some requests keep going to the dead backend for up to 15 seconds?
  2. haproxy notice the dead backend on the first request and mark it offline, but still lose that one request.
  3. haproxy notice the dead backend on the first request and mark it offline, and reroute that request to a good backend

If the answer is (3) and we lose no traffic at all, then we can do away with the cumbersome process of marking the backend as in maintenance.

Setup
haproxy 2.2.14
FreeBSD 11
Jetty powered application server
HTTP passthrough with SSL offloading on haproxy

2

Inflight transaction are NEVER interrupted, UNLESS on-marked-down shutdown-sessions is enabled.

If a cookies sticks a sessions to a specific server wich goes down, it will keep using that server, UNLESS option redispatch is enabled.

In fact, many people make the health check on a server manually fail by config changes on the server for such maintenance windows, so they don’t have to do it in haproxy manually. Haproxy by default will not overreact to this, it will merely stop distributing new session to the server in maintenance.

3

Do you know if haproxy will continue to send requests to the dead server until the next health check marks it as down?

4

If it’s a connection failure, and with option redispatch enabled and non-zero retries (default: 3), yes, it will try on another server despite the health check not yet seeing the server as down.

5

Great, thanks! So what I think I’ve learnt here:

  1. Set ‘retries’ to something greater than 0.
  2. Set ’ option redispatch’

Then if the application server behaves nicely and never abandons an request it is processing while shutting down (but it still rejects all new requests), then everything will work well and no requests should be lost at all during the shutdown of the application. All requests will be handed over to another server as soon as the dying server rejects the connections.

6

retries defaults to 3, and option redispatch should be necessary only if there is some kind of stickiness configured (like with cookies).