High availability with replay (retry-on) and backup servers

I have a conceptional question to the high availability setup with HAProxy 2.2.

Our server setup is simple, we have production server S1 in one data center and a second one S2 in a different data center within a different country. On failure within one data center, the HAProxy switches to the backup server in the other data center.

For that, our backend config looked like this:

  maxconn 30000
  tune.bufsize 1048576
  option http-buffer-request
  option http-keep-alive
  option forwardfor
  timeout http-request 10s
  timeout connect 5s
  timeout client 30s
  timeout queue 15s
  timeout server 100s
  timeout http-keep-alive 10s
  retries 3 
  unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
  unique-id-header X-HA-Request-Id
  option log-health-checks
  no option logasap
  timeout client-fin 30s
  timeout tarpit 15s

backend A
  balance first
  option httpchk GET /healthcheck HTTP/1.0

  server S1_5557 check inter 5000
  server S2_5558 check inter 5000 backup

  retries 3
  option redispatch 1
  retry-on 502 503 504 empty-response conn-failure 0rtt-rejected

  http-request disable-l7-retry if METH_POST

We’ve noticed that in this setup, as soon as we shut down the S1 server, it takes 5sec until the S2 server takes over. Requests coming in between two healthchecks get a HTTP 503 and are not forwarded to the backup automatically.

The assumption was, that the retry-on option will trigger a replay of current request against the backup server as soon as a http 503 code is detected.

It works, if we remove the backup option from the S2 server. As the backup server is in a different country, we do not want to have it in constant use (by removing the backup option from it).

How would someone set up such a use case as we have it? The only requirement would be that no GET request gets lost and is always served by one of the environments.