Intermittent request failure with termination code "KH"

Hi,

We are using HAProxy 1.7.9, with keep-alive connection and have overridden connection-timeout to 10m and keep-live is 10s.

Intermittently request is failing with termination code KH, with response code 502.please find below a sample of request timings.
0/0/0/-1/85072
0/0/0/-1/60879
0/0/0/-1/93077
0/0/0/-1/82248
0/0/0/-1/112680
0/0/0/-1/72457
0/0/0/-1/72262
0/0/0/-1/66217
0/0/0/-1/63302
0/0/1/-1/65023
0/0/0/-1/297599
0/0/0/-1/84934
0/0/1/-1/86788
0/0/0/-1/86192
0/0/1/-1/69071
0/0/0/-1/260637
0/0/0/-1/120751

So my understanding of failure is:

  • Backend is still fetching the data, but HAproxy thinks the connection is idle and closing the premature connection?
  • Failover/ load balancing: Requests which are being served by HAProxy leader instance, and afterload balancing are no longer a leader, and the existing connection is being treated as idle

If anyone has faced a similar problem? and if you know the workaround.

I am also wondering if it is worth to try option prefer-last-server – to redirect calls to the same box
or `option srvtcpka’ – to get packets from backed to HA

Thanks in advance!