Backend NOSRV issue

Help!
1

Dear community,

Recently I have noticed an issue with our HAproxy serving backend on our nginx servers.
Basically we have haproxy in front of 2 nginx servers and every morning for like less than minute I am getting 503 NOSRV errors in haproxy logs, but at the same time fron nginx access logs I can see that there are requests going through with no issues. This is in the logs -

Dec  9 10:03:53 boost-2 haproxy[15199]: 80.232.255.170:59993 [09/Dec/2019:10:03:53.904] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 369/89/14/0/0 0/0 "GET /images/svg/days-recipe.svg HTTP/1.1"
Dec  9 10:03:53 boost-2 haproxy[15199]: 194.213.9.26:35652 [09/Dec/2019:10:03:53.917] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 371/91/14/0/0 0/0 "GET /export/mixed/ HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 87.110.50.14:57217 [09/Dec/2019:10:03:54.072] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 369/85/14/0/0 0/0 "GET /images/svg/days-recipe.svg HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 46.109.52.151:53813 [09/Dec/2019:10:03:54.115] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 366/84/14/0/0 0/0 "GET /example/example-url-1 HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 213.175.124.141:44632 [09/Dec/2019:10:03:54.186] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 365/86/14/0/0 0/0 "GET /export/mixed/ HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 81.198.123.2:53243 [09/Dec/2019:10:03:54.508] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 358/77/14/0/0 0/0 "GET /?_ga=2.48308399.820674435.1575875672-152970791.1575875669 HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 78.28.224.67:61335 [09/Dec/2019:10:03:54.705] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 367/80/14/0/0 0/0 "GET /export/mixed/ HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 212.3.196.55:39024 [09/Dec/2019:10:03:54.709] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 365/79/14/0/0 0/0 "GET /export/mixed/ HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 46.109.52.151:53812 [09/Dec/2019:10:03:54.775] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 355/76/14/0/0 0/0 "GET /favicon.ico HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 83.99.167.240:53850 [09/Dec/2019:10:03:54.897] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 354/75/14/0/0 0/0 "GET /export/mixed/ HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 84.237.170.220:34939 [09/Dec/2019:10:03:54.956] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/1 503 213 - - SC-- 362/84/14/0/0 0/0 "GET /example/example-url-2 HTTP/1.1"

And the config is as below

frontend frontend-http
  bind 222.155.222.242:80
  bind 222.155.222.243:80
  bind 222.155.222.242:443 ssl crt /etc/haproxy/ssl/certs/example.com.pem crt /etc/haproxy/ssl/certs/i.example.com.pem
  bind 222.155.222.243:443 ssl crt /etc/haproxy/ssl/certs/example.com.pem crt /etc/haproxy/ssl/certs/i.example.com.pem
  redirect scheme https code 301 if !{ ssl_fc }
  reqadd X-Forwarded-Proto:\ https if { ssl_fc }
  reqadd X-Forwarded-Port:\ 443 if { ssl_fc }
  reqadd X-Forwarded-Proto:\ http if !{ ssl_fc }
  reqadd X-Forwarded-Port:\ 80 if !{ ssl_fc }

  option forwardfor header X-GW-Client-IP
  http-request set-header X-GW-Client-IP %[src]

  acl host-img-example-com hdr_dom(host) -i -m reg ^(i.example.com)(:[0-9]+)?$
  acl host-www-example-com  hdr_dom(host) -i -m reg ^((www\.)?example.com)(:[0-9]+)?$
  acl host-example-com hdr_dom(host) -i -m reg ^(example.com)(:[0-9]+)?$
  use_backend backend-img-example-com if host-img-example-com
  use_backend backend-example-com if host-example-com


backend backend-example-com
  retries 3
  option redispatch
  option httpclose
  option forwardfor
  balance roundrobin
  option httpchk HEAD / HTTP/1.1\r\nHost:example.com
  # we use ssl cause backend checks generates sometiems output based on _backend_ was ssl or not
  server example-1  10.111.111.2:443 check ssl verify none
  server example-2  10.111.111.3:443 check ssl verify none

I would really appreciate any help/advice where should I begin to look at, thank you!

2

The SC error flag means that the network or the server refused the connection:

 SC   The server or an equipment between it and haproxy explicitly refused
      the TCP connection (the proxy received a TCP RST or an ICMP message
      in return). Under some circumstances, it can also be the network
      stack telling the proxy that the server is unreachable (e.g. no route,
      or no ARP response on local network). When this happens in HTTP mode,
      the status code is likely a 502 or 503 here.

Sure, just because some requests are rejected/refused doesn’t mean all of the requests are rejected. For example, if there are any rate limits on the backend server or intermediate devices rejected connections when crossing a threshold, that would explain why select connections are discarded.

You probably want to capture the traffic between haproxy and nginx from the haproxy box, to confirm.