Frontend 503 errors not produced by the balanced servers

Help!
1

I’m experiencing an issue where 503 errors are being logged by haproxy, specifically by the frontend (not the backend). There are no related errors logged by the backend servers (nginx). Here is some information:

defaults
log global
mode http
option httplog
option dontlognull
option log-separate-errors
maxconn 8000
timeout connect 5000
timeout client 1h
timeout server 1h
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend appssl
bind 192.168.2.52:443 ssl crt
log-format frontend:%f/%H/%fi:%fp\ %hrl\ %hsl\ CST:%Tl\ client:%ci:%cp\ backend:%s\ body:%[capture.req.hdr(0)]\ request:%r\ %ST
declare capture request len 40000
http-request capture req.body id 0
capture request header Referrer len 64
capture request header Content-Length len 10
capture request header User-Agent len 64
capture request header Host len 64
option http-buffer-request
option http-server-close
option forwardfor
reqadd X-Forwarded-Proto:\ https
reqadd X-Forwarded-Port:\ 443
# set HTTP Strict Transport Security (HTST) header
rspadd Strict-Transport-Security:\ max-age=15768000
default_backend ssl_pool

backend ssl_pool
balance roundrobin
balance roundrobin
cookie BALANCEID insert indirect nocache
option httpchk GET /load_balancer_check.php HTTP/1.1\r\nHost:\ oursite.com
http-check expect string pass
option http-buffer-request
server app01 192.168.2.174:443 ssl verify none check port 80 inter 30s cookie app01
server app02 192.168.2.109:443 ssl verify none check port 80 inter 30s cookie app02
server app03 192.168.2.247:443 ssl verify none check port 80 inter 30s cookie app03
server app04 192.168.2.122:443 ssl verify none check port 80 inter 30s cookie app04
server app05 192.168.2.102:443 ssl verify none check port 80 inter 30s cookie app05
server app06 192.168.2.160:443 ssl verify none check port 80 inter 30s cookie app06
server app07 192.168.2.201:443 ssl verify none check port 80 inter 30s cookie app07
server app08 192.168.2.208:443 ssl verify none check port 80 inter 30s cookie app08
server app09 192.168.2.112:443 ssl verify none check port 80 inter 30s cookie app09
server app10 192.168.2.189:443 ssl verify none check port 80 inter 30s cookie app10
server app11 192.168.2.148:443 ssl verify none check port 80 inter 30s cookie app11
server app12 192.168.2.163:443 ssl verify none check port 80 inter 30s cookie app12
server app13 192.168.2.187:443 ssl verify none check port 80 inter 30s cookie app13
server app14 192.168.2.233:443 ssl verify none check port 80 inter 30s cookie app14
server app15 192.168.2.164:443 ssl verify none check port 80 inter 30s cookie app15
server app16 192.168.2.117:443 ssl verify none check port 80 inter 30s cookie app16

And what the error looks like:

May 24 19:02:37
haproxy2 haproxy[7666]: frontend:appssl/haproxy2/192.168.2.52:443 - -
Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko mesval1234.oursite.com
CST:24/May/2016:19:02:36 -0500 client:75.173.99.247:58691 backend:app04 body:- request:GET
/js/zeroclipboard/ZeroClipboard.js HTTP/1.1 503

This is never logged on the backend (app04 here).

2

It clipped my entry here. The crt is a list of a few certs, as we utilize SNI.

3

Please remove your “log-format” configuration for the time beeing, as the log is useless this way.

4

We use that for information we want logwatch to pick up and email to us (like when these errors occur).

5

I understand that, but the reason of the 503 is not indicated the log line this way. If you want to know the reason of the 503, you need to revert to standard logging.

6

Gotcha, that makes more sense.

7

Removing the log format line actually didn’t revert back to default logging, it still logs in the same manner. However, it may not matter as I added redispatch (we rely heavily upon inserted cookie data to maintain session stickiness), and we haven’t had one of these since Wednesday (we were getting 3-5 per day).