For the issue mchesler mentioned, here is a sample log of the 5xx responses:
Oct 10 19:09:53 nikita haproxy: 188.8.131.52:20049 [10/Oct/2017:19:09:53.881] node_http node/10_0_1_237 0/0/-1/-1/0 503 212 - - CC-- 6193/6097/550/55/0 0/0 "GET /p/rp/239842934823952.png HTTP/1.1"
Oct 10 19:09:53 nikita haproxy: 184.108.40.206:29793 [10/Oct/2017:19:09:53.884] node_http node/10_0_1_245 0/0/-1/-1/0 503 212 - - CC-- 6190/6094/548/47/0 0/0 "GET /p/rp/348293248248349.png HTTP/1.1"
Oct 10 19:09:53 nikita haproxy: 220.127.116.11:48497 [10/Oct/2017:19:09:53.892] node_http node/10_0_0_120 0/0/-1/-1/1 503 212 - - CC-- 6185/6089/548/46/0 0/0 "GET /p/rp/435435324324232.png HTTP/1.1"
HAProxy is logging it as a 503, but the connection string is
CC, which I believe means that the client disconnected before the backend server was able to establish a connection. But the more curious part was that the total request time is 0ms, which would seem to mean that the backend wasn’t given any time to accept the connection anyway.
If my understanding of it is correct it seems like these errors are safe to ignore, except I can’t reproduce these errors synthetically, only in production. If these are not real errors, is there a way to prevent them from showing up in our stats, or if they are real errors is there a way to track down what the errors actually are?