I am looking for some enlightenment on the topic of session response codes. Today I was trying to troubleshoot a problem and opened the stats page, as one does. I have used the stats page many times but today I noticed something that I had not before.
If I look at the HTTP 5xx responses on the frontend, I see many counted. In the example screenshot below the total is “5493”:
default frontend session totals:
If I look at the same HTTP 5xx responses row on the backend, I see zero counted.
backend server session totals:
To bear this out further, I opened the Prometheus metrics page. The metrics, of course, mirror the count that we saw in the screenshots:
haproxy_frontend_http_responses_total{proxy="default-frontend",code="5xx"} 5493
haproxy_server_http_responses_total{proxy="backend",server="srv001",code="5xx"} 0
haproxy_server_http_responses_total{proxy="backend",server="srv002",code="5xx"} 0
... etc
My question then is, if these 500 level responses are not coming from the backend servers, what is causing HAProxy to return those responses?
Am I to understand that the frontend responses are what HAProxy receives from trying to communicate with the various backend servers? And then the backend responses are only what is returned specifically from the target instances?
Thank you for helping me understand these data better. 