Context - running v2.0.5, with one frontend, and about 5 backends, all HTTP mode. Requests to frontend are routed to one of the backends based on URL pattern matching.
In one of the backends, the statistics page is reporting that over 50% of requests (of ~8 million, currently) are responding with 4xx codes, spread evenly across the 8 servers behind this. However, the HTTP access logs for those servers have no record of anywhere near that number of 4xx - they only contain a few 1000 rather than the ~4.4m that haproxy is currently reporting. The server http access logs do match up to the number of HTTP 2xx responses being reported by by haproxy. however.
I had thought haproxy would only report 4xx errors like this on a backend if it actually received this code back from the servers involved. Am I incorrect in i that? Any help explaining this situation appreciated!