Thanks again for such great software
I am using haproxy 2.0.17 with such configuration:
global pidfile haproxy.pid master-worker log rsyslogd.sck local0 defaults log global mode http option httplog timeout queue 60s timeout server 600s timeout client 600s timeout connect 5s retries 3 frontend http-backend bind <haproxyipv4>:21080 acl is_BACKEND_NN hdr_reg(host) -i ^NN.example.com($|:.*) use_backend BACKEND_NN-http if is_BACKEND_NN frontend https-backend bind <haproxyipv4>:21443 acl is_BACKEND_NN hdr_reg(host) -i ^NN.example.com($|:.*) use_backend BACKEND_NN-https if is_BACKEND_NN backend BACKEND_NN-http timeout server 600s timeout connect 5s retries 3 server backend ip:port backend BACKEND_NN-https timeout server 600s timeout connect 5s retries 3 server backend ip2:port
Where NN results with about 500 entries.
Such configuration client is Apache Traffic Server 7.1. All is running fine for 99,95% of cases, but I have sometimes issue logged like this in my haproxy log:
ipA:38644 [24/Nov/2020:15:13:24.782] https-backend https-backend/<NOSRV> -1/-1/-1/-1/0 400 211 - - CR-- 31/2/0/0/0 0/0 "<BADREQ>" ipB:57014 [24/Nov/2020:15:13:26.248] http-backend http-backend/<NOSRV> -1/-1/-1/-1/0 400 211 - - CR-- 31/30/0/0/0 0/0 "<BADREQ>"
I am trying to find out more information on haproxy side regarding those client errors (if I read correctly the CR-- termination status).
While running with
-d mode I was able to see such entries:
00000000:https-backend.accept(000a)=0010 from [ipA:38644] ALPN=<none> 00000000:https-backend.clicls[0010:ffffffff] 00000000:https-backend.closed[0010:ffffffff] 00000001:http-backend.accept(0009)=0010 from [ipB:57014] ALPN=<none> 00000001:http-backend.clicls[0010:ffffffff] 00000001:http-backend.closed[0010:ffffffff]
So, it does not give me enough of information on the haproxy side.
My questions are:
- Is it possible to get more information in debugging log regarding those requests? (Using tcpdump or something might be hard, having in mind, that this is so small percentage of problems.)
- Is it known issue with the configuration I have? Maybe I shall use option clitcpka or something similar, to keep the client not resetting the connection?
- It seems not related to haproxy timeout, as then the termination would have small ‘c’ instead of capital ‘C’, am I right?
For reference the ATS side reports:
20201124.15h07m16s CONNECT: could not connect to <haproxyipv4> for 'http://<haproxyipv4>:21080/...PATH...' (setting last failure time) 20201124.15h07m16s RESPONSE: sent 10.0.241.200 status 502 (Server Hangup) for 'http://<haproxyipv4>:21080/...PATH.../
As you can see times here are a bit before the things seen in haproxy itself (it’s running on the same machine).
Also ATS was working well with it’s current configuration when it was connecting to different software, but maybe putting haproxy there just revealed wrong configuration, but before continuing, I’d like to have more information from haproxy itself.
Note: ATS neither haproxy are NOT reloaded around this problem occurs.