Is there a way to log the HTTP headers going to a backend? I’m getting HTTP 400 Bad Request from a backend server and I need to figure what headers we are sending towards a backend.
You run haproxy in debug mode, but it’s way easier to just run tcpdump and capture it yourself, that will also certainly not be biased, but haproxy may be (if there is a bug).
That’s said if haproxy is rejecting a request, you can use the show errors command on the admin socket to understand the reason haproxy rejected the request (or response). But that doesn’t seem to be your use-case here.
Just to add if it helps, I usually use this tcpdump command:
tcpdump -A -s 10240 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' | egrep --line-buffered "^........(GET |HTTP\/|POST |HEAD )|^[A-Za-z0-9-]+: " | sed -r 's/^........(GET |HTTP\/|POST |HEAD )/\n\1/g'
Unfotunately I dont have root access to run TCPDUMP and it’s not HAproxy rejecting the request. one of the HAproxy backend is rejecting the request, so I need to capture the headers going south bound.
when I ran the “show errors” on nc- its always shows as ‘0’.
any other suggestions? Appreciate your help!
How do you change the haproxy configuration when you are not root?
I’m running HAProxy as non-root
Ok, then the only possibility is to run this request through haproxy in debug mode.
hmm-when you say HAProxy in debug mode- all I need to add “debug” word in global section? Possible to give a sample please?
No, stop haproxy, and start it manually from the command line with the -d argument:
haproxy -f /path/to/haproxy.cfg -d
You will then see the entire requests and response in your terminal. Stop haproxy with ctrl-C and start haproxy again via your process manager (or whatever method you use to start haproxy usually).
I did that- all I see is incoming headers and the 400 error. here is what Im seeing-
0000004f:http_frontend.clireq[000b:ffffffff]: POST https://x.x.x.x:443/Event/Publish/AccountActivityEvent HTTP/1.1
0000004f:http_frontend.clihdr[000b:ffffffff]: SOAPAction: “createAccountActivityEvent”
0000004f:http_frontend.clihdr[000b:ffffffff]: Content-Type: text/xml;charset=UTF-8
0000004f:http_frontend.clihdr[000b:ffffffff]: User-Agent: Jakarta Commons-HttpClient/3.1
0000004f:http_frontend.clihdr[000b:ffffffff]: Host:x.x.x.x:443
0000004f:http_frontend.clihdr[000b:ffffffff]: Content-Length:1351
0000004f:AccountActivityEvent.createAccountActivityEvent.srvrep[000b:000d]: HTTP/1.1 400 Bad Request
0000004f:AccountActivityEvent.createAccountActivityEvent.srvhdr[000b:000d]: Content-Type: text/plain
0000004f:AccountActivityEvent.createAccountActivityEvent.srvhdr[000b:000d]: Connection: close
0000004f:AccountActivityEvent.createAccountActivityEvent.srvcls[000b:000d]
0000004f:AccountActivityEvent.createAccountActivityEvent.clicls[000b:000d]
0000004f:AccountActivityEvent.createAccountActivityEvent.closed[000b:000d]
I relied to your thread yesterday - but for some reason my post is hidden. I started HAP in foreground with “-d”
and all I could see is following-
0000004f:http_frontend.clireq[000b:ffffffff]: POST https://x.x.x.x:443/Event/Publish/AccountActivityEvent HTTP/1.1
0000004f:http_frontend.clihdr[000b:ffffffff]: SOAPAction: “createAccountActivityEvent”
0000004f:http_frontend.clihdr[000b:ffffffff]: Content-Type: text/xml;charset=UTF-8
0000004f:http_frontend.clihdr[000b:ffffffff]: User-Agent: Jakarta Commons-HttpClient/3.1
0000004f:http_frontend.clihdr[000b:ffffffff]: Host:x.x.x.x:443
0000004f:http_frontend.clihdr[000b:ffffffff]: Content-Length:1351
0000004f:AccountActivityEvent.createAccountActivityEvent.srvrep[000b:000d]: HTTP/1.1 400 Bad Request
0000004f:AccountActivityEvent.createAccountActivityEvent.srvhdr[000b:000d]: Content-Type: text/plain
0000004f:AccountActivityEvent.createAccountActivityEvent.srvhdr[000b:000d]: Connection: close
0000004f:AccountActivityEvent.createAccountActivityEvent.srvcls[000b:000d]
0000004f:AccountActivityEvent.createAccountActivityEvent.clicls[000b:000d]
0000004f:AccountActivityEvent.createAccountActivityEvent.closed[000b:000d]
So you got what you needed.
Or do you need help with the actual troubleshooting as well? In that case, please upload the log output to something like pastebin, because the forum will mess with this logs so that it is impossible for me to tell what the forum did and what is really in the original request.
Also share your entire configuration please (replace actual hostnames and IPs, but share everything else).
Finally figured out that- there was an extra space in URI sent by client. Removed that extraspace using reqrep