Receiving NOSRV BADREQ when discrepancy between CONNECT and Host header

olifur · October 20, 2020, 3:31pm

Dear community,

We have a haproxy 2.2.4-de45672 2020/09/30 working as a router proxy for another proxy (just for the records: a squid webproxy, but which should not matter here)
Here is our haproxy config

global
log stdout format raw local0 info
stats socket /var/run/haproxy.stat
daemon
maxconn 256

defaults
log global
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms

listen stats
bind :9999
stats enable
stats hide-version
stats uri /stats
stats auth xxx:yyyy

frontend proxy_in
log global
option httplog
option logasap
bind :8888
use_backend proxies_out

backend proxies_out
cookie SERVERID insert indirect nocache
option forceclose
option forwardfor
balance roundrobin
mode http
server webproxy squid:8080

Now the problem with this haproxy and configuration is that using curl, we can test and send requests to the haproxy which work and returns us the websites content
(going through the squid webproxy)

$ curl https://abcdefghi.jklmnopqrs.de -v -x http://:8888

Output in Haproxy Log:

10.60.1.131:43080 [19/Oct/2020:09:50:37.111] proxy_in proxies_out/squid 0/0/4/32/+36 200 +124 - - --NI 1/1/1/1/0 0/0 "CONNECT abcdefghi.jklmnopqrs.de:443 HTTP/1.1"

Now the problem is from a application different than curl we get the following Haproxy error, when running the same request as we did with curl.

10.60.1.131:43080 [19/Oct/2020:09:58:54.111] proxy_in proxy_in/<NOSRV> -1/-1/-1/-1/+0 400 +211 - - PR-- 1/1/0/0/0 0/0 "<BADREQ>"

We can identify the only difference between the two requests using tcpdump, which is the port number in the Host header.

Non-Working version, see the difference between CONNECT and Host header, CONNECT has the port, the host header not.
The Application is a standard Java Webapplication using default HTTP libraries.

P…CONNECT.
abcdefghi.jklmno
pqrs.de:443.HTTP
/1.1…User-Agent
:.Java/1.8.0_252
…Host:.abcdefgh
i.jklmnopqrs.de.
.Accept:.text/ht
ml,.image/gif,.i
mage/jpeg,.;.q=
.2,./*;.q=.2…P
roxy-Connection:
.keep-alive…

Working curl version, see there is no difference between the CONNECT and Host header, both have the port appended.

E…@.?..dC…
dC…“…S”.:.
P…CONNECT.
abcdefghi.jklmno
pqrs.de:443.HTTP
/1.1…Host:.abcd
efghi.jklmnopqrs
.de:443…User-Ag
ent:.curl/7.29.0
…Proxy-Connecti
on:.Keep-Alive…

As this is the only remarkable difference between those two requests, we think it is due to that difference between CONNECT and Host header.

Now since we cannot change the applications code I wonder if there is an HAProxy option / flag or setting which disable this check between CONNECT and Host
header resulting in a BADREQ.

Thank you very much for your help
Oliver

Note: 10/21/20 edited the haproxy log output from blockquote to preformatted_text as the blockquote killed its formatting.

lukastribus · October 20, 2020, 5:07pm

I disagree with your analysis, I don’t think haproxy looks at the Host header at all and it certainly doesn’t care about the port.

Can you provide a .cap file with the full HTTP request and response in a working and non-working configuration?

olifur · October 21, 2020, 6:38am

Hi Lukas,

Thank you very much for your answer.
We could distill the test even more and reproduce the problem with only curl, have a look:

Negative Test (without Host port)

curl -H “Host: google.com” https://google.com/ -v -x http://haproxy-url:8888

Result:

10.60.1.131:43080 [21/Oct/2020:06:32:11.775] proxy_in proxy_in/<NOSRV> -1/-1/-1/-1/+0 400 +211 - - > PR-- 2/2/0/0/0 0/0 "<BADREQ>"

Positive Test:

curl -H “Host: google.com:443” https://google.com/ -v -x http://100.68.183.121:8888

Result:

10.60.1.131:43080 [21/Oct/2020:06:32:18.298] proxy_in proxies_out/squid 0/0/4/19/+23 200 +124 - - --NI 1/1/1/1/0 0/0 "CONNECT google.com:443 HTTP/1.1"

Can someone confirm this problem using our config and haproxy version?

We cannot provide you with a PCAP file at the moment, as we need to remove all cooperate information in it.

Best,
Oliver

rage2dev · October 21, 2020, 7:23am

Hej Guys,

I was able to recreate the problem.

Setup:

Newest Manjaro (5.9.1)
haproxy (2.2.3-0e58a34 2020/09/08)
tinyproxy

haproxy.cfg:

global
log stdout format raw local0 debug
daemon
maxconn 256

defaults
log global
mode http
option httpclose
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms

listen stats
bind :9999
stats enable
stats hide-version
stats uri /stats
stats auth admin:admin

frontend proxy_in
option httplog
option logasap
bind :8888
use_backend proxies_out

backend proxies_out
cookie SERVERID insert indirect nocache
option forwardfor header X-Client
balance roundrobin
server prox 127.0.0.1:8080

Working:
curl --proxy-header “Host: www.google.de:443” https://www.google.de -v -x localhost:8888

*   Trying 127.0.0.1:8888...
* Connected to localhost (127.0.0.1) port 8888 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to www.google.de:443
> CONNECT www.google.de:443 HTTP/1.1
> User-Agent: curl/7.72.0
> Proxy-Connection: Keep-Alive
> Host: www.google.de:443
> 
< HTTP/1.0 200 Connection established
< proxy-agent: tinyproxy/1.10.0
< set-cookie: SERVERID=; Expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!

Not Working:

 curl --proxy-header "Host: www.google.de" https://www.google.de -v -x localhost:8888
*   Trying 127.0.0.1:8888...
* Connected to localhost (127.0.0.1) port 8888 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to www.google.de:443
> CONNECT www.google.de:443 HTTP/1.1
> User-Agent: curl/7.72.0
> Proxy-Connection: Keep-Alive
> Host: www.google.de
> 
< HTTP/1.1 400 Bad request
< content-length: 90
< cache-control: no-cache
< content-type: text/html
< connection: close
< 
* Received HTTP code 400 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 400 from proxy after CONNECT

I could upload a pcap-file, but there is nothing to see. If the Port in the Host-Header is missing while connecting to a 443 URI, there will be a instant HTTP code 400 from haproxy. Same http-package with port appended to the Host-header, and it works.

EDIT:

haproxy with debug:

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
[CACHE] cache
[FCGI] fcgi-app
Using epoll() as the polling mechanism.
Proxy stats started.
Proxy proxy_in started.
Proxy proxies_out started.
[NOTICE] 294/093433 (20479) : New worker #1 (20480) forked

Working request:

00000000:proxy_in.accept(0007)=0037 from [127.0.0.1:39792] ALPN=
00000000:proxy_in.clireq[0037:ffffffff]: CONNECT www.google.de:443 HTTP/1.1
00000000:proxy_in.clihdr[0037:ffffffff]: user-agent: curl/7.72.0
00000000:proxy_in.clihdr[0037:ffffffff]: proxy-connection: Keep-Alive
00000000:proxy_in.clihdr[0037:ffffffff]: host: www.google.de:443
00000000:proxies_out.srvrep[0037:0038]: HTTP/1.0 200 Connection established
00000000:proxies_out.srvhdr[0037:0038]: proxy-agent: tinyproxy/1.10.0
127.0.0.1:39792 [21/Oct/2020:09:34:39.636] proxy_in proxies_out/pro-int 0/0/0/33/+33 200 +151 - - --NI 1/1/1/1/0 0/0 “CONNECT www.google.de:443 HTTP/1.1”
00000000:proxies_out.srvcls[0037:0038]
00000000:proxies_out.clicls[0037:0038]
00000000:proxies_out.closed[0037:0038]

Not working request (Missing :443 in Host-Header)

00000001:proxy_in.accept(0007)=0037 from [127.0.0.1:39804] ALPN=
00000001:proxy_in.clicls[0037:ffffffff]
00000001:proxy_in.closed[0037:ffffffff]
127.0.0.1:39804 [21/Oct/2020:09:34:43.399] proxy_in proxy_in/ -1/-1/-1/-1/+0 400 +211 - - PR-- 1/1/0/0/0 0/0 “”

Best,
rage2dev

lukastribus · October 21, 2020, 7:56am

As far as I can see, this is RFC mandated behavior:

A client sending a CONNECT request MUST send the authority form of request-target (Section 5.3 of [RFC7230]); i.e., the request-target consists of only the host name and port number of the tunnel destination, separated by a colon. For example,
CONNECT server.example.com:80 HTTP/1.1
Host: server.example.com:80

rage2dev · October 21, 2020, 10:34am

Hi Lukas,

I guess you are right. Looks like haproxy is really strict about the http-request. For example Tinyproxy works with the same request.

option accept-invalid-http-request in the frontend will “fix/workaround” the problem.

Thanks for the help!

olifur · October 22, 2020, 11:12am

Hi Rage2dev,

Awesome, using your option "accept-invalid-http-request " the request now works. It still is a workaround but for our case it is all we need here.

Thx again,
Oli

Topic		Replies	Views
Haproxy returns 403 NOSRV error intermittently Help!	6	2164	September 25, 2023
Getting 503 errors with <NOSRV> in backend and Layer6 timeout error log Help!	1	349	April 19, 2024
What can be cause for /NOSRV Help!	2	959	November 29, 2023
Need help with better understanding CR---- termination state in http mode Help!	0	908	November 24, 2020
Haproxy load testing on h2 responce with NOSRV after 6000 hit/s Help!	1	474	November 6, 2019

Receiving NOSRV BADREQ when discrepancy between CONNECT and Host header

Related topics