<NOSRV> issue with TLS termination

Help!
1

Hi everybody,

I got an issue with HAProxy with SSL/TLS termination.

When I do a HTTPS request, I got the following error :
Oct 16 10:39:28 localhost haproxy[724]: xxx.xxx.xxx.xxx:62597 [16/Oct/2020:16 10:39:28.658] w-https~ w-backend/<NOSRV> -1/-1/-1/-1/0 400 188 - - PR-- 2/2/0/0/3 0/0 "GET / HTTP/1.1"

HAProxy blocks the request (PR), but I don’t why .

Can you help me ?

This is my haproxy.cfg :

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        user haproxy
        group haproxy
        daemon
        tune.ssl.default-dh-param 4096

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option forwardfor
        option http_proxy
        timeout connect 5000
        timeout client  50000
        timeout server  5000

frontend w-http
        bind *:80
        reqadd X-Forwarded-Proto:\ http
        default_backend w-backend
        # le request
        acl le-acl path_beg /.well-known/acme-challenge/
        use_backend le-backend if letsencrypt-acl

frontend w-https
        bind *:8000-9000 ssl crt /etc/haproxy/certs/www.example.com.pem
        bind *:443 ssl crt /etc/haproxy/certs/www.example.com.pem
        reqadd X-Forwarded-Proto:\ https
        default_backend w-backend

backend w-backend
        mode http
        http-request set-header X-Forwarded-For %[src]
        reqadd X-Forwarded-Proto:\ https
        option http-server-close
        balance roundrobin
        redirect scheme https if !{ ssl_fc }
        server w1 aaa.aaa.aaa.aaa check port 80
        server w2 bbb.bbb.bbb.bbb check port 80

backend le-backend
        server le 127.0.0.1:54321
2

As per:

The PR error code means

The proxy blocked the client's HTTP request, either because of an
invalid HTTP syntax, in which case it returned an HTTP 400 error to
the client, or because a deny filter matched, in which case it
returned an HTTP 403 error.

Since you see 400, it means the HTTP request is invalid.

If you are sure about the client and want to investigate further on haproxy, use the show errors command on the admin socket.

3

When I run : sudo echo "show errors" | sudo socat unix-connect:/run/haproxy/admin.sock stdio , I got this result : Total events captured on [19/Oct/2020:10:51:34.759] : 0

Could you help me ?

Also, I ran : haproxy -vv to give more informations about my haproxy

HA-Proxy version 1.8.19-1+deb10u3 2020/08/01
Copyright 2000-2019 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-OkbS59/haproxy-1.8.19=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-format-truncation -Wno-null-dereference -Wno-unused-label
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_NS=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE2 version : 10.32 2018-09-10
PCRE2 library supports JIT : yes
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with network namespace support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

4

Can you provide the output of a curl -vv https://destination call? Is this happening with 100% of the HTTPS requests?

Are you sure the modified configuration provided corresponds with what you are running here?

Are you accessing port 443 or one of those high ports 8000-9000 ?

5

Hi,

Yes, I can do it , this is the result : (curl -vv https://www.example.com )

* Expire in 0 ms for 6 (transfer 0x7fffbb5cff50)
* Expire in 1 ms for 1 (transfer 0x7fffbb5cff50)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Expire in 1 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 2 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 1 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 1 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 4 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 2 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 2 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 4 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 3 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 3 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 4 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 3 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 3 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 4 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 4 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 4 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 4 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 4 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 4 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 8 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 6 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 6 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 8 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 6 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 6 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 8 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 8 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 8 ms for 1 (transfer 0x7fffbb5cff50)
* Expire in 9 ms for 1 (transfer 0x7fffbb5cff50)
*   Trying 78.222.220.190...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7fffbb5cff50)
* Connected to www.example.com (78.222.220.190) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2816 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [520 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=www.example.com
*  start date: Oct 16 13:36:07 2020 GMT
*  expire date: Jan 14 13:36:07 2021 GMT
*  subjectAltName: host "www.example.com" matched cert's "www.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x7fffbb5cff50)
} [5 bytes data]
> GET / HTTP/2
> Host: www.example.com
> User-Agent: curl/7.64.0
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 400
< cache-control: no-cache
< content-type: text/html
<
{ [90 bytes data]
100    90    0    90    0     0    967      0 --:--:-- --:--:-- --:--:--   978
* Connection #0 to host www.example.com left intact

Is this happening with 100% of the HTTPS requests?
Are you accessing port 443 or one of those high ports 8000-9000 ?
Are you sure the modified configuration provided corresponds with what you are running here?

I kept looking, and yes, I changed the configuration, but the result is the same.

All my https requests on port 443 or in the range 8000-9000 give me the same result : 400 Bad request.

This my haproxy.cgf at this point :

global
        log /dev/log    local0
        log /dev/log    local1 debug

        chroot /var/lib/haproxy
        #stats socket /var/run/haproxy.sock mode 660 level admin expose-fd listeners
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 5s
        user haproxy
        group haproxy
        daemon

        tune.ssl.default-dh-param 2048

        # generated 2020-10-19, Mozilla Guideline v5.6, HAProxy 1.8.19, OpenSSL 1.1.1d, modern configuration, no HSTS
        # https://ssl-config.mozilla.org/#server=haproxy&version=1.8.19&config=modern&openssl=1.1.1d&hsts=false&guideline=5.6
        # modern configuration
        ssl-default-bind-options prefer-client-ciphers no-sslv3 no-tlsv10 no-tlsv11 no-tlsv12 no-tls-tickets

        ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tlsv12 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option forwardfor
        option http_proxy
        option accept-invalid-http-request
        timeout connect 5000
        timeout client  50000
        timeout server  5000
        stats enable
        stats hide-version
        stats refresh 5s
        stats uri /hastats



frontend w-http
        bind *:80
        reqadd X-Forwarded-Proto:\ http
        default_backend w-backend
        option accept-invalid-http-request
        # Test URI to see if its a le request
        acl le-acl path_beg /.well-known/acme-challenge/
        use_backend le-backend if le-acl


frontend w-https
        mode http
        option forwardfor
        option http_proxy
        option accept-invalid-http-request
 
        bind *:8000-9000 ssl crt /etc/haproxy/certs/radio4.lips.ovh.pem
        bind *:443 ssl crt /etc/haproxy/certs/radio4.lips.ovh.pem alpn h2,http/1.1
        reqadd X-Forwarded-Proto:\ https
        http-request set-header X-Real-IP %[src]
        #http-request add-header X-Forwarded-Proto https
        #http-request add-header X-Forwarded-Port 443
        # set HTTP Strict Transport Security (HTST) header
        http-response add-header Strict-Transport-Security max-age=15768000

        default_backend w-backend


backend w-backend
        mode http
        http-request set-header X-Forwarded-For %[src]
        reqadd X-Forwarded-Proto:\ https
        option http-server-close
        option accept-invalid-http-request
        balance roundrobin
        redirect scheme https if !{ ssl_fc }
        server web1 192.168.56.101:80 check
        server web2 192.168.56.102:80 check

backend le-backend
        server le 127.0.0.1:8080
6

option http_proxy is wrong, you need to remove it from the entire configuration.

I also suggest you remove option http-server-close everywhere, it makes no real sense.

7

Yes, you’re right.
I cleaned up my haproxy.cfg, but the result it’s the same.

This is my haproxy.cfg :

global

        log /dev/log    local0

        log /dev/log    local1 debug

        chroot /var/lib/haproxy

        #stats socket /var/run/haproxy.sock mode 660 level admin expose-fd listeners

        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners

        stats timeout 5s

        user haproxy

        group haproxy

        daemon

        tune.ssl.default-dh-param 2048

        # generated 2020-10-19, Mozilla Guideline v5.6, HAProxy 1.8.19, OpenSSL 1.1.1d, modern configuration, no HSTS

        # https://ssl-config.mozilla.org/#server=haproxy&version=1.8.19&config=modern&openssl=1.1.1d&hsts=false&guideline=5.6

        # modern configuration

        ssl-default-bind-options prefer-client-ciphers no-sslv3 no-tlsv10 no-tlsv11 no-tlsv12 no-tls-tickets

        ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tlsv12 no-tls-tickets

defaults

        log     global

        mode    http

        option  httplog

        option  dontlognull

        option forwardfor

        option http_proxy

        timeout connect 5000

        timeout client  50000

        timeout server  5000

        stats enable

        stats hide-version

        stats refresh 5s

        stats uri /hastats

frontend w-http

        bind *:80

        reqadd X-Forwarded-Proto:\ http

        default_backend w-backend

        # Test URI to see if its a le request

        acl le-acl path_beg /.well-known/acme-challenge/

        use_backend le-backend if le-acl

frontend w-https

        mode http

        option forwardfor

        option http_proxy

        bind *:8000-9000 ssl crt /etc/haproxy/certs/www.example.com.pem

        bind *:443 ssl crt /etc/haproxy/certs/www.example.com.pem alpn h2,http/1.1

        reqadd X-Forwarded-Proto:\ https

        http-request set-header X-Real-IP %[src]

        # set HTTP Strict Transport Security (HTST) header

        http-response add-header Strict-Transport-Security max-age=15768000

        default_backend w-backend

backend w-backend

        mode http

        http-request set-header X-Forwarded-For %[src]

        reqadd X-Forwarded-Proto:\ https

        balance roundrobin

        redirect scheme https if !{ ssl_fc }

        server web1 192.168.56.101:80 check

        server web2 192.168.56.102:80 check

backend le-backend

        server le 127.0.0.1:8080

And in the logs /var/logs/haproxy.log , I got this :

Oct 21 18:57:18 cmash haproxy[487]: 192.168.1.254:64312 [21/Oct/2020:18:57:18.335] w-https~ w-backend/ -1/-1/-1/-1/0 400 187 - - PR-- 1/1/0/0/3 0/0 “GET /favicon.ico HTTP/1.1”

To set up my haproxy , I followed this tutorial :
https://www.digitalocean.com/community/tutorials/how-to-secure-haproxy-with-let-s-encrypt-on-ubuntu-14-04

8

Hi,

I just found the mistake. The trouble was from the option proxy in the default’s section.

Thank you @lukastribus for your time and your help.

1 Like