Following a tutorial but gone wrong (haproxy.service: Start request repeated too quickly.)

ffuentes · February 2, 2021, 6:41pm

I was following this tutorial (I use Ubuntu 20.04 minimal) to run a DNS over HTTPS which is very close to my use case: A experimental server with just only so many applications inside and nothing production worth. The thing is I need to have both the dnsdist service and nginx using port 443.

frontend https
   bind 12.34.56.78:443
   mode tcp
   tcp-request inspect-delay 5s
   tcp-request content accept if { req_ssl_hello_type 1 }

   use_backend dnsdist if { req_ssl_sni -i doh.example.com }
   use_backend nginx if { req_ssl_sni -i www.example.com }
   use_backend nginx if { req_ssl_sni -i example.com }

   default_backend dnsdist

backend dnsdist
   mode tcp
   option ssl-hello-chk
   server dnsdist 127.0.0.1:443

backend nginx
   mode tcp
   option ssl-hello-chk
   server nginx 127.0.0.2:443 check

What the tutorial doesn’t show is the content above this block (because it comes preinstalled I guess). I have this right above this block:

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    tcp
        option  tcplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

I added both of them in the same position and IP but I couldn’t manage them to work together. Testing the configuration it’s fine but running the process on systemctl it throws this message:

Feb 02 18:31:25 instance-20210130-1326 systemd[1]: haproxy.service: Start request repeated too quickly.
Feb 02 18:31:25 instance-20210130-1326 systemd[1]: haproxy.service: Failed with result 'exit-code'.
Feb 02 18:31:25 instance-20210130-1326 systemd[1]: Failed to start HAProxy Load Balancer.

So I don’t know if something can be done. (Both subdomains are created under let’s encrypt but none of them is the main domain which is in another server)

TimScotland · February 3, 2021, 9:54am

To my mind (right or wrong) there are three elements that I think are wrong.

there is no certificate for the binding of https
you should be useing mode http (as the dns over https)
the use backend statements should be done differently

frontend https
bind *:443 ssl crt /etc/ssl/certs/server.pem alpn h2,http/1.1
mode http
option httpclose

    ### Sort by ACL ### 
    acl is_dns hdr_dom(host) -i doh.example.com
    acl is_nginx hdr_dom(host) -i www.iqxanywhere.net

    ### Select what backend to use
    use_backend dns if is_dns
    use_backend nginx if is_nginx

    default_backend IQXAnywhere-bk

HTH

ffuentes · February 5, 2021, 12:02am

I amended the mistakes but it still can’t bind the socket

[ALERT] 034/235748 (36381) : Starting frontend https: cannot bind socket [12.34.56.78:443]

(made up ip)

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    maxconn 2048
    tune.ssl.default-dh-param 2048
    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    tcp
        option  tcplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
                                                                     
frontend https
   bind 12.34.56.78:443 ssl crt /etc/haproxy/certs/doh.example.com.pem alpn h2,http/1.1
   mode http
# tcp-request inspect-delay 10s
# tcp-request content accept if { req_ssl_hello_type 1 }
   option httpclose

   acl is_dns hdr_dom(host) -i doh.example.com
   acl is_nginx hdr_dom(host) -i cl.example.com

#   use_backend dnsdist if { req_ssl_sni -i doh.example.com }
#   use_backend nginx if { req_ssl_sni -i cl.example.com }

   ### Select what backend to use
    use_backend dnsdist if is_dns
    use_backend nginx if is_nginx
   default_backend dnsdist

backend dnsdist
   mode http
   option ssl-hello-chk
   server dnsdist 127.0.0.1:443

backend nginx
   mode http
   option ssl-hello-chk
   server nginx 127.0.0.2:443 check

TimScotland · February 5, 2021, 11:07am

I assume that you have checked that “ss -atl” shows nothing listening or 443 (or netstat if no ss)?

When you start haproxy what is the error shown when you “journalctl -xe”

T

ffuentes · February 5, 2021, 2:14pm

Hi, you were right in that I had to check that. However, I still have
problems now that I can make it run.

bind *:443 doesn’t let my nginx or dnsdist run, neither does bind
0.0.0.0:443 but only the private ip of the network.

On the other hand, a huge difficulty is that whenever I try to open the
website it complains about getting the wrong cert, even though it’s
appropiately configurated in nginx because it’s reading the haproxy one
and not being able to validate it somehow.


frontend https
    bind 10.0.0.3:443 ssl crt /etc/haproxy/certs/cl.example.com.pem alpn 
h2,http/1.1
#   bind 129.151.98.55:443 ssl crt 
/etc/haproxy/certs/doh.example.com.pem alpn h2,http/1.1
    mode http
# tcp-request inspect-delay 10s
# tcp-request content accept if { req_ssl_hello_type 1 }
    option httpclose

    acl is_dns hdr_dom(host) -i doh.example.com
    acl is_nginx hdr_dom(host) -i cl.example.com

    use_backend dnsdist if { req_ssl_sni -i doh.example.com }
    use_backend nginx if { req_ssl_sni -i cl.example.com }

    ### Select what backend to use
    # use_backend dnsdist if is_dns
    # use_backend nginx if is_nginx
    default_backend nginx

backend dnsdist
    mode http
    option ssl-hello-chk
    server dnsdist 127.0.0.1:443

backend nginx
    mode http
    option ssl-hello-chk
    option forwardfor
    reqadd x-forwarded-proto:\ https
    server nginx 127.0.0.2:443 check

TimScotland · February 5, 2021, 2:17pm

Try disabling the SSL verification - it will allow a TLS connection no matter what the certificate used

server nginx 127.0.0.2:443 check ssl verify none

Tim

ffuentes · February 5, 2021, 2:34pm

I added that check ssl verify none but now browsers are checking the
opposite certificate!

ffuentes · February 5, 2021, 2:47pm

Also, I get from the nginx server:

The plain HTTP request was sent to HTTPS port

TimScotland · February 5, 2021, 2:55pm

We are fighting a number of issues simultaneously - I always prefer to deal with one at a time

Lets start by getting the nginx to respond on port 80 to start with. Then you know that the only certificate has to be that which you have on haproxy. When that is working we can start working back up the queue. Having said which, as you are using 127.0.02 and 3 there is nothing on the wire so using HAproxy for the SSL offload and port 80 for the backend isn’t an issue.

Get that working first.

Tim

ffuentes · February 5, 2021, 3:24pm

I didn’t understand very well but I changed the 443 in the backend for
80 because the nginx server was set up as usual with a port 80 and a 443
with a 301 redirection from the first to the second.

backend dnsdist
    mode http
    option ssl-hello-chk
    server dnsdist 127.0.0.1:443

backend nginx
    mode http
    option ssl-hello-chk
    option forwardfor
    reqadd x-forwarded-proto:\ https
    server nginx 127.0.0.1:80 check

It complains that it’s getting the wrong certificate. I’m lost now.

Topic		Replies	Views
Haproxy with nginx, ocserver and letsencrypt ssl Help!	3	1439	June 27, 2022
[SOLVED] One IP, fistful of domains, pack of subdomains and HAProxy in front of it Help!	12	2884	November 4, 2018
Need help with HAPROXY https when apps share the SSL cert Help!	2	274	August 25, 2023
TCP mode and http/2 backend Help!	6	3588	June 4, 2020
Use tcp backend based on map and ssl_c_s_dn Help!	3	1000	October 30, 2020

Following a tutorial but gone wrong (haproxy.service: Start request repeated too quickly.)

Related topics