Path_beg not matching - what am I doing wrong?

Help!
1

I’m kind of at wit’s end attempting to figure out what I’m doing wrong, but damned if I can figure it out.

First, my config file:

global
    log /dev/log	local0
    log /dev/log	local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    ca-base /etc/ssl/certs
    crt-base /etc/haproxy/ssl

    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
    ssl-dh-param-file /etc/haproxy/ssl/dhparam.pem

defaults
    log	global
    mode	tcp
    option	tcpka
    option	dontlognull
    option http-server-close
        timeout connect 5000
        timeout client  50000
        timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

frontend redirect
    mode http
    bind :::80 v4v6
    acl letsencryptrequest path_beg -i /.well-known/acme-challenge/
    acl http ssl_fc,not
    http-request redirect scheme https if http
    use_backend letsencrypt if letsencryptrequest
    default_backend dummy

frontend liquidfront
    mode tcp
    bind :::443 v4v6 ssl crt (redacted long list of certificates and hosts) ecdhe secp384r1 alpn h2,http/1.1
    default_backend tovarnish

backend tovarnish
    mode tcp
    server local 127.0.0.1:6081 send-proxy-v2

backend letsencrypt
    mode http
    server letsencrypt 127.0.0.1:54321

backend dummy
    mode http
    server local 127.0.0.1:9999

Stack:
HAProxy 1.7.5 <-> Varnish 5.1 <-> Nginx 1.13.0

The reason there are multiple front-ends is for HTTP/2 support. The way I understood the documentation between HAPropxy and Varnish, I need to use TCP mode on my Varnish backend stanza in order to use the proxy protocol; Varnish then can extract the protocol and route to the appropriate nginx listening port.

I have to separate the frontends until HAProxy support http/2 in HTTP mode—at least, that’s how I understand it.

Problem behavior:
I use LetsEncrypt, and I use HAProxy to watch for incoming ACME challenges by using the same method that’s all over the web: matching to an ACL based on path_beg -i /.well-known/acme-challenge/, and then kicking all matching traffic to a LetsEncrypt backend that points to 127.0.0.1:54321, where the local certbot is listening.

However, the ACL doesn’t appear to be matching anymore. Instead of the requests going to the LE backend, they go to the tovarnish backend, which is where everything except LE traffic should be going. I can watch the varnishncsa logs and see the LE requests hitting Varnish:

66.133.109.36 - - [10/Jun/2017:20:09:17 -0400] GET http://myhost.redacted.com/.well-known/acme-challenge/7euurczH9HL4Q6_DV_Yn1eG7LSFpqYsvXQm8vvcv3UY HTTP/1.1 403 162 http://myhost.redacted.com/.well-known/acme-challenge/7euurczH9HL4Q6_DV_Yn1eG7LSFpqYsvXQm8vvcv3UY Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)

66.133.109.36 - - [10/Jun/2017:20:12:18 -0400] GET http://myhost.redacted.com/.well-known/acme-challenge/MJI1xHqUO8VpgNKLzhfJddYw1iz2IVuZUAXAGBiWLBs HTTP/1.1 403 162 http://myhost.redacted.com/.well-known/acme-challenge/MJI1xHqUO8VpgNKLzhfJddYw1iz2IVuZUAXAGBiWLBs Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)

66.133.109.36 - - [10/Jun/2017:20:12:43 -0400] GET http://myhost.redacted.com/.well-known/acme-challenge/vpatZfBKKygczculNMVksQEeko1T8wuYaDdJk9riEL4 HTTP/1.1 403 162 http://myhost.redacted.com/.well-known/acme-challenge/vpatZfBKKygczculNMVksQEeko1T8wuYaDdJk9riEL4 Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)

66.133.109.36 - - [10/Jun/2017:20:15:21 -0400] GET http://myhost.redacted.com/.well-known/acme-challenge/VJI1Y-OyKoe9KmJOOlZ3_RxvLfDDlHv7Xzqz0Xy7hO4 HTTP/1.1 403 162 http://myhost.redacted.com/.well-known/acme-challenge/VJI1Y-OyKoe9KmJOOlZ3_RxvLfDDlHv7Xzqz0Xy7hO4 Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)

66.133.109.36 - - [10/Jun/2017:20:16:38 -0400] GET http://myhost.redacted.com/.well-known/acme-challenge/HT4lhg0ixP981CTTPP95A6wPD2ddbj0jAEPMuoksmCQ HTTP/1.1 403 162 http://myhost.redacted.com/.well-known/acme-challenge/HT4lhg0ixP981CTTPP95A6wPD2ddbj0jAEPMuoksmCQ Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)

66.133.109.36 - - [10/Jun/2017:20:17:58 -0400] GET http://myhost.redacted.com/.well-known/acme-challenge/l5UAdVGTwLJalvi7U9uSi4B-riJP4bEg33mhEt34Zhk HTTP/1.1 403 162 http://myhost.redacted.com/.well-known/acme-challenge/l5UAdVGTwLJalvi7U9uSi4B-riJP4bEg33mhEt34Zhk Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)

66.133.109.36 - - [10/Jun/2017:20:21:04 -0400] GET http://myhost.redacted.com/.well-known/acme-challenge/yxB_gxKWooknSgjCCkTJNk0ac1SMHqGUQiu3lSQY5bY HTTP/1.1 403 162 http://myhost.redacted.com/.well-known/acme-challenge/yxB_gxKWooknSgjCCkTJNk0ac1SMHqGUQiu3lSQY5bY Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)

(hostname redacted, obviously)

I don’t know exactly when the problem behavior started, and I’ve been going over HAProxy’s docs for a few hours now without figuring out my problem. I’ve tried using hdr_beg(host) and matching hostname, so that the ACL looks like this instead:

acl letsencryptrequest hdr_beg(host) -i myhost.redacted.com

…and it still doesn’t match, even when definitely using the correct hostname.

I’m completely baffled. What on earth am I doing wrong? Any help would be greatly appreciated.

2

This problem eventually just went away on its own, but has now resurfaced. I still have absolutely no idea what’s wrong or why, but path_beg is simply flat-out not matching.

3

I think your ACL matches just fine, you are just using it wrong.

I assume haproxy redirects to https before even looking at the letsencryptrequest ACL.

Also the http ACL is strange, unnecessary (you have a port 80 only frontend, you don’t have to check if the request was SSL as by definition there never is on a port 80 non-SSL frontend) and invalid (I believe http is a reserved keyword / internal ACL).

Just simplify, and most importantly use the letsencrypt backend BEFORE redirecting to HTTPS.

frontend redirect
    mode http
    bind :::80 v4v6
    acl letsencryptrequest path_beg -i /.well-known/acme-challenge/
    use_backend letsencrypt if letsencryptrequest
    http-request redirect scheme https
    default_backend dummy

If it still fails, negative match letsencryptrequest in the https redirect so it will never redirect to https for ACME:

frontend redirect
    mode http
    bind :::80 v4v6
    acl letsencryptrequest path_beg -i /.well-known/acme-challenge/
    use_backend letsencrypt if letsencryptrequest
    http-request redirect scheme https if !letsencryptrequest
    default_backend dummy
4

Sigh.

The problem is something with cloudflare. For the site in question, I’m using cloudflare to help absorb some significant load, and surprise surprise, disabling cloudflare’s dns redirection for the site causes everything to work perfectly.

So, guess I need to take this support thread over to the cloudflare and letsencrypt forums to figure out wtf page rules I need to put in place (did some quick experimenting with --dry-run and couldn’t make it work).

Thanks for the assist, @lukastribus - sorry for wasting time!

5

If you enabled HTTPS redirect in Cloudflare you definitely need to that turn it off (otherwise it cannot reach your backend via HTTP).
Also do make sure you use HTTP verification verification (not SNI verification).

Also see:

6

Yep, definitely using http-01 and not SNI. I don’t have HTTPS redirection enabled via Cloudflare since I do that at HAProxy anyway (primarily because I’m hosting multiple non-cloudflare sites on the box with the same stack). I did indeed read that article, which suggests using webroot instead of standalone w/http-01; tried their recommended config and was unable to get it to work, but I didn’t have time to start troubleshooting.

I’ll have some time after work today to sit down and grind on it a bit to see if I can tweak it into working. I appreciate your help :slight_smile: I don’t want to drag this posting further off-topic for HAProxy, though.

7

There is one last advice I can give you here which is to do this via the DNS verification method instead.

There are a number of letsencrypt clients that can use the DNS verification method and support the Cloudflare API at the same time. Then all you would have to do is to put your Cloudflare API key into your letsencrypt client, and the verification would be done via DNS instead of HTTP.

You could remove all those configurations from your HTTP stack as it would no longer be used for verification and you could even request certificates for subdomains not pointing to cloudflare or your HTTP servers.

8

Yep, that’s my goal. I wanted to do this six months ago, but there just wasn’t that much documentation out there and IIRC certbot didn’t support the necessary options back then. Fortunately, now that some time has passed, there are a number of reference configs I can crib from.

The one hiccup was the fact that the DNS validation TXT record changes at every renewal, but as you note, I believe I can automate that as well via a script and cloudflare’s API. It’s just a question of how much time I want to spend making it work. But, yep, you’re exactly right. And DNS validation would finally let me start using a LE certificate for my mail server, as well (which is a giant pain in the ass to get working with http validation).

9

Take a look at acme.sh:
https://github.com/Neilpang/acme.sh

Its pure shell based (no python, perl, not even bash is needed), and it supports the Cloudflare API directly for DNS verification. This should be very easy to setup.

1 Like