Cert renews, https+http redirects and send-proxy on ssl

I have a simple requirement & semi-working setup which made me think I knew what I was doing, but this is clearly not true.
I am adequately savvy i.t.o CLI/linux, but in HAproxy terms, I am (as I discovered), at best, a complete moron - apologies.

  1. OpenWRT( port forward of 80/443 → HAproxy( which must decide where to send things
  2. HAproxy( v2.6.15 on a debian VM
  3. ~4 websites which are on debian VM’s
    1. (nextcloud.domain.co.za:443) configured for SSL in nginx
    2. (jellyfin.domain.co.za) configured in nginx as a reverse proxy to mangle 80 ->jellyfin:8096
    3. (domain.co.za) configured as a http:80 service
    4. (nginx.domain.co.za:443) configured for SSL in nginx (shared SNI nginx vhost with above machine)
  • nextcloud: config has a few extra config bits that I found on the NC setup docs - but it works OK. However, if I attempt to use the send-proxy, the whole thing fails - which I’d like to address if possible.
  • jellyfin: also works OK - with the send-proxy which I assume is because it’s :80, but I’d like to get that to work “properly” via HAproxy.
  • nginx:443 also works OK.
  • The http://domain.co.za:80 link fails - if I try to force it to https, it bypasses this and goes to https://nginx.domain.co.za:443
    • I assume it has to do with the scheme redirect but despite how dismally simple the config seems, I cannot get it to stay on http for that link

I currently use DNS01 wildcard LE certs which I manually update every ~3 months (and often forget to do, hence the desire to automate).

What I’d like to achieve is to

  • get any http://domain:80 to work (with and/or without redirect) so that I can experiment with auto-certs
  • apply the SSL certs via HAproxy instead of nginx and let HAproxy renew them
    • I tried the acme.sh but since I can’t get the basic :80 working, I didn’t get very far
  • get send-proxy to work on SSL items too
  • figure out how/where to put multiple certs (possibly wildcard certs too?) so that SNI works automatically for multiple sites and still does renews

Thanks and sorry in advance

HAproxy config looks like this…

  log       /dev/log  local0
  log       /dev/log  local1 notice
  chroot    /var/lib/haproxy
  stats     socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
  stats     timeout 30s

  user      haproxy
  group     haproxy
  # Default SSL material locations
  ca-base   /etc/ssl/certs
  crt-base  /etc/ssl/private

  ssl-server-verify none
    # this is for letsencrypt/acme requests

  log       global
  option    httplog
  option    dontlognull
  # option    forwardfor       except
  option    redispatch
  option    http-server-close
  retries   3
  timeout   http-request    10s
  timeout   queue           1m
  timeout   connect         10s
  timeout   client          1m
  timeout   server          1m
  timeout   http-keep-alive 10s
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

frontend stats
  bind                *:9000
  mode                http
  stats               enable
  stats               uri /stats
  stats               refresh 30s
  stats               auth admin:password
  stats               hide-version
  stats               realm HAproxy\ Statistics

frontend http_in
  bind                *:80 alpn h2,h2c,http/1.1
  mode                http
  option              forwardfor
  acl                 test_acme       path_beg /.well-known/acme-challenge/
  acl	                domain_acl      hdr_end(host) -i domain.co.za
  # redirect scheme https code 301      if !test_acme
  use_backend         letsencrypt_BE  if test_acme
  use_backend         domain_http     if domain_acl !test_acme
  default_backend     default

frontend https_in
  bind                *:443 transparent
  mode                tcp
  option              tcplog
  tcp-request         inspect-delay 5s
  tcp-request         content accept  if { req_ssl_hello_type 1 }
  use_backend         nextcloud       if { req_ssl_sni -i nextcloud.domain.co.za }
  use_backend         jellyfin        if { req_ssl_sni -i jellyfin.domain.co.za }
  default_backend     default

backend letsencrypt_BE
  mode        http
  server      letsencrypt

backend domain_http
  mode        http
  server      domain         domain.co.za:80 check send-proxy-v2

backend jellyfin
  mode        tcp
  option      tcp-check
  option      ssl-hello-chk
  option      httpchk GET /
  server      jellyfin      jellyfin.domain.co.za:80 verify none send-proxy-v2

backend nextcloud
  mode        tcp
  option      tcp-check
  option      ssl-hello-chk
  option      httpchk GET /
  http-check  send hdr  Host nextcloud.domain.co.za
  server      nextcloud     nextcloud.domain.co.za:443 check-ssl verify none check-sni nextcloud.domain.co.za sni str(nextcloud.domain.co.za) # send-proxy-v2

backend default
  mode        tcp
  option      tcp-check
  option      ssl-hello-chk
  option      httpchk GET /
  http-check  send hdr Host nginx.domain.co.za
  server      nginx         nginx.domain.co.za

and the failing nginx config looks like this…

server {
  listen      80;
  server_name domain.co.za www.domain.co.za;
  root        /var/www/html/domain.co.za;
  index       index.html index.htm index.php;
  # return 301 https://$server_name$request_uri;  # Enforce HTTPS

  access_log /var/log/nginx/domain.co.za-access.log;
  error_log  /var/log/nginx/domain.co.za-error.log error;

  location / {
    try_files $uri $uri/ =404;
    # try_files $uri $uri/ /index.php$is_args$args;

  real_ip_header proxy_protocol;

the nginx logs specific to this vhost never get reached, so I guess the error is in my HAproxy config.
If I enable the redirect scheme then the requests go to nginx.domain.co.za:443
If I don’t then attempts to reach the domain.co.za:80 give a browser error (but nothing in the nginx vhost logs)

400 Bad Request


Personally, I think you were on the right track with acme.sh. I use DNS validation using the DNS API’s tho. It’s got a sharp learning curve, but auto-renewal is much easier in my opinion as it no longer depends on a web server.

Anyway, on with the goals!

Your frontend is mode http, but your default backend is mode tcp. Those two will not play well together.

Is domain.co.za pointing to the HAProxy instance? If so, you’re asking this backend to open a connection to itself on the same frontend. This might cause HAProxy to become unstable or even crash on the first connection attempt if it had any success. Your saving grace might be that the backend is set to send-proxy-v2 but the frontend is not set to accept it. What is this backend intended to do?

I know HAProxy can renew certificates, but I had acme.sh in place before that was a feature, so I can’t speak to that part. Applying the SSL certificates means that your listener on 443 needs to be in mode http. After that, your bind line can include a file with the key, cert, and chain all combined. It can also just be a directory where all of those things exist (I’ve also not done this one, but the docs say it can be done).

This is one I would recommend staying away from unless you know your application explicitly supports it. The docs actually recommend the same.:

This setting must not be used if the server isn't aware of the protocol.

This can also be done as part of your bind. If you bind a directory and multiple certs are in the directory, I think HAProxy will load as many of them as it can. You can also be explicit with something like:

	mode http
	bind *:443 ssl strict-sni crt /path/to/cert1.pem crt /path/to/cert2.pem

Just a quick note, alpn is a TLS extension and doesn’t do anything if SSL isn’t present and in mode http.

Hi storm. Thanks for the reply.
You actually replied to my first (similar question some time back and answered most of that). Sadly (being a dumbass) I did not do backups and had to start again. It was a while back so i couldn’t make head or tail of what I had done, so I resorted to groveling for help again :blush:.

I tried both on mode http, but then jellyfin/nextcloud fail dismally and unpredictably (though I suspect lack of knowledge is the problem)
domain.co.za point to the content server LXC/VM. I had it on :80 but since I can control the DNS via the pihole and/or HAproxy /etc/hosts, I thought it may be more scalable this way.

If i use send-proxy-v2 in any https backends, the it causes a fail, but it seems OK on http. How then, does one get the SRC-IP at the httpd point?

I will remove the 80 alpn if it serves no purpose (I just put it in there for consistency without knowing why or what it did).

I will keep playing/experimenting, but thanks again for replying

I did create a stripped down version with almost nothing in it, and the /path/to/certs containing a lot of them did appear to work correctly, but I gave up in frustration and stupidity.

Hey! I thought I recognized that face, but I’m horrible with names. :slight_smile:

Don’t be so hard on yourself. Mistakes happen, and there’s nothing wrong with asking for help. We all came here for help.

Typically in mode http, HAProxy will offload all SSL and connect to the backend server in plain text. If this is not desirable, you can add SSL back to the backend connection by adding ssl to your server lines. You have ssl-server-verify none in your global section, so HAProxy will not care if the certs are valid or not.

Typically, you find the real source IP using the header X-Forwarded-For, which most apps support. You have it in your defaults section but commented out. I would stick with that.

Don’t give up! Keep trying, and keep asking questions! To help keep you moving, here’s an excerpt from my very own HAProxy. I specifically included Jellyfin and Nextcloud since I, too, run both of those. “FTBRanks” is a static HTML page that I don’t want to lose (modded Minecraft documentation), and the “homepage” is an app called Grav, which is just a databaseless/flat-file blog CMS. Help yourself to any or all of it, and feel free to keep asking questions! Should disaster ever strike, this thread can be your documentation to get started again. There’s a good chance others might find this useful too. :grin:

# All incoming requests
frontend entrypoint
# Normal HTTP (no SSL/TLS)
	bind ipv4@*:80 name "Non-Secure Port 80"
# Normal HTTPS (http/2)
	bind ipv4@*:443 name "SSL on HTTP/2" ssl strict-sni crt haproxy.pem
	option socket-stats
	http-request del-header x-forwarded-for
	option forwardfor
	option http-buffer-request
	option http-ignore-probes
# Grabbing source IP from Cloudflare.
# It's stored in two extra headers: cf-connecting-ip and cf-ipcountry
	capture request header Host len 100
	capture request header cf-connecting-ip len 100
	capture request header cf-ipcountry len 2
# Redirect http without SSL to use SSL
	http-request redirect scheme https if !{ ssl_fc }
# Some Security Headers
	http-request set-header X-Forwarded-Proto https if { ssl_fc }
	http-response set-header X-Frame-Options SAMEORIGIN
	http-response set-header X-XSS-Protection "1;mode=block"
	http-response set-header X-Content-Type-Options nosniff
	http-response set-header Referrer-Policy strict-origin-when-cross-origin
# Security Rejects
	acl reject path_end -i /xmlrpc.php
	acl reject path -i .env
	acl reject path -i .user.ini
	acl reject path_end -i wlwmanifest.xml
	acl reject path -i /status/internal-only
# Hostname checks
	acl host_jellyfin hdr(host) -i jellyfin.example.net
	acl host_nextcloud hdr(host) -i cloud.example.net
	acl host_homepage hdr(host) -i example.net
	acl host_ftbranks hdr(host) -i ftbranks.example.net
# Enable HTTP caching of any cacheable content
	http-request cache-use cache
	http-response cache-store cache
# Return 404 if not on the internal network.
	use_backend no_route if reject
# Routing requests based on rules above:
	# Public Sites proxied by Cloudflare
	use_backend ftbranks if host_ftbranks
	use_backend homepage if host_homepage
	use_backend nextcloud if host_nextcloud
	use_backend jellyfin if host_jellyfin

	# Enable HTTP compression of text contents
	compression algo deflate gzip
	compression type text/ application/javascript application/xhtml+xml image/x-icon

	# Default to 404!
	default_backend no_route

cache cache
	total-max-size 256			# RAM cache size in megabytes
	max-object-size 10485760	# max cacheable object size in bytes
	max-age 3600				# max cache duration in seconds
	process-vary on				# handle the Vary header (otherwise don't cache)

backend homepage
	http-response set-header Content-Security-Policy "default-src 'none'; base-uri 'none'; manifest-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob: https://www.gravatar.com/avatar/; font-src 'self' data:; connect-src 'self'; media-src 'self'; frame-src 'self'; frame-ancestors 'self'; worker-src 'self' blob:; form-action 'self';"
	http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
	http-response add-header alt-svc "h3=\":443\"; ma=86400"
	option httpchk HEAD /
	timeout check 5s
	timeout connect 10s
	timeout server 1m
	server docker2 check maxconn 200

backend jellyfin
	http-request set-header X-Forwarded-Port %[dst_port]
	http-response set-header Content-Security-Policy "base-uri 'none'; connect-src 'self'; default-src 'none'; font-src 'self'; form-action 'self'; frame-ancestors 'self'; frame-src 'self'; img-src 'self' image.tmdb.org m.media-amazon.com; media-src 'self' data:; object-src 'none'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'"
	http-response add-header alt-svc "h3=\":443\"; ma=86400"
	http-response set-header Strict-Transport-Security "max-age=31536000; preload;"
	http-response set-header Referrer-Policy strict-origin-when-cross-origin
	option httpchk GET /health
	timeout connect 30s
	timeout server 5m
	server mediaserver check maxconn 1000

backend nextcloud
# CSP provided by application
	http-request set-path /remote.php/dav if { path_beg -i /.well-known/carddav }
	http-request set-path /remote.php/dav if { path_beg -i /.well-known/caldav }
	http-response add-header alt-svc "h3=\":443\"; ma=86400"
	http-response set-header Strict-Transport-Security "max-age=31536000; preload;"
	timeout check 5s
	balance roundrobin
	option tcp-check
	tcp-check connect
	server docker1 check
	server docker2 check
	server docker3 check

backend ftbranks
	http-response set-header Content-Security-Policy "script-src 'none'; object-src 'none'; base-uri 'none';"
	http-response add-header alt-svc "h3=\":443\"; ma=86400"
	http-response set-header Strict-Transport-Security "max-age=31536000; preload;"
	option httpchk HEAD /
	server docker1 check
	server docker2 check
	server docker3 check

backend no_route
	option httpclose
	http-request deny deny_status 404

1 Like

Thanks storm! I will try this out and see if I can salvage some pride.
I have dabbled with Grav - was quite impressed, so will resurrect all the test files and retry.
Again, my thanks for the help & patience