Getting pfsense/HAproxy to work behind Cloudflare

[kevdog]

Hi - I’m really new to using HAproxy as I’ve been proxy either Apache/Ngnix as reverse proxies.

I’m using HA proxy though the pfsense configuration.

My setup is basically

client—>Cloudflare---->pfsense/HAproxy---->Web Server

I’m only interested in using HAproxy as a reverse proxy at this time. I have working Lets Encrypt SSL certs installed on pfsense.

I’ll post my configuration, but in a nutshell I’m getting a Cloudflare 522 error saying there is a connection timeout to the server.

Here is my config with come of the details redacted:

# Automaticaly generated, dont edit manually.
# Generated on: 2020-01-20 18:00
global
	maxconn			1000
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	log-send-hostname		HA
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend front
	bind			<WANIP>:443 name <WANIP>:443   ssl crt-list /var/etc/haproxy/gohilton.com.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	acl			ACL1	var(txn.txnhost) -m str -i <URL>
	http-request set-var(txn.txnhost) hdr(host)
	use_backend back_ipvANY  if  ACL1 

frontend http-to-https-WAN
	bind			<WANIP>:80 name <WANIP>:80   
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000

backend back_ipvANY
	mode			http
	id			102
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			back 10.0.1.158:80 id 103 check inter 1000

My only concern is that the WAN IP is different than the proxied Cloudflare IP I have listed.
Thanks for any help

[lukastribus]

This has probably nothing to do with haproxy, but with Cloudflare unable to actually open TCP connections, as 522 means TCP times out while connecting:

Troubleshooting Cloudflare 5XX errors

You should check your pfsense rules and confirm that the allow connections to port 80 and 443.

[kevdog]

@lukastribus
I’m still confused about what to allow through in the firewall

HA proxy is going to take a request on WAN 80/443 and forward it in my case to LAN 10.0.1.158:80
I have the serverlist from cloudflare however do they need access to the proxy or the actual webserver?

Thanks for tip.

[lukastribus]

The proxy. Cloudflare needs to access port 80 and 443 on your WAN IP.

[kevdog]

Thanks for Clarification however I’m not sure what I’ve setup wrong. I’ve allowed all WAN traffic to WAN address on ports 80/443.

Unfortunately when doing this I’m still getting a 525 handshake error from cloudflare which I don’t know how to rectify.

Question — What do I do for computers within the LAN that need to go through the proxy to the internal website. I’m able to access the machine within the LAN directly and the ip address: http://10.0.1.158, however for SSL access here is what I’ve tried.

1. Created a DNS host override to point my domain name to the 10.0.1.1 (the pfsense/HA proxy address).
2. Created a frontend that not only listens on WAN IP Port 80/443, but also LAN IP Port 80/433
3. Created frontend acl/condition that if host matches either <domain.com> or www.<domain.com> the connection will be forwarded to the backend.
4. Kept the backend the same – forward to 10.0.1.158:80.

I’m assuming this is correct?

In terms of securing the site, mozilla recommends:

global
# intermediate configuration, tweak to your needs
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

# curl https://ssl-config.mozilla.org/ffdhe2048.txt > /path/to/dhparam.pem
ssl-dh-param-file /path/to/dhparam.pem

frontend ft_test
    mode    http
    bind    :443 ssl crt /path/to/<cert+privkey+intermediate> alpn h2,http/1.1
    bind    :80
    redirect scheme https code 301 if !{ ssl_fc }

# HSTS (63072000 seconds)
http-response set-header Strict-Transport-Security max-age=63072000

Unfortunately my version of HA proxy does not support ssl-default-bind-ciphersuites or ssl-default-server-ciphersuites so I omitted these.

Here is ha-config:

# Generated on: 2020-01-21 16:47
global
	maxconn			1000
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	log-send-hostname		domain.com-HA
	server-state-file /tmp/haproxy_server_state
	ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
	ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
	ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
	ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend shared-frontend-WAN-and-LAN-merged
	bind			69.xxx.xxx.xxx:443 name 69.xxx.xxx.xxx:443   ssl crt-list /var/etc/haproxy/shared-frontend-WAN-and-LAN.crt_list
	bind			10.0.1.1:443 name 10.0.1.1:443   ssl crt-list /var/etc/haproxy/shared-frontend-WAN-and-LAN.crt_list
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			ACL1	var(txn.txnhost) -m str -i domain.com
	acl			ACL2	var(txn.txnhost) -m str -i www.domain.com
	http-request set-var(txn.txnhost) hdr(host)
	use_backend domain.com_ipvANY  if  ACL1
	use_backend domain.com_ipvANY  if  ACL2

frontend http-to-https
	bind			69.xxx.xxx.xxx:80 name 69.xxx.xxx.xxx:80
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	redirect scheme https code 301 if !{ ssl_fc }

backend domain.com_ipvANY
	mode			http
	id			102
	log			global
	option			log-health-checks
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS /
	server			domain.com 10.0.1.158:80 id 103 check inter 100

With these settings however I can not connect to server either from WAN or LAN:

Here is a wget from LAN side:

--2020-01-21 16:50:47--  https://domain.com/
Resolving gohilton.com (domain.com)... 10.0.1.1
Connecting to gohilton.com (domain.com)|10.0.1.1|:443... connected.
HTTP request sent, awaiting response...

And it sits at this point until a timeout occurs after about 30 seconds or so ( along time) and I finally receive a:

--2020-01-21 16:50:47--  https://domain.com/
Resolving gohilton.com (domain.com)... 10.0.1.1
Connecting to gohilton.com (domain.com)|10.0.1.1|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2020-01-21 16:52:47 ERROR 503: Service Unavailable.

I’m unclear about cipher settings.

[lukastribus]

Stop doing everything at once.

1. Don’t restrict access to Cloudflare IPs only, you can do that later, once you got it all figured out
2. Don’t try from within the LAN to access the public-IP; depending on the NAT stack in pfsense, this may or may not work (NAT loopback)
3. Try from a different connection (like 3G/4G smartphone with Wifi turned off) to open the website (port 80 and port 443)

Does pfsense run any webserver itself for its own interface? Does that run on port 80 or 443? Make sure that you are not trying to run 2 different things on the same ports.

@kevdog

And it sits at this point until a timeout occurs after about 30 seconds or so ( along time) and I finally receive a:

Perhaps your backend server doesn’t like the OPTIONS check. Remove health checking and read the haproxy logs.

Simplify your configuration and start with small steps.

[kevdog]

This will be the thread I’ll use

1. I opened all sources to WAN and didn’t restrict to cloudflare. In fact I turned cloudflare proxy off not to confuse things
2. I’m using a phone with a 4g connection (wifi off) to test external connection.
3. pfsense runs internal ngnix webserver however I switched port to 81.
4. Backend server is reported as up, and in fact I can verify that within logs of apache server and within HAproxy itself.
5. I removed all the SSL options as specified by mozilla since those didn’t seem to work
6. Logs – Yikes. I’m having a hard time viewing them. Not a lot of output being produced.

I’m aware on the logs at the http server however. Access logs are full of statements like:
10.0.1.1 - - [21/Jan/2020:17:54:13 -0600] “OPTIONS / HTTP/1.0” 200 - “-” “-”
10.0.1.1 - - [21/Jan/2020:17:54:13 -0600] “HEAD / HTTP/1.0” 200 - “-” “-”
10.0.1.1 - - [21/Jan/2020:17:54:13 -0600] “OPTIONS / HTTP/1.0” 200 - “-” “-”
10.0.1.1 - - [21/Jan/2020:17:54:13 -0600] “HEAD / HTTP/1.0” 200 - “-” “-”
10.0.1.1 - - [21/Jan/2020:17:54:13 -0600] “OPTIONS / HTTP/1.0” 200 - “-” “-”

In terms of testing
About 75% of time I’m able to access the index.html file from the LAN side – I’m aware possible with NAT reflection etc, however most of the time things work.

From WAN side – I never get a connection. I usually get a timeout error. I’m unsure why the proxy isn’t passing traffic.

[lukastribus]

Check this posts for a basic syslog config:

How to enable HAProxy Stats Log

You can find the details of the logging system in the documentation, section 8 [1]. As an example, here is a basic logging configuration: global log 127.0.0.1 syslog debug defaults log global mode http option httplog [1] https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#8

@kevdog

From WAN side – I never get a connection. I usually get a timeout error. I’m unsure why the proxy isn’t passing traffic.

Ok, but which timeout? Do you get a 50x http error back after 30 seconds, or do you get a connection error directly in the browser?

The former means you can reach haproxy but it doesn’t go any further, the latter means you are not reaching haproxy at all (firewall issue).

Make sure you don’t have multiple haproxy processes running in the background.

[kevdog]

I’m getting a too long to respond ERR_CONNECTION_TIMED_OUT from mobile phone browser.

Hey thanks for pointing me in the right direction of telling me it was a firewall issue. The firewall rules were set up correctly, but I had a left over NAT that was forwarding connections from port 80/443 to the backend webserver. So basically it seemed like I had a race condition between HA proxy and the NAT table.

After deactivating the NAT statements, traffic now passes. I went ahead and then quickly made changes adding back in the SSL statements back into the proxy config and things also worked.

I’m posting my settings here for historical reasons and for those they may have similar problems in the future (including me).

pfsense WAN firewall rules:

pfsense Aliases to Define Cloudflare Networks

Make sure you do not have or have deactivated any NAT redirection on ports 80/443 for the firewall. Picture below shows the NAT rules deactivated (greyed out)

Haproxy.cfg (This is applicable to only one backend. Very possible to add more)

Please note my LAN network is on the 10.0.1.0/24 subnet. Adjust accordingly to your needs:

# Automaticaly generated, dont edit manually.
# Generated on: 2020-01-21 18:54
global
	maxconn			1000
	log			/var/run/log	local0	debug
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state
	ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
	ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
	ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
	ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend shared-frontend-WAN-and-LAN-merged
	bind			69.xxx.xxx.xxx.xxx:443 name 69.xxx.xxx.xxx.xxx:443   ssl crt-list /var/etc/haproxy/shared-frontend-WAN-and-LAN.crt_list
	bind			10.0.1.1:443 name 10.0.1.1:443   ssl crt-list /var/etc/haproxy/shared-frontend-WAN-and-LAN.crt_list
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			ACL1	var(txn.txnhost) -m str -i domain.com
	acl			ACL2	var(txn.txnhost) -m str -i www.domain.com
	http-request set-var(txn.txnhost) hdr(host)
	use_backend domain.com_ipvANY  if  ACL1
	use_backend domain.com_ipvANY  if  ACL2

frontend http-to-https
	bind			69.xxx.xxx.xxx.xxx:80 name 69.xxx.xxx.xxx.xxx:80
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	redirect scheme https code 301 if !{ ssl_fc }

backend domain.com_ipvANY
	mode			http
	id			102
	log			global
	option			log-health-checks
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS /
	server			domain.com 10.0.1.158:80 id 103 check inter 600000

Lastly @lukastribus – Thanks a lot for your help. Helping beginners really stinks sometimes since they are oftentimes uninformed and don’t give you all the information needed. Thanks for your patience. If you could mark the thread solved – or edit the title of the thread to include SOLVED that would be great. Thanks.