Config not working after upgrade from 1.5 to 1.8.13 (pfsense upgrade to 2.4.4)

Last year I followed this great tutorial, and I got openvpn, ssh, and some websites to work from a single 443 port.

Everything was working great until last week, when I upgraded my pfsense box from 2.4.3 to 2.4.4. It also upgraded the haproxy package from 1.5 to 1.8, and now haproxy does not proxy anything except for openvpn that still works.

Running the config through the config check says it is OK.

I tried to get some debug logging but nothing is sent to my syslog server (it was working before). And when I launch haproxy in debug mode with “-d”, I get the default log format instead of the tcp or http as defined in the config.

Is there any major changes in the config from 1.5 to 1.8?

My current configuration:

haproxy.cfg
# Automaticaly generated, dont edit manually.
# Generated on: 2018-10-04 00:12
global
	maxconn			500
	log			log.example.com:514	local0	debug
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	log-send-hostname		pfs
	server-state-file /tmp/haproxy_server_state
	# Modern browser compatibility only as mentioned here:
	# https://wiki.mozilla.org/Security/Server_Side_TLS
	ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
	tune.ssl.default-dh-param 2048
	
	# Time-to-first-Byte (TTFB) value needs to be optimized based on
	# the actual public certificate chain see
	# https://www.igvita.com/2013/10/24
	# /optimizing-tls-record-size-and-buffering-latency/
	tune.ssl.maxrecord 1370

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend WAN_HTTPS_offload-merged
	bind			127.0.0.1:2043 name 127.0.0.1:2043  no-sslv3 no-tls-tickets no-tlsv10 no-tlsv11 ssl crt-list /var/etc/haproxy/WAN_HTTPS_offload.crt_list ca-file /var/etc/haproxy/clientca_WAN_HTTPS_offload.pem verify optional  accept-proxy npn http/1.1
	mode			http
	log			global
	option			httplog
	option			http-keep-alive
	timeout client		7200000
	# Remove headers that expose security-sensitive information.
	rspidel ^Server:.*$
	rspidel ^X-Powered-By:.*$
	rspidel ^X-AspNet-Version:.*$
	
	# add some security related headers
	#rspadd Content-Security-Policy:\ default-src\ https:\ wss:\ data:\ \'unsafe-inline\'\ \'unsafe-eval\'
	
	rspadd X-Frame-Options:\ SAMEORIGIN
	rspadd X-Content-Type-Options:\ nosniff
	rspadd X-Xss-Protection:\ 1;\ mode=block
	
	capture request header X-Forwarded-For len 15
	capture request header host  len 50
	capture request header X-Plex-Device-Name  len 50
	
	declare capture response len 32
	declare capture response len 32
	http-response capture res.hdr(X-Forwarded-For) id 0
	http-response capture res.hdr(host) id 1
	acl			aclsystem_ssl_c_used	ssl_c_used
	acl			is_local	src 192.168.10.0/24
	acl			CF_cert_ok	ssl_c_verify 0
	acl			is_gateway	src 192.168.10.254
	acl			is_local	src 10.105.8.0/24
	acl			Plex_random	var(txn.txnhost) -m str -i plex.example.com
	acl			Plex_random	var(txn.txnhost) -m str -i plex.example.com:443
	http-request set-var(txn.txnhost) hdr(host)
	http-request add-header X-Forwarded-For %[req.hdr(CF-Connecting-IP)]  if  CF_cert_ok aclsystem_ssl_c_used 
	http-request add-header X-Forwarded-For %ci  if  is_local 
	use_backend plex_nas_ipvANY  if  Plex_random is_local !is_gateway 
	use_backend plex_nas_ipvANY  if  Plex_random CF_cert_ok aclsystem_ssl_c_used 
	default_backend None_backend_ipvANY

frontend WAN_SSLH-merged
	bind			127.0.0.1:2022 name 127.0.0.1:2022  no-sslv3 ssl crt-list /var/etc/haproxy/WAN_SSLH.crt_list  accept-proxy npn ssh/2.0
	mode			tcp
	log			global
	timeout client		7200000
	acl			ssl_acl	ssl_fc_npn -i ssh/2.0
	acl			proxmox	ssl_fc_sni_reg  prox.ssh.example.com
	use_backend ssh_proxmox_b_ipvANY  if  proxmox ssl_acl 

frontend WAN_443-merged
	bind			192.168.10.100:443 name 192.168.10.100:443   
	mode			tcp
	log			global
	timeout client		30000
	tcp-request content accept if { req.ssl_hello_type 1 } or !{ req.ssl_hello_type 1 }
	tcp-request inspect-delay	5s
	acl			is_local	src 192.168.10.0/24
	acl			vpn_acl	req.len 0
	acl			vpn_acl_1	req.ssl_hello_type 1
	acl			ssh_acl	req_ssl_hello_type 1
	acl			ssh_acl_1	req_ssl_sni -m end -i ssh.example.com
	acl			https	req.ssl_ver gt 0
	acl			offload_hostnames	req.ssl_sni -i plex.example.com
	acl			proxy_hostnames	req.ssl_sni -i bw.example.com
	tcp-request content accept if { req.ssl_ver gt 0 }
	use_backend OpenVPN_local_tun_b_ipvANY  if  !vpn_acl !vpn_acl_1 
	use_backend WAN_SSLH_b_ipvANY  if  ssh_acl ssh_acl_1 
	use_backend WAN_HTTPS_offload_b_ipvANY  if  https offload_hostnames 
	use_backend WAN_HTTPS_proxy_b_ipvANY  if  https proxy_hostnames 
	default_backend None_backend_ssl_ipvANY

frontend WAN_HTTPS_proxy-merged
	bind			127.0.0.1:2044 name 127.0.0.1:2044   accept-proxy npn http/1.1
	mode			tcp
	log			global
	option			tcplog
	timeout client		7200000
	tcp-request inspect-delay	5s
	acl			is_local	src 192.168.10.0/24
	acl			is_gateway	src 192.168.10.254
	acl			bitwarden	req.ssl_sni -i bw.example.com
	tcp-request content accept if { req.ssl_hello_type 1 }
	use_backend bitwarden_nas_ipvANY  if  bitwarden 
	default_backend None_backend_ssl_ipvANY

frontend test
	bind			192.168.10.100:80 name 192.168.10.100:80   
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000

backend None_backend_ipvANY
	mode			http
	id			101
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			none_backend_server 127.0.0.1:81 id 100  

backend plex_nas_ipvANY
	mode			http
	id			112
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			plex_nas 192.168.20.2:32400 id 113 check inter 1000  

backend ssh_proxmox_b_ipvANY
	mode			tcp
	id			114
	log			global
	timeout connect		30000
	timeout server		30000
	retries			2
	server			proxmox_server 192.168.10.30:22 id 115  

backend None_backend_ssl_ipvANY
	mode			tcp
	id			102
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			none_backend_server 127.0.0.1:81 id 103  

backend OpenVPN_local_tun_b_ipvANY
	mode			tcp
	id			104
	log			global
	timeout connect		3000
	timeout server		7200000
	retries			2
	server			OpenVPN_local_tun 192.168.10.100:100911 id 105  

backend WAN_SSLH_b_ipvANY
	mode			tcp
	id			110
	log			global
	timeout connect		30000
	timeout server		7200000
	retries			3
	server			WAN_SSLH 127.0.0.1:2022 id 111 ssl  verify none send-proxy 

backend WAN_HTTPS_offload_b_ipvANY
	mode			tcp
	id			108
	log			global
	timeout connect		30000
	timeout server		7200000
	retries			3
	server			WAN_HTTPS_offload 127.0.0.1:2043 id 109 ssl  verify none send-proxy 

backend WAN_HTTPS_proxy_b_ipvANY
	mode			tcp
	id			116
	log			global
	timeout connect		30000
	timeout server		7200000
	retries			3
	server			WAN_HTTPS_proxy 127.0.0.1:2044 id 117 ssl  verify none send-proxy 

backend bitwarden_nas_ipvANY
	mode			tcp
	id			118
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			bitwarden_nas_server 192.168.20.2:8091 id 119 ssl check inter 1000  verify none

Does anyone know if there are any changes that need to be done so that this config can work for 1.8?

This configuration cannot work in 1.5, for multiple reasons.

Clearly pfsense changed/recreated this configuration from scratch with different and new features enabled, so a comparison between the two cannot be made.

Use the actual configuration that works in 1.5 and try that in 1.8, first of all.

When you upgraded pfsense, both haproxy release and configuration was heavily modified, so first of all you need to normalize the situation.

I checked that the configuration that worked in 1.5 does works with 1.8, it just needed some adjustment for certificates that pfsense moved after the upgrade.

I checked more carefully the difference between this working configuration and the one generated by pfsense 2.4.4. And I found out that the migration to 2.4.4 changed a setting for the backend that previously generated an “check-ssl” on the backend server to a “ssl”.

Disabling this “Encrypt(SSL)” option on the pfsense backends panels made everything work again!