pfSense + haProxy 1 ip, 2 servers

maxtim · July 24, 2020, 2:42am

Hello,
The scenario seems pretty simple, but I am having a very difficult time implementing. Briefly:
WAN → pfSense(haproxy) -1> x.x.x.249 example1.com → x.x.x.246 example2.com

I have certs on both servers using certbot/letsencrypt. Currently, both sites are returning error 500.
I’m really not sure how to troubleshoot/where to go from here as I’ve been feverishly searching for about two days now to get this to work.

pfsense v 2.4.5-RELEASE-p1
– haproxy v 1.8.25

Relevant configs:

Automaticaly generated, dont edit manually.
Generated on: 2020-07-23 22:17
global
 maxconn			10000
 stats socket /tmp/haproxy.socket level admin  expose-fd listeners
 uid			80
 gid			80
 nbproc			1
 nbthread			1
 hard-stop-after		15m
 chroot				/tmp/haproxy_chroot
 daemon
 tune.ssl.default-dh-param	2048
 server-state-file /tmp/haproxy_server_state
 ssl-server-verify none

listen HAProxyLocalStats
 bind 127.0.0.1:2200 name localstats
 mode http
 stats enable
 stats admin if TRUE
 stats show-legends
 stats uri /haproxy/haproxy_stats.php?haproxystats=1
 timeout client 5000
 timeout connect 5000
 timeout server 5000

frontend SplitFinleyCorbett
 bind			<snip>:443 name <snip>:443   ssl crt-list /var/etc/haprox /SplitFinleyCorbett.crt_list  
 mode			http
 log			global
 option			http-keep-alive
 timeout client		30000
 acl			TimFinley	var(txn.txnhost) -m str -i example1.com
 acl			CorbettCloud	var(txn.txnhost) -m str -i example2.com
 http-request set-var(txn.txnhost) hdr(host)
 use_backend DebianServer_ipvANY  if  TimFinley aclcrt_SplitFinleyCorbett
 use_backend CorbettCloud_ipvANY  if  CorbettCloud aclcrt_SplitFinleyCorbett

backend DebianServer_ipvANY
 mode			http
 id			100
 log			global
 timeout connect		30000
 timeout server		30000
 retries			3
 option			httpchk OPTIONS / 
 server			DebianServer 192.168.1.249:443 id 101 check inter 1000  ssl verify none 

backend CorbettCloud_ipvANY
 mode			http
 id			102
 log			global
 timeout connect		30000
 timeout server		30000
 retries			3
 option			httpchk OPTIONS / 
 server			CorbettCloud 192.168.1.246:443 id 103 check inter 1000  ssl verify none

stats show:

pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses,
HAProxyLocalStats,FRONTEND,1,1,2000,2,135,24816,0,0,0,OPEN,1,2,0,0,1,0,1,0,1,0,0,0,0,1,1,2,0,0,0,0,http,1,1,2,2,0,0,
HAProxyLocalStats,BACKEND,0,0,0,0,200,0,135,24816,0,0,0,0,0,0,UP,0,0,0,0,221,1,2,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,http,roundrobin,
SplitFinleyCorbett,FRONTEND,0,0,2000,0,0,0,0,0,0,OPEN,1,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,http,0,0,0,0,0,0,
DebianServer_ipvANY,DebianServer,0,0,0,0,0,0,0,0,0,0,0,0,UP,1,1,0,0,0,221,0,1,100,101,0,2,0,0,L7OK,200,1,0,0,0,0,0,0,0,0,-1,OK,0,0,0,0,Layer7 check passed,2,3,4,192.168.1.249:443,http,
DebianServer_ipvANY,BACKEND,0,0,0,0,200,0,0,0,0,0,0,0,0,0,UP,1,1,0,0,221,0,1,100,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,http,roundrobin,
CorbettCloud_ipvANY,CorbettCloud,0,0,0,0,0,0,0,0,0,0,0,0,DOWN,1,1,0,1,1,221,221,1,102,103,0,2,0,0,L7STS,500,9,0,0,0,0,0,0,0,0,-1,Internal Server Error,0,0,0,0,Layer7 wrong status,2,3,0,192.168.1.246:443,http,
CorbettCloud_ipvANY,BACKEND,0,0,0,0,200,0,0,0,0,0,0,0,0,0,DOWN,0,0,0,1,221,221,1,102,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,http,roundrobin,

Server Information:
both servers are running debian 10 buster, apache 2.4
example2.com is a nextcloud installation.
example1.com hosts a handful of random websites and apps that will be split up if I can successfully get haproxy to work.

regardless, both apache sites-enabled configs look similar:

000-default.conf

<VirtualHost *:80>
  ServerName example(x).com
  ServerAlias www.example(x).com

  ServerAdmin webmaster@localhost
  DocumentRoot /var/www/html

  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined

  RewriteEngine on
  RewriteCond %{SERVER_NAME} =example(x).com [OR]
  RewriteCond %{SERVER_NAME} =www.example(x).com
  RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

000-default-le-ssl.conf

 <IfModule mod_ssl.c>
   <VirtualHost *:443>
      ServerName example(x).com
      ServerAlias www.example(x).com

      ServerAdmin webmaster@localhost
      DocumentRoot /var/www/html

      ErrorLog ${APACHE_LOG_DIR}/error.log
      CustomLog ${APACHE_LOG_DIR}/access.log combined

      SSLCertificateFile /etc/letsencrypt/live/example(x).com/fullchain.pem
      SSLCertificateKeyFile /etc/letsencrypt/live/example(x).com/privkey.pem
      Include /etc/letsencrypt/options-ssl-apache.conf
  </VirtualHost>
</IfModule>

Did I miss anything?

maxtim · July 25, 2020, 3:50am

I’ve come to a better understanding about how this all works. I do not want haproxy to handle certs, so I will pass that duty to the backend servers. To do so, I must use TCP. Here is my current configuration:

Automaticaly generated, dont edit manually.
Generated on: 2020-07-24 23:14
global
 maxconn			1000
 stats socket /tmp/haproxy.socket level admin  expose-fd listeners
 uid			80
 gid			80
 nbproc			1
 nbthread			1
 hard-stop-after		15m
 chroot				/tmp/haproxy_chroot
 daemon
 tune.ssl.default-dh-param	2048
 server-state-file /tmp/haproxy_server_state
 ssl-server-verify none

listen HAProxyLocalStats
 bind 127.0.0.1:2200 name localstats
 mode http
 stats enable
 stats refresh 10
 stats admin if TRUE
 stats show-legends
 stats uri /haproxy/haproxy_stats.php?haproxystats=1
 timeout client 5000
 timeout connect 5000
 timeout server 5000

frontend mainTLS
 bind			xx.xx.xx.xx:443 name xx.xx.xx.xx:443   
 mode			tcp
 log			global
 timeout client		30000
 default_backend DebianServer_ipvANY
 frontend http-to-https
 bind			xx.xx.xx.xx:80 name xx.xx.xx.xx:80   
 mode			http
 log			global
 option			http-keep-alive
 timeout client		30000
 http-request redirect scheme https 

backend DebianServer_ipvANY
 mode			tcp
 id			100
 log			global
 timeout connect		30000
 timeout server		30000
 retries			3
 server			DebianServer 192.168.1.249:443 id 101 check inter 1000

Notice

However, that there is no other servers besides just the one. How, then, can I route traffic to example2.com based on domain name via TCP without handling certifications on haproxy?

maxtim · July 26, 2020, 5:27am

I have gotten things to work using mode tcp rather than http. However, sometimes one site gets a security error when it gets the incorrect cert. Why does one site get the incorrect cert? is haproxy sending a request to the other server first? how is this possible?

Automaticaly generated, dont edit manually.
Generated on: 2020-07-25 00:06
global
 maxconn			1000
 stats socket /tmp/haproxy.socket level admin  expose-fd listeners
 uid			80
 gid			80
 nbproc			1
 nbthread			1
 hard-stop-after		15m
 chroot				/tmp/haproxy_chroot
 daemon
 tune.ssl.default-dh-param	2048
 server-state-file /tmp/haproxy_server_state
 ssl-server-verify none

listen HAProxyLocalStats
 bind 127.0.0.1:2200 name localstats
 mode http
 stats enable
 stats refresh 10
 stats admin if TRUE
 stats show-legends
 stats uri /haproxy/haproxy_stats.php?haproxystats=1
 timeout client 5000
 timeout connect 5000
 timeout server 5000

frontend mainTLS
 bind			xx.xx.xx.xx:443 name xx.xx.xx.xx:443   
 mode			tcp
 log			global
 timeout client		30000
 default_backend DebianServer_ipvANY

frontend http-to-https
 bind			xx.xx.xx.xx:80 name xx.xx.xx.xx:80   
 mode			http
 log			global
 option			http-keep-alive
 timeout client		30000
 http-request redirect scheme https 

backend DebianServer_ipvANY
 mode			tcp
 id			100
 log			global
 timeout connect		30000
 timeout server		30000
 retries			3
 server			DebianServer 192.168.1.249:443 id 101 check inter 1000  
 server			CorCloud 192.168.1.246:443 id 103 check inter 1000

lukastribus · July 28, 2020, 10:20pm

You are just load-balancing between the two servers which is clearly NOT what you want, since those have different applications, right?

You can route based on the SNI value of the client_hello, but for this to work you need non-overlapping certificates on the backends.

Then put each server in its own backend and instead of default_backend you use use_backend based on SNI, like:

tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }

use_backend backend1 if { req_ssl_sni -i backend1.example.org }
use_backend backend2 if { req_ssl_sni -i backend2.example.org }

Also see:

https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#7.3.5-req.ssl_sni

Topic		Replies	Views
Getting pfsense/HAproxy to work behind Cloudflare Help!	8	10888	January 22, 2020
SNI HTTPS Reverse Proxy on pfSense Not Working Help!	1	4341	May 13, 2018
What is wrong with this config? Help!	6	3160	September 6, 2018
Single IP to separate internal servers SSL Help!	9	3191	September 1, 2018
Handling http to an https site only (pfsense/haproxy) Help!	3	422	January 9, 2024

pfSense + haProxy 1 ip, 2 servers

Notice

Related Topics