HAProxy community

pfSense + haProxy 1 ip, 2 servers

Hello,
The scenario seems pretty simple, but I am having a very difficult time implementing. Briefly:
WAN -> pfSense(haproxy) -1> x.x.x.249 example1.com -> x.x.x.246 example2.com

I have certs on both servers using certbot/letsencrypt. Currently, both sites are returning error 500.
I’m really not sure how to troubleshoot/where to go from here as I’ve been feverishly searching for about two days now to get this to work.

  • pfsense v 2.4.5-RELEASE-p1
    – haproxy v 1.8.25

  • Relevant configs:

    Automaticaly generated, dont edit manually.
    Generated on: 2020-07-23 22:17
    global
     maxconn			10000
     stats socket /tmp/haproxy.socket level admin  expose-fd listeners
     uid			80
     gid			80
     nbproc			1
     nbthread			1
     hard-stop-after		15m
     chroot				/tmp/haproxy_chroot
     daemon
     tune.ssl.default-dh-param	2048
     server-state-file /tmp/haproxy_server_state
     ssl-server-verify none
    
    listen HAProxyLocalStats
     bind 127.0.0.1:2200 name localstats
     mode http
     stats enable
     stats admin if TRUE
     stats show-legends
     stats uri /haproxy/haproxy_stats.php?haproxystats=1
     timeout client 5000
     timeout connect 5000
     timeout server 5000
    
    frontend SplitFinleyCorbett
     bind			<snip>:443 name <snip>:443   ssl crt-list /var/etc/haprox /SplitFinleyCorbett.crt_list  
     mode			http
     log			global
     option			http-keep-alive
     timeout client		30000
     acl			TimFinley	var(txn.txnhost) -m str -i example1.com
     acl			CorbettCloud	var(txn.txnhost) -m str -i example2.com
     http-request set-var(txn.txnhost) hdr(host)
     use_backend DebianServer_ipvANY  if  TimFinley aclcrt_SplitFinleyCorbett
     use_backend CorbettCloud_ipvANY  if  CorbettCloud aclcrt_SplitFinleyCorbett
    
    backend DebianServer_ipvANY
     mode			http
     id			100
     log			global
     timeout connect		30000
     timeout server		30000
     retries			3
     option			httpchk OPTIONS / 
     server			DebianServer 192.168.1.249:443 id 101 check inter 1000  ssl verify none 
    
    backend CorbettCloud_ipvANY
     mode			http
     id			102
     log			global
     timeout connect		30000
     timeout server		30000
     retries			3
     option			httpchk OPTIONS / 
     server			CorbettCloud 192.168.1.246:443 id 103 check inter 1000  ssl verify none
    
  • stats show:

pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses,
HAProxyLocalStats,FRONTEND,1,1,2000,2,135,24816,0,0,0,OPEN,1,2,0,0,1,0,1,0,1,0,0,0,0,1,1,2,0,0,0,0,http,1,1,2,2,0,0,
HAProxyLocalStats,BACKEND,0,0,0,0,200,0,135,24816,0,0,0,0,0,0,UP,0,0,0,0,221,1,2,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,http,roundrobin,
SplitFinleyCorbett,FRONTEND,0,0,2000,0,0,0,0,0,0,OPEN,1,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,http,0,0,0,0,0,0,
DebianServer_ipvANY,DebianServer,0,0,0,0,0,0,0,0,0,0,0,0,UP,1,1,0,0,0,221,0,1,100,101,0,2,0,0,L7OK,200,1,0,0,0,0,0,0,0,0,-1,OK,0,0,0,0,Layer7 check passed,2,3,4,192.168.1.249:443,http,
DebianServer_ipvANY,BACKEND,0,0,0,0,200,0,0,0,0,0,0,0,0,0,UP,1,1,0,0,221,0,1,100,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,http,roundrobin,
CorbettCloud_ipvANY,CorbettCloud,0,0,0,0,0,0,0,0,0,0,0,0,DOWN,1,1,0,1,1,221,221,1,102,103,0,2,0,0,L7STS,500,9,0,0,0,0,0,0,0,0,-1,Internal Server Error,0,0,0,0,Layer7 wrong status,2,3,0,192.168.1.246:443,http,
CorbettCloud_ipvANY,BACKEND,0,0,0,0,200,0,0,0,0,0,0,0,0,0,DOWN,0,0,0,1,221,221,1,102,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,http,roundrobin,

Server Information:
both servers are running debian 10 buster, apache 2.4
example2.com is a nextcloud installation.
example1.com hosts a handful of random websites and apps that will be split up if I can successfully get haproxy to work.

regardless, both apache sites-enabled configs look similar:

  • 000-default.conf

    <VirtualHost *:80>
      ServerName example(x).com
      ServerAlias www.example(x).com
    
      ServerAdmin webmaster@localhost
      DocumentRoot /var/www/html
    
      ErrorLog ${APACHE_LOG_DIR}/error.log
      CustomLog ${APACHE_LOG_DIR}/access.log combined
    
      RewriteEngine on
      RewriteCond %{SERVER_NAME} =example(x).com [OR]
      RewriteCond %{SERVER_NAME} =www.example(x).com
      RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
    </VirtualHost>
    
  • 000-default-le-ssl.conf

     <IfModule mod_ssl.c>
       <VirtualHost *:443>
          ServerName example(x).com
          ServerAlias www.example(x).com
    
          ServerAdmin webmaster@localhost
          DocumentRoot /var/www/html
    
          ErrorLog ${APACHE_LOG_DIR}/error.log
          CustomLog ${APACHE_LOG_DIR}/access.log combined
    
          SSLCertificateFile /etc/letsencrypt/live/example(x).com/fullchain.pem
          SSLCertificateKeyFile /etc/letsencrypt/live/example(x).com/privkey.pem
          Include /etc/letsencrypt/options-ssl-apache.conf
      </VirtualHost>
    </IfModule>
    

Did I miss anything?

I’ve come to a better understanding about how this all works. I do not want haproxy to handle certs, so I will pass that duty to the backend servers. To do so, I must use TCP. Here is my current configuration:

Automaticaly generated, dont edit manually.
Generated on: 2020-07-24 23:14
global
 maxconn			1000
 stats socket /tmp/haproxy.socket level admin  expose-fd listeners
 uid			80
 gid			80
 nbproc			1
 nbthread			1
 hard-stop-after		15m
 chroot				/tmp/haproxy_chroot
 daemon
 tune.ssl.default-dh-param	2048
 server-state-file /tmp/haproxy_server_state
 ssl-server-verify none

listen HAProxyLocalStats
 bind 127.0.0.1:2200 name localstats
 mode http
 stats enable
 stats refresh 10
 stats admin if TRUE
 stats show-legends
 stats uri /haproxy/haproxy_stats.php?haproxystats=1
 timeout client 5000
 timeout connect 5000
 timeout server 5000

frontend mainTLS
 bind			xx.xx.xx.xx:443 name xx.xx.xx.xx:443   
 mode			tcp
 log			global
 timeout client		30000
 default_backend DebianServer_ipvANY
 frontend http-to-https
 bind			xx.xx.xx.xx:80 name xx.xx.xx.xx:80   
 mode			http
 log			global
 option			http-keep-alive
 timeout client		30000
 http-request redirect scheme https 

backend DebianServer_ipvANY
 mode			tcp
 id			100
 log			global
 timeout connect		30000
 timeout server		30000
 retries			3
 server			DebianServer 192.168.1.249:443 id 101 check inter 1000

Notice

However, that there is no other servers besides just the one. How, then, can I route traffic to example2.com based on domain name via TCP without handling certifications on haproxy?

I have gotten things to work using mode tcp rather than http. However, sometimes one site gets a security error when it gets the incorrect cert. Why does one site get the incorrect cert? is haproxy sending a request to the other server first? how is this possible?

Automaticaly generated, dont edit manually.
Generated on: 2020-07-25 00:06
global
 maxconn			1000
 stats socket /tmp/haproxy.socket level admin  expose-fd listeners
 uid			80
 gid			80
 nbproc			1
 nbthread			1
 hard-stop-after		15m
 chroot				/tmp/haproxy_chroot
 daemon
 tune.ssl.default-dh-param	2048
 server-state-file /tmp/haproxy_server_state
 ssl-server-verify none

listen HAProxyLocalStats
 bind 127.0.0.1:2200 name localstats
 mode http
 stats enable
 stats refresh 10
 stats admin if TRUE
 stats show-legends
 stats uri /haproxy/haproxy_stats.php?haproxystats=1
 timeout client 5000
 timeout connect 5000
 timeout server 5000

frontend mainTLS
 bind			xx.xx.xx.xx:443 name xx.xx.xx.xx:443   
 mode			tcp
 log			global
 timeout client		30000
 default_backend DebianServer_ipvANY

frontend http-to-https
 bind			xx.xx.xx.xx:80 name xx.xx.xx.xx:80   
 mode			http
 log			global
 option			http-keep-alive
 timeout client		30000
 http-request redirect scheme https 

backend DebianServer_ipvANY
 mode			tcp
 id			100
 log			global
 timeout connect		30000
 timeout server		30000
 retries			3
 server			DebianServer 192.168.1.249:443 id 101 check inter 1000  
 server			CorCloud 192.168.1.246:443 id 103 check inter 1000

You are just load-balancing between the two servers which is clearly NOT what you want, since those have different applications, right?

You can route based on the SNI value of the client_hello, but for this to work you need non-overlapping certificates on the backends.

Then put each server in its own backend and instead of default_backend you use use_backend based on SNI, like:

tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }

use_backend backend1 if { req_ssl_sni -i backend1.example.org }
use_backend backend2 if { req_ssl_sni -i backend2.example.org }

Also see:

https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#7.3.5-req.ssl_sni