SNI Routing - Session Counts Doubled


#1

Hi,

We recently deployed a change to our HAProxy configuration for a client that needed to handle multiple different SSL certs and behaviours over the same IP/Port combination. For reasons I won’t go into, we couldn’t considered additional IPs, so instead we looked to utilise the SNI routing discussed in this blog to route the requests to different internal front-ends where we could terminate and handle the SSL as appropriate.

The configuration works and we have had no issues with the routing, however, the number of reported connections within the haproxy stats doubled for the Front Ends and Backends operating in TCP mode; whereas the Front Ends/Backends performing the SSL termination continue to report a similar number of connections to the figures we have prior to the change.

For example, using the names in the below, fe_TCP_SNI_Entry and be_Entry_B report 330 connections where as fe_Entry_Web_B reports 165 connections.

Has anyone experienced anything similar or have any suggestions as to why this difference (and exact doubling of the connections)? The increase in reported connections is playing havoc with our monitoring and scaling configuration.

HAProxy version:

$ haproxy -v
HA-Proxy version 1.6.4 2016/03/13

(yes, we know that we need to update)

Sanitised Configuration

global
    user                    haproxy
    group                   haproxy

    maxconn                 100000
    spread-checks           5

    pidfile                 /var/run/haproxy.pid
    chroot                  /var/lib/haproxy
    stats socket            /var/run/haproxy.sock mode 600 user haproxy level admin
    stats socket            /caci/haproxy/stats/haproxy.sock mode 600 user shinken level operator

    log                     127.0.0.1    local3     info

    #default SSL locations
    ca-base                 /etc/ssl/certs
    crt-base                /etc/ssl/private
    
    <snip ... removed SSL opts and ciphers />
    
defaults
    log                     global
    mode                    http

    #Don't log messages with no data exchange - relying on BrightSolid to protect from port scans
    option                  dontlognull
    option                  log-separate-errors

    option                  splice-auto
    option                  http-server-close
    option                  redispatch
    option                  contstats

    retries                 3

   <snip. ... removed timeouts and errors />

frontend fe_TCP_SNI_Entry
    mode tcp
    
    bind 0.0.0.0:443
    
    tcp-request inspect-delay 2s
    tcp-request content accept if { req_ssl_hello_type 1 }
	
    acl admin  req.ssl_sni -i admin."${HAPROXY_DOMAIN}"
	
    no log
    use_backend be_Entry_A if admin
    use_backend be_Entry_B
    
backend be_Entry_B
    ## Performs an internal proxy redirect to the conventional HTTP SSL termination.
	
    mode tcp
    server localhost localhost:46870 send-proxy
	
frontend fe_Web_Https_B
    mode http

    ## Perform SSL termination
    bind localhost:46870 accept-proxy ssl crt /etc/ssl/private/web.pem

   <snip ... removed server def />

backend be_Entry_A
    ## Performs an internal proxy redirect to the HTTP frontend where ACL rules are applied.
	
    mode tcp
    server localhost localhost:46869 send-proxy

frontend fe_Web_Https_A
    bind localhost:46869 accept-proxy ssl crt /etc/ssl/private/web2.pem ca-file /etc/ssl/private/client.crt verify optional crt-ignore-err all

   <snip ... remove SSL client certs and server defintions />

Thanks in advance,
MrBasset


#2

You can easily specify multiple certificates or even a directory full of certificates in a single frontend, haproxy will take of all the SNI routing required. No need for complicated TCP frontends with manual ACL based SNI routing.

This is only required if you have to pass SSL through haproxy WITHOUT terminating SSL or if you have some requirement to specify different SSL options based on SNI (for example, client certificate authentication, which I can see you indeed use in one frontend but not the other one).

Can you provide the exact output of
show stat

from the admin socket?


#3

Thanks for the reply. Indeed, the client cert requirement is what drove us down this otherwise I’d have stuck well clear. Stats output below - I’ve amended the names to match the example above (connections aren’t the nice neat doubling I saw earlier):

# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,                                                                                   lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1x                                                                                   x,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,c                                                                                   omp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,

fe_TCP_SNI_Entry,FRONTEND,,,281,597,2000,3589514,50973748996,154705389710,0,0,44,,,,,OPEN,,,,,,,,,1,4,0,,,,0,20,0,89,,,,,,,,,,,0,0,0,                                                                                   ,,0,0,0,0,,,,,,,,
be_Entry_B,localhost,0,0,280,597,,3563575,50866076275,154264675360,,0,,0,15844,0,0,no check,1,1,0,,,,,,1,5,1,,3563575,,2,20,,88,,                                                                                   ,,,,,,,,0,,,,814720,16862,,,,,0,,,25,0,0,12326,
be_Entry_B,BACKEND,0,0,280,597,200,3572760,50866076275,154264675360,0,0,,0,15844,0,0,UP,1,1,0,,0,369740,0,,1,5,0,,3563575,,1,20,,                                                                                   88,,,,,,,,,,,,,,814764,16862,0,0,0,0,0,,,25,0,0,12326,
fe_Web_Https_B,FRONTEND,,,175,423,2000,3514895,49033259628,154796569700,0,0,408833,,,,,OPEN,,,,,,,,,1,6,0,,,,0,19,0,82,,,,0,11860                                                                                   027,504567,426945,12577,6501,,44,224,12810626,,,0,0,0,0,,,,,,,,
be_Entry_A,localhost,0,0,1,29,,16754,107672721,440714350,,0,,0,5,0,0,no check,1,1,0,,,,,,1,9,1,,16754,,2,0,,24,,,,,,,,,,0,,,,2                                                                                   06,19,,,,,27,,,5,0,0,9167,
be_Entry_a,BACKEND,0,0,1,29,200,16754,107672721,440714350,0,0,,0,5,0,0,UP,1,1,0,,0,369740,0,,1,9,0,,16754,,1,0,,24,,,,,,,,,,,,                                                                                   ,,206,19,0,0,0,0,27,,,5,0,0,9167,
fe_Web_Https_A,FRONTEND,,,1,29,2000,15050,93951506,398114404,0,0,797,,,,,OPEN,,,,,,,,,1,10,0,,,,0,0,0,24,,,,0,21183,8607,3455,71,0,,                                                                                   1,53,33316,,,0,0,0,0,,,,,,,,