SSL Termination with accept-proxy/send-proxy-v2

Greetings,

I’m currently searching for a way to implement accept-proxy & send-proxy-v2 to my haproxy instance.
My goal is that nginx (reverse proxy) is able to receive the IP address of the caller from haproxy instead of the haproxy ip.

Currently I have an issue that whenever I enable this, I get a SSL handshake error (more Details below).

    HA-Proxy version 2.0.18-1 2020/09/30 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-OKgPAa/haproxy-2.0.18=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=4).
Built with OpenSSL version : OpenSSL 1.1.1g  21 Apr 2020
Running on OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.34 2019-11-21
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE     mux=H2
              h2 : mode=HTTP       side=FE        mux=H2
       <default> : mode=HTX        side=FE|BE     mux=H1
       <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS

Available services :
	prometheus-exporter

Available filters :
	[SPOE] spoe
	[COMP] compression
	[CACHE] cache
	[TRACE] trace

The error I get is:

● haproxy.service - HAProxy Load Balancer
     Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2020-11-01 10:55:52 EST; 23s ago
       Docs: man:haproxy(1)
             file:/usr/share/doc/haproxy/configuration.txt.gz
    Process: 7573 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
   Main PID: 7575 (haproxy)
      Tasks: 5 (limit: 4689)
     Memory: 41.6M
     CGroup: /system.slice/haproxy.service
             ├─7575 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
             └─7577 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock

Nov 01 10:55:52 aurora-gw haproxy[7575]: Proxy cisco-vpn started.
Nov 01 10:55:52 aurora-gw haproxy[7575]: Proxy cisco-vpn started.
Nov 01 10:55:52 aurora-gw haproxy[7575]: Proxy gw-web-ssl started.
Nov 01 10:55:52 aurora-gw haproxy[7575]: [NOTICE] 305/105552 (7575) : New worker #1 (7577) forked
Nov 01 10:55:52 aurora-gw haproxy[7575]: Proxy gw-web-ssl started.
Nov 01 10:55:52 aurora-gw haproxy[7575]: Proxy gw-web started.
Nov 01 10:55:52 aurora-gw haproxy[7575]: Proxy gw-web started.
Nov 01 10:55:52 aurora-gw systemd[1]: Started HAProxy Load Balancer.
Nov 01 10:55:52 aurora-gw haproxy[7577]: [WARNING] 305/105552 (7577) : Server gw-web-ssl/server1 is DOWN, reason: Socket error, info: "SSL handshake failure", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Nov 01 10:55:52 aurora-gw haproxy[7577]: [ALERT] 305/105552 (7577) : backend 'gw-web-ssl' has no server available!

My configuration looks like this:

global

  log /dev/log  local0

  log /dev/log  local1 notice

  chroot /var/lib/haproxy

  stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners

  stats timeout 30s

  user haproxy

  group haproxy

  daemon

  # Default SSL material locations

  ca-base /etc/ssl/certs

  crt-base /etc/ssl/private

  # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate

        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384

        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults

  maxconn 1000

  mode http

  log global

  option dontlognull

  timeout http-request 5s

  timeout connect 5000

  timeout client 2000000 # ddos protection

  timeout server 2000000 # stick-table type ip size 100k expire 30s store conn_cur

  errorfile 400 /etc/haproxy/errors/400.http

  errorfile 403 /etc/haproxy/errors/403.http

  errorfile 408 /etc/haproxy/errors/408.http

  errorfile 500 /etc/haproxy/errors/500.http

  errorfile 502 /etc/haproxy/errors/502.http

  errorfile 503 /etc/haproxy/errors/503.http

  errorfile 504 /etc/haproxy/errors/504.http

listen www-http

  bind *:80

  mode http

  use_backend gw-web

listen www-https

  bind *:443 accept-proxy ssl crt /home/stpuser/vault/kicb.org.pem ciphers ECDHE-RSA-AES256-SHA:-RC4-SHA:HIGH:!MD5:!aNULL:!EDH

  mode tcp

  option tcplog

  tcp-request inspect-delay 5s

  tcp-request content accept if { req_ssl_hello_type 1 }

  use_backend gw-web-ssl if { req.ssl_sni -m end .kicb.org }  # <--- Website trough 443 SSL

  use_backend cisco-vpn if { req.ssl_sni gate.knight-industries.org }  # <--- OpenConnect VPN 

  default_backend gw-web-ssl

backend cisco-vpn

  mode tcp

  option tcp-check

  server vpn-server 10.0.0.9:443 send-proxy-v2

backend gw-web-ssl

  mode tcp

  server server1 10.0.0.120:444 send-proxy-v2 check maxconn 20 ssl ca-file /home/stpuser/.acme.sh/kicb.org/fullchain.cer verify none

backend gw-web

  mode http

  server server2 10.0.0.120:88

The certificate is a public trusted by LetsEncrypt. Any ideas what could be wrong?

Pretty much everything about this configuration is wrong.

Can you reexplain your setup in your own words? Including where the clients connect to directly, and where you actually want to terminate the SSL connection?

Do you want to terminate SSL on haproxy, and therefor switch haproxy -> nginx to plaintext? What about
the cisco-vpn backend? Do you want to terminate SSL for that on haproxy as well?

Or do you to passthrough SSL, with SSL enabled on cisco-vpn and nginx backends?

Hello Lukas,

The cisco-vpn backend actually is no longer in use, I forgot to remove it from the config.
The config works when I remove the accept proxy && send-proxy-v2.

What is my goal:

I have a nginx reverse proxy that servers some internal resources in my home-environment.
I want to deny specific IPs from accessing said resources and currently doing this using the location block in nginx. This way I can allow for specific subfolders to be still accessible.

The clients connect on HAProxy 10.0.0.120 and then the request gets sent to nginx.
nginx runs on that same dedicated box used solely for reverse-proxy/load-balancing stuff.

The connection therefore is done on the same machine, 127.0.0.1 would probably explained it better then using the 10.0.0.120 IP for nginx on the backends…

nginx can stay with SSL, I have always a valid certificate in place there. The only thing I really need is being able to pass the requester IP to nginx.

Does this make sense to you?

Got it.

Let’s get the small stuff out of the way first:

tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
use_backend gw-web-ssl if { req.ssl_sni -m end .kicb.org }

This is a configuration for SSL passthrough. You are terminating SSL here.

Now the primary issue is that you put accept-proxy on haproxy. It doesn’t belong there. accept-proxy means it accepting and expecting the proxy protocol there, which is wrong, here your clients connect and they don’t send the proxy protocol.

Remove accept-proxy from the haproxy configuration.

To configure haproxy to send the proxy protocol to the backend, one configuration change is enough and that is send-proxy or send-proxy-v2 on the backend server. That’s it.

Now what you also have to configure nginx, so you need to adjust the nginx listen configuration, adding the proxy_protocol keyword. v2 is supported depending on your nginx release. Better not use v2 to be sure.

Hi Lukas,

Sorry for the late reply, was pretty busy. I have now implemented said changes.
I don’t get any error, however I’m now not sure in which variable the requester’s IP is located.

In nginx I define the block rules like this:

location / {
    
        allow 10.0.0.14;
        allow 10.0.0.21;
        allow 10.0.0.120;
        deny all;

        }

But I probably now need to have some sort of mechanism that tells nginx what is the requesters IP rather then using the haproxy IP for that.

As soon as I remove the haproxy IP (10.0.0.120) I get a 401. I’m connecting from IP 10.0.0.21

Share your nginx configuration completely.

Hi,

Sorry I have not received an e-mail notification and I missed your reply.
Here is the vhost configuration:

        map $http_upgrade $connection_upgrade {                                                                                                                                                                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                                                                                                                                                                                    
                default upgrade;                                                                                                                                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                                                                                                                                    
                ''      close;                                                                                                                                                                                                                                                                                                                                                                                                      
        }                                                                                                                                                                                                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                                                                                                                                                                    
        map $host $upstream_dnsmanager {                                                                                                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                                                                                                                    
        default 10.0.0.2;                                                                                                                                                                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                                                                                                                                                                    
        }                                                                                                                                                                                                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                                                                                                                                                                    
        server {                                                                                                                                                                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                                                                                                                                    
                listen 444 ssl proxy_protocol;                                                                                                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                                                                    
                include includes/certs/cert-wildcard-kicb.org.conf;                                                                                                                                                                                                                                                                                                                                                                 
                include includes/certs/ssl-base.conf;                                                                                                                                                                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                                                                                                                                                                                    
                server_name dns-manager.kicb.org;                                                                                                                                                                                                                                                                                                                                                                                   
                location ~/(.*)$ {                                                                                                                                                                                                                                                                                                                                                                                                  
                proxy_pass http://$upstream_dnsmanager;                                                                                                                                                                                                                                                                                                                                                                             
                include includes/ki-proxy.conf;                                                                                                                                                                                                                                                                                                                                                                                     
                include /etc/nginx/includes/cors-setup.conf;                                                                                                                                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                                                    
                allow 10.0.0.14;                                                                                                                                                                                                                                                                                                                                                                                                    
                allow 10.0.0.21;                                                                                                                                                                                                                                                                                                                                                                                                    
                allow 10.0.0.120;                                                                                                                                                                                                                                                                                                                                                                                                   
                deny all;                                                                                                                                                                                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                                                                                                                                                                    
                }                                                                                                                                                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                                                                                                                                    
                include includes/inc-ki.conf;                                                                                                                                                                                                                                                                                                                                                                                       
                location ~ /\.ht {                                                                                                                                                                                                                                                                                                                                                                                                  
                                deny all;                                                                                                                                                                                                                                                                                                                                                                                         
                                }                                                                                                                                                                                                                                                                                                                                                                                                   
                }                                         

Hope that is sufficient.
Happy new year.

Just wanted to say that the question is still relevant.
If someone has an answer to it please share.

Many thanks.

Likely allow/deny rules in nginx look at the actual source IP, not what’s send in the proxy protocol. You really need to reach out to nginx support.

1 Like

Sorry to bump this thread, just wanted to share the resolution / fix that needs to be applied on nginx to get it to work with HAProxy:

set_real_ip_from 10.0.0.120;
set_real_ip_from 10.0.0.121;

real_ip_header proxy_protocol;
real_ip_recursive on;

Where as 10.0.0.120 & 10.0.0.121 are the HAProxy IPs.
With that nginx then knows the haproxy IPs and gets the actual caller IP.

More detailed explanation: Aebian - Work with nginx deny rules and HAproxy