HAProxy RDP/Outlook Anywhere NTLM Issues


#1

I have done my due diligence and searched both here and Google for an answer to my conundrum, but I am coming up empty with my exact problem.

I will say first off, major props to the devs for this program. Amazing software that does what Micro-don’t. On to the issue…
I have a Reverse Proxy (FreeBSD 10.2 running HAProxy 1.6.4 from pkg repository) in my DMZ that listens for incoming traffic on 443. I initially ran using the SSL Passthrough method and everything worked great functionality wise. I, however, wanted to leverage some of the security features in HAProxy. So I changed my configs over to using HTTP mode and perform SSL Bridging instead. After many hours of fiddling with the configs I was finally able to get HTTPS working with SSL Bridging, so I thought. My ActiveSync and my EWS clients are working fine in this method and my Android RDP client connects successfully too, however that ugly beast NTLM is causing issues when it comes to the desktop version of the RDP client and using Outlook Anywhere external to the network.
I poured over the logs inside Remote Desktop Gateway (running on Server 2012 R2 up-to-date on patches) and quadruple checked my configs and ensuring I had the SSL Bridging mode in RD Gateway set and determined that the Android Remote Desktop client is authenticating using Kerberos over HTTP. Fancy that! But the stupid desktop RDP client uses NTLMV2 and fails connection every single time. The traffic makes it through the proxy, hits the RD Gateway server, and fails authentication throwing an Audit Failure for the user account. The same user account that was working under the TCP SSL Passthrough mode.
This gets weirder though…I decided to embark on a quest to figure this out and get SSL Bridging operating correctly. I ran across a config posted on the ALOHA documentation about the different types of HTTP and how the default is option http-keep-alive and such. The config showed a system operating in SSL Offloading mode with option http-keep-alive. I decided to give that a try. After a few updates to the config in HAProxy and an update for Remote Desktop Gateway to expect an incoming HTTP connection instead of HTTPS, lo and behold it worked! I successfully logged in with the same user and was brought to my remote desktop from a desktop RDP client.
Something hinky is definitely going on here. The software is doing what it is supposed to, but for some reason NTLMv2 refuses to work in SSL Bridge mode. I can only get this to work in SSL Offload mode. I checked Outlook Anywhere and it is having the same issue as RD Gateway. I see the traffic and the HTTPS requests logged in HAProxy, but no joy.
I have pasted a copy of my config here and I am more than happy to provide any logs to help track this issue down:

global
log /var/run/log local0 debug
#chroot /usr/jail/haproxy
stats socket /var/run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon

    # Default SSL material locations
    ca-base /path/to/cafiles
    crt-base /path/to/certs

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL).
    #ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
    #ssl-default-bind-options no-sslv3

    #Secure SSL Cipher Settings
    tune.ssl.default-dh-param 2048

    ssl-default-bind-options no-sslv3 no-tls-tickets
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

    ssl-default-server-options no-sslv3 no-tls-tickets
    ssl-default-server-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

defaults
log /var/run/log local0 debug
mode http
option httplog
option dontlognull
#timeout connect 4s
#timeout client 300s
#timeout server 300s
#errorfile 400 /usr/local/etc/haproxy/errors/400.http
#errorfile 403 /usr/local/etc/haproxy/errors/403.http
#errorfile 408 /usr/local/etc/haproxy/errors/408.http
#errorfile 500 /usr/local/etc/haproxy/errors/500.http
#errorfile 502 /usr/local/etc/haproxy/errors/502.http
#errorfile 503 /usr/local/etc/haproxy/errors/503.http
#errorfile 504 /usr/local/etc/haproxy/errors/504.http

frontend ssl_relay
bind reverseIPaddress:443 ssl crt /path/to/cert/
option http-keep-alive
capture request header Host len 32
log global
option httplog
maxconn 300

use_backend ssl_mail if { ssl_fc_sni mailserver }
use_backend ssl_rd if { ssl_fc_sni rdserver }
use_backend ssl_chat if { ssl_fc_sni chatserver }

default_backend ssl_mail

backend ssl_mail
option http-keep-alive
log global
option httplog

server mail servername:443 ssl verify none

backend ssl_rd
option http-keep-alive
log global
option httplog

server rd servername:80 check

backend ssl_chat
option http-keep-alive
log global
option httplog

server chat servername:443 ssl verify none

#2

Not really familiar with all this NTLM/Microsoft stuff here, but a quick guess would be that you have to put your backend into “mode tcp”.

Also, setting option prefer-last-server would probably be a good idea for this setup.


#3

I have a working setup right now with NTLMv2 and am using Exchange and RD Gateway. I’ve removed some config as new users can only post 2 links and some of the config is showing up as links,obviously.

Exchange is 2013 SP1 Enterprise
Servers are all 2012 R2

I noticed for some reason I couldn’t get newer RDP clients to use HTTP and had to use RPC-HTTP…I think this is actually a bug in the RDP 8 and newer clients.

When a client requests HTTP, I force a fallback to RPC-HTTP by dropping the connection. I haven’t noticed any problems with this aside from being forced to use RPC-HTTP. I’ve had it setup this way, in a production environment for months.

For what its worth I setup automated certificate renewal with LetsEncrypt as well.

Here is the relevant config:

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
daemon
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-server-options no-sslv3
ssl-default-bind-options no-tls-tickets no-sslv3
ssl-server-verify required
tune.ssl.default-dh-param 2048
ca-base /etc/ssl/certs
crt-base /etc/ssl/haproxy

defaults
mode http
balance source
option httplog
option http-keep-alive
option dontlognull
option redispatch
option contstats
timeout http-keep-alive 60s
timeout http-request 30s
timeout queue 30s
timeout connect 1800s
timeout client 1800s
timeout server 1800s
log global
retries 3
maxconn 32000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http

################################# STATUS CONFIGURATION ##################################

listen stats 10.0.0.1:8080
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /

############################## HTTP FRONTEND CONFIGURATION ##############################

frontend SSL_FRONTEND_REDIRECT
mode http
option httplog
option forwardfor
option http-server-close
option httpclose
bind MYIPADDRESS:80
acl WWW_HOST_0 hdr_beg(host) -i www.
reqirep ^Host:\ www.(.*)$ Host:\ \1 if WWW_HOST_0
redirect code 301 prefix / if WWW_HOST_0
redirect scheme https code 301 if !{ ssl_fc }

frontend SSL_FRONTEND_LOADBALANCER_0
mode http
rspidel ^Server:. rspidel ^X-Powered-By:.*
rspidel ^X-AspNet-Version:.
$
option httplog
option forwardfor
option http-no-delay
capture request header Host len 32
rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubdomains;\ preload
rspadd X-Frame-Options:\ SAMEORIGIN
rspadd Content-Security-Policy:\ default-src\ https:\ data:\ ‘unsafe-inline’\ 'unsafe-eval’
description SSL FRONTEND LOADBALANCER
bind MYIPADDRESS:443 transparent ssl crt /etc/haproxy/certificates/whatever.pem crt /etc/haproxy/certificates/ no-tls-tickets
acl EXMX_DOMAIN hdr(host) -i DOMAIN1 -i DOMAIN2 -i AUTODISCOVER1 -i AUTODISCOVER2 -i DOMAIN3 -i AUTODISCOVER3
acl RDGW_DOMAIN hdr(host) -i RDPDOMAIN -i WWW.RDPDOMAIN
use_backend EXMX_BACKEND if EXMX_DOMAIN
use_backend RDGW_BACKEND if RDGW_DOMAIN

backend EXMX_BACKEND
mode http
balance source
option httplog
option forwardfor
source 0.0.0.0 usesrc clientip
option httpchk GET /owa/healthcheck.htm
option httpchk GET /ecp/healthcheck.htm
option httpchk GET /ews/healthcheck.htm
option httpchk GET /rpc/healthcheck.htm
option httpchk GET /mapi/healthcheck.htm
option httpchk GET /oab/healthcheck.htm
option httpchk GET /microsoft-server-activesync/healthcheck.htm
option httpchk GET /autodiscover/healthcheck.htm
default-server inter 3s rise 2 fall 2
redirect prefix httpsDOMAIN1 code 301 if { hdr(host) -i WWW.DOMAIN1 }
redirect prefix httpsWWW.DOMAIN2 code 301 if { hdr(host) -i WWW.DOMAIN2 }
server EXMX0 10.0.0.20:443 check check ssl verify none weight 80
server EXMX1 10.0.0.21:443 check check ssl verify none weight 20

backend RDGW_BACKEND
mode http
log global
option httplog
balance source
option httpchk GET /RDWeb
cookie RDGW insert nocache
source 0.0.0.0 usesrc clientip
default-server inter 2s rise 3 fall 2
acl RDGW_PATH_0 path_beg -i /remoteDesktopGateway/
http-request deny if RDGW_PATH_0
redirect prefix httpsRDPDOMAIN code 301 if { hdr(host) -i WWW.RDPDOMAIN }
server RDGW0 10.0.0.11:443 check check ssl verify none check cookie RDGW0 weight 50
server RDGW1 10.0.0.12:443 check check ssl verify none check cookie RDGW1 weight 50 backup


#4

Thank you for the information.
I have tried very similar configs in my troubleshooting of this issue, but was unsuccessfully able to get them to work. In my testing I did not use any acl rules or any rspadd options or such yet to keep things simple and ease troubleshooting nor did I use the “option http-no-delay”.
What version HAPROXY are you running, jcbhltz? Also, how are you blocking HTTP requests and forcing a fallback to RPC-HTTP? Is that a setting you changed in your firewall, HAPROXY, or the Remote Desktop Gateway software? That sounds like it might be critical to my issue.


#5

I have tried setting mode tcp and it does work in that mode, but I would like to leverage some of the features of HAPROXY in the mode http setting, like a single area to control all of the HTTPS ciphers (IIS is a HUGE pain to turn on and off ciphers, requiring registry edits and such) and one central certificate to worry about renewing instead of each web server requiring an external cert.
“option prefer-last-server” I thought was relevant to load balancing only, according to the documentation.


#6

acl RDGW_PATH_0 path_beg -i /remoteDesktopGateway/
http-request deny if RDGW_PATH_0

RDP over HTTP requests the path “/remoteDesktopGateway/” so, as you can see in the above rule, I use http-request-deny to ensure a fallback.

FYI, https://www.nartac.com/Products/IISCrypto for setting up Windows OS ciphers.


#7

HA-Proxy version 1.5.8 2014/10/31
Copyright 2000-2014 Willy Tarreau w@1wt.eu


#8

You can still terminate TLS on haproxy even when using TCP mode, with all the advantages like preferred ciphers, central certificate, etc). You don’t give up anything here.