HAproxy and Outlook 2010 on Exchange 2016

Hello, I’m very new to haproxy, managed to get a working configuration by combining a week of Googling and surfing the web.

The one issue I have is OA - RPC over HTTPS is not working on L7 configuration.

Outlook 2010 won’t connect. The only error I get is Can’t connect to Exchange, mailbox unavailable (pardon, don’t recall the excact error) , outlook logging stops after autodiscover and nothing in the logs, seems like it won’t connect to the DAG.

haproxy version 2.0.8 on Debian Buster, Exchange 2016 (migrated from 2010), all the namespaces point to mail.domain.com, had a round robin DNS (yeah I know it’s 2019) which worked , won’t say perfectly but decently on the non caching ISPs, the caching ones…
hence haproxy…

snipped config :

ssl-default-bind-options no-sslv3
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-server-options no-sslv3
ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
tune.ssl.default-dh-param 2048

defaults
mode http
log global
option httplog
option dontlognull
option forwardfor except 127.0.0.0/8
option redispatch

option contstats

retries                 3
timeout http-request    10s
timeout queue           1m
timeout connect         10s
timeout client          15m # this value should be rather high with Exchange
timeout server          15m # this value should be rather high with Exchange
timeout http-keep-alive 10s
timeout check           10s
maxconn                 100000

frontend fe_ex2016
http-response set-header X-Frame-Options SAMEORIGIN
http-response set-header X-Content-Type-Options nosniff
mode http
bind 1.2.3.4:80
bind aaa:bbb:cccc:dddd::30:80 transparent
bind 1.2.3.4:443 ssl crt /etc/ssl/certs/exchange.pem
bind aaa:bbbb:cccc:dddd::30:443 transparent ssl crt /etc/ssl/certs/exchange.pem
redirect scheme https code 301 if !{ ssl_fc } # redirect 80 -> 443 (for owa)
acl autodiscover url_beg /Autodiscover
acl autodiscover url_beg /autodiscover
acl mapi url_beg /mapi
acl rpc url_beg /rpc/rpcproxy.dll
acl owa url_beg /owa
acl owa url_beg /OWA
acl eas url_beg /Microsoft-Server-ActiveSync
acl ecp url_beg /ecp
acl ews url_beg /EWS
acl ews url_beg /ews
acl oab url_beg /OAB
use_backend be_ex2016_autodiscover if autodiscover
use_backend be_ex2016_mapi if mapi
use_backend be_ex2016_rpc if rpc
option accept-invalid-http-request
use_backend be_ex2016_owa if owa
use_backend be_ex2016_eas if eas
use_backend be_ex2016_ecp if ecp
use_backend be_ex2016_ews if ews
use_backend be_ex2016_oab if oab
default_backend be_ex2016

backend be_ex2016_rpc
mode http
balance roundrobin
option httpchk GET /rpc/healthcheck.htm
option log-health-checks
http-check expect status 200
server exch1 2.3.4.5:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-certificates.crt
server exch2 2.3.4.5:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-certificates.crt

the things I’ve tried so far:
acl rpc url_beg /rpc/ -> acl rpc url_beg /rpc/rpcproxy.dll
ssl-default-bind-ciphers added ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES
and to the fe
option accept-invalid-http-request

haven’t taken this to production, tests been made offsite with hosts file, on the FW I have opened 443 to the VIP of haproxy.

Any and all help much appreciated, maybe I’ve been just staring at this for too long and can’t see the forest for the trees…

If You can, I would suggest trying to use https://testconnectivity.microsoft.com/ – this would show at exact step where the connection fails. But since You said this setup is on a test lab, then next thing for You is to turn on debug log of haproxy and also check IIS logs on exchange CAS server it get’s routed to. Could be something as trivial as some wrong port closed or a route missing.

For starters, let me apologize for making a very poor problem description.
At the time I posted this I was very tired and frustrated. The problem explanation itself was lacking almost all of the actual data needed to resolve this.

Let’s start from the beginning.

The Exchange DAG cluster I’m connecting to is a production system.
I’m testing haproxy on it via host files , setting the clusters IPV4 and IPV6 on a host file as not to break anything in production. The cluster has mail.domain.com set for all services, both internally, which I pointed to haproxys VIP when I started testing, and externally with round robin DNS pointing to the public ip addresses of both servers.

This problem is happening only with Outlook 2010 sp2+ , the one that use MAPI not RPC over https, Outlook 2010 w sp1 would connect fine with RPC over https. Outlook 2013+ all connect fine, tried on 3 different OSes, win7,win10 and win server 2008 R2.

Upon further testing, CAS connections work, (as far as I can tell) but DAG won’t connect.
I ran into the problem when trying to connect from an offsite Windows 2008 R2 server with Outlook 2010. Tried another W2K8R2 server with Outlook 2010 from a different site. This led me to falsely believe that the problem was with RPC over https when the actual problem was MAPI connections from Outlook 2010.
Upgraded Outlook on one of the servers to 2013, and it connected right away.

At first I thought it was a cipher problem or windows trying to use tls 1.0 even if 1.2 was enabled
(https://support.quovadisglobal.com/kb/a433/how-to-enable-tls-1_2-on-windows-server-2008-r2.aspx)
Then found the winhttp reg edit, even if tls 1.2 is enabled outlook 2010 might not use it by default (https://blogs.technet.microsoft.com/schrimsher/2016/07/08/enabling-tls-1-1-and-1-2-in-outlook-on-windows-7/)
Did a wireshark capture that showed Outlook was using tls 1.2, tried to connect 3 times (had 3 set as retries in haproxy.cfg) and every time the client would drop the connection. It would go application data-application data, the RST from the client.
Creating a new Outlook profile would do autodiscover fine, the give the outlook can’t connect to exchange server.
Trying to create a profile manually that uses RPC over https won’t work either, Outlook just white screens for a few minutes untill it give the same error.
This could be seen on outlook also if I opened connection status (CTRL, right click), a brief flash of connecting (to the DAG I presume) with auth part showing error, was for a half a second so coudn’t screen shot it.

If I connected directly to the servers w/o haproxy in between everything works fine.

I then installed a W7 and a W10 wks VMs to rule out W2K8R2 specific problems.
Both would connect fine untill I updated Outlooks on the VM to the one that uses MAPI. No matter what version of tls I was using.

On the logs I don’t see any reason for the RST from the client… It won’t show on haproxy logs or exchange mapi logs.

A little update,
downgraded haproxy to 1.8, and Outlook 2010 started working.
So something in 2.0.8 doesn’t like Outlook 2010s :slight_smile:

The Outlook version that I used when getting the problem was 14.0.7237.5000 (32-bit).

Pre mapi Outlook 2010s would connect via RPC fine , apologies don’t have the exact Outlook version number where this problem starts.

I discovered I could configure RPC connection manually with 2.0.8 as long and I entered one of the exchange servers names on the mail server page (server1.domain.com) and in additional setting, connect using http -> I would put the clusters name (mail.domain.com)

I would like to go with 2.0.8 but guess I have to go with 1.8 for now. Anyone have any idea what might have caused this ? I’d be really happy to know…

One issue I’ve found with old (and broken) clients is they wrongly assume headers are case sensitive, which is an expectation that is not valid with HAProxy 2.0.

You could try disabling HTX in both your front end and back end (or defaults):

no option http-use-htx

This makes 2.0 handle thing the same as 1.8 and 1.9.

FYI version 2.1 (dev version) has an option to keep headers in their original case while still using the new HTX mode.

That seems to have done it, thank you very much.
I’ve been stress testing a dev system for a week now. Without no option http-use-htx I would start getting mapi connection errors (client side, the outlook can’t connect to mailbox error) after simulating 600 clients connecting for 8 hours. After adding no option http-use-htx haven’t had problem since.

The funny thing is that testing it small scale, with htx, like from a couple of workstations , everything would work fine, when hitting the servers with more clients, as in my stress test, it would suddenly break and stop connecting to the mailbox.

The header case issue either affects the specific broken client 100% of the time, or not at all. If the issue only shows under high load and hours of testing, it’s probably not about the header case, but caused by a different issue. By disabling HTX you change the code path in haproxy completely, it’s not only about the header case.

That said haproxy 2.0.10 (just released) fixes another bunch of bugs. Also the the h1-case-adjust and
h1-case-adjust-file have been backported from 2.1-dev, so you may want to give it a try with HTX enabled (and if you have doubts about header case, you can use the adjust feature).

Thank you lukastribus.
I did begin suspect it had nothing to do with header cases, but not knowing how haproxy works I wasn’t sure. At first I suspected there was some caching going on and the broken clients would “poison” the cache with broken urls, this from when I put haproxy in front of the production servers over the weekend internally a week ago. (stress testing can’t reliably,consistently simulate the working enviroment hence the test). Mapi connections would still start to fail after around 12-24hrs but for only some clients maybe around 10%. Reverted the DNS changes before monday and ofc announced the test so not to cause too much aggravation with users :slight_smile:
This made me suspect it had something to do with HTX itself not case sensitivity. I upgraded my dev servers to 2.10, haven’t had time to run more that 2 stress tests , but haven’t been able to reproduce the mapi connection error so far. And i should add these without no option http-use-htx…
This is an A+ grade product you got here, I suspect I just need to RTFM way more…

After going through the dev Exchange server logs I found , that the failing connections did not have a sequence cookie on the Exchange servers. With HTX on this happens and so far only with Outlook 2010 using MAPI. Outlook 2013+ work fine, turning HTX off will show the cookie on Exchange logs with Outlook 2010.
In the image below, the first lines are connecting through haproxy, after the red line connecting directly to Exchange.

Did you upgrade to 2.0.10?

Yes I did and the above is using 2.0.10.

Edit:
Apologies, forgot to add:
I can reproduce this without stress testing the system on the dev system. With HTX on, Outlook 2010 will fail every time to connect. If I create an Outlook profile using RPC, it will work the 1st time, as it’s using RPC, but will “upgrade” itself to mapi and on second start it will fail. If I try to create a mapi profile on Outlook 2010 it will set itself up fine but fail to connect.
Doing a stress test on 2.0.8 with HTX on I also managed to get same response from Outlook 2013 client after the stress test once. The production enviroment test did have 2.0.8 also with atleast 20ish Outlook 2013s failing to connect after about 12-24 hrs. Haven’t been able to reproduce that with Outlook 2013 + since in dev running 2.0.10…

If you can reliably reproduce it against 2.0.10, please provide a full packet capture from both frontend and backend traffic. Also full logs would be good.

lukastribus, did you get all the needed info I sent via PM or do you need additional data ?

Thank you good Sir :slight_smile:

Hello jasejk,

I have exactly the same, how did you finally solve the problem?

I ended up with no option http-use-htx.
That has worked for the very very few 2010 users… Tempted to just comment that out and make them upgrade.

edit:
We’re scheduled for Exchange 2019 upgrade this year anyways so Outlook 2010 won’t work anyways. Or will work in a limited fashion, you can read mails, but sending them is a different matter. You mileage might vary tho. Those are just my observations with ex2019 and Outlook 2010

Because we encountered the same problem with Legacy-Outlook on the clients and apparently nobody delivers a working solution so far, although the problem is well known, I searched for documentation on the topic and found it at Microsoft:

I then took all the potentially used header fields - in the notation specified in the document - and inserted them into the default config as h1-case-adjust:

h1-case-adjust accept Accept
h1-case-adjust authorization Authorization
h1-case-adjust authrequired AuthRequired
h1-case-adjust cache-control Cache-Control
h1-case-adjust client-request-id Client-Request-Id
h1-case-adjust connection Connection
h1-case-adjust content-length Content-Length
h1-case-adjust content-type Content-Type
h1-case-adjust cookie Cookie
h1-case-adjust date Date
h1-case-adjust host Host
h1-case-adjust persistent-auth Persistent-Auth
h1-case-adjust pragma Pragma
h1-case-adjust request-header Request-Header
h1-case-adjust response-header Response-Header
h1-case-adjust server Server
h1-case-adjust set-cookie Set-Cookie
h1-case-adjust status-code Status-Code
h1-case-adjust transfer-encoding Transfer-Encoding
h1-case-adjust user-agent User-Agent
h1-case-adjust www-authenticate WWW-Authenticate
h1-case-adjust x-anchormailbox X-AnchorMailbox
h1-case-adjust x-clientapplication X-ClientApplication
h1-case-adjust x-clientInfo X-ClientInfo
h1-case-adjust x-content-type-options X-Content-Type-Options
h1-case-adjust x-deviceinfo X-DeviceInfo
h1-case-adjust x-elapsedtime X-ElapsedTime
h1-case-adjust x-expirationinfo X-ExpirationInfo
h1-case-adjust x-feserver X-FEServer
h1-case-adjust x-mapihttpcapability X-MapiHttpCapability
h1-case-adjust x-pendingperiod X-PendingPeriod
h1-case-adjust x-powered-by X-Powered-By
h1-case-adjust x-requestid X-RequestId
h1-case-adjust x-requesttype X-RequestType
h1-case-adjust x-responsecode X-ResponseCode
h1-case-adjust x-serverapplication X-ServerApplication
h1-case-adjust x-starttime X-StartTime
h1-case-adjust x-user-identity X-User-Identity

in addition to
option h1-case-adjust-bogus-client
in the frontend config.

And - by magic - it works.

2 Likes

Great Stuff, thank you Sir !!

1 Like

lukastribus did offer the solution a while back ago…

That said haproxy 2.0.10 (just released) fixes another bunch of bugs. Also the the h1-case-adjust and
h1-case-adjust-file have been backported from 2.1-dev, so you may want to give it a try with HTX enabled (and if you have doubts about header case, you can use the adjust feature).

I was lazy and didn’t RTFM…

I still do wonder why it does work for 12-24 hrs… Once you get actual client connections it seems to work for a while, then start acting up on 2010 and in a few cases 2013 of Outlook… Something on the Exchange side or just a DNS thing…

Well, lukastribus gave a hint what to do - but I wouldn’t call it “the” solution, since “h1-case-adjust” was already mentioned everywhere when searching for “no option http-use-htx”. No offense intended.

To your problem: it’s probably a DNS thing - but probably a bit different than you might think:
If Outlook is already connected and the client switches to HAPROXY because of DNS, it continues to work despite the header problem. But it is not able to reconnect if the connection is completely disconnected.