HAProxy 2.07 problems with Exchange Server 2016 and Outlook

Help!
1

I’ve been working on setting up HAProxy as a Layer 7 NLB for our Microsoft Exchange 2016 cluster to replace a DNS round-robin (for internal) + firewall random DNAT (external) configuration.

Using CentOS 7, I opted to install the latest available RPM version from the IUS yum repository, which turned out to be HAProxy version 2.07.

Relying on a number of different HOWTO and blog articles, I created a configuration which seemed to be well supported (though based on pre-2.0 versions of HAProxy), using http mode and frontend acls to route traffic to individual backends for each Exchange service with backend health checks for each.

Using local hosts file overrides, I was able to test some clients and determined that everything was working well (OWA, ECP, ActiveSync, etc.) with the exception of Outlook clients, which couldn’t seem to authenticate and kept prompting for credentials or claiming the server was unavailable.

I set up a Layer 4 tcp mode configuration as a test, and everything worked perfectly including Outlook, so I figured the issue was related to something I’d missed with the http configuration.

After perusing Discourse for a bit, I added:

option accept-invalid-http-request to the frontend configuration, and
no option http-use-htx to the default configuration.

That seemed to work; the test Outlook clients were now able to authenticate and no more errors or server disconnects were evident.

I updated the internal DNS to point to the HAPRoxy NLB rather than the Exchange servers directly, and at first everything seemed to be working perfectly.

However, after an hour or so as more and more clients switched over to the NLB (DNS propagation delays) all Outlook clients started receiving “diconnected from server” warnings. Restarting Outlook on any given workstation would result in a successfull connection and an immediate burst of receiving/sending queued mail to/from the server, after which the server would show as disconnected once again.

I was unable to see any error messages in the haproxy.log, but figured there might be issues with connection limits, so I bumped up maxconn quite a bit but it didn’t help.

I switched back to my tcp mode configuration, and despite much lower maxconn settings everything is working flawlessly with no client issues reported.

How can I diagnose my http mode configuration for 2.07? Any help would be appreciated.

I have not messed with my system ulimit, as haproxy -vv indicates epoll is supported. I’ve checked SELinux and it’s not currently running in enforcing mode. Firewall logging shows nothing unexpectedly blocked.

I’ve considered trying a 1.8.x install as a diagnostic step, but wanted to check in here first as I expect I’m still missing something.

1 Like
2

Update: Since I haven’t received any feedback as of yet, I decided to try rolling back to HAProxy v1.8 with a Layer 7 (http mode) configuration. The only change I made to the configuration file I’d been attempting with 2.07 was to comment out the no option http-use-htx directive.

Everything has been running flawlessly for several days now under production load.

Any thoughts re: what might be going on with 2.07? Would posting my configuration file be helpful?

3

Hi,

i will not help you, but did you make it works even with v1.8 with outlook login ?

I faced outlook keeping prompting for auth…

4

Yes, we have haproxy 1.8 working fine for (previously; they’re now all upgraded) Outlook 2010 SP3 and Outlook 2016 clients.

1 Like
5

Same issue here. With v2.0 the /mapi requests didn’t work.

After a downgrade to v1.8 everything runs like a charm (same config).

6

Hi,

We have same issue with Haproxy since version 2.0.0 on Ubuntu Xenial (try with Bionic also). We use package from this source (https://haproxy.debian.net/).

Since 2.0.0-1 package (try with 2.0.10-1 until 2.1.13-1) outlook 2013, 2016 connectivity (Mapi over HTTP) doesn’t work any more. We use Exchange 2016 with last update.

Here is the resultat of haproxy --v

HA-Proxy version 2.0.13-1ppa1~xenial 2020/02/15 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=1).
Built with OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.1
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.21 2016-01-12
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE     mux=H2
              h2 : mode=HTTP       side=FE        mux=H2
       <default> : mode=HTX        side=FE|BE     mux=H1
       <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS

Available services :
        prometheus-exporter

Available filters :
        [SPOE] spoe
        [COMP] compression
        [CACHE] cache
        [TRACE] trace

We use this backend (for the mapi part)

backend bk_Exchange2016_servename_mapi
        mode http
        option http-keep-alive
        option prefer-last-server
        no option httpclose
        no option http-server-close
        no option forceclose
        no option http-tunnel
        option forwardfor
        option httpchk GET /mapi/HealthCheck.htm
        http-check expect string 200\ OK
        cookie ALBWA insert indirect nocache
        timeout server 600s
        server SERVERNAME X.X.X.X:443 ssl check

When Outlook try to connect, we can observe in haproxy.log :

Mar 24 15:33:34 localhost haproxy[10109]: X.X.X.X:51959 [24/Mar/2020:15:33:34.617] Web~ bk_Exchange2016_servenamemapi/SERVERNAME 43/0/0/3/+46 401 +728 - - --NI 177/8/1/1/0 0/0 {|Microsoft Office/16.0 (Windows NT 10.0; Microsoft Outlook 16.0.4|servername.fr {TLSv1.2/ECDHE-RSA-AES128-GCM-SHA256/servername.fr //C=US/O=xxxx Inc/OU=xxx/CN=xxx RSA CA XXXX/} "POST /mapi/nspi/?MailboxId=6ba0e895-0c9b-40e7-be3a-e6cf85f3d44a@servername.fr HTTP/1.1"

If we downgrade in 1.9.14 version no problem, outlook connects correctly. We see in this case “NI” in log.

Thank you in advance for your help.

7

Did you guys all read @nlindq post in its entirety, especially the part with:

  • no option http-use-htx to the default configuration

This should help with buggy HTTP implementations on both client as well as server side.

8

Hi,

Thanks for you reply.

If add no option http-use-htx, my configuration is not correct

http frontend ‘Web’ (/etc/haproxy/haproxy.cfg:68) tries to use http backend ‘xxxx_backend’ (/etc/haproxy/haproxy.cfg:11620) in a ‘use_backend’ rule, both of which disagree on 'option http-use-htx

Update : I had disable the backend (I don’t understand what is causing the problem on this backend). and connection work ! Thanks.

but the objective would be to leave HTX active by default but to revenig in legacy mode only on the backend, but by doing this I have a configuration error

http frontend ‘Web’ (/etc/haproxy/haproxy.cfg:68) tries to use http backend ‘bk_Exchange2016_servername_mapi’ (/etc/haproxy/haproxy.cfg:3596) in a ‘use_backend’ rule, both of which disagree on ‘option http-use-htx’.

9

Right, you have to configure it consistently.

Use the default configuration for that, and remove individual configurations of it in more specific sections.

10

Yes Lukas, it’s good, thanks.I would like to disable HTX only on Exchange MAPI backend. The aim is to keep this feature for all other backend.

Think it’s possible ?

Thanks

11

Sure, but the frontend that actually uses that backend must have the option also.

12

Another update: due to some indications in other threads that issues may have been resolved post 2.0.10, I tried upgrading from my functioning 1.8.x setup to 2.0.17. Unfortunately within 10 minutes the same symptoms showed up–Outlook unable to connect to Exchange; rebooting clears the symptom for any given client for a short time, but then the problem recurs.

I rolled back to 1.8.26 and everything is working fine again.

Is there anything else I might look at for required configuration differences introduced in 2.0+?