Inconsistent ACL resolution resulting in incorrect backend selection

Help!
1

I have a single frontend that I would like to send to two different backends based on the requested url.

Example Frontend Config

frontend example_https
bind 192.168.1.5:443 ssl crt /etc/ssl/private/certname.pem alpn h2,http/1.1
acl is_jim path -i /jim.asmx
option forwardfor
option http-server-close
use_backend jim_webservers_https if is_jim
default_backend bob_webservers_https

With the config above if I request www.example.com/jim.asmx, my requests are sometimes successful. By looking in the haproxy.logs, I can see that for the failed requests the wrong (default) backend is being selected.

I have tried various different approaches eg using path/path_beg/path_end but always appear to encounter inconsistent behaviour. I tends to either work/not work in a particular browser session until I change it/reopen/change browser. That said this pattern is not consistent, all the while haproxy.log shows the same uri being requested and being routed to different backends.

This would suggest my ACL is not being matched correctly
Any thoughts on what to try?

2

Can you share the complete configuration, the actual log line and the output of haproxy -vv please.

3

Thanks for the reply, please find output below

HA-Proxy version 1.8.8-1ubuntu0.3 2019/01/11
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -g -O2 -fdebug-prefix-map=/build/haproxy-_XfIfg/haproxy-1.8.8=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0g  2 Nov 2017
Running on OpenSSL version : OpenSSL 1.1.0g  2 Nov 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : yes
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
        [SPOE] spoe
        [COMP] compression
        [TRACE] trace

Below is an example of requests from the haproxy log, the same frontend is being used with the same page requested each time

[04/Mar/2019:10:37:13.146] vip7_https~ lead_webservers_https/web03 0/0/1/101/102 200 3466 - - ---- 1/1/0/0/0 0/0 "GET /Leads.asmx HTTP/1.1"
[04/Mar/2019:10:37:18.987] vip7_https~ lead_webservers_https/web04 0/0/0/124/124 200 3466 - - ---- 1/1/0/0/0 0/0 "GET /Leads.asmx HTTP/1.1"
[04/Mar/2019:10:37:20.756] vip7_https~ lead_webservers_https/web03 0/0/1/3/4 200 3466 - - ---- 1/1/0/0/0 0/0 "GET /Leads.asmx HTTP/1.1"
[04/Mar/2019:10:37:30.329] vip7_https~ vip7_webservers_https/web08 0/0/1/2/3 404 2210 - - ---- 1/1/0/0/0 0/0 "GET /Leads.asmx HTTP/1.1"
[04/Mar/2019:10:37:33.742] vip7_https~ vip7_webservers_https/web08 0/0/1/2/3 404 2210 - - ---- 1/1/0/0/0 0/0 "GET /Leads.asmx HTTP/1.1"
[04/Mar/2019:10:37:36.836] vip7_https~ vip7_webservers_https/web09 0/0/0/3/3 404 2210 - - ---- 1/1/0/0/0 0/0 "GET /Leads.asmx HTTP/1.1"
[04/Mar/2019:10:37:37.499] vip7_https~ vip7_webservers_https/web08 0/0/1/2/3 404 2210 - - ---- 1/1/0/0/0 0/0 "GET /Leads.asmx HTTP/1.1"
[04/Mar/2019:10:37:38.143] vip7_https~ vip7_webservers_https/web09 0/0/0/2/2 404 2210 - - ---- 1/1/0/0/0 0/0 "GET /Leads.asmx HTTP/1.1"

Below is the frontend and the non default backend

frontend vip7_https
    bind redacted:443  ssl crt /etc/ssl/private/redacted.pem alpn h2,http/1.1
    acl is_leads path -i /Leads.asmx
    acl is_leads_subdir path -i /LeadServiceMS/Leads.asmx
    option forwardfor
    option http-server-close
    use_backend lead_webservers_https if is_leads is_leads_subdir
    default_backend vip7_webservers_https

backend lead_webservers_https
    mode http
    balance roundrobin
    option http-server-close
    http-request set-header X-Forwarded-Proto https if  { ssl_fc }
    acl is_leads_subdir path_sub -i LeadServiceMS
    reqrep ^([^\ :]*)\ /(.*)     \1\ /LeadServiceMS/\2 if !is_leads_subdir
    server      web03 redacted:443 check weight 1 ssl verify none
    server      web04 redacted:443 check weight 1 ssl verify none

Any help is full appreciated.

4

Did you try swapping order?
Also, try building a new backend for the other instead of relying on default?
I find when I want specific, I need to be obtusely specific. Worth a shot :slight_smile:

5

The entire configuration is needed. Including global section, default section and especially the vip7_webservers_https backend.

Also double and triple check that you don’t have any obsolete, old haproxy instances running in the background. If you want to be sure, restart haproxy with noreuseport enabled.

6

Thank you both for your feedback. As it turns out I had loads of old HAproxy instances running in the background. After killing these off my rules were followed correctly.

To avoid these old instances, what are my options?
Is there an appreciable outage when one uses noreuseport?
Below is my global/default config for sockets/expose-fd listeners

 global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    # An alternative list with additional directives can be obtained from
    #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3 no-tls-tickets
    tune.ssl.default-dh-param 2048

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

Thanks again for your guidance

7

There reasons for instances still running in the background can be multiple; most likely reasons are issues and bugs in the process handling.

I’d suggest you upgrade to 1.8.19 first of all. You can do that on Ubuntu by using Vincent Bernat’s PPA:

https://launchpad.net/~vbernat/+archive/ubuntu/haproxy-1.8/

I’m not sure noreuseport really fixes this, it does guarantee though that a restart would fail if the ports are still bound by an old instance.

I’d therefor suggesting to switch to the PPA for the current release and monitor the situation.

8

Thanks for that, I’ll test this out and give it a go.

Cheers