Unable to start HAPROXY 2.2

Hi, I am uanble to start haproxy with 2.2 version,

We were earlier running 1.7.9 version and planned to upgrade haproxy to latest version 2.2, so therefore we are using the same configuration 1.7.9 file to the latest version of haproxy as well.

Haproxy 2.2 is unable to start with the following errors:

[WARNING] 300/184709 (2527) : parsing [/var/vcap/jobs/haproxy/config/haproxy.config:548] : a 'http-request' rule placed after a 'use_backend' rule will still be processed before.
[ALERT] 300/184709 (2527) : Error(s) found in configuration file : /var/vcap/jobs/haproxy/config/haproxy.config
[ALERT] 300/184709 (2527) : Fatal errors found in configuration.
Tue Oct 27 18:47:09 UTC 2020: Errored starting HAProxy

Not sure whats happening with the error log
haproxy -vv

HA-Proxy version 2.2.1 2020/07/23 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.2.1.html
Running on: Linux 4.15.0-117-generic #118~16.04.1-Ubuntu SMP Sat Sep 5 23:35:06 UTC 2020 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initial

izers -Wtype-limits
  OPTIONS = USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE +STATIC_PCRE -STATIC_PCRE2 +TPROXY +LI

NUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -

OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents =
Built with multi-threading support (MAX_THREADS=64, default=1).
Built with OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with network namespace support.
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.38 2015-11-23
Running on PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 5.4.0 20160609

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
            fcgi : mode=HTTP       side=BE        mux=FCGI
       <default> : mode=HTTP       side=FE|BE     mux=H1
              h2 : mode=HTTP       side=FE|BE     mux=H2
       <default> : mode=TCP        side=FE|BE     mux=PASS

Available services : none

Available filters :
        [SPOE] spoe
        [COMP] compression
        [TRACE] trace
        [CACHE] cache
        [FCGI] fcgi-app

From 1.7 to 2.2 is a large jump, can you elaborate how you upgraded haproxy?

Because the startup scripts are totally different, 1.7 uses a systemd wrapper while this is totally different on 2.2. So it needs a clean removal of all files, including the systemd wrapper and old init.d files.

What Ubuntu release is this?

Please share the output of systemctl status haproxy, journalctl -u haproxy. Most importantly, share the output of haproxy -c -f /var/vcap/jobs/haproxy/config/haproxy.config.

Hi lukas,

Thanks for reaching out.
So I have downloaded the HAPROXY source code from https://www.haproxy.org/download/2.2/src/
and for the startup process we are using below startup script
echo “Extracting pcre…”
tar xzf haproxy/pcre-8.38.tar.gz
cd pcre-8.38
./configure
make
make install
cd …

tar xzf haproxy/haproxy-2.2.1.tar.gz
cd haproxy-2.2.1
make TARGET=linux-glibc USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_ZLIB=1 USE_PCRE_JIT=1
mkdir ${BOSH_INSTALL_TARGET}/bin
cp haproxy ${BOSH_INSTALL_TARGET}/bin/
chmod 755 ${BOSH_INSTALL_TARGET}/bin/haproxy

Ubuntu_ release:
lsb_release -a

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

We are not running HAPROXY as a system service.

Output of haproxy -c -f /var/vcap/jobs/haproxy/config/haproxy.config
haproxy -c -f /var/vcap/jobs/haproxy/config/haproxy.config
acessed before.
[WARNING] 300/205411 (6606) : parsing [/var/vcap/jobs/haproxy/config/haproxy.config:701]: ‘option httpchk’ : hiding headers or body at the end of the version string
is deprecated. Please, consider to use ‘http-check send’ directive instead.
[ALERT] 300/205411 (6606) : Error(s) found in configuration file : /var/vcap/jobs/haproxy/config/haproxy.config
[ALERT] 300/205411 (6606) : Fatal errors found in configuration

HAPROXY config folder:
-rw-r----- 1 root vcap 12122 Oct 27 20:49 certs.ttar
drwxr-x— 2 root vcap 4096 Oct 27 20:49 errorfiles/
-rw-r----- 1 root vcap 61162 Oct 27 20:49 haproxy.config
-rw-r----- 1 root vcap 23838 Oct 27 20:49 ip_denyList.lst
-rw-r----- 1 root vcap 1222 Oct 27 20:49 ip_dmproxyAllowList.lst
drwxr-xr-x 2 root root 4096 Oct 27 20:49 ssl/
-rw-r----- 1 root vcap 1 Oct 27 20:49 ssl_redirect.map

please let me know if you need anything.

Thanks,
NIRISH M

Please always use the latest stable version, for 2.2 this is 2.2.4.

You will have to share your configuration, I don’t know how to help you otherwise.

Here is the configuration that we use

  global
    log 127.0.0.1 local1 info
    daemon
    user <value>
    group <value>
    maxconn 64000
    spread-checks 4
    tune.ssl.default-dh-param 2048
    ssl-default-bind-options force-tlsv12
    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

defaults
    maxconn 64000
    option http-server-close
    option httplog
    option forwardfor
    option contstats
    timeout connect         5000ms
    timeout client          30000ms
    timeout server          30000ms
    timeout tunnel          3600000ms
    timeout http-keep-alive 1000ms
    timeout http-request    30000ms
    timeout queue           30000ms
    errorfile 504 config/errorfiles/504.http

##########################################################################
# Frontends
##########################################################################

frontend http-in
    mode http
    bind :80
    log global
    log-format %ci\ %fi:%fp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %{+Q}r

    tcp-request connection reject if { src -f config/ip_denyList.lst }

    acl xfp_exists hdr_cnt(X-Forwarded-Proto) gt 0
    reqadd X-Forwarded-Proto:\ http if ! xfp_exists

    http-request redirect scheme https code 301

frontend https-in
    mode http
    bind :443 ssl crt config/ssl/server_cert.pem ca-file config/ssl/client_auth_cert.pem verify optional
    log global
    log-format %ci\ %fi:%fp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %{+Q}r

    tcp-request connection reject if { src -f config/ip_denylist.lst }

    
    
    # checks for "|" in certificate common name - if present it is no old user cert
    acl no_user_cert ssl_c_s_dn(cn) -m reg -i .*\|+

    http-request set-header ssl_client_user         %{+Q}[ssl_c_s_dn(cn)]
    http-request set-header ssl_client_cert         %{+Q}[ssl_c_der,base64]
    http-request set-header ssl_client_cert_used    %[ssl_c_used] if no_user_cert

    acl xfp_exists hdr_cnt(X-Forwarded-Proto) gt 0
    reqadd X-Forwarded-Proto:\ https if ! xfp_exists

    # use HSTS
    http-response set-header Strict-Transport-Security max-age=16000000;\ includeSubDomains;\ preload;

    # routing to IoT Services Cockpit


    http-request redirect location /URI/cockpit/ if { path /URI }
    http-request redirect location /URI/cockpit/ if { path /URI }
    http-request redirect location /URI/cockpit/ if { path /URI/ }
    http-request redirect location /URI/cockpit/ if { path /URI/ }
    http-request redirect location /URI/cockpit/ if { path /URI/cockpit }

    use_backend http-backend-tomcat if { path_reg ^\/.*?/URI/cockpit/.* }

    # we also allow paths without platform-tenant-id currently, add it automatically in the backend
    http-request redirect location /URI/cockpit/ if { path / }
    http-request redirect location /URI/cockpit/ if { path /URI }
    http-request redirect location /URI/cockpit/ if { path /URI/ }
    http-request redirect location /iot/cockpit/ if { path /URI/cockpit }

    use_backend http-backend-tomcat-no-ptenantid if { path_beg /URI/cockpit/ }


    # redirecting paths for Swagger API documenation of IoT Core to correct location

    acl api_unversioned           path /URI/api
    acl api_unversioned_slash     path /URI/api/
    acl api_v1                    path /URI/api/v1
    acl api_v1_slash              path /URI/api/v1/
    acl api_doc_unversioned       path /URI/api/doc
    acl api_doc_unversioned_slash path /URI/api/doc/

    
    http-request redirect location /URI/api/v1/doc if api_unversioned || api_unversioned_slash || api_v1 || api_v1_slash || api_doc_unversioned || api_doc_unversioned_slash

    # routing to IoT Spring Core
    use_backend http-backend-core if { path_reg ^\/.*?/URI/core/api/.* }
    # we also allow paths without platform-tenant-id currently, add it automatically in the backend
    use_backend http-backend-core if { path_beg /URI/core/api }
    # redirect legacy gateway sync API
    use_backend http-backend-core if { path_beg /URI/core/GatewaySync }
	
    # redirecting paths for Swagger API documenation of ELMO to correct location
    acl api_elm_unversioned           path /URI/api
    acl api_elm_unversioned_slash     path /URI/api/
    acl api_elm_v1                    path /URI/api/v1
    acl api_elm_v1_slash              path /URI/api/v1/
    acl api_elm_doc_unversioned       path /URI/api/doc
    acl api_elm_doc_unversioned_slash path /URI/api/doc/
    http-request redirect location /URI/api/v1/doc if api_elm_unversioned || api_elm_unversioned_slash || api_elm_v1 || api_elm_v1_slash || api_elm_doc_unversioned || api_elm_doc_unversioned_slash
    # routing to ELMO
    use_backend http-backend-elmo if { path_reg ^\/.*?/URI/elm/api/.* }
	# we also allow paths without platform-tenant-id currently, add it automatically in the backend
    use_backend http-backend-elmo if { path_beg /URI/elm/api }

    # redirecting paths for Swagger API documenation of IoT Processing to correct location
    acl processing_api_unversioned           path /URI/api
    acl processing_api_unversioned_slash     path /URI/api/
    acl processing_api_v1                    path /URI/api/v1
    acl processing_api_v1_slash              path /URI/api/v1/
    acl processing_api_doc_unversioned       path /URI/api/doc
    acl processing_api_doc_unversioned_slash path /URI/api/doc/

    
    http-request redirect location /URI/api/v1/doc if processing_api_unversioned || processing_api_unversioned_slash || processing_api_v1 || processing_api_v1_slash || processing_api_doc_unversioned || processing_api_doc_unversioned_slash

    # routing to IoT Processing
    use_backend http-backend-processing if { path_beg /URI/api }
	
    # routing to IoT Gateway REST
    use_backend http-backend-gateway-rest if { path_beg /URI/rest }

    default_backend error-backend-404

frontend tcp-frontend-gateway-mqtt-tls
    mode tcp
    bind :<port>
    log global
    log-format %ci\ %fi:%fp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq

    tcp-request connection reject if { src -f config/ip_denylist.lst }

    

    default_backend tcp-backend-gateway-mqtt-tls

frontend tcp-frontend-mms-nio-tls
    mode tcp
    bind :<port>
    log global
    log-format %ci\ %fi:%fp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq

    tcp-request connection reject if { src -f config/ip_denylist.lst }

    

    default_backend tcp-backend-mms-nio-tls

######################################################
# Frontends for connections
######################################################


frontend tcp-frontend-mms-nio
    mode tcp
    bind :<port>
    log-format %ci\ %fi:%fp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq

    tcp-request connection reject if { src -f config/ip_denylist.lst }

    default_backend tcp-backend-mms-nio



frontend rest-frontend-core-spring
    mode http
    bind :<port>
    log-format %ci\ %fi:%fp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %{+Q}r

    tcp-request connection reject if { src -f config/ip_denylist.lst }

    default_backend http-backend-core



frontend rest-frontend-elmo
    mode http
    bind :<port>
    log-format %ci\ %fi:%fp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %{+Q}r

    tcp-request connection reject if { src -f config/ip_denylist.lst }

    default_backend http-backend-elmo



frontend rest-frontend-processing
    mode http
    bind :<port>
    log-format %ci\ %fi:%fp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %{+Q}r

    tcp-request connection reject if { src -f config/ip_denylist.lst }

    default_backend http-backend-processing


# We need the resolvers section here, as HA-Proxy caches DNS entries. Pointing to BOSH DNS running locally forces HA-Proxy to retry DNS resolving
resolvers bosh-dns
    nameserver localhost-bosh-dns <Ipaddress>:53

##########################
# HTTP Backends 
##########################

backend http-backend-tomcat
    mode http
    log global
    balance roundrobin


    # replace "/<platform-tenant-id>/URI/cockpit" with "/cockpit" at the beginning of any request path. Cockpit runs internally on context /cockpit.
    http-request set-path %[path,regsub(^/.*?/URI/cockpit,/cockpit)]

    # rewrite the path from internal /cockpit to outward-facing /<platform-tenant-id>/URI/cockpit.
    # if we wouldn't do this, the browser could not use these cookies.
    http-response replace-header Set-Cookie (.*)Path=/cockpit(.*) \1Path=/URI/cockpit\2
    http-response replace-header Location /cockpit/ /URI/cockpit/

    # add cookie for session persistence
    cookie COCKPITID insert indirect nocache httponly


    server cockpit-node-0 <Ipaddress>:<port> resolvers bosh-dns cookie cockpit-node-0 check inter 1000




# we need an additional backend for calls against Tomcat where the platform-tenant-id is NOT part of the path:
# we have to rewrite the Set-Cookie header replacing internal /cockpit with the correct outward-facing path.
# unfortunately http-response has no means to access the request path. therefore the decision has already to be taken
# in the frontend. this means we target a different backend.
backend http-backend-tomcat-no-ptenantid
    mode http
    log global
    balance roundrobin

    # replace "/URI/cockpit" with "/cockpit" at the beginning of any request path. Cockpit runs internally on context /cockpit.
    http-request set-path %[path,regsub(^/URI/cockpit,/cockpit)]

    # rewrite the path from internal /cockpit to outward-facing /URI/cockpit.
    # if we wouldn't do this, the browser could not use these cookies.
    http-response replace-header Set-Cookie (.*)Path=/cockpit(.*) \1Path=/URI/cockpit\2
    http-response replace-header Location /cockpit/ /URI/cockpit/


    server cockpit-node-0 <Ipaddress>:<port> resolvers bosh-dns check inter 1000



backend http-backend-core
    mode http
    log global
    balance roundrobin



    # we also allow paths without platform-tenant-id currently
    # if request enters without platform-tenant-id, enhance path to contain this
    http-request set-path %[path,regsub(^/URI/core/api/,/URI/api/)]

    # support legacy gateway sync API: add platform tenant and /api/v1 to ensure, that the correct controller is hit
    http-request set-path %[path,regsub(^/URI/core/GatewaySync/,/URI/api/v1/GatewaySync/)]

    # add cookie for session persistence
    cookie SERVERID insert indirect nocache httponly

    # health check: as soon as /URI/core/api/v1/about contains "startupTimestamp", Core server is operational
    option httpchk GET /URI/api/v1/about HTTP/1.0\r\nAuthorization:\ Basic\ cm9vdDo4R2psaERlZDhSMHFSWnA=
    http-check expect string startupTimestamp


    server core-node-0 <Ipaddress>:<port> resolvers bosh-dns cookie core-node-0 check inter 1000



#

backend http-backend-elm
    mode http
    log global
    balance roundrobin

    # we also allow paths without platform-tenant-id currently
    # if request enters without platform-tenant-id, enhance path to contain this
    http-request set-path %[path,regsub(^/URI/elm/api/,/URI/elm/api/)]
	
    # add cookie for session persistence
    cookie SERVERID insert indirect nocache httponly


    server elmo-node-0 <server-Ipaddress>:<port-5> resolvers bosh-dns cookie elmo-node-0 check inter 1000




backend http-backend-processing
    mode http
    log global
    balance roundrobin


    server processing-node-0 <server-Ipaddress>:port-2 resolvers bosh-dns check inter 1000



backend http-backend-gateway-rest
    mode http
    log global
    balance roundrobin

    # replace "/URI/gateway/rest/" with "/" at the beginning of any request path. Gateway endpoints run internally on root context.
    http-request set-path %[path,regsub(/URI/gateway/rest/,/)]


    server gw-rest-node-0 <Ipaddress>:<port> resolvers bosh-dns check inter 1000


########################################################
# TCP-Backends
########################################################

backend tcp-backend-gateway-mqtt-tls
    mode tcp
    log global
    balance leastconn


    server gw-mqtt-node-0 <Ipaddress>:<port> resolvers bosh-dns check inter 1000


backend tcp-backend-mms-nio-tls
    mode tcp
    log global
    balance leastconn


    # health check: as MMS is not operational without Core, we check here for availability of Core too.
    # For details check: IOTP-5965, IOTP-6114, IOTP-6394
    option tcp-check
    tcp-check connect port 61620
    tcp-check connect port 8081
    tcp-check send GET\ /URI/api/v1/about\ HTTP/1.0\r\nAuthorization:\ Basic\ cm9vdDo4R2psaERlZDhSMHFSWnA=\r\n\r\n
    tcp-check expect string startupTimestamp



    server mms-node-0 <Ipaddress>:<port> resolvers bosh-dns check inter 1000


#################################################################
# Backends 
#################################################################

backend tcp-backend-mms-nio
    mode tcp
    log global
    balance leastconn

    # extended health check to Core not needed as this is only the internal interface.
    # enabling the extended health check would result in a deadlock during startup anyways, as Core will not start without Messaging


    server mms-node-0 <Ipaddress>:<port> resolvers bosh-dns check inter 1000


backend error-backend-404
    mode http
    errorfile 503 config/errorfiles/404.http

please let me know if this helps.

Hello,

Checking your config file i have found multiple errors.

your deny list is pointing to 2 different files (config/ip_denyList.lst and config/ip_denylist.lst). Is it intended ?

Also, you have a typo in one of your backend name (backend http-backend-elm) A “o” is missing i think.

You also need to correct the deprecated config like reqadd.

After correcting these errors, i was able to pass a config check without error, so haproxy should be able to start.

Hope it will help :slight_smile:

1 Like