Haproxy with pacemaker, haproxy fails to start after adding floating IP

Help!
1

I have a 2 node test setup with pacemaker 1.1.15 and haproxy 1.5.18 running on cento 7. I have a running cluster with 2 floating IP address and the haproxy service is running as a clone. I need to add a new floating IP address and make changes to the haproxy.cfg for a new site. So far in my testing, after I add the new IP address and set the pacemaker constraint, the haproxy service fails to start. This takes all of the sites down. Once I restart pacemaker on both nodes everything comes back up again. How can I prevent the haproxy service from failing? The IP address I am going to add is the 10.x.x.189. You will see it has already been added in the below file outputs.

I have copied and pastes in several files. haproxy.cfg, then output from pcs status and lastly the pcs config output.

 [root@t-haproxylb3 haproxy]# cat haproxy.cfg
peers QAHAproxypeers
        peer t-haproxylb3 10.x.x.185:1024
        peer t-haproxylb4 10.x.x.186:1024

global
        log     127.0.0.1 local0
#       log /dev/log local0
#       log /dev/log local1 notice
        chroot /var/lib/haproxy
        stats socket /var/lib/haproxy/stats
        stats timeout 30s
        tune.ssl.default-dh-param 2048
        user haproxy
        group haproxy
        daemon

defaults
        log global
        mode http
        option httplog
        option dontlognull
        option redispatch
        option forwardfor
        option http-server-close
        maxconn 5000
        timeout connect 5s
        timeout client 5h
        timeout server 5h
        timeout queue 30s
        timeout http-request 5s
        timeout http-keep-alive 15s

listen stats *:1936
        mode http
        stats enable
        stats hide-version
        stats realm Haproxy\ Statistics
        stats uri /haproxy_stats
        stats auth admin:password
        stats admin if TRUE

frontend http_in
        bind *:80
        ###Add new acl and use_backend entry for each new site
        ###new backend sections will be needed as well
        acl is_clients hdr(host) -i clients.qa.racingcars.com
        acl is_apps hdr(host) -i apps.qa.racingcars.com
        acl is_dad hdr(host) -i dad.qa.racingcars.com
        acl is_scripting hdr(host) -i scripting.qa.racingcars.com
        acl is_racingcarsnet hdr_end(host) -i racingcars.net
        use_backend http_client if is_clients
        use_backend http_apps if is_apps
        use_backend http_dad if is_dad
        use_backend http_scripting if is_scripting
        use_backend http_racingcarsnet if is_racingcarsnet
        option forwardfor
        option http-server-close

frontend https_in_ssl_apps
        bind 10.x.x.187:443 ssl crt /etc/ssl/private/apps.racingcars.com.pem
        mode http
        use_backend https_ssl_apps
        option forwardfor
        option http-server-close

frontend https_in_ssl_clients
        bind 10.x.x.188:443 ssl crt /etc/ssl/private/clients.racingcars.com.pem
        mode http
        use_backend https_ssl_clients
        option forwardfor
        option http-server-close

frontend https_in_ssl_scripting
        bind 10.x.x.189:443 ssl crt /etc/ssl/private/clients.racingcars.com.pem
        mode http
        use_backend https_ssl_scripting
        option forwardfor
        option http-server-close

frontend https_in_ssl
        mode http
#       bind *:443 ssl crt /etc/ssl/private/ no-sslv3
        bind *:443 ssl crt /etc/ssl/private/
        reqadd X-Forwarded-Proto:\ https
####### commented out below to enable https pass-through for apps
#       use_backend http_clients if { ssl_fc_sni clients.racingcars.com }
#       use_backend http_apps if { ssl_fc_sni apps.racingcars.com }

        acl is_ssl_racingcarsnet hdr_end(host) -i racingcars.net
        use_backend http_racingcarsnet if is_ssl_racingcarsnet

backend http_clients
        balance source
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers QAHAproxypeers
        stick on src
        ###This site does not use host header - only the page name is needed###
#       option httpchk HEAD /Default.aspx
        ###Added host header so haproxy can route around NLB - use below for checking###
        option httpchk HEAD /Default.aspx HTTP/1.1\r\nHost:\ clients.qa.racingcars.com
        server websvr03 10.x.x.183:80 cookie web3 weight 5 check
#        server websvr04 10.x.x.118:80 cookie web4 weight 5 check

backend https_ssl_clients
        balance source
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers QAHAproxypeers
        stick on src
        option httpchk HEAD /Default.aspx HTTP/1.1\r\nHost:\ clients.qa.racingcars.com
        server websvr03 10.x.x.183:443 cookie web3 weight 5 check ssl verify none
#       server websvr04 10.x.x.118:443 cookie web4 weight 5 check ssl verify none

backend http_apps
        balance roundrobin
        stick-table type ip size 1m expire 6h peers QAHAproxypeers
        stick on src
        ###This site uses host headers so this type of check is required###
        option httpchk HEAD /default.htm HTTP/1.1\r\nHost:\ apps.qa.racingcars.com
        server websvr03 10.x.x.182:80 cookie web3 weight 5 check
#        server websvr04 10.x.x.116:80 cookie web4 weight 5 check

backend https_ssl_apps
        balance roundrobin
        stick-table type ip size 1m expire 6h peers QAHAproxypeers
        stick on src
        ###This site uses host headers so this type of check is required###
        option httpchk HEAD /default.htm HTTP/1.1\r\nHost:\ apps.qa.racingcars.com
        server websvr03 10.x.x.182:443 cookie web3 weight 5 check ssl verify none
#        server websvr04 10.x.x.116:443 cookie web4 weight 5 check ssl verify none

backend http_dad
        balance roundrobin
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers QAHAproxypeers
        stick on src
        ###This site does not use host header - only the page name is needed###
        option httpchk HEAD /login.aspx HTTP/1.1\r\nHost:\ dad.qa.racingcars.com
        server websvr03 10.x.x.182:80 cookie web3 weight 5 check
#        server websvr04 10.x.x.116:80 cookie web4 weight 5 check

backend http_scripting
        balance roundrobin
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers QAHAproxypeers
        stick on src
        ###This site uses host header so this type of check is required###
        option httpchk HEAD /default.aspx HTTP/1.1\r\nHost:\ scripting.qa.racingcars.com
        server websvr03 10.x.x.184:80 cookie web3 weight 5 check
#        server websvr04 10.x.x.116:80 cookie web4 weight 5 check

backend https_ssl_scripting
        balance source
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers QAHAproxypeers
        stick on src
        option httpchk HEAD /Default.aspx HTTP/1.1\r\nHost:\ scripting.qa.racingcars.com
        server websvr03 10.x.x.184:443 cookie web3 weight 5 check ssl verify none
#       server websvr04 10.x.x.118:443 cookie web4 weight 5 check ssl verify none

backend http_racingcarsnet
        balance roundrobin
        cookie SRV_ID prefix
        stick-table type ip size 1m expire 6h peers QAHAproxypeers
        stick on src
        ###This site uses host header so this type of check is required###
        option httpchk HEAD /default.aspx HTTP/1.1\r\nHost:\ test.racingcars.net
#        server websvr03 10.x.x.115:80 cookie web3 weight 5 check
#        server websvr04 10.x.x.117:80 cookie web4 weight 5 check



[root@t-haproxylb3 haproxy]# pcs status
Cluster name: testcluster2
Stack: corosync
Current DC: t-haproxylb3 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum
Last updated: Tue Dec 20 16:55:37 2016          Last change: Tue Dec 20 14:15:59 2016 by root via cibadmin on t-haproxylb3

2 nodes and 5 resources configured

Online: [ t-haproxylb3 t-haproxylb4 ]

Full list of resources:

 Clone Set: haproxy-clone [haproxy]
     Started: [ t-haproxylb3 ]
     Stopped: [ t-haproxylb4 ]
 floatIP_187    (ocf::heartbeat:IPaddr2):       Started t-haproxylb3
 floatIP_188    (ocf::heartbeat:IPaddr2):       Started t-haproxylb3
 floatIP_189    (ocf::heartbeat:IPaddr2):       Started t-haproxylb3

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

[root@t-haproxylb3 haproxy]# pcs config
Cluster Name: testcluster2
Corosync Nodes:
t-haproxylb3 t-haproxylb4
Pacemaker Nodes:
t-haproxylb3 t-haproxylb4

Resources:
Clone: haproxy-clone
Resource: haproxy (class=systemd type=haproxy)
Operations: monitor interval=10s (haproxy-monitor-interval-10s)
Resource: floatIP_187 (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=10.x.x.187 cidr_netmask=32
Operations: start interval=0s timeout=20s (floatIP_187-start-interval-0s)
stop interval=0s timeout=20s (floatIP_187-stop-interval-0s)
monitor interval=30s (floatIP_187-monitor-interval-30s)
Resource: floatIP_188 (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=10.x.x.188 cidr_netmask=32
Operations: start interval=0s timeout=20s (floatIP_188-start-interval-0s)
stop interval=0s timeout=20s (floatIP_188-stop-interval-0s)
monitor interval=30s (floatIP_188-monitor-interval-30s)
Resource: floatIP_189 (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=10.x.x.189 cidr_netmask=32
Operations: start interval=0s timeout=20s (floatIP_189-start-interval-0s)
stop interval=0s timeout=20s (floatIP_189-stop-interval-0s)
monitor interval=30s (floatIP_189-monitor-interval-30s)

Stonith Devices:
Fencing Levels:

Location Constraints:
Ordering Constraints:
Colocation Constraints:
haproxy-clone with floatIP_187 (score:INFINITY) (id:colocation-haproxy-clone-f loatIP_187-INFINITY)
haproxy-clone with floatIP_188 (score:INFINITY) (id:colocation-haproxy-clone-f loatIP_188-INFINITY)
haproxy-clone with floatIP_189 (score:INFINITY) (id:colocation-haproxy-clone-f loatIP_189-INFINITY)
Ticket Constraints:

Alerts:
No alerts defined

Resources Defaults:
resource-stickiness: 100
Operations Defaults:
No defaults set

Cluster Properties:
cluster-infrastructure: corosync
cluster-name: testcluster2
dc-version: 1.1.15-11.el7_3.2-e174ec8
have-watchdog: false
stonith-enabled: false

Quorum:
Options:
[root@t-haproxylb3 haproxy]#

2

Looking at the error message that haproxy returns when failing to start would shed light on the issue.

But, I’m pretty sure it fails because you are trying to bind() to an IP address that is not yet configured.

Use the transparent keyword on the bind line to instruct the kernel to bind anyway:
https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#5.1-transparent

3

Thanks @lukastribus. I did get this to work sort of… What I did was; I stopped the passive node, added the float ip to the haproxy config, added the new floating IP address via cibadmin --query tmp.xml cmd then editing the file by adding in the new floating IP and running a cibadmin --replace --xml-file tmp.xml. I would need to fail over the cluster to the other node before the new IP would be active. But we are going to go down the path of rebuilding the configs with a “bind *:443” in the frontend part of the config since this is easier for future growth and maintainability.

4

Hi,

Looks like you do not have the “net.ipv4.ip_nonlocal_bind = 1” setting configured.
This will enable daemons to bind to an ip address that’s currently not available on local network interface card.

Cheers,
Patrick

5

@pderuiter Just enabling the transparent keyword on the bind line will do the same, and also works for IPv6, while net.ipv4.ip_nonlocal_bind applies to all IPv4 sockets on the system and does not have an IPv6 equivalent.

That’s why my suggestion is to use that keyword.