Issue with peers in HA Proxy deployment

govind · September 17, 2024, 4:17pm

Hi Team,

I m running multiple HA proxy pod and trying to understand if HA proxy pods are in sync using peers concept.
When trying to execute below show command, I m not able to find anything :-

/ # echo “show peers” | socat stdio /var/run/haproxy.sock

though peers info is set in the config but no output. below is my configuration :-

## HAProxy daemon configuration
# ref: https://www.haproxy.org/download/2.6/doc/configuration.txt
config: |
  global
    stats socket /var/run/haproxy.sock mode 600
    log stdout format raw local0
    maxconn 1024

  defaults
    log global
    timeout client 60s
    timeout connect 60s
    timeout server 60s

  frontend jms
       mode http
       bind *:80
       default_backend jms

  peers haproxy-peers
      # peer <peer_name> <ip:port>
      peer i-012345 haproxy-1.ha-proxy.svc.cluster.local:1024
      peer i-012367 haproxy-2.ha-proxy.svc.cluster.local:1024 

  backend jms
        mode http
        balance leastconn
        stick-table type string len 256 size 200k expire 120m peers haproxy-peers
        stick on url_param(room) table jms
        hash-type consistent
 
        server jitsi-1 10.22.33.22:80 id 88
        server jitsi-2 10.33.11.34:80 id 72

Any idea on this ?

govind · September 18, 2024, 3:36am

On further debugging, I m seeing Alert and warning in the haproxy command output, mentioned below :-

/ # 
/ #  haproxy -db -f /etc/haproxy/haproxy.cfg -L i-012367
[NOTICE]   (27) : haproxy version is 2.9.6-9eafce5
[ALERT]    (27) : Binding [/etc/haproxy/haproxy.cfg:21] for peers section i-012367: cannot bind socket (Address not available) for [172.25.78.105:1024]
[ALERT]    (27) : [haproxy.main()] Some protocols failed to start their listeners! Exiting.
/ # 
/ #

/ # haproxy -f /etc/haproxy/haproxy.cfg -c
[NOTICE]   (29) : haproxy version is 2.9.6-9eafce5
[WARNING]  (29) : config : Removing incomplete section 'peers haproxy-peers' (no peer named 'haproxy-1-7d79b7d54-mdvn5').
Warnings were found.
/ # 
/ #

Could you please guide me what is wrong here ?

Also how we can check the status of haproxy as I’m not able to see anything with using haproxy or haproxy.service in the pod.

govind · September 18, 2024, 3:59pm

Hi @lukastribus

Can you help me on this issue ?

lukastribus · September 18, 2024, 4:28pm

The error message is clear:

cannot bind socket (Address not available) for [172.25.78.105:1024]

Make sure that haproxy runs with root privileges, move the ports above privileged ports, or apply some workaround like --sysctl net.ipv4.ip_unprivileged_port_start=0 when using docker run.

But this forum is really about haproxy itself, there are really no docker folks here as far as I know.

govind · September 22, 2024, 7:39pm

Hi @lukastribus

To simplify the things, I have installed the HA-Proxy on 2 RHEL Servers, everything is working fine but again when I tried to setup peers in my haproxy config of rhel server then getting below error -

  peers haproxy-peers
      # peer <peer_name> <ip:port>
      peer ha1 XX.XX.XX.XX:1024
      peer ha2 YY.YY.YY.YY:1024

Error -
Proxy 'jms': unable to find local peer 'ip-XX-XX-XX-XX.us-west-2.compute.internal' in peers section 'haproxy-peers'

Any idea ?

lukastribus · September 23, 2024, 6:19am

Put it into /etc/hosts if it can’t lookup the name.

govind · September 23, 2024, 6:33am

I did but same error.

[root@ip bin]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

XX.XX.XX.XX  ip-XX-XX-XX-XX.us-west-2.compute.internal
YY.YY.YY.YY  ip-YY-YY-YY-YY.us-west-2.compute.internal

[root@ip bin]#
[root@ip bin]# sudo tail -f /var/log/messages
Sep 23 06:27:31  haproxy[7823]: [ALERT] 266/062731 (7823) : Fatal errors found in configuration.
Sep 23 06:27:31 ip-10-28-17-250 systemd[1]: haproxy.service: Control process exited, code=exited status=1
Sep 23 06:27:31 systemd[1]: haproxy.service: Failed with result 'exit-code'.
Sep 23 06:27:31  systemd[1]: Failed to start HAProxy Load Balancer.
Sep 23 06:29:00 systemd[1]: Starting HAProxy Load Balancer...
Sep 23 06:29:00 haproxy[7843]: [ALERT] 266/062900 (7843) : Proxy 'jms': unable to find local peer 'XX.XX.XX.XX.us-west-2.compute.internal' in peers section 'haproxy-peers'.

govind · September 24, 2024, 4:06am

Hi @lukastribus

I m able to fix that above error by setting the hostname using hostnamectl command. this error is no longer present and HA proxy started successfully.

However I m not able to find the information of peers , getting unknown command.

How I can validate my peers are in sync now and have all info ?

[root@haproxy2 bin]# echo "show peers" | sudo socat /run/haproxy/admin.sock stdio
Unknown command. Please enter one of the following commands only

Also don’t see any peers info when running the HA proxy UI though it is present in the configuration file.

lukastribus · September 24, 2024, 7:23am

Correct, the hostname needs to match. Further informations in the documentation:

http://docs.haproxy.org/2.8/configuration.html#3.5-peer

If is set to the local peer name (by default hostname, or forced using “-L” command line option or “localpeer” global configuration setting), HAProxy will listen for incoming remote peer connection on the provided address.

Regarding the stats socket, I assume you have to upgrade it to level admin:

http://docs.haproxy.org/2.8/configuration.html#5.1-level

govind · September 26, 2024, 8:56am

Hi @lukastribus

Above mentioned suggestion helped me to get rid of the cannot bind socket error, Now I m not getting that error when trying to execute this command - haproxy -db -f /etc/haproxy/haproxy.cfg.

However even that issue fixed but in the HA proxy pod log, i m getting below error, it is still not able to configure sync using peers.

sh-3.2# kubectl logs haproxy-1-6bc9c898f9-42srk -n ha-proxy 
[NOTICE]  (1) : haproxy version is 2.9.6-9eafce5
[WARNING] (1) : config : Removing incomplete section 'peers haproxy-peers' (no peer named 'haproxy-1-6bc9c898f9-42srk').
[NOTICE]  (1) : New worker (8) forked
[NOTICE]  (1) : Loading success.
sh-3.2# 
sh-3.2# 
sh-3.2# kubectl exec -it haproxy-1-6bc9c898f9-42srk -n ha-proxy /bin/sh
/ # 
/ # echo "show peers haproxy-peers" | socat stdio /var/run/haproxy.sock
No such peers
/ # 
/ # haproxy -db -f /etc/haproxy/haproxy.cfg -L ha1

-L parameter does not match any of peernames, and there is not localpeer global configuration setting either, so HAProxy is using default of local hostname which is haproxy-1-6bc9c898f9-42srk which didn’t match any of peer declarations either, hence the whole block is marked invalid.

Any advise on this error now ?

lukastribus · September 26, 2024, 9:01am

Please read the peer documentation in it’s entirety.

Either:

specify the correct hostname with -L
specify the correct hostname with localpeer gloabl configuration
or fix the actual local hostname

govind · September 26, 2024, 10:24am

Hi @lukastribus

Here I need some clarification, what do you mean by correct hostname ?

what I did is I provided ha1 and ha2 in localpeer section and with this no peer named error gone but if you see the

  global
    stats socket /var/run/haproxy.sock mode 600 level admin
    log stdout format raw local0
    maxconn 1024
    localpeer ha1
    localpeer ha2

  peers haproxy-peers
      # peer <peer_name> <ip:port>
      peer ha1 haproxy-1.ha-proxy.svc.cluster.local:1024
      peer ha2 haproxy-2.ha-proxy.svc.cluster.local:1024

But if you see below output of show peers, it is showing ha2(local,inactive).

echo “show peers haproxy-peers” | socat stdio /var/run/haproxy.sock

0x7f97cb677500: [06/Nov/2024:05:43:20] id=haproxy-peers disabled=0 flags=0x33 resync_timeout=<PAST> task_calls=293
  0x7f97cb682280: id=ha2(local,inactive) addr=172.22.33.44:1024 last_status=NONE last_hdshk=<NEVER>
        reconnect=<NEVER> heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=0 proto_err=0 coll=0
        flags=0x0
        shared tables:
          0x7f97cb121aa0 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
              last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
              table:0x7f97cb0a4f00 id=jms update=2 localupdate=2 commitupdate=0 refcnt=1
        Dictionary cache not dumped (use "show peers dict")
  0x7f97cb682000: id=ha1(remote,active) addr=172.44.55.66:1024 last_status=CONN last_hdshk=0s
        reconnect=4s heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=146 proto_err=0 coll=0
        flags=0x0 appctx:0x7f97ca4003c0 st0=6 st1=0 task_calls=2 state=TAR
        shared tables:
          0x7f97cb121b40 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
              last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
              table:0x7f97cb0a4f00 id=jms update=2 localupdate=2 commitupdate=0 refcnt=1
        Dictionary cache not dumped (use "show peers dict")

Note -
haproxy-1.ha-proxy.svc.cluster.local is resolving to service cluster IP i.e 172.44.55.66:1024
haproxy-2.ha-proxy.svc.cluster.local is resolving to service cluster IP i.e 172.22.33.44:1024

lukastribus · September 26, 2024, 10:31am

Again I am asking you that you read the peer keyword documentation in its entirety.

I cannot spoon feed you the documentation, you need to invest the 180 seconds required to read that part of the documentation yourself.

If you have 3 haproxy instances that peer with each other, the peer section needs to contain 3 peers. One local, two remote.

govind · September 26, 2024, 2:28pm

I’m sorry for the inconvenience this may cause and Thank you for your guidance on this issue so far

I’ve reviewed the doc thoroughly and tried multiple combinations based on what I understood, but I’m still facing issues with the configuration not working as expected…
Could you provide any additional insights or troubleshooting tips that might help me resolve this?

global
  stats socket /var/run/haproxy.sock mode 600 level admin
  log stdout format raw local0
  maxconn 1024
  localpeer ha2

 peers haproxy-peers
   # peer <peer_name> <ip:port>
   peer ha1 haproxy-1.ha-proxy.svc.cluster.local:1024
   peer ha2 0.0.0.0:1024

I also tried below config as well -

 peers haproxy-peers
   # peer <peer_name> <ip:port>
   peer ha1 haproxy-1.ha-proxy.svc.cluster.local:1024
   peer ha2

Show peers command output -

/ # echo "show peers haproxy-peers" | socat stdio /var/run/haproxy.sock
0x7f3adce77500: [06/Nov/2024:05:43:19] id=haproxy-peers disabled=0 flags=0x33 resync_timeout=<PAST> task_calls=39
 0x7f3adce82280: id=ha2(local,inactive) addr=0.0.0.0:1024 last_status=NONE last_hdshk=<NEVER>
    reconnect=<NEVER> heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=0 proto_err=0 coll=0
    flags=0x0
    shared tables:
     0x7f3adc921aa0 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
       last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
       table:0x7f3adc8a4f00 id=jms update=0 localupdate=0 commitupdate=0 refcnt=1
    Dictionary cache not dumped (use "show peers dict")
 0x7f3adce82000: id=ha1(remote,active) addr=172.10.144.89:1024 last_status=CONN last_hdshk=0s
    reconnect=4s heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=20 proto_err=0 coll=0
    flags=0x0 appctx:0x7f3adce86b00 st0=6 st1=0 task_calls=2 state=CON
    shared tables:
     0x7f3adc921b40 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
       last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
       table:0x7f3adc8a4f00 id=jms update=0 localupdate=0 commitupdate=0 refcnt=1
    Dictionary cache not dumped (use "show peers dict")

Seems like the localpeer is inactive only
I tried with 127.0.0.1 and also tried to bind to local ip address nothing work so far.

lukastribus · September 26, 2024, 4:58pm

No, I’m afraid I’m unable.

You can wait for others to chime in here or get commercial support from haproxy.com folks.

Topic		Replies	Views
Peers with nbproc > 1 in HAProxy version 2.0.4 Help!	0	1288	August 16, 2019
Some Listeners Failed to Start - How to Debug? Help!	6	5469	June 22, 2018
Haproxy peers section not respecting init-addr none Help!	3	396	October 23, 2020
Run Show commands on HAProxy Help!	4	1942	April 4, 2019
Cannot bind socket Help!	1	449	May 15, 2020

Issue with peers in HA Proxy deployment

Related Topics