govind
September 17, 2024, 4:17pm
1
Hi Team,
I m running multiple HA proxy pod and trying to understand if HA proxy pods are in sync using peers concept.
When trying to execute below show command, I m not able to find anything :-
/ # echo “show peers” | socat stdio /var/run/haproxy.sock
though peers info is set in the config but no output. below is my configuration :-
## HAProxy daemon configuration
# ref: https://www.haproxy.org/download/2.6/doc/configuration.txt
config: |
global
stats socket /var/run/haproxy.sock mode 600
log stdout format raw local0
maxconn 1024
defaults
log global
timeout client 60s
timeout connect 60s
timeout server 60s
frontend jms
mode http
bind *:80
default_backend jms
peers haproxy-peers
# peer <peer_name> <ip:port>
peer i-012345 haproxy-1.ha-proxy.svc.cluster.local:1024
peer i-012367 haproxy-2.ha-proxy.svc.cluster.local:1024
backend jms
mode http
balance leastconn
stick-table type string len 256 size 200k expire 120m peers haproxy-peers
stick on url_param(room) table jms
hash-type consistent
server jitsi-1 10.22.33.22:80 id 88
server jitsi-2 10.33.11.34:80 id 72
Any idea on this ?
govind
September 18, 2024, 3:36am
2
On further debugging, I m seeing Alert and warning in the haproxy command output, mentioned below :-
/ #
/ # haproxy -db -f /etc/haproxy/haproxy.cfg -L i-012367
[NOTICE] (27) : haproxy version is 2.9.6-9eafce5
[ALERT] (27) : Binding [/etc/haproxy/haproxy.cfg:21] for peers section i-012367: cannot bind socket (Address not available) for [172.25.78.105:1024]
[ALERT] (27) : [haproxy.main()] Some protocols failed to start their listeners! Exiting.
/ #
/ #
/ # haproxy -f /etc/haproxy/haproxy.cfg -c
[NOTICE] (29) : haproxy version is 2.9.6-9eafce5
[WARNING] (29) : config : Removing incomplete section 'peers haproxy-peers' (no peer named 'haproxy-1-7d79b7d54-mdvn5').
Warnings were found.
/ #
/ #
Could you please guide me what is wrong here ?
Also how we can check the status of haproxy as I’m not able to see anything with using haproxy or haproxy.service in the pod.
govind
September 18, 2024, 3:59pm
3
Hi @lukastribus
Can you help me on this issue ?
The error message is clear:
cannot bind socket (Address not available) for [172.25.78.105:1024]
Make sure that haproxy runs with root privileges, move the ports above privileged ports, or apply some workaround like --sysctl net.ipv4.ip_unprivileged_port_start=0
when using docker run.
But this forum is really about haproxy itself, there are really no docker folks here as far as I know.
1 Like
govind
September 22, 2024, 7:39pm
5
Hi @lukastribus
To simplify the things, I have installed the HA-Proxy on 2 RHEL Servers, everything is working fine but again when I tried to setup peers in my haproxy config of rhel server then getting below error -
peers haproxy-peers
# peer <peer_name> <ip:port>
peer ha1 XX.XX.XX.XX:1024
peer ha2 YY.YY.YY.YY:1024
Error -
Proxy 'jms': unable to find local peer 'ip-XX-XX-XX-XX.us-west-2.compute.internal' in peers section 'haproxy-peers'
Any idea ?
Put it into /etc/hosts
if it can’t lookup the name.
govind
September 23, 2024, 6:33am
7
I did but same error.
[root@ip bin]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
XX.XX.XX.XX ip-XX-XX-XX-XX.us-west-2.compute.internal
YY.YY.YY.YY ip-YY-YY-YY-YY.us-west-2.compute.internal
[root@ip bin]#
[root@ip bin]# sudo tail -f /var/log/messages
Sep 23 06:27:31 haproxy[7823]: [ALERT] 266/062731 (7823) : Fatal errors found in configuration.
Sep 23 06:27:31 ip-10-28-17-250 systemd[1]: haproxy.service: Control process exited, code=exited status=1
Sep 23 06:27:31 systemd[1]: haproxy.service: Failed with result 'exit-code'.
Sep 23 06:27:31 systemd[1]: Failed to start HAProxy Load Balancer.
Sep 23 06:29:00 systemd[1]: Starting HAProxy Load Balancer...
Sep 23 06:29:00 haproxy[7843]: [ALERT] 266/062900 (7843) : Proxy 'jms': unable to find local peer 'XX.XX.XX.XX.us-west-2.compute.internal' in peers section 'haproxy-peers'.
govind
September 24, 2024, 4:06am
8
Hi @lukastribus
I m able to fix that above error by setting the hostname using hostnamectl command. this error is no longer present and HA proxy started successfully.
However I m not able to find the information of peers
, getting unknown command.
How I can validate my peers are in sync now and have all info ?
[root@haproxy2 bin]# echo "show peers" | sudo socat /run/haproxy/admin.sock stdio
Unknown command. Please enter one of the following commands only
Also don’t see any peers info when running the HA proxy UI though it is present in the configuration file.
Correct, the hostname needs to match. Further informations in the documentation:
http://docs.haproxy.org/2.8/configuration.html#3.5-peer
If is set to the local peer name (by default hostname, or forced using “-L” command line option or “localpeer ” global configuration setting), HAProxy will listen for incoming remote peer connection on the provided address.
Regarding the stats socket, I assume you have to upgrade it to level admin:
http://docs.haproxy.org/2.8/configuration.html#5.1-level
1 Like
govind
September 26, 2024, 8:56am
10
Hi @lukastribus
Above mentioned suggestion helped me to get rid of the cannot bind socket error, Now I m not getting that error when trying to execute this command - haproxy -db -f /etc/haproxy/haproxy.cfg.
However even that issue fixed but in the HA proxy pod log, i m getting below error, it is still not able to configure sync using peers.
sh-3.2# kubectl logs haproxy-1-6bc9c898f9-42srk -n ha-proxy
[NOTICE] (1) : haproxy version is 2.9.6-9eafce5
[WARNING] (1) : config : Removing incomplete section 'peers haproxy-peers' (no peer named 'haproxy-1-6bc9c898f9-42srk').
[NOTICE] (1) : New worker (8) forked
[NOTICE] (1) : Loading success.
sh-3.2#
sh-3.2#
sh-3.2# kubectl exec -it haproxy-1-6bc9c898f9-42srk -n ha-proxy /bin/sh
/ #
/ # echo "show peers haproxy-peers" | socat stdio /var/run/haproxy.sock
No such peers
/ #
/ # haproxy -db -f /etc/haproxy/haproxy.cfg -L ha1
-L parameter does not match any of peernames, and there is not localpeer global configuration setting either, so HAProxy is using default of local hostname which is haproxy-1-6bc9c898f9-42srk which didn’t match any of peer declarations either, hence the whole block is marked invalid.
Any advise on this error now ?
Please read the peer documentation in it’s entirety.
govind:
-L parameter does not match any of peernames, and there is not localpeer global configuration setting either, so HAProxy is using default of local hostname which is haproxy-1-6bc9c898f9-42srk which didn’t match any of peer declarations either, hence the whole block is marked invalid.
Any advise on this error now ?
Either:
specify the correct hostname with -L
specify the correct hostname with localpeer gloabl configuration
or fix the actual local hostname
govind
September 26, 2024, 10:24am
12
Hi @lukastribus
Here I need some clarification, what do you mean by correct hostname ?
what I did is I provided ha1 and ha2 in localpeer section and with this no peer named
error gone but if you see the
global
stats socket /var/run/haproxy.sock mode 600 level admin
log stdout format raw local0
maxconn 1024
localpeer ha1
localpeer ha2
peers haproxy-peers
# peer <peer_name> <ip:port>
peer ha1 haproxy-1.ha-proxy.svc.cluster.local:1024
peer ha2 haproxy-2.ha-proxy.svc.cluster.local:1024
But if you see below output of show peers, it is showing ha2(local,inactive).
echo “show peers haproxy-peers” | socat stdio /var/run/haproxy.sock
0x7f97cb677500: [06/Nov/2024:05:43:20] id=haproxy-peers disabled=0 flags=0x33 resync_timeout=<PAST> task_calls=293
0x7f97cb682280: id=ha2(local,inactive) addr=172.22.33.44:1024 last_status=NONE last_hdshk=<NEVER>
reconnect=<NEVER> heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=0 proto_err=0 coll=0
flags=0x0
shared tables:
0x7f97cb121aa0 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
table:0x7f97cb0a4f00 id=jms update=2 localupdate=2 commitupdate=0 refcnt=1
Dictionary cache not dumped (use "show peers dict")
0x7f97cb682000: id=ha1(remote,active) addr=172.44.55.66:1024 last_status=CONN last_hdshk=0s
reconnect=4s heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=146 proto_err=0 coll=0
flags=0x0 appctx:0x7f97ca4003c0 st0=6 st1=0 task_calls=2 state=TAR
shared tables:
0x7f97cb121b40 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
table:0x7f97cb0a4f00 id=jms update=2 localupdate=2 commitupdate=0 refcnt=1
Dictionary cache not dumped (use "show peers dict")
Note -
haproxy-1.ha-proxy.svc.cluster.local is resolving to service cluster IP i.e 172.44.55.66:1024
haproxy-2.ha-proxy.svc.cluster.local is resolving to service cluster IP i.e 172.22.33.44:1024
Again I am asking you that you read the peer keyword documentation in its entirety.
I cannot spoon feed you the documentation, you need to invest the 180 seconds required to read that part of the documentation yourself.
If you have 3 haproxy instances that peer with each other, the peer section needs to contain 3 peers. One local, two remote.
govind
September 26, 2024, 2:28pm
14
I’m sorry for the inconvenience this may cause and Thank you for your guidance on this issue so far
I’ve reviewed the doc thoroughly and tried multiple combinations based on what I understood, but I’m still facing issues with the configuration not working as expected…
Could you provide any additional insights or troubleshooting tips that might help me resolve this?
global
stats socket /var/run/haproxy.sock mode 600 level admin
log stdout format raw local0
maxconn 1024
localpeer ha2
peers haproxy-peers
# peer <peer_name> <ip:port>
peer ha1 haproxy-1.ha-proxy.svc.cluster.local:1024
peer ha2 0.0.0.0:1024
I also tried below config as well -
peers haproxy-peers
# peer <peer_name> <ip:port>
peer ha1 haproxy-1.ha-proxy.svc.cluster.local:1024
peer ha2
Show peers command output -
/ # echo "show peers haproxy-peers" | socat stdio /var/run/haproxy.sock
0x7f3adce77500: [06/Nov/2024:05:43:19] id=haproxy-peers disabled=0 flags=0x33 resync_timeout=<PAST> task_calls=39
0x7f3adce82280: id=ha2(local,inactive) addr=0.0.0.0:1024 last_status=NONE last_hdshk=<NEVER>
reconnect=<NEVER> heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=0 proto_err=0 coll=0
flags=0x0
shared tables:
0x7f3adc921aa0 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
table:0x7f3adc8a4f00 id=jms update=0 localupdate=0 commitupdate=0 refcnt=1
Dictionary cache not dumped (use "show peers dict")
0x7f3adce82000: id=ha1(remote,active) addr=172.10.144.89:1024 last_status=CONN last_hdshk=0s
reconnect=4s heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=20 proto_err=0 coll=0
flags=0x0 appctx:0x7f3adce86b00 st0=6 st1=0 task_calls=2 state=CON
shared tables:
0x7f3adc921b40 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
table:0x7f3adc8a4f00 id=jms update=0 localupdate=0 commitupdate=0 refcnt=1
Dictionary cache not dumped (use "show peers dict")
Seems like the localpeer is inactive only
I tried with 127.0.0.1 and also tried to bind to local ip address nothing work so far.
No, I’m afraid I’m unable.
You can wait for others to chime in here or get commercial support from haproxy.com
folks.