I’m trying to test HAPRoxy to see if it would work well for a project … however I can’t seem to get it to start properly and I’m not entirely sure where I’m going wrong. I’m running this on an Ubuntu 16.04 system just for testing purposes on a private network. I installed it via apt and the HAProxy version that was installed was 1.6.13. I start haproxy and it says it started … but nothing ever happens. It never binds the port (according to netstat), never throws an errors (according the haproxy log or debug), and doesn’t do anything. I run haproxy manually with the -d flag and this is the results :
# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -d
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Using epoll() as the polling mechanism.
It never goes any further … it just sits there with no other output. I’ve left it running for 10 - 15 minutes with no other output while I constantly try to connect to the port I specified in the cfg file (7032). The config passes the config check and here is the config that I have setup :
# /usr/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid
# cat /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 392
user haproxy
group haproxy
daemon
defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000
listen mysql-cluster
bind *:7032
mode tcp
option mysql-check
balance roundrobin
server innodbNode1 192.168.1.111:3306 check
server innodbNode2 192.168.1.112:3306 check
server innodbNode3 192.168.1.113:3306 check
I’m really not sure what I’m doing wrong here … I’ve got no errors, no problem indicated, yet nothing works. Could someone please point me in the direction of what I’m doing wrong?
Update …
I’m still attempting to figure out what I’m doing wrong with no success. I removed the apt package and downloaded a more recent version. I pulled down haproxy-1.7.7 and compiled it with no change in results. It’s still not binding the port and giving no indications that it’s even trying.
# /usr/local/sbin/haproxy -v
HA-Proxy version 1.7.7 2017/06/25
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>
# /usr/local/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid
# /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -d
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe
Using epoll() as the polling mechanism.
I’ve checked and made sure that selinux is disabled on the system, so I know that is not the issue. I’m not getting any binding errors, I’m not getting any errors at all. After compiling 1.7.7 there is a change in the debug output (additional lines) but still nothing to indicate what the problem is …
Try without the pid file if you are just testing.
Does the appropriate start script fail as well?
sudo systemctl start haproxy
Tried without the pid file but it made no difference. I get the same results with the start script. Start script says it starts successfully with no errors the syslog files show no errors :
Jun 30 09:23:58 appTester systemd[1]: Starting HAProxy Load Balancer...
Jun 30 09:23:58 appTester systemd[1]: Started HAProxy Load Balancer.
Jun 30 09:38:34 appTester systemd[1]: Stopping HAProxy Load Balancer...
Jun 30 09:38:34 appTester systemd[1]: Stopped HAProxy Load Balancer.
The haproxy log also shows nothing …
Jun 30 09:23:58 appTester haproxy-systemd-wrapper[12953]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Jun 30 09:38:34 appTester haproxy-systemd-wrapper[12953]: haproxy-systemd-wrapper: SIGTERM -> 13051.
Jun 30 09:38:34 appTester haproxy-systemd-wrapper[13307]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Here is the status from the systemctl :
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2017-06-30 09:23:58 EDT; 25s ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 12752 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
Process: 12742 ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
Process: 29502 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
Main PID: 29506 (haproxy-systemd)
Tasks: 3
Memory: 1.2M
CPU: 24ms
CGroup: /system.slice/haproxy.service
├─29506 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─29507 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
└─29509 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
These are from this morning before I upgraded to haproxy 1.7.7 … I haven’t used the start script with new version yet, I have just been running the new one manually so I could run it with the debug flag.
Can you start with the “-dR” (no SO_REUSEPORT), hopefully that will generate an error.
If that still does not lead to an error, please start with “strace -tt”, so that would be (strace is in the Ubuntu repositories, you can just install it via the usual “apt-get install strace”):
strace -tt /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -d -dR
strace will show all syscalls, so there will be a lot of output, please provide the result (but consider that it may contain private data such as IP addresses).
I was finally able to get it successfully working. I’ve been digging around documentation, forums, and everything else I could find. I found a posting that mentioned haproxy not binding properly when bind * is used where IPv4 and IPv6 both exist. The solution on that forum was to apply v4v6 to the bind statement. I modified the config …
I changed :
bind *:7032
To the following :
bind *:7032 v4v6
Restarted the haproxy in debug mode and tested a connection and got the prompt I would expect from the port I specified in the config :
# telnet 127.0.0.1 7032
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
N
5.7.18-log▒]5krR_#o`c~(6>z1mysql_native_password^]
telnet> close
Connection closed.
And instantly saw the connection appear on the debug (running on another session) …
# /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -d
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe
Using epoll() as the polling mechanism.
00000000:mysql-cluster.accept(0004)=0006 from [127.0.0.1:49112]
00000000:mysql-cluster.srvcls[0006:0007]
00000000:mysql-cluster.clicls[0006:0007]
00000000:mysql-cluster.closed[0006:0007]
This appears to have resolved the issue. To make sure I hit the haproxy port with sysbench set at 48 threads and it ran with no issues. I can’t say if it’s a bug in the code or how it’s intended to operate … but without the v4v6 it doesn’t bind to any interface or address on the box. I removed the v4v6 statement, restarted haproxy, checked every IP on the box (127.0.0.1, 192.168.1.99, :::1) on the port in the config (7032) and got connection refused on every address. I added the v4v6 back then restarted and get a response on every IPv4 address.
Mmh, can you run it through “strace -tt” in the working and non-working case, so that I can compare to two? I would like to find out what happened here, and, at the very least make some amendments to the documentation.
The v4v6 was removed, haproxy started with “strace -tt”, connection on 127.0.0.1 7032 was attempted three times in another session, received connection refused on all three attempts. I’ve got the strace but it’s to large to post here and I don’t see a way to upload it. If you let me know how to send it I’ll be glad to send it to you.
You can just upload it to pastebin.com and post the link (or the paste ID, if for some reason the software here restricts the amount of links - I know it is restrictive especially to new users).
Thanks
Pastebin of failed attempts : https://pastebin.com/g9NrWvhH
Pastebin of successful attempt : https://pastebin.com/KCz3WJgZ
On the successful attempt, v4v6 was added, haproxy started, connection to 127.0.0.1 7032 was done a few moments after haproxy was restarted, and I received a mysql prompt.
If I can provide anything else let me know, if I can avoid others having this issue I’m happy to help.