HAProxy won't start properly


#1

I’m trying to test HAPRoxy to see if it would work well for a project … however I can’t seem to get it to start properly and I’m not entirely sure where I’m going wrong. I’m running this on an Ubuntu 16.04 system just for testing purposes on a private network. I installed it via apt and the HAProxy version that was installed was 1.6.13. I start haproxy and it says it started … but nothing ever happens. It never binds the port (according to netstat), never throws an errors (according the haproxy log or debug), and doesn’t do anything. I run haproxy manually with the -d flag and this is the results :

# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -d
Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.
Using epoll() as the polling mechanism.

It never goes any further … it just sits there with no other output. I’ve left it running for 10 - 15 minutes with no other output while I constantly try to connect to the port I specified in the cfg file (7032). The config passes the config check and here is the config that I have setup :

# /usr/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid
# cat /etc/haproxy/haproxy.cfg
global
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    maxconn 392
    user haproxy
    group haproxy
    daemon

defaults
    log global
    retries 2
    timeout connect 3000
    timeout server 5000
    timeout client 5000

listen mysql-cluster
    bind *:7032
    mode tcp
    option mysql-check
    balance roundrobin
    server innodbNode1 192.168.1.111:3306 check
    server innodbNode2 192.168.1.112:3306 check
    server innodbNode3 192.168.1.113:3306 check

I’m really not sure what I’m doing wrong here … I’ve got no errors, no problem indicated, yet nothing works. Could someone please point me in the direction of what I’m doing wrong?


#2

Update …

I’m still attempting to figure out what I’m doing wrong with no success. I removed the apt package and downloaded a more recent version. I pulled down haproxy-1.7.7 and compiled it with no change in results. It’s still not binding the port and giving no indications that it’s even trying.

# /usr/local/sbin/haproxy -v
HA-Proxy version 1.7.7 2017/06/25
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>
# /usr/local/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid
# /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -d
Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
        [COMP] compression
        [TRACE] trace
        [SPOE] spoe
Using epoll() as the polling mechanism.

I’ve checked and made sure that selinux is disabled on the system, so I know that is not the issue. I’m not getting any binding errors, I’m not getting any errors at all. After compiling 1.7.7 there is a change in the debug output (additional lines) but still nothing to indicate what the problem is …


#3

Try without the pid file if you are just testing.

Does the appropriate start script fail as well?
sudo systemctl start haproxy


#4

Tried without the pid file but it made no difference. I get the same results with the start script. Start script says it starts successfully with no errors the syslog files show no errors :

Jun 30 09:23:58 appTester systemd[1]: Starting HAProxy Load Balancer...
Jun 30 09:23:58 appTester systemd[1]: Started HAProxy Load Balancer.
Jun 30 09:38:34 appTester systemd[1]: Stopping HAProxy Load Balancer...
Jun 30 09:38:34 appTester systemd[1]: Stopped HAProxy Load Balancer.

The haproxy log also shows nothing …

Jun 30 09:23:58 appTester haproxy-systemd-wrapper[12953]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Jun 30 09:38:34 appTester haproxy-systemd-wrapper[12953]: haproxy-systemd-wrapper: SIGTERM -> 13051.
Jun 30 09:38:34 appTester haproxy-systemd-wrapper[13307]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

Here is the status from the systemctl :

● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2017-06-30 09:23:58 EDT; 25s ago
     Docs: man:haproxy(1)
           file:/usr/share/doc/haproxy/configuration.txt.gz
  Process: 12752 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
  Process: 12742 ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
  Process: 29502 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
 Main PID: 29506 (haproxy-systemd)
    Tasks: 3
   Memory: 1.2M
      CPU: 24ms
   CGroup: /system.slice/haproxy.service
           ├─29506 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           ├─29507 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
           └─29509 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

These are from this morning before I upgraded to haproxy 1.7.7 … I haven’t used the start script with new version yet, I have just been running the new one manually so I could run it with the debug flag.


#5

Can you start with the “-dR” (no SO_REUSEPORT), hopefully that will generate an error.

If that still does not lead to an error, please start with “strace -tt”, so that would be (strace is in the Ubuntu repositories, you can just install it via the usual “apt-get install strace”):
strace -tt /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -d -dR

strace will show all syscalls, so there will be a lot of output, please provide the result (but consider that it may contain private data such as IP addresses).


#6

I was finally able to get it successfully working. I’ve been digging around documentation, forums, and everything else I could find. I found a posting that mentioned haproxy not binding properly when bind * is used where IPv4 and IPv6 both exist. The solution on that forum was to apply v4v6 to the bind statement. I modified the config …

I changed :

bind *:7032

To the following :

bind *:7032 v4v6

Restarted the haproxy in debug mode and tested a connection and got the prompt I would expect from the port I specified in the config :

# telnet 127.0.0.1 7032
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
N
5.7.18-log▒]5krR_#o`c~(6>z1mysql_native_password^]
telnet> close
Connection closed.

And instantly saw the connection appear on the debug (running on another session) …

# /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -d
Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
        [COMP] compression
        [TRACE] trace
        [SPOE] spoe
Using epoll() as the polling mechanism.
00000000:mysql-cluster.accept(0004)=0006 from [127.0.0.1:49112]
00000000:mysql-cluster.srvcls[0006:0007]
00000000:mysql-cluster.clicls[0006:0007]
00000000:mysql-cluster.closed[0006:0007]

This appears to have resolved the issue. To make sure I hit the haproxy port with sysbench set at 48 threads and it ran with no issues. I can’t say if it’s a bug in the code or how it’s intended to operate … but without the v4v6 it doesn’t bind to any interface or address on the box. I removed the v4v6 statement, restarted haproxy, checked every IP on the box (127.0.0.1, 192.168.1.99, :::1) on the port in the config (7032) and got connection refused on every address. I added the v4v6 back then restarted and get a response on every IPv4 address.


#7

Mmh, can you run it through “strace -tt” in the working and non-working case, so that I can compare to two? I would like to find out what happened here, and, at the very least make some amendments to the documentation.


#8

The v4v6 was removed, haproxy started with “strace -tt”, connection on 127.0.0.1 7032 was attempted three times in another session, received connection refused on all three attempts. I’ve got the strace but it’s to large to post here and I don’t see a way to upload it. If you let me know how to send it I’ll be glad to send it to you.


#9

You can just upload it to pastebin.com and post the link (or the paste ID, if for some reason the software here restricts the amount of links - I know it is restrictive especially to new users).

Thanks


#10

Pastebin of failed attempts : https://pastebin.com/g9NrWvhH
Pastebin of successful attempt : https://pastebin.com/KCz3WJgZ

On the successful attempt, v4v6 was added, haproxy started, connection to 127.0.0.1 7032 was done a few moments after haproxy was restarted, and I received a mysql prompt.

If I can provide anything else let me know, if I can avoid others having this issue I’m happy to help.