Haproxy using systemctl

Hi

Did anyone using compiled version of haproxy on RHEL 7.x or derivatives? We have been running compiled version successfully last few years on RHEL 6.7 platform successfully. recently we migrated to RHEL 7.x, thus compiled using the same option which we are using on RHEL 7.x build system. i am having intermediate down every few days. Not seeing any relevent messages on logs ( /var/logs/messages or haproxy notiice logs) not sure why it is going down every few days and we have to issue reload to make it work again.

only noticeable difference is system-wrapper script and systemctl script. I am using system-wrapper generated from compiled version and using unit file from haproxy contribution directory.

unit file:

cat etc/systemd/system/haproxy-80.service

[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target

[Service]
Environment=“CONFIG=/opt/haproxy/haproxy-80/conf/haproxy.conf” "PIDFILE=/run/haproxy-80.pid"
ExecStartPre=/opt/haproxy/haproxy-80/sbin/haproxy-80 -f $CONFIG -c -q
ExecStart=/opt/haproxy/haproxy-80/sbin/haproxy-80-systemd-wrapper -f $CONFIG -p $PIDFILE
ExecReload=/opt/haproxy/haproxy-80/sbin/haproxy-80 -f $CONFIG -c -q
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=haproxy-80
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target

Syslog file

$ cat etc/rsyslog.d/haproxy-80.conf
if ($programname == ‘haproxy-80’ and $syslogseverity-text == ‘info’) then -/opt/haproxy/logs/haproxy-80/haproxy-80-info.log
& ~
if ($programname == ‘haproxy-80’ and $syslogseverity-text == ‘notice’) then -/opt/haproxy/logs/haproxy-80/haproxy-80-notice.log
& ~

haproxy config file:

$ cat opt/haproxy/haproxy-80/conf/haproxy.conf
global
daemon
user www
group www
pidfile /var/run/haproxy-80.pid
log /dev/log local0 info
log /dev/log local1 notice
maxconn 50000
tune.bufsize 128000
tune.maxrewrite 1024
stats socket /var/run/haproxy-80.stat mode 600 level admin
stats timeout 2m
nbproc 1
tune.comp.maxlevel 5

defaults
log global
mode http
option forwardfor except 127.0.0.0/8
option http-server-close
option dontlognull
option httplog clf
option tcp-smart-accept
option tcp-smart-connect
option splice-auto
no option logasap
balance roundrobin
retries 3
timeout client 1m
timeout http-keep-alive 5s
timeout server 1m
timeout queue 1m
timeout connect 10s
timeout check 10s
timeout http-request 10s
default-server maxconn 1024 inter 5s rise 5 fall 5

   errorfile 503 /opt/haproxy/errors/503.html
   errorfile 400 /opt/haproxy/errors/400.html
   errorfile 403 /opt/haproxy/errors/403.html
   errorfile 500 /opt/haproxy/errors/500.html
   errorfile 502 /opt/haproxy/errors/502.html
   errorfile 504 /opt/haproxy//errors/504.html
   errorfile 408 /dev/null

frontend IN
mode http
bind *:80
monitor-uri /proxy.html
log-format [%pid]\ [%Ts.%ms]\ %ac/%fc/%bc/%bq/%sc/%sq/%rc\ %Tq/%Tw/%Tc/%Tr/%Tt\ %tsc\ %ci:%cp\ %fi:%fp\ %si:%sp\ %ft\ %{+Q}r\ %ST\ %b:%s\ “%CC”\ “%hr”\ “%CS”\ “%hs”\ ireq_size=%U\ resp_size=%B
unique-id-format %{+X}o\ %ci:%cp
%fi:%fp_%Ts_%rt:%pid
unique-id-header X-Unique-ID

Blacklist: Deny access to some IPs before anything else is checked

tcp-request content reject if { src -f /opt/haproxy/security/rules/blacklist.lst }

Whitelist: Allow IPs to bypass the filters

http-request allow if { src -f /opt/haproxy/security/rules/whitelist.lst }

    reqadd                X-Forwarded-Proto:\ http
    reqadd                X-Forwarded-Port:\ 80

    rspdel                ^Server:.*
    rspdel                ^X-Powered-By:.*
    rspdel                ^X-Runtime:.*

capture request header X-Forwarded-For len 500
capture request header Host len 500
capture request header Accept-Encoding len 32
capture response header Location len 64
capture response header Content-Type len 64
capture request header Referrer len 64
capture request header Content-Length len 10
capture request header User-Agent len 64
capture request header X-Unique-ID len 64

default_backend sso_server

Compile flags:

./haproxy-80 -vv
HA-Proxy version 1.6.9 2016/08/30
Copyright 2000-2016 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = native
CC = gcc
CFLAGS = -O2 -march=native -g -fno-strict-aliasing -Wdeclaration-after-statement
OPTIONS = USE_LIBCRYPT=1 USE_ZLIB=yes USE_OPENSSL=1 USE_STATIC_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with OpenSSL version : OpenSSL 1.0.2j 26 Sep 2016
Running on OpenSSL version : OpenSSL 1.0.2j 26 Sep 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

PS output:

www 1451 18205 0 Oct18 ? 00:00:00 /opt/haproxy/haproxy-80/sbin/haproxy-80 -f /opt/haproxy/haproxy-80/conf/haproxy.conf -p /run/haproxy-80.pid -Ds -sf 18209
www 1452 1451 0 Oct18 ? 00:53:31 /opt/haproxy/haproxy-80/sbin/haproxy-80 -f /opt/haproxy/haproxy-80/conf/haproxy.conf -p /run/haproxy-80.pid -Ds -sf 18209
root 18205 1 0 Oct11 ? 00:00:00 /opt/haproxy/haproxy-80/sbin/haproxy-80-systemd-wrapper -f /opt/haproxy/haproxy-80/conf/haproxy.conf -p /run/haproxy-80.pid
root 18219 1 0 Oct11 ? 00:00:00 /opt/haproxy/haproxy-443/sbin/haproxy-443-systemd-wrapper -f /opt/haproxy/haproxy-443/conf/haproxy.conf -p /run/haproxy-443.pid
wwwssl 28231 18219 0 Oct28 ? 00:00:00 /opt/haproxy/haproxy-443/sbin/haproxy-443 -f /opt/haproxy/haproxy-443/conf/haproxy.conf -p /run/haproxy-443.pid -Ds -sf 26752
wwwssl 28232 28231 0 Oct28 ? 00:09:01 /opt/haproxy/haproxy-443/sbin/haproxy-443 -f /opt/haproxy/haproxy-443/conf/haproxy.conf -p /run/haproxy-443.pid -Ds -sf 26752

any did one had similar experience? or any one using haproxy successfully on rhel platform using systemd based service management?

any help is highly appreciated. unable to breakthrough this issue due to insufficient log why it is failing every few days.

Srinivas Kotaru