Logs are not going into log file

I am facing problem. I have two different harpxoy linux servers (in diff network range). In one server, i can see logs are getting generated in log file but on other server, the logs are not getting generated in log file.

I have compared almost all possible config file from two servers and they appear to be same but still cannot find the clue on what is the problem.

Working server -

[root@<hostname> ~]# tail -f  /var/log/haproxy*.log
 
==> /var/log/haproxy.log <==
 
==> /var/log/haproxy-status.log <==
 
==> /var/log/haproxy-traffic.log <==
Oct 28 20:32:48 localhost haproxy[1347]: source_ip:52675 [28/Oct/2021:20:32:46.269] ssl~ logs/es_server1 98/0/1/1391/1950 200 103475 - - --VN 43/0/0/0/0 0/0 "POST /api/telemetry/v2/clusters/_stats HTTP/1.1"
Oct 28 20:33:47 localhost haproxy[1347]: source_ip:52709 [28/Oct/2021:20:33:46.294] ssl~ logs/es_server1 125/0/0/1426/1589 200 103369 - - --VN 43/0/0/0/0 0/0 "POST /api/telemetry/v2/clusters/_stats HTTP/1.1"

ownership of files

-rw------- 1 root root       0 Jan 19  2020 /var/log/haproxy-status.log
-rw------- 1 root root 1980957 Oct 28 20:45 /var/log/haproxy-traffic.log
[root@<hostname> ~]#
ls -l /etc/rsyslog.d/haproxy.conf
-rw-r--r-- 1 root root 265 Jan 14  2020 /etc/rsyslog.d/haproxy.conf

/etc/rsyslog.d/haproxy.conf

# Collect log with UDP
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
 
# Creating separate log files based on the severity
local0.* /var/log/haproxy-traffic.log
local0.notice /var/log/haproxy-admin.log
 
# don't put anything in /var/log/messages
& stop

rsyslog.conf file -

[root@<hostname> ~]# cat /etc/rsyslog.conf |grep -v '#'

$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
local6.*,auth.*,authpriv.*      @10.1.x.x

/etc/haproxy/haproxy.cfg file -

global
    user haproxy
    group haproxy
    daemon
    stats socket 127.0.0.1:14567
    log 127.0.0.1:514 local0 debug
 
defaults
    mode http
    log /dev/log local1 notice
 
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    option httpclose
    option forwardfor except 127.0.0.0/8
    option redispatch
    option abortonclose
    option httplog
    option dontlognull
    .
    .

both ports mentioned in haproxy config is up.


[root@<hostname> ~]#  netstat -tulnp |grep 514
udp        0      0 127.0.0.1:514           0.0.0.0:*                           1309/rsyslogd
[root@<hostname> ~]#
 
[root@<hostname> ~]#  netstat -antp |grep 14567
tcp        0      0 127.0.0.1:14567         0.0.0.0:*               LISTEN      1347/haproxy
[root@<hostname> ~]#

logs -

journalctl -fu rsyslog

Oct 28 15:00:10 <hostname> rsyslogd[1309]: imjournal: journal reloaded... [v8.24.0-57.el7_9.1 try http://www.rsyslog.com/e/0 ]
Oct 28 20:40:54 <hostname> rsyslogd[1309]: imjournal: journal reloaded... [v8.24.0-57.el7_9.1 try http://www.rsyslog.com/e/0 ]

[root@<hostname> ~]# journalctl -fu haproxy

-- Logs begin at Wed 2021-10-27 02:30:13 +03. --
Oct 27 20:28:29 <hostname> haproxy[1347]: Server tomcat-apps/App_server1 is DOWN, reason: Layer7 timeout, check duration: 2001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

Access logs are also appearing in message log.

cat /var/log/messages | grep haproxy

Oct 28 21:27:47 localhost haproxy[1347]: source_ip:54604 [28/Oct/2021:21:27:46.309] ssl~ logs/es_server1 98/0/1/1364/1550 200 103362 - - --VN 43/0/0/0/0 0/0 "POST /api/telemetry/v2/clusters/_stats HTTP/1.1"
Oct 28 21:28:47 localhost haproxy[1347]: source_ip:54635 [28/Oct/2021:21:28:46.307] ssl~ logs/es_server1 151/0/0/1397/1647 200 103361 - - --VN 43/0/0/0/0 0/0 "POST /api/telemetry/v2/clusters/_stats HTTP/1.1"

Non-working server -

no logs in below files

[root@<hostname> ~]# tail -f  /var/log/haproxy*.log 
==> /var/log/haproxy.log <==
 
==> /var/log/haproxy-status.log <==
 
==> /var/log/haproxy-traffic.log <==

ownership is same as above

[root@<hostname> ~]# ls -l /var/log/haproxy*.log
-rw------- 1 root root 0 Sep  4  2020 /var/log/haproxy.log
-rw------- 1 root root 0 Sep  4  2020 /var/log/haproxy-status.log
-rw------- 1 root root 0 Oct 28 20:23 /var/log/haproxy-traffic.log
[root@<hostname> ~]#
ls -l /etc/rsyslog.d/haproxy.conf
-rw-r--r-- 1 root root 265 Aug  4 20:15 /etc/rsyslog.d/haproxy.conf

/etc/rsyslog.d/haproxy.conf

# Collect log with UDP
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
 
# Creating separate log files based on the severity
local0.* /var/log/haproxy-traffic.log
local0.notice /var/log/haproxy-admin.log
 
# don't put anything in /var/log/messages
& stop

cat /etc/rsyslog.conf |grep -v '#'

$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
local6.*    @@10.1.x.x

/etc/haproxy/haproxy.cfg file

global
    user haproxy
    group haproxy
    daemon
    stats socket 127.0.0.1:14567
    log 127.0.0.1:514 local0 debug
 
defaults
    mode http
    log /dev/log local1 notice
 
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    option httpclose
    option forwardfor except 127.0.0.0/8
    option redispatch
    option abortonclose
    option httplog
    option dontlognull

ports - here some connection are in TIME_WAIT state for port 14567

[root@<hostname> ~]# netstat -tulnp |grep 514
udp        0      0 127.0.0.1:514           0.0.0.0:*                           21740/rsyslogd
[root@<hostname> ~]#
 
 
[root@<hostname> ~]# netstat -antp |grep 14567
tcp        0      0 127.0.0.1:14567         0.0.0.0:*               LISTEN      18749/haproxy
tcp        0      0 127.0.0.1:14567         127.0.0.1:36168         TIME_WAIT   -
tcp        0      0 127.0.0.1:14567         127.0.0.1:36170         TIME_WAIT   -
tcp        0      0 127.0.0.1:14567         127.0.0.1:36242         TIME_WAIT   -
tcp        0      0 127.0.0.1:14567         127.0.0.1:36348         TIME_WAIT   -
tcp        0      0 127.0.0.1:14567         127.0.0.1:36332         TIME_WAIT   -

Checked the logs but not finding anything related to this issue.

journalctl -fu rsyslog
 
Oct 28 21:00:01 <hostname> rsyslogd[21740]: action 'action 9' resumed (module 'builtin:omfwd') [v8.24.0-57.el7_9.1 try http://www.rsyslog.com/e/2359 ]
Oct 28 21:00:01 <hostname> rsyslogd[21740]: action 'action 9' resumed (module 'builtin:omfwd') [v8.24.0-57.el7_9.1 try http://www.rsyslog.com/e/2359 ]
[root@<hostname> ~]#  journalctl -fu haproxy
-- Logs begin at Thu 2021-10-28 10:23:25 +03. --
Oct 28 16:34:17 <hostname> haproxy[11346]: Proxy stats started.
Oct 28 19:59:29 <hostname> systemd[1]: Stopping HAProxy Load Balancer...
Oct 28 19:59:29 <hostname> haproxy-systemd-wrapper[11345]: haproxy-systemd-wrapper: SIGTERM -> 11347.
Oct 28 19:59:29 <hostname> haproxy[18748]: Proxy ssl started.
Oct 28 19:59:29 <hostname> haproxy[18748]: Proxy stats started.

here message log file has haproxy access log but its not going to its individual file.

cat /var/log/messages | grep  haproxy
 
Oct 28 10:19:52 <hostname> audispd: node=<hostname> type=SERVICE_START msg=audit(1635405592.054:45): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=haproxy comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 28 10:19:52 <hostname> haproxy[1749]: Proxy ssl-pingaccess started.
Oct 28 10:19:52 <hostname> haproxy[1749]: Proxy ssl started.
 
Oct 28 10:19:52 <hostname> haproxy[1749]: Server ssl-pingaccess/192.168.157.41 is DOWN, reason: Layer4 connection problem, info: "Connection error during SSL handshake (Connection refused) at initial connection step of tcp-check", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Oct 28 10:19:52 <hostname> haproxy[1749]: backend ssl-pingaccess has no server available!

These files are same on both servers

both are same
 
/usr/lib/systemd/system/rsyslog.service
 
/usr/lib/systemd/system/haproxy.service
 
/etc/sysconfig/haproxy
 
/etc/sysconfig/rsyslog
 
/etc/logrotate.d/haproxy

Both have rsyslog-8.24.0-57.el7_9.1.x86_64 and haproxy-1.5.18-9.el7_9.1.x86_64 package.

Configuration is almost same on both the servers but on non-working servers, logs are not going to its own log file.

Can anyone please help.

Thanks,

Hi All,

Can someone please reply.

Thanks,

After adding below line under frontend section, logs are now generating in log file.

log 127.0.0.1:514 local0 debug