Alert emails not being sent

Help!
1

I have haproxy setup locally and I am using a SMTP server to send email alerts which works fine. I setup the mailers section and then add the config to the backend which sends alerts when any of the 3 backend servers are down.

I moved the configuration to our aws vm and it does not work. I take a backend server down and no alerts are sent. checked the stats page and the health check fails which means haproxy sees it as down and an alert should be sent. I manually connected to the smtp server with telnet and sent an email and it works fine, this means port 25 is open and the vm CAN connect to the SMTP server.

I then installed tcpdump and monitored port 25 then took one of the backend servers down. NO data on port 25 which i think means haproxy is not even trying to connect. any ideas ?
Im using haproxy 1.8
thanks for any help.

mailers mta
mailer smtp1 e2ksmtp01.e2k.mydomain.com:25

backend test_backend
mode tcp
balance leastconn
email-alert mailers mta
email-alert level notice
email-alert from v.san@mydomain.com
email-alert to testing1@mydomain.com

   server node1 1.1.1.1:443 fall 3 rise 2 maxconn 500
   server node2 1.1.3.168:443 fall 3 rise 2 maxconn 500
   server node3 1.1.77.1:443 fall 3 rise 2 maxconn 500
2

So you send an email via telnet from the AWS VM and actually received it too?

There is not much to go with here, I can only guess …

  • please provide the output of haproxy -vv (both from the local and from the AWS VM)
  • is the mailer hostname dual-stacked? Do you have connectivity to both the IPv4 and the IPv6 address of the mail server?
  • other than the backend going down, do you have any other messages in the haproxy log?
3

just to make sure it was not 1.8 i downgraded to 1.7 to see if it would work and no, same issue.

$ haproxy -vv
HA-Proxy version 1.7.11 2018/04/30
Copyright 2000-2018 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe

My local version is 1.5.
Mailer hostname is not dual stacked i believe. yes I can manually connect to the smtp server with telnet, and manually send an email, and yes I receive the email.

here is the haproxy.log after a fresh restart

Oct 25 13:03:27 localhost haproxy[23183]: Proxy api started.
Oct 25 13:03:27 localhost haproxy[23183]: Proxy test_backend started.
Oct 25 13:03:27 localhost haproxy[23183]: Proxy 429_slow_down started.
Oct 25 13:03:27 localhost haproxy[23183]: Proxy stats started.
Oct 25 13:03:27 localhost haproxy[23184]: Server test_backend/node1 is DOWN, reason: Layer4 connection problem, info: “Connection refused”, check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Oct 25 13:03:28 localhost haproxy[23184]: Server test_backend/node3 is DOWN, reason: Layer4 connection problem, info: “Connection refused”, check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Oct 25 13:03:29 localhost haproxy[23184]: Connect from 127.0.0.1:39058 to 127.0.0.1:8080 (stats/HTTP)
~

so it sees those 2 backend servers are down but its not attempting reaching out to the SMTP server.

4

So what works is haproxy 1.5 locally, and on AWS neither 1.7 nor 1.8 works.

Can you try if 1.7 or 1.8 are working locally?

Please check anyway, it’s a single command from the AWS VM:
host <hostname>

5

its not , hostname maps to 1 ip address.
I will install 1.7 and try that locally. seems to be some issue from AWS only. Do you know a way to test why haproxy would not try to connect to smtp server ? i think running tcpdump i should see it try to connect on port 25 but I dont. nothing shows.

6

I can’t think of a reason why this would not work in AWS. 1.7 and 1.8 did have bugs regarding email-alerts, but everything is fixed in 1.7.11+ and 1.8.6+.

7

ok thanks for trying.

8

forgot last question. Should I see in the haproxy.log the attempted connection to the SMTP server ?

9

The next step in troubleshooting is for you to try 1.7 locally. If that works, you’ll have to strace -tt it on AWS.

I’m not sure if, how and when the mail code should produce a log. I assumed that when it fails, something may be in there.