Haproxy causes "Communications link failure" with JDBC and MySQL

Help!
1

I’m running a dockerized metabase instance on rancher server and I’m using haproxy to front the application. Everything runs pretty fine if not that upon first page hit I get the following error page:

HTTP ERROR: 500
Problem accessing /. Reason:
    Communications link failure
The last packet successfully received from the server was 1,758,522 milliseconds ago.  The last packet sent successfully to the server was 2 milliseconds ago.

The fun thing is that a refresh fixes the issue, which comes back after a period of inactivity.
Any suggestion please?

2

Share more informations:

  • output of haproxy -vv
  • haproxy configuration
  • the output of haproxy logs
3

This is the most confusing part, as haproxy logs don’t tell much:

2/24/2017 4:49:14 PMtime="2017-02-24T15:49:14Z" level=info msg="Starting Rancher LB service"
2/24/2017 4:49:14 PMtime="2017-02-24T15:49:14Z" level=info msg="LB controller: rancher"
2/24/2017 4:49:14 PMtime="2017-02-24T15:49:14Z" level=info msg="LB provider: haproxy"
2/24/2017 4:49:14 PMtime="2017-02-24T15:49:14Z" level=info msg="starting rancher controller"
2/24/2017 4:49:14 PMtime="2017-02-24T15:49:14Z" level=info msg="Healthcheck handler is listening on :10241"
2/24/2017 4:49:16 PMtime="2017-02-24T15:49:16Z" level=info msg=" -- starting haproxy\n[WARNING] 054/154915 (28) : config : 'option forwardfor' ignored for proxy 'default' as it requires HTTP mode.\n"

Metabase is being more helpful:

2/24/2017 3:50:00 PM	at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:850)
2/24/2017 3:50:00 PM	at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:818)
2/24/2017 3:50:00 PM	at clojure.java.jdbc$query.invokeStatic(jdbc.clj:874)
2/24/2017 3:50:00 PM	at clojure.java.jdbc$query.invoke(jdbc.clj:854)
2/24/2017 3:50:00 PM	at metabase.db$query.invokeStatic(db.clj:534)
2/24/2017 3:50:00 PM	at metabase.db$query.doInvoke(db.clj:530)
2/24/2017 3:50:00 PM	at clojure.lang.RestFn.invoke(RestFn.java:410)
2/24/2017 3:50:00 PM	at metabase.db$simple_select.invokeStatic(db.clj:593)
2/24/2017 3:50:00 PM	at metabase.db$simple_select.invoke(db.clj:585)
2/24/2017 3:50:00 PM	at metabase.db$select.invokeStatic(db.clj:836)
2/24/2017 3:50:00 PM	at metabase.db$select.doInvoke(db.clj:830)
2/24/2017 3:50:00 PM	at clojure.lang.RestFn.invoke(RestFn.java:439)
2/24/2017 3:50:00 PM	at metabase.task.sync_databases.SyncDatabases.execute(sync_databases.clj:21)
2/24/2017 3:50:00 PM	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
2/24/2017 3:50:00 PM	... 1 more
2/24/2017 3:50:00 PMCaused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2/24/2017 3:50:00 PM	at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3004)
2/24/2017 3:50:00 PM	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3462)
2/24/2017 3:50:00 PM	... 24 more
2/24/2017 4:50:00 PM02-24 15:50:00 ERROR core.JobRunShell :: Job DEFAULT.metabase.task.sync-databases.job threw an unhandled Exception:
2/24/2017 4:50:00 PMcom.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
2/24/2017 4:50:00 PM
2/24/2017 4:50:00 PMThe last packet successfully received from the server was 2,999,993 milliseconds ago.  The last packet sent successfully to the server was 1 milliseconds ago.
2/24/2017 4:50:00 PM	at sun.reflect.GeneratedConstructorAccessor62.newInstance(Unknown Source)
2/24/2017 4:50:00 PM	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2/24/2017 4:50:00 PM	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:988)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3552)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3452)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3893)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1962)
2/24/2017 4:50:00 PM	at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
2/24/2017 4:50:00 PM	at clojure.java.jdbc$db_query_with_resultset$run_query_with_params__2997.invoke(jdbc.clj:831)
2/24/2017 4:50:00 PM	at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:850)
2/24/2017 4:50:00 PM	at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:818)
2/24/2017 4:50:00 PM	at clojure.java.jdbc$query.invokeStatic(jdbc.clj:874)
2/24/2017 4:50:00 PM	at clojure.java.jdbc$query.invoke(jdbc.clj:854)
2/24/2017 4:50:00 PM	at metabase.db$query.invokeStatic(db.clj:534)
2/24/2017 4:50:00 PM	at metabase.db$query.doInvoke(db.clj:530)
2/24/2017 4:50:00 PM	at clojure.lang.RestFn.invoke(RestFn.java:410)
2/24/2017 4:50:00 PM	at metabase.db$simple_select.invokeStatic(db.clj:593)
2/24/2017 4:50:00 PM	at metabase.db$simple_select.invoke(db.clj:585)
2/24/2017 4:50:00 PM	at metabase.db$select.invokeStatic(db.clj:836)
2/24/2017 4:50:00 PM	at metabase.db$select.doInvoke(db.clj:830)
2/24/2017 4:50:00 PM	at clojure.lang.RestFn.invoke(RestFn.java:439)
2/24/2017 4:50:00 PM	at metabase.task.sync_databases.SyncDatabases.execute(sync_databases.clj:21)
2/24/2017 4:50:00 PM	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
2/24/2017 4:50:00 PM	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
2/24/2017 4:50:00 PMCaused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3004)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3462)
2/24/2017 4:50:00 PM	... 24 more
2/24/2017 4:50:00 PM02-24 15:50:00 ERROR core.ErrorLogger :: Job (DEFAULT.metabase.task.sync-databases.job threw an exception.
2/24/2017 4:50:00 PMorg.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
2/24/2017 4:50:00 PM
2/24/2017 4:50:00 PMThe last packet successfully received from the server was 2,999,993 milliseconds ago.  The last packet sent successfully to the server was 1 milliseconds ago.]
2/24/2017 4:50:00 PM	at org.quartz.core.JobRunShell.run(JobRunShell.java:224)
2/24/2017 4:50:00 PM	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
2/24/2017 4:50:00 PMCaused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
2/24/2017 4:50:00 PM
2/24/2017 4:50:00 PMThe last packet successfully received from the server was 2,999,993 milliseconds ago.  The last packet sent successfully to the server was 1 milliseconds ago.
2/24/2017 4:50:00 PM	at sun.reflect.GeneratedConstructorAccessor62.newInstance(Unknown Source)
2/24/2017 4:50:00 PM	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2/24/2017 4:50:00 PM	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:988)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3552)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3452)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3893)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1962)
2/24/2017 4:50:00 PM	at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
2/24/2017 4:50:00 PM	at clojure.java.jdbc$db_query_with_resultset$run_query_with_params__2997.invoke(jdbc.clj:831)
2/24/2017 4:50:00 PM	at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:850)
2/24/2017 4:50:00 PM	at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:818)
2/24/2017 4:50:00 PM	at clojure.java.jdbc$query.invokeStatic(jdbc.clj:874)
2/24/2017 4:50:00 PM	at clojure.java.jdbc$query.invoke(jdbc.clj:854)
2/24/2017 4:50:00 PM	at metabase.db$query.invokeStatic(db.clj:534)
2/24/2017 4:50:00 PM	at metabase.db$query.doInvoke(db.clj:530)
2/24/2017 4:50:00 PM	at clojure.lang.RestFn.invoke(RestFn.java:410)
2/24/2017 4:50:00 PM	at metabase.db$simple_select.invokeStatic(db.clj:593)
2/24/2017 4:50:00 PM	at metabase.db$simple_select.invoke(db.clj:585)
2/24/2017 4:50:00 PM	at metabase.db$select.invokeStatic(db.clj:836)
2/24/2017 4:50:00 PM	at metabase.db$select.doInvoke(db.clj:830)
2/24/2017 4:50:00 PM	at clojure.lang.RestFn.invoke(RestFn.java:439)
2/24/2017 4:50:00 PM	at metabase.task.sync_databases.SyncDatabases.execute(sync_databases.clj:21)
2/24/2017 4:50:00 PM	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
2/24/2017 4:50:00 PM	... 1 more
2/24/2017 4:50:00 PMCaused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3004)
2/24/2017 4:50:00 PM	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3462)
2/24/2017 4:50:00 PM	... 24 more

So far I’ve been able to only mitigate a lot the issue (not solve it) with this config for haproxy which I grabbed and tweaked from this thread.

default
    timeout connect 1000
    timeout check 1000
    timeout client 10000
    timeout server 10000

frontend http-frontend
    bind *:80
    mode http
    redirect scheme https code 302 if !{ ssl_fc }

Here I found the TTL settings on the sql driver connection pools for metabase.

And the output of haproxy -vv from inside the container:

HA-Proxy version 1.6.11-1ppa1~xenial 2016/12/27
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-securi
ty -Wdate-time -D_FORTIFY_SOURCE=2
  OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw
-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.38 2015-11-23
Running on PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_F
REEBIND
Built with network namespace support

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.
4

Probably haproxy is not setup correctly for logging.

Please share the complete configuration. You can replace private data like IP addresses and hostnames with placeholders, but unless we can see the actual configuration, its gonna be difficult to provide any help.

5

I see what you mean @lukastribus, sorry about that, but I’m not actually hiding anything. Haproxy is run in a container by Rancher and here’s a screenshot attached with almost all there is related to configuration. But I can open a shell and inside of the container and tell you all you need to.

screenshot.png2524×2854 141 KB

6

Thats the same you posted above. You gonna have to post the content of the config file that rancher actually generates on the filesystem.

7

Thank you so much for your support on this :blush:
Here’s what I found:

global
    chroot /var/lib/haproxy
    daemon
    group haproxy
    maxconn 4096
    maxpipes 1024
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
    ssl-default-bind-options no-sslv3 no-tlsv10
    ssl-default-server-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
    tune.ssl.default-dh-param 2048
    user haproxy

defaults
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http
    maxconn 4096
    mode tcp
    option forwardfor
    option http-server-close
    option redispatch
    retries 3
    timeout client 50000
    timeout connect 5000
    timeout server 50000

frontend http-frontend
        bind *:80
        mode http
        redirect scheme https code 302 if !{ ssl_fc }


resolvers rancher
 nameserver dnsmasq 169.254.169.250:53

listen default
bind *:42

frontend 443
bind *:443 ssl crt /etc/haproxy/certs/current
mode http
acl 443_XXX_XXXXXXXXX_XXX__host hdr(host) -i XXX.XXXXXXXXX.XXX
acl 443_XXX_XXXXXXXXX_XXX__host hdr(host) -i XXX.XXXXXXXXX.XXX:443
acl 443_XXX_XXXXXXXXX_XXX__path path_beg -i /
use_backend 443_XXX_XXXXXXXXX_XXX_ if 443_XXX_XXXXXXXXX_XXX__host 443_XXX_XXXXXXXXX_XXX__path
acl 443_XXXX_XXXXXXXXX_XXX__host hdr(host) -i XXXX.XXXXXXXXX.XXX
acl 443_XXXX_XXXXXXXXX_XXX__host hdr(host) -i XXXX.XXXXXXXXX.XXX:443
acl 443_XXXX_XXXXXXXXX_XXX__path path_beg -i /
use_backend 443_XXXX_XXXXXXXXX_XXX_ if 443_XXXX_XXXXXXXXX_XXX__host 443_XXXX_XXXXXXXXX_XXX__path

backend 443_XXX_XXXXXXXXX_XXX_
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    timeout check 2000
mode http
server cf30e808b48e285aa06bc60564eee154e2fb5281 10.42.86.210:3100  check port 3100 inter 2000 rise 2 fall 3

backend 443_XXXX_XXXXXXXXX_XXX_
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
    http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
    http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
    timeout check 2000
mode http
server cd08d0098117003700fda5e439fd35466f7cef8c 10.42.122.58:3000  check port 3000 inter 2000 rise 2 fall 3
8

There is no logging configured, you will have to set it up, so that we can see if/what timeout is triggering:

https://cbonte.github.io/haproxy-dconv/1.6/configuration.html#8

You probably want to send your message via IP/UDP to a syslog server, as you run in chroot.

9

Ok so I configured logs like so:

global
    log 127.0.0.1:8514 local0
    log 127.0.0.1:8514 local1 notice

defaults
    log global
    mode http
    option httplog

I triggered the error on metabase and got back to check the logs, but no error was found:

ls -l /var/log/haproxy/
total 20
-rw-r--r-- 1 root root    0 Jan 24 22:27 errors
-rw-r--r-- 1 root root 6031 Feb 26 23:23 events
-rw-r--r-- 1 root root 8224 Feb 26 23:34 traffic

These are the events logged:

Feb 26 23:10:40 localhost haproxy[169]: Proxy http-frontend started.
Feb 26 23:10:40 localhost haproxy[169]: Proxy default started.
Feb 26 23:10:40 localhost haproxy[169]: Proxy 443 started.
Feb 26 23:10:40 localhost haproxy[169]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:10:40 localhost haproxy[169]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:10:48 localhost haproxy[31]: Proxy http-frontend started.
Feb 26 23:10:48 localhost haproxy[31]: Proxy default started.
Feb 26 23:10:48 localhost haproxy[31]: Proxy 443 started.
Feb 26 23:10:48 localhost haproxy[31]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:10:48 localhost haproxy[31]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy http-frontend started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy http-frontend started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy default started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy default started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy 443 started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy 443 started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:28 localhost haproxy[32]: Stopping frontend http-frontend in 0 ms.
Feb 26 23:20:28 localhost haproxy[32]: Stopping proxy default in 0 ms.
Feb 26 23:20:28 localhost haproxy[32]: Stopping frontend 443 in 0 ms.
Feb 26 23:20:28 localhost haproxy[32]: Stopping backend 443_XXX_XXXXXXXXX_XXX_ in 0 ms.
Feb 26 23:20:28 localhost haproxy[32]: Stopping backend 443_XXXX_XXXXXXXXX_XXX_ in 0 ms.
Feb 26 23:20:28 localhost haproxy[32]: Proxy http-frontend stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:20:28 localhost haproxy[32]: Proxy default stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:20:28 localhost haproxy[32]: Proxy 443 stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:20:28 localhost haproxy[32]: Proxy 443_XXX_XXXXXXXXX_XXX_ stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:20:28 localhost haproxy[32]: Proxy 443_XXXX_XXXXXXXXX_XXX_ stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:20:36 localhost haproxy[29]: Proxy http-frontend started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy http-frontend started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy default started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy default started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy 443 started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy 443 started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy http-frontend started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy http-frontend started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy default started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy default started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy 443 started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy 443 started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:23 localhost haproxy[30]: Stopping frontend http-frontend in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping frontend http-frontend in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping proxy default in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping proxy default in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping frontend 443 in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping frontend 443 in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping backend 443_XXX_XXXXXXXXX_XXX_ in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping backend 443_XXX_XXXXXXXXX_XXX_ in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping backend 443_XXXX_XXXXXXXXX_XXX_ in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping backend 443_XXXX_XXXXXXXXX_XXX_ in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Proxy http-frontend stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy http-frontend stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy default stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy default stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy 443 stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy 443 stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy 443_XXX_XXXXXXXXX_XXX_ stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy 443_XXX_XXXXXXXXX_XXX_ stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy 443_XXXX_XXXXXXXXX_XXX_ stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy 443_XXXX_XXXXXXXXX_XXX_ stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:30 localhost haproxy[29]: Proxy http-frontend started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy http-frontend started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy default started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy default started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy 443 started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy 443 started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.

And traffic:

Feb 26 23:23:38 localhost haproxy[30]: XXX.XXX.XXX.XXX:53291 [26/Feb/2017:23:23:38.406] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 424/0/1/145/570 500 863 - - ---- 6/6/0/0/0 0/0 "GET / HTTP/1.1"
Feb 26 23:23:39 localhost haproxy[30]: XXX.XXX.XXX.XXX:53291 [26/Feb/2017:23:23:38.977] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 233/0/1/68/309 200 1635 - - ---- 5/5/0/0/0 0/0 "GET /favicon.ico HTTP/1.1"
Feb 26 23:23:42 localhost haproxy[30]: XXX.XXX.XXX.XXX:53294 [26/Feb/2017:23:23:38.406] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 4027/0/0/114/4229 200 4683 - - ---- 5/5/0/0/0 0/0 "GET / HTTP/1.1"
Feb 26 23:23:43 localhost haproxy[30]: XXX.XXX.XXX.XXX:53293 [26/Feb/2017:23:23:38.406] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 4903/0/1/101/5005 500 885 - - ---- 5/5/1/2/0 0/0 "GET /api/session/properties HTTP/1.1"
Feb 26 23:23:43 localhost haproxy[30]: XXX.XXX.XXX.XXX:53296 [26/Feb/2017:23:23:38.413] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 4890/0/1/111/5003 200 589 - - ---- 4/4/0/0/0 0/0 "GET /api/user/current HTTP/1.1"
Feb 26 23:23:43 localhost haproxy[30]: XXX.XXX.XXX.XXX:53293 [26/Feb/2017:23:23:43.412] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 142/0/0/41/184 200 589 - - ---- 3/3/0/0/0 0/0 "GET /api/user/current HTTP/1.1"
Feb 26 23:23:43 localhost haproxy[30]: XXX.XXX.XXX.XXX:53292 [26/Feb/2017:23:23:38.415] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 5219/0/0/24/5333 200 1635 - - ---- 2/2/0/0/0 0/0 "GET /favicon.ico HTTP/1.1"
Feb 26 23:23:44 localhost haproxy[30]: XXX.XXX.XXX.XXX:53299 [26/Feb/2017:23:23:42.324] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1463/0/1/136/1692 200 1033 - - ---- 3/3/1/1/0 0/0 "GET /api/setup/admin_checklist HTTP/1.1"
Feb 26 23:23:44 localhost haproxy[30]: XXX.XXX.XXX.XXX:53295 [26/Feb/2017:23:23:38.412] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 5342/0/1/601/5944 200 971 - - ---- 2/2/2/2/0 0/0 "GET /api/dashboard?f=all HTTP/1.1"
Feb 26 23:23:44 localhost haproxy[30]: XXX.XXX.XXX.XXX:53303 [26/Feb/2017:23:23:43.779] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 277/0/0/734/1095 200 787 - - ---- 1/1/1/1/0 0/0 "GET /api/activity/recent_views HTTP/1.1"
Feb 26 23:23:45 localhost haproxy[30]: XXX.XXX.XXX.XXX:53302 [26/Feb/2017:23:23:43.775] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 269/0/0/1190/1464 200 2222 - - ---- 0/0/0/0/0 0/0 "GET /api/activity HTTP/1.1"
Feb 26 23:34:00 localhost haproxy[30]: XXX.XXX.XXX.XXX:53727 [26/Feb/2017:23:33:59.490] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 615/0/0/46/661 500 874 - - ---- 1/1/0/0/0 0/0 "GET /api/dashboard/1 HTTP/1.1"
Feb 26 23:34:12 localhost haproxy[30]: XXX.XXX.XXX.XXX:53737 [26/Feb/2017:23:34:12.325] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 360/0/1/117/479 200 4683 - - ---- 5/5/0/0/0 0/0 "GET /dash/1 HTTP/1.1"
Feb 26 23:34:13 localhost haproxy[30]: XXX.XXX.XXX.XXX:53738 [26/Feb/2017:23:34:12.326] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1218/0/0/46/1264 500 881 - - ---- 5/5/1/1/0 0/0 "GET /api/session/properties HTTP/1.1"
Feb 26 23:34:13 localhost haproxy[30]: XXX.XXX.XXX.XXX:53736 [26/Feb/2017:23:34:12.327] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1212/0/1/54/1336 200 589 - - ---- 4/4/0/0/0 0/0 "GET /api/user/current HTTP/1.1"
Feb 26 23:34:13 localhost haproxy[30]: XXX.XXX.XXX.XXX:53738 [26/Feb/2017:23:34:13.591] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 87/0/1/20/193 200 589 - - ---- 3/3/0/0/0 0/0 "GET /api/user/current HTTP/1.1"
Feb 26 23:34:13 localhost haproxy[30]: XXX.XXX.XXX.XXX:53741 [26/Feb/2017:23:34:12.333] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1529/0/1/23/1563 200 1635 - - ---- 2/2/2/2/0 0/0 "GET /favicon.ico HTTP/1.1"
Feb 26 23:34:14 localhost haproxy[30]: XXX.XXX.XXX.XXX:53739 [26/Feb/2017:23:34:12.329] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1491/0/1/454/2037 200 971 - - ---- 1/1/1/1/0 0/0 "GET /api/dashboard?f=all HTTP/1.1"
Feb 26 23:34:15 localhost haproxy[30]: XXX.XXX.XXX.XXX:53740 [26/Feb/2017:23:34:12.333] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1493/0/0/1239/2732 500 2873 - - ---- 0/0/0/0/0 0/0 "GET /api/dashboard/1 HTTP/1.1"
Feb 26 23:34:26 localhost haproxy[30]: XXX.XXX.XXX.XXX:53752 [26/Feb/2017:23:34:26.252] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 386/0/0/53/444 200 4683 - - ---- 5/5/0/0/0 0/0 "GET /dash/1 HTTP/1.1"
Feb 26 23:34:27 localhost haproxy[30]: XXX.XXX.XXX.XXX:53753 [26/Feb/2017:23:34:26.256] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1364/0/0/29/1394 200 2933 - - ---- 4/4/1/0/0 0/0 "GET /api/session/properties HTTP/1.1"
Feb 26 23:34:27 localhost haproxy[30]: XXX.XXX.XXX.XXX:53755 [26/Feb/2017:23:34:26.254] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1356/0/0/28/1479 200 589 - - ---- 3/3/0/0/0 0/0 "GET /api/user/current HTTP/1.1"
Feb 26 23:34:27 localhost haproxy[30]: XXX.XXX.XXX.XXX:53756 [26/Feb/2017:23:34:26.256] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1586/0/1/22/1610 200 589 - - ---- 2/2/0/0/0 0/0 "GET /api/user/current HTTP/1.1"
Feb 26 23:34:27 localhost haproxy[30]: XXX.XXX.XXX.XXX:53754 [26/Feb/2017:23:34:26.259] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1632/0/0/11/1644 200 1635 - - ---- 1/1/0/0/0 0/0 "GET /favicon.ico HTTP/1.1"
Feb 26 23:34:28 localhost haproxy[30]: XXX.XXX.XXX.XXX:53751 [26/Feb/2017:23:34:26.255] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1849/0/1/194/2047 200 971 - - ---- 1/1/0/0/0 0/0 "GET /api/dashboard?f=all HTTP/1.1"
Feb 26 23:34:28 localhost haproxy[30]: XXX.XXX.XXX.XXX:53759 [26/Feb/2017:23:34:28.047] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 378/0/1/80/461 200 2045 - - ---- 0/0/0/0/0 0/0 "GET /api/dashboard/1 HTTP/1.1"
Feb 26 23:34:29 localhost haproxy[30]: XXX.XXX.XXX.XXX:53762 [26/Feb/2017:23:34:28.696] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 357/0/0/266/727 200 6030 - - ---- 5/5/4/4/0 0/0 "GET /api/database/2/metadata HTTP/1.1"
Feb 26 23:34:31 localhost haproxy[30]: XXX.XXX.XXX.XXX:53763 [26/Feb/2017:23:34:28.692] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 643/0/0/1589/2328 200 907 - - ---- 5/5/5/4/0 0/0 "POST /api/card/1/query HTTP/1.1"
Feb 26 23:34:31 localhost haproxy[30]: XXX.XXX.XXX.XXX:53761 [26/Feb/2017:23:34:28.691] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 663/0/1/1580/2332 200 843 - - ---- 4/4/4/4/0 0/0 "POST /api/card/3/query HTTP/1.1"
Feb 26 23:34:31 localhost haproxy[30]: XXX.XXX.XXX.XXX:53760 [26/Feb/2017:23:34:28.691] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 652/0/0/1797/2544 200 904 - - ---- 5/5/3/3/0 0/0 "POST /api/card/2/query HTTP/1.1"
Feb 26 23:34:31 localhost haproxy[30]: XXX.XXX.XXX.XXX:53765 [26/Feb/2017:23:34:28.695] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1516/0/1/1235/2856 200 1136 - - ---- 4/4/2/1/0 0/0 "POST /api/card/6/query HTTP/1.1"
Feb 26 23:34:31 localhost haproxy[30]: XXX.XXX.XXX.XXX:53764 [26/Feb/2017:23:34:28.695] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 665/0/0/2179/2937 200 1166 - - ---- 3/3/1/1/0 0/0 "POST /api/card/4/query HTTP/1.1"
Feb 26 23:34:31 localhost haproxy[30]: XXX.XXX.XXX.XXX:53769 [26/Feb/2017:23:34:29.515] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 215/0/0/1948/2261 200 1319 - - ---- 2/2/0/0/0 0/0 "POST /api/card/5/query HTTP/1.1"
Feb 26 23:34:32 localhost haproxy[30]: XXX.XXX.XXX.XXX:53772 [26/Feb/2017:23:34:31.111] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 756/0/0/110/972 200 866 - - ---- 1/1/1/1/0 0/0 "POST /api/card/84/query HTTP/1.1"
Feb 26 23:34:32 localhost haproxy[30]: XXX.XXX.XXX.XXX:53771 [26/Feb/2017:23:34:31.107] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 899/0/0/290/1270 200 1067 - - ---- 0/0/0/0/0 0/0 "POST /api/card/7/query HTTP/1.1"
10

Ok so I configured logs like this:

global
    log 127.0.0.1:8514 local0
    log 127.0.0.1:8514 local1 notice

defaults
    log global
    mode http
    option httplog

I triggered the error on metabase and got back to check the logs, but no error was found:

ls -l /var/log/haproxy/
total 20
-rw-r--r-- 1 root root    0 Jan 24 22:27 errors
-rw-r--r-- 1 root root 6031 Feb 26 23:23 events
-rw-r--r-- 1 root root 8224 Feb 26 23:34 traffic

These are the events logged:

Feb 26 23:10:40 localhost haproxy[169]: Proxy http-frontend started.
Feb 26 23:10:40 localhost haproxy[169]: Proxy default started.
Feb 26 23:10:40 localhost haproxy[169]: Proxy 443 started.
Feb 26 23:10:40 localhost haproxy[169]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:10:40 localhost haproxy[169]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:10:48 localhost haproxy[31]: Proxy http-frontend started.
Feb 26 23:10:48 localhost haproxy[31]: Proxy default started.
Feb 26 23:10:48 localhost haproxy[31]: Proxy 443 started.
Feb 26 23:10:48 localhost haproxy[31]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:10:48 localhost haproxy[31]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy http-frontend started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy http-frontend started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy default started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy default started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy 443 started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy 443 started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:28 localhost haproxy[78]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:28 localhost haproxy[32]: Stopping frontend http-frontend in 0 ms.
Feb 26 23:20:28 localhost haproxy[32]: Stopping proxy default in 0 ms.
Feb 26 23:20:28 localhost haproxy[32]: Stopping frontend 443 in 0 ms.
Feb 26 23:20:28 localhost haproxy[32]: Stopping backend 443_XXX_XXXXXXXXX_XXX_ in 0 ms.
Feb 26 23:20:28 localhost haproxy[32]: Stopping backend 443_XXXX_XXXXXXXXX_XXX_ in 0 ms.
Feb 26 23:20:28 localhost haproxy[32]: Proxy http-frontend stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:20:28 localhost haproxy[32]: Proxy default stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:20:28 localhost haproxy[32]: Proxy 443 stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:20:28 localhost haproxy[32]: Proxy 443_XXX_XXXXXXXXX_XXX_ stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:20:28 localhost haproxy[32]: Proxy 443_XXXX_XXXXXXXXX_XXX_ stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:20:36 localhost haproxy[29]: Proxy http-frontend started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy http-frontend started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy default started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy default started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy 443 started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy 443 started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:20:36 localhost haproxy[29]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy http-frontend started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy http-frontend started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy default started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy default started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy 443 started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy 443 started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:23 localhost haproxy[68]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:23 localhost haproxy[30]: Stopping frontend http-frontend in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping frontend http-frontend in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping proxy default in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping proxy default in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping frontend 443 in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping frontend 443 in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping backend 443_XXX_XXXXXXXXX_XXX_ in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping backend 443_XXX_XXXXXXXXX_XXX_ in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping backend 443_XXXX_XXXXXXXXX_XXX_ in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Stopping backend 443_XXXX_XXXXXXXXX_XXX_ in 0 ms.
Feb 26 23:23:23 localhost haproxy[30]: Proxy http-frontend stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy http-frontend stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy default stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy default stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy 443 stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy 443 stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy 443_XXX_XXXXXXXXX_XXX_ stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy 443_XXX_XXXXXXXXX_XXX_ stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy 443_XXXX_XXXXXXXXX_XXX_ stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:23 localhost haproxy[30]: Proxy 443_XXXX_XXXXXXXXX_XXX_ stopped (FE: 0 conns, BE: 0 conns).
Feb 26 23:23:30 localhost haproxy[29]: Proxy http-frontend started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy http-frontend started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy default started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy default started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy 443 started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy 443 started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Feb 26 23:23:30 localhost haproxy[29]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.

And traffic:

Feb 26 23:23:38 localhost haproxy[30]: XXX.XXX.XXX.XXX:53291 [26/Feb/2017:23:23:38.406] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 424/0/1/145/570 500 863 - - ---- 6/6/0/0/0 0/0 ""GET / HTTP/1.1""
Feb 26 23:23:39 localhost haproxy[30]: XXX.XXX.XXX.XXX:53291 [26/Feb/2017:23:23:38.977] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 233/0/1/68/309 200 1635 - - ---- 5/5/0/0/0 0/0 ""GET /favicon.ico HTTP/1.1""
Feb 26 23:23:42 localhost haproxy[30]: XXX.XXX.XXX.XXX:53294 [26/Feb/2017:23:23:38.406] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 4027/0/0/114/4229 200 4683 - - ---- 5/5/0/0/0 0/0 ""GET / HTTP/1.1""
Feb 26 23:23:43 localhost haproxy[30]: XXX.XXX.XXX.XXX:53293 [26/Feb/2017:23:23:38.406] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 4903/0/1/101/5005 500 885 - - ---- 5/5/1/2/0 0/0 ""GET /api/session/properties HTTP/1.1""
Feb 26 23:23:43 localhost haproxy[30]: XXX.XXX.XXX.XXX:53296 [26/Feb/2017:23:23:38.413] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 4890/0/1/111/5003 200 589 - - ---- 4/4/0/0/0 0/0 ""GET /api/user/current HTTP/1.1""
Feb 26 23:23:43 localhost haproxy[30]: XXX.XXX.XXX.XXX:53293 [26/Feb/2017:23:23:43.412] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 142/0/0/41/184 200 589 - - ---- 3/3/0/0/0 0/0 ""GET /api/user/current HTTP/1.1""
Feb 26 23:23:43 localhost haproxy[30]: XXX.XXX.XXX.XXX:53292 [26/Feb/2017:23:23:38.415] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 5219/0/0/24/5333 200 1635 - - ---- 2/2/0/0/0 0/0 ""GET /favicon.ico HTTP/1.1""
Feb 26 23:23:44 localhost haproxy[30]: XXX.XXX.XXX.XXX:53299 [26/Feb/2017:23:23:42.324] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1463/0/1/136/1692 200 1033 - - ---- 3/3/1/1/0 0/0 ""GET /api/setup/admin_checklist HTTP/1.1""
Feb 26 23:23:44 localhost haproxy[30]: XXX.XXX.XXX.XXX:53295 [26/Feb/2017:23:23:38.412] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 5342/0/1/601/5944 200 971 - - ---- 2/2/2/2/0 0/0 ""GET /api/dashboard?f=all HTTP/1.1""
Feb 26 23:23:44 localhost haproxy[30]: XXX.XXX.XXX.XXX:53303 [26/Feb/2017:23:23:43.779] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 277/0/0/734/1095 200 787 - - ---- 1/1/1/1/0 0/0 ""GET /api/activity/recent_views HTTP/1.1""
Feb 26 23:23:45 localhost haproxy[30]: XXX.XXX.XXX.XXX:53302 [26/Feb/2017:23:23:43.775] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 269/0/0/1190/1464 200 2222 - - ---- 0/0/0/0/0 0/0 ""GET /api/activity HTTP/1.1""
Feb 26 23:34:00 localhost haproxy[30]: XXX.XXX.XXX.XXX:53727 [26/Feb/2017:23:33:59.490] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 615/0/0/46/661 500 874 - - ---- 1/1/0/0/0 0/0 ""GET /api/dashboard/1 HTTP/1.1""
Feb 26 23:34:12 localhost haproxy[30]: XXX.XXX.XXX.XXX:53737 [26/Feb/2017:23:34:12.325] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 360/0/1/117/479 200 4683 - - ---- 5/5/0/0/0 0/0 ""GET /dash/1 HTTP/1.1""
Feb 26 23:34:13 localhost haproxy[30]: XXX.XXX.XXX.XXX:53738 [26/Feb/2017:23:34:12.326] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1218/0/0/46/1264 500 881 - - ---- 5/5/1/1/0 0/0 ""GET /api/session/properties HTTP/1.1""
Feb 26 23:34:13 localhost haproxy[30]: XXX.XXX.XXX.XXX:53736 [26/Feb/2017:23:34:12.327] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1212/0/1/54/1336 200 589 - - ---- 4/4/0/0/0 0/0 ""GET /api/user/current HTTP/1.1""
Feb 26 23:34:13 localhost haproxy[30]: XXX.XXX.XXX.XXX:53738 [26/Feb/2017:23:34:13.591] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 87/0/1/20/193 200 589 - - ---- 3/3/0/0/0 0/0 ""GET /api/user/current HTTP/1.1""
Feb 26 23:34:13 localhost haproxy[30]: XXX.XXX.XXX.XXX:53741 [26/Feb/2017:23:34:12.333] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1529/0/1/23/1563 200 1635 - - ---- 2/2/2/2/0 0/0 ""GET /favicon.ico HTTP/1.1""
Feb 26 23:34:14 localhost haproxy[30]: XXX.XXX.XXX.XXX:53739 [26/Feb/2017:23:34:12.329] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1491/0/1/454/2037 200 971 - - ---- 1/1/1/1/0 0/0 ""GET /api/dashboard?f=all HTTP/1.1""
Feb 26 23:34:15 localhost haproxy[30]: XXX.XXX.XXX.XXX:53740 [26/Feb/2017:23:34:12.333] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1493/0/0/1239/2732 500 2873 - - ---- 0/0/0/0/0 0/0 ""GET /api/dashboard/1 HTTP/1.1""
Feb 26 23:34:26 localhost haproxy[30]: XXX.XXX.XXX.XXX:53752 [26/Feb/2017:23:34:26.252] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 386/0/0/53/444 200 4683 - - ---- 5/5/0/0/0 0/0 ""GET /dash/1 HTTP/1.1""
Feb 26 23:34:27 localhost haproxy[30]: XXX.XXX.XXX.XXX:53753 [26/Feb/2017:23:34:26.256] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1364/0/0/29/1394 200 2933 - - ---- 4/4/1/0/0 0/0 ""GET /api/session/properties HTTP/1.1""
Feb 26 23:34:27 localhost haproxy[30]: XXX.XXX.XXX.XXX:53755 [26/Feb/2017:23:34:26.254] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1356/0/0/28/1479 200 589 - - ---- 3/3/0/0/0 0/0 ""GET /api/user/current HTTP/1.1""
Feb 26 23:34:27 localhost haproxy[30]: XXX.XXX.XXX.XXX:53756 [26/Feb/2017:23:34:26.256] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1586/0/1/22/1610 200 589 - - ---- 2/2/0/0/0 0/0 ""GET /api/user/current HTTP/1.1""
Feb 26 23:34:27 localhost haproxy[30]: XXX.XXX.XXX.XXX:53754 [26/Feb/2017:23:34:26.259] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1632/0/0/11/1644 200 1635 - - ---- 1/1/0/0/0 0/0 ""GET /favicon.ico HTTP/1.1""
Feb 26 23:34:28 localhost haproxy[30]: XXX.XXX.XXX.XXX:53751 [26/Feb/2017:23:34:26.255] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1849/0/1/194/2047 200 971 - - ---- 1/1/0/0/0 0/0 ""GET /api/dashboard?f=all HTTP/1.1""
Feb 26 23:34:28 localhost haproxy[30]: XXX.XXX.XXX.XXX:53759 [26/Feb/2017:23:34:28.047] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 378/0/1/80/461 200 2045 - - ---- 0/0/0/0/0 0/0 ""GET /api/dashboard/1 HTTP/1.1""
Feb 26 23:34:29 localhost haproxy[30]: XXX.XXX.XXX.XXX:53762 [26/Feb/2017:23:34:28.696] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 357/0/0/266/727 200 6030 - - ---- 5/5/4/4/0 0/0 ""GET /api/database/2/metadata HTTP/1.1""
Feb 26 23:34:31 localhost haproxy[30]: XXX.XXX.XXX.XXX:53763 [26/Feb/2017:23:34:28.692] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 643/0/0/1589/2328 200 907 - - ---- 5/5/5/4/0 0/0 ""POST /api/card/1/query HTTP/1.1""
Feb 26 23:34:31 localhost haproxy[30]: XXX.XXX.XXX.XXX:53761 [26/Feb/2017:23:34:28.691] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 663/0/1/1580/2332 200 843 - - ---- 4/4/4/4/0 0/0 ""POST /api/card/3/query HTTP/1.1""
Feb 26 23:34:31 localhost haproxy[30]: XXX.XXX.XXX.XXX:53760 [26/Feb/2017:23:34:28.691] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 652/0/0/1797/2544 200 904 - - ---- 5/5/3/3/0 0/0 ""POST /api/card/2/query HTTP/1.1""
Feb 26 23:34:31 localhost haproxy[30]: XXX.XXX.XXX.XXX:53765 [26/Feb/2017:23:34:28.695] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 1516/0/1/1235/2856 200 1136 - - ---- 4/4/2/1/0 0/0 ""POST /api/card/6/query HTTP/1.1""
Feb 26 23:34:31 localhost haproxy[30]: XXX.XXX.XXX.XXX:53764 [26/Feb/2017:23:34:28.695] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 665/0/0/2179/2937 200 1166 - - ---- 3/3/1/1/0 0/0 ""POST /api/card/4/query HTTP/1.1""
Feb 26 23:34:31 localhost haproxy[30]: XXX.XXX.XXX.XXX:53769 [26/Feb/2017:23:34:29.515] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 215/0/0/1948/2261 200 1319 - - ---- 2/2/0/0/0 0/0 ""POST /api/card/5/query HTTP/1.1""
Feb 26 23:34:32 localhost haproxy[30]: XXX.XXX.XXX.XXX:53772 [26/Feb/2017:23:34:31.111] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 756/0/0/110/972 200 866 - - ---- 1/1/1/1/0 0/0 ""POST /api/card/84/query HTTP/1.1""
Feb 26 23:34:32 localhost haproxy[30]: XXX.XXX.XXX.XXX:53771 [26/Feb/2017:23:34:31.107] 443~ 443_XXX_XXXXXXXXX_XXX_/86ebec440a5825d229b3934052b42be07b77beee 899/0/0/290/1270 200 1067 - - ---- 0/0/0/0/0 0/0 ""POST /api/card/7/query HTTP/1.1""
11

As of today the errors log is still empty, but the events log shows some problems I think, that I hope can point to the right direction:

Mar  3 17:15:32 localhost haproxy[41]: Proxy http-frontend started.
Mar  3 17:15:32 localhost haproxy[41]: Proxy http-frontend started.
Mar  3 17:15:32 localhost haproxy[41]: Proxy default started.
Mar  3 17:15:32 localhost haproxy[41]: Proxy default started.
Mar  3 17:15:32 localhost haproxy[41]: Proxy 443 started.
Mar  3 17:15:32 localhost haproxy[41]: Proxy 443 started.
Mar  3 17:15:32 localhost haproxy[41]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Mar  3 17:15:32 localhost haproxy[41]: Proxy 443_XXX_XXXXXXXXX_XXX_ started.
Mar  3 17:15:32 localhost haproxy[41]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Mar  3 17:15:32 localhost haproxy[41]: Proxy 443_XXXX_XXXXXXXXX_XXX_ started.
Mar  3 17:15:32 localhost haproxy[42]: Server 443_XXX_XXXXXXXXX_XXX_/d347863e1f7269bbb70506aa6b7e30aaf3a1c592 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Mar  3 17:15:32 localhost haproxy[42]: Server 443_XXX_XXXXXXXXX_XXX_/d347863e1f7269bbb70506aa6b7e30aaf3a1c592 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Mar  3 17:15:32 localhost haproxy[42]: backend 443_XXX_XXXXXXXXX_XXX_ has no server available!
Mar  3 17:15:32 localhost haproxy[42]: backend 443_XXX_XXXXXXXXX_XXX_ has no server available!
Mar  3 17:15:33 localhost haproxy[42]: Server 443_XXXX_XXXXXXXXX_XXX_/d894fd041078668291cb4ce88996721c79a1e3d5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Mar  3 17:15:33 localhost haproxy[42]: Server 443_XXXX_XXXXXXXXX_XXX_/d894fd041078668291cb4ce88996721c79a1e3d5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Mar  3 17:15:33 localhost haproxy[42]: backend 443_XXXX_XXXXXXXXX_XXX_ has no server available!
Mar  3 17:15:33 localhost haproxy[42]: backend 443_XXXX_XXXXXXXXX_XXX_ has no server available!
Mar  3 17:16:04 localhost haproxy[42]: Server 443_XXX_XXXXXXXXX_XXX_/d347863e1f7269bbb70506aa6b7e30aaf3a1c592 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Mar  3 17:16:04 localhost haproxy[42]: Server 443_XXX_XXXXXXXXX_XXX_/d347863e1f7269bbb70506aa6b7e30aaf3a1c592 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Mar  3 17:16:11 localhost haproxy[42]: Server 443_XXXX_XXXXXXXXX_XXX_/d894fd041078668291cb4ce88996721c79a1e3d5 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Mar  3 17:16:11 localhost haproxy[42]: Server 443_XXXX_XXXXXXXXX_XXX_/d894fd041078668291cb4ce88996721c79a1e3d5 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
12

Looks like the health check fails (Connection refused), and therefor, new connections will fail as no servers are available anymore plus there may be an impact to active sessions depending on the root cause of the failure.

Do you see anything in the logs of the backend server (both application and system/kernel logs)?

13

Metabase application logs the following whenever the issue arises, but the MariaDB container involved in the setup is very silent and no error is logged:

3/6/2017 1:50:00 PMCaused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
3/6/2017 1:50:00 PM	at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3004)
3/6/2017 1:50:00 PM	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3462)
3/6/2017 1:50:00 PM	... 24 more
3/6/2017 2:50:00 PM03-06 13:50:00 ERROR core.JobRunShell :: Job DEFAULT.metabase.task.sync-databases.job threw an unhandled Exception:
3/6/2017 2:50:00 PMcom.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
3/6/2017 2:50:00 PM
3/6/2017 2:50:00 PMThe last packet successfully received from the server was 679,975 milliseconds ago.  The last packet sent successfully to the server was 1 milliseconds ago.
3/6/2017 2:50:00 PM	at sun.reflect.GeneratedConstructorAccessor67.newInstance(Unknown Source)
3/6/2017 2:50:00 PM	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
3/6/2017 2:50:00 PM	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
3/6/2017 2:50:00 PM	at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
3/6/2017 2:50:00 PM	at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:988)
3/6/2017 2:50:00 PM	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3552)
3/6/2017 2:50:00 PM	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3452)
3/6/2017 2:50:00 PM	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3893)
3/6/2017 2:50:00 PM	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
3/6/2017 2:50:00 PM	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
3/6/2017 2:50:00 PM	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
3/6/2017 2:50:00 PM	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
3/6/2017 2:50:00 PM	at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1962)
3/6/2017 2:50:00 PM	at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
3/6/2017 2:50:00 PM	at clojure.java.jdbc$db_query_with_resultset$run_query_with_params__2997.invoke(jdbc.clj:831)
3/6/2017 2:50:00 PM	at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:850)
3/6/2017 2:50:00 PM	at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:818)
3/6/2017 2:50:00 PM	at clojure.java.jdbc$query.invokeStatic(jdbc.clj:874)
3/6/2017 2:50:00 PM	at clojure.java.jdbc$query.invoke(jdbc.clj:854)
3/6/2017 2:50:00 PM	at metabase.db$query.invokeStatic(db.clj:534)
3/6/2017 2:50:00 PM	at metabase.db$query.doInvoke(db.clj:530)
3/6/2017 2:50:00 PM	at clojure.lang.RestFn.invoke(RestFn.java:410)
3/6/2017 2:50:00 PM	at metabase.db$simple_select.invokeStatic(db.clj:593)
3/6/2017 2:50:00 PM	at metabase.db$simple_select.invoke(db.clj:585)
3/6/2017 2:50:00 PM	at metabase.db$select.invokeStatic(db.clj:836)
3/6/2017 2:50:00 PM	at metabase.db$select.doInvoke(db.clj:830)
3/6/2017 2:50:00 PM	at clojure.lang.RestFn.invoke(RestFn.java:439)
3/6/2017 2:50:00 PM	at metabase.task.sync_databases.SyncDatabases.execute(sync_databases.clj:21)
3/6/2017 2:50:00 PM	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
3/6/2017 2:50:00 PM	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
3/6/2017 2:50:00 PMCaused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
3/6/2017 2:50:00 PM	at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3004)
3/6/2017 2:50:00 PM	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3462)
3/6/2017 2:50:00 PM	... 24 more
3/6/2017 2:50:00 PM03-06 13:50:00 ERROR core.ErrorLogger :: Job (DEFAULT.metabase.task.sync-databases.job threw an exception.
3/6/2017 2:50:00 PMorg.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

So frustrating :persevere: