Haproxy mysql gone away with version 3.0 and not 1.8

Help!
1

Hi all,

I’ve upgraded my version haproxy from 1.8 to 3.0.5 and now i get error on applications “mysql gone away”. I reverted to 1.8 and the error is gone, but with version 3.0.5 get back the error.

One solution is stay with version 1.8 but i want upgrade to 3.0.5 for security issues.

I used haproxy on rocky 8 and the service is used only for mysql (galera)

My configuration:

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           5m
    timeout connect         10s
    timeout client          5m
    timeout server          5m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 20000	
    load-server-state-from-file global

....

frontend galera
    bind x.x.x.x:3306
	mode tcp
    option tcpka
    option tcplog
    timeout client          480m

    acl ipgalera src -f /etc/haproxy/galera.lst
    use_backend galera if ipgalera    
    default_backend trash  
	
backend galera
    mode tcp
	timeout server 480m
	timeout connect 60s
	option external-check
    external-check path "/usr/bin:/bin:/usr/local/bin"
    external-check command /usr/local/bin/mysql_galera.sh
	
    server g1prod y.y.y.y:3306 check inter 5s fall 2 rise 1
    server g2prod z.z.z.z:3306 check inter 5s fall 2 rise 1 backup

Any idea about the problem exist with version 3.0.5 and not with 1.8?

Thank you for your help

2

So many things changed between 1.8 and 3.0.x that it’s hard to tell what could cause this timeout to appear after the upgrade.

Can the issue be reproduced easily? I would suggest to try to progressively upgrade from 1.8 to 2.6 then 2.8, to help spot where it first appears. If you could at least make it up to 2.6 without errors, then you would benefit from latest bugfixes as 2.6 will be maintained until Q2-2027 (while waiting to find out why it fails on 3.0)

3

Hi,
I’m come back to 1.8 and the problem is fixed.
I’ve tried 2.2 and 2.4 and the problem mysql gone aways is back.

The configuration is the same.

Any help, appreciate.

4

Thanks for the update

Any worth-mentioning difference in the logs produced by haproxy, 1.8 vs 2.2?

I’m suspecting a change related to checks or perhaps timeout handling, but nothing obvious so far

5

Hi adarragon,

Sorry for late response.
I’ve checked the log but don’t help anything to find the difference between 1.8 and other versions of haproxy.
My configuration is exactly the same on haproxy between 1.8 and other version.

When put version 2.2 or 2.4 or 3.0.5 i get many errors mysql gone away. With 1.8 (i’ve compiled the last version from 2022), 0 errors.

I’ve mentionned mysql, but with postgresql (patroni), i’ve checked the logs and get the same errors on many applications java try to connect do DB postgresql. With version 1.8, don’t get any errors.
On mysql, many of my application is on apache.

It’s very strange but don’t find any solution or difference

Thank you for your help.

6

Thanks for the update

Another thoughts:

on 1.8, did you use nbproc ? Because back then the total maxconn would be maxconn * nbproc, but with nbthreads this number is shared and must be tuned correctly

When looking at “gone away” errors, do they start to happen rapidly after haproxy is started or it takes a long time after haproxy was started to get the first error, and then other errors keep flowing?

7

When your applications are saying “mysql is gone”, what is HAproxy saying? Are the defined backends up with your external health-checks?
In your situation, I would try it first without health-checks :wink:

best regards,
Markus