HaProxy self-reloading: possible culprint behing job crashing

Help!
1

Hi guys,

We use HaProxy to manage our applications access to the microservices.
We are talking about some 21 microservices, which extend to 63 virtual machines and 63 ACLs

What we have noticed tho is that from time to time, it self-reload itself.
We have been having some issues where Rundeck jobs will crash due to timeout, HaProxy might be the culprit.
The timeouts don’t match the ones in place. It is like it just happens from nowhere.

We have another HaProxy, the same stable version, 1.8.14-52e4d43, running MariaDB. It has been running fine for months without any self-reload.
The only difference is that the database one uses TCP while the microservices one is using HTTP/HTTPS.

The image below is part of a long static page. Observe the status :frowning:

haproxy
haproxy399×741 44.9 KB

I don’t know what could be possible causing this.
I am sharing part of the config file showing the common places to have problems.
I appreciate any help.

        defaults
mode                    http
log                     global
option                  httplog
option                  dontlognull
option http-server-close
option forwardfor       except 127.0.0.0/8
option                  redispatch
retries                 3
timeout http-request    10s
timeout queue           1m
timeout connect         10s
timeout client          15m
timeout server          15m
timeout http-keep-alive 10s
timeout check           10s
maxconn                 4000
        frontend http
        bind *:80
        mode http
    	option http-server-close
        option forwardfor       except 127.0.0.0/8

    acl SOMETHING_IP src IP

    A dozen ACLs like above

    acl SOMETHING_SERVICE path_reg ^/SERVICE
    use_backend SOMETHING_SERVICE if SOMETHING_IP SOMETHING_SERVICE
    
    Dozens ACLs like above

backend SOMETHING_SERVICE
  mode http
  option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www
  balance roundrobin
  server SOMETHING_IP SOMETHING_IP:80 weight 10 check port 80 inter 1s fall 1 rise 3
  server SOMETHING_IP SOMETHING_IP:80 weight 10 check port 80 inter 1s fall 1 rise 3
  server SOMETHING_IP SOMETHING_IP:80 weight 0 check port 80 inter 1s fall 1 rise 3

Thanks guys