HAProxy dynamic discovery with Consul dns and Prometheus

Help!
1

Can someone pls help with this?

I’ve setup a consul based dynamic discovery for haproxy backends. - something like:
server-template test-ui 8 _test-ui._tcp.service.consul check resolvers ConsulAutoDiscovery check init-addr none

Auto discovery runs good and i can see 8 backends in the stats page showing and whichever ones are discovered are green. So far great. Here’s the issue:

  1. All the backends that do not have any ip associated show up as “down” in stats page as opposed to MAINT. I read that they will show up as MAINT.
  2. We’ve Prometheus scraping configured but now that these non-ip backends show up as down, we get constant alerts for haproxy backend down.

Did anyone encounter this problem? What is a good solution?

Thanks!

2

about #1: where did you read that? I think they are DOWN at start up and switch to MAINT when you scale in (so basically UP servers being unconfigured by DNS)

about #2: what is your goal with prometheus counting DOWN server?

I think I understand and you would like to make the difference between a MAINT server and a DOWN server to easily report in prometheus a real outage from a normal scaled down backend.
Can you confirm?

3

Thanks for responding. Here’s where I read about the backends being in maintenance mode if they’ve no ip assigned:

https://www.haproxy.com/blog/dns-service-discovery-haproxy/

Also, yes, would like to distinguish from no-ip backends (scale up/scale down) and real service down scenario.

Thank you!

4

So Ideally, servers should start up in MAINT mode and be DOWN only when there is a failed health check.

I’ll check how this could be done.

5

Thanks for the help!

6

Any update on this? To add to this, noticed that when a backend does go down, HAProxy shows that the backend that died is in MAINT mode. I think it should be the other way around. I’m using HAProxy 2.0.9 since we can’t upgrade to the latest due to incompatibilities