We use HAPROXY with SRV records, it works great but we have a small issue where it supposed to have only 2 Backend active (can scale up to 4) but we only had 2 at the current time.
We use haproxy 2.4.17 and this is the behavior we observed.
We have the following server-template:
server-template domain-x 4 "_tcp_.entrypoint.my-custom-discovery-dns.local" resolvers dns resolve-opts allow-dup-ip resolve-prefer ipv4 fastinter 10s rise 2 fall 2 on-error fastinter check inter 10000
Somehow we started to get some ‘errors’ on one of our system where some traffic were serving wrong ‘Website’.
The show stat unix command gave us the following:
domain-x-4_back,domain-x-41,0,0,0,0,0,0,0,0,0,0,0,0,UP,1,1,0,0,0,410,0,1,861,1,0,2,0,0,L4OK,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,Layer4 check passed,2,2,3,10.10.12.203:49156,http,0,0,0,0,0,0,0,0,0,0,0,0,0,1,-,0,0,0,
domain-x-4_back,domain-x-42,0,0,0,0,0,0,0,0,0,0,0,0,UP,1,1,0,0,0,103995,0,1,861,2,0,2,0,0,L4OK,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,Layer4 check passed,2,2,3,10.10.21.180:49157,http,0,0,0,0,0,0,0,0,0,0,0,0,0,1,-,0,0,0,
domain-x-4_back,domain-x-43,0,0,0,0,0,0,0,0,0,0,0,0,UP,1,1,0,0,0,410,0,1,861,3,0,2,0,0,L4OK,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,Layer4 check passed,2,2,3,10.10.12.157:49295,http,0,0,0,0,0,0,0,0,0,0,0,0,0,1,-,0,0,0,
domain-x-4_back,domain-x-44,0,0,0,0,0,0,0,0,0,0,0,0,UP,1,1,0,0,0,103488,0,1,861,4,0,2,0,0,L4OK,0,0,0,0,0,0,0,0,0,0,-1,0,0,0,0,Layer4 check passed,2,2,3,10.10.12.44:49160,http,0,0,0,0,0,0,0,0,0,0,0,0,0,1,-,0,0,0,
As we can see somehow we have 4 Servers, but there should be only 2 (valid).
When we run an ‘haproxy reload’ the following occurs:
socat /run/haproxy/haproxy_1_.sock - <<< “show servers state” > /var/state/haproxy/state
… reload haproxy
What we are thinking maybe we could “run and” sed command ( to remove the “IP” and “Port” to be “-”) after the saving of the haproxy state…
IF ( the srv_check_status srv_check_result srv_check_health srv_check_state ) is not healthy ? aka (in a bad state??)
Not sure what would be the condition yet …
Basically is there a way to tell haproxy to 'revalidate if the IP are still part of the SRV Record ?
Because when i was doing an dig tcp.entrypoint.my-custom-discovery-dns.local only two ip address are showing up… but the server-state is kept with the other IP’ / Port and these IP/Port may get re-used by the newer docker launches…