Haproxy no-check config not working as intended

Help!
1

Not sure if this is a bug or some misconfiguration

System:
HAProxy lrunnign as a K8s container pod and load balancing services in a k8s cluster with HAProxy backends pointing at k8s services

Desired behavior :

  • Some services might not exist when HAProxy is started .
  • HAProxy should start without erroring out because some services/backends do not exist
  • When the services are created, HAProxy should serve traffic to them

Actual Behavior:
HAProxy disables the services at startup

  • When the services are created, HAProxy does not recognize that they are up and errors out (503) when web requests are issued (503) unless HAProxy container is restarted or configuration reloaded
  • I tried this with a variety of HAPRoxy 2.x versions and 3.1
  • I tried using the no-check option to prevent disabling servers but it didn’t seem to work - HAProxy version 3.1.1 - Configuration Manual

Log

[NOTICE]   (1) : [/etc/haproxy/haproxy.cfg:285] : 'server mysvc/mysvc' : could not resolve address 'mysvc-abc123.mynamespace.svc', disabling server.

Config

defaults
  mode               http
  balance            roundrobin
  option             redispatch

  # Add X-Forwarded-For headers to each request
  # so backend sees request as coming from the client's IP address, not LB's.
  option             forwardfor except 127.0.0.1/8
  option             abortonclose
  retries            1
  timeout            client 10m
  timeout            server 10m
  timeout            connect 4s
  timeout            http-keep-alive 20s
  default-server     inter 30s slowstart 10s rise 1 fall 60 maxconn 500 weight 100 init-addr libc,none


relevant config 
backend mysvc ======> LINE 285
  server mysvc mysvc-abc123.mynamespace.svc:8081 no-check
2

You need to enable run time name resolution and disable libc based startup resolution (init-addr none) which cause the startup error:

resolvers default
 parse-resolv-conf
 timeout resolve 1m

backend mysvc
 default-server init-addr none resolvers default

This way the name is dynamically reresolved during run time.