Haproxy 1.8.13 & kubernetes service Discovery


#1

hello

i’m trying to use DNS for Service Discovery but without success
my config

global
debug

resolvers kube
  nameserver dns1 10.96.0.10:53

defaults
  mode http
  timeout connect 5000ms
  timeout client 50000ms
  timeout server 50000ms

listen stats
  bind *:1936
  mode http
  stats enable
  stats refresh 5s
  stats show-node
  stats show-legends
  stats auth admin:admin
  stats uri /stats

backend vccrm
  #server-template www 0-10 _vccrm-latest-qa._tcp.vccrm-latest-qa.qa:8000 check resolvers kube
  #server-template www 10 _vccrm-latest-qa._tcp.vccrm-latest-qa.qa.svc.cluster.local:8000 check resolvers kube
  #server-template www 10 _8000._tcp.vccrm-latest-qa.qa.svc.cluster.local:8000 check resolvers kube
  server-template www 0-8 _http._tcp.vccrm-latest-qa.qa.svc.cluster.local:8000 check resolvers kube resolve-prefer ipv4
  #server-template www 10 _vccrm-latest-qa._tcp.qa.svc.cluster.local:8000 check resolvers kube
  #server-template www 10 vccrm-latest-qa.qa.svc.cluster.local:8000 check resolvers kube
  #server-template www 10 _vccrm-latest-qa.vccrm-latest-qa.qa.svc.cluster.local.:8000 check resolvers kube
frontend vccrm
  bind *:80
  mode http
  default_backend vccrm

what i’m doing wrong ?


#2

Hi,

The SRV name associated to a service in kubernete is:
_http._tcp...svc.cluster.local

Of course, if you changed your cluster name (default is ‘cluster.local’), then update the above accordingly.

What happens when you 'dig -t NS @10.96.0.10 -p 53 ?
Could you share the output here?

What version of HAProxy are you using?


#3

the version is 1.8.13


#4

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> @10.96.0.10 vccrm-latest-qa.qa.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59303
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;vccrm-latest-qa.qa.svc.cluster.local. IN A

;; ANSWER SECTION:
vccrm-latest-qa.qa.svc.cluster.local. 5 IN A 10.103.48.106

;; Query time: 4 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Aug 10 12:08:34 CEST 2018
;; MSG SIZE rcvd: 117


#5

im having
Sorry, new users can only put 2 links in a post.

https://pastebin.com/c3GfRVWy


#6

My bad… (and badly formated)


#7

is supposed to be:

(need to add some padding)


#8

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> -t SRV @10.96.0.10 -p 53 _http._tcp.vccrm-latest-qa.qa.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 6596
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;_http._tcp.vccrm-latest-qa.qa.svc.cluster.local. IN SRV

;; AUTHORITY SECTION:
cluster.local. 7 IN SOA ns.dns.cluster.local. hostmaster.cluster.local. 1533897700 7200 1800 86400 30

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Aug 10 12:46:42 CEST 2018
;; MSG SIZE rcvd: 169


#9

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> -t SRV @10.96.0.10 -p 53 vccrm-latest-qa.qa.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13201
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;vccrm-latest-qa.qa.svc.cluster.local. IN SRV

;; ANSWER SECTION:
vccrm-latest-qa.qa.svc.cluster.local. 5 IN SRV 0 50 8000 vccrm-latest-qa.qa.svc.cluster.local.
vccrm-latest-qa.qa.svc.cluster.local. 5 IN SRV 0 50 4848 vccrm-latest-qa.qa.svc.cluster.local.

;; ADDITIONAL SECTION:
vccrm-latest-qa.qa.svc.cluster.local. 5 IN A 10.103.48.106

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Aug 10 12:47:13 CEST 2018
;; MSG SIZE rcvd: 301


#10

sorry my bad, i named http8000

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> -t SRV @10.96.0.10 -p 53 _http8000._tcp.vccrm-latest-qa.qa.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43860
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;_http8000._tcp.vccrm-latest-qa.qa.svc.cluster.local. IN SRV

;; ANSWER SECTION:
_http8000._tcp.vccrm-latest-qa.qa.svc.cluster.local. 1 IN SRV 0 100 8000 vccrm-latest-qa.qa.svc.cluster.local.

;; ADDITIONAL SECTION:
vccrm-latest-qa.qa.svc.cluster.local. 1 IN A 10.103.48.106

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Aug 10 12:54:03 CEST 2018
;; MSG SIZE rcvd: 239


#11

I thought with the dns of the service , haproxy would manage to put the all pods of the service :frowning:
i only see one ip the service ip


#12

Currently, it seems this service is delivered by a single pod in your cluster, so it’s normal HAProxy get a single server available for it.

Baptiste

So " _http8000._tcp.vccrm-latest-qa.qa.svc.cluster.local" is the hostname you want to use in HAProxy’s configuration.


#13

i have 3 pods on that service.


#14

So why does dig reports only one?


#15

reports the ip of the service, isnt any pod :frowning:

kubernets 1.11.0 with coreDNS


#16

[root@app197 ~] kubectl get services -n=vault NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE consul-backend-vault NodePort 10.111.202.194 <none> 8500:30060/TCP,8443:32403/TCP,8400:30002/TCP,8301:31636/TCP,8301:31636/UDP,8302:30304/TCP,8302:30304/UDP,8300:30765/TCP,8600:30422/TCP 17d vault NodePort 10.107.214.96 <none> 8200:30070/TCP,8201:30237/TCP 17d [root@app197 ~]

[root@app197 ~]$ kubectl get pods -n=vault
NAME READY STATUS RESTARTS AGE
consul-backend-vault-0 1/1 Running 0 1d
consul-backend-vault-1 1/1 Running 0 1d
consul-backend-vault-2 1/1 Running 0 1d
vault-0 2/2 Running 0 1d
vault-1 2/2 Running 0 1d
vault-2 2/2 Running 0 1d


#17

[root@app197 ~]$ dig @10.96.0.10 -t SRV _vaultport._tcp.vault.vault.svc.cluster.local

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> @10.96.0.10 -t SRV _vaultport._tcp.vault.vault.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20000
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;_vaultport._tcp.vault.vault.svc.cluster.local. IN SRV

;; ANSWER SECTION:
_vaultport._tcp.vault.vault.svc.cluster.local. 5 IN SRV 0 100 8200 vault.vault.svc.cluster.local.

;; ADDITIONAL SECTION:
vault.vault.svc.cluster.local. 5 IN A 10.107.214.96

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Aug 10 16:19:03 CEST 2018
;; MSG SIZE rcvd: 213


#18

[root@app197 ~]$ kubectl get pods -n=qa |grep vccrm
vccrm-latest-qa-6d7c8dbdf4-jg8ss 2/2 Running 0 2d
vccrm-latest-qa-6d7c8dbdf4-nx87t 2/2 Running 0 1d
vccrm-latest-qa-6d7c8dbdf4-r8qbg 2/2 Running 0 1d
vccrm-latest-qa-6d7c8dbdf4-sl7bd 2/2 Running 0 2d
vccrm-latest-qa-6d7c8dbdf4-vmnd8 2/2 Running 0 2d


#19

Ah ok, you must run your kubernetes service in “headless” mode.


#20

ho, thanks a lot
whats the disadvantage of the “headless” mode ?