Haproxy 1.8.13 & kubernetes service Discovery

gabrielsousa · August 9, 2018, 5:59pm

hello

i’m trying to use DNS for Service Discovery but without success
my config

global
debug

resolvers kube
  nameserver dns1 10.96.0.10:53

defaults
  mode http
  timeout connect 5000ms
  timeout client 50000ms
  timeout server 50000ms

listen stats
  bind *:1936
  mode http
  stats enable
  stats refresh 5s
  stats show-node
  stats show-legends
  stats auth admin:admin
  stats uri /stats

backend vccrm
  #server-template www 0-10 _vccrm-latest-qa._tcp.vccrm-latest-qa.qa:8000 check resolvers kube
  #server-template www 10 _vccrm-latest-qa._tcp.vccrm-latest-qa.qa.svc.cluster.local:8000 check resolvers kube
  #server-template www 10 _8000._tcp.vccrm-latest-qa.qa.svc.cluster.local:8000 check resolvers kube
  server-template www 0-8 _http._tcp.vccrm-latest-qa.qa.svc.cluster.local:8000 check resolvers kube resolve-prefer ipv4
  #server-template www 10 _vccrm-latest-qa._tcp.qa.svc.cluster.local:8000 check resolvers kube
  #server-template www 10 vccrm-latest-qa.qa.svc.cluster.local:8000 check resolvers kube
  #server-template www 10 _vccrm-latest-qa.vccrm-latest-qa.qa.svc.cluster.local.:8000 check resolvers kube
frontend vccrm
  bind *:80
  mode http
  default_backend vccrm

what i’m doing wrong ?

Baptiste · August 10, 2018, 9:57am

Hi,

The SRV name associated to a service in kubernete is:
_http._tcp...svc.cluster.local

Of course, if you changed your cluster name (default is ‘cluster.local’), then update the above accordingly.

What happens when you 'dig -t NS @10.96.0.10 -p 53 ?
Could you share the output here?

What version of HAProxy are you using?

gabrielsousa · August 10, 2018, 10:10am

the version is 1.8.13

gabrielsousa · August 10, 2018, 10:11am

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> @10.96.0.10 vccrm-latest-qa.qa.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59303
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;vccrm-latest-qa.qa.svc.cluster.local. IN A

;; ANSWER SECTION:
vccrm-latest-qa.qa.svc.cluster.local. 5 IN A 10.103.48.106

;; Query time: 4 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Aug 10 12:08:34 CEST 2018
;; MSG SIZE rcvd: 117

gabrielsousa · August 10, 2018, 10:14am

im having
Sorry, new users can only put 2 links in a post.

https://pastebin.com/c3GfRVWy

Baptiste · August 10, 2018, 10:42am

My bad… (and badly formated)

Baptiste · August 10, 2018, 10:43am

is supposed to be:

(need to add some padding)

gabrielsousa · August 10, 2018, 10:46am

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> -t SRV @10.96.0.10 -p 53 _http._tcp.vccrm-latest-qa.qa.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 6596
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;_http._tcp.vccrm-latest-qa.qa.svc.cluster.local. IN SRV

;; AUTHORITY SECTION:
cluster.local. 7 IN SOA ns.dns.cluster.local. hostmaster.cluster.local. 1533897700 7200 1800 86400 30

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Aug 10 12:46:42 CEST 2018
;; MSG SIZE rcvd: 169

gabrielsousa · August 10, 2018, 10:47am

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> -t SRV @10.96.0.10 -p 53 vccrm-latest-qa.qa.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13201
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;vccrm-latest-qa.qa.svc.cluster.local. IN SRV

;; ANSWER SECTION:
vccrm-latest-qa.qa.svc.cluster.local. 5 IN SRV 0 50 8000 vccrm-latest-qa.qa.svc.cluster.local.
vccrm-latest-qa.qa.svc.cluster.local. 5 IN SRV 0 50 4848 vccrm-latest-qa.qa.svc.cluster.local.

;; ADDITIONAL SECTION:
vccrm-latest-qa.qa.svc.cluster.local. 5 IN A 10.103.48.106

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Aug 10 12:47:13 CEST 2018
;; MSG SIZE rcvd: 301

gabrielsousa · August 10, 2018, 10:54am

sorry my bad, i named http8000

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> -t SRV @10.96.0.10 -p 53 _http8000._tcp.vccrm-latest-qa.qa.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43860
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;_http8000._tcp.vccrm-latest-qa.qa.svc.cluster.local. IN SRV

;; ANSWER SECTION:
_http8000._tcp.vccrm-latest-qa.qa.svc.cluster.local. 1 IN SRV 0 100 8000 vccrm-latest-qa.qa.svc.cluster.local.

;; ADDITIONAL SECTION:
vccrm-latest-qa.qa.svc.cluster.local. 1 IN A 10.103.48.106

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Aug 10 12:54:03 CEST 2018
;; MSG SIZE rcvd: 239

gabrielsousa · August 10, 2018, 10:56am

I thought with the dns of the service , haproxy would manage to put the all pods of the service
i only see one ip the service ip

Baptiste · August 10, 2018, 1:24pm

Currently, it seems this service is delivered by a single pod in your cluster, so it’s normal HAProxy get a single server available for it.

Baptiste

So " _http8000._tcp.vccrm-latest-qa.qa.svc.cluster.local" is the hostname you want to use in HAProxy’s configuration.

gabrielsousa · August 10, 2018, 1:35pm

i have 3 pods on that service.

Baptiste · August 10, 2018, 1:49pm

So why does dig reports only one?

gabrielsousa · August 10, 2018, 1:53pm

reports the ip of the service, isnt any pod

kubernets 1.11.0 with coreDNS

gabrielsousa · August 10, 2018, 2:06pm

[root@app197 ~]$ kubectl get services -n=vault
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul-backend-vault NodePort 10.111.202.194 8500:30060/TCP,8443:32403/TCP,8400:30002/TCP,8301:31636/TCP,8301:31636/UDP,8302:30304/TCP,8302:30304/UDP,8300:30765/TCP,8600:30422/TCP 17d
vault NodePort 10.107.214.96 8200:30070/TCP,8201:30237/TCP 17d
[root@app197 ~]$

[root@app197 ~]$ kubectl get pods -n=vault
NAME READY STATUS RESTARTS AGE
consul-backend-vault-0 1/1 Running 0 1d
consul-backend-vault-1 1/1 Running 0 1d
consul-backend-vault-2 1/1 Running 0 1d
vault-0 2/2 Running 0 1d
vault-1 2/2 Running 0 1d
vault-2 2/2 Running 0 1d

gabrielsousa · August 10, 2018, 2:19pm

[root@app197 ~]$ dig @10.96.0.10 -t SRV _vaultport._tcp.vault.vault.svc.cluster.local

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> @10.96.0.10 -t SRV _vaultport._tcp.vault.vault.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20000
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;_vaultport._tcp.vault.vault.svc.cluster.local. IN SRV

;; ANSWER SECTION:
_vaultport._tcp.vault.vault.svc.cluster.local. 5 IN SRV 0 100 8200 vault.vault.svc.cluster.local.

;; ADDITIONAL SECTION:
vault.vault.svc.cluster.local. 5 IN A 10.107.214.96

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Aug 10 16:19:03 CEST 2018
;; MSG SIZE rcvd: 213

gabrielsousa · August 10, 2018, 2:20pm

[root@app197 ~]$ kubectl get pods -n=qa |grep vccrm
vccrm-latest-qa-6d7c8dbdf4-jg8ss 2/2 Running 0 2d
vccrm-latest-qa-6d7c8dbdf4-nx87t 2/2 Running 0 1d
vccrm-latest-qa-6d7c8dbdf4-r8qbg 2/2 Running 0 1d
vccrm-latest-qa-6d7c8dbdf4-sl7bd 2/2 Running 0 2d
vccrm-latest-qa-6d7c8dbdf4-vmnd8 2/2 Running 0 2d

Baptiste · August 10, 2018, 3:37pm

Ah ok, you must run your kubernetes service in “headless” mode.

gabrielsousa · August 10, 2018, 3:48pm

ho, thanks a lot
whats the disadvantage of the “headless” mode ?

Topic		Replies	Views
Resolvers inside kubernetes Help!	11	5249	July 7, 2017
Config for Service Discovery using DNS and SRV records? Help!	3	4357	December 3, 2018
HAProxy won't connect resolving with Kubernetes DNS HAProxy 1.8.4 Help!	8	5745	April 4, 2019
Service Discovery using DNS and SRV，Is it good on 2.4？ Help!	1	526	October 19, 2021
Understnd DNS resolver behaviour Configuration Samples	0	402	March 15, 2023

Haproxy 1.8.13 & kubernetes service Discovery

Related topics