Haproxy 1.8.2; 1.8.3 DNS auto discover stop working

Help!
1

resolvers awsdns
nameserver dns 172.30.0.2:53
resolve_retries 3
timeout retry 1s
hold valid 10s
accepted_payload_size 8192
global
maxconn 41666
nbproc 6

backend hosts-us-east-1e
balance leastconn

http-reuse aggressive

http-check expect status 200
option httpchk GET /status_emserver

server-template host-rr 70 _http._tcp.us-east-1e.com resolvers awsdns resolve-prefer ipv4 check inter 1s downinter 60s fall 3 weight 1

haproxy -vv
HA-Proxy version 1.8.3-205f675 2017/12/30
Copyright 2000-2017 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.1k-fips 8 Jan 2015
Running on OpenSSL version : OpenSSL 1.0.1k-fips 8 Jan 2015
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.21 2011-12-12
Running on PCRE version : 8.21 2011-12-12
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with network namespace support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

for i in $(seq 1 6); do echo “show servers state” | socat unix-connect:/var/run/haproxy-$i.sock stdio|grep .int|wc -l; done
25
37
25
25
25
25

echo “show servers state” | socat unix-connect:/var/run/haproxy-1.sock stdio
1
be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port
4 hosts-us-east-1e 1 host-rr1 192.168.12.234 2 0 25 1 75258 15 3 4 6 0 0 0 ip-192-168-12-234.internal 80
4 hosts-us-east-1e 2 host-rr2 192.168.13.108 2 0 25 1 75540 15 3 4 6 0 0 0 ip-192-168-13-108.internal 80
4 hosts-us-east-1e 3 host-rr3 192.168.13.243 2 64 25 1 10287 15 3 4 6 0 0 0 ip-192-168-13-243.internal 80
4 hosts-us-east-1e 4 host-rr4 192.168.15.41 2 64 25 1 70953 15 3 4 6 0 0 0 ip-192-168-15-41.internal 80
4 hosts-us-east-1e 5 host-rr5 192.168.15.26 2 64 25 1 10654 15 3 4 6 0 0 0 ip-192-168-15-26.internal 80
4 hosts-us-east-1e 6 host-rr6 192.168.13.106 2 64 25 1 18817 15 3 4 6 0 0 0 ip-192-168-13-106.internal 80
4 hosts-us-east-1e 7 host-rr7 192.168.13.215 2 64 25 1 18821 15 3 4 6 0 0 0 ip-192-168-13-215.internal 80
4 hosts-us-east-1e 8 host-rr8 192.168.14.114 2 64 25 1 10288 15 3 4 6 0 0 0 ip-192-168-14-114.internal 80
4 hosts-us-east-1e 9 host-rr9 192.168.15.175 2 64 25 1 25514 15 3 4 6 0 0 0 ip-192-168-15-175.internal 80
4 hosts-us-east-1e 10 host-rr10 192.168.14.5 2 0 25 1 76103 15 3 4 6 0 0 0 ip-192-168-14-5.internal 80
4 hosts-us-east-1e 11 host-rr11 192.168.15.189 2 64 25 1 71322 15 3 4 6 0 0 0 ip-192-168-15-189.internal 80
4 hosts-us-east-1e 12 host-rr12 192.168.12.87 2 0 25 1 75511 15 3 4 6 0 0 0 ip-192-168-12-87.internal 80
4 hosts-us-east-1e 13 host-rr13 192.168.13.86 2 0 25 1 75612 15 3 4 6 0 0 0 ip-192-168-13-86.internal 80
4 hosts-us-east-1e 14 host-rr14 192.168.13.88 2 0 25 1 75367 15 3 4 6 0 0 0 ip-192-168-13-88.internal 80
4 hosts-us-east-1e 15 host-rr15 192.168.15.228 2 64 25 1 49206 15 3 4 6 0 0 0 ip-192-168-15-228.internal 80
4 hosts-us-east-1e 16 host-rr16 192.168.12.25 2 64 25 1 23400 15 3 4 6 0 0 0 ip-192-168-12-25.internal 80
4 hosts-us-east-1e 17 host-rr17 192.168.14.29 2 0 25 1 75771 15 3 4 6 0 0 0 ip-192-168-14-29.internal 80
4 hosts-us-east-1e 18 host-rr18 192.168.12.50 2 64 25 1 15702 15 3 4 6 0 0 0 ip-192-168-12-50.internal 80
4 hosts-us-east-1e 19 host-rr19 192.168.15.184 0 96 25 1 32707 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 20 host-rr20 192.168.15.46 2 0 25 1 76010 15 3 4 6 0 0 0 ip-192-168-15-46.internal 80
4 hosts-us-east-1e 21 host-rr21 192.168.15.68 0 96 25 1 29974 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 22 host-rr22 192.168.12.103 2 0 25 1 75015 15 3 4 6 0 0 0 ip-192-168-12-103.internal 80
4 hosts-us-east-1e 23 host-rr23 192.168.14.234 0 96 25 1 71661 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 24 host-rr24 192.168.12.154 2 0 25 1 75085 15 3 4 6 0 0 0 ip-192-168-12-154.internal 80
4 hosts-us-east-1e 25 host-rr25 192.168.12.190 2 0 25 1 75171 15 3 4 6 0 0 0 ip-192-168-12-190.internal 80
4 hosts-us-east-1e 26 host-rr26 192.168.12.206 0 96 25 1 70922 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 27 host-rr27 192.168.12.222 2 0 25 1 75354 15 3 4 6 0 0 0 ip-192-168-12-222.internal 80
4 hosts-us-east-1e 28 host-rr28 192.168.12.230 0 96 25 1 25499 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 29 host-rr29 192.168.12.231 2 0 25 1 75787 15 3 4 6 0 0 0 ip-192-168-12-231.internal 80
4 hosts-us-east-1e 30 host-rr30 192.168.12.139 0 96 25 1 34178 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 31 host-rr31 192.168.14.72 0 96 25 1 30499 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 32 host-rr32 192.168.14.141 0 96 25 1 30476 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 33 host-rr33 192.168.15.160 0 96 25 1 68571 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 34 host-rr34 192.168.13.8 0 96 25 1 32355 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 35 host-rr35 192.168.13.79 0 96 25 1 32326 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 36 host-rr36 192.168.14.254 0 96 25 1 27167 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 37 host-rr37 192.168.15.74 0 96 25 1 27190 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 38 host-rr38 192.168.12.198 0 96 25 1 23412 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 39 host-rr39 192.168.12.117 0 96 25 1 29348 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 40 host-rr40 192.168.14.86 0 96 25 1 29328 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 41 host-rr41 192.168.12.134 0 96 25 1 29976 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 42 host-rr42 192.168.15.104 0 96 25 1 68576 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 43 host-rr43 192.168.14.137 0 96 25 1 95956 8 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 44 host-rr44 192.168.13.60 0 96 25 1 26844 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 45 host-rr45 192.168.12.44 0 96 25 1 112406 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 46 host-rr46 192.168.13.222 0 96 25 1 101111 8 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 47 host-rr47 192.168.15.95 0 96 25 1 71291 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 48 host-rr48 192.168.14.74 0 96 25 1 113828 8 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 49 host-rr49 192.168.13.151 0 96 25 1 113117 8 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 50 host-rr50 192.168.12.70 0 96 25 1 115651 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 51 host-rr51 192.168.13.203 0 96 25 1 114944 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 52 host-rr52 192.168.12.109 0 96 25 1 103180 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 53 host-rr53 192.168.15.157 0 96 25 1 56958 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 54 host-rr54 192.168.12.112 0 96 25 1 114180 8 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 55 host-rr55 192.168.15.233 0 96 25 1 56963 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 56 host-rr56 192.168.13.237 0 96 25 1 132812 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 57 host-rr57 192.168.12.126 0 96 25 1 26847 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1e 58 host-rr58 192.168.12.228 2 0 25 1 75936 15 3 4 6 0 0 0 ip-192-168-12-228.internal 80
4 hosts-us-east-1e 59 host-rr59 0 0 1 1 171432 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1e 60 host-rr60 0 0 1 1 171432 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1e 61 host-rr61 0 0 1 1 171432 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1e 62 host-rr62 0 0 1 1 171432 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1e 63 host-rr63 0 0 1 1 171432 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1e 64 host-rr64 0 0 1 1 171432 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1e 65 host-rr65 0 0 1 1 171432 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1e 66 host-rr66 0 0 1 1 171432 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1e 67 host-rr67 0 0 1 1 171432 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1e 68 host-rr68 0 0 1 1 171432 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1e 69 host-rr69 0 0 1 1 171432 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1e 70 host-rr70 0 0 1 1 171432 5 2 0 6 0 0 0 - 0

After restart

for i in $(seq 1 6); do echo “show servers state” | socat unix-connect:/var/run/haproxy-$i.sock stdio|grep ec2.int|wc -l; done
37
37
37
37
37
37

CC: @lukastribus @Baptiste

2

Ok, a few steps to get the troubleshooting going:

  • enable syslogging and review the logs related to backend servers going down
  • provide a complete, anonymized configuration
  • confirm that you are not using nbthread
  • confirm that this does not happen because of a reload
  • confirm whether haproxy recovers from this condition when reloading (as opposed to restarting) - careful about hitting the new admin socket, not the old one when checking
  • confirm whether this gets worse over time when you don’t restart/reload
  • are you deprovisioning backend servers and could this be related (like deprovisioined server are down and occupy a server slot in haproxy but not reused for future dns resolution?)
  • try the current 1.8 tree (at least commmit 945f4cf08), although I don’t think you are affected by fixed bugs

thanks

3

resolvers dns1
nameserver dns 172.30.0.2:53
resolve_retries 3
timeout retry 1s
hold valid 10s
accepted_payload_size 8192

global
maxconn 41666
nbproc 6

pidfile /var/run/haproxy.pid
stats socket /var/run/haproxy-1.sock uid 0 gid 0 mode 0666 level admin process 1
stats socket /var/run/haproxy-2.sock uid 0 gid 0 mode 0666 level admin process 2
stats socket /var/run/haproxy-3.sock uid 0 gid 0 mode 0666 level admin process 3
stats socket /var/run/haproxy-4.sock uid 0 gid 0 mode 0666 level admin process 4
stats socket /var/run/haproxy-5.sock uid 0 gid 0 mode 0666 level admin process 5
stats socket /var/run/haproxy-6.sock uid 0 gid 0 mode 0666 level admin process 6

user haproxy
group haproxy
daemon
quiet
tune.ssl.default-dh-param 2048
spread-checks 5s
log 127.0.0.1 local0 info

tune.bufsize 10000
tune.maxaccept -1

defaults
mode http

log global
option httplog
option dontlog-normal

Forward request headers from the original client to the backend

option forwardfor

These make a big difference in the rate at which we can process requests

option tcp-smart-connect
option tcp-smart-accept
option splice-auto

Set pretty aggressive timeouts

timeout connect 5s
timeout client 30s
timeout server 30s
timeout queue 5s
timeout http-request 30s
timeout http-keep-alive 60s

keep backend connections alive

no option http-server-close

Retry if you can’t connect immediately

option redispatch 1 # redispatch on every retry
retries 3

frontend http-in
bind *:80
mode http

maxconn 31249
backlog 62498

compression algo gzip

acl is_204 status eq 204
acl no_content hdr_val(content-length) lt 1
http-response del-header Content-Length if is_204
http-response del-header Content-Type if is_204
http-response del-header Content-Type if no_content

default_backend *-us-east-1

frontend https-in
bind :443 ssl crt /etc/haproxy/star_.pem
reqadd X-Forwarded-Proto:\ https

maxconn 10416
backlog 20832

acl is_204 status eq 204
acl no_content hdr_val(content-length) lt 1
http-response del-header Content-Length if is_204
http-response del-header Content-Type if no_content

default_backend *-us-east-1

backend *-us-east-1
balance leastconn

http-reuse aggressive

http-check expect status 200
option httpchk GET /status_server

server-template -rr 70 _http._tcp.us-east-1.com resolvers dns1 resolve-prefer ipv4 check inter 1s downinter 60s fall 3 weight 1

listen stats
bind *:1936
stats enable
bind-process 1
stats uri /stats
stats hide-version
stats auth :
stats refresh 30s

  1. Enabled; there is nothing interesting, just like new servers don’t come up
  2. Done
  3. Not using nbthread
  4. no reload, no restart
  5. I was wrong. recovers on restart(should check reload)
  6. gets worse over time, was better when there was no more than 25-30 servers
  7. old servers just been terminated
4

In some point of time it’s just stop working

Jan 25 02:37:46 localhost haproxy[16807]: hosts-us-east-1e/host-rr3 changed its IP from to 192.168.13.6 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16807]: hosts-us-east-1e/host-rr2 changed its IP from to 192.168.12.6 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16811]: hosts-us-east-1e/host-rr2 changed its IP from to 192.168.15.175 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16811]: hosts-us-east-1e/host-rr5 changed its IP from to 192.168.14.5 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16811]: hosts-us-east-1e/host-rr7 changed its IP from to 192.168.12.87 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16811]: hosts-us-east-1e/host-rr6 changed its IP from to 192.168.12.45 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16811]: hosts-us-east-1e/host-rr9 changed its IP from to 192.168.13.30 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16811]: hosts-us-east-1e/host-rr10 changed its IP from to 192.168.13.35 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16811]: hosts-us-east-1e/host-rr8 changed its IP from to 192.168.13.11 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16811]: hosts-us-east-1e/host-rr3 changed its IP from to 192.168.12.6 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16811]: hosts-us-east-1e/host-rr4 changed its IP from to 192.168.13.6 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr1 changed its IP from to 192.168.15.155 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr3 changed its IP from to 192.168.15.175 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr5 changed its IP from to 192.168.13.6 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr6 changed its IP from to 192.168.14.5 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr7 changed its IP from to 192.168.12.45 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr8 changed its IP from to 192.168.12.87 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr9 changed its IP from to 192.168.13.11 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr10 changed its IP from to 192.168.13.30 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr13 changed its IP from to 192.168.13.74 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr4 changed its IP from to 192.168.12.6 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr15 changed its IP from to 192.168.13.86 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr16 changed its IP from to 192.168.13.88 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr18 changed its IP from to 192.168.14.29 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr17 changed its IP from to 192.168.14.13 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr19 changed its IP from to 192.168.14.33 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr11 changed its IP from to 192.168.13.35 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr21 changed its IP from to 192.168.14.85 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr14 changed its IP from to 192.168.13.80 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr46 changed its IP from to 192.168.15.126 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr24 changed its IP from to 192.168.15.68 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr47 changed its IP from to 192.168.15.144 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16807]: hosts-us-east-1e/host-rr10 changed its IP from to 192.168.13.58 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16811]: hosts-us-east-1e/host-rr11 changed its IP from to 192.168.13.58 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr23 changed its IP from to 192.168.15.20 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr22 changed its IP from to 192.168.14.89 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr20 changed its IP from to 192.168.14.45 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr2 changed its IP from to 192.168.15.126 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr1 changed its IP from to 192.168.15.123 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr6 changed its IP from to 192.168.15.175 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr3 changed its IP from to 192.168.15.144 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr8 changed its IP from to 192.168.13.6 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr9 changed its IP from to 192.168.14.5 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr10 changed its IP from to 192.168.12.45 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr11 changed its IP from to 192.168.12.87 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr4 changed its IP from to 192.168.15.155 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr7 changed its IP from to 192.168.12.6 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr14 changed its IP from to 192.168.13.35 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr13 changed its IP from to 192.168.13.30 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr15 changed its IP from to 192.168.13.58 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr16 changed its IP from to 192.168.13.74 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr17 changed its IP from to 192.168.13.80 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16810]: hosts-us-east-1e/host-rr4 changed its IP from to 192.168.15.175 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16810]: hosts-us-east-1e/host-rr5 changed its IP from to 192.168.12.6 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr1 changed its IP from to 192.168.15.126 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr2 changed its IP from to 192.168.15.144 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr3 changed its IP from to 192.168.15.155 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr7 changed its IP from to 192.168.13.6 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr5 changed its IP from to 192.168.15.175 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr6 changed its IP from to 192.168.12.6 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr8 changed its IP from to 192.168.14.5 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr10 changed its IP from to 192.168.12.87 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr9 changed its IP from to 192.168.12.45 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr11 changed its IP from to 192.168.13.11 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr24 changed its IP from to 192.168.14.89 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr26 changed its IP from to 192.168.15.68 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr25 changed its IP from to 192.168.15.20 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr29 changed its IP from to 192.168.12.111 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr27 changed its IP from to 192.168.15.69 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr30 changed its IP from to 192.168.12.140 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr32 changed its IP from to 192.168.12.154 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr34 changed its IP from to 192.168.12.167 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16811]: hosts-us-east-1e/host-rr1 changed its IP from to 192.168.15.159 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16810]: hosts-us-east-1e/host-rr3 changed its IP from to 192.168.15.159 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16809]: hosts-us-east-1e/host-rr2 changed its IP from to 192.168.15.159 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16808]: hosts-us-east-1e/host-rr5 changed its IP from to 192.168.15.159 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr4 changed its IP from to 192.168.15.159 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr28 changed its IP from to 192.168.12.109 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr33 changed its IP from to 192.168.12.160 by dns1/dns.
Jan 25 02:37:46 localhost haproxy[16812]: hosts-us-east-1e/host-rr31 changed its IP from to 192.168.12.152 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16807]: hosts-us-east-1e/host-rr13 changed its IP from to 192.168.13.86 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16807]: hosts-us-east-1e/host-rr16 changed its IP from to 192.168.14.29 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16807]: hosts-us-east-1e/host-rr14 changed its IP from to 192.168.13.88 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16807]: hosts-us-east-1e/host-rr12 changed its IP from to 192.168.13.80 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16807]: hosts-us-east-1e/host-rr17 changed its IP from to 192.168.14.33 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16807]: hosts-us-east-1e/host-rr18 changed its IP from to 192.168.14.45 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16807]: hosts-us-east-1e/host-rr19 changed its IP from to 192.168.14.85 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16807]: hosts-us-east-1e/host-rr20 changed its IP from to 192.168.14.89 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16807]: hosts-us-east-1e/host-rr22 changed its IP from to 192.168.15.68 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16807]: hosts-us-east-1e/host-rr21 changed its IP from to 192.168.15.20 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16812]: hosts-us-east-1e/host-rr13 changed its IP from to 192.168.13.35 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16812]: hosts-us-east-1e/host-rr12 changed its IP from to 192.168.13.30 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16812]: hosts-us-east-1e/host-rr16 changed its IP from to 192.168.13.80 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16812]: hosts-us-east-1e/host-rr17 changed its IP from to 192.168.13.86 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16812]: hosts-us-east-1e/host-rr19 changed its IP from to 192.168.14.13 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16812]: hosts-us-east-1e/host-rr18 changed its IP from to 192.168.13.88 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16812]: hosts-us-east-1e/host-rr21 changed its IP from to 192.168.14.33 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16812]: hosts-us-east-1e/host-rr20 changed its IP from to 192.168.14.29 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16812]: hosts-us-east-1e/host-rr14 changed its IP from to 192.168.13.58 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16812]: hosts-us-east-1e/host-rr23 changed its IP from to 192.168.14.85 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16808]: hosts-us-east-1e/host-rr20 changed its IP from to 192.168.14.13 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16808]: hosts-us-east-1e/host-rr21 changed its IP from to 192.168.14.29 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16808]: hosts-us-east-1e/host-rr22 changed its IP from to 192.168.14.33 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16808]: hosts-us-east-1e/host-rr19 changed its IP from to 192.168.13.88 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16808]: hosts-us-east-1e/host-rr24 changed its IP from to 192.168.14.85 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16809]: hosts-us-east-1e/host-rr12 changed its IP from to 192.168.13.58 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16808]: hosts-us-east-1e/host-rr23 changed its IP from to 192.168.14.45 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16808]: hosts-us-east-1e/host-rr26 changed its IP from to 192.168.15.20 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16809]: hosts-us-east-1e/host-rr26 changed its IP from to 192.168.12.109 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16808]: hosts-us-east-1e/host-rr25 changed its IP from to 192.168.14.89 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16809]: hosts-us-east-1e/host-rr25 changed its IP from to 192.168.15.69 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16808]: hosts-us-east-1e/host-rr27 changed its IP from to 192.168.15.68 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16809]: hosts-us-east-1e/host-rr28 changed its IP from to 192.168.12.140 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16808]: hosts-us-east-1e/host-rr28 changed its IP from to 192.168.15.69 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16809]: hosts-us-east-1e/host-rr35 changed its IP from to 192.168.12.234 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16810]: hosts-us-east-1e/host-rr15 changed its IP from to 192.168.13.80 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16810]: hosts-us-east-1e/host-rr11 changed its IP from to 192.168.13.30 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16810]: hosts-us-east-1e/host-rr16 changed its IP from to 192.168.13.86 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16810]: hosts-us-east-1e/host-rr12 changed its IP from to 192.168.13.35 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16810]: hosts-us-east-1e/host-rr21 changed its IP from to 192.168.14.45 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16810]: hosts-us-east-1e/host-rr18 changed its IP from to 192.168.14.13 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16810]: hosts-us-east-1e/host-rr23 changed its IP from to 192.168.14.89 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16810]: hosts-us-east-1e/host-rr22 changed its IP from to 192.168.14.85 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16810]: hosts-us-east-1e/host-rr25 changed its IP from to 192.168.15.68 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16810]: hosts-us-east-1e/host-rr24 changed its IP from to 192.168.15.20 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16811]: hosts-us-east-1e/host-rr14 changed its IP from to 192.168.13.86 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16811]: hosts-us-east-1e/host-rr13 changed its IP from to 192.168.13.80 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16811]: hosts-us-east-1e/host-rr12 changed its IP from to 192.168.13.74 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16811]: hosts-us-east-1e/host-rr15 changed its IP from to 192.168.13.88 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16811]: hosts-us-east-1e/host-rr16 changed its IP from to 192.168.14.13 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16811]: hosts-us-east-1e/host-rr18 changed its IP from to 192.168.14.33 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16811]: hosts-us-east-1e/host-rr17 changed its IP from to 192.168.14.29 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16811]: hosts-us-east-1e/host-rr19 changed its IP from to 192.168.14.45 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16811]: hosts-us-east-1e/host-rr20 changed its IP from to 192.168.14.85 by dns1/dns.
Jan 25 02:37:47 localhost haproxy[16811]: hosts-us-east-1e/host-rr21 changed its IP from to 192.168.14.89 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16810]: hosts-us-east-1e/host-rr26 changed its IP from to 192.168.15.69 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16810]: hosts-us-east-1e/host-rr27 changed its IP from to 192.168.12.109 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16810]: hosts-us-east-1e/host-rr29 changed its IP from to 192.168.12.140 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16810]: hosts-us-east-1e/host-rr28 changed its IP from to 192.168.12.111 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16810]: hosts-us-east-1e/host-rr30 changed its IP from to 192.168.12.152 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16810]: hosts-us-east-1e/host-rr32 changed its IP from to 192.168.12.160 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16810]: hosts-us-east-1e/host-rr34 changed its IP from to 192.168.12.195 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16810]: hosts-us-east-1e/host-rr35 changed its IP from to 192.168.12.231 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16810]: hosts-us-east-1e/host-rr36 changed its IP from to 192.168.12.234 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16810]: hosts-us-east-1e/host-rr33 changed its IP from to 192.168.12.167 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16807]: hosts-us-east-1e/host-rr24 changed its IP from to 192.168.12.109 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16807]: hosts-us-east-1e/host-rr23 changed its IP from to 192.168.15.69 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16807]: hosts-us-east-1e/host-rr26 changed its IP from to 192.168.12.140 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16807]: hosts-us-east-1e/host-rr27 changed its IP from to 192.168.12.152 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16807]: hosts-us-east-1e/host-rr29 changed its IP from to 192.168.12.160 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16807]: hosts-us-east-1e/host-rr30 changed its IP from to 192.168.12.167 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16807]: hosts-us-east-1e/host-rr31 changed its IP from to 192.168.12.195 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16807]: hosts-us-east-1e/host-rr25 changed its IP from to 192.168.12.111 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16807]: hosts-us-east-1e/host-rr32 changed its IP from to 192.168.12.231 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16807]: hosts-us-east-1e/host-rr33 changed its IP from to 192.168.12.234 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16812]: hosts-us-east-1e/host-rr22 changed its IP from to 192.168.14.45 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16812]: hosts-us-east-1e/host-rr15 changed its IP from to 192.168.13.74 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16812]: hosts-us-east-1e/host-rr35 changed its IP from to 192.168.12.195 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16812]: hosts-us-east-1e/host-rr36 changed its IP from to 192.168.12.231 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16812]: hosts-us-east-1e/host-rr37 changed its IP from to 192.168.12.234 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16812]: hosts-us-east-1e/host-rr38 changed its IP from to 192.168.12.241 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16812]: hosts-us-east-1e/host-rr40 changed its IP from to 192.168.13.127 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16812]: hosts-us-east-1e/host-rr39 changed its IP from to 192.168.13.108 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16812]: hosts-us-east-1e/host-rr42 changed its IP from to 192.168.13.217 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16812]: hosts-us-east-1e/host-rr41 changed its IP from to 192.168.13.158 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16809]: hosts-us-east-1e/host-rr27 changed its IP from to 192.168.12.111 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16809]: hosts-us-east-1e/host-rr34 changed its IP from to 192.168.12.231 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16809]: hosts-us-east-1e/host-rr36 changed its IP from to 192.168.12.241 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16809]: hosts-us-east-1e/host-rr37 changed its IP from to 192.168.13.108 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16809]: hosts-us-east-1e/host-rr38 changed its IP from to 192.168.13.127 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16809]: hosts-us-east-1e/host-rr39 changed its IP from to 192.168.13.158 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16809]: hosts-us-east-1e/host-rr42 changed its IP from to 192.168.14.161 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16809]: hosts-us-east-1e/host-rr43 changed its IP from to 192.168.14.202 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16809]: hosts-us-east-1e/host-rr45 changed its IP from to 192.168.15.123 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16809]: hosts-us-east-1e/host-rr44 changed its IP from to 192.168.14.239 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16808]: hosts-us-east-1e/host-rr30 changed its IP from to 192.168.12.111 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16808]: hosts-us-east-1e/host-rr29 changed its IP from to 192.168.12.109 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16808]: hosts-us-east-1e/host-rr31 changed its IP from to 192.168.12.140 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16808]: hosts-us-east-1e/host-rr33 changed its IP from to 192.168.12.154 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16808]: hosts-us-east-1e/host-rr34 changed its IP from to 192.168.12.160 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16808]: hosts-us-east-1e/host-rr35 changed its IP from to 192.168.12.167 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16808]: hosts-us-east-1e/host-rr38 changed its IP from to 192.168.12.234 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16808]: hosts-us-east-1e/host-rr37 changed its IP from to 192.168.12.231 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16808]: hosts-us-east-1e/host-rr36 changed its IP from to 192.168.12.195 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16808]: hosts-us-east-1e/host-rr39 changed its IP from to 192.168.12.241 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16811]: hosts-us-east-1e/host-rr22 changed its IP from to 192.168.15.20 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16811]: hosts-us-east-1e/host-rr24 changed its IP from to 192.168.15.69 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16811]: hosts-us-east-1e/host-rr23 changed its IP from to 192.168.15.68 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16811]: hosts-us-east-1e/host-rr26 changed its IP from to 192.168.12.111 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16811]: hosts-us-east-1e/host-rr28 changed its IP from to 192.168.12.152 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16811]: hosts-us-east-1e/host-rr27 changed its IP from to 192.168.12.140 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16811]: hosts-us-east-1e/host-rr29 changed its IP from to 192.168.12.154 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16811]: hosts-us-east-1e/host-rr30 changed its IP from to 192.168.12.160 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16811]: hosts-us-east-1e/host-rr31 changed its IP from to 192.168.12.167 by dns1/dns.
Jan 25 02:37:48 localhost haproxy[16811]: hosts-us-east-1e/host-rr32 changed its IP from to 192.168.12.195 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16812]: hosts-us-east-1e/host-rr43 changed its IP from to 192.168.13.249 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16812]: hosts-us-east-1e/host-rr44 changed its IP from to 192.168.14.161 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16812]: hosts-us-east-1e/host-rr45 changed its IP from to 192.168.14.202 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16812]: hosts-us-east-1e/host-rr47 changed its IP from to 192.168.15.123 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16812]: hosts-us-east-1e/host-rr46 changed its IP from to 192.168.14.239 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16809]: hosts-us-east-1e/host-rr40 changed its IP from to 192.168.13.217 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16809]: hosts-us-east-1e/host-rr41 changed its IP from to 192.168.13.249 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16807]: hosts-us-east-1e/host-rr15 changed its IP from to 192.168.14.13 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16807]: hosts-us-east-1e/host-rr28 changed its IP from to 192.168.12.154 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16807]: hosts-us-east-1e/host-rr37 changed its IP from to 192.168.13.158 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16807]: hosts-us-east-1e/host-rr36 changed its IP from to 192.168.13.127 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16807]: hosts-us-east-1e/host-rr38 changed its IP from to 192.168.13.217 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16807]: hosts-us-east-1e/host-rr39 changed its IP from to 192.168.13.249 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16807]: hosts-us-east-1e/host-rr40 changed its IP from to 192.168.14.161 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16807]: hosts-us-east-1e/host-rr41 changed its IP from to 192.168.14.202 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16807]: hosts-us-east-1e/host-rr34 changed its IP from to 192.168.12.241 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16807]: hosts-us-east-1e/host-rr42 changed its IP from to 192.168.14.239 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16810]: hosts-us-east-1e/host-rr37 changed its IP from to 192.168.12.241 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16810]: hosts-us-east-1e/host-rr31 changed its IP from to 192.168.12.154 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16810]: hosts-us-east-1e/host-rr41 changed its IP from to 192.168.13.217 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16810]: hosts-us-east-1e/host-rr40 changed its IP from to 192.168.13.158 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16810]: hosts-us-east-1e/host-rr42 changed its IP from to 192.168.13.249 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16810]: hosts-us-east-1e/host-rr43 changed its IP from to 192.168.14.161 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16810]: hosts-us-east-1e/host-rr44 changed its IP from to 192.168.14.202 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16810]: hosts-us-east-1e/host-rr45 changed its IP from to 192.168.14.239 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16810]: hosts-us-east-1e/host-rr38 changed its IP from to 192.168.13.108 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16810]: hosts-us-east-1e/host-rr47 changed its IP from to 192.168.15.126 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16808]: hosts-us-east-1e/host-rr32 changed its IP from to 192.168.12.152 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16808]: hosts-us-east-1e/host-rr40 changed its IP from to 192.168.13.108 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16808]: hosts-us-east-1e/host-rr43 changed its IP from to 192.168.13.217 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16808]: hosts-us-east-1e/host-rr44 changed its IP from to 192.168.13.249 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16808]: hosts-us-east-1e/host-rr45 changed its IP from to 192.168.14.161 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16808]: hosts-us-east-1e/host-rr46 changed its IP from to 192.168.14.202 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16808]: hosts-us-east-1e/host-rr47 changed its IP from to 192.168.14.239 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16808]: hosts-us-east-1e/host-rr41 changed its IP from to 192.168.13.127 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16808]: hosts-us-east-1e/host-rr42 changed its IP from to 192.168.13.158 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16811]: hosts-us-east-1e/host-rr33 changed its IP from to 192.168.12.231 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16811]: hosts-us-east-1e/host-rr34 changed its IP from to 192.168.12.234 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16811]: hosts-us-east-1e/host-rr36 changed its IP from to 192.168.13.108 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16811]: hosts-us-east-1e/host-rr37 changed its IP from to 192.168.13.127 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16811]: hosts-us-east-1e/host-rr39 changed its IP from to 192.168.13.217 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16811]: hosts-us-east-1e/host-rr38 changed its IP from to 192.168.13.158 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16811]: hosts-us-east-1e/host-rr41 changed its IP from to 192.168.14.161 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16811]: hosts-us-east-1e/host-rr40 changed its IP from to 192.168.13.249 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16811]: hosts-us-east-1e/host-rr35 changed its IP from to 192.168.12.241 by dns1/dns.
Jan 25 02:37:50 localhost haproxy[16811]: hosts-us-east-1e/host-rr42 changed its IP from to 192.168.14.202 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16807]: hosts-us-east-1e/host-rr35 changed its IP from to 192.168.13.108 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16807]: hosts-us-east-1e/host-rr43 changed its IP from to 192.168.15.123 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16807]: hosts-us-east-1e/host-rr46 changed its IP from to 192.168.15.155 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16807]: hosts-us-east-1e/host-rr47 changed its IP from to 192.168.15.159 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16807]: hosts-us-east-1e/host-rr44 changed its IP from to 192.168.15.126 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16807]: hosts-us-east-1e/host-rr45 changed its IP from to 192.168.15.144 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16810]: hosts-us-east-1e/host-rr39 changed its IP from to 192.168.13.127 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16810]: hosts-us-east-1e/host-rr46 changed its IP from to 192.168.15.123 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16811]: hosts-us-east-1e/host-rr43 changed its IP from to 192.168.14.239 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16811]: hosts-us-east-1e/host-rr25 changed its IP from to 192.168.12.109 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16811]: hosts-us-east-1e/host-rr46 changed its IP from to 192.168.15.144 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16811]: hosts-us-east-1e/host-rr47 changed its IP from to 192.168.15.155 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16811]: hosts-us-east-1e/host-rr45 changed its IP from to 192.168.15.126 by dns1/dns.
Jan 25 02:37:51 localhost haproxy[16811]: hosts-us-east-1e/host-rr44 changed its IP from to 192.168.15.123 by dns1/dns.
root@prod-loadbalancer-0.com [~]# date
Thu Jan 25 07:18:23 UTC 2018

for i in $(seq 1 6); do echo “show servers state” | socat unix-connect:/var/run/haproxy-$i.sock stdio|grep ec2.int|wc -l; done;echo;host -t srv _http._tcp.us-east.com|grep ec2.int|wc -l
47
47
47
47
47
47

29

As you can see haproxy has 47 servers in backend but srv record only 29. Testing on the latest stable build

haproxy -vv

HA-Proxy version 1.8.3-945f4cf 2018/01/23
Copyright 2000-2017 Willy Tarreau willy@haproxy.org

5

I don’t see how you can count this correctly with the grep command you are using.

Most of your servers are probably in SRV_ADMF_RMAINT.

Replace the the first grep “grep ec2.int” with (checking for servers with srv_admin_state = 0):
grep -P " \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} \d 0 "

6

Maybe I’ll agree with you but why in the first example I gave processes had different numbers of backend servers?

for i in $(seq 1 6); do echo “show servers state” | socat unix-connect:/var/run/haproxy-$i.sock stdio|grep .int|wc -l; done
25
37
25
25
25
25

And after restart they all have 37 servers

7

Sounds like you just added the 12 servers and haproxy was in the process of catching up with the new DNS entries. Or you actually did hit a bug, but that was with the vanilla release 1.8.3, which contains a number of bugs.

I suggest you monitor the situation now and in the case you do still see any issues (considering only servers with srv_admin_state = 0), you provide the entire output of every single process, not just the count of the grep output. Otherwise there is simply no way to tell what is going on.

8

Hi guys,

I can see than each server get a new IP very frequently. It is possible
that the server does not send enough records and some of them get obsolete
quickly.
You can try to enforce a “hold obsolete 30s” so if HAProxy does not see the
IP address in the response for 30s, it will consider it’s is definitively
dead.

Baptiste

9

Ok, i’m leaving host with yesterday release w/o reboots and going to add new information on Monday.

10

Somehow build from 24-January was working fine for me. And now i’m testing 26-January(1.8.3-2dd90ea) build.

11

@lukastribus Here we are

be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port
4 hosts-us-east-1d 1 host-rr1 192.168.11.133 2 64 1 1 204813 15 3 4 6 0 0 0 ip-192-168-11-133.ec2.internal 80
4 hosts-us-east-1d 2 host-rr2 192.168.10.242 2 0 1 1 261203 15 3 4 6 0 0 0 ip-192-168-10-242.ec2.internal 80
4 hosts-us-east-1d 3 host-rr3 192.168.8.62 2 64 1 1 212524 15 3 4 6 0 0 0 ip-192-168-8-62.ec2.internal 80
4 hosts-us-east-1d 4 host-rr4 192.168.11.108 2 0 1 1 261203 15 3 4 6 0 0 0 ip-192-168-11-108.ec2.internal 80
4 hosts-us-east-1d 5 host-rr5 192.168.9.66 2 64 1 1 18161 15 3 4 6 0 0 0 ip-192-168-9-66.ec2.internal 80
4 hosts-us-east-1d 6 host-rr6 192.168.11.101 2 64 1 1 40842 15 3 4 6 0 0 0 ip-192-168-11-101.ec2.internal 80
4 hosts-us-east-1d 7 host-rr7 192.168.11.130 2 64 1 1 33785 15 3 4 6 0 0 0 ip-192-168-11-130.ec2.internal 80
4 hosts-us-east-1d 8 host-rr8 192.168.11.132 2 64 1 1 39641 15 3 4 6 0 0 0 ip-192-168-11-132.ec2.internal 80
4 hosts-us-east-1d 9 host-rr9 192.168.8.63 2 64 1 1 40865 15 3 4 6 0 0 0 ip-192-168-8-63.ec2.internal 80
4 hosts-us-east-1d 10 host-rr10 192.168.8.73 2 64 1 1 37599 15 3 4 6 0 0 0 ip-192-168-8-73.ec2.internal 80
4 hosts-us-east-1d 11 host-rr11 192.168.8.126 2 64 1 1 37611 15 3 4 6 0 0 0 ip-192-168-8-126.ec2.internal 80
4 hosts-us-east-1d 12 host-rr12 192.168.11.174 2 64 1 1 212137 15 3 4 6 0 0 0 ip-192-168-11-174.ec2.internal 80
4 hosts-us-east-1d 13 host-rr13 192.168.11.165 2 64 1 1 25365 15 3 4 6 0 0 0 ip-192-168-11-165.ec2.internal 80
4 hosts-us-east-1d 14 host-rr14 192.168.9.107 2 64 1 1 41342 15 3 4 6 0 0 0 ip-192-168-9-107.ec2.internal 80
4 hosts-us-east-1d 15 host-rr15 192.168.9.16 2 0 1 1 261203 15 3 4 6 0 0 0 ip-192-168-9-16.ec2.internal 80
4 hosts-us-east-1d 16 host-rr16 192.168.11.55 2 64 1 1 118851 15 3 4 6 0 0 0 ip-192-168-11-55.ec2.internal 80
4 hosts-us-east-1d 17 host-rr17 192.168.9.95 2 0 1 1 261203 15 3 4 6 0 0 0 ip-192-168-9-95.ec2.internal 80
4 hosts-us-east-1d 18 host-rr18 192.168.10.43 2 0 1 1 261203 15 3 4 6 0 0 0 ip-192-168-10-43.ec2.internal 80
4 hosts-us-east-1d 19 host-rr19 192.168.9.236 2 64 1 1 39271 15 3 4 6 0 0 0 ip-192-168-9-236.ec2.internal 80
4 hosts-us-east-1d 20 host-rr20 192.168.10.178 2 64 1 1 201690 15 3 4 6 0 0 0 ip-192-168-10-178.ec2.internal 80
4 hosts-us-east-1d 21 host-rr21 192.168.9.67 0 96 1 1 18143 8 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 22 host-rr22 192.168.11.26 0 96 1 1 33776 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 23 host-rr23 192.168.11.79 2 0 1 1 261203 15 3 4 6 0 0 0 ip-192-168-11-79.ec2.internal 80
4 hosts-us-east-1d 24 host-rr24 192.168.11.80 0 96 1 1 194248 8 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 25 host-rr25 192.168.8.103 0 96 1 1 109472 8 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 26 host-rr26 192.168.8.150 0 96 1 1 204816 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 27 host-rr27 192.168.8.205 2 0 1 1 261199 15 3 4 6 0 0 0 ip-192-168-8-205.ec2.internal 80
4 hosts-us-east-1d 28 host-rr28 192.168.8.207 0 96 1 1 208701 8 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 29 host-rr29 192.168.8.235 0 96 1 1 208731 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 30 host-rr30 192.168.9.126 0 96 1 1 199268 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 31 host-rr31 192.168.9.172 0 96 1 1 206276 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 32 host-rr32 192.168.9.178 2 0 1 1 261200 15 3 4 6 0 0 0 ip-192-168-9-178.ec2.internal 80
4 hosts-us-east-1d 33 host-rr33 192.168.9.207 0 96 1 1 210366 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 34 host-rr34 192.168.10.108 0 96 1 1 210366 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 35 host-rr35 192.168.10.114 0 96 1 1 201688 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 36 host-rr36 192.168.10.116 0 96 1 1 230202 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 37 host-rr37 192.168.10.154 0 96 1 1 127780 7 2 0 14 0 0 0 - 80
4 hosts-us-east-1d 38 host-rr38 192.168.10.167 2 0 1 1 261199 15 3 4 6 0 0 0 ip-192-168-10-167.ec2.internal 80
4 hosts-us-east-1d 39 host-rr39 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 40 host-rr40 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 41 host-rr41 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 42 host-rr42 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 43 host-rr43 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 44 host-rr44 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 45 host-rr45 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 46 host-rr46 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 47 host-rr47 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 48 host-rr48 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 49 host-rr49 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 50 host-rr50 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 51 host-rr51 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 52 host-rr52 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 53 host-rr53 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 54 host-rr54 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 55 host-rr55 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 56 host-rr56 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 57 host-rr57 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 58 host-rr58 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 59 host-rr59 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 60 host-rr60 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 61 host-rr61 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 62 host-rr62 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 63 host-rr63 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 64 host-rr64 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 65 host-rr65 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 66 host-rr66 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 67 host-rr67 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 68 host-rr68 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 69 host-rr69 0 0 1 1 261202 5 2 0 6 0 0 0 - 0
4 hosts-us-east-1d 70 host-rr70 0 0 1 1 261202 5 2 0 6 0 0 0 - 0

haproxy -vv
HA-Proxy version 1.8.4-1deb90d 2018/02/08
Copyright 2000-2018 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.1k-fips 8 Jan 2015
Running on OpenSSL version : OpenSSL 1.0.1k-fips 8 Jan 2015
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.21 2011-12-12
Running on PCRE version : 8.21 2011-12-12
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with network namespace support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

for i in $(seq 1 6); do echo “show servers state” | socat unix-connect:/var/run/haproxy-$i.sock stdio|grep ec2.int|wc -l; done;host -t srv _http._tcp|grep ec2.int|wc -l
24
24
38
38
24
38
38

Last time we saw logs from 1st pid in logs

Feb 12 11:13:05 localhost haproxy[388]: hosts-us-east-1d/host-rr5 changed its IP from 172.30.9.168 to 172.30.9.66 by dns/dns.
Feb 12 11:13:05 localhost haproxy[388]: Server host-us-east-1d/host-rr5 administratively READY thanks to valid DNS answer.
another pids still present in the log file

Feb 12 16:38:29 localhost haproxy[393]: Server host-us-east-1d/host-rr38 administratively READY thanks to valid DNS answer.

date
Mon Feb 12 16:56:39 UTC 2018

This is stable state. I see the same numbers during ten minutes at least

12

Thanks. The delta are 14 servers and they are in srv_op_state = 0x0 (SRV_ST_STOPPED) and srv_admin_state = 0x60 (dec 96), the latter being a invalid state as far as I can tell. You should never see a server with the srv_admin_state set to 96.

0x60 (96) is a combination between SRV_ADMF_RMAINT and SRV_ADMF_HMAINT state, which I doubt is expected behavior. So we hit a code path here that brings some server slots into a bogus state.

@Baptiste do you see anything in the code that could cause this?

13

@lukastribus Hi! Is there anything I can do from my said to help with resolving this bug?

14

At this point we will need input from @Baptiste

15

@ASA please try to remove the automatic payload downgrade on packet loss, by removing line 1742 and 1743 from src/dns.c:

http://git.haproxy.org/?p=haproxy-1.8.git;a=blob;f=src/dns.c;hb=HEAD#l1742

1742                         if (resolvers->accepted_payload_size > 1280)
1743                                 resolvers->accepted_payload_size = 1280;
16

Got it. Going to test it today.

17

@ASA, did you try adding a “hold obsolete 30s” in your resolvers section?

I also sent a patch on the ML, could you try to apply it and say if you
still have this behavior:
https://www.mail-archive.com/haproxy@formilux.org/msg29040.html

18

@Baptiste Trying your patch + config parameter on one server
@lukastribus Works fine for me.

19

@Baptiste Your patch is not working for me:

for i in $(seq 1 6); do echo "show servers state"| socat unix-connect:/var/run/haproxy-$i.sock stdio|grep ec2.int|wc -l; done;host -t srv _http._tcp|grep ec2.int|wc -l
47
47
47
47
47
47
49

I fully agree with @lukastribus that such quite behavior of changing payload to default value is very hard to debug and quite uncomfortable in production environment. If you do not plan to add support of TCP for DNS resolution in 1.8.x it is better to add “fallback to 1280” as option (enabled or disabled by default) and at least message to logs about this problem.

20

@ASA the downgrade has been officially removed in 1.8.5.