503 service unavail for ES ingest backend

Help!
1

Trying to put HAp 1.8.19 in between elastic clients and elastic 6.8.1 ingest/coordinate nodes, but even though HAp have a healthy backend, it claims no service available, when hitting the frontend, hints appreciated, TIA.

tcpdumped reply to clients on HAp

E…@.@…J>.)…R.#…9d…`…P…F…HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html

503 Service Unavailable

No server is available to handle this request.

tcpdumped HAp health check reply on backend:

E..f..@.@..L>.)E>.)L#.....n.g...P.......HTTP/1.0 200 OK
content-type: application/json; charset=UTF-8
content-length: 487
{
  "name" : "es-i1",
  "cluster_name" : "mxes2data",
  "cluster_uuid" : "SmpMKhEwQs2r0G4eNOBJTA",
  "version" : {
    "number" : "6.8.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "1fad4e1",
    "build_date" : "2019-06-18T13:16:52.517138Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

HAproxy config snippet:

defaults
   maxconn 1024
   email-alert mailers sysadminmailer
   email-alert from "${INSTANCE}@${DOMAIN}"
   email-alert to "${ALERTEE}"
   mode http
   rate-limit sessions 20
   #log global
   #option  httplog
   option  dontlognull             # Do not log connections with no requests
   option  dontlog-normal          # Do not log successfull connections, ie. log on servers
   option  contstats               # Enable continuous traffic statistics updates
   retries 2                       # Try to connect up to 2 times in case of failure
   timeout connect 30s             # max to connect or to stay in queue
   timeout client 30s
   timeout server 30s
   timeout http-keep-alive 1s      # max for the client to post next request
   timeout http-request 15s        # max for the client to send a request
   timeout queue 30s               # max queued on load balancer
   timeout check 60s               # timeout health check read
   backlog 1000                    # Size of SYN backlog queue
   source 0.0.0.0 usesrc clientip  # transparent proxy mode
   balance leastconn

frontend fe-esi
   bind-process 6
   bind ipv4@*:9200 transparent mss 1460
   default_backend be-esi

backend be-esi
   bind-process 6
   option httpchk GET /
   default-server inter 30s downinter 60s rise 2
   server i1 ipv4@es-i1:9200 check

06
061418×167 32.3 KB

2

I assume you did not properly setup up the transparent mode.

Please remove source 0.0.0.0 usesrc clientip to confirm.

3

Thanks, only has had transperant proxy for ages on this HAp cluster and it works on other fe/be even under the same default config section I’ve got another fe/be for kibana which works just fine and be server are seen true client IPs.

my ES nodes are even cloned from kibana vhosts

/etc/rc.local snippet:
# required by haproxy to do transparent proxying
/sbin/modprobe nf_synproxy_core
/sbin/ip rule add fwmark 1 lookup haprxmark
/sbin/ip route add local 0.0.0.0/0 dev lo table haprxmark


cli# ip route list table haprxmark
local default dev lo  scope host
4

Well you are here to troubleshoot a problem, right? Whether this worked for ages and you are only cloning VMs not modifying there configuration is irrelevant if it doesn’t actually work for this specific VM. You still have to troubleshoot why it doesn’t work for that VM.

Health checks don’t spoof source IP’s, therefor are not impacted if transparent mode is incorrectly setup.

Check haproxy and backend logs and capture the traffic between them, if you want certainty.

But I assume the backend network configuration is wrong.

5

Yes yes :slight_smile: U’re right, appreciates, I may have forgotten what might be needed on the be servers for proxy to work other that a proper default route back to a HAp floating GW IP (floating between HAp cluster nodes)…

Anyway as you expected it works for the new ES be, if I move the source 0.0.0.0 setting from default to other be section…