Issues with connecting to backend server though URL redirection happens

Help!
1

Hi Team,

I have the following frontend and backend settings in my haproxy configuration file. Even though haproxy redirection to the backend URL it is failing to connect to the backend server.

Error detail :-Unable to connect to Ambari Server. Confirm Ambari Server is running and you can reach Ambari Server from this machine.

#---------------------------------------------------------------------

main frontend which proxys to the backends

#---------------------------------------------------------------------

frontend main *:80

acl url_dcos path_beg /dcos
acl url_ambari path_beg /ambari

use_backend dcos-backend if url_dcos
use_backend ambari-backend if url_ambari

#---------------------------------------------------------------------

round robin balancing between the various backends

#---------------------------------------------------------------------
backend dcos-backend
reqrep ^([^\ ])\ /dcos(/.) \1\ \2
cookie SERVERID insert indirect nocache
{% for master in groups[‘master-nodes’] %}
server dcos_master{{ loop.index0 }} {{ hostvars[master].ansible_host }}:80 check cookie dcos_master{{ loop.index0 }}
{% endfor %}

backend ambari-backend
reqrep ^([^\ ])\ /ambari(/.) \1\ \2
cookie SERVERID insert indirect nocache
{% for mgmt in groups[‘management-nodes’] %}
server ambari_server{{ loop.index0 }} {{ hostvars[mgmt].ansible_host }}:8080 check cookie ambari_server{{ loop.index0 }}
{% endfor %}

Please Guide

2

Don’t post your configuration templates. Post the ACTUAL and complete haproxy configuration. We cannot possible imagine what your provisioning tools deploy to the box.

Also, enable logging and post the haproxy error message.

3

Below is the error collated from tcpdump related to haproxy issue

"
_timestamp=1507287513523 HTTP/1.1
Host: 35.157.210.233
Accept: application/json, text/javascript
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
Content-Type: application/json; charset=utf-8
Referer: http://35.157.210.233/dcos/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: AMBARISESSIONID=91jec39gbymua8pkezshb7v; SERVERID=dcos_master1; ajs_anonymous_id=%22d372abdf-2ed9-42c9-9a9c-f1c6fa37c028%22; ajs_user_id=null; ajs_group_id=null; _ga=GA1.1.935682048.1507278288; _gid=GA1.1.357852378.1507278288
X-Origin: ASML
Via: 1.1 CWS_proxy
Cache-Control: max-age=259200
Connection: keep-alive

HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html

503 Service Unavailable

No server is available to handle this request. "
4

As the message says:

No server is available to handle this request

5

But when i set the default_backend to “dcos-backend” it works the issue seems to be with ACL redirection

6

Complete HA configuration details is as detaied below, its seems like ACL defined is not redirection to the backend URL properly.

#---------------------------------------------------------------------

Global settings

#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the ‘-r’ option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2

chroot      /var/lib/haproxy
pidfile     /var/run/haproxy.pid
maxconn     4000
user        haproxy
group       haproxy
daemon

# turn on stats unix socket
stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------

common defaults that all the ‘listen’ and ‘backend’ sections will

use if not designated in their block

#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000

#---------------------------------------------------------------------

main frontend which proxys to the backends

#---------------------------------------------------------------------

frontend main *:80
acl url_dcos path_sub /dcos
use_backend dcos-backend if url_dcos

acl url_ambari path_sub /ambari
use_backend ambari-backend if url_ambari
#default_backend ambari-backend

#---------------------------------------------------------------------

static backend for serving up images, stylesheets and such

#---------------------------------------------------------------------
backend static
balance roundrobin
server static 127.0.0.1:4331 check

#---------------------------------------------------------------------

round robin balancing between the various backends

#---------------------------------------------------------------------
backend dcos-backend
reqrep ^([^\ ])\ /dcos(/.) \1\ \2
cookie HAPROXY_DCOS insert indirect nocache
{% for master in groups[‘master-nodes’] %}
server dcos_master{{ loop.index0 }} {{ hostvars[master].ansible_host }}:80 check cookie dcos_master{{ loop.index0 }}
{% endfor %}

backend ambari-backend
reqrep ^([^\ ])\ /ambari(/.) \1\ \2
cookie HAPROXY_AMBARI insert indirect nocache
{% for mgmt in groups[‘management-nodes’] %}
server ambari_server{{ loop.index0 }} {{ hostvars[mgmt].ansible_host }}:8080 check cookie ambari_server{{ loop.index0 }}
{% endfor %}

7

So, what are you saying? What is it that you want to match?
The URL begins with /dcos but the ACL url_dcos does not work?

You still only posted your template configuration with Ansible variables. Post the configuration that haproxy actually uses. Also post that http log, and when you post the curl output, do not cut out the most important information, that is the request URI.

8

yes iam trying to see if the URL “eg:-http://35.157.207.3:80/dcos”(matches with /dcos) is requested it should ideally redirect to the backend server after removing the URL part “/dcos” on request

The same for Ambari server from the same management node “http://35.157.207.3:80/ambari” it should redirect to ambari backend.

35.157.207.3 is the management server IP, both the request comes to Port 80

9

The configuration which is send is the one installed using Ansible playbook which is copied to the management server below /etc/haproxy/haproxy.cfg , would you like to see the haproxy.cfg content after installation?

10

Yes, I would like the see the configuration file that haproxy reads from your hard disk before starting.
And the log output, as well as the COMPLETE request from the tcpdump output, not only the last part of it that you posted above.

11

Can i mail the tcpdump as well the configuration details from the server to your email id as it is huge to paste here. Kindly send me your email id?

12

Hi Lukas i have dropped a mail with the HAProxy Configuration post installation from the management server and tcpdump collated

13

When haproxy answers with

HTTP/1.0 503 Service Unavailable

It is because the requests does not contain /dcos.

For example, here’s a session from your capture:

--> GET /dcos-metadata/ui-config.json?_timestamp=1507287512865 HTTP/1.1
<-- HTTP/1.1 200 OK
--> GET /dcos-history-service/history/minute?_timestamp=1507287513315 HTTP/1.1
<-- HTTP/1.1 200 OK
--> GET /metadata?_timestamp=1507287513523 HTTP/1.1
<-- HTTP/1.0 503 Service Unavailable
14

Hi Lukas below is one the log details which i have seen in tcpdump file which contain /dcos in the path but it still fails

Accept: application/json, text/javascript
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
Content-Type: application/json; charset=utf-8
Referer: http://35.157.210.233/dcos/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: AMBARISESSIONID=91jec39gbymua8pkezshb7v; SERVERID=dcos_master1; ajs_anonymous_id=%22d372abdf-2ed9-42c9-9a9c-f1c6fa37c028%22; ajs_user_id=null; ajs_group_id=null; _ga=GA1.1.935682048.1507278288; _gid=GA1.1.357852378.1507278288
X-Origin: ASML
Via: 1.1 CWS_proxy
Cache-Control: max-age=259200
Connection: keep-alive

HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html

15

Then why do you omit the request part here?

16

Ideally the requirment is like if the user gives the request “http://management-node:80/dcos/” it should redirect to http://dcos_master URL , the same way for Ambari the user request expected is “http://management-node:80/ambari/” which should redirect to “http://management-node::8080” and it is expected to remove the “/dcos” or “/ambari” from the URL string in the backend.

17

Redirect or rewrite?

You have configured haproxy to reject requests that do not contain /dcos or /ambari.

Now you are saying you want to redirect the request and remove those strings, but then haproxy won’t be able match /dcos or /ambari anymore, because it would not be in the request any longer.

Also your trace contains a lot of requests that do not contain those strings at all, which is why the request obviously fails.

I’m not sure what it is that you are asking here. Haproxy behaves exactly what you configured it to do.

18

Yes my requirement is basically to redirect the URL to the respective backend server if the URL is in the form “http://management-node:80/dcos/” or “http://management-node:80/ambari/”

Please guide what correction should i make in my ansible HAProxy configuration file

19

What you are saying doesn’t make any sense and I explained the reason above.

Again, if you redirect from /dcos to /somethingelse, your ACL (matching the /dcos string) does no longer match.

20

ok so you mean it’s not possible to redirect from http://management-node:80/dcos/ to a backend URL with the mentioned ACL and regular expressions. Is there any alternative to achieve my goal