Kibana behind Haproxy: GUI not loading fully

Help!
1

Hello,

Using the deviantony/docker-elk repo, i recently setup ELK on an EC2 instance running Docker (version 17.05.0-ce, build 89658be) and docker-compose (docker-compose version 1.6.2, build 4d72027). I also used the docker-elk-filebeat repo to install filebeat.

After configuring everything, I’m able to see basic log file entries flowing thru such that I can view them in Kibana: I have to access the GUI, though, via an ssh tunnel into my Docker host.

I would like, though, to be able to access the Kibana GUI via an haproxy container that’s also running on my Docker host. That way I’d be able to just hit “http://<EC2 instance’s IP>/logs” and have this redirect to Kibana, without having to open port 5601 on the host and modify the current security group.

When I try, though, the UI loads partially, then just stops. Clicking on the different options (Discover, Visualize, etc) changes the URL listed (to one that contains “myapp-demo/app/kibana#/home?_g=()”, “myapp-demo/app/kibana#/discover” etc), but that’s it. Oh, and I have a simple entry in my local /etc/hosts file mapping myapp-demo to the AWS instance’s IP.

Security groups are setup, temporarily, on the EC2 instance such that ports 22, 5601, 9200, 9100 are all open to my admin workstation’s IP. iptables has only the entries that were put in place by the installation of Docker (iptables wasn’t even there on the Docker host, before I installed Docker).

Here’s an excerpt from my haproxy.cfg file (I’ve created a pastebin at https://pastebin.com/yavnXyEw, just in case my posting proves difficult to read):

frontend web
bind *:80
reqadd X-Forwarded-Proto:\ http if !{ ssl_fc }
reqadd X-Forwarded-Proto:\ https if { ssl_fc }

default_backend myapp_ui

…
acl is_kibana path_sub logs
acl is_kibana path_sub kibana
acl is_kibana path_sub bundles
acl is_kibana path_sub app/kibana
acl is_kibana path_sub plugins/kibana/assets

use_backend kibana if is_kibana
use_backend es_head if is_es_head

…

backend myapp_ui
server ui1 myapp:8080 check

backend kibana
mode http
reqrep ^([^\ ] \ /)logs[/]?(. ) \1\2
reqrep ^([^\ ] \ /)kibana[/]?(. ) \1\2
server kibana kibana:5601 check

And here’s an excerpt from my docker-compose.yml file:

haproxy:
image: < personal repo >
restart: always
links:

  • myapp
  • kibana
  • es_head
    volumes:
  • ./docker/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
    ports:
  • “80:80”

kibana:
image: < personal repo >
ports:

  • “5601:5601”

Now, since I’m able to get the GUI to load partially, I know that the setup is mostly functional. The issue seems to be, perhaps, with my haproxy acls. I’m not sure what else I need to add in that area.

'Anyone out there have experience with this kind of setup and any ideas on what I can do to get around the issue that I’m seeing?

2

Check your browser log to see what requests fail. I assume your ACL’s don’t cover everything.

Why is it that you need all this ACLs anyway? What is it that you are trying to accomplish with this haproxy configuration, other than accessing kibana?

3

Thank you for you reply.

ELK is an add-on, here. The main job of the haproxy container is to grant access to “myapp”. All of the ACLs and both of the regrep calls I added as part of troubleshooting the problem with loading the Kibana GUI.

Yeah, I thought that gathering log information might be helpful. So, I carried out the following steps:

1.) Added the following to my ~/.bash_profile:

alias ogc-debug=‘open -a Google\ Chrome --args --enable-logging --v=2’

2.) Quit and restarted iTerm

3.) Ran “ogc-debug” to launch Chrome.

I’m going thru the log file, now, but have pasted it at https://pastebin.com/HgdCCDLF, just in case you have a sense of how to quickly determine what ACLs need to be added. I wonder if there’s a way of specifying a wild card… something like myapp/*, to cover myapp/kibana, myapp/bundles, myapp/plugins, myapp/api. I’m not even sure if that would do the trick, though.

4

Hmm, it seems that sections like the one that I’ve pasted/re-pasted at https://pastebin.com/1g4AtY7G might also be significant. It got me to thinking that adding a " reqrep ^([^\ ] \ /)myapp[/]?(. ) \1\2" might help, but this doesn’t seem to have made a difference.

5

I didn’t mean debug log, I just meant open the developer console in your browser with F12, go to the network tab, repeat the request and check the failed request that are marked red.

The haproxy config should be as simple as:

frontend web
bind *:80
default_backend kibana

backend kibana
 mode http
 server kibana kibana:5601 check

All those ACL and reqrep are just messing with the requests and probably denying some of them, likely being the root cause in the first place.

6

Ok, I’ll give that I try. I’ve not used either of these logging mechanisms. So, I had no idea what you meant.

The main thing behind this haproxy container is the myapp container/service. I started with something similar to what you’ve described but default_backend kibana won’t work, because with that, requests won’t make it thru to the myapp backend.

7

default_backend redirects everything, if that doesn’t work any other configuration will fail as well. Making the configuration more complicated is not a step in the right direction.

8

Ok, now I’m thoroughly confused. I wasn’t simply making things more complicated. I was just trying to get things flowing to where it should be going.

If default_backend redirects everything to kibana, how will traffic meant for myapp get to that backend?

9

Isn’t myapp kibana? Also what is es_head and the ACL is_es_head?

What you are saying is that you want haproxy to forward everything to kabana, but what you are doing is something else entirely.

Is Kibana the only application haproxy needs to forward or not?

10

Oh, heck, I’ve miscommunicated, here, somewhere.

Before I even brought ELK into the picture, a couple of weeks ago, this was a docker’ized application with the following containers/services: myapp, mongo, webcalc, haproxy. It’s essentially a custom app to which I’m attempting to add ELK, for log aggregation. So, where haproxy formerly stood in front of the custom “myapp”, I am now trying to have it stand in front of Kibana, as well. So, a user would hit http://myapp, to get to the custom app, and http://myapp/logs to get to Kibana.

es_head is for elasticsearch-head, the ES cluster health webapp. I wasn’t even sure I’d be able to get that to work, but I was hoping to make it such that a person could hit …/myapp/es_health and be redirected to the elasticsearch-head service that’s on docker-host:9100.

11

It’s a terrible idea to put an application into a subdir like /logs and then expect the proxy to handle everything.

Unless you know the application inside out and adjusted every single api endpoint, every single css, png, jpeg and js reference in the html code manually, this will not work.

Haproxy will not replace URLs in your HTTP payload to replace the wrong references in the page.

I strongly suggest you use different Host’s and content-switch based on that. That will make everything very easy and reliable.

12

Well, I am working with the the actual Developers of the app. So, they certainly know it inside out.

“It’s a terrible idea to put an application into a subdir like /logs and then expect the proxy to handle everything.” the tone in this is not too in keeping with the word "community.

13

I was not trying to come off as aggressive, my intention was to explain that this is a bad idea, and that haproxy does not support rewriting links within the HTTP payload.

Now, if you can configure kibana in a way so that it understands everything is under /logs instead of /, then all you need in haproxy is:

use_backend kibana if { path_beg /logs }

no other acls, not other rules and certainly no reqrep.

If kibana can’t be configured to do this, than it’s difficult to accomplish this, because you need to know every wrong URI call and catch in haproxy, and it’s possible it will collide the other application.

That’s where dedicated hostname would come in handy, which makes this very easy, without the need to configure anything on the backend application.