Styling fails when proxying with path

I’m trying to document the correct way to proxy to netdata with haproxy, on 127.0.0.1:19999. The problem I’m having is - only when running with a path - the styling fails. I assume I have to do something to rewrite the path, I’m not sure how to translate the known working configuration from nginx to haproxy.

The accepted nginx configuration is:

upstream netdata {
    # the Netdata server
    server 127.0.0.1:19999;
    keepalive 64;
}

server {
    listen 81;
    # the virtual host name of this subfolder should be exposed
    server_name netdata.domain.tld;

       location = /netdata {
                return 301 /netdata/;
        }

        location ~ /netdata/(?<ndpath>.*) {
                proxy_redirect off;
                proxy_set_header Host $host;

                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Server $host;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass_request_headers on;
                proxy_http_version 1.1;
                proxy_set_header Connection "keep-alive";
                proxy_store off;
                proxy_pass http://netdata/$ndpath$is_args$args;
        }
}

The configuration I’ve tried using is:

frontend http_frontend
    ## HTTP ipv4 and ipv6 on all ips ##
    bind :::80 v4v6
    # URL begins with /netdata
    acl is_netdata url_beg  /netdata
    ## Backends ##
    use_backend     netdata_backend       if is_netdata
    # Other requests go here (optional)
    default_backend www_backend

backend netdata_backend
    option forwardfor
    server      netdata_local     127.0.0.1:19999
    http-request set-header Host %[src]
    http-request set-header X-Forwarded-For %[src]
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request set-header Connection "keep-alive"

I tried using a rewrite to remove the sub path but wasn’t able to get it to work. Is there a equivalent configuration I should be using to the nginx configuration to get the styling working?

By looking at your NGinx configuration I see that you are in fact dropping the /netdata part of the path when sending your request to the actual server. Meanwhile you don’t do the same for the HAProxy backend.

You should add inside the backend section something like this:

http-request set-path %[path,regsub(^/netdata/,/)]

Although the application has to be written to work with relative paths (i.e. ./whatever.css instead of /whatever.css), else it won’t work properly.

(HAProxy doesn’t have support for rewriting paths inside the response body.)

I end up getting:

File does not exist, or is not accessible: /usr/share/netdata/web/netdata

Where it should be trying to access /usr/share/netdata/web. Is there any way to modify that path?

HAProxy does not actually serve anything from the filesystem, therefore that error is from the downstream backend server. (Check the HTTP response headers to see which server generates this error message.)

Yes, I’m getting the message from the downstream server.

The response headers are:

HTTP/1.1 404 Not Found
server: NetData Embedded HTTP Server v1.16.0-61-nightly
access-control-allow-origin: *
access-control-allow-credentials: true
content-type: text/html; charset=utf-8
date: Thu, 18 Jul 2019 20:56:02 GMT
cache-control: no-cache, no-store, must-revalidate
pragma: no-cache
expires: Thu, 18 Jul 2019 20:56:03 GMT
content-encoding: gzip
transfer-encoding: chunked

Request headers:

GET /netdata HTTP/1.1
Host: REDACTED
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Authorization: Basic REDACTED
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
DNT: 1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: ajs_group_id=null; ajs_user_id=%225nhkts7umffupk98j7yhn8t36o%22; ajs_anonymous_id=%2200000000000000000000000000%22; SRVNAME=REDACTED

Please note that the regsub pattern I gave you above doesn’t also replace /netdata, but requires /netdata/, you should try ^/netdata(/|$) as regular expression.

1 Like

Brilliant! That did it, thank you so much!

So it works when I add the trailing slash but when I don’t add the trailing slash, I got it to work, as in to send me to the right page, but styling is still broken.

I had to modify your example because haproxy doesn’t support parentheses in regular expressions, so I used the following:

acl nd_path_slash path_end /

http-request set-path %[path,regsub(^/netdata/,/)] if nd_path_slash
http-request set-path %[path,regsub(^/netdata,/)] if ! nd_path_slash

Not sure if you have a more ideal solution.

Here is the response I get with styling removed when I visit the URL without the trailing slash. For some reason it’s removing the subdomain which is breaking CSS and JavaScript:

General

Request URL: https://example.com/main.css?v=5
Request Method: GET
Status Code: 404 Not Found
Remote Address: REDACTED:443
Referrer Policy: no-referrer-when-downgrade

Request Headers

Provisional headers are shown
DNT: 1
Referer: https://subdomain.example.com/netdata

With the trailing slash these are the request headers:

GET /netdata/main.css?v=5 HTTP/1.1
Host: subdomain.example.com
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Authorization: Basic REDACTED
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
DNT: 1
Accept: text/css,*/*;q=0.1
Referer: https://subdomain.example.com/netdata/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: ajs_group_id=null; ajs_user_id=%225nhkts7umffupk98j7yhn8t36o%22; ajs_anonymous_id=%2200000000000000000000000000%22; SRVNAME=REDACTED

So only for styling it removes the subdomain.

It is really hard for me to make a mental map of what works and what doesn’t. Cloud you provide a simple “list” like:

  • when the URL browser is /netdata/, and HAProxy rewrites it as /, it works;
  • when the URL browser is /netdata, and HAProxy rewrites it as /, it doesn’t work;
  • etc.

Indeed, reviewing the documentation it seems that closing ) doesn’t work.

I would approach this a different way:

  • redirect if the URL is exactly /netdata;
  • do the replace of /netdata/ with /; it keeps things simpler;

As expected, and as hinted in my previous replies, just by rewriting the request path won’t magically make your web application work if it was not designed to support this method of serving.

The reason is how the browser works when it requests relative paths. If your URL in the browser is /netdata, and the page embeds an ./main.css (or just main.css), then the browser would ask for /main.css (as it is relative to the “folder” of the /netdata which is /); if however your browser URL is /netdata/, then that relative URL becomes /netdata/main.css. However if your webserver returns in its body an URL like /main.css, i.e. an absolute one, then the browser will just ignore the current URL, and always ask for /main.css.

Therefore I would strongly advise to double check the documentation and see if they support this way of working, or alternatively just use an entire subdomain dedicated to the monitoring tool, and don’t do any path mangling.

I’ve asked about whether or not the application will work with relative paths.

Regarding what works and does not work.

  • /netdata/ works with my code snippet above
  • /netdata/OTHER_ARGS works with my code snippet above
  • /netdata sends me to the application, but the styling is missing.

If need be I will use a subdomain, but I would really prefer not to since all of my other infrastructure uses a path.

Then if /netdata/ works, use the redirect to /netdata/ if it is only /netdata, and use that single regular expression to replace it with /.


I would strongly suggest to use a different subdomain, dedicated to each “internal” service, not only it simplifies things, but especially since the entire web security model (as implemented in the browsers) is based on the concept of origin, which includes only the schema, domain, and port.

(For example if your applications use cookies for authentication, and they don’t mention a path when they set the cookie (which is by default in many cases), then if someone compromises say one of your least important services running on /whatever, then they could also steal your cookies for the service running on /important.)

(Not to mention that a compromised application running on /whatever can easily initiate AJAX requests to your /important directly from the browser.)

Thank you, I’ll definitely keep that in mind and consider using a subdomain. For now at least I’ve got this working using in the frontend:

acl is_netdata       url_beg  /netdata
http-request redirect scheme https drop-query append-slash if is_netdata ! { path_beg /netdata/ }

And then just replacing using the regex:

http-request set-path %[path,regsub(^/netdata/,/)]