Reverse http/ws from public ip to lan

Help!
1

Hi I m a junior engineer trying to get better with haproxy, so far I had a good experience until this :grimacing:
I’m trying to setup a haproxy as reverse proxy in our dmz, to route http/s and wss to our webappplications. We would like to get rid of our nginx reverse proxy as nobody knows it well, thats why I have a working configuration for nginx to do the reversing. I figured it would be easiest to try to translate this nginx config as best as I can to a haproxy conf, but the best I’ve got out of this is a 502 error… my guess so far is, that it has something to do with the change of URL from external to internal or the webapplication (but it works with nginx though :man_shrugging:) …

The setup:
public IP <-> demo.crm.example.com
internal IP <-> demo.crm.example.local

haproxy listens on 443 and should do tls termination for demo.crm.example.com and proxy the traffic to the webapplication which does its own tls termination for demo.crm.example.lan . As much as I understand this webapplication, it builds up a login page on http traffic and then as you logged in, starts a wss tunnel.

Firewall rules and networking in general works, I m able to reverse simple http to a webserver. Maybe I m getting something wrong her or it has something to do with SSL or maybe it is just this webapplication which is playing stupid… I don’t have any clue what I m doing wrong her.

for comparison this is the nginx config:

server {  
  limit_req zone=limitbyaddr burst=100 nodelay;
  listen 0.0.0.0:443 ssl;
  server_name demo.crm.example.com;
  gzip on;
  access_log /tmp/cxw/logs/demo.crm.example.com_access.log;
  error_log /tmp/cxw/logs/demo.crm.example.com_error.log;
  ssl_certificate /etc/letsencrypt/live/demo.crm.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/demo.crm.example.com/privkey.pem;
  include /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
  underscores_in_headers on;  
  
  location / {
      proxy_ssl_server_name on;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Original-URI $request_uri;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Port 443;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_ssl_protocols TLSv1.1 TLSv1.2;
      proxy_pass https://demo.crm.example.local/;
  }
 
  location /nrpc-wss {
     proxy_ssl_server_name on;
     proxy_pass https://demo.crm.example.local/nrpc-wss;
     proxy_http_version 1.1;
     proxy_set_header Upgrade $http_upgrade;
     proxy_set_header Connection $connection_upgrade;
  }
  
  location /.well-known/acme-challenge/ {
     root /var/www/certbot;
  }

}

and here the haproxy conf:

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log /dev/log local0
    log /dev/log local1 notice
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode http
    log global
    option httplog
    option  http-server-close
    option  dontlognull
    option  redispatch
    option  contstats
    retries 3
    backlog 10000
    option forwardfor
    timeout http-request    15s
    timeout queue           30s
    timeout connect         5s
    timeout client          25s
    timeout server          25s
    timeout check           10s
    timeout tunnel          43200s
    timeout tarpit          60s
    timeout http-keep-alive  1s

#---------------------------------------------------------------------
# haproxy stats
#---------------------------------------------------------------------
frontend stats
    bind *:9000
    mode http
    log global
    maxconn 10
    stats enable
    stats hide-version
    stats refresh 30s
    stats show-node
    stats show-desc Stats for CXW Reverse Proxy
    stats uri /stats

#-------------------------------------------------------
# Lets Encrypt Backend ---------------------------------
#-------------------------------------------------------

backend letsencrypt-backend
    server letsencrypt 127.0.0.1:7080

#-------------------------------------------------------
# prd-k8s ----------------------------------------------
#-------------------------------------------------------
frontend demo-fe
    mode http
    bind *:443 ssl crt /etc/ssl/demo.crm.example.com/demo.crm.example.com.pem
    default_backend demo.crm
    acl letsencrypt-acl path_beg /.well-known/acme-challenge/
    use_backend letsencrypt-backend if letsencrypt-acl
    acl is_websocket hdr(Upgrade) -i WebSocket
    acl is_websocket hdr_beg(Host) -i ws
    use_backend demo.crm-nrpc-wss if is_websocket

backend demo.crm
    mode http
    balance roundrobin
    http-request set-path /
    http-request set-header Host demo.crm.example.local
    http-request set-header X-Client-IP %[src]
    http-request set-header X-Original-URI https://demo.crm.example.local/
    http-request set-header X-Forwarded-Port 443
    http-request set-header X-Forwarded-Proto https
    server demo.crm.example.local demo.crm.example.local:443 check sni req.hdr(Host)

backend demo.crm-nrpc-wss
    mode http
    balance roundrobin
    http-request set-path /nrpc-wss
    http-request set-header Host demo.crm.example.local
    server demo.crm.example.local demo.crm.example.local:443 check sni req.hdr(Host)

Let me know if I can provide further informations that could help! :grinning:

2

Maybe I 've a lot of unnecessary things in this conf, I don’t know I tried a lot and read many blogs and discussions but nothing helped so far.

3

Greetins @jmbizkit!

If I recall correctly, no special configuration is required for Web Sockets.

Knowing this and wanting to keep things simple, I would try commenting out all the acl and use_backend lines in your frontend (effectively sending everything to the default_backend), and see what happens.

If that’s a bit too drastic, I would definitely comment out http-request set-path /nrpc-wss in the backend. That sets the path for everything going to the server behind HAProxy, but this does not set the path with the client (usually a browser). More often than not, this causes problems. If this was intended to be a redirect, I would use redirect location /nrpc-wss instead.

Edit: Initial response was incomplete.

1 Like
4

Thanks @stormrover for the fast reply. I commented all the acl and use_backend out. It is still not working, but I will use the conf like that for further troubleshooting, as it looks like, traffic is passing through haproxy and going to the webapplication.
Maybe it has to do with the headers, because there is a Istio-Mesh (forgot to mention this) in front of the webapplication. Which should route traffic to the right service based on SNI.

5

Did you take out the http-request set-path /nrpc-wss in the backend also? I’ve seen setting request paths in the backend like this break stuff.

6

The only way I was able to make it work, is to let it through in tcp mode, but as it looks like WSS isn’t working. As I only reach the login page and after login where WSS tunnel opens, it doesn’t happen :frowning:

TCP mode isn’t really what I need, as I need to have TLS termination for the external URL (don’t know an other way to make that happen) and it is just weird that WSS isn’t even working. We are running a similar application behind HAProxy as internal LoadBalancer in TCP mode on which WSS works perfect…

This is the fe and be config:

frontend demo-fe
    mode tcp
    bind *:443
    default_backend demo.crm

backend demo.crm
    mode tcp
    balance roundrobin
    server demo.crm.example.local demo.crm.example.local:443
7

Does WSS work if you connect to demo.crm.example.local directly? Neither HTTP mode nor TCP mode should interfere with WSS.

8

Yep, if I connect to demo.crm.example.local everything works and if I put it behind that mentioned HAProxy Load Balancer, it works as well. The Load Balancer is also in TCP and the config is pretty much exactly the same.
The main differences between the Reverse Proxy (DMZ) and the Load Balancer (Server LAN), is the network they are in and the fact that the Reverse Proxy has NAT to make it external available. Also the HAProxy versions are different: RP version 2.4.18 and LB version 2.0.29

9

Could you share some logs from HAProxy? I’m specifically interested to see the session state at disconnection. If HAProxy has a problem with any request or response, it will show letter codes, for example CD-- which means the client unexpectedly aborted during data transfer. Generally, if it only shows four dashes like ----, then HAProxy is fulfilling the request as the backend server instructs it to.

10

I did two tests, one in TCP and the other in HTTP Mode, I’ve censored the requester ip address…

This is the fe and be in TCP mode and the logs:

frontend demo-fe
    mode tcp
    bind *:443
    default_backend demo.crm

backend demo.crm
    mode tcp
    balance roundrobin
    server demo.crm.example.local demo.crm.example.local:443

Feb  8 08:25:36 hap-reverse01 haproxy[36324]: 1......:54504 [08/Feb/2023:08:25:31.006] demo-fe demo.crm/demo.crm.example.local 1/0/5317 6648 -- 6/5/4/4/0 0/0
Feb  8 08:25:36 hap-reverse01 haproxy[36324]: 1......:54505 [08/Feb/2023:08:25:31.321] demo-fe demo.crm/demo.crm.example.local 1/0/5028 3816 -- 5/4/3/3/0 0/0
Feb  8 08:25:36 hap-reverse01 haproxy[36324]: 1......:54506 [08/Feb/2023:08:25:31.321] demo-fe demo.crm/demo.crm.example.local 1/0/5030 3818 -- 4/3/2/2/0 0/0

and in HTTP Mode:

frontend demo-fe
    mode http
    bind *:443 ssl crt /etc/ssl/demo.crm.example.com/demo.crm.example.com.pem
    default_backend demo.crm
    acl letsencrypt-acl path_beg /.well-known/acme-challenge/
    use_backend letsencrypt-backend if letsencrypt-acl

backend demo.crm
    mode http
    balance roundrobin
    server demo.crm.example.local demo.crm.example.local:443

backend letsencrypt-backend
    server letsencrypt 127.0.0.1:7080

Feb  8 13:42:21 hap-reverse01 haproxy[60052]: 1......:54126 [08/Feb/2023:13:42:21.042] demo-fe/1: SSL handshake failure
Feb  8 13:42:21 hap-reverse01 haproxy[60052]: 1......:54127 [08/Feb/2023:13:42:21.052] demo-fe~ demo.crm/demo.crm.example.local 0/0/1/-1/4 502 209 - - SH-- 1/1/0/0/0 0/0 "GET /nomad/ HTTP/1.1"
Feb  8 13:42:21 hap-reverse01 haproxy[60052]: 1......:54128 [08/Feb/2023:13:42:21.929] demo-fe/1: SSL handshake failure
Feb  8 13:42:21 hap-reverse01 haproxy[60052]: 1......:54129 [08/Feb/2023:13:42:21.938] demo-fe~ demo.crm/demo.crm.example.local 0/0/1/-1/3 502 209 - - SH-- 1/1/0/0/0 0/0 "GET /nomad/ HTTP/1.1"

As I understand this SH-- from the documentation, there is an issue with my certificate or can this indicate something else?
I’ve created the cert with certbot, I don’t know if you have any experience with it, but this is the command I used to setup the cert:

sudo certbot certonly --standalone -d demo.crm.example.com --non-interactive --agree-tos --email mailaddress@example.com --http-01-port=7080
11

Okay, I guess I should have specified: You only get two letters in TCP mode because there’s no HTTP inspection going on. In TCP mode, your logs show HAProxy is passing the request through as instructed.

As for the SH-- part:

     SH   The server aborted before sending its full HTTP response headers, or
          it crashed while processing the request. Since a server aborting at
          this moment is very rare, it would be wise to inspect its logs to
          control whether it crashed and why. The logged request may indicate a
          small set of faulty requests, demonstrating bugs in the application.
          Sometimes this might also be caused by an IDS killing the connection
          between HAProxy and the server.

It appears your backend server expects SSL since it’s listening on 443, but you did not specify SSL on the server (and I can’t belive I missed that this whole time). It should probably look like:

backend demo.crm
    mode http
    balance roundrobin
    server demo.crm.example.local demo.crm.example.local:443 check ssl

check: poll the server every few seconds and make sure it’s up. (optional)
ssl: use SSL to connect to this server

I don’t think that will fully resolve your issue, as TCP mode shows HAProxy is handling your request, but elliminating that error might expose another error that gives better info.

1 Like
12

Thanks a lot @stormrover that was it! :grinning:

The other issue I had that producced the handshake to fail:

was an issue with me testing the auto renewal of certbot, the cert that was issued was a staging cert. Solved that as well and now everything is as it should be.

Again thanks a lot!

1 Like