503 Service Unavailable | The plain HTTP request was sent to HTTPS port

Help!
1

Hello there. I’m in need of a reverse proxy, using only HTTPS.

I have my VM-HaProxy on 192.168.10.5
and my VM-Git with a web interface (Gogs), with NGINX listening to 443 with let’s encrypt crt which has been validated.

I’m only using HTTPS, always rewriting http to https.

The thing is, i’m getting a 400 error (The plain HTTP request was sent to HTTPS port) with the backend line :
server vm-git 192.168.10.11:443 weight 1 maxconn 8192
And a 503 Service Unavailable with the backend line :
server vm-git 192.168.10.11:443 weight 1 maxconn 8192 check ssl verify none

I went to a lot of forums searching for a solution, but found none.

My Iptables rules are OK :
#HTTP, HTTPS
iptables -t filter -A OUTPUT -p tcp --dport 80 -j ACCEPT
iptables -t filter -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 443 -j ACCEPT
iptables -t filter -A INPUT -p tcp --dport 443 -j ACCEPT

Debian 8.9 64bits
HA-Proxy version 1.5.8 2014/10/31

haproxy.cfg :

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon

    # Default SSL material locations
    # ca-base /etc/ssl/certs
    # crt-base /etc/ssl/private

    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3

defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
option forwardfor except 127.0.0.0/8
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

############################################

frontend https-in
bind 192.168.10.5:443 ssl crt /etc/haproxy/cert/gogs.pem
acl gogs hdr(host) gogs.imperial-legion.fr
use_backend backendgogs if gogs
default_backend backendgogs

############################################

backend backendgogs
option httpchk
http-request add-header X-Forwarded-Proto https if { ssl_fc }
server vm-git 192.168.10.11:443 weight 1 maxconn 8192
#server vm-git 192.168.10.11:443 weight 1 maxconn 8192 check ssl verify none

2

And this is my nginx file to manage the gogs interface on VM-Git :

NGINX file on VM-Git :

server {
listen 80;
server_name gogs.imperial-legion.fr;
location / {
proxy_pass http://localhost:3000;
}

listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/gogs.imperial-legion.fr/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/gogs.imperial-legion.fr/privkey.pem; # managed by Certbot
    ssl_session_cache shared:le_nginx_SSL:1m; # managed by Certbot
    ssl_session_timeout 1440m; # managed by Certbot

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # managed by Certbot
    ssl_prefer_server_ciphers on; # managed by Certbot

    ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-ECDSA-AES128-SHA ECDHE-ECDSA-AES256-SHA ECDHE-ECDSA-AES128-SHA256 ECDHE-ECDSA-AES256-SHA384 ECDHE-RSA-AES$



if ($scheme != "https") {
    return 301 https://$host$request_uri;
} # managed by Certbot

}

3

You need to decide:

Would like to terminate SSL on nginx or haproxy? Seems you are trying to do it on both, which of course doesn’t work.

4

I’m pretty sure that I don’t understand everything about the reverse-proxy technology.

If I have SSL/TLS on NGINX, is everything secured ? Like, i’m sending logins informations, do it pass plain text through the HaProxy ?

I first heard that If I had SSL/TLS set on my NGINX, HaProxy was needing the ssl cert, so I guess I was wrong ?

5

You have an SSL/TLS session between 2 endpoints, those endpoints can be:
the browser and the nginx backend server (haproxy only sees encrypted traffic)
the browser and haproxy (haproxy sees plaintext, and forwards plaintext HTTP to nginx)

That’s wrong; the opoosite is true: you only configured the server certificate either on nginx (former case above) or haproxy (latter case above).

In the former case you remove all SSL configurations from haproxy and put it into “mode tcp” (as opposed to “mode http”).

In the latter case you configure nginx to only listen on port 80, and connect to the backend via plaintext HTTP traffic.

6

I would like to have this endpoint :
the browser and the nginx backend server (haproxy only sees encrypted traffic)

I think it’s the best.

I removed every ssl configuration line, now I got (indent do now appear, but it’s OK) :

frontend https-in-ssl-relay 192.168.10.5:443
mode tcp
acl gogs hdr(host) my-url.tld
use_backend backendgogs if gogs
############################################

backend backendgogs
mode tcp
server vm-git 192.168.10.11:443 maxconn 32

But when I go to my https url, I get a Connexion secured failed.
What’s wrong ?

7

Change:

frontend https-in-ssl-relay 192.168.10.5:443
to:

frontend port443-relay
 bind 192.168.10.5:443

And make sure you are not binding port 443 somewhere else on this box.

8

I made the changes, but I still get a Connexion secured failed.
There is no other binding port 443, no other haproxy file, the VM is dedicated to that.

In the /var/log/haproxy.log I got :

port443-relay port443-relay/ -1/-1/0 0 SC 0/0/0/0/0 0/0

I tried to reboot, but no changes.
It doesnt seems to see my webserver… hmm… idk why. Is there something to add somewhere?

9

http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#8.5

The server or an equipment between it and haproxy explicitly refused
the TCP connection (the proxy received a TCP RST or an ICMP message
in return). Under some circumstances, it can also be the network
stack telling the proxy that the server is unreachable (eg: no route,
or no ARP response on local network). When this happens in HTTP mode,
the status code is likely a 502 or 503 here.

From the haproxy box, try curl’ing to the webserver:

curl -vk https://192.168.10.11/

10

Seems okay ?

  • Hostname was NOT found in DNS cache
  • Trying 192.168.10.11…
  • Connected to 192.168.10.11 (192.168.10.11) port 443 (#0)
  • successfully set certificate verify locations:
  • CAfile: none
    CApath: /etc/ssl/certs
  • SSLv3, TLS handshake, Client hello (1):
  • SSLv3, TLS handshake, Server hello (2):
  • SSLv3, TLS handshake, CERT (11):
  • SSLv3, TLS handshake, Server key exchange (12):
  • SSLv3, TLS handshake, Server finished (14):
  • SSLv3, TLS handshake, Client key exchange (16):
  • SSLv3, TLS change cipher, Client hello (1):
  • SSLv3, TLS handshake, Finished (20):
  • SSLv3, TLS change cipher, Client hello (1):
  • SSLv3, TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
  • Server certificate:
  •    subject: CN=gogs.imperial-legion.fr

  •    start date: 2017-08-21 17:37:00 GMT

  •    expire date: 2017-11-19 17:37:00 GMT

  •    issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3

  •    SSL certificate verify ok.

GET / HTTP/1.1
User-Agent: curl/7.38.0
Host: 192.168.10.11
Accept: /

< HTTP/1.1 302 Found

  • Server nginx is not blacklisted
    < Server: nginx
    < Date: Thu, 24 Aug 2017 15:52:05 GMT
    < Content-Type: text/html; charset=utf-8
    < Content-Length: 34
    < Connection: keep-alive
    < Location: /user/login
    < Set-Cookie: lang=en-US; Path=/; Max-Age=2147483647
    < Set-Cookie: i_like_gogits=04f99f1d5f76b02c; Path=/; HttpOnly
    < Set-Cookie: _csrf=SLztMZEb4EKG9wU6Q-v9jD7MZcg6MTUwMzU4OTkyNTIyODY1NzI0Mw%3D%3D; Path=/; Expires=Fri, 25 Aug 2017 15:52:05 GMT; HttpOnly
    < Set-Cookie: redirect_to=%252F; Path=/
    <
    Found.

  • Connection #0 to host 192.168.10.11 left intact

Both servers are on the same network, both HTTP and HTTPS allowed in iptables. Is there any other ports I need to allow ?

11

Can you share the complete haproxy configuration?

12

Here it is :

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3

defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
option forwardfor except 127.0.0.0/8
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

############################################

frontend port443-relay
bind 192.168.10.5:443
mode tcp
acl gogs hdr(host) my-url
use_backend backendgogs if gogs
############################################

backend backendgogs
mode tcp
server vm-git 192.168.10.11:443 maxconn 32

13

This doesn’t work in TCP mode. You need a default-backend, without ACL:

default_backend backendgogs
14

YES ! It’s working ! :smile:

First of all, thanks you, very much for you help, but i’m not done here.

It’s working when I use the https url, but not when I type the http one, because there is not http to https redirect obviously. I tried to add theses lines but it’s not correct :

frontend port80-relay
mode tcp
bind 192.168.10.5:80
redirect scheme https if !{ ssl_fc }
default_backend backendgogs

Showing me a “The plain HTTP request was sent to HTTPS port” error.

My second issue is the following :

I’m setting up a reverse proxy, because I have several web interfaces listening on 443 on several machines.

With the setup using only default_backend, how I am supposed to redirect to another machine with another url ? It doesnt seems possible.

15

The redirect doesn’t work in TCP mode, and the backend is HTTPS, so it doesn’t work this way.

This should be the correct configuration (also you don’t need a backend, since you just redirect to https):

frontend port80-redirect
 mode http
 bind 192.168.10.5:80 
 redirect scheme https

Correct, you cannot look at the HTTP headers (inlucding hostname and URI).

What is your requirement exactly - do you need a different backend based on URI, or just hostname?
How do your certificates look like if you have multiple hostnames? Do you have multiple certificates or one certificate with multiple SANs?

16

HTTP to HTTPS working. :wink:

I have two VM with their own web interfaces, listening on 443 with theirs own let’s encrypt certificates.

I want to access first interface with subdom1.dom.tld and access first interface with subdom2.dom.tdl.

I have only two web interfaces for now, but it’s going to grow as my infrastructure is growing as well.

17

Ok, routing via SNI should suffice in this case, here’s an example:

frontend port443-relay
 bind 192.168.10.5:443
 mode tcp
 tcp-request inspect-delay 5s
 tcp-request content accept if { req_ssl_hello_type 1 }
 use_backend backendgogs if { req_ssl_sni -i subdom1.dom.tld }
 use_backend bk_subdom2 if { req_ssl_sni -i subdom2.dom.tld }
 use_backend bk_subdom3 if { req_ssl_sni -i subdom3.dom.tld }
 default_backend bk_default_backend
18

Seems okay !

Do I need to edit the frontend port80-redirect part too ?

It’s redirecting to https, so is it going to the frontend port443-relay right after ?

19

No need, unless there are backends where you do not want to redirect to https (port 443).

port80-redirect unconditionally redirects the browser to https, without looking at the hostname.

20

Well Okay, I just need to try with the second backend, but I think everything will be OK.

I sincerly thanks you, I even… bend the knee ! :smile: