Haproxy ssl termination and upstream forward proxy

ja8zyjits · March 2, 2020, 12:21pm

Problem:

Iam trying to build a forward proxy with ssl termination, further it upstreams to my proxy servers eg: TOR. My upstream proxy services are non-https.

Client → Network-Haproxy → Uptstream-Proxy → Internet

I could easily succeed in tcp mode of HAproxy without ssl termination, but when I terminate ssl and forward, things don’t work.

Steps Followed:

I followed the below steps to generate self-certified ssl certificates.

$ openssl req -new -newkey rsa:2048 -sha256 -days 365 -nodes -x509 -extensions v3_ca -keyout haproxy-ca-key.pem -out haproxy-ca-cert.pem -subj "/C=GB/ST=London/L=London/O=Global Security/OU=IT Department/CN=example.com"

combined them for creating final .pem file

$ cat haproxy-ca-cert.pem haproxy-ca-key.pem >> mysite.pem

The above file is used in my haproxy.cfg for ssl termination.

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256::RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
	ssl-default-bind-options no-sslv3

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

	stats enable
	stats uri /stats
   	stats realm Haproxy\ Statistics
   	stats auth user:password


frontend www.mysite.com
    mode http
    bind 0.0.0.0:8443
    bind 0.0.0.0:443 ssl crt /home/ubuntu/haproxy/mysite.pem crt-ignore-err all
    redirect scheme https if !{ ssl_fc }
    default_backend web_servers

backend web_servers
    mode http
    balance roundrobin
    server server1 xx.xx.xx.xx:xxxx #my upstream server which is not ssl protected

When I try to curl from my client machine to use the above proxy I get following error.

$ curl -k --proxy https://my-haproxy-server:443 --cacert haproxy-ca-cert.pem  https://httpbin.org/ip -vvv
*   Trying my-haproxy-server...
* TCP_NODELAY set
* Connected to my-haproxy-server (my-haproxy-server) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: self signed certificate
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

If you have read the cfg file, you can see I have redirected :8443 to :443, so I can send request to non https proxy, but that too doesn’t work

$ curl -k --proxy http://my-haproxy-server:8443 --cacert haproxy-ca-cert.pem  https://httpbin.org/ip -vvv
*   Trying my-haproxy-server...
* TCP_NODELAY set
* Connected to my-haproxy-server (my-haproxy-server) port 8443 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to httpbin.org:443
> CONNECT httpbin.org:443 HTTP/1.1
> Host: httpbin.org:443
> User-Agent: curl/7.58.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 302 Found
< content-length: 0
< location: https://httpbin.org:443/
< cache-control: no-cache
< connection: close
< 
* Received HTTP code 302 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 302 from proxy after CONNECT

Any lead would be appreciated.

Extra Info:

HAPROXY: HA-Proxy version 2.0.13-1ppa1~bionic 2020/02/15 - https://haproxy.org/
OPENSSL: OpenSSL 1.1.1 11 Sep 2018
CURL: curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
SF post: HAproxy forward proxy with SSL termination - Server Fault

lukastribus · March 3, 2020, 5:48pm

When you instruct curl to use a HTTPS proxy, you need to use the correct curl arguments for the SSL parameters:

It’s --proxy-cacert instead of --cacert:

https://curl.haxx.se/docs/manpage.html#--proxy-cacert

And it’s --proxy-insecure instead of -k or --insecure:

https://curl.haxx.se/docs/manpage.html#--proxy-insecure

ja8zyjits · March 4, 2020, 6:01am

@lukastribus Thank you for your response.

that didn’t work, Iam getting error

$ curl --proxy https://my-haproxy-server:443 --proxy-cacert haproxy-ca-cert.pem https://httpbin.org/ip -vvv
*   Trying my-haproxy-server...
* TCP_NODELAY set
* Connected to my-haproxy-server (my-haproxy-server) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: haproxy-ca-cert.pem
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unsupported certificate purpose
* Closing connection 0
curl: (60) SSL certificate problem: unsupported certificate purpose
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This too dint work

$ curl --proxy https://my-haproxy-server:443 --proxy-cacert haproxy-ca-cert.pem https://httpbin.org/ip -vvv --proxy-insecure
*   Trying my-haproxy-server...
* TCP_NODELAY set
* Connected to my-haproxy-server (my-haproxy-server) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: haproxy-ca-cert.pem
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Proxy certificate:
*  subject: CN=haproxy; O=haproxy
*  start date: Feb 29 11:55:39 2020 GMT
*  expire date: Mar  2 11:55:39 2023 GMT
*  issuer: CN=haproxy; O=haproxy
*  SSL certificate verify result: unsupported certificate purpose (26), continuing anyway.
* allocate connect buffer!
* Establish HTTP proxy tunnel to httpbin.org:443
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> CONNECT httpbin.org:443 HTTP/1.1
> Host: httpbin.org:443
> User-Agent: curl/7.58.0
> Proxy-Connection: Keep-Alive
> 
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/1.0 400 Bad request
< Cache-Control: no-cache
< Content-Type: text/html
< 
* Received HTTP code 400 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
* TLSv1.3 (OUT), TLS Unknown, Unknown (21):
curl: (56) Received HTTP code 400 from proxy after CONNECT

lukastribus · March 4, 2020, 4:55pm

Something is wrong with the certificate, but that’s probably not important (if you can ignore proxy SSL issues with --proxy-insecure).

I think it doesn’t make sense to use mode http here. CONNECT is probably interpreted by haproxy instead of your backend proxy server.

Why do you need HTTP mode? Can’t you just use SSL termination with TCP mode?

ja8zyjits · March 11, 2020, 9:06am

I think termination only works in HTTP mode. In TCP mode it directly forwards the connection right?

lukastribus · March 11, 2020, 9:18am

SSL termination works just fine with TCP mode. Just because you CAN passthrough SSL in tcp mode, doesn’t mean you cannot SSL terminate with TCP mode, and pass only the actual unencrypted payload to the backend.

ja8zyjits · April 27, 2020, 6:28pm

Hey,

 Sorry for late response.

It seems your ideas worked. Need to verify, with another tool like squid, my main purpose. But for now it worked.
So the final configuration in this. Will update once squid starts working, for which i need to fix the ca-certificate issue, since squid does not allow insecure connection.

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256::RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
	ssl-default-bind-options no-sslv3

defaults
	log	global
	mode	tcp
	option	tcplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

	stats enable
	stats uri /stats
   	stats realm Haproxy\ Statistics
   	stats auth user:password


frontend www.mysite.com
    mode tcp
    bind 0.0.0.0:8443
    bind 0.0.0.0:443 ssl crt /the/directory/of/mysite.pem crt-ignore-err all
    redirect scheme https if !{ ssl_fc }
    default_backend web_servers

backend web_servers
    mode tcp 
    balance roundrobin
    server server1 xx.xx.xx.xx:xxxx #my upstream server which is not ssl protected

Running it.

$ sudo haproxy -f haproxy.cfg 
[WARNING] 117/234016 (13634) : config : 'stats' statement ignored for frontend 'www.mysite.com' as it requires HTTP mode.
[WARNING] 117/234016 (13634) : config : 'redirect' rules ignored for frontend 'www.mysite.com' as they require HTTP mode.
[WARNING] 117/234016 (13634) : config : 'stats' statement ignored for backend 'web_servers' as it requires HTTP mode.
[WARNING] 117/234016 (13634) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.

and run it to check

$ curl --proxy-insecure --proxy https://localhost:443 --proxy-cacert haproxy-ca-cert.pem  https://httpbin.org/ip -vvv
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: haproxy-ca-cert.pem
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Proxy certificate:
*  subject: C=GB; ST=London; L=London; O=Global Security; OU=IT Department; CN=example.com
*  start date: Apr 27 17:51:36 2020 GMT
*  expire date: Apr 27 17:51:36 2021 GMT
*  issuer: C=GB; ST=London; L=London; O=Global Security; OU=IT Department; CN=example.com
*  SSL certificate verify ok.
* allocate connect buffer!
* Establish HTTP proxy tunnel to httpbin.org:443
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> CONNECT httpbin.org:443 HTTP/1.1
> Host: httpbin.org:443
> User-Agent: curl/7.58.0
> Proxy-Connection: Keep-Alive
> 
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/1.1 200 Connection established
< X-PROXY-IP: xx.xx.xx.xx
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=httpbin.org
*  start date: Jan 18 00:00:00 2020 GMT
*  expire date: Feb 18 12:00:00 2021 GMT
*  subjectAltName: host "httpbin.org" matched cert's' "httpbin.org"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x55572c686580)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET /ip HTTP/2
> Host: httpbin.org
> User-Agent: curl/7.58.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/2 200 
< date: Mon, 27 Apr 2020 18:13:25 GMT
< content-type: application/json
< content-length: 32
< server: gunicorn/19.9.0
< access-control-allow-origin: *
< access-control-allow-credentials: true
< 
{
  "origin": "xx.xx.xx.xx"
}
* Connection #0 to host localhost left intact

ja8zyjits · April 27, 2020, 8:15pm

I guess I found the issue with the certificate generation, I replaced example.com to localhost and it worked in local system. So now I guess, even that issue is nearly fixed. Will let you know if my configuration of localsystem → squid → haproxy worked or not.

mindeswx · January 12, 2021, 1:45am

@ja8zyjits Have you figured it out? Working on a similar project…

Topic		Replies	Views
Apache Creating Non HTTPS Link When Using SSL Termination Help!	2	1708	September 14, 2018
SSL termination does not work correctly (v2.0.8) Help!	3	1692	November 13, 2019
Certificate/SSL termination problem Help!	3	1798	June 3, 2016
Is this possible - HAProxy 2 Way SSL - Cert Auth Reverse Proxy Help!	4	5454	January 10, 2019
SSL Termination Help!	8	2667	October 4, 2017

Haproxy ssl termination and upstream forward proxy

Problem:

Steps Followed:

Extra Info:

Related topics