HAProxy Download/Upload Rates issue

Help!
1

Hey All,

firstly i like to say that I am quite new to haproxying and would like to display what i have set up so you guys know what my infrastructure looks like.

My overall system looks like the following and is setup to function in 1Gbit full duplex (no jumbo frames within the network, MTU 1500 MSS 1460)

WAN -- PFSENSE (DNS Resolver and HAProxy) -- SWITCH __ Webserver 1 -- Many Services on different Ports
                                                    \__Webserver 2 -- Many Services on differnt Ports
                                                     \__ Pc and Wifi things

Or in Ip Related Terms

W.A.N.IP -- PFSENSE (192.168.0.1) -- SWITCH __ Webserver 1 (192.168.0.19) -- Many Services on different Ports
                                             \__Webserver 2 (192.168.0.21) -- Many Services on differnt Ports
                                              \__ PC (192.168.0.172)

On my Webserver 1 under port 2020 i have setup a html5 speedtest namely (GitHub - librespeed/speedtest: Self-hosted Speed Test for HTML5 and more. Easy setup, examples, configurable, mobile friendly. Supports PHP, Node, Multiple servers, and more) which works fairly nicely, i.e., if i visti 192.168.0.19:2020 i will be greeted with my speed test interface, and if i execute it i, i’ll get upload and download rates close to the maximum of 1Gbit, its in the lan so that great! When i visit my W.A.N.IP:2021 which is forwarded to 192.168.0.19:2020 the same applies.

However, now the HAproxy part comes in, I also have a domain with speedtest.example.wtf and a vailid ssl certificate which is offloaded with HAproxy so when I visit https://speedtest.example.wtf i’ll be greated with my speedtest interface. However when i execute the speedtest i’ll get download rates of close and constant to 500Mbit (webservers uploadspeed) and Upload rates of 1000Mbit (servers downloadspeed)

Now to my questions:
It is weird to me that my download rates are limited as soon as the HAproxy is in between, are there some stupid things i forgot to setup, does ssl offloading influence this at all?

My PFSENSE:

Intel(R) Core(TM) i5-5250U CPU @ 1.60GHz Current: 1400 MHz, Max: 1601 MHz 4 Gig Ram 4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads State table size 0.4% (1640/396000) MBUF Usage 4% (10386/246072)

here my Ha.cfg

# Automaticaly generated, dont edit manually.
# Generated on: 2017-11-30 17:34
global
	maxconn			10000
	stats socket /tmp/haproxy.socket level admin
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state
	tune.ssl.maxrecord 1370
	ssl-default-bind-options no-sslv3 no-tls-tickets
	ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend frontend1_http
	bind			192.168.0.15:80 name 192.168.0.15:80   
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			acl-https	hdr_beg(host) -i chat
	acl			acl-https	hdr_beg(host) -i ombi
	acl			acl-https	hdr_beg(host) -i mb
	acl			acl-https	hdr_beg(host) -i rss
	acl			acl-https	hdr_beg(host) -i latex
	acl			acl-https	hdr_beg(host) -i blog
	acl			acl-https	hdr_beg(host) -i plex
	acl			acl-https	hdr_beg(host) -i jd
	acl			acl-https	hdr_beg(host) -i home
	acl			acl-https	hdr_beg(host) -i fab
	acl			acl-https	hdr_beg(host) -i pf
	acl			acl-https	hdr_beg(host) -i hydra
	http-request redirect scheme https  if  acl-https 

frontend frontend2_sni
	bind			192.168.0.15:443 name 192.168.0.15:443   
	mode			tcp
	log			global
	timeout client		30000
	tcp-request inspect-delay	5s
	acl			cloud_sni	req.ssl_sni -m beg -i cloud
	tcp-request content accept if { req.ssl_hello_type 1 }

	use_backend backend_cloud_https_ipvANY  if  cloud_sni 
	default_backend backend_offloading_https_ipvANY

frontend frontend3_offloading
	bind			127.0.0.1:1443 name 127.0.0.1:1443 ssl  crt /var/etc/haproxy/frontend3_offloading.pem  
	bind /tmp/haproxy_chroot/frontend3_offloading.socket name unixsocket uid 80 accept-proxy ssl  crt /var/etc/haproxy/frontend3_offloading.pem 
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	#
	rspidel ^Server:.*$
	rspidel ^X-Powered-By:.*$
	rspidel ^X-AspNet-Version:.*$
	http-response set-header Strict-Transport-Security max-age=31536000;\ includeSubdomains;\ preload
	#http-response set-header X-Frame-Options SAMEDOMAIN
	http-response set-header X-Content-Type-Options nosniff	
	acl			chatACL	hdr_beg(host) -i chat
	acl			ombiACL	hdr_beg(host) -i ombi
	acl			latexACL	hdr_beg(host) -i latex
	acl			rssACL	hdr_beg(host) -i rss
	acl			mbACL	hdr_beg(host) -i mb
	acl			embyACL	hdr_beg(host) -i emby
	acl			plexACL	hdr_beg(host) -i plex
	acl			jdACL	hdr_beg(host) -i jd
	acl			homeACL	hdr_beg(host) -i home
	acl			fabACL	hdr_beg(host) -i fab
	acl			pfACL	hdr_beg(host) -i pf
	acl			hydraACL	hdr_beg(host) -i hydra
	acl			aclcrt_frontend3_offloading	hdr_reg(host) -i ^([^\.]*)\.example\.de(:([0-9]){1,5})?$
	acl			aclcrt_frontend3_offloading	hdr_reg(host) -i ^example\.de(:([0-9]){1,5})?$
	use_backend backend-chat_http_ipv4  if  chatACL aclcrt_frontend3_offloading
	use_backend backend-ombi_http_ipv4  if  ombiACL aclcrt_frontend3_offloading
	use_backend backend-latex_http_ipv4  if  latexACL aclcrt_frontend3_offloading
	use_backend backend-rss_http_ipv4  if  rssACL aclcrt_frontend3_offloading
	use_backend backend-emby_http_ipv4  if  embyACL aclcrt_frontend3_offloading
	use_backend backend-plex_http_ipv4  if  plexACL aclcrt_frontend3_offloading
	use_backend backend-jd_http_ipv4  if  jdACL aclcrt_frontend3_offloading
	use_backend backend-emby_http_ipv4  if  mbACL aclcrt_frontend3_offloading
	use_backend backend-home_http_ipv4  if  homeACL aclcrt_frontend3_offloading
	use_backend backend-fabian-blog_http_ipv4  if  fabACL aclcrt_frontend3_offloading
	use_backend backend-pf_http_ipv4  if  pfACL aclcrt_frontend3_offloading
	use_backend backend-hydra_http_ipv4  if  hydraACL aclcrt_frontend3_offloading
	use_backend backend-kai-blog_http_ipv4  if   aclcrt_frontend3_offloading

backend backend_cloud_https_ipvANY
	mode			tcp
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			cloud 192.168.0.21:2443 check-ssl  verify none 

backend backend_offloading_https_ipvANY
	mode			tcp
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			backend-to-frontend /frontend3_offloading.socket send-proxy-v2-ssl-cn check inter 1000  

backend backend-chat_http_ipv4
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			chat 192.168.0.21:3000 check inter 1000  

backend backend-ombi_http_ipv4
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			ombi 192.168.0.19:3579 check inter 1000  

backend backend-latex_http_ipv4
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			latex 192.168.0.21:5000 check inter 1000  

backend backend-rss_http_ipv4
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			rss 192.168.0.21:2280 check inter 1000  

backend backend-emby_http_ipv4
	mode			http
	log			global
	rspadd Strict-Transport-Security:\ max-age=31536000;
	errorfile			503 /var/etc/haproxy/errorfile_backend-emby_http_ipv4_503_customerror
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	# add some security related headers
	# rspadd Content-Security-Policy:\ default-src\ https:\ data:\ 'unsafe-inline'\ 'unsafe-eval'
	rspadd X-Frame-Options:\ SAMEORIGIN
	rspadd X-Content-Type-Options:\ nosniff
	rspadd X-Xss-Protection:\ 1;\ mode=block
	server			emby_warden 192.168.0.102:8096 check inter 1000  

backend backend-plex_http_ipv4
	mode			http
	log			global
	rspadd Strict-Transport-Security:\ max-age=31536000;
	timeout connect		30000
	timeout server		30000
	retries			3
	server			emby_warden 192.168.0.102:8096 check inter 1000  
	server			emby_wardenssl 192.168.0.102:8920 ssl check inter 1000  verify none 

backend backend-jd_http_ipv4
	mode			http
	log			global
	rspadd Strict-Transport-Security:\ max-age=31536000;
	errorfile			503 /var/etc/haproxy/errorfile_backend-jd_http_ipv4_503_customerror
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	server			jd 192.168.0.19:2020 check inter 1000  

backend backend-home_http_ipv4
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			homepi 192.168.0.191:8123 check inter 1000  

backend backend-fabian-blog_http_ipv4
	mode			http
	log			global
	rspadd Strict-Transport-Security:\ max-age=31536000;
	timeout connect		30000
	timeout server		30000
	retries			3
	server			chat 192.168.0.21:2370 check inter 1000  

backend backend-pf_http_ipv4
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			pf-rancherqnap 192.168.0.19:9001 check inter 1000  

backend backend-hydra_http_ipv4
	mode			http
	log			global
	rspadd Strict-Transport-Security:\ max-age=31536000;
	timeout connect		30000
	timeout server		30000
	retries			3
	server			hydra 192.168.0.19:6050 check inter 1000  

backend backend-kai-blog_http_ipv4
	mode			http
	log			global
	rspadd Strict-Transport-Security:\ max-age=31536000;
	timeout connect		30000
	timeout server		30000
	retries			3
	server			chat 192.168.0.21:2369 check inter 1000

Thanks Guys! Cheers

2

Well haproxy does not artificially limit the upload speed of course.

When you visit 192.168.0.19:2020 from your PC, you only go through your switch, and pfsense is not involved at all. When you go through haproxy, your entire traffic flows in both direction through the 1x Gbit/s link of the pfsense firewall. Maybe the ACK’s don’t get through in time, maybe pfsense is unable to handle the bidirectional traffic or maybe the stack is optimized for actual ip forwarding (the primary use case for a firewall) as opposed to host TCP performance. SSL can be a factor, for both pfsense and your client (since you don’t use HTTPS when testing locally).

Try just relaying traffic on another TCP port, without using SSL and see if that changes anything. Maybe it is your client that is unable to encrypt 1Gbit/s of traffic.

3

Thank you so much for your reply, as i stated i am quite new to all of this:) What i have tried is just connect to my site thought my wanip:2020 to just guide some traffic through the pfsense and rule out some limitations and here i also reach the 1gbit up/down.

So, a follow up questions, is there a difference between, in terms of speed, if i offload ssl on the haproxy machine or terminate it directly on the machine that runs the service?

how can i check on the ACK thingy :smiley:

4

Yes, and I explained the differences in the previous post. What I suggest you do is to route the traffic through haproxy without SSL. So you can see if SSL is the problem or not.

5

Awesome, I’ll try that, and report back

6

Hi there,

what i have tried now is:

                            d gbit  u gbit
http://internalip:2020    -> 1     1             pc -> switch -> server
http://wanip:2020         -> 1     1             pc -> pfsense -> server
http://speed.example.com  -> 0.5   1             pc -> pfsense -> haproxy -> server
https://speed.example.com -> 0.5   1             pc -> pfsense -> haproxy ssl -> server

From an outside network

                            gbit gbit
http://wanip:2020         -> 1     1             work pc (outside network) -> pfsense -> server
http://speed.example.com  -> 0.5   1             work pc -> pfsense -> haproxy -> server
https://speed.example.com -> 0.5   1             work pc -> pfsense -> haproxy ssl -> server

I was hoping it was the ssl part of it, but now i am kind of confused.

7

In this configuration, due to the SNI routing you are forwarding the traffic through haproxy twice. When you tested with port 80, did you have a single frontend -> backend? Or a reiteration like you have with SSL (frontend -> “backend-to-frontend” --> unix socket --> frontend --> backend)?

Anyway looks like raw host tcp performance is limiting you here. Check system and userspace cpuload while testing the upload. But this likely depends on pfsense/freebsd kernel itself, other than haproxy.

8

Hey Lukas,

when i tested this on port 80, I used a frontend -> backend scenario

here are the relevant modifications on

frontend frontend1_http
	bind			192.168.0.15:80 name 192.168.0.15:80   
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			acl-https	hdr_beg(host) -i rss
	http-request redirect scheme https  if  acl-https 
	use_backend backend-jd_http_ipv4  if  acl-http 

backend backend-jd_http_ipv4
	mode			http
	log			global
	rspadd Strict-Transport-Security:\ max-age=31536000;
	errorfile			503 /var/etc/haproxy/errorfile_backend-jd_http_ipv4_503_customerror
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	server			jd 192.168.0.19:2020 check inter 1000

In terms of CPU and Such on the PFSense, my cpu barley goes above 25% usage

Untitled
Untitled.png2046×467 51.8 KB
![Untitled2|690x185]

not sure if any of the information i provided will help you to help me :smiley:

9

You have 4 threads, 25% may mean a fully saturated CPU. I’m not entirely sure about the meaning of WCPU on freebsd though.

Just a few things I would suggest:

  • show the top output when running with -C (top -C) while the upload occurs
  • try what happens when you remove “source ipv4@ usesrc clientip” from the configuration
  • try with the “nokqueue” global configuration parameter (this usually degrades performance, if it improves performance for you, that may be an indication for bugs)
  • provide the pfsense and freebsd kernel release
  • provide the output of haproxy -vv

Thanks

10

Hey Lukas,

first of all thanks so much for all the help, if I can buy you a beer!

  • top -C
    image
    image1085×1069 42.9 KB
  • ipv4 client ip parameter remove - same as before
  • nokqueue (besides that it take 40% cpu now no difference)
  • haproxy -vv
    haproxy -vv
HA-Proxy version 1.8.0 2017/11/26
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = freebsd
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-null-dereference -Wno-unused-label -DFREEBSD_PORTS
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.40 2017-01-11
Running on PCRE version : 8.40 2017-01-11
PCRE library supports JIT : yes
Built with multi-threading support.
Encrypted password support via crypt(3): yes
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with Lua version : Lua 5.3.4
Built with OpenSSL version : OpenSSL 1.0.2m-freebsd  2 Nov 2017
Running on OpenSSL version : OpenSSL 1.0.2m-freebsd  2 Nov 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2

Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

Available filters :
        [TRACE] trace
        [COMP] compression
        [SPOE] spoe
  • freebsd Kernel and pfsense kerbel
    `

FreeBSD 11.1-RELEASE-p4 FreeBSD 11.1-RELEASE-p4 #5 r313908+79c92265a31(RELENG_2_4): Mon Nov 20 08:18:22 CST 2017 root@buildbot2.netgate.com:/builder/ce-242/tmp/obj/builder/ce-242/tmp/FreeBSD-src/sys/pfSense amd64

`

11

I tried an nth option as well.

From an outside network

                                 gbit gbit
http://wanip:2020              -> 1     1             work pc (outside network) -> pfsense -> server
http://speed.example.com:2020  -> 1     1             work pc -> pfsense -> server  
http://speed.example.com:80    -> 0.5   1             work pc -> pfsense -> haproxy -> server

From my poor understanding is that as soon as i navigate the traffic throught the HAproxy is that the upload speed is somehow directly reduced to 50% but download is

12

This has almost certainly nothing to do with haproxy itself. I suggest you take a look at pfsense itself, especially QoS settings and pf configuration.

Not sure what else could be done.

13

Hi,

I dont use any traffic shaping atm, however that was my first guess as well. I am really puzzled :frowning:

maybe it has something to do with my DNS resolver? its the only config part i really touched.

Thank you again for your help. !