Slow throughput with HAproxy and SSL offload

I setup HA proxy for SSL offload on a Pfsense VM and it is working great but I have noticed that throughput seems slower then it should be for my internet connection for file transfers. So I setup a speed test server and did a test with SSL offload enabled and without. The speed loss with SSL offload is 78Mbs down and 177Mbs up.

With SSL offload enabled I’m getting 22/23Mbs down/up
Without SSL offload I’m getting 100/200Mbs down/up

Anybody Experience this issue before? Any ideas on what could cause it or how to troubleshoot it?

My install is a Pfsense VM on Proxmox.
Pfsense 2.7.2-RELEASE
haproxy-devel net 0.63_2
Made sure all these settings were disabled.

Hardware Checksum Offloading
Hardware TCP Segmentation Offloading
Hardware Large Receive Offloading