Deny access for HAProxy VIP address

strict-sni is what you want.

http://docs.haproxy.org/2.8/configuration.html#5.1-strict-sni