Most likely you have overlapping certificates on your backend servers (wildcard certificate *.domain.com or multiple SAN matching both server1.domain.com and server2.domain.com), this will NOT work correctly.
Use distinct and dedicated, non-overlapping certificate or terminate SSL on haproxy and look at the Host header instead.