The health check is the problem. The health check does not send the SNI header, while regular traffic would correctly handle it (disable health checking or revert to layer 4 health check would work around this problem).
Also see:
https://www.mail-archive.com/haproxy@formilux.org/msg22938.html