I figured it out. You first have to try checking for passthrough, using SNI. If nothing matches with the SNI info, it goes to a backend that redirects to an alternate frontend bound to a different port with certs, and will then perform SSL termination.
I also have 2 files for mapping domains, to make it easier to add new servers without messing with the config file too much, based on this link: http://blog.haproxy.com/2015/01/26/web-application-name-to-backend-mapping-in-haproxy/
Anyway - here is my config that seems to work right now:
http://pastebin.com/pkRsp9cc