You can’t.
Haproxy does not know about SMTP or IMAP, and therefor cannot make load-balancing decision based on that. Also, the destination email address is unknown when the connection establishes first (and when the first packet is send) - which is when a load-balancing decision has to take place (unless the proxy itself is a full-blown SMTP/IMAP speaker).
So this really requires a software that is actually able to dialog in SMTP/IMAP with the client, which haproxy isn’t.