Filter and load balance certain traffic?

Is it possible to only load balance certain traffic?

i.e.

IF traffic is from X subnet and is port Y then loadbalance
ELSE passthrough transparently