Of course everything will slow down if you hit maxconn, that’s the point of queuing. Configure maxconn based on your backend’s capability, but if you tell haproxy “my backend cannot handle more than 128 connections”, than haproxy will respect that and queue the connections so that your backend is not overloaded.
From your description it isn’t clear if with higher maxconn configuration you reach your performance capabilities of your backend or not; however you configuration and keepalive modes are also unkown.
Please provide the full configuration and the full output of haproxy -vv if you still don’t reach the performance you are looking for.