HA Proxy being slow

No, you should see for every single request a log line along with a number of timers that will help finding the issue.

Please see the following post for details on the logging configuration: