Haproxy 503 with termination set to CD

Following are the observations
Seeing intermittent 503 in haproxy with termination code set to CD
Backend is chosen and set correctly by haproxy
http_total_time, tcp_total_time and server_response_time is high compared to requests served with 200 status code but still less than timeout configured in haproxy configuration file.
client_request_time is 0
Server chosen has active servers.
At the same time where 503 is seen 200 is also served for other requests. Please help/suggest ways to recover from this and fix this