High latency after backend reconnect (Minecraft)

I use haproxy as edge tcp proxy in front of a Minecraft server.

The minecraft server has to restart daily. When that happens, some users complaint that the latency shoots up to 1000% until a couple of reconnects where it calms down.

Why is this potentially happening and how can I mitigate?