I configured HAproxy so it forwards tcp requests to set of 3 backend servers and created health check that is working fine. At a time only one of the backend servers is available as others are slaves and not allowed for writing.
Each of the 3 servers runs nginx proxy to a python app that connects to postgresql. As postgresql does not support multi-master i configured it with master and slaves and automatic failover. In case a node is down, it promotes another node to master.
The health check seems to be working as i see in the logs it always shows 2 backend nodes down and the other up.
If i open the app in web browser, it connects to HAproxy and then forwards to the healthy backend node.
Next i simulate node failure and automatic failover promotes another node as master. The health check detects this fine, it changes the live backend node in HAproxy and sets the other 2 down.
However if i reload the web page in the browser it still connects the old backend node. I can see in it’s logs the request is going to it instead of the healthy node.
If i open new incognito window in the browser it is working fine, request is processed by the healthy node but reload in previous browser window goes to the down node.
Why is it this happening?