When the ejabberd backend goes down


#1

I have rather typical setup:

1 frontend + 1 backend, tcp mode, in backend I’ve several ejabberd servers.

Ejabberd backends are stateless and when some of them go down, I’d like client not to know about it and to be redispatched to another ejabberd backend.

Unfortunately it seems that when backend goes down, it send FIN packet to haproxy. This is great, since it allows to remove connection from haproxy to unavailable backend. But then haproxy send FIN packet to client, what makes client to reconnect.

When we use netfilter on haproxy and block sending FIN packet to client, everything seems to be OK: client is unconscious of situation, and continues its session on another ejabberd backend - redispatched by haproxy. Since - as I wrote above - ejabberd cluster is stateless, from application point of view, nothing wrong happen.

Therefore: is there any way to block on haproxy that session termination between client and haproxy? That iptables-way doesn’t look like production approach for me, and we’re looking for more proper way.

Stay cool!


#2

No, this cannot be done. Depending on the protocol this could be very dangerous, and I have my doubts about your backend being 100% stateless.

HTTP mode can redispatched, TCP mode should not. You probably hitting a bug in haproxy that permits TCP mode connections to be redispatched.


#3

Thanks for your reply.
Tell me please, what is wrong about dispatching connections to backends in tcp mode, if application is stateless. Does it make difference, tcp, or http mode? All is about application layer, am I wrong?
When we test it, it seems, that jabberd/xmpp were cool about that situation. From their perspective, nothing weird had taken place.


#4

An application always maintains state for a specific TCP sessions, simply because the TCP sessions is supposed to break when something happens, restarting the state machine on both end-points. By redispatching TCP sessions you are breaking this assumption.

I don’t know this protocol, but I assume there is some kind of authentication or at least identification (whats the username of this client?) happening at the beginning of the TCP session. How would the server know which client is connected, if you arbitrarily connect existing TCP sessions to different servers?

HTTP is different: a HTTP transaction (request / response) can be served from any TCP sessions, and must not depend on a particular TCP session, thats why HTTP keep-alive and pooling is possible in the first place. Only a few applications don’t comply with this (the HTTP standard), like NTLM authentication.