HAProxy community

Bingbot SNI handling (terminate at TLS layer)

#1

My server solely serves SSL SNI domains and therefore no default fallback for non sni clients configured. Clients which do not support SNI will get a 503 because of that reason.

However, I’m having some problems with bingbot to work with SNI and after talking with their product team they replied the following:

Basically, how it works is that our bots first offer a TLS handshake to the server withoutSNI info. If the web server refuses this at the TLS layer, Bingbot will know to then offer a TLS handshake withSNI info. Returning a 503 will prevent Bingbot from detecting that it needs to send SNI info. Also, if the handshake is terminated at a deeper layer than the TLS layer, Bingbot will also not take proper action on this.

As I understand it in simpler terms, Bingbot comes to a secret club and asks to come in without saying the password. Because Bingbot didn’t say the password, the bouncer lets Bingbot in but shows him an empty room and then kicks him out the backdoor. Bingbot leaves confused and upset. If the bouncer had refused entry, Bingbot would have remembered to say the password and the bouncer would have lead him to the party.

The Product Team can’t offer anything else besides this explanation on properly terminating the HTTPS handshake so that our bots can make the correct handshake request.

Please configure the server so that HTTPS handshake is terminated if the client “hello” doesn’t contain the expected SNI server name. Basically, if there is an error on TLS layer, the TLS handshake should terminate. Returning info through the higher HTTP layer is too late.
If the web server refuses the TLS handshake when the TLS request doesn’t contain SNI info, Bingbot will be able to determine that SNI might be required and do a retry.

So my question is: how can I configure haproxy to refuse at the TLS layer?

My config is as follows:

frontend www-ssl
mode http
bind 0.0.0.0:443 ssl crt /usr/local/etc/haproxy/certs
use_backend %[ssl_fc_sni,lower,map_end(/usr/local/etc/haproxy2/ssl_domains.map)]

Thanks!

#2

Easy, just configure strict-sni on your bind line.

#3

Yep tried that, but bingbot says in that case :

    One or more fetch attempts were performed but they failed.

:frowning:

#4

Then talk about that with their support. strict-sni is doing exactly what they are asking above.

#5

Thanks, will do and let you know.

#6

To be more specific: when strict-sni is set and a invalid or no SNI value is set in the client_hello, haproxy aborts the handshake by sending a fatal-level unrecognized_name(112) alert as per RFC6066 section-3:

If the server understood the ClientHello extension but
does not recognize the server name, the server SHOULD take one of two
actions: either abort the handshake by sending a fatal-level
unrecognized_name(112) alert or continue the handshake.

#7

Yep I confirmed that using wireshark. Haproxy behaves correctly. I did some more tests with bingbot and it’s behavior seems to be really inconsistent. Some SNI domains are working perfectly and others not. While it’s the exact same HAproxy config.

So it has to be something really weird on Bingbot ends and not my config in Haproxy. Therefore I consider this topic closed :slight_smile: