HAProxy HTTP frontend and backend

Help!
1

Hi ,
We have an HAProxy setup running in Production for some time which supports access to Confluent Kafka cloud purpose in TCP for both Kafka brokers , port 9092 and Kafka Admin API , port 443 ,
as explained - both are TCP and that setup works for a while …
following is the snippet from the configuration -

frontend ccloud
mode tcp
bind *:9092
bind *:443
log global

    tcp-request inspect-delay 5s
    tcp-request content accept if { req.ssl_hello_type 1 }

**** there are several backends here additionally at the same format … brokers 1-8

backend broker1-nonprod
mode tcp
server broker1 b1-confluent.cloud:9092 check resolvers mynameservers init-addr last,libc,none
…
…
…
…
backend bootstrap
mode tcp
server bootstrap confluent.cloud:9092 check resolvers mynameservers init-addr last,libc,none

backend admin_api
mode tcp
server admin_api admin_api.confluent.cloud:443 check resolvers mynameservers init-addr last,libc,none

additionally to that we have a frontend for the stats API (again that’s also been running successfully for a while)

frontend stats
mode http
bind *:8404
http-request use-service prometheus-exporter if { path /metrics }
stats enable
stats uri /stats
stats refresh 20s

Now - we wanted to add support to use HAProxy to connect to a different backend in Confluent to support Rest API calls using HTTPS , note that the URL to connect is same URL as the other general bootstrap server URL for Confluent used in the TCP 9092 - for instance in the example here -
URL is - confluent.cloud

The difference is the protocol + port ,
as in the new addition we need to support that URL in port 443 HTTPS

So we added the following -

frontend ccloud_rest
mode http
bind *:443
use_backend rest_proxy_cloud

backend rest_proxy_cloud
server rest_proxy_cloud confluent.cloud:443 check resolvers mynameservers init-addr last,libc,none

Tried also to add several adjustments such as -
mode http - in the backend
or - default_backend rest_proxy_cloud
and some other adjustments …

Issue is that none of the attempt work in routing the request to the backend of the Rest-Proxy in the cloud -

  • not seeing anything in the logs with that frontend / backend (anything that can be added to the configuration to add it to the logs) ?

  • not getting a correct response when invoking a request to that cloud from the client -
    getting response - The underlying connection was closed: The connection was closed unexpectedly.

  • when looking in the stats page - able to detect that there are incoming sessions in the frontend for that new frontend ccloud_rest , however in the backend not getting any sessions , does that say that the sessions are not being routed correctly to the appropriate backend ?

  • on the same stats page , on the frontend - getting that the response is some 4XX HTTP response code , not getting the accurate code , only 4XX
    is there an option to tell the exact code or response returned by that call ?

Can anyone explain the reason why it’s not working or how to debug it based on the configuration that posted here ?

Thanks

2

Hi there,

Could you pls share how did you tested it and what was reply ?
It is hard to guess what is happening but based on provided info I wonder how do you connect to the haproxy ? via http or https ? e.g. your frontend configuration doesn’t specify any certificate so https will not likely work.

3

Hi , yes ,
Tested it using some Azure utility which should invoke the Kafka Confluent Cloud ,
internally there is routing which translates the URL to the HAProxy LB , which then routes it to 1 of the 2 HAProxy servers ,
so the URL that Azure tool is using for Confluent cloud is -
POST - https:// confluent.cloud/kafka/v3/clusters/xxxx/topics/xxxx/records

The Frontend doesn’t specify any certificate , that’s correct , didn’t know that we need to add it , actually it’s a middleware here , and we don’t have any certificate ,

Why we don’t get any informative exception ?
Why the call doesn’t reach the Backend ?

Thanks

4

You can’t have 2 frontends bound to the same port.

5

Hi ,
even if the 2 Frontends are listening in different protocol ?
One is TCP and other is HTTP …
Tried to remove the 443 TCP Frontend and now getting different exception -
The handshake failed due to an unexpected packet format.

does that say that it’s indeed the issue ?
if so - what should we do in order to fix it ? create a new HAProxy instance to run for that purpose ?
and if that’s correct - how can we fix the exception ? is it related to adding a certificate to the trust store - the server certificate ?

Thanks

6

Hi ,
we thought of the idea of using a different port for the HTTP Frontend , let’s say 8443 and then listen on that port for the HTTPS connection ,
and in the Backend use 443 which is needed as the backend server support 443 ,
can that work ?
is the exception we get related to the server certificate that need to be added to the HAProxy trust store ?

Thanks