LXD + LXD proxy device + haproxy + nginx results in mix of normal page loads + ERR_EMPTY_RESPONSE & ERR_CONNECTION_RESET errors

Having some sort of sporadic connection issues with LXD proxy device + HAProxy + Nginx. The errors mostly seem to be ERR_EMPTY_RESPONSE but also sometimes are ERR_CONNECTION_RESET.

Those errors come up when I initially access via IP or domain, but after refreshing the ‘welcome to Nginx’ page does display. Sometimes it takes many page reloads to get the Nginx welcome page to come up. Sometimes the two different errors get in some sort of loop, then the Nginx welcome page is displayed. Once the Nginx welcome page displays I can rapidly reload many times and it continues working properly.

If I try the IP & domain again maybe 5 to 10 minutes later, the same errors initially occur again. Also if I close and restart the browser.

At this point I’m only working on port 80 proxying and the HAProxy config I’ve used is below, just the part I added on after the default settings. In that config “[IP address]” is the actual server IP and “subdomain.example.com” is a working domain which is also the hostname.

##########################################
frontend localhost80
    bind *:80
    mode tcp

    # Set acl based on domain name. 
    acl host_[IP address] hdr(host) -i [IP address] subdomain.example.com

    # Set backend for each acl.
    use_backend [IP address]_http if host_[IP address]

backend [IP address]_http
   mode tcp
   server nginx 10.0.0.50:80 check
##########################################

I’ve tested this using both nat=true & nat=false as well as the Ubuntu 20.04 default version 2.0.13 of HAProxy as well as version 2.4 from haproxy.debian.net

The LXC proxy device I am apply via profile which was generally configured with the following:

##########################################
sudo lxc profile device add proxy-80 hostport80 proxy connect="tcp:10.0.0.2:80" listen="tcp:[IP address]:80" nat=true
sudo lxc profile add haproxy proxy-80
##########################################

From inside the host OS, haproxy container & nginx container I tried “curl [IP address]”, “curl subdomain.example.com” and “curl localhost”.

“curl localhost” on the host OS always gets ‘connection refused’. On the haproxy container it always gets ‘empty reply from server’. And on the nginx container it always loads the nginx welcome page.

“curl [IP address]” in both the host OS & haproxy container receives nearly a 50/50 mix of ‘empty reply from server’ and the nginx welcome page. Nginx container does nothing, just a long delay that I break with ctrl+c.

“curl subdomain.example.com” has same results as “curl [IP address]”

When I do the same test “curl domain.com” on an active site on my current non-lxd non-vm/container server, the page loads every time consistently from the host OS, haproxy container and nginx container.

I also tried “curl 10.0.0.50” (the private IP of the nginx container) from host OS, haproxy container and nginx container. This appeared to work perfectly without any ERR_* errors as well.

When I stop using LXD & stop proxying port 80 and install Nginx directly in the host OS it appears to work perfectly. No ERR_* errors. The page loads quickly every time without error.

I further tried testing LXD proxy device forwarding directly to an Nginx container (with haproxy not involved). In this test, connecting from my computer appeared to work perfectly as well. With this additional test it seems to narrow the ERR_* issues down to something related to the HAProxy segment, either haproxy directly or maybe something in the connection. Not sure yet.

HAProxy on the host OS, without LXD proxy device, forwarding to container still has the intermittent ERR_* errors. Works pretty much the same as when haproxy is in a container forwarding to the nginx container.

In case this is pertinent, when I setup ‘lxd init’ I used IP 10.0.0.1/24 and had the init script setup the default bridge.

Not sure what else to do but the ERR_EMPTY_RESPONSE & ERR_CONNECTION_RESET issues need to be sorted out before I can transfer my sites over to this new server where I’ll be using LXD & HAProxy to direct traffic to different LXC Ubuntu containers. Seems like a high percentage of page views could be lost with these ERR_* page loading error issues.

Edited again to add screenshot clips of the browser error messages:

LXDproxy-error1

LXDproxy-error2

LXDproxy-error3

Those errors only occur initially when accessing the sites through my web browsers, then the nginx welcome site loads & refreshes normally. But it does repeat if I wait a bit of time or restart the browsers.

Any ideas what might be causing this or what I should try troubleshooting next?

Thanks in advance for any help.

John

I believe I finally tracked down the issue. I’ve been updating my first post while investigating further. Will post this as a separate reply as I believe this may be the solution. Also leaving this thread up in case it’s useful for anyone in the future.

For haproxy I followed a recent tutorial, from January 2022, and as part of that the setting ‘mode tcp’ was used in the frontend and backend settings. Ran a check on the config file & determined that ‘mode http’ was needed for something.

Solution appears to be replacing ‘mode tcp’ in frontend & backend and changing that to ‘mode http’. After that I haven’t seen any ERR_* messages since.

Now going to track down the tutorial I used to post a reply there linking to this article for the issue I ran into.