HTTP connection upgrade to TCP seems broken since 2.4

proudier · August 12, 2024, 6:24pm

Summary

Upgrading a HTTP connection to TCP no long work for versions greater than v2.3.
This is NOT about upgrading a connection in a TCP frontend using switch-mode http, nor is it about WebSocket. It’s about the Connection: Upgrade and Upgrade: tcp HTTP headers.

With HAProxy 2.2: client and server could upgrade the connection and converse bidirectionnaly using a proprietary binary format on the connection.

With HAProxy 3.0: proxy returns HTTP 200 (instead of expected 101) and connection upgrade fails.

Looking at traces, the problem seems to originate from the parsing of the HTTP request. The Upgrade header seems ignored and is not passed to the backend, so no upgrade takes place.

Looking at the code, the h1_parse_upgrade_header seems to ignore the Upgrade header unless it’s set to websocket.

Do you think this is the intended behavior? Or is it a shortcoming?

Context

The docker-socket-proxy project uses HAproxy to filter requests to the Docker daemon using ACLs (like here). It worked well with HAproxy v2.2 and broke when I updated HAproxy to v3.0.

The proxied protocol is the Docker Engine API. It’s mostly HTTP REST but for some endpoints, the connection is “hijacked”: upgraded to TCP, then used for streaming a proprietary binary format in both directions. Hijacking is detailed here in Docker doc.

An example of problematic call is the POST to /v1.46/exec/{id}/start.

Observations

Note: output of various commands were redacted for brevity

When working (HAproxy 2.2)

Proxy logs

dockerhttp dockerhttp/dockersocket 0/0/0/0/29 101 316 - - ---- 2/2/0/0/0 0/0 "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1"

Wireshark capture, between client and proxy

Hypertext Transfer Protocol
    POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1\r\n
    Connection: Upgrade\r\n
    Content-Type: application/json\r\n
    Upgrade: tcp\r\n

Hypertext Transfer Protocol
    HTTP/1.1 101 UPGRADED\r\n
    content-type: application/vnd.docker.multiplexed-stream\r\n
    connection: Upgrade\r\n
    upgrade: tcp\r\n

Transmission Control Protocol
    # A paquet with binary data that respect the docker.multiplexed-stream content-type

Traces

[04|h1|0|mux_h1.c:1534] H1 request fully rcvd : [F] [MSG_DONE, MSG_RPBEFORE] - "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1" - h1c=0x7551d2ecf090(0x00000000) h1s=0x7551d2ecf7b0(0x00000410) ibuf=280@0x7551d2a02f00+48/16384 obuf=0@0+0/0
	htx=0x7551d2a07810(size=16336,data=276,used=10,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=9,tail_addr=276,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_REQ_SL    - size=118    - addr=0     	POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1
		[1] type=HTX_BLK_HDR       - size=18     - addr=118   	host: 127.0.0.1:2375
		[2] type=HTX_BLK_HDR       - size=38     - addr=136   	user-agent: Docker-Client/27.1.1 (linux)
		[3] type=HTX_BLK_HDR       - size=16     - addr=174   	content-length: 29
		[4] type=HTX_BLK_HDR       - size=17     - addr=190   	connection: Upgrade
		[5] type=HTX_BLK_HDR       - size=28     - addr=207   	content-type: application/json
		[6] type=HTX_BLK_HDR       - size=10     - addr=235   	upgrade: tcp
		[7] type=HTX_BLK_EOH       - size=1      - addr=245   	<empty>
		[8] type=HTX_BLK_DATA      - size=29     - addr=246
		[9] type=HTX_BLK_EOM       - size=1      - addr=275   	<empty>

# snip

[04|h1|0|mux_h1.c:1739] sending request headers : [B] [MSG_RQBEFORE, MSG_RPBEFORE] - "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1" - h1c=0x7551d2ecf680(0x00000000) h1s=0x7551d2ecf9a0(0x00000010) ibuf=0@0+0/0 obuf=0@0x7551d2a10a50+0/16384
	htx=0x7551d2a07810(size=16336,data=276,used=10,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=9,tail_addr=276,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_REQ_SL    - size=118    - addr=0     	POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1
		[1] type=HTX_BLK_HDR       - size=18     - addr=118   	host: 127.0.0.1:2375
		[2] type=HTX_BLK_HDR       - size=38     - addr=136   	user-agent: Docker-Client/27.1.1 (linux)
		[3] type=HTX_BLK_HDR       - size=16     - addr=174   	content-length: 29
		[4] type=HTX_BLK_HDR       - size=17     - addr=190   	connection: Upgrade
		[5] type=HTX_BLK_HDR       - size=28     - addr=207   	content-type: application/json
		[6] type=HTX_BLK_HDR       - size=10     - addr=235   	upgrade: tcp
		[7] type=HTX_BLK_EOH       - size=1      - addr=245   	<empty>
		[8] type=HTX_BLK_DATA      - size=29     - addr=246
		[9] type=HTX_BLK_EOM       - size=1      - addr=275   	<empty>

# snip

[04|h1|0|mux_h1.c:1501] rcvd H1 response headers : [B] [MSG_TUNNEL, MSG_TUNNEL] - VAL=210 - "HTTP/1.1 101 UPGRADED" - h1c=0x7551d2ecf680(0x00000000) h1s=0x7551d2ecf9a0(0x00004020) ibuf=210@0x7551d2a02f00+0/16384 obuf=0@0+0/0
	htx=0x7551d2a07810(size=16336,data=201,used=9,wrap=NO,flags=0x00000000,extra=18446744073709551615,first=0,head=0,tail=8,tail_addr=201,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_RES_SL    - size=43     - addr=0     	HTTP/1.1 101 UPGRADED
		[1] type=HTX_BLK_HDR       - size=53     - addr=43    	content-type: application/vnd.docker.multiplexed-stream
		[2] type=HTX_BLK_HDR       - size=17     - addr=96    	connection: Upgrade
		[3] type=HTX_BLK_HDR       - size=10     - addr=113   	upgrade: tcp
		[4] type=HTX_BLK_HDR       - size=15     - addr=123   	api-version: 1.46
		[5] type=HTX_BLK_HDR       - size=24     - addr=138   	docker-experimental: false
		[6] type=HTX_BLK_HDR       - size=11     - addr=162   	ostype: linux
		[7] type=HTX_BLK_HDR       - size=27     - addr=173   	server: Docker/27.1.1 (linux)
		[8] type=HTX_BLK_EOH       - size=1      - addr=200   	<empty>

When broken (HAproxy 3.0)

Proxy logs

dockerhttp dockerhttp/dockersocket 0/0/0/0/20 200 156 - - ---- 1/1/0/0/0 0/0 "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1"

Wireshark capture, between client and proxy

Hypertext Transfer Protocol
    POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1\r\n
    Connection: Upgrade\r\n
    Content-Type: application/json\r\n
    Upgrade: tcp\r\n

Hypertext Transfer Protocol
    HTTP/1.1 200 OK\r\n
    content-type: application/vnd.docker.raw-stream\r\n
    connection: close\r\n

# No TCP packet with binary data

Traces

[02|h1|1|mux_h1.c:2087] H1 request fully rcvd : [F,EMB] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=29)  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1" - h1c=0x7341fce5e1f0(0x00000000) conn=0x7341fce5e100(0x80000300) h1s=0x7341fce5e2e0(0x00000010) sd=0x7341fd80ec90(0x00500021) ibuf=280@0x7341fce8a530+48/16384 obuf=0@0+0/0
	htx=0x7341fce7ca00(size=16336,data=261,used=8,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=7,tail_addr=261,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_REQ_SL    - size=114    - addr=0     	POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1
		[1] type=HTX_BLK_HDR       - size=18     - addr=114   	host: 127.0.0.1:2375
		[2] type=HTX_BLK_HDR       - size=38     - addr=132   	user-agent: Docker-Client/27.1.1 (linux)
		[3] type=HTX_BLK_HDR       - size=16     - addr=170   	content-length: 29
		[4] type=HTX_BLK_HDR       - size=17     - addr=186   	connection: Upgrade
		[5] type=HTX_BLK_HDR       - size=28     - addr=203   	content-type: application/json
		[6] type=HTX_BLK_EOH       - size=1      - addr=231   	<empty>
		[7] type=HTX_BLK_DATA      - size=29     - addr=232


# snip

[02|h1|1|mux_h1.c:2291] sending request headers : [B,RUN] [MSG_RQBEFORE, MSG_RPBEFORE] - req=(.fl=0x00001400 .curr_len=0 .body_len=0)  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1" - h1c=0x7341fce5e4c0(0x80000000) conn=0x7341fce5e3d0(0x00000300) h1s=0x7341fce5e5b0(0x00100010) sd=0x7341fd80ec10(0x40400001) sc=0x7341fd515f00(0x00001c31) ibuf=0@0+0/0 obuf=0@0x7341fce85c20+0/16384
	htx=0x7341fce7ca00(size=16336,data=261,used=8,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=7,tail_addr=261,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_REQ_SL    - size=114    - addr=0     	POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1
		[1] type=HTX_BLK_HDR       - size=18     - addr=114   	host: 127.0.0.1:2375
		[2] type=HTX_BLK_HDR       - size=38     - addr=132   	user-agent: Docker-Client/27.1.1 (linux)
		[3] type=HTX_BLK_HDR       - size=16     - addr=170   	content-length: 29
		[4] type=HTX_BLK_HDR       - size=17     - addr=186   	connection: Upgrade
		[5] type=HTX_BLK_HDR       - size=28     - addr=203   	content-type: application/json
		[6] type=HTX_BLK_EOH       - size=1      - addr=231   	<empty>
		[7] type=HTX_BLK_DATA      - size=29     - addr=232

# snip

[02|h1|1|mux_h1.c:2376] sending response headers : [F,RUN] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=29)  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "HTTP/1.1 200 OK" - h1c=0x7341fce5e1f0(0x00000000) conn=0x7341fce5e100(0x80000300) h1s=0x7341fce5e2e0(0x00100010) sd=0x7341fd80ec90(0x50404001) sc=0x7341fd515ea0(0x00001422) ibuf=0@0+0/0 obuf=0@0x7341fce8a530+0/16384
	htx=0x7341fce7ca00(size=16336,data=156,used=7,wrap=NO,flags=0x00000000,extra=18446744073709551615,first=0,head=0,tail=6,tail_addr=156,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_RES_SL    - size=33     - addr=0     	HTTP/1.1 200 OK
		[1] type=HTX_BLK_HDR       - size=45     - addr=33    	content-type: application/vnd.docker.raw-stream
		[2] type=HTX_BLK_HDR       - size=15     - addr=78    	api-version: 1.46
		[3] type=HTX_BLK_HDR       - size=24     - addr=93    	docker-experimental: false
		[4] type=HTX_BLK_HDR       - size=11     - addr=117   	ostype: linux
		[5] type=HTX_BLK_HDR       - size=27     - addr=128   	server: Docker/27.1.1 (linux)
		[6] type=HTX_BLK_EOH       - size=1      - addr=155   	<empty>

How to reproduce

haproxy.cfg:

global
    log stdout len 4096 format raw daemon debug
    stats socket ipv4@:10000 level admin
    maxconn 100

defaults
    log global
    timeout connect 10s
    timeout client 10m
    timeout server 10m
    errorfile 400 /usr/local/etc/haproxy/errors/400.http
    errorfile 403 /usr/local/etc/haproxy/errors/403.http
    errorfile 408 /usr/local/etc/haproxy/errors/408.http
    errorfile 500 /usr/local/etc/haproxy/errors/500.http
    errorfile 502 /usr/local/etc/haproxy/errors/502.http
    errorfile 503 /usr/local/etc/haproxy/errors/503.http
    errorfile 504 /usr/local/etc/haproxy/errors/504.http

frontend dockerhttp
    bind :2375
    mode http
    option httplog
    default_backend dockerhttp

backend dockerhttp
    mode http
    server dockersocket /var/run/docker.sock

As an user that has access to /var/run/docker.sock (upgrade version as needed and make sure Docker daemon is up)

 docker run --rm --name haproxy \
    --user root --privileged \
    -v ./haproxy.cfg:/haproxy.cfg \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p 127.0.0.1:2375:2375 \
    -p 127.0.0.1:10000:10000 \
    haproxy:2.2-alpine haproxy -f /haproxy.cfg

In another terminal:

export DOCKER_HOST=tcp://127.0.0.1:2375
docker exec haproxy ls

lukastribus · August 12, 2024, 7:10pm

The affecting change is probably (based on the code that you found) this one:

github.com/haproxy/haproxy

MINOR: h1: reject websocket handshake if missing key

committed 03:37PM - 28 Jan 21 UTC

a-denoyelle

+97 -0

If a request is identified as a WebSocket handshake, it must contains a websocke…t key header or else it can be reject, following the rfc6455. A new flag H1_MF_UPG_WEBSOCKET is set on such messages. For the request te be identified as a WebSocket handshake, it must contains the headers: Connection: upgrade Upgrade: websocket This commit is a compagnon of "MEDIUM: h1: generate WebSocket key on response if needed" and "MEDIUM: h1: add a WebSocket key on handshake if needed". Indeed, it ensures that a WebSocket key is added only from a http/2 side and not for a http/1 bogus peer.

I would suggest you file a bug at Github:

Do make sure to emphasize the docker docs regarding the connection hijacking.

edit: this has been filed as:

github.com/haproxy/haproxy

HTTP connection upgrade to TCP seems broken since 2.4

opened 08:40PM - 13 Aug 24 UTC

proudier

type: bug status: needs-triage

### Detailed Description of the Problem **Summary:** [Upgrading](https://d…eveloper.mozilla.org/en-US/docs/Web/HTTP/Headers/Upgrade) a HTTP connection to TCP no longer work for versions greater than v2.3. This is NOT about upgrading a connection in a TCP frontend using switch-mode http, nor is it about WebSocket. It’s about the Connection: Upgrade and Upgrade: tcp HTTP headers. The proxied protocol is the Docker Engine API. It’s mostly HTTP REST but for some endpoints, the connection is “hijacked”: upgraded to TCP, then used for streaming a proprietary binary format in both directions. Hijacking is detailed [here in Docker doc](https://docs.docker.com/engine/api/v1.46/#tag/Container/operation/ContainerAttach). An example of problematic call is the POST to /v1.46/exec/{id}/start. With HAProxy 3.0: proxy returns HTTP 200 (instead of expected 101) and connection upgrade fails. **Proxy logs:** ``` dockerhttp dockerhttp/dockersocket 0/0/0/0/20 200 156 - - ---- 1/1/0/0/0 0/0 "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1" ``` **Wireshark capture, between client and proxy:** ``` Hypertext Transfer Protocol POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1\r\n Connection: Upgrade\r\n Content-Type: application/json\r\n Upgrade: tcp\r\n Hypertext Transfer Protocol HTTP/1.1 200 OK\r\n content-type: application/vnd.docker.raw-stream\r\n connection: close\r\n # No TCP packet with binary data ``` **Traces:** ``` [02|h1|1|mux_h1.c:2087] H1 request fully rcvd : [F,EMB] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=29) res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1" - h1c=0x7341fce5e1f0(0x00000000) conn=0x7341fce5e100(0x80000300) h1s=0x7341fce5e2e0(0x00000010) sd=0x7341fd80ec90(0x00500021) ibuf=280@0x7341fce8a530+48/16384 obuf=0@0+0/0 htx=0x7341fce7ca00(size=16336,data=261,used=8,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=7,tail_addr=261,head_addr=0,end_addr=0) [0] type=HTX_BLK_REQ_SL - size=114 - addr=0 POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1 [1] type=HTX_BLK_HDR - size=18 - addr=114 host: 127.0.0.1:2375 [2] type=HTX_BLK_HDR - size=38 - addr=132 user-agent: Docker-Client/27.1.1 (linux) [3] type=HTX_BLK_HDR - size=16 - addr=170 content-length: 29 [4] type=HTX_BLK_HDR - size=17 - addr=186 connection: Upgrade [5] type=HTX_BLK_HDR - size=28 - addr=203 content-type: application/json [6] type=HTX_BLK_EOH - size=1 - addr=231 <empty> [7] type=HTX_BLK_DATA - size=29 - addr=232 # snip [02|h1|1|mux_h1.c:2291] sending request headers : [B,RUN] [MSG_RQBEFORE, MSG_RPBEFORE] - req=(.fl=0x00001400 .curr_len=0 .body_len=0) res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1" - h1c=0x7341fce5e4c0(0x80000000) conn=0x7341fce5e3d0(0x00000300) h1s=0x7341fce5e5b0(0x00100010) sd=0x7341fd80ec10(0x40400001) sc=0x7341fd515f00(0x00001c31) ibuf=0@0+0/0 obuf=0@0x7341fce85c20+0/16384 htx=0x7341fce7ca00(size=16336,data=261,used=8,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=7,tail_addr=261,head_addr=0,end_addr=0) [0] type=HTX_BLK_REQ_SL - size=114 - addr=0 POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1 [1] type=HTX_BLK_HDR - size=18 - addr=114 host: 127.0.0.1:2375 [2] type=HTX_BLK_HDR - size=38 - addr=132 user-agent: Docker-Client/27.1.1 (linux) [3] type=HTX_BLK_HDR - size=16 - addr=170 content-length: 29 [4] type=HTX_BLK_HDR - size=17 - addr=186 connection: Upgrade [5] type=HTX_BLK_HDR - size=28 - addr=203 content-type: application/json [6] type=HTX_BLK_EOH - size=1 - addr=231 <empty> [7] type=HTX_BLK_DATA - size=29 - addr=232 # snip [02|h1|1|mux_h1.c:2376] sending response headers : [F,RUN] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=29) res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "HTTP/1.1 200 OK" - h1c=0x7341fce5e1f0(0x00000000) conn=0x7341fce5e100(0x80000300) h1s=0x7341fce5e2e0(0x00100010) sd=0x7341fd80ec90(0x50404001) sc=0x7341fd515ea0(0x00001422) ibuf=0@0+0/0 obuf=0@0x7341fce8a530+0/16384 htx=0x7341fce7ca00(size=16336,data=156,used=7,wrap=NO,flags=0x00000000,extra=18446744073709551615,first=0,head=0,tail=6,tail_addr=156,head_addr=0,end_addr=0) [0] type=HTX_BLK_RES_SL - size=33 - addr=0 HTTP/1.1 200 OK [1] type=HTX_BLK_HDR - size=45 - addr=33 content-type: application/vnd.docker.raw-stream [2] type=HTX_BLK_HDR - size=15 - addr=78 api-version: 1.46 [3] type=HTX_BLK_HDR - size=24 - addr=93 docker-experimental: false [4] type=HTX_BLK_HDR - size=11 - addr=117 ostype: linux [5] type=HTX_BLK_HDR - size=27 - addr=128 server: Docker/27.1.1 (linux) [6] type=HTX_BLK_EOH - size=1 - addr=155 <empty> ``` **More context:** The [docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy) project uses HAproxy to filter requests to the Docker daemon using ACLs (like [here](https://github.com/Tecnativa/docker-socket-proxy/blob/e37525d1577226e518a93adbdcedc2b2fd377c58/haproxy.cfg#L49_)). It worked well with HAproxy v2.2 and [broke](https://github.com/Tecnativa/docker-socket-proxy/issues/132) when I updated HAproxy to v3.0. [Post in the forum](https://discourse.haproxy.org/t/http-connection-upgrade-to-tcp-seems-broken-since-2-4/10115/1) ### Expected Behavior With HAProxy 2.2: client and server could upgrade the connection and converse bidirectionally using a proprietary binary format on the connection. **Proxy Logs:** ``` dockerhttp dockerhttp/dockersocket 0/0/0/0/29 101 316 - - ---- 2/2/0/0/0 0/0 "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1" ``` **Wireshark capture, between client and proxy:** ``` Hypertext Transfer Protocol POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1\r\n Connection: Upgrade\r\n Content-Type: application/json\r\n Upgrade: tcp\r\n Hypertext Transfer Protocol HTTP/1.1 101 UPGRADED\r\n content-type: application/vnd.docker.multiplexed-stream\r\n connection: Upgrade\r\n upgrade: tcp\r\n Transmission Control Protocol # A paquet with binary data that respect the docker.multiplexed-stream content-type ``` **Traces:** ``` [04|h1|0|mux_h1.c:1534] H1 request fully rcvd : [F] [MSG_DONE, MSG_RPBEFORE] - "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1" - h1c=0x7551d2ecf090(0x00000000) h1s=0x7551d2ecf7b0(0x00000410) ibuf=280@0x7551d2a02f00+48/16384 obuf=0@0+0/0 htx=0x7551d2a07810(size=16336,data=276,used=10,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=9,tail_addr=276,head_addr=0,end_addr=0) [0] type=HTX_BLK_REQ_SL - size=118 - addr=0 POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1 [1] type=HTX_BLK_HDR - size=18 - addr=118 host: 127.0.0.1:2375 [2] type=HTX_BLK_HDR - size=38 - addr=136 user-agent: Docker-Client/27.1.1 (linux) [3] type=HTX_BLK_HDR - size=16 - addr=174 content-length: 29 [4] type=HTX_BLK_HDR - size=17 - addr=190 connection: Upgrade [5] type=HTX_BLK_HDR - size=28 - addr=207 content-type: application/json [6] type=HTX_BLK_HDR - size=10 - addr=235 upgrade: tcp [7] type=HTX_BLK_EOH - size=1 - addr=245 <empty> [8] type=HTX_BLK_DATA - size=29 - addr=246 [9] type=HTX_BLK_EOM - size=1 - addr=275 <empty> # snip [04|h1|0|mux_h1.c:1739] sending request headers : [B] [MSG_RQBEFORE, MSG_RPBEFORE] - "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1" - h1c=0x7551d2ecf680(0x00000000) h1s=0x7551d2ecf9a0(0x00000010) ibuf=0@0+0/0 obuf=0@0x7551d2a10a50+0/16384 htx=0x7551d2a07810(size=16336,data=276,used=10,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=9,tail_addr=276,head_addr=0,end_addr=0) [0] type=HTX_BLK_REQ_SL - size=118 - addr=0 POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1 [1] type=HTX_BLK_HDR - size=18 - addr=118 host: 127.0.0.1:2375 [2] type=HTX_BLK_HDR - size=38 - addr=136 user-agent: Docker-Client/27.1.1 (linux) [3] type=HTX_BLK_HDR - size=16 - addr=174 content-length: 29 [4] type=HTX_BLK_HDR - size=17 - addr=190 connection: Upgrade [5] type=HTX_BLK_HDR - size=28 - addr=207 content-type: application/json [6] type=HTX_BLK_HDR - size=10 - addr=235 upgrade: tcp [7] type=HTX_BLK_EOH - size=1 - addr=245 <empty> [8] type=HTX_BLK_DATA - size=29 - addr=246 [9] type=HTX_BLK_EOM - size=1 - addr=275 <empty> # snip [04|h1|0|mux_h1.c:1501] rcvd H1 response headers : [B] [MSG_TUNNEL, MSG_TUNNEL] - VAL=210 - "HTTP/1.1 101 UPGRADED" - h1c=0x7551d2ecf680(0x00000000) h1s=0x7551d2ecf9a0(0x00004020) ibuf=210@0x7551d2a02f00+0/16384 obuf=0@0+0/0 htx=0x7551d2a07810(size=16336,data=201,used=9,wrap=NO,flags=0x00000000,extra=18446744073709551615,first=0,head=0,tail=8,tail_addr=201,head_addr=0,end_addr=0) [0] type=HTX_BLK_RES_SL - size=43 - addr=0 HTTP/1.1 101 UPGRADED [1] type=HTX_BLK_HDR - size=53 - addr=43 content-type: application/vnd.docker.multiplexed-stream [2] type=HTX_BLK_HDR - size=17 - addr=96 connection: Upgrade [3] type=HTX_BLK_HDR - size=10 - addr=113 upgrade: tcp [4] type=HTX_BLK_HDR - size=15 - addr=123 api-version: 1.46 [5] type=HTX_BLK_HDR - size=24 - addr=138 docker-experimental: false [6] type=HTX_BLK_HDR - size=11 - addr=162 ostype: linux [7] type=HTX_BLK_HDR - size=27 - addr=173 server: Docker/27.1.1 (linux) [8] type=HTX_BLK_EOH - size=1 - addr=200 <empty> ``` ### Steps to Reproduce the Behavior Pre-requisites: - On a computer with the docker daemon started - As an user that can assess `/var/run/docker.sock` - Save config shared below in haproxy.cfg In terminal 1: ``` docker run --rm --name haproxy \ --user root --privileged \ -v ./haproxy.cfg:/haproxy.cfg \ -v /var/run/docker.sock:/var/run/docker.sock \ -p 127.0.0.1:2375:2375 \ -p 127.0.0.1:10000:10000 \ haproxy:3.0-alpine haproxy -f /haproxy.cfg ``` In terminal 2: ``` export DOCKER_HOST=tcp://127.0.0.1:2375 docker exec haproxy ls ``` The message `unable to upgrade to tcp, received 200` indicates an error ### Do you have any idea what may have caused this? Looking at traces, the problem seems to originate from the parsing of the HTTP request. The Upgrade header seems ignored and is not passed to the backend, so no upgrade takes place. Looking at the code, the [h1_parse_upgrade_header](https://github.com/haproxy/haproxy/blob/5590ada4731a1f75004675680b4bdca61fa4c507/src/h1.c#L349) seems to ignore the Upgrade header unless it’s set to websocket. In the forum, lukastribus [pointed](https://discourse.haproxy.org/t/http-connection-upgrade-to-tcp-seems-broken-since-2-4/10115/2) at [this commit](https://github.com/haproxy/haproxy/commit/18ee5c3eb0c8afc18627a0d9a096aabb4c61e34f). ### Do you have an idea how to solve the issue? _No response_ ### What is your configuration? ```haproxy global log stdout len 4096 format raw daemon debug stats socket ipv4@:10000 level admin maxconn 100 defaults log global timeout connect 10s timeout client 10m timeout server 10m errorfile 400 /usr/local/etc/haproxy/errors/400.http errorfile 403 /usr/local/etc/haproxy/errors/403.http errorfile 408 /usr/local/etc/haproxy/errors/408.http errorfile 500 /usr/local/etc/haproxy/errors/500.http errorfile 502 /usr/local/etc/haproxy/errors/502.http errorfile 503 /usr/local/etc/haproxy/errors/503.http errorfile 504 /usr/local/etc/haproxy/errors/504.http frontend dockerhttp bind :2375 mode http option httplog default_backend dockerhttp backend dockerhttp mode http server dockersocket /var/run/docker.sock ``` ### Output of `haproxy -vv` ```plain HAProxy version 3.0.3-95a607c 2024/07/11 - https://haproxy.org/ Status: long-term supported branch - will stop receiving fixes around Q2 2029. Known bugs: http://www.haproxy.org/bugs/bugs-3.0.3.html Running on: Linux 6.10.4-arch2-1 #1 SMP PREEMPT_DYNAMIC Sun, 11 Aug 2024 16:19:06 +0000 x86_64 Build options : TARGET = linux-musl CC = cc CFLAGS = -O2 -g -fwrapv OPTIONS = USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 DEBUG = Feature list : -51DEGREES +ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=8). Built with OpenSSL version : OpenSSL 3.3.1 4 Jun 2024 Running on OpenSSL version : OpenSSL 3.3.1 4 Jun 2024 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 OpenSSL providers loaded : default Built with Lua version : Lua 5.4.6 Built with the Prometheus exporter as a service Built with network namespace support. Built with libslz for stateless compression. Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with PCRE2 version : 10.43 2024-02-16 PCRE2 library supports JIT : yes Encrypted password support via crypt(3): yes Built with gcc compiler version 13.2.1 20240309 Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG <default> : mode=HTTP side=FE|BE mux=H1 flags=HTX h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG <default> : mode=TCP side=FE|BE mux=PASS flags= none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG Available services : prometheus-exporter Available filters : [BWLIM] bwlim-in [BWLIM] bwlim-out [CACHE] cache [COMP] compression [FCGI] fcgi-app [SPOE] spoe [TRACE] trace ``` ### Last Outputs and Backtraces ```plain n/a ``` ### Additional Information n/a

Rickard_505 · September 20, 2024, 1:45pm

wow - finally someone else (that knew the right lingo) saw this

Topic		Replies	Views
TCP and HTTP Connections - TCP got considered as HTTP and throws Handshake error Help!	5	29	November 18, 2024
Haproxy 1.7.5-2 doesn't update http_req_cnt properly in stick-tables Help!	0	1073	October 2, 2017
Server Disconnects / TCP Window Full Help!	6	970	May 9, 2023
Performance Degraded after upgrade from 2.2.24 container image to 2.2.29 Help!	0	223	June 12, 2023
HAProxy and TCP Mode not working correctly on Layer 7 Help!	0	469	February 15, 2020

HTTP connection upgrade to TCP seems broken since 2.4

Summary

Context

Observations

When working (HAproxy 2.2)

Proxy logs

Wireshark capture, between client and proxy

Traces

When broken (HAproxy 3.0)

Proxy logs

Wireshark capture, between client and proxy

Traces

How to reproduce

Related topics