# HTTP connection upgrade to TCP seems broken since 2.4

**URL:** <https://discourse.haproxy.org/t/http-connection-upgrade-to-tcp-seems-broken-since-2-4/10115>\
**Category:** Help!\
**Created:** [August 12, 2024, 6:24pm UTC](https://discourse.haproxy.org/t/http-connection-upgrade-to-tcp-seems-broken-since-2-4/10115 "2024-08-12T18:24:41Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![proudier](https://avatars.discourse-cdn.com/v4/letter/p/c4cdca/32.png) [@proudier](https://discourse.haproxy.org/u/proudier)\
**Post date:** [August 12, 2024, 6:24pm UTC](https://discourse.haproxy.org/t/http-connection-upgrade-to-tcp-seems-broken-since-2-4/10115/1 "2024-08-12T18:24:41Z")

</div>

# Summary

[Upgrading](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Upgrade) a HTTP connection to TCP no long work for versions greater than v2.3.  
This is NOT about upgrading a connection in a TCP frontend using `switch-mode http`, nor is it about WebSocket. It’s about the `Connection: Upgrade` and `Upgrade: tcp` HTTP headers.

With HAProxy 2.2: client and server could upgrade the connection and converse bidirectionnaly using a proprietary binary format on the connection.

With HAProxy 3.0: proxy returns HTTP 200 (instead of expected 101) and connection upgrade fails.

Looking at traces, the problem seems to originate from the parsing of the HTTP request. The `Upgrade` header seems ignored and is not passed to the backend, so no upgrade takes place.

Looking at the code, the [h1\_parse\_upgrade\_header](https://github.com/haproxy/haproxy/blob/5590ada4731a1f75004675680b4bdca61fa4c507/src/h1.c#L349) seems to ignore the `Upgrade` header unless it’s set to `websocket`.

Do you think this is the intended behavior? Or is it a shortcoming?

# Context

The [docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy) project uses HAproxy to filter requests to the Docker daemon using ACLs (like [here](https://github.com/Tecnativa/docker-socket-proxy/blob/e37525d1577226e518a93adbdcedc2b2fd377c58/haproxy.cfg#L49_)). It worked well with HAproxy v2.2 and [broke](https://github.com/Tecnativa/docker-socket-proxy/issues/132) when I updated HAproxy to v3.0.

The proxied protocol is the Docker Engine API. It’s mostly HTTP REST but for some endpoints, the connection is “hijacked”: upgraded to TCP, then used for streaming a proprietary binary format in both directions. Hijacking is detailed [here in Docker doc](https://docs.docker.com/engine/api/v1.46/#tag/Container/operation/ContainerAttach).

An example of problematic call is the POST to `/v1.46/exec/{id}/start`.

# Observations

Note: output of various commands were redacted for brevity

## When working (HAproxy 2.2)

### Proxy logs

```auto
dockerhttp dockerhttp/dockersocket 0/0/0/0/29 101 316 - - ---- 2/2/0/0/0 0/0 "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1"

```

### Wireshark capture, between client and proxy

```auto
Hypertext Transfer Protocol
    POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1\r\n
    Connection: Upgrade\r\n
    Content-Type: application/json\r\n
    Upgrade: tcp\r\n

Hypertext Transfer Protocol
    HTTP/1.1 101 UPGRADED\r\n
    content-type: application/vnd.docker.multiplexed-stream\r\n
    connection: Upgrade\r\n
    upgrade: tcp\r\n

Transmission Control Protocol
    # A paquet with binary data that respect the docker.multiplexed-stream content-type

```

### Traces

```auto
[04|h1|0|mux_h1.c:1534] H1 request fully rcvd : [F] [MSG_DONE, MSG_RPBEFORE] - "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1" - h1c=0x7551d2ecf090(0x00000000) h1s=0x7551d2ecf7b0(0x00000410) ibuf=280@0x7551d2a02f00+48/16384 obuf=0@0+0/0
	htx=0x7551d2a07810(size=16336,data=276,used=10,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=9,tail_addr=276,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_REQ_SL - size=118 - addr=0 POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1
		[1] type=HTX_BLK_HDR - size=18 - addr=118 host: 127.0.0.1:2375
		[2] type=HTX_BLK_HDR - size=38 - addr=136 user-agent: Docker-Client/27.1.1 (linux)
		[3] type=HTX_BLK_HDR - size=16 - addr=174 content-length: 29
		[4] type=HTX_BLK_HDR - size=17 - addr=190 connection: Upgrade
		[5] type=HTX_BLK_HDR - size=28 - addr=207 content-type: application/json
		[6] type=HTX_BLK_HDR - size=10 - addr=235 upgrade: tcp
		[7] type=HTX_BLK_EOH - size=1 - addr=245 <empty>
		[8] type=HTX_BLK_DATA - size=29 - addr=246
		[9] type=HTX_BLK_EOM - size=1 - addr=275 <empty>

# snip

[04|h1|0|mux_h1.c:1739] sending request headers : [B] [MSG_RQBEFORE, MSG_RPBEFORE] - "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1" - h1c=0x7551d2ecf680(0x00000000) h1s=0x7551d2ecf9a0(0x00000010) ibuf=0@0+0/0 obuf=0@0x7551d2a10a50+0/16384
	htx=0x7551d2a07810(size=16336,data=276,used=10,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=9,tail_addr=276,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_REQ_SL - size=118 - addr=0 POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1
		[1] type=HTX_BLK_HDR - size=18 - addr=118 host: 127.0.0.1:2375
		[2] type=HTX_BLK_HDR - size=38 - addr=136 user-agent: Docker-Client/27.1.1 (linux)
		[3] type=HTX_BLK_HDR - size=16 - addr=174 content-length: 29
		[4] type=HTX_BLK_HDR - size=17 - addr=190 connection: Upgrade
		[5] type=HTX_BLK_HDR - size=28 - addr=207 content-type: application/json
		[6] type=HTX_BLK_HDR - size=10 - addr=235 upgrade: tcp
		[7] type=HTX_BLK_EOH - size=1 - addr=245 <empty>
		[8] type=HTX_BLK_DATA - size=29 - addr=246
		[9] type=HTX_BLK_EOM - size=1 - addr=275 <empty>

# snip

[04|h1|0|mux_h1.c:1501] rcvd H1 response headers : [B] [MSG_TUNNEL, MSG_TUNNEL] - VAL=210 - "HTTP/1.1 101 UPGRADED" - h1c=0x7551d2ecf680(0x00000000) h1s=0x7551d2ecf9a0(0x00004020) ibuf=210@0x7551d2a02f00+0/16384 obuf=0@0+0/0
	htx=0x7551d2a07810(size=16336,data=201,used=9,wrap=NO,flags=0x00000000,extra=18446744073709551615,first=0,head=0,tail=8,tail_addr=201,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_RES_SL - size=43 - addr=0 HTTP/1.1 101 UPGRADED
		[1] type=HTX_BLK_HDR - size=53 - addr=43 content-type: application/vnd.docker.multiplexed-stream
		[2] type=HTX_BLK_HDR - size=17 - addr=96 connection: Upgrade
		[3] type=HTX_BLK_HDR - size=10 - addr=113 upgrade: tcp
		[4] type=HTX_BLK_HDR - size=15 - addr=123 api-version: 1.46
		[5] type=HTX_BLK_HDR - size=24 - addr=138 docker-experimental: false
		[6] type=HTX_BLK_HDR - size=11 - addr=162 ostype: linux
		[7] type=HTX_BLK_HDR - size=27 - addr=173 server: Docker/27.1.1 (linux)
		[8] type=HTX_BLK_EOH - size=1 - addr=200 <empty>

```

## When broken (HAproxy 3.0)

### Proxy logs

```auto
dockerhttp dockerhttp/dockersocket 0/0/0/0/20 200 156 - - ---- 1/1/0/0/0 0/0 "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1"

```

### Wireshark capture, between client and proxy

```auto
Hypertext Transfer Protocol
    POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1\r\n
    Connection: Upgrade\r\n
    Content-Type: application/json\r\n
    Upgrade: tcp\r\n

Hypertext Transfer Protocol
    HTTP/1.1 200 OK\r\n
    content-type: application/vnd.docker.raw-stream\r\n
    connection: close\r\n

# No TCP packet with binary data

```

### Traces

```auto
[02|h1|1|mux_h1.c:2087] H1 request fully rcvd : [F,EMB] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=29) res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1" - h1c=0x7341fce5e1f0(0x00000000) conn=0x7341fce5e100(0x80000300) h1s=0x7341fce5e2e0(0x00000010) sd=0x7341fd80ec90(0x00500021) ibuf=280@0x7341fce8a530+48/16384 obuf=0@0+0/0
	htx=0x7341fce7ca00(size=16336,data=261,used=8,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=7,tail_addr=261,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_REQ_SL - size=114 - addr=0 POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1
		[1] type=HTX_BLK_HDR - size=18 - addr=114 host: 127.0.0.1:2375
		[2] type=HTX_BLK_HDR - size=38 - addr=132 user-agent: Docker-Client/27.1.1 (linux)
		[3] type=HTX_BLK_HDR - size=16 - addr=170 content-length: 29
		[4] type=HTX_BLK_HDR - size=17 - addr=186 connection: Upgrade
		[5] type=HTX_BLK_HDR - size=28 - addr=203 content-type: application/json
		[6] type=HTX_BLK_EOH - size=1 - addr=231 <empty>
		[7] type=HTX_BLK_DATA - size=29 - addr=232

# snip

[02|h1|1|mux_h1.c:2291] sending request headers : [B,RUN] [MSG_RQBEFORE, MSG_RPBEFORE] - req=(.fl=0x00001400 .curr_len=0 .body_len=0) res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1" - h1c=0x7341fce5e4c0(0x80000000) conn=0x7341fce5e3d0(0x00000300) h1s=0x7341fce5e5b0(0x00100010) sd=0x7341fd80ec10(0x40400001) sc=0x7341fd515f00(0x00001c31) ibuf=0@0+0/0 obuf=0@0x7341fce85c20+0/16384
	htx=0x7341fce7ca00(size=16336,data=261,used=8,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=7,tail_addr=261,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_REQ_SL - size=114 - addr=0 POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1
		[1] type=HTX_BLK_HDR - size=18 - addr=114 host: 127.0.0.1:2375
		[2] type=HTX_BLK_HDR - size=38 - addr=132 user-agent: Docker-Client/27.1.1 (linux)
		[3] type=HTX_BLK_HDR - size=16 - addr=170 content-length: 29
		[4] type=HTX_BLK_HDR - size=17 - addr=186 connection: Upgrade
		[5] type=HTX_BLK_HDR - size=28 - addr=203 content-type: application/json
		[6] type=HTX_BLK_EOH - size=1 - addr=231 <empty>
		[7] type=HTX_BLK_DATA - size=29 - addr=232

# snip

[02|h1|1|mux_h1.c:2376] sending response headers : [F,RUN] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=29) res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "HTTP/1.1 200 OK" - h1c=0x7341fce5e1f0(0x00000000) conn=0x7341fce5e100(0x80000300) h1s=0x7341fce5e2e0(0x00100010) sd=0x7341fd80ec90(0x50404001) sc=0x7341fd515ea0(0x00001422) ibuf=0@0+0/0 obuf=0@0x7341fce8a530+0/16384
	htx=0x7341fce7ca00(size=16336,data=156,used=7,wrap=NO,flags=0x00000000,extra=18446744073709551615,first=0,head=0,tail=6,tail_addr=156,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_RES_SL - size=33 - addr=0 HTTP/1.1 200 OK
		[1] type=HTX_BLK_HDR - size=45 - addr=33 content-type: application/vnd.docker.raw-stream
		[2] type=HTX_BLK_HDR - size=15 - addr=78 api-version: 1.46
		[3] type=HTX_BLK_HDR - size=24 - addr=93 docker-experimental: false
		[4] type=HTX_BLK_HDR - size=11 - addr=117 ostype: linux
		[5] type=HTX_BLK_HDR - size=27 - addr=128 server: Docker/27.1.1 (linux)
		[6] type=HTX_BLK_EOH - size=1 - addr=155 <empty>

```

## How to reproduce

haproxy.cfg:

```auto
global
    log stdout len 4096 format raw daemon debug
    stats socket ipv4@:10000 level admin
    maxconn 100

defaults
    log global
    timeout connect 10s
    timeout client 10m
    timeout server 10m
    errorfile 400 /usr/local/etc/haproxy/errors/400.http
    errorfile 403 /usr/local/etc/haproxy/errors/403.http
    errorfile 408 /usr/local/etc/haproxy/errors/408.http
    errorfile 500 /usr/local/etc/haproxy/errors/500.http
    errorfile 502 /usr/local/etc/haproxy/errors/502.http
    errorfile 503 /usr/local/etc/haproxy/errors/503.http
    errorfile 504 /usr/local/etc/haproxy/errors/504.http

frontend dockerhttp
    bind :2375
    mode http
    option httplog
    default_backend dockerhttp

backend dockerhttp
    mode http
    server dockersocket /var/run/docker.sock

```

As an user that has access to `/var/run/docker.sock` (upgrade version as needed and make sure Docker daemon is up)

```auto
 docker run --rm --name haproxy \
    --user root --privileged \
    -v ./haproxy.cfg:/haproxy.cfg \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p 127.0.0.1:2375:2375 \
    -p 127.0.0.1:10000:10000 \
    haproxy:2.2-alpine haproxy -f /haproxy.cfg

```

In another terminal:

```auto
export DOCKER_HOST=tcp://127.0.0.1:2375
docker exec haproxy ls

```

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 12, 2024, 7:10pm UTC](https://discourse.haproxy.org/t/http-connection-upgrade-to-tcp-seems-broken-since-2-4/10115/2 "2024-08-12T19:10:50Z")

</div>

The affecting change is probably (based on the code that you found) this one:

> <https://github.com/haproxy/haproxy/commit/18ee5c3eb0c8afc18627a0d9a096aabb4c61e34f>
>
> If a request is identified as a WebSocket handshake, it must contains a
> websocke…t key header or else it can be reject, following the rfc6455.
> 
> A new flag H1\_MF\_UPG\_WEBSOCKET is set on such messages. For the request
> te be identified as a WebSocket handshake, it must contains the headers:
> Connection: upgrade
> Upgrade: websocket
> 
> This commit is a compagnon of
> "MEDIUM: h1: generate WebSocket key on response if needed" and
> "MEDIUM: h1: add a WebSocket key on handshake if needed".
> 
> Indeed, it ensures that a WebSocket key is added only from a http/2 side
> and not for a http/1 bogus peer.

I would suggest you file a bug at Github:

> **[Issues · haproxy/haproxy](https://github.com/haproxy/haproxy/issues)**
>
> HAProxy Load Balancer's development branch (mirror of git.haproxy.org) - Issues · haproxy/haproxy

Do make sure to emphasize the docker docs regarding the connection hijacking.

edit: this has been filed as:

> <https://github.com/haproxy/haproxy/issues/2684>
>
> \### Detailed Description of the Problem
> 
> \*\*Summary:\*\*
> 
> \[Upgrading\](https://d…eveloper.mozilla.org/en-US/docs/Web/HTTP/Headers/Upgrade) a HTTP connection to TCP no longer work for versions greater than v2.3. This is NOT about upgrading a connection in a TCP frontend using switch-mode http, nor is it about WebSocket. It’s about the Connection: Upgrade and Upgrade: tcp HTTP headers.
> 
> The proxied protocol is the Docker Engine API. It’s mostly HTTP REST but for some endpoints, the connection is “hijacked”: upgraded to TCP, then used for streaming a proprietary binary format in both directions. Hijacking is detailed \[here in Docker doc\](https://docs.docker.com/engine/api/v1.46/#tag/Container/operation/ContainerAttach). An example of problematic call is the POST to /v1.46/exec/{id}/start.
> 
> With HAProxy 3.0: proxy returns HTTP 200 (instead of expected 101) and connection upgrade fails.
> 
> \*\*Proxy logs:\*\*
> 
> \`\`\`
> dockerhttp dockerhttp/dockersocket 0/0/0/0/20 200 156 - - ---- 1/1/0/0/0 0/0 "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1"
> \`\`\`
> 
> \*\*Wireshark capture, between client and proxy:\*\*
> 
> \`\`\`
> Hypertext Transfer Protocol
> POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1\\r\\n
> Connection: Upgrade\\r\\n
> Content-Type: application/json\\r\\n
> Upgrade: tcp\\r\\n
> 
> Hypertext Transfer Protocol
> HTTP/1.1 200 OK\\r\\n
> content-type: application/vnd.docker.raw-stream\\r\\n
> connection: close\\r\\n
> 
> \# No TCP packet with binary data
> \`\`\`
> 
> \*\*Traces:\*\*
> 
> \`\`\`
> \[02|h1|1|mux\_h1.c:2087\] H1 request fully rcvd : \[F,EMB\] \[MSG\_DONE, MSG\_RPBEFORE\] - req=(.fl=0x00001511 .curr\_len=0 .body\_len=29) res=(.fl=0x00001404 .curr\_len=0 .body\_len=0) - "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1" - h1c=0x7341fce5e1f0(0x00000000) conn=0x7341fce5e100(0x80000300) h1s=0x7341fce5e2e0(0x00000010) sd=0x7341fd80ec90(0x00500021) ibuf=280@0x7341fce8a530+48/16384 obuf=0@0+0/0
> htx=0x7341fce7ca00(size=16336,data=261,used=8,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=7,tail\_addr=261,head\_addr=0,end\_addr=0)
> \[0\] type=HTX\_BLK\_REQ\_SL - size=114 - addr=0 POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1
> \[1\] type=HTX\_BLK\_HDR - size=18 - addr=114 host: 127.0.0.1:2375
> \[2\] type=HTX\_BLK\_HDR - size=38 - addr=132 user-agent: Docker-Client/27.1.1 (linux)
> \[3\] type=HTX\_BLK\_HDR - size=16 - addr=170 content-length: 29
> \[4\] type=HTX\_BLK\_HDR - size=17 - addr=186 connection: Upgrade
> \[5\] type=HTX\_BLK\_HDR - size=28 - addr=203 content-type: application/json
> \[6\] type=HTX\_BLK\_EOH - size=1 - addr=231 \<empty\>
> \[7\] type=HTX\_BLK\_DATA - size=29 - addr=232
> 
> 
> \# snip
> 
> \[02|h1|1|mux\_h1.c:2291\] sending request headers : \[B,RUN\] \[MSG\_RQBEFORE, MSG\_RPBEFORE\] - req=(.fl=0x00001400 .curr\_len=0 .body\_len=0) res=(.fl=0x00001404 .curr\_len=0 .body\_len=0) - "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1" - h1c=0x7341fce5e4c0(0x80000000) conn=0x7341fce5e3d0(0x00000300) h1s=0x7341fce5e5b0(0x00100010) sd=0x7341fd80ec10(0x40400001) sc=0x7341fd515f00(0x00001c31) ibuf=0@0+0/0 obuf=0@0x7341fce85c20+0/16384
> htx=0x7341fce7ca00(size=16336,data=261,used=8,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=7,tail\_addr=261,head\_addr=0,end\_addr=0)
> \[0\] type=HTX\_BLK\_REQ\_SL - size=114 - addr=0 POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1
> \[1\] type=HTX\_BLK\_HDR - size=18 - addr=114 host: 127.0.0.1:2375
> \[2\] type=HTX\_BLK\_HDR - size=38 - addr=132 user-agent: Docker-Client/27.1.1 (linux)
> \[3\] type=HTX\_BLK\_HDR - size=16 - addr=170 content-length: 29
> \[4\] type=HTX\_BLK\_HDR - size=17 - addr=186 connection: Upgrade
> \[5\] type=HTX\_BLK\_HDR - size=28 - addr=203 content-type: application/json
> \[6\] type=HTX\_BLK\_EOH - size=1 - addr=231 \<empty\>
> \[7\] type=HTX\_BLK\_DATA - size=29 - addr=232
> 
> \# snip
> 
> \[02|h1|1|mux\_h1.c:2376\] sending response headers : \[F,RUN\] \[MSG\_DONE, MSG\_RPBEFORE\] - req=(.fl=0x00001511 .curr\_len=0 .body\_len=29) res=(.fl=0x00001404 .curr\_len=0 .body\_len=0) - "HTTP/1.1 200 OK" - h1c=0x7341fce5e1f0(0x00000000) conn=0x7341fce5e100(0x80000300) h1s=0x7341fce5e2e0(0x00100010) sd=0x7341fd80ec90(0x50404001) sc=0x7341fd515ea0(0x00001422) ibuf=0@0+0/0 obuf=0@0x7341fce8a530+0/16384
> htx=0x7341fce7ca00(size=16336,data=156,used=7,wrap=NO,flags=0x00000000,extra=18446744073709551615,first=0,head=0,tail=6,tail\_addr=156,head\_addr=0,end\_addr=0)
> \[0\] type=HTX\_BLK\_RES\_SL - size=33 - addr=0 HTTP/1.1 200 OK
> \[1\] type=HTX\_BLK\_HDR - size=45 - addr=33 content-type: application/vnd.docker.raw-stream
> \[2\] type=HTX\_BLK\_HDR - size=15 - addr=78 api-version: 1.46
> \[3\] type=HTX\_BLK\_HDR - size=24 - addr=93 docker-experimental: false
> \[4\] type=HTX\_BLK\_HDR - size=11 - addr=117 ostype: linux
> \[5\] type=HTX\_BLK\_HDR - size=27 - addr=128 server: Docker/27.1.1 (linux)
> \[6\] type=HTX\_BLK\_EOH - size=1 - addr=155 \<empty\>
> \`\`\`
> 
> \*\*More context:\*\*
> 
> The \[docker-socket-proxy\](https://github.com/Tecnativa/docker-socket-proxy) project uses HAproxy to filter requests to the Docker daemon using ACLs (like \[here\](https://github.com/Tecnativa/docker-socket-proxy/blob/e37525d1577226e518a93adbdcedc2b2fd377c58/haproxy.cfg#L49\_)). It worked well with HAproxy v2.2 and \[broke\](https://github.com/Tecnativa/docker-socket-proxy/issues/132) when I updated HAproxy to v3.0.
> 
> \[Post in the forum\](https://discourse.haproxy.org/t/http-connection-upgrade-to-tcp-seems-broken-since-2-4/10115/1)
> 
> \### Expected Behavior
> 
> With HAProxy 2.2: client and server could upgrade the connection and converse bidirectionally using a proprietary binary format on the connection.
> 
> \*\*Proxy Logs:\*\* 
> 
> \`\`\`
> dockerhttp dockerhttp/dockersocket 0/0/0/0/29 101 316 - - ---- 2/2/0/0/0 0/0 "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1"
> \`\`\`
> 
> \*\*Wireshark capture, between client and proxy:\*\*
> 
> \`\`\`
> Hypertext Transfer Protocol
> POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1\\r\\n
> Connection: Upgrade\\r\\n
> Content-Type: application/json\\r\\n
> Upgrade: tcp\\r\\n
> 
> Hypertext Transfer Protocol
> HTTP/1.1 101 UPGRADED\\r\\n
> content-type: application/vnd.docker.multiplexed-stream\\r\\n
> connection: Upgrade\\r\\n
> upgrade: tcp\\r\\n
> 
> Transmission Control Protocol
> # A paquet with binary data that respect the docker.multiplexed-stream content-type
> \`\`\`
> 
> \*\*Traces:\*\* 
> 
> \`\`\`
> \[04|h1|0|mux\_h1.c:1534\] H1 request fully rcvd : \[F\] \[MSG\_DONE, MSG\_RPBEFORE\] - "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1" - h1c=0x7551d2ecf090(0x00000000) h1s=0x7551d2ecf7b0(0x00000410) ibuf=280@0x7551d2a02f00+48/16384 obuf=0@0+0/0
> htx=0x7551d2a07810(size=16336,data=276,used=10,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=9,tail\_addr=276,head\_addr=0,end\_addr=0)
> \[0\] type=HTX\_BLK\_REQ\_SL - size=118 - addr=0 POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1
> \[1\] type=HTX\_BLK\_HDR - size=18 - addr=118 host: 127.0.0.1:2375
> \[2\] type=HTX\_BLK\_HDR - size=38 - addr=136 user-agent: Docker-Client/27.1.1 (linux)
> \[3\] type=HTX\_BLK\_HDR - size=16 - addr=174 content-length: 29
> \[4\] type=HTX\_BLK\_HDR - size=17 - addr=190 connection: Upgrade
> \[5\] type=HTX\_BLK\_HDR - size=28 - addr=207 content-type: application/json
> \[6\] type=HTX\_BLK\_HDR - size=10 - addr=235 upgrade: tcp
> \[7\] type=HTX\_BLK\_EOH - size=1 - addr=245 \<empty\>
> \[8\] type=HTX\_BLK\_DATA - size=29 - addr=246
> \[9\] type=HTX\_BLK\_EOM - size=1 - addr=275 \<empty\>
> 
> \# snip
> 
> \[04|h1|0|mux\_h1.c:1739\] sending request headers : \[B\] \[MSG\_RQBEFORE, MSG\_RPBEFORE\] - "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1" - h1c=0x7551d2ecf680(0x00000000) h1s=0x7551d2ecf9a0(0x00000010) ibuf=0@0+0/0 obuf=0@0x7551d2a10a50+0/16384
> htx=0x7551d2a07810(size=16336,data=276,used=10,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=9,tail\_addr=276,head\_addr=0,end\_addr=0)
> \[0\] type=HTX\_BLK\_REQ\_SL - size=118 - addr=0 POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1
> \[1\] type=HTX\_BLK\_HDR - size=18 - addr=118 host: 127.0.0.1:2375
> \[2\] type=HTX\_BLK\_HDR - size=38 - addr=136 user-agent: Docker-Client/27.1.1 (linux)
> \[3\] type=HTX\_BLK\_HDR - size=16 - addr=174 content-length: 29
> \[4\] type=HTX\_BLK\_HDR - size=17 - addr=190 connection: Upgrade
> \[5\] type=HTX\_BLK\_HDR - size=28 - addr=207 content-type: application/json
> \[6\] type=HTX\_BLK\_HDR - size=10 - addr=235 upgrade: tcp
> \[7\] type=HTX\_BLK\_EOH - size=1 - addr=245 \<empty\>
> \[8\] type=HTX\_BLK\_DATA - size=29 - addr=246
> \[9\] type=HTX\_BLK\_EOM - size=1 - addr=275 \<empty\>
> 
> \# snip
> 
> \[04|h1|0|mux\_h1.c:1501\] rcvd H1 response headers : \[B\] \[MSG\_TUNNEL, MSG\_TUNNEL\] - VAL=210 - "HTTP/1.1 101 UPGRADED" - h1c=0x7551d2ecf680(0x00000000) h1s=0x7551d2ecf9a0(0x00004020) ibuf=210@0x7551d2a02f00+0/16384 obuf=0@0+0/0
> htx=0x7551d2a07810(size=16336,data=201,used=9,wrap=NO,flags=0x00000000,extra=18446744073709551615,first=0,head=0,tail=8,tail\_addr=201,head\_addr=0,end\_addr=0)
> \[0\] type=HTX\_BLK\_RES\_SL - size=43 - addr=0 HTTP/1.1 101 UPGRADED
> \[1\] type=HTX\_BLK\_HDR - size=53 - addr=43 content-type: application/vnd.docker.multiplexed-stream
> \[2\] type=HTX\_BLK\_HDR - size=17 - addr=96 connection: Upgrade
> \[3\] type=HTX\_BLK\_HDR - size=10 - addr=113 upgrade: tcp
> \[4\] type=HTX\_BLK\_HDR - size=15 - addr=123 api-version: 1.46
> \[5\] type=HTX\_BLK\_HDR - size=24 - addr=138 docker-experimental: false
> \[6\] type=HTX\_BLK\_HDR - size=11 - addr=162 ostype: linux
> \[7\] type=HTX\_BLK\_HDR - size=27 - addr=173 server: Docker/27.1.1 (linux)
> \[8\] type=HTX\_BLK\_EOH - size=1 - addr=200 \<empty\>
> \`\`\`
> 
> \### Steps to Reproduce the Behavior
> 
> Pre-requisites:
> \- On a computer with the docker daemon started 
> \- As an user that can assess \`/var/run/docker.sock\`
> \- Save config shared below in haproxy.cfg
> 
> In terminal 1:
> 
> \`\`\`
> docker run --rm --name haproxy \\
> --user root --privileged \\
> -v ./haproxy.cfg:/haproxy.cfg \\
> -v /var/run/docker.sock:/var/run/docker.sock \\
> -p 127.0.0.1:2375:2375 \\
> -p 127.0.0.1:10000:10000 \\
> haproxy:3.0-alpine haproxy -f /haproxy.cfg
> \`\`\`
> 
> In terminal 2:
> 
> \`\`\`
> export DOCKER\_HOST=tcp://127.0.0.1:2375
> docker exec haproxy ls
> \`\`\`
> 
> The message \`unable to upgrade to tcp, received 200\` indicates an error
> 
> \### Do you have any idea what may have caused this?
> 
> Looking at traces, the problem seems to originate from the parsing of the HTTP request. The Upgrade header seems ignored and is not passed to the backend, so no upgrade takes place.
> 
> Looking at the code, the \[h1\_parse\_upgrade\_header\](https://github.com/haproxy/haproxy/blob/5590ada4731a1f75004675680b4bdca61fa4c507/src/h1.c#L349) seems to ignore the Upgrade header unless it’s set to websocket.
> 
> In the forum, lukastribus \[pointed\](https://discourse.haproxy.org/t/http-connection-upgrade-to-tcp-seems-broken-since-2-4/10115/2) at \[this commit\](https://github.com/haproxy/haproxy/commit/18ee5c3eb0c8afc18627a0d9a096aabb4c61e34f).
> 
> \### Do you have an idea how to solve the issue?
> 
> \_No response\_
> 
> \### What is your configuration?
> 
> \`\`\`haproxy
> global
> log stdout len 4096 format raw daemon debug
> stats socket ipv4@:10000 level admin
> maxconn 100
> 
> defaults
> log global
> timeout connect 10s
> timeout client 10m
> timeout server 10m
> errorfile 400 /usr/local/etc/haproxy/errors/400.http
> errorfile 403 /usr/local/etc/haproxy/errors/403.http
> errorfile 408 /usr/local/etc/haproxy/errors/408.http
> errorfile 500 /usr/local/etc/haproxy/errors/500.http
> errorfile 502 /usr/local/etc/haproxy/errors/502.http
> errorfile 503 /usr/local/etc/haproxy/errors/503.http
> errorfile 504 /usr/local/etc/haproxy/errors/504.http
> 
> frontend dockerhttp
> bind :2375
> mode http
> option httplog
> default\_backend dockerhttp
> 
> backend dockerhttp
> mode http
> server dockersocket /var/run/docker.sock
> \`\`\`
> 
> 
> \### Output of \`haproxy -vv\`
> 
> \`\`\`plain
> HAProxy version 3.0.3-95a607c 2024/07/11 - https://haproxy.org/
> Status: long-term supported branch - will stop receiving fixes around Q2 2029.
> Known bugs: http://www.haproxy.org/bugs/bugs-3.0.3.html
> Running on: Linux 6.10.4-arch2-1 #1 SMP PREEMPT\_DYNAMIC Sun, 11 Aug 2024 16:19:06 +0000 x86\_64
> Build options :
> TARGET = linux-musl
> CC = cc
> CFLAGS = -O2 -g -fwrapv
> OPTIONS = USE\_GETADDRINFO=1 USE\_OPENSSL=1 USE\_LUA=1 USE\_PROMEX=1 USE\_PCRE2=1 USE\_PCRE2\_JIT=1
> DEBUG =
> 
> Feature list : -51DEGREES +ACCEPT4 -BACKTRACE -CLOSEFROM +CPU\_AFFINITY +CRYPT\_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX\_CAP +LINUX\_SPLICE +LINUX\_TPROXY +LUA +MATH -MEMORY\_PROFILING +NETFILTER +NS -OBSOLETE\_LINKER +OPENSSL -OPENSSL\_AWSLC -OPENSSL\_WOLFSSL -OT -PCRE +PCRE2 +PCRE2\_JIT -PCRE\_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD\_EMULATION -QUIC -QUIC\_OPENSSL\_COMPAT +RT +SHM\_OPEN +SLZ +SSL -STATIC\_PCRE -STATIC\_PCRE2 -SYSTEMD +TFO +THREAD +THREAD\_DUMP +TPROXY -WURFL -ZLIB
> 
> Default settings :
> bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
> 
> Built with multi-threading support (MAX\_TGROUPS=16, MAX\_THREADS=256, default=8).
> Built with OpenSSL version : OpenSSL 3.3.1 4 Jun 2024
> Running on OpenSSL version : OpenSSL 3.3.1 4 Jun 2024
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
> OpenSSL providers loaded : default
> Built with Lua version : Lua 5.4.6
> Built with the Prometheus exporter as a service
> Built with network namespace support.
> Built with libslz for stateless compression.
> Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
> Built with transparent proxy support using: IP\_TRANSPARENT IPV6\_TRANSPARENT IP\_FREEBIND
> Built with PCRE2 version : 10.43 2024-02-16
> PCRE2 library supports JIT : yes
> Encrypted password support via crypt(3): yes
> Built with gcc compiler version 13.2.1 20240309
> 
> Available polling systems :
> epoll : pref=300, test result OK
> poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 3 (3 usable), will use epoll.
> 
> Available multiplexer protocols :
> (protocols marked as \<default\> cannot be specified using 'proto' keyword)
> h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL\_RISK|NO\_UPG
> \<default\> : mode=HTTP side=FE|BE mux=H1 flags=HTX
> h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO\_UPG
> fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL\_RISK|NO\_UPG
> \<default\> : mode=TCP side=FE|BE mux=PASS flags=
> none : mode=TCP side=FE|BE mux=PASS flags=NO\_UPG
> 
> Available services : prometheus-exporter
> Available filters :
> \[BWLIM\] bwlim-in
> \[BWLIM\] bwlim-out
> \[CACHE\] cache
> \[COMP\] compression
> \[FCGI\] fcgi-app
> \[SPOE\] spoe
> \[TRACE\] trace
> \`\`\`
> 
> 
> \### Last Outputs and Backtraces
> 
> \`\`\`plain
> n/a
> \`\`\`
> 
> 
> \### Additional Information
> 
> n/a

---

<div class="post-metadata">

**Author:** ![Rickard\_505](https://avatars.discourse-cdn.com/v4/letter/r/5f9b8f/32.png) [@Rickard\_505](https://discourse.haproxy.org/u/Rickard_505)\
**Post date:** [September 20, 2024, 1:45pm UTC](https://discourse.haproxy.org/t/http-connection-upgrade-to-tcp-seems-broken-since-2-4/10115/3 "2024-09-20T13:45:57Z")

</div>

wow - finally someone else (that knew the right lingo) saw this 🙂