# Unable to receive gRPC error during a response streaming call using HAProxy in HTTP mode

**URL:** <https://discourse.haproxy.org/t/unable-to-receive-grpc-error-during-a-response-streaming-call-using-haproxy-in-http-mode/6102>\
**Category:** Help!\
**Created:** [January 12, 2021, 3:45pm UTC](https://discourse.haproxy.org/t/unable-to-receive-grpc-error-during-a-response-streaming-call-using-haproxy-in-http-mode/6102 "2021-01-12T15:45:47Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mgetka](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.haproxy.org/mgetka/32/1346_2.png) [@mgetka](https://discourse.haproxy.org/u/mgetka)\
**Post date:** [January 12, 2021, 3:45pm UTC](https://discourse.haproxy.org/t/unable-to-receive-grpc-error-during-a-response-streaming-call-using-haproxy-in-http-mode/6102/1 "2021-01-12T15:45:47Z")

</div>

I’m working on a gRPC application that will be served behind HAProxy using the http mode. If the server application immediately (ie. before sending any responses) aborts the response streaming call with specific error, then the client application will receive `CANCELLED` error instead of the one that was sent. The error details will be _Received RST\_STREAM with error code 8_. I’m working with HAProxy 2.3.2 and grpc 1.34.0.

For each of such requests, HAProxy log entry have `SD--` flags set in the _session state at disconnection_ field, eg.

```auto
<134>Jan 9 18:09:39 1a8328663d74 haproxy[8]: 172.28.0.4:41698 [09/Jan/2021:18:09:39.346] grpc-server-fe grpc-server-fe/grpc-server-be 0/0/0/-1/+0 -1 +115 - - SD-- 1/1/0/0/0 0/0 "POST http://proxy:6000/service.Service/StreamStream HTTP/2.0"

```

In the HAProxy documentation those flags are defined as follows:

- S: the TCP session was unexpectedly aborted by the server, or the server explicitly refused it.
- D: the session was in the DATA phase.

Additionally:

> **SD** The connection to the server died with an error during the data transfer. This usually means that haproxy has received an RST from the server or an ICMP message from an intermediate equipment while exchanging data with the server. This can be caused by a server crash or by a network issue on an intermediate equipment.

Knowing that, and having in mind that the HTTP connection was opened with an intention of data streaming, as a workaround I have tried sending an empty gRPC message before raising the error. The solution have helped partially - the error codes could have been received by the client for the most requests, but the issue still happened from time to time.

As a next step I have inspected the network traffic using wireshark. Following is a trace of the HTTP response served by the gRPC server on an event of immediate call abortion:

```auto
HyperText Transfer Protocol 2
    Stream: SETTINGS, Stream ID: 0, Length 0
        Length: 0
        Type: SETTINGS (4)
        Flags: 0x01
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0000 = Stream Identifier: 0
    Stream: HEADERS, Stream ID: 1, Length 88, 200 OK
        Length: 88
        Type: HEADERS (1)
        Flags: 0x05
            .... ...1 = End Stream: True
            .... .1.. = End Headers: True
            .... 0... = Padded: False
            ..0. .... = Priority: False
            00.0 ..0. = Unused: 0x00
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0001 = Stream Identifier: 1
        [Pad Length: 0]
        Header Block Fragment: 88400c636f6e74656e742d74797065106170706c69636174...
        [Header Length: 120]
        [Header Count: 4]
        Header: :status: 200 OK
        Header: content-type: application/grpc
        Header: grpc-status: 7
        Header: grpc-message: Details sent by the server
    Stream: RST_STREAM, Stream ID: 1, Length 4
        Length: 4
        Type: RST_STREAM (3)
        Flags: 0x00
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0001 = Stream Identifier: 1
        Error: NO_ERROR (0)

```

So the server sends response headers with details on error with `End Stream` and `End Headers` flags set. And then closes the stream with `NO_ERROR` code. According to the answer provided in [_should gRPC server-side half-closing implicitly terminate the client?_](https://stackoverflow.com/questions/55511528/should-grpc-server-side-half-closing-implicitly-terminate-the-client) Stack Overflow question, everything is ok at this stage. I have also briefly reviewed the [RFC 7540](https://tools.ietf.org/html/rfc7540) and couldn’t find anything that is off in the terms of HTTP/2 protocol.

Quoted gRPC server HTTP response is followed by HAProxy originating TCP `ACK`, and next, HAProxy dispatches its response to the client.

```auto
HyperText Transfer Protocol 2
    Stream: HEADERS, Stream ID: 1, Length 75, 200 OK
        Length: 75
        Type: HEADERS (1)
        Flags: 0x05
            .... ...1 = End Stream: True
            .... .1.. = End Headers: True
            .... 0... = Padded: False
            ..0. .... = Priority: False
            00.0 ..0. = Unused: 0x00
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0001 = Stream Identifier: 1
        [Pad Length: 0]
        Header Block Fragment: 885f106170706c69636174696f6e2f67727063000b677270...
        [Header Length: 120]
        [Header Count: 4]
        Header: :status: 200 OK
        Header: content-type: application/grpc
        Header: grpc-status: 7
        Header: grpc-message: Details sent by the server
    Stream: RST_STREAM, Stream ID: 1, Length 4
        Length: 4
        Type: RST_STREAM (3)
        Flags: 0x00
            0000 0000 = Unused: 0x00
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0001 = Stream Identifier: 1
        Error: CANCEL (8)

```

It can be seen that the flags and all the contents of the `HEADERS` frame are in place, so the error details are passed to the client, but the code of `RST_STREAM` has changed to `CANCEL`. In effect the client recieves all the expected data, but after that it receives unexpected `RST_STREAM(CANCEL)` that is being mapped on gRPC `CANCELLED` error, as specified in the [gRPC documentation](https://github.com/grpc/grpc/blob/86f93801e5d452cff40d6958d21a8eb211da0654/doc/PROTOCOL-HTTP2.md#http2-transport-mapping).

In the course of further investigation, I have referred to HAProxy source code. I have discovered that the code is set in the [`h2_do_shutr` function of `mux_h2.c`](https://github.com/haproxy/haproxy/blob/master/src/mux_h2.c#L4236) (experiments with custom HAProxy builds have proven that it is indeed this place). The involved code branch has the following comment:

> a final response was already provided, we don’t want this stream anymore. This may happen when the server responds before the end of an upload and closes quickly (redirect, deny, …)

So those are the details I have managed to gather on the issue. I’m not entirely sure whether the problem lies in the gRPC core (being too messy in the terms of HTTP2 streams handling) or HAProxy (being too careless while rewriting `RST_STREAM` codes).

The final question is, how can I tune the configuration of HAProxy and gRPC core server to work correctly on an event of immediate call abortion? At the moment, to overcome the issue, I have prepared custom HAProxy build that responds with `RST_STREAM(NO_ERROR)` on the described circumstances, but this solution feels rather like technology debt.

The minimal HAProxy configuration that reproduces the issue is as follows:

```auto
global
    log stdout local0

listen grpc-server-fe
    bind *:6000 proto h2

    mode http
    log global
    option logasap
    option httplog

    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

    server grpc-server-be server:6000 proto h2

```

I have prepared a [repository with minimal example](https://github.com/mgetka/haproxy-grpc-issue) containing simple python client & server. It also contains `docker-compose` with network environment including configured HAProxy.

---

<div class="post-metadata">

**Author:** ![SunSparc](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.haproxy.org/sunsparc/32/1609_2.png) [@SunSparc](https://discourse.haproxy.org/u/SunSparc)\
**Post date:** [January 6, 2022, 9:43pm UTC](https://discourse.haproxy.org/t/unable-to-receive-grpc-error-during-a-response-streaming-call-using-haproxy-in-http-mode/6102/2 "2022-01-06T21:43:11Z")

</div>

@mgetka: Excellent description of the problem and good job with all the research. I am running into this problem as well on a project that I am working on. I am wondering if there was any resolution yet.

---

<div class="post-metadata">

**Author:** ![mgetka](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.haproxy.org/mgetka/32/1346_2.png) [@mgetka](https://discourse.haproxy.org/u/mgetka)\
**Post date:** [March 1, 2022, 6:30am UTC](https://discourse.haproxy.org/t/unable-to-receive-grpc-error-during-a-response-streaming-call-using-haproxy-in-http-mode/6102/3 "2022-03-01T06:30:47Z")

</div>

As a follow up, for anybody who ends up in this topic looking for answers. The issue is being tracked in this github issue thread:

> <https://github.com/haproxy/haproxy/issues/1219>
>
> \<!--
> 
> Welcome! - We kindly ask that you:
> 
> 1. Fill out the issue template b…elow - not doing so needs a good reason.
> 2. Use the forum or the mailing list if you have a question rather than a bug or feature request.
> 
> The forum is at: https://discourse.haproxy.org/
> 
> The mailing list (no need to subscribe) is: haproxy@formilux.org
> Subscribe to the list: haproxy+subscribe@formilux.org
> Unsubscribe from the list: haproxy+unsubscribe@formilux.org
> 
> Forum and mailing list are correct places for questions about HAProxy or general suggestions
> and topics, e.g. usage or documentation questions! This issue tracker is for tracking bugs and
> feature requests directly relating to the development of the software itself.
> 
> Thanks for understanding, and for contributing to the project!
> 
> \--\>
> 
> 
> \## Detailed description of the problem
> 
> If the gRPC server application (served from behind HAProxy) immediately (ie. before sending any responses) aborts the response streaming call with specific error, then the client application will receive \`CANCELLED\` error instead of the one that was sent. The error details will be \_Received RST\_STREAM with error code 8\_.
> 
> \## Expected behavior
> 
> \<!--
> Explain why you consider the described behavior (above) to be wrong. What did you expect instead?
> 
> Most likely this is a mismatch between HAProxy's documentation and HAProxy's behavior.
> \--\>
> 
> The error data produced by the gRPC server reaches the client unchanged in the terms of gRPC protocol..
> 
> \## Steps to reproduce the behavior
> 
> \<!--
> The more time you spend describing an easy way to reproduce the behavior (if
> this is possible), the easier it is for the project developers to fix it!
> \--\>
> 
> I have prepared a \[repository with minimal example\](https://github.com/mgetka/haproxy-grpc-issue) containing simple python client & server. It also contains docker-compose with network environment including configured HAProxy.
> 
> \## Do you have any idea what may have caused this?
> 
> To terminate the call gRPC server sends \`RST\_STREAM\` with error code \`NO\_ERROR\`. The client application expects the error code to be like that. HAProxy rewrite the error code to \`CANCEL\` which breaks the client logic. The rewriting is done in \`h2\_do\_shutr\` function of \`mux\_h2.c\`.
> 
> \## What is your configuration?
> 
> \<!--
> Include as much configuration as possible, including global and default sections.
> Replace confidential data like domain names and IP addresses.
> 
> Put it in code blocks \`\`\`:
> \--\>
> 
> \`\`\`
> global
> log stdout local0
> 
> listen grpc-server-fe
> bind \*:6000 proto h2
> 
> mode http
> log global
> option logasap
> option httplog
> 
> timeout connect 5000ms
> timeout client 50000ms
> timeout server 50000ms
> 
> server grpc-server-be server:6000 proto h2
> \`\`\`
> 
> \## Output of \`haproxy -vv\` and \`uname -a\`
> 
> 
> 
> \`\`\`
> \# haproxy -vv
> HA-Proxy version 2.3.2-d522db7 2020/11/28 - https://haproxy.org/
> Status: stable branch - will stop receiving fixes around Q1 2022.
> Known bugs: http://www.haproxy.org/bugs/bugs-2.3.2.html
> Running on: Linux 5.4.0-70-generic #78~18.04.1-Ubuntu SMP Sat Mar 20 14:10:07 UTC 2021 x86\_64
> Build options :
> TARGET = linux-glibc
> CPU = generic
> CC = cc
> CFLAGS = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
> OPTIONS = USE\_PCRE2=1 USE\_PCRE2\_JIT=1 USE\_GETADDRINFO=1 USE\_OPENSSL=1 USE\_LUA=1 USE\_ZLIB=1
> DEBUG = 
> 
> Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE\_JIT +PCRE2 +PCRE2\_JIT +POLL -PRIVATE\_CACHE +THREAD -PTHREAD\_PSHARED +BACKTRACE -STATIC\_PCRE -STATIC\_PCRE2 +TPROXY +LINUX\_TPROXY +LINUX\_SPLICE +LIBCRYPT +CRYPT\_H +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU\_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE\_LINKER +PRCTL +THREAD\_DUMP -EVPORTS
> 
> Default settings :
> bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
> 
> Built with multi-threading support (MAX\_THREADS=64, default=8).
> Built with OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
> Running on OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
> Built with Lua version : Lua 5.3.3
> Built with network namespace support.
> Built with the Prometheus exporter as a service
> Built with zlib version : 1.2.11
> Running on zlib version : 1.2.11
> Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
> Built with transparent proxy support using: IP\_TRANSPARENT IPV6\_TRANSPARENT IP\_FREEBIND
> Built with PCRE2 version : 10.32 2018-09-10
> PCRE2 library supports JIT : yes
> Encrypted password support via crypt(3): yes
> Built with gcc compiler version 8.3.0
> 
> Available polling systems :
> epoll : pref=300, test result OK
> poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 3 (3 usable), will use epoll.
> 
> Available multiplexer protocols :
> (protocols marked as \<default\> cannot be specified using 'proto' keyword)
> h2 : mode=HTTP side=FE|BE mux=H2
> fcgi : mode=HTTP side=BE mux=FCGI
> \<default\> : mode=HTTP side=FE|BE mux=H1
> \<default\> : mode=TCP side=FE|BE mux=PASS
> 
> Available services :
> prometheus-exporter
> 
> Available filters :
> \[SPOE\] spoe
> \[CACHE\] cache
> \[FCGI\] fcgi-app
> \[COMP\] compression
> \[TRACE\] trace
> 
> \# uname -a
> Linux 1fccc9cf174c 5.4.0-70-generic #78~18.04.1-Ubuntu SMP Sat Mar 20 14:10:07 UTC 2021 x86\_64 GNU/Linux
> \`\`\`
> 
> In my repository containing the minimal issue example I've received report that the issue is also present with HAProxy 2.4 and gRPC 1.37.0.
> 
> \## Additional information (if helpful)
> 
> - \[Forum thread with my insights on the issue\](https://discourse.haproxy.org/t/unable-to-receive-grpc-error-during-a-response-streaming-call-using-haproxy-in-http-mode/6102).
> - \[Repository containing the minimal issue example\](https://github.com/mgetka/haproxy-grpc-issue).

At the moment there is no easy fix on the horizon.