# 502 Bad gateway where large files are uploaded

**URL:** <https://discourse.haproxy.org/t/502-bad-gateway-where-large-files-are-uploaded/827>\
**Category:** Help!\
**Created:** [December 7, 2016, 3:32pm UTC](https://discourse.haproxy.org/t/502-bad-gateway-where-large-files-are-uploaded/827 "2016-12-07T15:32:02Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![usergoodvery](https://avatars.discourse-cdn.com/v4/letter/u/cc9497/32.png) [@usergoodvery](https://discourse.haproxy.org/u/usergoodvery)\
**Post date:** [December 7, 2016, 3:32pm UTC](https://discourse.haproxy.org/t/502-bad-gateway-where-large-files-are-uploaded/827/1 "2016-12-07T15:32:02Z")

</div>

Hi,  
I get intermittent failures when uploading largish files (5M) via haproxy. I have a fairly simple setup at this stage with haproxy fronting two servers (custom) with SSL termination.  
The server is gobbling data at high rate, close to 64k per read() invocation and has no trouble keeping up, however, at some point it reads zero before all data is transmitted through from the client, because haproxy has terminated the connection.

I upgraded to the latest stable release, but that did not make a difference:

HA-Proxy version 1.7.0-e59fcdd 2016/11/25  
Copyright 2000-2016 Willy Tarreau [willy@haproxy.org](mailto:willy@haproxy.org)

Build options :  
TARGET = linux2628  
CPU = generic  
CC = gcc  
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement  
OPTIONS = USE\_ZLIB=1 USE\_OPENSSL=1

Default settings :  
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes  
Built with zlib version : 1.2.7  
Running on zlib version : 1.2.7  
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)  
Built with OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013  
Running on OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013  
OpenSSL library supports TLS extensions : yes  
OpenSSL library supports SNI : yes  
OpenSSL library supports prefer-server-ciphers : yes  
Built without PCRE support (using libc’s regex instead)  
Built without Lua support  
Built with transparent proxy support using: IP\_TRANSPARENT IPV6\_TRANSPARENT IP\_FREEBIND

Available polling systems :  
epoll : pref=300, test result OK  
poll : pref=200, test result OK  
select : pref=150, test result OK  
Total: 3 (3 usable), will use epoll.

Available filters :  
[COMP] compression  
[TRACE] trace  
[SPOE] spoe

This is a log entry for a failed upload:

Dec 8 02:01:14 localhost haproxy[1791]: 139.162.56.133:33638 [08/Dec/2016:02:01:12.807] api\_farm~ bk\_api/ufsrv0 0/0/1/-1/1204 502 204 - - SH-- 0/0/0/0/0 0/0 {} {} “PUT /V1/Account/Attachment/8de29bf901bcb895c3a287821 HTTP/1.1”

Config file:

global  
stats socket /var/run/haproxy.sock mode 600 level admin  
stats timeout 2m  
log 127.0.0.1 local0  
maxconn 10024  
tune.ssl.default-dh-param 2048  
tune.maxrewrite 1024  
tune.bufsize 17408 #16k+1k  
debug

defaults  
log global  
mode http  
option http-server-close  
option httplog  
option tcplog  
option dontlognull  
retries 3  
option redispatch  
option contstats  
option forwardfor  
timeout client 60s  
timeout connect 5s  
timeout server 60s  
timeout tunnel 3600s  
timeout http-keep-alive 1s  
timeout http-request 60s  
timeout queue 30s  
timeout tarpit 60s

frontend ufsrvapi\_farm  
bind \*:20080 ssl crt /opt/etc/ssl/xxxxx.pem  
mode http  
option httplog  
capture cookie session len 100  
capture request header X-UFSRVCID len 64  
capture response header X-UFSRVCID len 64  
default\_backend bk\_ws  
stats enable  
stats uri /haproxy\_ufsrvapi  
stats refresh 30s

# stats auth admin:admin

```
    default_backend bk_ufsrvapi

```

backend bk\_ufsrvapi  
mode http  
balance roundrobin

```
    server ufsrv0 127.0.0.1:20081 weight 10 check
    server ufsrv1 127.0.0.1:20082 weight 10 check
```

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [December 8, 2016, 1:54pm UTC](https://discourse.haproxy.org/t/502-bad-gateway-where-large-files-are-uploaded/827/2 "2016-12-08T13:54:59Z")

</div>

> [@usergoodvery](#):
>
> The server is gobbling data at high rate, close to 64k per read() invocation and has no trouble keeping up, however, at some point it reads zero before all data is transmitted through from the client, because haproxy has terminated the connection.

Or is it the other way around? The application is eager to read(), the kernel has no data on this socket yet, and the application erroneously terminates the connection? How does the application wait for data before read()ing?

> [@usergoodvery](#):
>
> Dec 8 02:01:14 localhost haproxy[1791]: 139.162.56.133:33638 [08/Dec/2016:02:01:12.807] api\_farm~ bk\_api/ufsrv0 0/0/1/-1/1204 502 204 - - SH-- 0/0/0/0/0 0/0 {} {} “PUT /V1/Account/Attachment/8de29bf901bcb895c3a287821 HTTP/1.1”

SH translates to this (see doc [1]):

> The server aborted before sending its full HTTP response headers, or  
> it crashed while processing the request. Since a server aborting at  
> this moment is very rare, it would be wise to inspect its logs to  
> control whether it crashed and why. The logged request may indicate a  
> small set of faulty requests, demonstrating bugs in the application.  
> Sometimes this might also be caused by an IDS killing the connection  
> between haproxy and the server.

I strongly suggest to triple check that read() handling in the application.

[1] [HAProxy version 1.7.14 - Configuration Manual](https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#8.5)

---

<div class="post-metadata">

**Author:** ![usergoodvery](https://avatars.discourse-cdn.com/v4/letter/u/cc9497/32.png) [@usergoodvery](https://discourse.haproxy.org/u/usergoodvery)\
**Post date:** [December 9, 2016, 8:43am UTC](https://discourse.haproxy.org/t/502-bad-gateway-where-large-files-are-uploaded/827/3 "2016-12-09T08:43:37Z")

</div>

Thanks for your perspective, especially the link to the document. At this stage I have ruled out haproxy as being the culprit, but I am yet to figure out why I am getting zero reads. The server doesn’t crash or anything just occasionally reads zero prematurely on otherwise non blocking sockets.

Curiously, I did a tcpdump trace and the client (curl in this instance) was not initiating close when the server read zero. I did that whilst the server was under the instrumentation of valgrind to ensure there was no funny memory business affecting the internals of the server. I think I am in for some painful debugging on this one.

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [December 9, 2016, 9:12am UTC](https://discourse.haproxy.org/t/502-bad-gateway-where-large-files-are-uploaded/827/4 "2016-12-09T09:12:45Z")

</div>

You probably want to run your application through strace, that will help understanding what happens on a system call level.