Nextcloud behind haproxy: problem with big files


#1

Hello,

I’m running Nextcloud behind HAproxy 1.5 on a CentOS 7 virtual machine. All seems to work fine except when I try to download some large files. The download starts but randomly the downstream drops to 0Kbps. Some times it drops when it has downloaded 1MB and someother times drops when the download is more than 100MB. I couldn’t find any correlation about when the download drops nor anything in logfiles.
Sometimes the downloads works as expected, I succesfully downloaded files bigger than 2GB. It makes a mess and I don’t know where to look for.

any suggestion?

I post some details about the server:
haproxy.conf
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
tune.ssl.default-dh-param 2048
tune.ssl.cachesize 1000000
daemon
ssl-default-bind-options no-sslv3 no-tls-tickets
ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
stats socket /var/lib/haproxy/stats

defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 30s
timeout queue 1m
timeout connect 30s
timeout client 5000
timeout server 30s
timeout http-keep-alive 30s
timeout check 30s
maxconn 30000

frontend http
bind 0.0.0.0:80
option http-server-close
option forwardfor
option http-keep-alive

#ACLs
acl host_cloud_cmp hdr_beg(host) -i cloud.company.com

use_backend cloud_cmp if host_cloud_cmp

frontend https
option http-keep-alive
http-response set-header Strict-Transport-Security max-age=31536000;\ includeSubdomains;\ preload
http-response set-header X-Frame-Options DENY
http-response set-header X-Content-Type-Options nosniff
bind 0.0.0.0:443 ssl crt /etc/haproxy/wildcard.company.com.pem
mode http
reqadd X-Forwarded-Proto:\ https
#ACLs
acl host_cloud_cmp hdr_beg(host) -i cloud.company.com

use_backend cloud_cmp-https if host_cloud_cmp

backend cloud_cmp
redirect scheme https if !{ ssl_fc }

backend cloud_cmp-https
mode http
balance roundrobin
option http-server-close
option http-pretend-keepalive
option forwardfor header X-Real-IP
cookie NEXTCLOUD insert indirect nocache
stick-table type ip size 200k expire 30m
stick on src
http-response set-header Strict-Transport-Security max-age=16000000;\ includeSubDomains;\ preload;
server NEXTCLOUDSERVER NEXTCLOUDSERVER:80 check cookie NEXTCLOUDSERVER

haproxy -v

HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau

uname -ir

3.10.0-514.16.1.el7.x86_64 x86_64

4CPU, 4GB of RAM

Low CPU Usage
09:20:01 PM CPU %user %nice %system %iowait %steal %idle
09:30:01 PM all 0.21 0.01 0.05 0.00 0.00 99.73
09:40:01 PM all 0.21 0.01 0.05 0.00 0.00 99.72
09:50:02 PM all 0.22 0.01 0.06 0.00 0.00 99.72
10:00:01 PM all 0.21 0.01 0.05 0.00 0.00 99.72
10:10:01 PM all 0.21 0.01 0.06 0.00 0.00 99.72
10:20:01 PM all 0.21 0.01 0.06 0.00 0.00 99.72
10:30:01 PM all 0.21 0.01 0.06 0.00 0.00 99.73
10:40:01 PM all 0.21 0.01 0.06 0.00 0.00 99.72
10:50:01 PM all 0.21 0.01 0.06 0.00 0.00 99.72
11:00:01 PM all 0.21 0.01 0.06 0.00 0.00 99.73
11:10:01 PM all 0.22 0.01 0.06 0.00 0.00 99.72
11:20:01 PM all 0.21 0.01 0.06 0.00 0.00 99.72
11:30:01 PM all 0.21 0.01 0.06 0.00 0.00 99.72
11:40:01 PM all 0.22 0.01 0.06 0.00 0.00 99.72
11:50:01 PM all 0.21 0.01 0.05 0.00 0.00 99.73


#2

You need to pull the log line of the failed request. I understand this is not easy in a high traffic environment, but it is the only way to understand what happens.