cas
December 20, 2016, 5:40pm
1
I have some mysterious problem. There are a lot of valid http requests resulting 400 code. Can’t replay that requests
[20/Dec/2016:12:31:29.058] frontend http-in (#3 ): invalid request#-1 ), server (#-1 ), event #3 #3747 , session flags 0x00000080
00000 GET {NORMAL URL}/ \r\n22gmail.com %22%2C%22landing%22%3A%20%22default%22%2C%22regdservert1.com \r\nsoironport.txfb-ins.com:8080 (Cisco-WSA/8.0.6-119)\r\n
Could someone help me?
cas
December 20, 2016, 5:44pm
2
I’ve disable request validation. Everything is fine now. But “show errors” shows such errors
cas
December 20, 2016, 5:51pm
3
global
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 20000
user haproxy
group haproxy
daemon
defaults
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 1m
timeout queue 5m
timeout connect 10s
timeout client 5m
timeout server 5m
timeout http-keep-alive 10s
timeout check 10s
maxconn 10000
option accept-invalid-http-request
listen stats
maxconn 10
timeout client 100s
timeout server 100s
timeout connect 100s
timeout queue 100s
backend servert1
cas
December 20, 2016, 6:40pm
4
log record (another one)
81.101.104.134:47386 [20/Dec/2016:13:34:32.553] http-in http-in/ -1/-1/-1/-1/0 400 187 - - PR-- 239/238/0/0/0 0/0 “”
[20/Dec/2016:13:34:32.553] frontend http-in (#3 ): invalid request#-1 ), server (#-1 ), event #160 #312029 , session flags 0x00000080
00000 GET {NORMAL+URL+WAS+HERE+NORMAL+URL+WAS+HERE+NORMAL+URL+WAS+HERE+NOR}servert1.com \r\n/ ;q=0.il.com %22%2C%22landing%22%3A%20%22default_emtest%22%2C%22regdate%22%3A
That’s expected behavior.
It allows troubleshooting the issue, while the workaround is in place.
The issue seems to be with your huge cookies, can you try bumping tune.bufsize [1] and see what happens?
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#3.2-tune.bufsize
cas
December 21, 2016, 12:19pm
7
Disabled splicing for testing purposes. Didn`t help
cas
December 21, 2016, 12:24pm
8
Os Centos 7 with all updates
HA-Proxy version 1.7.1 2016/12/13willy@haproxy.org
Build options :
Default settings :
Encrypted password support via crypt(3): yes
Available polling systems :
Available filters :
cas
December 21, 2016, 12:26pm
9
After disabling request some requests are working, other - not. 400 Error Code.
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#3.2-tune.bufsize
didn’t help
cas
December 21, 2016, 2:18pm
10
I’ve tried to replay some of that requests using yandex tank (load test). It runs on 1000 rps without any 400 code error. Server can’t run faster. So may be some symbols are hidden and there is request problem.
Yandex very likely doesn’t produce those huge cookie headers.
Can you try tune.bufsize 32768 but without any maxrewrite configuration?
edit: Also what you posted here may be incomplete. Can you confirm the output of affected lines is as is here in discourse as it was on your command line? For example, the line that starts with “01430+” (containing byte 1472), do it look exactly the same? Because it seems we are missing characters.
cas
December 22, 2016, 4:28am
13
Can you try tune.bufsize 32768 but without any maxrewrite configuration?
Doesn’t help problem still here. You can see error using that link
32768 without maxrewrite,
Total events captured on [22/Dec/2016:05:26:27.506] : 4
[22/Dec/2016:05:25:08.535] frontend http-in (#3): invalid request
backend <NONE> (#-1), server <NONE> (#-1), event #3
src 74.85.204.128:4976, session #4909, session flags 0x00000080
HTTP msg state 26, msg flags 0x00000000, tx flags 0x80000000
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 2699 bytes
pending 2699 bytes, wrapping at 32768, error at position 1361:
show original
haproxy
Total events captured on [21/Dec/2016:18:23:11.623] : 1
[21/Dec/2016:18:22:53.546] frontend http-in (#3): invalid request
backend <NONE> (#-1), server <NONE> (#-1), event #0
src 99.186.17.71:50566, session #461, session flags 0x00000080
HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 3145 bytes
pending 3145 bytes, wrapping at 65536, error at position 2921:
show original
About requests Yes, i’ve modified them. Please use previous link to see them
Hi,
I looked through your traces, and I cannot find anything wrong. Replaying the same exact request in local repro works fine, I’m unable to reproduce this problem.
Can you provide the complete configuration and can you tell if this happens only under load only or does it also happen on on an “empty” server?
I believe we need to move this discussion to the mailing list, as I’m unable to pin point the root cause.
Can you send the github “gist” link along with a short description of what we have tried here to the mailing list at:
You don’t have to subscribe, just send an email to this address.
cas
December 27, 2016, 12:34pm
15
can’t send
Diagnostic-Code: smtp; 451 Temporary local problem - please try later
is it working?
It works fine, I just send a test email:
https://www.mail-archive.com/haproxy@formilux.org/msg24455.html
Double check the email address:haproxy@formilux.org
If it still doesn’t work, can you provide the full error message please.
cas
December 28, 2016, 10:39am
17
I’m sorry , I’ve made a mistake -formilux.dot.org
cas
January 8, 2017, 12:02pm
18
Bug is found. Fix soon be ready.
cas
January 10, 2017, 7:40am
19