400 error on cookie string

#1

I have some mysterious problem. There are a lot of valid http requests resulting 400 code. Can’t replay that requests
Tested on haproxy version 1.6.10 1.7.0 1.7.1

[20/Dec/2016:12:31:29.058] frontend http-in (#3): invalid request
backend (#-1), server (#-1), event #3
src 72.48.177.194:29536, session #3747, session flags 0x00000080
HTTP msg state 26, msg flags 0x00000000, tx flags 0x80000000
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 3187 bytes
pending 3187 bytes, wrapping at 16384, error at position 1377:

00000 GET {NORMAL URL}
00070+ {URL} HTTP/1.1\r\n
00092 Connection: keep-alive\r\n
00116 Accept: text/html, application/xhtml+xml, /\r\n
00163 Referer: {NORMAL REF
00233+ ERER}/\r\n
00251 Accept-Language: en-US\r\n
00275 User-Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64;
00345+ Trident/6.0)\r\n
00360 Accept-Encoding: gzip, deflate\r\n
00392 DNT: 1\r\n
00400 Cookie: 323d7f8f696ae3e7=53da53acf098e03e; fqz=6597e447-b306-447a-910b
00470+ -3032cfdb6e28; .AspNet.ApplicationCookie=U8GkR7sPHN-1Dgw819lpBrj8X137G
00540+ 6hhJVCDC12WMUD3BfwyMSBsAqSbqo_odWfUX88m1VnKPNMcBq-HTywYBxxPBHRQyqRokKj
00610+ agIa0AE8dSOHRuRZoM_ZZ0-ftBxG2jDI-T2tPXwAu2dc2rkwFh2NxrdqaCH2WnnMWRk4kn
00680+ h70epNm2CcHkA9mzcuBIPPTA2W9ua74Kke9HW6l4i2hBabi7xucesic-q-SNmtRIzXATB4
00750+ vOPdL-CDLLsIFlE-aEZ6yyxYGOUmznXIES0-p9_5oZ86MEQgPsSIODhf3F4FuFRrw6HiKm
00820+ BRNDiFGqCKmBe_GJOCv6EotwYFDP5iTgg-WPgL9nNFwc80l4Rxaoln5HYArcrGjw10Li-L
00890+ OW3rjMXe5PjXXzyqR0Gw2FWcImbd2gWzOvsRNG76kZi7-TsEZ3UzXMZhPSdwqftR5LF7qt
00960+ d7byW3pjeZOYMqDMaDccwdpGr8mYbiS1KiF_slEXrKv55sF4_h2rv10zCjdweXm; _sp_i
01030+ d=febcfb57-f372-4052-ae5c-b1c052fdde82; __qca=P0-1274716697-1473360368
01100+ 455; _ga=GA1.2.631100383.1473360369; _abc_lsdid1=5404e6b1415a4dc0bb7f5
01170+ 5bd356a5a9d; _abc_sds1=8774a36259495c91ce9f683236d75182%2C5404e6b1415a
01240+ 4dc0bb7f55bd356a5a9d; userid=CC8743A6-9645-2BCA-FEB5-9C358E9B7940; crt
01310+ g_rta=ad4g300250060%3D1%3Bad4g300250080%3D1%3Bad4g300250100%3D1%3Bad4g
01380+ 300250120%3D1%3Bad4g300250140%3D1%3Bad4g300250160%3D1%3Bad4g300250180%
01450+ 3D1%3Bad4g300250200%3D1%3Bad4g300250300%3D1%3Bad4g300250320%3D1%3Bad4g
01520+ 72890020%3D1%3Bad4g336280020%3D1%3Bad4g160600020%3D1%3Bad4g160600040%3
01590+ D1%3Bad4g160600060%3D1%3Bad4g160600080%3D1%3Bad4g160600100%3D1%3Bad4g1
01660+ 60600120%3D1%3Bad4g160600140%3D1%3Bad4g160600160%3D1%3Bad4g160600180%3
01730+ D1%3Bad4g160600200%3D1%3Bad4g160600220%3D1%3Bad4g300600160%3D1%3Bad4g3
01800+ 00600180%3D1%3Bad4g3006002%3D1%3Bad4g300600220%3D1%3Bad4g3205001%3D1%3
01870+ Bad4g32050025%3D1%3Bad4g3205008%3D1%3B; _pk_id.1.6d88=cf5277c0edfa87ff
01940+ .1478276299.11.1481213238.1481213238.; mp_5c2767339ff8b27b54ee2f71c633
02010+ bd56_mixpanel=%7B%22distinct_id%22%3A%20%223acf9e7a-6c17-e611-80c1-4c7
02080+ 2b97cbdb1%22%2C%22tref1%22%3A%20%22xmonetize%22%2C%22tref2%22%3A%20%22
02150+ emailsmaster%22%2C%22tref3%22%3A%20%22Daily%20Quiz%20%7C%20Trivia%20Qu
02220+ izzClub%20Email4%20JSON%22%2C%22tref4%22%3A%20%22NA%22%2C%22emaildomai
02290+ n%22%3A%20%22gmail.com%22%2C%22landing%22%3A%20%22default%22%2C%22regd
02360+ ate%22%3A%20%222016-05-11%22%2C%22tref12%22%3A%20%22facebook-ad%22%2C%
02430+ 22tref13%22%3A%20%22can-you-pass-psychology-101%22%2C%22tref14%22%3A%2
02500+ 0%22us%22%2C%22tref15%22%3A%20%22personality%22%2C%22%24initial_referr
02570+ er%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%2
02640+ 2%24direct%22%7D; amplitude_idservert1.com=eyJkZXZpY2VJZCI6IjYzMDRkYW
02710+ NjLWM2YjktNDRmMi1iZGUxLTQwMjI4YWFmZDM2M1IiLCJ1c2VySWQiOiIzYWNmOWU3YS02
02780+ YzE3LWU2MTEtODBjMS00YzcyYjk3Y2JkYjEiLCJvcHRPdXQiOmZhbHNlLCJzZXNzaW9uSW
02850+ QiOjE0ODE5MDgzODc1ODQsImxhc3RFdmVudFRpbWUiOjE0ODE5MDgzODkxNzIsImV2ZW50
02920+ SWQiOjEsImlkZW50aWZ5SWQiOjEsInNlcXVlbmNlTnVtYmVyIjoyfQ==; QSID=s1; _sp
02990+ _ses=955e8c13-30ae-42ae-8a69-d57339f14837; _gat=1; _abc_uid1=68756ec0f
03060+ 5094d1da12ba8238c429ffd\r\n
03085 Host: servert1.com\r\n
03106 X-IMForwards: 20\r\n
03124 Via: 1.1 soironport.txfb-ins.com:8080 (Cisco-WSA/8.0.6-119)\r\n
0

Could someone help me?

#2

I’ve disable request validation. Everything is fine now. But “show errors” shows such errors

#3

global
log 127.0.0.1 local0

chroot      /var/lib/haproxy
pidfile     /var/run/haproxy.pid
maxconn     20000
user        haproxy
group       haproxy
daemon

defaults
mode http
log global
option httplog

option dontlognull

option http-server-close
option forwardfor       except 127.0.0.0/8
option                  redispatch
retries                 3
timeout http-request    1m
timeout queue           5m
timeout connect         10s
timeout client          5m
timeout server          5m
timeout http-keep-alive 10s
timeout check           10s
maxconn                 10000
option                  accept-invalid-http-request

listen stats
bind *:8080
mode http
log global

    maxconn 10

    timeout client    100s
timeout server      100s
    timeout connect      100s
    timeout queue   100s

backend servert1
balance roundrobin
cookie QSID insert indirect nocache
server v6 s1:80 weight 100 check cookie s1

#4

log record (another one)

81.101.104.134:47386 [20/Dec/2016:13:34:32.553] http-in http-in/ -1/-1/-1/-1/0 400 187 - - PR-- 239/238/0/0/0 0/0 “”

[20/Dec/2016:13:34:32.553] frontend http-in (#3): invalid request
backend (#-1), server (#-1), event #160
src 81.101.104.134:47386, session #312029, session flags 0x00000080
HTTP msg state 26, msg flags 0x00000000, tx flags 0x80000000
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 2670 bytes
pending 2670 bytes, wrapping at 16384, error at position 1472:

00000 GET {NORMAL+URL+WAS+HERE+NORMAL+URL+WAS+HERE+NORMAL+URL+WAS+HERE+NOR}
00070+ {NORMAL+URL+WAS+HERE} HTTP/1.1\r
00138+ \n
00139 Host: servert1.com\r\n
00160 User-Agent: Mozilla/5.0 (Android 6.0.1; Tablet; rv:50.0) Gecko/50.0 Fi
00230+ refox/50.0\r\n
00242 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.
00312+ 8\r\n
00315 Accept-Language: en-GB,en;q=0.5\r\n
00348 Accept-Encoding: gzip, deflate\r\n
00380 Cookie: 323d7f8f696ae3e7=53da53acf098e03e; fqz=ba2d29e0-96d3-4b29-a909
00450+ -c8f9e0ce3eee; .AspNet.ApplicationCookie=9mjoU_f1QdlJmmdRMqq1lRWRxEDFc
00520+ ojb6GZ4hq2QGSn3LyuWfosEM-s7j7SuIl5O_2BdO2645zKSAR5W69CJQMr9mDaCzRhyFJb
00590+ CeqQdQf5_CmvKqqM_uvtWZfv3ixiv2gcq-VPTmd2IP6NTbNYvfhkzmxy_8lQ5NgRJmpLAO
00660+ vlGGoyPJ3-vWcX8W11lFQqp_PvFr_bqTjPG0bLyhq2gHxVDxK04BpuTsp6n-02uQv3V04U
00730+ zWs0HiATg_DP9sNEpTHQU2v9bCcGWd05mYLLKHO3Iw5qVq3GhwCaOZLMylDqJoGF6JZOR2
00800+ UuFkEh32F4FCwnwhw3zJ4ngk8ypJ7vZKPNTPvMGU66zp805dyYoEDNkafjYJ9f_jwQM0cS
00870+ Ew1DGW_gbsmITYqqd79stZMpTbYfoXGSCNYc8iv_-WogLFyk2MGFVc0vz1xQe5o2c4AdPK
00940+ QXsihnLezCTq-YLCAucn97A3C2diBkcCnRvAqNBHKo1UcivcopBswxmIeHTMS5tj3UcjoQ
01010+ r9lEs-M64tuScog; _sp_id=a4e0dbfa-816e-448d-aed3-f6643e7aa3d0; __qca=P0
01080+ -2087919155-1476051864231; _ga=GA1.2.1689700816.1476051864; _abc_lsdid
01150+ 1=709ae5670f1a4db280f7ebf80f7c2117; _abc_sds1=62749d4b385255c9d2d2a578
01220+ 8ba8d1cf%2C709ae5670f1a4db280f7ebf80f7c2117; crtg_rta=; __lx205879_loa
01290+ d_cnt=166; __lx205879_load_tmr=1480209945294; __lx205879_load_tmr_pre=
01360+ 1480209961221; cookieconsent_status=dismiss; _pk_id.1.6d88=8048e2533a7
01430+ 7458f.1478208657.81.1481255309.1481255305.; amplitude_idservert1.com=
01500+ eyJkZXZpY2VJZCI6IjNiMWM4MzUyLTg1NTYtNGY0Yy1iMTA4LWI1YWZjYTdhZjE3NFIiLC
01570+ J1c2VySWQiOiI5OTM0MjA0Mi0yNWVhLWU1MTEtODBiZS00YzcyYjk3Y2JkYjEiLCJvcHRP
01640+ dXQiOmZhbHNlLCJzZXNzaW9uSWQiOjE0ODIyMDkwODI5MzYsImxhc3RFdmVudFRpbWUiOj
01710+ E0ODIyMDkyMzM2MDAsImV2ZW50SWQiOjE2NiwiaWRlbnRpZnlJZCI6MTAsInNlcXVlbmNl
01780+ TnVtYmVyIjoxNzZ9; mp_5c2767339ff8b27b54ee2f71c633bd56_mixpanel=%7B%22d
01850+ istinct_id%22%3A%20%2299342042-25ea-e511-80be-4c72b97cbdb1%22%2C%22tre
01920+ f1%22%3A%20%22xmonetize%22%2C%22tref2%22%3A%20%22emailsmaster%22%2C%22
01990+ tref3%22%3A%20%22Daily%20Quiz%20%7C%20Trivia%20Clickers%200-14d%20%7C%
02060+ 208d%2B%20%7C%20Email1%20Theme%20%7C%20ByActivityTime%20Since%2012%2F8
02130+ %2F2016%22%2C%22tref4%22%3A%20%22NA%22%2C%22emaildomain%22%3A%20%22gma
02200+ il.com%22%2C%22landing%22%3A%20%22default_emtest%22%2C%22regdate%22%3A
02270+ %20%222016-03-14%22%2C%22tref12%22%3A%20%22facebook-ad%22%2C%22tref13%
02340+ 22%3A%20%22TriviaQuestions-2%22%2C%22tref14%22%3A%20%22GB10%25Clickers
02410+ Conv%22%2C%22tref15%22%3A%20%22series%22%2C%22%24initial_referrer%22%3
02480+ A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24dir
02550+ ect%22%7D; QSID=s1; _abc_uid1=c38793b9138943aead7e366480682bfa\r\n
02614 Connection: keep-alive\r\n
02638 Upgrade-Insecure-Requests: 1\r\n
0

#5

That’s expected behavior.

It allows troubleshooting the issue, while the workaround is in place.

The issue seems to be with your huge cookies, can you try bumping tune.bufsize [1] and see what happens?

https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#3.2-tune.bufsize

#7

Disabled splicing for testing purposes. Didn`t help

#8

Os Centos 7 with all updates
Current build info

HA-Proxy version 1.7.1 2016/12/13
Copyright 2000-2016 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -DTCP_USER_TIMEOUT=18
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 USE_PCRE_JIT=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with OpenSSL version : OpenSSL 1.0.2j 26 Sep 2016
Running on OpenSSL version : OpenSSL 1.0.2j 26 Sep 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : yes
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe

#9

After disabling request some requests are working, other - not. 400 Error Code.

https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#3.2-tune.bufsize

didn’t help
I’ve set tune.bufsize 32768 tune.maxrewrite 1024
Errors still here. After analysing requests I see that problem appears in random position. Cookie is just big field. I saw it in referrer and others. all requests are abour 2-5 Kb. I’ve tried to move traffic to another haproxy server in case hardware problems. Problem is still there.

#10

I’ve tried to replay some of that requests using yandex tank (load test). It runs on 1000 rps without any 400 code error. Server can’t run faster. So may be some symbols are hidden and there is request problem.

#11

Yandex very likely doesn’t produce those huge cookie headers.

Can you try tune.bufsize 32768 but without any maxrewrite configuration?

edit: Also what you posted here may be incomplete. Can you confirm the output of affected lines is as is here in discourse as it was on your command line? For example, the line that starts with “01430+” (containing byte 1472), do it look exactly the same? Because it seems we are missing characters.

#13

Can you try tune.bufsize 32768 but without any maxrewrite configuration?

Doesn’t help problem still here. You can see error using that link

About requests Yes, i’ve modified them. Please use previous link to see them

#14

Hi,

I looked through your traces, and I cannot find anything wrong. Replaying the same exact request in local repro works fine, I’m unable to reproduce this problem.

Can you provide the complete configuration and can you tell if this happens only under load only or does it also happen on on an “empty” server?

I believe we need to move this discussion to the mailing list, as I’m unable to pin point the root cause.

Can you send the github “gist” link along with a short description of what we have tried here to the mailing list at:
haproxy at formilux dot org

You don’t have to subscribe, just send an email to this address.

#15

can’t send

Diagnostic-Code: smtp; 451 Temporary local problem - please try later

is it working?

#16

It works fine, I just send a test email:

https://www.mail-archive.com/haproxy@formilux.org/msg24455.html

Double check the email address:
haproxy@formilux.org

If it still doesn’t work, can you provide the full error message please.

#17

I’m sorry , I’ve made a mistake -formilux.dot.org

#18

Bug is found. Fix soon be ready.

#19

fix

http://git.haproxy.org/?p=haproxy-1.7.git;a=commit;h=934600081df66e44ecda2a1c22edbf9725d1e228