Constantly changing sessions under H2


#1

I am trying to deploy 1.8 to make use of h2. We have a single HAProxy install in front of several PHP web application servers. We have enjoyed HAProxy for the last several years and have upgraded from 1.5-1.7 without issue. I have been trying to get 1.8-rcX working and everything is OK except under h2 the ajax calls on the site a breaking. We use a CSRF tokens which are linked to the session, the problem is under h2 each ajax request gets a different session ID. Removing h2 fixes the problem.If I request the URLS directly under h2 it works, it is only we they are called via AJAX

This probably isn’t a HAProxy issue, but is there any reason that I would be seeing this behavour under h2, and not http/1.1?

haproxy -vv
HA-Proxy version 1.8-rc3-34650d5 2017/11/11
Copyright 2000-2017 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = x86_64
CC = gcc
CFLAGS = -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with network namespace support.
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Encrypted password support via crypt(3): yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with multi-threading support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace


config

global
log 127.0.0.1 local0
maxconn 20000
ssl-server-verify none
user haproxy
group haproxy

# set default parameters to the intermediate configuration
tune.ssl.default-dh-param 2048


# STATS SOCKET
stats socket /var/run/haproxy.stats level admin

# ACCEPT LARGE REQUESTS
tune.bufsize 128000

defaults
log global
mode http
retries 3

maxconn                 20000
timeout connect         15s
timeout client          15s
timeout server          90s
timeout http-request    5s
timeout http-keep-alive 15s

option forwardfor
option httplog
option http-keep-alive
option http-server-close

Redirect all HTTP traffice to HTTPS.

frontend WEB-HTTP-IN
bind :80
option forwardfor

# Redirection Everything else to HTTPS
redirect code 301 scheme https if !{ ssl_fc }

Main HTTS Frontend for our sites.

frontend WEB-HTTPS-IN
option forwardfor

bind 10.0.0.1:443 ssl crt /etc/haproxy/ssl/cert.pem no-sslv3 alpn h2,http/1.1

############# RATE LIMITNG BRUTE FORCE #######################

# Table definition  

acl login_request path_beg -i /account/login
tcp-request inspect-delay 10s
acl brute_force        sc1_inc_gpc0 gt 20
stick-table type binary len 20 size 100k expire 300s store gpc0
tcp-request content track-sc1 base32+src if METH_POST login_request
http-request deny if brute_force

############################################################

# Block bad IPs
acl bad_ip hdr_ip(X-Forwarded-For) -f /etc/haproxy/bad_ips.lst
http-request deny if bad_ip

# CAPTURE HEADERS FOR LOGGING
capture request header Host len 64
capture request header x-csrf-token len 64

# Send all other traffic that does match anything else to the WEB-FARM
default_backend WEB-FARM

backend WEB-FARM
balance static-rr

server WEB-011 192.168.70.221:80 check maxconn 12

l


#2

How do you generate those session ID’s exactly and can show show us a request/response that shows how those session ID’s look like?

Also, which Browsers are affected by this problem?

Please give us more information about this session ID.


#3

Chrome and firefox are both affected, Edge works as expected. Example request/response from chrome. Sessions are generated via PHP using the memcached extension, but the problem persists if we use the default file, or redis as the session manager. The fact that it work with Edge makes me think its a JS issue but I am not sure why it would act differently under H2.

You can see in the requests that the session ID is changing, which means the CSRF token can’t be validated. This doesn’t occur when using HTTP/1.1

Below is the H2 request/response from Chrome. The 400 is returned from our CSFR checks, rather then an actual bad/malformed request.

Request URL:https://www.domain.com/account/sessionTimeoutConf
Request Method:POST
Status Code:400 
Remote Address:66.171.196.7:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
cache-control:max-age=864000, private, must-revalidate
content-encoding:gzip
content-length:31
content-type:text/html; charset=UTF-8
date:Mon, 13 Nov 2017 21:37:39 GMT
expires:Thu, 19 Nov 1981 08:52:00 GMT
pragma:no-cache
server:Apache
set-cookie:PHPSESSID=8E565789-E58C-4F9D-A1F7-84FD3DE54F15; path=/; secure; httponly;Secure
set-cookie:PHPSESSID=8E565789-E58C-4F9D-A1F7-84FD3DE54F15; path=/; secure; HttpOnly;Secure
status:400
vary:Accept-Encoding,User-Agent
x-frame-options:SAMEORIGIN

Request Headers
:authority:www.domain.com
:method:POST
:path:/account/sessionTimeoutConf
:scheme:https
accept:text/plain, */*; q=0.01
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.9
content-length:0
cookie:showCookieNotification=1; __lc.visitor_id.5967301=S1500852004.57dda70ae1; lc_window_state=minimized; PHPSESSID=79979F1B-4F0D-4C4E-BF2E-4685CA514FFF
origin:https://www.domain.com
referer:https://www.domain.com/
user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36
x-csrf-token:fad09fac13185416393dc459f7b4ec7c299f352f
x-requested-with:XMLHttpRequest

Next AJAX Request

Request URL:https://www.domain.com/tag/renderWidget
Request Method:POST
Status Code:400 
Remote Address:66.171.196.7:443
Referrer Policy:no-referrer-when-downgrade
Response Headers
cache-control:max-age=864000, private, must-revalidate
content-encoding:gzip
content-length:31
content-type:text/html; charset=UTF-8
date:Mon, 13 Nov 2017 21:37:40 GMT
expires:Thu, 19 Nov 1981 08:52:00 GMT
pragma:no-cache
server:Apache
set-cookie:PHPSESSID=574BC7D3-A09A-47FC-9D9F-A1CE30A116DC; path=/; secure; httponly;Secure
set-cookie:PHPSESSID=574BC7D3-A09A-47FC-9D9F-A1CE30A116DC; path=/; secure; HttpOnly;Secure
status:400
vary:Accept-Encoding,User-Agent
x-frame-options:SAMEORIGIN

Request Headers
:authority:www.domain.com
:method:POST
:path:/tag/renderWidget
:scheme:https
accept:text/html, */*; q=0.01
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.9
content-length:20
content-type:application/x-www-form-urlencoded; charset=UTF-8
cookie:showCookieNotification=1; __lc.visitor_id.5967301=S1500852004.57dda70ae1; lc_window_state=minimized; PHPSESSID=8E565789-E58C-4F9D-A1F7-84FD3DE54F15
origin:https://www.domain.com
referer:https://www.domain.com/
user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36
x-csrf-token:fad09fac13185416393dc459f7b4ec7c299f352f
x-requested-with:XMLHttpRequest

Same request but with HTTP/1.1

Request URL:https://www.domain.com/account/sessionTimeoutConf
Request Method:POST
Status Code:200 OK
Remote Address:66.171.196.7:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
view source
Cache-Control:max-age=864000, private, must-revalidate
Content-Encoding:gzip
Content-Length:79
Content-Type:text/html; charset=UTF-8
Date:Mon, 13 Nov 2017 21:45:31 GMT
Expires:Thu, 19 Nov 1981 08:52:00 GMT
Pragma:no-cache
Server:Apache
Vary:Accept-Encoding,User-Agent
X-Frame-Options:SAMEORIGIN

Request Headers
view source
Accept:text/plain, */*; q=0.01
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.9
Cache-Control:no-cache
Connection:keep-alive
Content-Length:0
Cookie:showCookieNotification=1; __lc.visitor_id.5967301=S1500852004.57dda70ae1; lc_window_state=minimized; PHPSESSID=C968D468-429E-4051-837A-DA4B137CF38A
Host:www.domain.com
Origin:https://www.domain.com
Pragma:no-cache
Referer:https://www.domain.com/
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36
X-CSRF-Token:1cb3658f3ef90ce1e3003b3e853fca3b14508fa5
X-Requested-With:XMLHttpRequest

Next Request

Request URL:https://www.domain.com/tag/renderWidget
Request Method:POST
Status Code:200 OK
Remote Address:66.171.196.7:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
view source
Cache-Control:max-age=864000, private, must-revalidate
Content-Encoding:gzip
Content-Length:94
Content-Type:text/html; charset=UTF-8
Date:Mon, 13 Nov 2017 21:45:32 GMT
Expires:Thu, 19 Nov 1981 08:52:00 GMT
Pragma:no-cache
Server:Apache
Vary:Accept-Encoding,User-Agent
X-Frame-Options:SAMEORIGIN

Request Headers
view source
Accept:text/html, */*; q=0.01
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.9
Cache-Control:no-cache
Connection:keep-alive
Content-Length:22
Content-Type:application/x-www-form-urlencoded; charset=UTF-8
Cookie:showCookieNotification=1; __lc.visitor_id.5967301=S1500852004.57dda70ae1; PHPSESSID=C968D468-429E-4051-837A-DA4B137CF38A; lc_window_state=minimized
Host:www.domain.com
Origin:https://www.domain.com
Pragma:no-cache
Referer:https://www.domain.com/
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36
X-CSRF-Token:1cb3658f3ef90ce1e3003b3e853fca3b14508fa5
X-Requested-With:XMLHttpRequest

#4

I guess this could be it:

Maybe Chrome and FF already split cookies headers into different fields, as per the suggestion in the RFC, and Edge doesn’t.
Since haproxy does not implement the necessary concatenation, and your backend probably is unable to handle it, it doesn’t see the cookies, therefor rewriting them at every transaction, leading to this behavior.

Capture and Analyze the cleartext HTTP traffic between haproxy and your backend; if Chrome and FF requests have multiple “Cookie:” lines in the HTTP/1.1 cleartext traffic towards the backend and Edge has not, than this could be it:
https://tools.ietf.org/html/rfc7540#section-8.1.2.5


#5

Thanks, this is indeed the problem.