HAproxy returns 408 or 504 error when timeout client value is every 25d

amiee · May 28, 2019, 10:21am

Hello,
I am using HAproxy with version 1.5.18.
There is a weird pattern for timeout client value which cause HAproxy works or return 408/504 error when added it to the fronted section.

frontend http_vip
mode http
bind x.x.x.x:80 transparent
option httplog clf
log global

timeout client 25d
default_backend http-pool

For example:
Values equal or above 1us and below 25d are working (application behind the HAproxy can be accessed).
Values equal or above 25d and below 50d are NOT working (application behind the HAproxy can NOT be accessed).
Values equal or above 50d and below 75d are working
Values equal or above 75d and below 100d are NOT working
Values equal or above 100d and below 125d are working
Values equal or above 125d and below 150d are NOT working
and this pattern continues forever every 25 days.

When the application cannot be accessed, a 408 or 504 error is returned.
In the access log, the session is terminated with “cC” state as following:
“GET /index.html HTTP/1.1” 504 194 “” “” 58568 614 “http_vip” “http-pool” “m1” 0 0 0 -1 0 cC-- 0 0 0 0 0 0 0 “” “”

I have tried with tcp mode, still has the same result.

Pls help to debug on this.

Thanks.

lukastribus · May 28, 2019, 12:29pm

Can you provide the output of haproxy -vv?

amiee · May 29, 2019, 2:49am

Build options :
TARGET = linux26
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing
OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_OPENSSL=1 USE_STATIC_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built without zlib support (USE_ZLIB not set)
Compression algorithms supported : identity
Built with OpenSSL version : OpenSSL 1.0.2p-fips 14 Aug 2018
Running on OpenSSL version : OpenSSL 1.0.2p-fips 14 Aug 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.42 2018-03-20
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

lukastribus · May 29, 2019, 10:05pm

Timeouts with huge values like this are certainly not something that was expected. Tick handling will wrap at 24.85 days, see:

github.com

haproxy/haproxy/blob/master/include/common/ticks.h#L39


* relative to the current date. Since it is not practical to update all values
* each time the current date changes, instead we use the absolute date rounded
* down to fit in a tick. We then have to compare a tick to the current date to
* know whether it is in the future or in the past. If a tick is below the
* current date, it is in the past. If it is above, it is in the future. The
* values will wrap so we can't compare that easily, instead we check the sign
* of the difference between a tick and the current date.
*
* Proceeding like this allows us to manipulate dates that are stored in
* scalars with enough precision and range. For this reason, we store ticks in
* 32-bit integers. This is enough to handle dates that are between 24.85 days
* in the past and as much in the future.
* 
* We must both support absolute dates (well in fact, dates relative to now+/-
* 24 days), and intervals (for timeouts). Both types need an "eternity" magic
* value. For optimal code generation, we'll use zero as the magic value
* indicating that an expiration timer or a timeout is not set. We have to
* check that we don't return this value when adding timeouts to <now>. If a
* computation returns 0, we must increase it to 1 (which will push the timeout
* 1 ms further). For this reason, timeouts must not be added by hand but via
* the dedicated tick_add() function.

Maybe we can document this maximum tick value and reject such invalid configurations, so there is no doubt about this.

Any thoughts on this @willy ?

willy · May 30, 2019, 5:06am

Hi Lukas!
Well I’m embarrassed, I was absolutely certain it was documented… and I was wrong! The only place where I find such a mention in the doc is on the stick-table expire keyword. Not only we need to add to add it in the configuration, but we also need to check it during parsing (and yes, I was pretty sure it was tested as well). Let’s fix this before 2.0. Ideally we need to report an error if a non-null timeout is null after conversion, and if a timeout overflows. We can probably have parse_time_err() report two dummy pointers for “underflow” and “overflow” so that callers can report errors.

willy · May 30, 2019, 5:18am

I created issue #109 with this.

willy · June 7, 2019, 5:34pm

This is now fixed in 2.0-dev.

Topic		Replies	Views
Backend server not avaliable for layer6 timeout Help!	1	846	June 8, 2022
Haproxy 2.7.1 logging a ton of 0/400/408 errors with <BADREQ> Help!	4	359	January 11, 2023
Debugging connection and session issues - Haproxy TCP Mode Help!	2	10304	March 24, 2016
Server timeout past 60 seconds? (FIXED - php-fpm) Help!	2	3022	October 28, 2019
Haproxy throwing intermittent 503, Backend server not available for layer6 timeout Help!	1	655	June 17, 2022

HAproxy returns 408 or 504 error when timeout client value is every 25d

Related Topics