"show map" command return wrong value on haproxy 1.9.x

Hi.
I have used haproxy 1.7.x with lua to calculate a successful response of accumulate counts of each ts file of hls stream and update to map data.
I have run “show map” command every 1min by crontab to get accumulate count value and there is no problem for a long time without a problem.

But, recently we updated haproxy version to 1.9.14 and 1.9.15, I seldom got a problem when I use the “show map” command to get map data.


case1

“show map” command return unexpected string instead of number.

0x7f63ba6da4b0 hls-m.example.com/ad1 À<91>^\ºc^?
0x7f63b1094480 hls-m.example.com/ad1/ad1.smil 27956
0x7f63cdf5c0e0 hls-m.example.com/ad1/ad1.smil/b496000 1358
0x7f63b10a2500 hls-m.example.com/ad1/ad1.smil/b1328000 3632
0x7f63b10937a0 hls-m.example.com/ad1/ad1.smil/b2692000 22966
0x7f63ba6ea960 hls-m.example.com/ad1/ad1_300.stream 8774
0x7f63b0ed1010 hls-m.example.com/ad1/ad1_800.stream 197021
0x7f63ba6d0780 hls-m.example.com/ad1/ad1_2000.stream 60520

bug

0x7f63ba6da4b0 hls-m.example.com/ad1 À<91>^\ºc^?

the value must be a number in my map data.


case2.

“show map” command returns a key without a value.

example

0x1cafa70 hls-m.example.com/ad5 370954
0x7f23f8f48150 hls-m.example.com/ad5/ad5.smil 44122
0x7f23f8f48480 hls-m.example.com/ad5/ad5.smil/b496000 1152
0x1cae3e0 hls-m.example.com/ad5/ad5.smil/b1328000 11444
0x1cafc70 hls-m.example.com/ad5/ad5.smil/b2692000 31526
0x7f23f8f245d0 hls-m.example.com/ad5/ad5_300.stream 8230
0x7f23f8f4c3d0 hls-m.example.com/ad5/ad5_800.stream 229211
0x1ca19a0 hls-m.example.com/ad5/ad5_2000.stream 89391
0x1c9cf80 hls.example.com/ad5
0x1ca8520 hls.example.com/ad5/ad5_800.stream 115320
0x7f23f92ea300 hls.example.com/ad5/ad5_5000.stream 5005

bug

0x1c9cf80 key “hls.example.com/ad5” exist, but no value.

0x1c9cf80 hls.example.com/ad5

If I run “show map” command once more, I can get correct value

0x1c9cf80 hls.example.com/ad5 120325

case3.

“show map” command returns a value that looks like stripped from the origin.

example

0x22de590 hls-m.example.com/ad5 398640
0x7f0bb8bd79d0 hls-m.example.com/ad5/ad5.smil 48757
0x7f0bb8bd7c90 hls-m.example.com/ad5/ad5.smil/b496000 9070
0x7f0bb8bbcb10 hls-m.example.com/ad5/ad5.smil/b1328000 8868
0x7f0bd473dd30 hls-m.example.com/ad5/ad5.smil/b2692000 30819
0x22db280 hls-m.example.com/ad5/ad5_300.stream 9633
0x22de760 hls-m.example.com/ad5/ad5_800.stream 253635
0x7f0bb8bd7120 hls-m.example.com/ad5/ad5_2000.stream 86615
0x7f0bbd51d7f0 hls.example.com/ad5 1744
0x7f0bbd4e9ed0 hls.example.com/ad5/ad5_800.stream 163192
0x7f0bbd5b9970 hls.example.com/ad5/ad5_2000.stream 61
0x7f0bbd4eb310 hls.example.com/ad5/ad5_5000.stream 11147
0x186ad50 hls.example.com/ch4/ch4_800.stream 9643

bug

0x7f0bbd51d7f0 key “hls.example.com/ad5” exist, but the value (1744) is striped from 174400.
163192 + 61 + 11147 = 174400

0x7f0bbd51d7f0 hls.example.com/ad5 1744

If I run “show map” command once more, I can get correct value

0x22de590 hls-m.example.com/ad5 398640
0x7f0bb8bd79d0 hls-m.example.com/ad5/ad5.smil 48757
0x7f0bb8bd7c90 hls-m.example.com/ad5/ad5.smil/b496000 9070
0x7f0bb8bbcb10 hls-m.example.com/ad5/ad5.smil/b1328000 8868
0x7f0bd473dd30 hls-m.example.com/ad5/ad5.smil/b2692000 30819
0x22db280 hls-m.example.com/ad5/ad5_300.stream 9633
0x22de760 hls-m.example.com/ad5/ad5_800.stream 253635
0x7f0bb8bd7120 hls-m.example.com/ad5/ad5_2000.stream 86615
0x7f0bbd51d7f0 hls.example.com/ad5 174400
0x7f0bbd4e9ed0 hls.example.com/ad5/ad5_800.stream 163192
0x7f0bbd5b9970 hls.example.com/ad5/ad5_2000.stream 61
0x7f0bbd4eb310 hls.example.com/ad5/ad5_5000.stream 11147
0x186ad50 hls.example.com/ch4/ch4_800.stream 9643


These problems happened seldom but it makes me hard to parse a correct value from map data by using the “show map” command.

I think map data that haproxy is handling is correct but there is some bug when haproxy return map data by using the “show map” command on haproxy 1.9.x.


Please help me.
below is info of my system that I using.

uname -a

Linux SI8221-130 2.6.32-754.17.1.el6.x86_64 #1 SMP Tue Jul 2 12:42:48 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

haproxy -vv

HA-Proxy version 1.9.15 2020/04/02 - https://haproxy.org/
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
OPTIONS = USE_DL=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.1g 21 Apr 2020
Running on OpenSSL version : OpenSSL 1.1.1g 21 Apr 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity(“identity”)
Built with PCRE version : 7.8 2008-09-05
Running on PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with multi-threading support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as cannot be specified using ‘proto’ keyword)
h2 : mode=HTX side=FE|BE
h2 : mode=HTTP side=FE
: mode=HTX side=FE|BE
: mode=TCP|HTTP side=FE|BE

Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace

if you need more info, please let me know.
I hope your response. thanks.

Could you file a bug at Github?

Also could you try disabling multithreading by putting nbthread 1 into the global section of your configuration, see if that works around the problem?

Thanks

hi, lukastribus.

I upload this issue at Github.

nbproc          2
nbthread        20

cpu-map         1/all   0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38
cpu-map         2/all   1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

I use 2 processes with nbthread 20 and could try disabling multithreading but I wonder 1 process with nbthread 1 can handle same load.

is this should be ok?

nbproc          2
nbthread        1

cpu-map         1/all   0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38
cpu-map         2/all   1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
1 Like

You should decide whether to use multi-threading or multi-processing. Enabling both exposes you to the disadvantages of both.

I’m just asking to disable threading, you can keep nbproc 2 with nbthread 1.

Whether or not you need all those threads depends on what your configuration and your load is. Are you terminating SSL there?

Process 1 is used to terminate SSL on the frontend and process 2 is used to handle other work on the backend.

I can try to disable multi-threading. I used multi-processing in haproxy 1.7.x, but I had no problems, so I think disabling threading can mitigate this problem.

Anyway, it is not a real solution, so I hope this bug will be fixed as soon as possible.

Confirming whether multi-threading affects the issue or not will help the developers understand how to fix this bug.