HAProxy community

Haproxy-1.8 segfault

#1

Hi,

We’ve been using haproxy-1.8.18 happily since it was released.

It sits in front our app server and gets requested to route them to our backend farm depending on the desired app service.

We use:

  • nbproc 5;
  • 1 unix frontend attached to each process;
  • communicate with a few hundred (~300) backends. We use server-templates for a few cases with DNS resolution and IP addresses (IPv4 and IPv6) for the rest.

All frontends are non-encrypted HTTP/1.1 and backends vary between TLS are HTTP.

This is the very same (boring but long) config file we’ve always used.

And this weekend a few haproxy processes started dying. “show errors” don’t show anything (and can’t be used after the process dies), so not a clue besides this dmesg messages:

[12313007.354629] haproxy[1010574]: segfault at 58 ip 000000000048de73 sp 00007ffd7e218950 error 4 in haproxy[400000+146000]
[12313007.355575] Code: 44 24 18 48 8b 00 48 85 c0 74 1f 89 4c 24 28 48 89 54 24 20 4c 89 e7 4c 89 04 24 ff d0 8b 4c 24 28 48 8b 54 24 20 4c 8b 04 24 <48> 8b 42 58 48 85 c0 0f 84 4c 08 00 00 4d 85 c0 74 0d 41 83 78 10 
[12316538.140456] haproxy[1013602]: segfault at 58 ip 000000000048de73 sp 00007ffe09fbe5e0 error 4 in haproxy[400000+146000]
[12316538.141250] Code: 44 24 18 48 8b 00 48 85 c0 74 1f 89 4c 24 28 48 89 54 24 20 4c 89 e7 4c 89 04 24 ff d0 8b 4c 24 28 48 8b 54 24 20 4c 8b 04 24 <48> 8b 42 58 48 85 c0 0f 84 4c 08 00 00 4d 85 c0 74 0d 41 83 78 10 
[12424725.217771] haproxy[1112079]: segfault at 58 ip 000000000048de73 sp 00007fff1d1e87c0 error 4 in haproxy[400000+146000]
[12424725.218582] Code: 44 24 18 48 8b 00 48 85 c0 74 1f 89 4c 24 28 48 89 54 24 20 4c 89 e7 4c 89 04 24 ff d0 8b 4c 24 28 48 8b 54 24 20 4c 8b 04 24 <48> 8b 42 58 48 85 c0 0f 84 4c 08 00 00 4d 85 c0 74 0d 41 83 78 10 
[12444059.954893] haproxy[1112083]: segfault at 58 ip 000000000048de73 sp 00007fff1d1e87c0 error 4 in haproxy[400000+146000]
[12444059.955708] Code: 44 24 18 48 8b 00 48 85 c0 74 1f 89 4c 24 28 48 89 54 24 20 4c 89 e7 4c 89 04 24 ff d0 8b 4c 24 28 48 8b 54 24 20 4c 8b 04 24 <48> 8b 42 58 48 85 c0 0f 84 4c 08 00 00 4d 85 c0 74 0d 41 83 78 10 
[12473582.800870] haproxy[1162962]: segfault at 58 ip 000000000048de73 sp 00007fff3d196f00 error 4 in haproxy[400000+146000]
[12473582.801908] Code: 44 24 18 48 8b 00 48 85 c0 74 1f 89 4c 24 28 48 89 54 24 20 4c 89 e7 4c 89 04 24 ff d0 8b 4c 24 28 48 8b 54 24 20 4c 8b 04 24 <48> 8b 42 58 48 85 c0 0f 84 4c 08 00 00 4d 85 c0 74 0d 41 83 78 10 
[12489985.349159] haproxy[1162959]: segfault at 58 ip 000000000048de73 sp 00007fff3d196f00 error 4 in haproxy[400000+146000]
[12489985.350112] Code: 44 24 18 48 8b 00 48 85 c0 74 1f 89 4c 24 28 48 89 54 24 20 4c 89 e7 4c 89 04 24 ff d0 8b 4c 24 28 48 8b 54 24 20 4c 8b 04 24 <48> 8b 42 58 48 85 c0 0f 84 4c 08 00 00 4d 85 c0 74 0d 41 83 78 10 

I’m not sure what info I can provide since this is live traffic, but obviously I’ll try and get as much as possible. I since updated to 1.8.19 with the latest patches on top of it (as of March 23rd git tree) but can’t spot a single problem there that’s related. And segfault is…weird!

Any info you need to help figure this out would be greatly appreciated :slight_smile:

0 Likes

#2

So does 1.8.19 + patches crash or not? 1.8.18 has a known crash bug with unique_id that was fixed in 1.8.19.

If you keep seeing crashes with the current 1.8 git tree (or 1.8.19), please generate a core and a backtrace of it.

0 Likes

#3

Hi Lukas,

No, I updated to 1.8.19 + Latest patches after a few crashes. But we don’t use unique-id, which is the only crash-fix on that tree.

I’ll keep my peeled for any future crashes and report back. If you have any insight on what those code might indicate, please let me know.

0 Likes

#4

Get ready to enable core dumping on that infrastructure, so that next time it happens, you can generate a backtrace with gdb.

The output from dmesg is pretty much useless.

0 Likes

#5

Hi Lukas,

Show to let you know, we got a few segfaults even with 1.8.19 + commits up to a3cfe8f4.

I’ll go onto tracking this a little further and let you know.

0 Likes

#7

It could be this:

Could your try 1.9.6 or apply the patch manually to your 1.8 tree?

0 Likes

#8

It indeed fixed it!

Thanks,

0 Likes