Seeing extremely large numbers in gpc0_rate counter in stick table


We are using HAProxy community version 2.2.24. We are using the gpc counters for rate limiting and it works well, however it seems every once in a while, we see extremely large numbers for gpc_rate in our stick tables without the data to back it up. For example, we are using gpc0 to track the number of “400” events for token request error returned from ADFS. We use gpc0_rate to track the error rate. If the error rate exceeds our threshold, any incoming requests for that specific entity are blocked. We are using peers to have the stick tables replicated between HAProxy servers. This works perfectly 99% of the time. Every once is a great while we are seeing extremely large numbers in gpc0 & gpc0_rate that have no databacking them up, meaning I do not see a large number of “400” events coming from ADFS.

I see this same scenario occur with our TokenRate limiters that also use gpc counters. The rates are nowhere near the numbers captured in the stick table.

I have the deny’s prior to my increments so that the counters will not increase when a request is denied.

Has anyone seen this behavior before?

Haproxy 2.2.24 has a number of bugs:

Specifically it has a few nasty ones with stick tables, like the the 49.7 days uptime bug or a race condition.

I suggest to upgrade to the latest bugfix release as per as a first step.