# ACL for Bad User-Agents with SubString Match Fails

**URL:** <https://discourse.haproxy.org/t/acl-for-bad-user-agents-with-substring-match-fails/9907>\
**Category:** Help!\
**Created:** [June 11, 2024, 10:12am UTC](https://discourse.haproxy.org/t/acl-for-bad-user-agents-with-substring-match-fails/9907 "2024-06-11T10:12:02Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![georgeg](https://avatars.discourse-cdn.com/v4/letter/g/779978/32.png) [@georgeg](https://discourse.haproxy.org/u/georgeg)\
**Post date:** [June 11, 2024, 10:12am UTC](https://discourse.haproxy.org/t/acl-for-bad-user-agents-with-substring-match-fails/9907/1 "2024-06-11T10:12:02Z")

</div>

Hello,

We created an ACL for blocking bad user-agents/bots but it gives a false positive 403 in a specific cases which I will explain below.

Here is the config:  
acl is-blockedagent-http hdr\_sub(user-agent) -f /etc/haproxy/agentblock.lst  
http-request deny if is-blockedagent-http

The agentblock.lst list consists of user-agents not in their full form, sample below:  
Disco  
Discobot  
Discoverybot  
ZumBot  
ZyBorg

So the ACL provided above will normally give a 403 when it finds an exact match of “ZyBorg” anywhere within a user-agent , and again normally will give 200 for “ZyBor” which is missing the last “g” character.  
But in cases where similar words exist in the list like “Disco” , “Discobot” and “Discoverybot”,  
the ACL will false-fully give a 403 when a user-agent includes the word “Discov”, “Discove” , “Discover”, “Discovery” ,“Discoveryb”, “Discob”, “Discobo”.  
So it will trigger a 403 for anything starting with “Disco” and matching any letter from bot or very which are included in “Discobot” and “Discoverybot”

We also tried adding another ACL and list for fixing the above but it doesn’t work:  
acl is-blockedagent-http hdr\_sub(user-agent) -f /etc/haproxy/agentblock.lst  
acl is-goodagent-http hdr\_sub(user-agent) -f /etc/haproxy/goodagents.lst  
http-request deny if is-blockedagent-http !is-goodagent-http

goodagent.lst:  
Discob  
Discobo  
Discov  
Discove  
Discover  
Discovery  
Discoveryb  
Discoverybo

Any ideas on how to resolve the above which only occurs when there are user-agents with similar names in the list?  
The only alternative working solution we can think of is using req.fhdr(user-agent) but we would need a list of user-agents in their full form which is hard to find.

Thank you

---

<div class="post-metadata">

**Author:** ![adarragon](https://avatars.discourse-cdn.com/v4/letter/a/7bcc69/32.png) [@adarragon](https://discourse.haproxy.org/u/adarragon)\
**Post date:** [June 11, 2024, 12:12pm UTC](https://discourse.haproxy.org/t/acl-for-bad-user-agents-with-substring-match-fails/9907/2 "2024-06-11T12:12:31Z")

</div>

Hi,

I’m assuming the user agent names always start with the word you want to match them against, right?

```auto
"Disco *blabla*"
"Zyborg *blabla*"
...

```

If so, then you could leverage the `field()` converter to only extract the first “word” of the user agent and compare it against the pattern file using the strict string (-m str) matching method:

```auto
# use space, slash, underscore and dash as word delimiter using field():
acl is-blockedagent-http "hdr(user-agent),field(1,' /-_')" -m str -f /etc/haproxy/agentblock.lst
http-request deny if is-blockedagent-http

```

---

<div class="post-metadata">

**Author:** ![georgeg](https://avatars.discourse-cdn.com/v4/letter/g/779978/32.png) [@georgeg](https://discourse.haproxy.org/u/georgeg)\
**Post date:** [June 11, 2024, 1:54pm UTC](https://discourse.haproxy.org/t/acl-for-bad-user-agents-with-substring-match-fails/9907/3 "2024-06-11T13:54:04Z")

</div>

Hello,

Thank you for your response.  
In most cases the user-agents don`t start with the word we would like to match, examples of complete user-agent names look as follows:

(1) Mozilla/4.0 compatible ZyBorg/1.0 ([wn-14.zyborg@looksmart.net](mailto:wn-14.zyborg@looksmart.net); [http://www.WISEnutbot.com](http://www.WISEnutbot.com))  
(2)Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)

Thank you

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [June 11, 2024, 2:57pm UTC](https://discourse.haproxy.org/t/acl-for-bad-user-agents-with-substring-match-fails/9907/4 "2024-06-11T14:57:36Z")

</div>

Add more data. For example leading space (needs to be escaped) and trailing slash.

```
\ ZyBorg/
\ discobot/

```

Otherwise there is no choice but to revert to regex but it will be more expensive:

> **[RegExr: Learn, Build, & Test RegEx](https://regexr.com/81hk2)**
>
> RegExr is an online tool to learn, build, & test Regular Expressions (RegEx / RegExp).

```
-m reg -f /etc/haproxy/agentblock.regex

```

Matching:

```
\WZyBorg\W
\Wdiscobot\W

```

---

<div class="post-metadata">

**Author:** ![georgeg](https://avatars.discourse-cdn.com/v4/letter/g/779978/32.png) [@georgeg](https://discourse.haproxy.org/u/georgeg)\
**Post date:** [June 13, 2024, 11:54am UTC](https://discourse.haproxy.org/t/acl-for-bad-user-agents-with-substring-match-fails/9907/5 "2024-06-13T11:54:18Z")

</div>

Tried the below with the following config

```auto
\ ZyBorg/
\ discobot/

```

hdr\_sub(user-agent) -m reg -f filename ← Gives 403 only when the user-agent includes the slash and backslash.  
req.fhdr(user-agent) -m reg -f filename ← Same as above.  
hdr\_sub(user-agent) -f filename ← Same as above.

I also tried

```auto
\WZyBorg\W
\Wdiscobot\W

```

With  
`hdr(user-agent) -m reg -f filename`  
and  
`hdr_reg(user-agent) -f filename`

Thank you for the suggestions.

---

<div class="post-metadata">

**Author:** ![adarragon](https://avatars.discourse-cdn.com/v4/letter/a/7bcc69/32.png) [@adarragon](https://discourse.haproxy.org/u/adarragon)\
**Post date:** [June 13, 2024, 1:16pm UTC](https://discourse.haproxy.org/t/acl-for-bad-user-agents-with-substring-match-fails/9907/6 "2024-06-13T13:16:10Z")

</div>

It doesn’t make sense to use both hdr\_ **match-method** + `-m match-method`

Indeed, `hdr_sub(name)` is already an alias for `hdr(name) -m sub`

So if you do this:

```auto
hdr_sub(user-agent) -m reg -f filename 

```

It ends up being the same as if you used:

```auto
hdr(user-agent) -m sub -m reg -f filename 

```

And considering that haproxy only considers the last `-m`, it won’t work as expected.

Now to complete Lukas answer:

For his first suggestion, escaping the “space” with a backslash will only work for patterns provided directly in the config file but not in dedicated pattern/acl files loaded using `-f`: character escaping is handled at the HAProxy config parser level, while `-f` leverages the acl/pattern file parser, which is pretty basic and doesn’t offer advanced features such as escaping. And since leading spaces are ignored when loading patterns from a file using `-f` (as per the documentation), this cannot work for you.

However with the second suggestion (involving regexp), it should work for your use case:

```auto
hdr(user-agent) -m reg -i -f filename

```

filename:

```auto
\WZyBorg\W
\Wdiscobot\W

```

I tested the regexp example provided by Lukas against the full user agents examples you provided in your previous post, and it seems to do the job, and if not, you may want to adjust the regexp expressions to better fit your needs using an online regexp tester for instance

---

<div class="post-metadata">

**Author:** ![georgeg](https://avatars.discourse-cdn.com/v4/letter/g/779978/32.png) [@georgeg](https://discourse.haproxy.org/u/georgeg)\
**Post date:** [June 17, 2024, 12:32pm UTC](https://discourse.haproxy.org/t/acl-for-bad-user-agents-with-substring-match-fails/9907/7 "2024-06-17T12:32:50Z")

</div>

Hello!

Indeed this seems to resolve the issue!  
According to our initial tests we dont get false 403s with user-agents including “Discob, Discobo” etc, and it does work properly while using the user-agents with their complete names.

Thank you all for your help!