Rate limiting by host (domain)

Help!
1

Hi there,

I’m wondering if there is a way to set per-host (that is: per domain) rate-limiting in HAProxy, using maps?

My frontend setup is as follows (it’s essentially the example given on the HAProxy website):

# Create a 100,000-strong, ten-second expiry stick table that tracks HTTP requests over a sliding ten second window
    stick-table  type binary  len 8  size 100k  expire 10s  store http_req_rate(10s)
    # Track client by base32+src (Host header + URL path + src IP)
    http-request track-sc0 base32+src
    # Check map file to get rate limit for paths; default to 200 for all others
    http-request set-var(req.rate_limit)  path,map_beg(/etc/haproxy/rates.map,200)
    # Ensure that the client's request rate is tracked
    http-request set-var(req.request_rate)  base32+src,table_http_req_rate()
    # Subtract the current request rate from the limit; if less than zero, set rate_abuse to true
    acl rate_abuse var(req.rate_limit),sub(req.request_rate) lt 0
    # If rate abuse is detected, give status 429
    http-request deny deny_status 429 if rate_abuse

In the maps file, I can set rates per path like this:

/path1 20
/path2 10

But what I’d really like to be able to do is set them per domain, too. As it is, the paths in the example above apply to both foo.com and bar.com. Ideally, I’d like to be able to set different limits on foo.com/path1 and bar.com/path1. Is there any way of doing this within the same frontend?

Thanks!

Charles

2

From what I can see, the answer to this is either to create a new map for each domain, as such:

    # Check map file to get rate limit for paths; default to 200 for all others
    http-request set-var(req.rate_limit)  path,map_beg(/etc/haproxy/maps/rates.map,200)
    # For domain2.com, check map file to get rate limit for paths; default to 600 for all others
    acl host_domain2 hdr(host) -i domain2.com
    http-request set-var(req.rate_limit)  path,map_beg(/etc/haproxy/maps/domain2.com-rates.map,600) if host_domain2

Or, if you only want to change the default number, but want to keep the same path variables for each domain, you can use the same map. So:

    # Check map file to get rate limit for paths; default to 200 for all others
    http-request set-var(req.rate_limit)  path,map_beg(/etc/haproxy/maps/rates.map,200)
    # For domain2.com, check map file to get rate limit for paths; default to 600 for all others
    acl host_domain2 hdr(host) -i domain2.com
    http-request set-var(req.rate_limit)  path,map_beg(/etc/haproxy/maps/rates.map,600) if host_domain2

I’m assuming that I can’t define per domain in the map file itself, but would be interested to know if I’m wrong.

3

You can use the base sample fetch for your map lookups instead of path, and in your map your keys are domain/path instead of path.

4

Thank you. Just to make sure I’m understanding correctly. I’d do, e.g.:

http-request set-var(req.rate_limit)  base,map_beg(/etc/haproxy/maps/rates.map,200)

And then in the map, e.g.:

domain1.com/path1 10
domain2.com/path1 20

Presumably, this means that every single path in the map needs to have a domain attached, even if identical?

5

Thank you. Just to make sure I’m understanding correctly. I’d do, e.g.:

correct.

Presumably, this means that every single path in the map needs to have a domain attached, even if identical?

if some of your limits are not per domain you can use base if you found it in the map, or path.

http-request set-var(req.rate_limit)  base,map_beg(/etc/haproxy/maps/rates.map) if { base,map_beg(/etc/haproxy/maps/rates.map -m found }
http-request set-var(req.rate_limit)  path,map_beg(/etc/haproxy/maps/rates.map,200) unless { base,map_beg(/etc/haproxy/maps/rates.map -m found }

you can use the same map file for both, or different files.

6

Thank you. I appreciate your help.

7

Final question (promise!). The above example allows me to set paths that apply to all domains, but also to set specific paths for a domain.

As such, if I wanted a different default rate for a given domain, I could put it in the maps file like this:

domain.com 40

But if I do that, that domain will stop inheriting all the generally applicable path rules.

Is there a trick I can use that will allow me to:

  • Set path rules that apply to every domain;
  • Set domain-specific rules that override those general path rules;
  • Set a different default value for a domain without negating all the general path rules
8

If path must have precedence over domain, then look for path, if you don’t find it look for base, and default to a value of your choosing if you don’t find base.

  http-request set-var(req.rl) path,map_beg(./bar.map) if  { path,map_beg(./bar.map) -m found }
  http-request set-var(req.rl) base,map_beg(./bar.map,200) if ! { var(req.rl) -m found }
9

That makes perfect sense. Thank you!

10

It made sense to me and then I tried it and it didn’t work. I’ll take another look tomorrow after some rest.

11

Ah! Thanks.

12

It’s easy with a separate map for the domains. I can’t find a solution with a single map file.

  http-request set-var(req.rl) base,map_beg(./bar.map) if { base,map_beg(./bar.map) -m found }
  http-request set-var(req.rl) path,map_beg(./bar.map) if { path,map_beg(./bar.map) -m found } !{ var(req.rl) -m found }
  http-request set-var(req.rl) base,map_beg(./bar.dom.map,200) if !{ var(req.rl) -m found }
13

Thank you.