HAPROXY with AWS AUTOSCALING


#1

hi guys!
I want to use haproxy with AWS autoscaling groups. I’m using balancing by ID (hash-consistency). I ran into the following problem:
When autoscaling group UP Haproxy reloaded clear cache (hash tables) and we have a problem with connecting to needed instance. The students pass exam and send request with ID to haproxy -> Haproxy send request to the backend and associate hash with backend -> proctor which should view the exam send the request with same ID and goes to same backend. How i can save hash state when scaling up? Thanks

haproxy -v
HA-Proxy version 1.8.13-1ppa1~xenial 2018/08/01
Copyright 2000-2018 Willy Tarreau <willy@haproxy.

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
server-state-file global
server-state-base /run/haproxy/server-state/
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
tune.ssl.default-dh-param 2048
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
node lb
nbproc 1
maxconn 2000000

defaults
log global
mode http
option forwardfor
option http-server-close
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
load-server-state-from-file global

frontend www-http
bind *:80
redirect scheme https code 301 if !{ ssl_fc }
default_backend java
maxconn 2000000

frontend www-https
bind *:443 ssl crt /etc/ssl/private/test.pem
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
capture request header origin len 50
default_backend java
maxconn 2000000

backend java
balance url_param fid
hash-type consistent
option forwardfor # This sets X-Forwarded-For
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
http-response set-header Access-Control-Allow-Origin %[capture.req.hdr(0)]
option httpchk
default-server inter 3s fall 3 rise 2
option httpchk HEAD /students HTTP/1.0

autoscaling group instances will be

dynamically added below


#2

Configure peers for cross-instance hash-table synchronization.


#3

But if I have only 1 instance with HAproxy?


#4

If you have just one instance you can specify the local hostname, and it will talk to itself (check the documentation from earlier), so a haproxy reload will maintain the hash-table.

Not sure if that answers your question at this point. I don’t know what AWS autoscaling is and how it works, so maybe you can explain what happens at system and userspace level?


#5

I’ve ruby script which updated haproxy.cfg (added and removed new backends) and reload haproxy


#6

Ok, so peers will solve your issue then, as explained.


#7

I’m sorry, but how I can stick by url_parameter?

backend java
balance url_param fid
hash-type consistent
stick-table type string len 64 size 10k expire 8h
stick on url_param(???) table simulate

Request url looks like https://domain.com/recording&fid=123123jsd21312


#8

I was under the assumption that you already have a working configuration, and that you need to find a solution to your autoscaling problem.

Do you not have a working configuration right now?


#9

The configuration which I’ve sent in my first message works fine , but when scaling up i get a problem


#10

I don’t understand what you are asking, sorry.

Please explain your problem again, explain what you actually want haproxy to do, and avoid the xy-problem.


#11

I’ve attached my config file in my first message. Now Haproxy is balancing by hash (url_parameter) and it works fine with any number of the backends
The problem is when scaling up and new backends is adding to haproxy conf. My script check autoscaling groups and add new backends to haproxy configuration then haproxy is reloading. And hash table which associate hash with the backend is cleared.
How i can add new backend instances without loses hash data?


#12

Ok, but why do you need to load-balance based on the URL? What is the reason you cannot achieve stickiness with cookies, for example, or with source-IP?


#13

Students passes exams in our system. When exam starts students sent a request with ID to HaProxy which associates ID with backend. Also we have a proctors who should monitor exam. In this way when proctor openned exam he sent request to haproxy with same ID and he can view student session.


#14

Remove:

balance url_param fid
hash-type consistent

After all, you don’t want hashing, you are using the stick table to achieve this.

As there is no question mark in this URL, you need to set the url_param delimiter to &:

stick-table type string len 64 size 10k expire 8h
stick on url_param(fid,&)

And of course, setup the peers section so that the stick-table is kept while reloading.


#15

Hi, my finally config looks like

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
server-state-file global
server-state-base /run/haproxy/server-state/
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
tune.ssl.default-dh-param 2048
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
node lb
nbproc 2
maxconn 2000000
defaults
log global
mode http
option forwardfor
option http-server-close
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
load-server-state-from-file global
frontend www-http
bind *:80
bind-process 1,2
redirect scheme https code 301 if !{ ssl_fc }
default_backend java
maxconn 2000000
frontend www-https
bind-process 1,2
bind *:443 ssl crt /etc/ssl/private/tests.com.pem
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
capture request header origin len 50
default_backend java
maxconn 2000000
peers article
peer ithcy 127.0.0.1:1023

backend java
bind-process 1,2
stick-table type string len 512 size 20k expire 8h peers article
stick on url_param(fid,?)
option forwardfor # This sets X-Forwarded-For
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
http-response set-header Access-Control-Allow-Origin %[capture.req.hdr(0)]
option httpchk
default-server inter 3s fall 3 rise 2
option httpchk HEAD /students HTTP/1.0

; autoscaling group instances will be
;dynamically added below

It works fine , when I’ve reloaded haproxy and count of the backend instances doesn’t change.
But If I run script which added new backends to config and reloaded haproxy it doen’t work(. Haproxy begin send requests to other instances and proctor doesn’t go to needed student


#16

But if I’ve added new backends manually it works without problem.
I have a template with haproxy config (without backends). My script copys template everytime and added backends it the end of file and run systemctl reload haproxy. Uh, this is very strange


#17

Looks like your automation does something that you don’t expect. Finding the difference between the manual, working sequence of events and the automated, failing sequence is something that I can’t help you with.


#18

There are problems when I manual adding backend isntances too.
How do you think there can be a problem because of the peers is localhost?


#19

Can you elaborate what “a problem” means exactly?


#20

First case: I’ve added 5 backend instances and run video stream, haproxy associate each session with backend and when I’ve reloaded haproxy - sticky tables doesn’t clear.
Second case: I’ve run video streams then i add new backend instances to config and reload haproxy. After that sticky tables re-associate and new requests spread randomly.

UP: the problem exist when I added new instances and change backends of order than sticky tables are re-associated , when I just add new backends to end of file all ok.