# HAPROXY with AWS AUTOSCALING

**URL:** <https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902>\
**Category:** Help!\
**Created:** [August 22, 2018, 12:44pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902 "2018-08-22T12:44:56Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![andrewgricuk](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@andrewgricuk](https://discourse.haproxy.org/u/andrewgricuk)\
**Post date:** [August 22, 2018, 12:44pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/1 "2018-08-22T12:44:56Z")

</div>

hi guys!  
I want to use haproxy with AWS autoscaling groups. I’m using balancing by ID (hash-consistency). I ran into the following problem:  
When autoscaling group UP Haproxy reloaded clear cache (hash tables) and we have a problem with connecting to needed instance. The students pass exam and send request with ID to haproxy -\> Haproxy send request to the backend and associate hash with backend -\> proctor which should view the exam send the request with same ID and goes to same backend. How i can save hash state when scaling up? Thanks

haproxy -v  
HA-Proxy version 1.8.13-1ppa1~xenial 2018/08/01  
Copyright 2000-2018 Willy Tarreau \<willy@haproxy.

global  
log /dev/log local0  
log /dev/log local1 notice  
chroot /var/lib/haproxy  
server-state-file global  
server-state-base /run/haproxy/server-state/  
stats socket /run/haproxy/admin.sock mode 660 level admin  
stats timeout 30s  
user haproxy  
group haproxy  
daemon  
# Default SSL material locations  
ca-base /etc/ssl/certs  
crt-base /etc/ssl/private  
tune.ssl.default-dh-param 2048  
# Default ciphers to use on SSL-enabled listening sockets.  
# For more information, see ciphers(1SSL). This list is from:  
# [https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/](https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/)  
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS  
ssl-default-bind-options no-sslv3  
node lb  
nbproc 1  
maxconn 2000000

defaults  
log global  
mode http  
option forwardfor  
option http-server-close  
timeout connect 5000  
timeout client 50000  
timeout server 50000  
errorfile 400 /etc/haproxy/errors/400.http  
errorfile 403 /etc/haproxy/errors/403.http  
errorfile 408 /etc/haproxy/errors/408.http  
errorfile 500 /etc/haproxy/errors/500.http  
errorfile 502 /etc/haproxy/errors/502.http  
errorfile 503 /etc/haproxy/errors/503.http  
errorfile 504 /etc/haproxy/errors/504.http  
load-server-state-from-file global

frontend www-http  
bind \*:80  
redirect scheme https code 301 if !{ ssl\_fc }  
default\_backend java  
maxconn 2000000

frontend www-https  
bind \*:443 ssl crt /etc/ssl/private/test.pem  
acl is\_websocket hdr(Upgrade) -i WebSocket  
acl is\_websocket hdr\_beg(Host) -i ws  
capture request header origin len 50  
default\_backend java  
maxconn 2000000

backend java  
balance url\_param fid  
hash-type consistent  
option forwardfor # This sets X-Forwarded-For  
http-request set-header X-Forwarded-Port %[dst\_port]  
http-request add-header X-Forwarded-Proto https if { ssl\_fc }  
http-response set-header Access-Control-Allow-Origin %[capture.req.hdr(0)]  
option httpchk  
default-server inter 3s fall 3 rise 2  
option httpchk HEAD /students HTTP/1.0

# autoscaling group instances will be

# dynamically added below

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 22, 2018, 1:49pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/2 "2018-08-22T13:49:39Z")

</div>

> [@andrewgricuk](#):
>
> How i can save hash state when scaling up?

Configure [peers](https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.5) for cross-instance hash-table synchronization.

---

<div class="post-metadata">

**Author:** ![andrewgricuk](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@andrewgricuk](https://discourse.haproxy.org/u/andrewgricuk)\
**Post date:** [August 22, 2018, 1:52pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/3 "2018-08-22T13:52:00Z")

</div>

> [@lukastribus](#):
>
> > [@andrewgricuk](#):
> >
> > How i can save hash state when scaling up?
> 
> Configure [peers](https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.5) for cross-instance hash-table synchronization.

But if I have only 1 instance with HAproxy?

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 22, 2018, 1:57pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/4 "2018-08-22T13:57:22Z")

</div>

If you have just one instance you can specify the local hostname, and it will talk to itself (check the documentation from earlier), so a haproxy reload will maintain the hash-table.

Not sure if that answers your question at this point. I don’t know what AWS autoscaling is and how it works, so maybe you can explain what happens at system and userspace level?

---

<div class="post-metadata">

**Author:** ![andrewgricuk](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@andrewgricuk](https://discourse.haproxy.org/u/andrewgricuk)\
**Post date:** [August 22, 2018, 1:59pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/5 "2018-08-22T13:59:13Z")

</div>

> [@lukastribus](#):
>
> If you have just one instance you can specify the local hostname, and it will talk to itself (check the documentation from earlier), so a haproxy reload will maintain the hash-table.
> 
> Not sure if that answers your question at this point. I don’t know what AWS autoscaling is and how it works, so maybe you can explain what happens at system and userspace level?

I’ve ruby script which updated haproxy.cfg (added and removed new backends) and reload haproxy

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 22, 2018, 2:04pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/6 "2018-08-22T14:04:56Z")

</div>

Ok, so peers will solve your issue then, as explained.

---

<div class="post-metadata">

**Author:** ![andrewgricuk](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@andrewgricuk](https://discourse.haproxy.org/u/andrewgricuk)\
**Post date:** [August 22, 2018, 2:45pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/7 "2018-08-22T14:45:33Z")

</div>

> [@lukastribus](#):
>
> Ok, so peers will solve your issue then, as explained.

I’m sorry, but how I can stick by url\_parameter?

backend java  
balance url\_param fid  
hash-type consistent  
stick-table type string len 64 size 10k expire 8h  
stick on url\_param(???) table simulate

Request url looks like [https://domain.com/recording&fid=123123jsd21312](https://domain.com/recording&fid=123123jsd21312)

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 22, 2018, 3:01pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/8 "2018-08-22T15:01:26Z")

</div>

I was under the assumption that you already have a working configuration, and that you need to find a solution to your autoscaling problem.

Do you not have a working configuration right now?

---

<div class="post-metadata">

**Author:** ![andrewgricuk](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@andrewgricuk](https://discourse.haproxy.org/u/andrewgricuk)\
**Post date:** [August 22, 2018, 3:04pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/9 "2018-08-22T15:04:17Z")

</div>

> [@lukastribus](#):
>
> I was under the assumption that you already have a working configuration, and that you need to find a solution to your autoscaling problem.
> 
> Do you not have a working configuration right now?

The configuration which I’ve sent in my first message works fine , but when scaling up i get a problem

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 22, 2018, 7:15pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/10 "2018-08-22T19:15:45Z")

</div>

I don’t understand what you are asking, sorry.

Please explain your problem again, explain what you actually want haproxy to do, and avoid the [xy-problem](http://xyproblem.info/).

---

<div class="post-metadata">

**Author:** ![andrewgricuk](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@andrewgricuk](https://discourse.haproxy.org/u/andrewgricuk)\
**Post date:** [August 22, 2018, 8:05pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/11 "2018-08-22T20:05:28Z")

</div>

> [@lukastribus](#):
>
> I don’t understand what you are asking, sorry.
> 
> Please explain your problem again, explain what you actually want haproxy to do, and avoid the [xy-problem](http://xyproblem.info/).

I’ve attached my config file in my first message. Now Haproxy is balancing by hash (url\_parameter) and it works fine with any number of the backends  
The problem is when scaling up and new backends is adding to haproxy conf. My script check autoscaling groups and add new backends to haproxy configuration then haproxy is reloading. And hash table which associate hash with the backend is cleared.  
How i can add new backend instances without loses hash data?

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 22, 2018, 8:09pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/12 "2018-08-22T20:09:38Z")

</div>

Ok, but why do you need to load-balance based on the URL? What is the reason you cannot achieve stickiness with cookies, for example, or with source-IP?

---

<div class="post-metadata">

**Author:** ![andrewgricuk](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@andrewgricuk](https://discourse.haproxy.org/u/andrewgricuk)\
**Post date:** [August 22, 2018, 8:21pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/13 "2018-08-22T20:21:46Z")

</div>

> [@lukastribus](#):
>
> Ok, but why do you need to load-balance based on the URL? What is the reason you cannot achieve stickiness with cookies, for example, or with source-IP?

Students passes exams in our system. When exam starts students sent a request with ID to HaProxy which associates ID with backend. Also we have a proctors who should monitor exam. In this way when proctor openned exam he sent request to haproxy with same ID and he can view student session.

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 22, 2018, 10:58pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/14 "2018-08-22T22:58:57Z")

</div>

> [@andrewgricuk](#):
>
> ```
> backend java
> balance url_param fid
> hash-type consistent
> stick-table type string len 64 size 10k expire 8h
> stick on url_param(???) table simulate
> 
> ```

Remove:

```
balance url_param fid
hash-type consistent

```

After all, you don’t want hashing, you are using the stick table to achieve this.

> [@andrewgricuk](#):
>
> Request url looks like [https://domain.com/recording&amp;fid=123123jsd21312](https://domain.com/recording&amp;fid=123123jsd21312)

As there is no question mark in this URL, you need to set the [url\_param delimiter](https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#7.3.6-url_param) to `&`:

```
stick-table type string len 64 size 10k expire 8h
stick on url_param(fid,&)

```

And of course, setup the peers section so that the stick-table is kept while reloading.

---

<div class="post-metadata">

**Author:** ![andrewgricuk](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@andrewgricuk](https://discourse.haproxy.org/u/andrewgricuk)\
**Post date:** [August 23, 2018, 8:49am UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/15 "2018-08-23T08:49:43Z")

</div>

Hi, my finally config looks like

global  
log /dev/log local0  
log /dev/log local1 notice  
chroot /var/lib/haproxy  
server-state-file global  
server-state-base /run/haproxy/server-state/  
stats socket /run/haproxy/admin.sock mode 660 level admin  
stats timeout 30s  
user haproxy  
group haproxy  
daemon  
# Default SSL material locations  
ca-base /etc/ssl/certs  
crt-base /etc/ssl/private  
tune.ssl.default-dh-param 2048  
# Default ciphers to use on SSL-enabled listening sockets.  
# For more information, see ciphers(1SSL). This list is from:  
# [https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/](https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/)  
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS  
ssl-default-bind-options no-sslv3  
node lb  
nbproc 2  
maxconn 2000000  
defaults  
log global  
mode http  
option forwardfor  
option http-server-close  
timeout connect 5000  
timeout client 50000  
timeout server 50000  
errorfile 400 /etc/haproxy/errors/400.http  
errorfile 403 /etc/haproxy/errors/403.http  
errorfile 408 /etc/haproxy/errors/408.http  
errorfile 500 /etc/haproxy/errors/500.http  
errorfile 502 /etc/haproxy/errors/502.http  
errorfile 503 /etc/haproxy/errors/503.http  
errorfile 504 /etc/haproxy/errors/504.http  
load-server-state-from-file global  
frontend www-http  
bind \*:80  
bind-process 1,2  
redirect scheme https code 301 if !{ ssl\_fc }  
default\_backend java  
maxconn 2000000  
frontend www-https  
bind-process 1,2  
bind \*:443 ssl crt /etc/ssl/private/tests.com.pem  
acl is\_websocket hdr(Upgrade) -i WebSocket  
acl is\_websocket hdr\_beg(Host) -i ws  
capture request header origin len 50  
default\_backend java  
maxconn 2000000  
peers article  
peer ithcy 127.0.0.1:1023

backend java  
bind-process 1,2  
stick-table type string len 512 size 20k expire 8h peers article  
stick on url\_param(fid,?)  
option forwardfor # This sets X-Forwarded-For  
http-request set-header X-Forwarded-Port %[dst\_port]  
http-request add-header X-Forwarded-Proto https if { ssl\_fc }  
http-response set-header Access-Control-Allow-Origin %[capture.req.hdr(0)]  
option httpchk  
default-server inter 3s fall 3 rise 2  
option httpchk HEAD /students HTTP/1.0

; autoscaling group instances will be  
;dynamically added below

**It works fine , when I’ve reloaded haproxy and count of the backend instances doesn’t change.**  
**But If I run script which added new backends to config and reloaded haproxy it doen’t work(. Haproxy begin send requests to other instances and proctor doesn’t go to needed student**

---

<div class="post-metadata">

**Author:** ![andrewgricuk](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@andrewgricuk](https://discourse.haproxy.org/u/andrewgricuk)\
**Post date:** [August 23, 2018, 9:17am UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/16 "2018-08-23T09:17:54Z")

</div>

But if I’ve added new backends manually it works without problem.  
I have a template with haproxy config (without backends). My script copys template everytime and added backends it the end of file and run systemctl reload haproxy. Uh, this is very strange

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 23, 2018, 3:09pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/17 "2018-08-23T15:09:37Z")

</div>

Looks like your automation does something that you don’t expect. Finding the difference between the manual, working sequence of events and the automated, failing sequence is something that I can’t help you with.

---

<div class="post-metadata">

**Author:** ![andrewgricuk](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@andrewgricuk](https://discourse.haproxy.org/u/andrewgricuk)\
**Post date:** [August 23, 2018, 4:23pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/18 "2018-08-23T16:23:28Z")

</div>

There are problems when I manual adding backend isntances too.  
How do you think there can be a problem because of the peers is localhost?

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 23, 2018, 5:04pm UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/19 "2018-08-23T17:04:21Z")

</div>

Can you elaborate what “a problem” means exactly?

---

<div class="post-metadata">

**Author:** ![andrewgricuk](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@andrewgricuk](https://discourse.haproxy.org/u/andrewgricuk)\
**Post date:** [August 24, 2018, 8:09am UTC](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902/20 "2018-08-24T08:09:28Z")

</div>

First case: I’ve added 5 backend instances and run video stream, haproxy associate each session with backend and when I’ve reloaded haproxy - sticky tables doesn’t clear.  
Second case: I’ve run video streams then i add new backend instances to config and reload haproxy. After that sticky tables re-associate and new requests spread randomly.

UP: the problem exist when I added new instances and change backends of order than sticky tables are re-associated , when I just add new backends to end of file all ok.

[Next page](https://discourse.haproxy.org/t/haproxy-with-aws-autoscaling/2902.md?page=2)