Hey everybody,
I have multiple domain and their corresponding back-end server’s . I want to Use HaProxy as Reverse Proxy for my domains.
whenever I hit that domain it sometime it give me response and sometime it give me 503 HaProxy error,*
below is the log of that error
-
Jun 12 09:50:07 HAProxy haproxy[101105]: 192.168.37.1:48766 [12/Jun/2024:09:50:07.719] http_frontend default_backend/ 0/-1/-1/-1/0 503 216 - - SC-- 1/1/0/0/0 0/0 “GET /industry/advertising-translation-services HTTP/1.1”
-
Jun 12 10:03:53 HAProxy haproxy[101092]: 192.168.37.1:48196 [12/Jun/2024:10:03:50.785] http_frontend default_backend/localhost 0/0/-1/-1/3006 503 216 - - SC-- 1/1/0/0/3 0/0 “GET /industry/advertising-translation-services HTTP/1.1”
after multiple refreshed it works and the log is
- Jun 12 09:41:19 HAProxy haproxy[125808]: 192.168.37.1:54442 [12/Jun/2024:09:41:19.551] http_front servers/qadev 0/0/0/44/45 200 510642 - - ---- 2/2/0/0/0 0/0 “GET /industry/advertising-translation-services HTTP/1.1”
I have spend almost 1.5 week on this error but failed to find the solution.
this is my cnf of Haproxy.
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend marshub_com
bind *:8095
acl ccjkcrmdev.marshub.com hdr(host) -i -m beg ccjkcrmdev.marshub.com
use_backend ccjkcrmdev_backend if ccjkcrmdev.marshub.com
acl logstash.marshub.com hdr(host) -i -m beg logstash.marshub.com
use_backend logstash_backend if logstash.marshub.com
backend ccjkcrmdev_backend
server ccjkcrmdev_server 192.156.65.24:8090 check
backend logstash_backend
server logstash_server 192.156.23.78:8092 check
Kindly please help to resolve that issue, i also tried adding a default back-end server but still same error.
my main goal is to use HaProxy to route request to their corresponding back-end server’s based on the domain name.
It possibile you have parallel haproxy instances still running with an older configuration.
I suggest stopping and killing all haproxy instances and restart it:
systemctl stop haproxy
killall haproxy
sleep 1
killall haproxy
sleep 3
systemctl start haproxy
I don’t have any instances,
I assume you are saying that you used all those commands above and the problem is unchanged.
In this case you’d have to provide the actual configuration that you are running and the failing and working loss related to each other and the configuration you post.
In the first post the configuration posted does not match the log files you provided, and the log files are also tens of minutes a part in different PID’s, so there is really nothing that we can troubleshoot.
Actually this is the original cnf that I am using.
Thats not really possibile as the name do not match.
But anyway, the failure logs have SC code set which means:
SC The server or an equipment between it and HAProxy explicitly refused
the TCP connection (the proxy received a TCP RST or an ICMP message
in return). Under some circumstances, it can also be the network
stack telling the proxy that the server is unreachable (e.g. no route,
or no ARP response on local network). When this happens in HTTP mode,
the status code is likely a 502 or 503 here.
I will tell you the whole scenario,
I have multiple domains and there respective back-end server’s.
firstly I am using nginx as a reverse proxy and now I want to use Haproxy as a reverse proxy.
I have configured HaProxy to act as a reverse proxy
and thats the whole conf below
global
log stdout format raw local0 debug
maxconn 4096
tune.bufsize 3276800 # Increase buffer size to 32 KB
tune.maxrewrite 2024 # Increase maximum rewrite size
tune.http.maxhdr 228 # Increase the maximum number of headers
defaults
log global
mode http
option httplog
timeout connect 5m
timeout client 3m
timeout server 3m
retries 5
option log-health-checks
frontend marshub_com
bind *:8095
acl ccjkcrmdev.marshub.com hdr(host) -i -m beg ccjkcrmdev.marshub.com
use_backend ccjkcrmdev_backend if ccjkcrmdev.marshub.com
acl pcp.dev.marshub.com hdr(host) -i -m beg pcp.dev.marshub.com
use_backend pcp_backend if pcp.dev.marshub.com
acl logstash.marshub.com hdr(host) -i -m beg logstash.marshub.com
use_backend logstash_backend if logstash.marshub.com
acl qadev.marstranslation.com hdr(host) -i qadev.marstranslation.com
use_backend qadev_backend if qadev.marstranslation.com
backend ccjkcrmdev_backend
server ccjkcrmdev_server 192.168.76.97:8010 check
backend pcp_backend
server pcp_server 192.168.76.28:8011 check
backend logstash_backend
server logstash_server 192.168.76.86:8012 check
backend qadev_backend
server qadev_server 192.168.76.91:8013 check```
now the main issue is, it sometime through error of 503 and after multiple refreshes it successfully route to that backend server.
I am failed to find the solution why sometime it gives 503 error and sometime it successfully route the request.
I hope you understand my scenerio now :)
It could be good to share the true logs corresponding to this config, all the 503s followed by a 200, without any reload/restart in the mean time. Idea is to check haproxy’s pids and frontend/backend/server name.
Because your are matching on the request host header only, it could be a good idea to first capture it to have in the logs. To do so, you should add following line in your frontend section:
capture request header Host len 64
tune.bufsize is set to 3 MB (not 32 KB as the comment suggests), this should probably be fixed.