Server is DOWN, reason: Layer7 wrong status, code: 403

Help!
1

Hello,
I am running HAproxy version 2.3.0
My backend servers are listening on port 9020.
Connection to the haproxy(ecsproxy.mydomain) port 9020 failed.

And I’ve got the following errors in my log file:
[WARNING] 351/121109 (19213) : Server web_servers/secsdata01 is DOWN, reason: Layer7 wrong status, code: 403, info: “Forbidden”, check duration: 1ms. 6 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 351/121111 (19213) : backend ‘web_servers’ has no server available!

  1. Proxy version:
    ecsproxy# haproxy -v
    HA-Proxy version 2.3.0-1c0a722 2020/11/05 - https://haproxy.org/
    Running on: Linux 3.10.0-1160.2.2.el7.x86_64 #1 SMP Sat Oct 17 05:06:47 UTC 2020 x86_64

  2. My HAproxy configuration
    #---- Global settings
    global
    maxconn 256
    user haproxy
    group haproxy
    log 127.0.0.1 local2
    chroot /var/lib/haproxy
    pidfile /var/run/haproxy.pid
    daemon
    #---- Defaults settings
    defaults HTTP
    log global
    mode http
    option httplog
    option dontlognull
    timeout connect 5s
    timeout client 50s
    timeout server 50s
    timeout http-request 10s
    timeout http-keep-alive 10s
    option forwardfor
    balance leastconn
    #---- Frontend Definitions for Object Access
    frontend http-in
    bind *:80
    default_backend web_servers
    # Define the hostnames
    acl host_s3 hdr(host) -i -m dom ecsproxy.mydomain

    # Route to backend
use_backend web_servers if host_s3

#---- Backend Definitions
backend web_servers
# Use the s3 Ping extension to make sure ECS services on node arealive.
option httpchk
http-check connect
http-check send meth GET uri / ver HTTP/1.1 hdr host ecsproxy.mydomain
http-check expect status 200-399

    # List of data nodes.  S3 runs on port 9020.
    server secsdata01 192.168.143.53:9020 check
    server secsdata02 192.168.143.54:9020 check
    server secsdata03 192.168.143.55:9020 check
    server secsdata04 192.168.143.56:9020 check
    server secsdata05 192.168.143.57:9020 check
    server secsdata06 192.168.143.58:9020 check

  1. Log file:
    Using epoll() as the polling mechanism.
    [WARNING] 351/121109 (19213) : Server web_servers/secsdata01 is DOWN, reason: Layer7 wrong status, code: 403, info: “Forbidden”, check duration: 1ms. 6 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    [WARNING] 351/121109 (19213) : Server web_servers/secsdata02 is DOWN, reason: Layer7 wrong status, code: 403, info: “Forbidden”, check duration: 1ms. 5 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    [WARNING] 351/121109 (19213) : Server web_servers/secsdata03 is DOWN, reason: Layer7 wrong status, code: 403, info: “Forbidden”, check duration: 1ms. 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    [WARNING] 351/121109 (19213) : Server web_servers/secsdata04 is DOWN, reason: Layer7 wrong status, code: 403, info: “Forbidden”, check duration: 1ms. 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    [WARNING] 351/121109 (19213) : Server web_servers/secsdata05 is DOWN, reason: Layer7 wrong status, code: 403, info: “Forbidden”, check duration: 1ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    [WARNING] 351/121109 (19213) : Server web_servers/secsdata06 is DOWN, reason: Layer7 wrong status, code: 403, info: “Forbidden”, check duration: 3ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    [NOTICE] 351/121111 (19213) : haproxy version is 2.3.0-1c0a722
    [NOTICE] 351/121111 (19213) : path to executable is /usr/local/sbin/haproxy
    [ALERT] 351/121111 (19213) : backend ‘web_servers’ has no server available!

  2. Check port 9020 on backend: successsful
    ecsproxy# telnet secsdata01 9020
    Trying 192.168.143.53…
    Connected to secsdata01.
    Escape character is ‘^]’.
    ^]
    telnet>

  3. Check HAproxy port 9020: failed
    root@kaya:~# telnet ecsproxy 9020
    Trying 192.168.160.115…
    telnet: Unable to connect to remote host: Connection refused

Please, how can I configure my haproxy to set the backend servers UP?
Any help would be appreciated.

Benarson

2

Hello,

The log seems clear, your backends respond with a 403 http code which is considered as not healthy for haproxy.

You could check on your backends logs and adapt your configuration to allow default haproxy health checks or you could tune health check to requests in a way your backend allows.

3

Hello rhada,
thank you ofr your answer.
I will check the backend servers.