Find the server behind but shows 503 SC

Help!
1

I have 2 minio servers with the following config:

backend s3_ssl_back
  mode http
  balance roundrobin
  option redispatch
  option httpchk GET /minio/health/live
  retries 3
  timeout connect 5s
  timeout server 31s
  server ops-object01 192.168.210.233:80 check port 80 inter 5s fall 4 rise 3 verify none
  server ops-object02 192.168.210.140:80 check port 80 inter 5s fall 4 rise 3 verify none

And from the HAProxy server I have no issues connecting to both of the servers (using minio client), also the health checks from HAProxy shows them fine and up with L7 check.
But then the errors I see are the following:

s3_ssl_back/ops-object01 0/3003/-1/-1/3004 503 237 - - SC-- 172/119/0/0/3 0/0 "GET /minio/admin/v3/info HTTP/1.1

What could be wrong here? It use to be before with 443 over TLS, now I have an issue with SSL on the servers that I had to disable it. Also HAProxy and both backend servers are in the same LAN.

on the HAProxy server, doing:

[root@opsproxy01 haproxy]# curl https://s3.domain.de
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

also is not allowed. I would appreciate any help.

Thanks.

2

As per the docs:

https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#8.5

 SC   The server or an equipment between it and haproxy explicitly refused
      the TCP connection (the proxy received a TCP RST or an ICMP message
      in return). Under some circumstances, it can also be the network
      stack telling the proxy that the server is unreachable (e.g. no route,
      or no ARP response on local network). When this happens in HTTP mode,
      the status code is likely a 502 or 503 here.

I understand. Capturing the traffic between haproxy and your backend servers will be required to understand where the problem comes from.

Also post the outputs of:

curl -vv -H "Host: s3.domain.de" "192.168.210.233/minio/admin/v3/info"
curl -vv -H "Host: s3.domain.de" "192.168.210.140/minio/admin/v3/info"
3

The outputs look fine, the last curl shows an endpoint which doesn’t need authentication:

[root@opsproxy01 haproxy]# curl -vv -H "Host: s3.domain.de" "192.168.210.233/minio/admin/v3/info"
* About to connect() to 192.168.210.233 port 80 (#0)
*   Trying 192.168.210.233...
* Connected to 192.168.210.233 (192.168.210.233) port 80 (#0)
> GET /minio/admin/v3/info HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host: s3.domain.de
> 
< HTTP/1.1 403 Forbidden
< Accept-Ranges: bytes
< Content-Length: 164
< Content-Security-Policy: block-all-mixed-content
< Content-Type: application/json
< Server: MinIO/RELEASE.2020-10-28T08-16-50Z
< Vary: Origin
< X-Amz-Request-Id: 16422CC5B1785647
< X-Xss-Protection: 1; mode=block
< Date: Wed, 28 Oct 2020 14:00:44 GMT
< 
{"Code":"AccessDenied","Message":"Access Denied.","Resource":"/minio/admin/v3/info","RequestId":"16422CC5B1785647","HostId":"50f73ea1-7aa8-4b55-8b58-9776bac7048a"}
* Connection #0 to host 192.168.210.233 left intact
[root@opsproxy01 haproxy]# curl -vv -H "Host: s3.domain.de" "192.168.210.140/minio/admin/v3/info"
* About to connect() to 192.168.210.140 port 80 (#0)
*   Trying 192.168.210.140...
* Connected to 192.168.210.140 (192.168.210.140) port 80 (#0)
> GET /minio/admin/v3/info HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host: s3.domain.de
> 
< HTTP/1.1 403 Forbidden
< Accept-Ranges: bytes
< Content-Length: 164
< Content-Security-Policy: block-all-mixed-content
< Content-Type: application/json
< Server: MinIO/RELEASE.2020-10-28T08-16-50Z
< Vary: Origin
< X-Amz-Request-Id: 16422CC85EDC549C
< X-Xss-Protection: 1; mode=block
< Date: Wed, 28 Oct 2020 14:00:56 GMT
< 
{"Code":"AccessDenied","Message":"Access Denied.","Resource":"/minio/admin/v3/info","RequestId":"16422CC85EDC549C","HostId":"50f73ea1-7aa8-4b55-8b58-9776bac7048a"}
* Connection #0 to host 192.168.210.140 left intact
[root@opsproxy01 haproxy]# curl -vv -H "Host: s3.domain.de" "192.168.210.140/minio/health/live
> "
* Illegal characters found in URL
* Closing connection -1
curl: (3) Illegal characters found in URL
[root@opsproxy01 haproxy]# curl -vv -H "Host: s3.domain.de" "192.168.210.140/minio/health/live"
* About to connect() to 192.168.210.140 port 80 (#0)
*   Trying 192.168.210.140...
* Connected to 192.168.210.140 (192.168.210.140) port 80 (#0)
> GET /minio/health/live HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host: s3.domain.de
> 
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 0
< Content-Security-Policy: block-all-mixed-content
< Server: MinIO/RELEASE.2020-10-28T08-16-50Z
< Vary: Origin
< X-Amz-Request-Id: 16422CD3312E117B
< X-Xss-Protection: 1; mode=block
< Date: Wed, 28 Oct 2020 14:01:42 GMT
< 
* Connection #0 to host 192.168.210.140 left intact

Here is a tcpdump on haproxy filtering by one of the backend’s IP:

[root@opsproxy01 ~]# tcpdump -i team0  tcp port 80|grep 192.168.210.233
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on team0, link-type EN10MB (Ethernet), capture size 262144 bytes


15:04:45.532071 IP opsproxy01.domain.de.28342 > 192.168.210.233.http: Flags [S], seq 2228800307, win 29200, options [mss 1460,sackOK,TS val 15764215 ecr 0,nop,wscale 9], length 0
15:04:45.532159 IP 192.168.210.233.http > opsproxy01.domain.de.28342: Flags [S.], seq 1192694361, ack 2228800308, win 28644, options [mss 9560,sackOK,TS val 4229312 ecr 15764215,nop,wscale 7], length 0
15:04:45.532181 IP opsproxy01.domain.de.28342 > 192.168.210.233.http: Flags [.], ack 1, win 58, options [nop,nop,TS val 15764215 ecr 4229312], length 0
15:04:45.532194 IP opsproxy01.domain.de.28342 > 192.168.210.233.http: Flags [P.], seq 1:37, ack 1, win 58, options [nop,nop,TS val 15764215 ecr 4229312], length 36: HTTP: HEAD /minio/health/live HTTP/1.0
15:04:45.532230 IP 192.168.210.233.http > opsproxy01.domain.de.28342: Flags [.], ack 37, win 224, options [nop,nop,TS val 4229313 ecr 15764215], length 0
15:04:45.532666 IP 192.168.210.233.http > opsproxy01.domain.de.28342: Flags [P.], seq 1:275, ack 37, win 224, options [nop,nop,TS val 4229313 ecr 15764215], length 274: HTTP: HTTP/1.0 200 OK
15:04:45.532677 IP opsproxy01.domain.de.28342 > 192.168.210.233.http: Flags [.], ack 275, win 60, options [nop,nop,TS val 15764216 ecr 4229313], length 0
15:04:45.532685 IP 192.168.210.233.http > opsproxy01.domain.de.28342: Flags [F.], seq 275, ack 37, win 224, options [nop,nop,TS val 4229313 ecr 15764215], length 0
15:04:45.532710 IP opsproxy01.domain.de.28342 > 192.168.210.233.http: Flags [F.], seq 37, ack 276, win 60, options [nop,nop,TS val 15764216 ecr 4229313], length 0
15:04:45.532745 IP 192.168.210.233.http > opsproxy01.domain.de.28342: Flags [.], ack 38, win 224, options [nop,nop,TS val 4229313 ecr 15764216], length 0
^C^C^C^C^C^C85 packets captured
373 packets received by filter
216 packets dropped by kernel

All of the servers involved here are physical and I see no errors on their interfaces. Also all the servers here have LACP connected to the same hardware (extreme network black diamond switch)

4

I see the above traffic is being generated by the healthcheck to the backend, but actual traffic never leaves haproxy

5

Here is the traffic between my machine (generating traffic towards minio servers through haproxy) (443 to haproxy, and 80 to the backend:

[root@opsproxy01 ~]# tcpdump -i team0  |grep alatxa.pool.domain.de.
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on team0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:23:51.773235 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [S], seq 1028973919, win 65025, options [mss 1275,sackOK,TS val 1383309940 ecr 0,nop,wscale 7], length 0
16:23:51.773256 IP opsproxy01.domain.de.https > alatxa.pool.domain.de.35800: Flags [S.], seq 590334942, ack 1028973920, win 28960, options [mss 1460,sackOK,TS val 20509865 ecr 1383309940,nop,wscale 9], length 0
16:23:51.801768 IP dnsint3.domain.de.domain > opsproxy01.domain.de.59119: 8323* 1/1/1 PTR alatxa.pool.domain.de. (114)
16:23:51.872103 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [.], ack 1, win 509, options [nop,nop,TS val 1383310006 ecr 20509865], length 0
16:23:51.872391 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [P.], seq 1:259, ack 1, win 509, options [nop,nop,TS val 1383310006 ecr 20509865], length 258
16:23:51.874774 IP opsproxy01.domain.de.https > alatxa.pool.domain.de.35800: Flags [.], seq 1:2527, ack 259, win 59, options [nop,nop,TS val 20509967 ecr 1383310006], length 2526
16:23:51.874786 IP opsproxy01.domain.de.https > alatxa.pool.domain.de.35800: Flags [.], seq 2527:5053, ack 259, win 59, options [nop,nop,TS val 20509967 ecr 1383310006], length 2526
16:23:51.874797 IP opsproxy01.domain.de.https > alatxa.pool.domain.de.35800: Flags [P.], seq 5053:6387, ack 259, win 59, options [nop,nop,TS val 20509967 ecr 1383310006], length 1334
16:23:51.942407 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [.], ack 3790, win 503, options [nop,nop,TS val 1383310115 ecr 20509967], length 0
16:23:51.942461 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [.], ack 2527, win 494, options [nop,nop,TS val 1383310115 ecr 20509967], length 0
16:23:51.942513 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [.], ack 1, win 509, options [nop,nop,TS val 1383310115 ecr 20509865,nop,nop,sack 1 {1264:2527}], length 0
16:23:51.971910 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [.], ack 3790, win 503, options [nop,nop,TS val 1383310116 ecr 20509967,nop,nop,sack 1 {6316:6387}], length 0
16:23:51.971929 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [.], ack 5053, win 500, options [nop,nop,TS val 1383310117 ecr 20509967,nop,nop,sack 1 {6316:6387}], length 0
16:23:51.971935 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [.], ack 6387, win 494, options [nop,nop,TS val 1383310117 ecr 20509967], length 0
16:23:51.977793 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [P.], seq 259:385, ack 6387, win 503, options [nop,nop,TS val 1383310131 ecr 20509967], length 126
16:23:51.978172 IP opsproxy01.domain.de.https > alatxa.pool.domain.de.35800: Flags [P.], seq 6387:6438, ack 385, win 59, options [nop,nop,TS val 20510070 ecr 1383310131], length 51
16:23:52.042882 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [P.], seq 385:878, ack 6438, win 503, options [nop,nop,TS val 1383310211 ecr 20510070], length 493
16:23:52.082462 IP opsproxy01.domain.de.https > alatxa.pool.domain.de.35800: Flags [.], ack 878, win 61, options [nop,nop,TS val 20510175 ecr 1383310211], length 0
16:23:55.046784 IP opsproxy01.domain.de.https > alatxa.pool.domain.de.35800: Flags [P.], seq 6438:6700, ack 878, win 61, options [nop,nop,TS val 20513139 ecr 1383310211], length 262
16:23:55.046804 IP opsproxy01.domain.de.https > alatxa.pool.domain.de.35800: Flags [P.], seq 6700:6731, ack 878, win 61, options [nop,nop,TS val 20513139 ecr 1383310211], length 31
16:23:55.046815 IP opsproxy01.domain.de.https > alatxa.pool.domain.de.35800: Flags [F.], seq 6731, ack 878, win 61, options [nop,nop,TS val 20513139 ecr 1383310211], length 0
16:23:55.211264 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [.], ack 6438, win 503, options [nop,nop,TS val 1383313384 ecr 20510175,nop,nop,sack 1 {6700:6731}], length 0
16:23:55.211329 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [.], ack 6732, win 503, options [nop,nop,TS val 1383313384 ecr 20513139], length 0
16:23:55.211485 IP alatxa.pool.domain.de.35800 > opsproxy01.domain.de.https: Flags [.], ack 6438, win 503, options [nop,nop,TS val 1383313384 ecr 20510175,nop,nop,sack 1 {6700:6732}], length 0
16:23:55.431707 IP alatxa.pool.domain.de.35802 > opsproxy01.domain.de.https: Flags [.], ack 792578726, win 503, options [nop,nop,TS val 1383313602 ecr 20513381,nop,nop,sack 1 {4294963437:4294964700}], length 0
16:23:55.431722 IP alatxa.pool.domain.de.35802 > opsproxy01.domain.de.https: Flags [.], ack 1, win 503, options [nop,nop,TS val 1383313602 ecr 20513381,nop,nop,sack 1 {4294965963:4294967226}], length 0
16:23:55.432078 IP alatxa.pool.domain.de.35802 > opsproxy01.domain.de.https: Flags [.], ack 1, win 503, options [nop,nop,TS val 1383313602 ecr 20513381,nop,nop,sack 1 {4294960911:4294962174}], length 0
16:23:55.432090 IP alatxa.pool.domain.de.35802 > opsproxy01.domain.de.https: Flags [.], ack 1, win 503, options [nop,nop,TS val 1383313602 ecr 20513381,nop,nop,sack 1 {4294962174:4294963437}], length 0
16:23:55.432093 IP alatxa.pool.domain.de.35802 > opsproxy01.domain.de.https: Flags [.], ack 52, win 503, options [nop,nop,TS val 1383313602 ecr 20513459], length 0
^C^C^C^C^C^C^C^C^C17698 packets captured