Haproxy with Tomcat issue

Help!
1

Hi,

I encounter an issue using Haproxy (1.5.18) with Tomcat

Here a basic description of my test infra:

(server1)10.0.1.238:443 ==> Haproxy ==> 192.168.0.10:10080(/server1) >>>(server2) Tomcat listening on 192.168.0.10:10080 (/server2)

On my server1, an iptables is used as NAT, routing and firewall.

HAproxy returns a 503 Service Unavailable.

Things I’ve already checked/tried
-Redirect port 10080 of my server1 to port 10080 of my server2 ==> it works so my app is running and OK
-Wget from server1 to server2 (bypassing haproxy) ==> it works so no firewall issue
-Install an Apache on server2 listining on port 10080 ==> it works so Haproxy conf seems to be OK
-using TCPDump on both server==> see packet on my server1 but nothing on my server2 so it seems that HAproxy don’t redirect the flow

I have the feeling that Haproxy consider server2 dead and don’t redirect packet.

Do you have any idea of what happens?

Thanks.

2

Show the complete configuration of haproxy, its logs when it’s failing and the tcpdumps you captured.

3

Thanks for your answer.

It doesn’t work for backend back1 and back2

HAProxy.cfg

global
log 127.0.0.1 local2

chroot      /var/lib/haproxy
pidfile     /var/run/haproxy.pid
maxconn     300
user        ************
group       ************
daemon

stats socket /var/lib/haproxy/stats

defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 300
mode http

frontend HTTPS-in-clientcert
bind 10.0.1.237:443 ssl crt *************************************** no-sslv3 no-tlsv10 ca-file ************************************ verify optional crl-file *****************************************
option httplog
use_backend back1 if { ssl_c_verify 0 } { ssl_c_used }

default_backend DENYALL

frontend HTTPS-in-noclientcert
bind 10.0.1.238:443 ssl crt ************************* no-sslv3 no-tlsv10 ca-file *****************************
option httplog
use_backend back2

default_backend DENYALL

backend DENYALL
http-request deny

backend back1
option httpchk
option forwardfor except 127.0.0.1
http-request add-header X-Forwarded-Proto https if { ssl_fc }
server web-server1 server1:10080 maxconn 32

backend back2
option httpchk
option forwardfor except 127.0.0.1
http-request add-header X-Forwarded-Proto https if { ssl_fc }
server web-server1 server2:10080 maxconn 32

TCPDump on server which host Haproxy

tcpdump -n -i ens192 src host 10.0.4.79
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
16:28:33.822581 IP 10.0.4.79.49350 > 10.0.1.238.https: Flags [F.], seq 482250663, ack 1243566557, win 255, length 0
16:28:33.822697 IP 10.0.4.79.49350 > 10.0.1.238.https: Flags [R.], seq 1, ack 1, win 0, length 0
16:28:33.824604 IP 10.0.4.79.49354 > 10.0.1.238.https: Flags [S], seq 137659877, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
16:28:33.824889 IP 10.0.4.79.49355 > 10.0.1.238.https: Flags [S], seq 332126156, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
16:28:33.824951 IP 10.0.4.79.49354 > 10.0.1.238.https: Flags [.], ack 3519986653, win 256, length 0
16:28:33.825102 IP 10.0.4.79.49356 > 10.0.1.238.https: Flags [S], seq 3747773893, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
16:28:33.825144 IP 10.0.4.79.49355 > 10.0.1.238.https: Flags [.], ack 3456042091, win 256, length 0
16:28:33.825272 IP 10.0.4.79.49356 > 10.0.1.238.https: Flags [.], ack 2372076757, win 256, length 0
16:28:33.825706 IP 10.0.4.79.49354 > 10.0.1.238.https: Flags [P.], seq 0:517, ack 1, win 256, length 517
16:28:33.825876 IP 10.0.4.79.49355 > 10.0.1.238.https: Flags [P.], seq 0:517, ack 1, win 256, length 517
16:28:33.828190 IP 10.0.4.79.49356 > 10.0.1.238.https: Flags [P.], seq 0:517, ack 1, win 256, length 517
16:28:33.828399 IP 10.0.4.79.49354 > 10.0.1.238.https: Flags [.], ack 2836, win 256, length 0
16:28:33.830807 IP 10.0.4.79.49355 > 10.0.1.238.https: Flags [.], ack 2836, win 256, length 0
16:28:33.833128 IP 10.0.4.79.49354 > 10.0.1.238.https: Flags [P.], seq 517:643, ack 2836, win 256, length 126
16:28:33.833198 IP 10.0.4.79.49356 > 10.0.1.238.https: Flags [.], ack 2836, win 256, length 0
16:28:33.836083 IP 10.0.4.79.49355 > 10.0.1.238.https: Flags [P.], seq 517:643, ack 2836, win 256, length 126
16:28:33.838680 IP 10.0.4.79.49356 > 10.0.1.238.https: Flags [P.], seq 517:643, ack 2836, win 256, length 126
16:28:33.840195 IP 10.0.4.79.49354 > 10.0.1.238.https: Flags [P.], seq 643:1139, ack 3094, win 255, length 496
16:28:33.840878 IP 10.0.4.79.49354 > 10.0.1.238.https: Flags [.], ack 3367, win 254, length 0
16:28:33.843379 IP 10.0.4.79.49354 > 10.0.1.238.https: Flags [F.], seq 1139, ack 3367, win 254, length 0
16:28:33.859926 IP 10.0.4.79.49355 > 10.0.1.238.https: Flags [P.], seq 643:1094, ack 3094, win 255, length 451
16:28:33.860552 IP 10.0.4.79.49355 > 10.0.1.238.https: Flags [.], ack 3367, win 254, length 0
16:28:33.862859 IP 10.0.4.79.49355 > 10.0.1.238.https: Flags [F.], seq 1094, ack 3367, win 254, length 0
16:28:34.034670 IP 10.0.4.79.49356 > 10.0.1.238.https: Flags [.], ack 3094, win 255, length 0
16:28:43.840983 IP 10.0.4.79.49356 > 10.0.1.238.https: Flags [.], ack 3367, win 254, length 0

TCPDump on server hosting Tomcat

tcpdump -n -i ens192 src host 10.0.4.79
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
nothing…

HAPROXY log

tail -f /var/log/haproxy*.log
Feb 7 16:46:11 localhost haproxy[14726]: Proxy HTTPS-in-clientcert started.
Feb 7 16:46:11 localhost haproxy[14726]: Proxy HTTPS-in-noclientcert started.
Feb 7 16:46:11 localhost haproxy[14726]: Proxy DENYALL started.
Feb 7 16:46:11 localhost haproxy[14726]: Proxy ******************** started.
Feb 7 16:46:11 localhost haproxy[14726]: Proxy ******************** started.
Feb 7 16:46:21 localhost haproxy[14727]: 10.0.4.79:49394 [07/Feb/2018:16:46:21.282] HTTPS-in-noclientcert~ /web-server1 14/0/-1/-1/15 503 212 - - SC-- 2/2/0/0/3 0/0 "GET /***** HTTP/1.1"
Feb 7 16:46:21 localhost haproxy[14727]: 10.0.4.79:49395 [07/Feb/2018:16:46:21.282] HTTPS-in-noclientcert~ **************/web-server1 34/0/-1/-1/34 503 212 - - SC-- 1/1/0/0/3 0/0 “GET /favicon.ico HTTP/1.1”

4

Like I said, we also need logs and capture.

5

I’ve updated my previous post to centralized informations

6

The tomcat server refuses the connection from haproxy, you can see that based on the “SC” error code in the log, which means:

The server or an equipment between it and haproxy explicitly refused
the TCP connection (the proxy received a TCP RST or an ICMP message
in return). Under some circumstances, it can also be the network
stack telling the proxy that the server is unreachable (eg: no route,
or no ARP response on local network). When this happens in HTTP mode,
the status code is likely a 502 or 503 here.

If you want to see the backend server in action refusing the connection from haproxy, you need to focus on the backend traffic, therefor modify the capture filter to “host 192.168.0.10”.

7

Thanks for your really helpful answer.

I’ll investigate and update this post with solution.

8

My server 1 and server 2 are on the same ESX, directly connected on the same vswitch.

When I try an arping, request and reply are well received on both side.

I can also ping both side using name or ip.

I’ve also changed my default iptables rules to “ACCEPT” on both side to be sure that nothing is dropped.

I also try to disable ssl.

Issue still pending.

9

Like I said, modify the tcpdump command and you will see what really happens.

10

Hi,

I’ve finally found the solution.

I had to set “haproxy_connect_any” to 1 to allow haproxy to connect to all tcp port.

setsebool -P haproxy_connect_any 1

https://www.systutorials.com/docs/linux/man/8-haproxy_selinux/

Everything is working fine now.

Thanks again for your help.

Seb.