for service A1 and A2, there are sharing two backend servers, we can name it as HostA1 , HostA2 (all service are based on tomcat web service)
for service B1 and B2, there are sharing two backend servers, we can name it as HostB1 , HostB2
10.10.10.2 is the local proxy server.
All service are running successfully before 23:13:13, suddently, another haproxy was started.
2022-04-23T23:13:13 localhost haproxy[31928]: 817827 100.65.194.161:46086 10.10.10.2 US_site1 serviceA2 10.62.129.81:443 200 POST /serviceA2 HTTP/1.1
2022-04-23T23:13:21 localhost haproxy[26675]: Proxy stats started.
2022-04-23T23:13:21 localhost haproxy[26675]: Proxy US_site1 started.
2022-04-23T23:13:21 localhost haproxy[26675]: Proxy serviceA1 started.
2022-04-23T23:13:21 localhost haproxy[26675]: Proxy serviceA2 started.
2022-04-23T23:13:21 localhost haproxy[26675]: Proxy serviceB1 started.
2022-04-23T23:13:21 localhost haproxy[26675]: Proxy serviceB2 started.
2022-04-23T23:13:21 localhost haproxy[26675]: Server serviceA1/web is DOWN, reason: Layer4 connection problem, info: “Connection error during SSL handshake (Connection refused)”, check duration: 21ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2022-04-23T23:13:21 localhost haproxy[26675]: backend serviceA1 has no server available!
2022-04-23T23:13:21 localhost haproxy[26675]: Server serviceA2/web is DOWN, reason: Layer4 connection problem, info: “Connection error during SSL handshake (Connection refused)”, check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2022-04-23T23:13:21 localhost haproxy[26675]: backend serviceA2 has no server available!
2022-04-23T23:13:28 localhost haproxy[31928]: 817828 10.62.100.89:62336 10.10.10.2 US_site1 serviceB2 10.62.129.82:443 200 POST /serviceB2/ForHtmlTest HTTP/1.1
2022-04-23T23:13:29 localhost haproxy[31928]: 817829 10.99.231.214:56008 10.10.10.2 US_site1 serviceB2 10.62.129.88:443 200 HEAD /serviceB2/ HTTP/1.1
2022-04-23T23:13:29 localhost haproxy[31928]: 817830 10.62.98.148:40272 10.10.10.2 US_site1 serviceA2 10.62.129.80:443 200 POST /serviceA2 HTTP/1.1
2022-04-23T23:13:29 localhost haproxy[31928]: 817831 10.62.100.89:62338 10.10.10.2 US_site1 serviceB2 10.62.129.83:443 200 POST /serviceB2/ForHtmlTest HTTP/1.1
2022-04-23T23:13:30 localhost haproxy[26676]: 10.62.142.62:36726 - US_site1 serviceA2 -:- 503 POST /serviceA2 HTTP/1.1\
2022-04-23T23:13:35-04:00 localhost haproxy[26676]: 1 10.62.98.160:54908 - US_PRD serviceA2 -:- 503 POST /serviceA2 HTTP/1.1
2022-04-23T23:13:47-04:00 localhost haproxy[26676]: 3 100.65.194.160:15524 - US_PRD serviceA2 -:- 503 POST /serviceA2 HTTP/1.1
2022-04-23T23:13:49-04:00 localhost haproxy[26676]: 4 100.65.194.161:46130 - US_PRD serviceA2 -:- 503 POST /serviceA2 HTTP/1.1
2022-04-23T23:13:53-04:00 localhost haproxy[31928]: 817833 10.99.231.214:56098 10.10.10.2 US_PRD serviceA1 10.62.129.80:443 200 POST /serviceA2 HTTP/1.1
2022-04-23T23:13:56-04:00 localhost haproxy[31928]: 817834 100.65.194.161:46140 10.10.10.2 US_PRD serviceA2 10.62.129.81:443 200 POST /serviceA2 HTTP/1.1
2022-04-23T23:14:06-04:00 localhost haproxy[26676]: 5 100.65.194.161:46164 - US_PRD serviceA2 -:- 503 POST serviceA2 HTTP/1.1
2022-04-23T23:14:06-04:00 localhost haproxy[31928]: 817836 100.65.194.160:15556 10.10.10.2 US_PRD serviceA2 10.62.129.81:443 200 POST /serviceA2 HTTP/1.1
2022-04-23T23:14:07-04:00 localhost haproxy[31928]: 817835 100.65.194.161:46162 10.10.10.2 US_PRD serviceA2 10.62.129.80:443 200 POST /serviceA2 HTTP/1.1
2022-04-23T23:14:08-04:00 localhost haproxy[31928]: 817837 100.65.194.160:15558 10.10.10.2 US_PRD serviceA2 10.62.129.80:443 200 POST /serviceA2 HTTP/1.1
2022-04-23T23:14:31-04:00 localhost haproxy[31928]: 817838 10.62.6.36:58292 10.10.10.2 US_PRD serviceB2 10.62.129.84:443 200 POST serviceB2 HTTP/1.1
2022-04-23T23:14:34-04:00 localhost haproxy[31928]: 817839 10.62.100.83:45204 10.10.10.2 US_PRD serviceB2 10.62.129.89:443 200 POST serviceB2 HTTP/1.1
2022-04-23T23:14:35-04:00 localhost haproxy[26676]: 6 10.62.100.88:43658 - US_PRD serviceA2 -:- 503 POST /serviceA2 HTTP/1.1
2022-04-23T23:14:41-04:00 localhost haproxy[31928]: 817840 10.62.6.36:58297 10.10.10.2 US_PRD serviceA2 10.62.129.81:443 200 POST serviceA2 HTTP/1.1
2022-04-23T23:14:41-04:00 localhost haproxy[26676]: 8 10.62.6.36:58299 - US_PRD serviceA2 -:- 503 POST /serviceA2 HTTP/1.1
2022-04-23T23:14:42-04:00 localhost haproxy[26676]: 9 100.65.194.160:15598 - US_PRD serviceA2 -:- 503 POST /serviceA2 HTTP/1.1
2022-04-23T23:14:42-04:00 localhost haproxy[26676]: 10 100.65.194.160:15600 - US_PRD serviceA2 -:- 503 POST /serviceA2 HTTP/1.1
2022-04-23T23:14:44-04:00 localhost haproxy[26676]: 11 100.65.194.160:15604 - US_PRD serviceA2 -:- 503 POST /serviceA2 HTTP/1.1
2022-04-23T23:14:45-04:00 localhost haproxy[31928]: 817841 100.65.194.161:46212 10.10.10.2 US_PRD serviceA2 10.62.129.80:443 200 POST /serviceA2 HTTP/1.1\
in the later logs , service B1 and service B2 are running successfully , but for service A1 and A2, there are two many 503 error.
almost 50% are 503 error, also the first request is ok, then next failed(503 error).
the solution :
at the first time, i restarted the tomcat for service on both nodes, the issue still exists. then i restarted the haproxy on the active node,
then the actice (i also deployed the keepalived on both nodes), then issue disappeared.
what i’m comfused :
- why the proxy was started without restarting ?
- why the counter for service2 is increased by 1,3, but most is increased from 810000 ?
- why there is SSL handshake error ? but the service is not always down.
- it seems two haproxy daemon id are started, the first is 31928 , when the second haproxy daemon id started, two many 503 error were create.
the version is 1.5.18, is it a production bug ?