# 503 with NOSRV but no tcp reset found

**URL:** <https://discourse.haproxy.org/t/503-with-nosrv-but-no-tcp-reset-found/2666>\
**Category:** Help!\
**Created:** [June 28, 2018, 7:33am UTC](https://discourse.haproxy.org/t/503-with-nosrv-but-no-tcp-reset-found/2666 "2018-06-28T07:33:01Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![mainred](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@mainred](https://discourse.haproxy.org/u/mainred)\
**Post date:** [June 28, 2018, 7:33am UTC](https://discourse.haproxy.org/t/503-with-nosrv-but-no-tcp-reset-found/2666/1 "2018-06-28T07:33:01Z")

</div>

haproxy: 1.7.6

I found in haproxy log that occasional 503 error, and I tried to tcpdump traffic from haproxy node,  
but I did not find any tcp reset from server.

logs of 503 error looks like below:  
marathon\_http\_in marathon\_http\_in/ -1/-1/-1/-1/0 503 212 - - SC-- 22/22/0/0/0 0/0 "GET /api/item/

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [June 28, 2018, 7:37am UTC](https://discourse.haproxy.org/t/503-with-nosrv-but-no-tcp-reset-found/2666/2 "2018-06-28T07:37:06Z")

</div>

If you did not find any TCP reset, what DID you find in the tcpdump?

SC is the network stack telling haproxy that the destination is unreachable. Could be due to “no route to host”, or unresolved ARP entries, etc.

---

<div class="post-metadata">

**Author:** ![mainred](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@mainred](https://discourse.haproxy.org/u/mainred)\
**Post date:** [June 28, 2018, 8:01am UTC](https://discourse.haproxy.org/t/503-with-nosrv-but-no-tcp-reset-found/2666/3 "2018-06-28T08:01:55Z")

</div>

In fact, I got a sequence of NOSRV 503 error, here are just a few of them:  
Jun 28 11:09:19 marathon\_http\_in/ -1/-1/-1/-1/0 503 212 - - SC-- 35/35/0/0/0 0/0 “GET /api/user/”  
Jun 28 11:09:19 marathon\_http\_in marathon\_http\_in/ -1/-1/-1/-1/0 503 212 - - SC-- 35/35/0/0/0 0/0 “GET /api/user”

Static route is used, so route or arp is should ignored.

I only find TCP reset that was originated from haproxy, which is used for health check, syn sync&ack, rst&ack

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [June 28, 2018, 8:39am UTC](https://discourse.haproxy.org/t/503-with-nosrv-but-no-tcp-reset-found/2666/4 "2018-06-28T08:39:56Z")

</div>

> [@mainred](#):
>
> Static route is used, so route or arp is should ignored.

That’s not how it works. If you have static route, you still need to ARP the gateway. And you can always get a “no route to host” from the intermediate router or the end host.

You need to tcpdump everything, inlcluding ARP and ICMP from all IP’s. Then check it out. Or you run haproxy through `strace -tt` and point to the specific socket that had this problem.

But there will be no solution within haproxy. This is your kernel that is telling haproxy that the destination is unreachable.

---

<div class="post-metadata">

**Author:** ![mainred](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@mainred](https://discourse.haproxy.org/u/mainred)\
**Post date:** [June 28, 2018, 9:23am UTC](https://discourse.haproxy.org/t/503-with-nosrv-but-no-tcp-reset-found/2666/5 "2018-06-28T09:23:47Z")

</div>

Sorry, I simplified my words.  
By saying “route or arp should ignore”, I can make sure route and arp to gateway are ok.

I did not find from the source code how SC is parsed, so I am wondering if there are other logical conditions of haproxy that result in SC.

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [June 28, 2018, 9:47am UTC](https://discourse.haproxy.org/t/503-with-nosrv-but-no-tcp-reset-found/2666/6 "2018-06-28T09:47:51Z")

</div>

`strace -tt` it, and you will see how your kernel tells haproxy how the destination is unreachable.

---

<div class="post-metadata">

**Author:** ![mainred](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@mainred](https://discourse.haproxy.org/u/mainred)\
**Post date:** [June 29, 2018, 1:28am UTC](https://discourse.haproxy.org/t/503-with-nosrv-but-no-tcp-reset-found/2666/7 "2018-06-29T01:28:07Z")

</div>

thank you very much, I’ll try strace and report result if I made some progress.

---

<div class="post-metadata">

**Author:** ![mainred](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@mainred](https://discourse.haproxy.org/u/mainred)\
**Post date:** [June 29, 2018, 3:44am UTC](https://discourse.haproxy.org/t/503-with-nosrv-but-no-tcp-reset-found/2666/8 "2018-06-29T03:44:19Z")

</div>

I used strace and find that haproxy first close socket and read socket as show beflow, this fd is 68  
11:22:36.914496 socket(AF\_INET, SOCK\_STREAM, IPPROTO\_TCP) = 68  
11:22:36.914566 fcntl(68, F\_SETFL, O\_RDONLY|O\_NONBLOCK) = 0  
11:22:36.914631 setsockopt(68, SOL\_TCP, TCP\_NODELAY, [1], 4) = 0  
11:22:36.914699 setsockopt(68, SOL\_TCP, TCP\_QUICKACK, [0], 4) = 0  
11:22:36.914763 connect(68, {sa\_family=AF\_INET, sin\_port=htons(8080), sin\_addr=inet\_addr(“9.0.5.142”)}, 16) = -1 EINPROGRESS (Operation now in progress)  
11:22:36.914897 epoll\_wait(0, [], 200, 0) = 0  
11:22:36.914962 connect(68, {sa\_family=AF\_INET, sin\_port=htons(8080), sin\_addr=inet\_addr(“9.0.5.142”)}, 16) = -1 EALREADY (Operation already in progress)  
11:22:36.915056 epoll\_ctl(0, EPOLL\_CTL\_ADD, 68, {EPOLLOUT, {u32=68, u64=68}}) = 0  
11:22:36.915125 epoll\_wait(0, [{EPOLLOUT, {u32=68, u64=68}}], 200, 35) = 1  
11:22:36.915198 connect(68, {sa\_family=AF\_INET, sin\_port=htons(8080), sin\_addr=inet\_addr(“9.0.5.142”)}, 16) = 0  
11:22:36.915289 recvfrom(68, NULL, 2147483647, MSG\_TRUNC|MSG\_DONTWAIT|MSG\_NOSIGNAL, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)  
11:22:36.915356 setsockopt(68, SOL\_SOCKET, SO\_LINGER, {onoff=1, linger=0}, 8) = 0  
11:22:36.915421 close(68) = 0  
11:22:36.915515 epoll\_wait(0, [{EPOLLIN, {u32=10, u64=10}}], 200, 34) = 1  
11:22:36.937865 accept4(10, {sa\_family=AF\_INET, sin\_port=htons(32807), sin\_addr=inet\_addr(“100.109.215.128”)}, [16], SOCK\_NONBLOCK) = 68  
11:22:36.938009 setsockopt(68, SOL\_TCP, TCP\_NODELAY, [1], 4) = 0  
11:22:36.938104 accept4(10, 0x7ffed5fc07a0, 0x7ffed5fc079c, SOCK\_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)  
11:22:36.938225 read(68, 0x55690f500b83, 5) = -1 ECONNRESET (Connection reset by peer)  
11:22:36.938305 close(68) = 0

---

<div class="post-metadata">

**Author:** ![mainred](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@mainred](https://discourse.haproxy.org/u/mainred)\
**Post date:** [June 29, 2018, 5:02am UTC](https://discourse.haproxy.org/t/503-with-nosrv-but-no-tcp-reset-found/2666/9 "2018-06-29T05:02:30Z")

</div>

Let’s ignore my last reply, because there is no sendto, so it is more likely a health check.  
haproxy got an request, accept as fd 2, and sendto 503 without tcp-rst-like message.  
It seems to be that haproxy reply 503 automatically by internal logic without proxy request to backend server.

11:22:36.765962 accept4(8, {sa\_family=AF\_INET, sin\_port=htons(31516), sin\_addr=inet\_addr(“42.120.75.138”)}, [16], SOCK\_NONBLOCK) = 2  
11:22:36.766073 setsockopt(2, SOL\_TCP, TCP\_NODELAY, [1], 4) = 0  
11:22:36.766140 accept4(8, 0x7ffed5fc07a0, 0x7ffed5fc079c, SOCK\_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)  
11:22:36.766227 recvfrom(2, 0x55690f3efe04, 15360, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)  
11:22:36.766295 epoll\_ctl(0, EPOLL\_CTL\_ADD, 2, {EPOLLIN|EPOLLRDHUP, {u32=2, u64=2}}) = 0  
11:22:36.766369 epoll\_wait(0, [{EPOLLIN, {u32=76, u64=76}}], 200, 62) = 1  
11:22:36.766460 recvfrom(76, “GET /agent/0d908a29-bf91-4aa7-97”…, 15360, 0, NULL, NULL) = 842  
11:22:36.766606 getpid() = 22479  
11:22:36.766663 sendmsg(5, {msg\_name(110)={sa\_family=AF\_LOCAL, sun\_path="/dev/log"}, msg\_iov(8)=[{"\<135\>Jun 29 03:22:36 “, 21}, {“haproxy”, 7}, {”[", 1}, {“22479”, 5}, {"]: “, 3}, {”", 0}, {“add log-id header:01153024255676”…, 56}, {"\n", 1}], msg\_controllen=0, msg\_flags=0}, MSG\_DONTWAIT|MSG\_NOSIGNAL) = 94  
11:22:36.766806 socket(AF\_INET, SOCK\_STREAM, IPPROTO\_TCP) = 68  
11:22:36.766884 fcntl(68, F\_SETFL, O\_RDONLY|O\_NONBLOCK) = 0  
11:22:36.766946 setsockopt(68, SOL\_TCP, TCP\_NODELAY, [1], 4) = 0  
11:22:36.767005 connect(68, {sa\_family=AF\_INET, sin\_port=htons(80), sin\_addr=inet\_addr(“9.0.5.130”)}, 16) = -1 EINPROGRESS (Operation now in progress)  
11:22:36.767125 epoll\_wait(0, [{EPOLLIN, {u32=2, u64=2}}], 200, 0) = 1  
11:22:36.767249 sendto(68, “GET /agent/0d908a29-bf91-4aa7-97”…, 951, MSG\_DONTWAIT|MSG\_NOSIGNAL, NULL, 0) = 951  
11:22:36.767339 recvfrom(2, “GET / HTTP/1.1\r\nHost: yanan-stor”…, 15360, 0, NULL, NULL) = 464  
11:22:36.767490 getpid() = 22479  
11:22:36.767549 sendmsg(5, {msg\_name(110)={sa\_family=AF\_LOCAL, sun\_path="/dev/log"}, msg\_iov(8)=[{"\<135\>Jun 29 03:22:36 “, 21}, {“haproxy”, 7}, {”[", 1}, {“22479”, 5}, {"]: “, 3}, {”", 0}, {“add log-id header:01153024255676”…, 56}, {"\n", 1}], msg\_controllen=0, msg\_flags=0}, MSG\_DONTWAIT|MSG\_NOSIGNAL) = 94  
11:22:36.767698 epoll\_ctl(0, EPOLL\_CTL\_DEL, 2, 0x55690ed5f430) = 0  
11:22:36.767738 epoll\_wait(0, [], 200, 0) = 0  
11:22:36.767764 recvfrom(68, 0x55690f3ebde4, 15360, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)  
11:22:36.767805 sendto(2, “HTTP/1.0 503 Service Unavailable”…, 212, MSG\_DONTWAIT|MSG\_NOSIGNAL|MSG\_MORE, NULL, 0) = 212  
11:22:36.767836 shutdown(2, SHUT\_WR) = 0

---

<div class="post-metadata">

**Author:** ![mainred](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@mainred](https://discourse.haproxy.org/u/mainred)\
**Post date:** [June 29, 2018, 10:33am UTC](https://discourse.haproxy.org/t/503-with-nosrv-but-no-tcp-reset-found/2666/10 "2018-06-29T10:33:59Z")

</div>

This problem is solved. Except reset from backend server, if we dispatch request from host of http header，and find no frontend, 503 with NOSRV and SC can also occur.  
Thank for you help, @lukastribus.