Httpck on bash script results in Socket error

f1outsourcing · August 26, 2020, 7:53pm

I am using for dovecot this health-check pass through option. Where a bash script is just returning the http response. Although the curl gives me an ok connection, haproxy keeps giving me “Socket error: Connection reset by peer”

Is this because this bash script is not closing the connection properly?

[@temp]$ curl http://test2.local:5001/ -v
* About to connect() to test2.local port 5001 (#0)
*   Trying 192.168.10.22... connected
* Connected to test2.local (192.168.10.22) port 5001 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: test2.local:5001
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 47
< Content-Type: text/html
<
<html><body>MySQL is running.</body></html>

* Connection #0 to host test2.local left intact
* Closing connection #0

backend config

backend imap
  mode tcp
  balance leastconn
  stick store-request src
  stick-table type ip size 200k expire 30m
  option httpchk GET / HTTP/1.0
  http-check expect status 200
  default-server resolvers dnssvrs1
  server s2 test2.local:10143 check port 5001 send-proxy-v2

dovecot conf

service health-check {
  # example health-check.
  executable = script -p /bin/health-check.sh
  inet_listener health-check {
      # not working env:PORT0
      port = 5001
  }
}

lukastribus · August 26, 2020, 9:34pm

Capture the health check traffic and look at it. That is way faster than any other troubleshooting or guessing around this issue.

f1outsourcing · October 8, 2021, 12:00pm

Hi Lukas, I thought about giving this another try, I am still having this “SOCKERR” “Connection reset by peer”. I have tried multiple httpchk configs in the tcp mode backend but I can not get past this error.
In the curl case it looks like the health check is sending a 2 nd packet of 27 bytes

[@]$ curl -v http://192.168.10.46:5001
* About to connect() to 192.168.10.46 port 5001 (#0)
*   Trying 192.168.10.46... connected
* Connected to 192.168.10.46 (192.168.10.46) port 5001 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 192.168.10.46:5001
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200
< content-length: 6
<
* Closing connection #0
200 OK

these are tcpdumps, which indeed have different patterns

curl:
3:44:35.494210 IP 192.168.10.2.51154 > 192.168.10.46.5001: tcp 0
13:44:35.494287 IP 192.168.10.46.5001 > 192.168.10.2.51154: tcp 0
13:44:35.494644 IP 192.168.10.2.51154 > 192.168.10.46.5001: tcp 0
13:44:35.494732 IP 192.168.10.2.51154 > 192.168.10.46.5001: tcp 169
13:44:35.494748 IP 192.168.10.46.5001 > 192.168.10.2.51154: tcp 0
13:44:35.501400 IP 192.168.10.46.5001 > 192.168.10.2.51154: tcp 14
13:44:35.501837 IP 192.168.10.2.51154 > 192.168.10.46.5001: tcp 0
13:44:35.501856 IP 192.168.10.46.5001 > 192.168.10.2.51154: tcp 27
13:44:35.502157 IP 192.168.10.46.5001 > 192.168.10.2.51154: tcp 0
13:44:35.502238 IP 192.168.10.2.51154 > 192.168.10.46.5001: tcp 0
13:44:35.502281 IP 192.168.10.46.5001 > 192.168.10.2.51154: tcp 0


haproxy request:
13:44:02.401674 IP 192.168.10.231.47584 > 192.168.10.46.5001: tcp 0
13:44:02.401729 IP 192.168.10.46.5001 > 192.168.10.231.47584: tcp 0
13:44:02.401904 IP 192.168.10.231.47584 > 192.168.10.46.5001: tcp 37
13:44:02.401946 IP 192.168.10.46.5001 > 192.168.10.231.47584: tcp 0
13:44:02.409983 IP 192.168.10.46.5001 > 192.168.10.231.47584: tcp 14
13:44:02.410506 IP 192.168.10.46.5001 > 192.168.10.231.47584: tcp 0


13:53:02.997451 IP (tos 0x0, ttl 64, id 7415, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.10.231.49168 > 192.168.10.46.5001: tcp 0
        0x0000:  4500 003c 1cf7 4000 4006 875f c0a8 0ae7  E..<..@.@.._....
        0x0010:  c0a8 0a2e c010 1389 1318 d5e2 0000 0000  ................
        0x0020:  a002 7210 9694 0000 0204 05b4 0402 080a  ..r.............
        0x0030:  be27 c915 0000 0000 0103 0307            .'..........
13:53:02.997531 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.10.46.5001 > 192.168.10.231.49168: tcp 0
        0x0000:  4500 003c 0000 4000 4006 a456 c0a8 0a2e  E..<..@.@..V....
        0x0010:  c0a8 0ae7 1389 c010 6793 4365 1318 d5e3  ........g.Ce....
        0x0020:  a012 7120 9694 0000 0204 05b4 0402 080a  ..q.............
        0x0030:  ce72 0552 be27 c915 0103 0307            .r.R.'......
13:53:02.997779 IP (tos 0x0, ttl 64, id 7416, offset 0, flags [DF], proto TCP (6), length 89)
    192.168.10.231.49168 > 192.168.10.46.5001: tcp 37
        0x0000:  4500 0059 1cf8 4000 4006 8741 c0a8 0ae7  E..Y..@.@..A....
        0x0010:  c0a8 0a2e c010 1389 1318 d5e3 6793 4366  ............g.Cf
        0x0020:  8018 00e5 96b1 0000 0101 080a be27 c916  .............'..
        0x0030:  ce72 0552 4745 5420 2f20 4854 5450 2f31  .r.RGET./.HTTP/1
        0x0040:  2e30 0d0a 636f 6e74 656e 742d 6c65 6e67  .0..content-leng
        0x0050:  7468 3a20 300d 0a0d 0a                   th:.0....
13:53:02.997825 IP (tos 0x0, ttl 64, id 13572, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.10.46.5001 > 192.168.10.231.49168: tcp 0
        0x0000:  4500 0034 3504 4000 4006 6f5a c0a8 0a2e  E..45.@.@.oZ....
        0x0010:  c0a8 0ae7 1389 c010 6793 4366 1318 d608  ........g.Cf....
        0x0020:  8010 00e3 968c 0000 0101 080a ce72 0552  .............r.R
        0x0030:  be27 c916                                .'..
13:53:03.006101 IP (tos 0x0, ttl 64, id 13573, offset 0, flags [DF], proto TCP (6), length 66)
    192.168.10.46.5001 > 192.168.10.231.49168: tcp 14
        0x0000:  4500 0042 3505 4000 4006 6f4b c0a8 0a2e  E..B5.@.@.oK....
        0x0010:  c0a8 0ae7 1389 c010 6793 4366 1318 d608  ........g.Cf....
        0x0020:  8018 00e3 969a 0000 0101 080a ce72 055a  .............r.Z
        0x0030:  be27 c916 4854 5450 2f31 2e30 2032 3030  .'..HTTP/1.0.200
        0x0040:  0d0a                                     ..
13:53:03.006755 IP (tos 0x0, ttl 64, id 13574, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.10.46.5001 > 192.168.10.231.49168: tcp 0
        0x0000:  4500 0034 3506 4000 4006 6f58 c0a8 0a2e  E..45.@.@.oX....
        0x0010:  c0a8 0ae7 1389 c010 6793 4374 1318 d608  ........g.Ct....
        0x0020:  8014 00e3 968c 0000 0101 080a ce72 055b  .............r.[
        0x0030:  be27 c916                                .'..

lukastribus · October 8, 2021, 6:07pm

Save it to a file with -w filename.cap

f1outsourcing · October 9, 2021, 12:02pm

this is a capture only having these in mode tcp.

option httpchk

server mail1 mail.example.com check port 5001

http://www.f1-outsourcing.eu/files/output.cap

lukastribus · October 14, 2021, 7:09pm

curl uses GET command, while haproxy uses the OPTION method here.

Looking at the response in both capture file (OPTION) and the tcpdump output above with the GET method, I think the response is lacking:

the reason phrase: it’s HTTP/1.0 200 OK instead of HTTP/1.0 200
another CR LF (\r\n), there must be two: one to end the Status line, and another one to delimit the response headers

f1outsourcing · October 14, 2021, 8:21pm

I am now testing with “curl -X OPTIONS -v” all these curls give “Connection #0 to host left intact” and haproxy still gives socket error

With these scripts:

echo -ne "HTTP/1.1 200 OK\r\n"
echo -ne "Content-Length: 10\r\n"
echo -ne "\r\n"
echo -ne "200 OK\r\n"
echo -ne "\r\n"

echo -ne "HTTP/1.1 200 OK\r\n"
echo -ne "Content-Length: 6\r\n"
echo -ne "\r\n"
echo -ne "200 OK"

echo -ne "HTTP/1.1 200 OK\r\n"
echo -ne "Content-Length: 0\r\n"
echo -ne "\r\n"

This generates also an error in curl

echo -ne "HTTP/1.1 200 OK\r\n"
echo -ne "\r\n"
echo -ne "Content-Length: 0\r\n"
echo -ne "\r\n"

PS I tried copying this from an exchange server but the exchange server gives an auth error if I try curl with an option request.
PPS using haproxy 2.2

lukastribus · October 15, 2021, 7:58am

Capture the traffic again.

f1outsourcing · October 15, 2021, 8:49am

With this script

echo -ne "HTTP/1.1 200 OK\r\n"
echo -ne "Content-Length: 0\r\n"
echo -ne "\r\n"
exit 0

http://www.f1-outsourcing.eu/files/output2-curl-options.cap
http://www.f1-outsourcing.eu/files/output2-ha.cap

lukastribus · October 15, 2021, 9:39am

In the output2-ha.cap there is wrong at least:

there are two spaces as opposed to one between OPTIONS and / (in the request)
Content-Length: 0 is in the request, which makes no sense
the response still does not contain proper line feed endings (its fine in the curl call)

Can you please share the haproxy healthcheck configuration you used, that generate the latest traces?

f1outsourcing · October 15, 2021, 10:06am

option log-health-checks
option httpchk
server mail1 example.com check port 5001

I have to mention that I forgot to comment out my tcp check, but I guess that is of no influence here.

#option tcp-check
#tcp-check connect port 143
#tcp-check expect string *\ OK

This script is being used in the dovecot health check mode as passthrough, maybe there is some difference being generated. However my first thought is still that if the curl seems to be ok, than the output is ok.

lukastribus · October 15, 2021, 11:34am

Both the request and the response are totally different between curl and haproxy. The capture file provided also shows content-length: 0 in the haproxy request which doesn’t make sense, so I think there is something else going on.

Are you capturing the health check traffic on the haproxy box, or the dovecot box? Please tcpdump directly on the haproxy box.

capflam · October 15, 2021, 11:49am

Sorry guys, I 'm a bit busy and I haven’t read the full story but I’ve checked and the “Content-Length: 0” is systematically added to the HTTP request, independently on the method.

lukastribus · October 15, 2021, 12:01pm

This is wrong, spaces are correct. I was confused by some wireshark formatting earlier.

Ok thanks.

The dovecot script is probably confused for whatever reason.

Could you try:

option httpchk GET / HTTP/1.1

f1outsourcing · October 15, 2021, 1:17pm


backend mail
  mode tcp
  balance first
  stick store-request src
  stick-table type ip size 200k expire 30m
  # tcp imap health check
  #option tcp-check
  #tcp-check connect port 143
  #tcp-check expect string *\ OK
  option log-health-checks
  #option httpchk
  option httpchk GET / HTTP/1.1
  #http-check connect
  #http-check send meth GET uri / HTTP/1.1
  #http-check expect status 200
  default-server check resolvers dnssvrs1 maxconn 200 weight 100 send-proxy-v2
  #default-server resolvers dnssvrs1 maxconn 200 weight 100
  server mail1 mail.local check port 5001

This still generates the same error

capflam · October 15, 2021, 1:32pm

@f1outsourcing, what is your HAProxy version ?

f1outsourcing · October 15, 2021, 4:37pm

2.2.13-5f3eb59, released 2021/04/02

lukastribus · October 15, 2021, 7:05pm

The fact of the matter is that the HTTP response contains only a single CR LF (terminating the HTTP status line). However it would have to have another CR LF sequence, to terminate the response headers. I don’t see how haproxy would be able to interpret this as a valid response.

Now, why the backend returns a correct HTTP response to curl, and an incorrect response to the haproxy health check, that I don’t know.

Maybe we need to replicate the request completely …

option httpchk GET / HTTP/1.1
http-check send hdr Host mail06.local:5001
http-check send hdr User-Agent curl/7.61.1
http-check send hdr Accept */*

f1outsourcing · October 15, 2021, 7:50pm

I will see if I can make this bash script run on a port directly without dovecot being in between. The first " “http-check send hdr” statement I can add. But as soon as I add the 2nd or 3rd, haproxy.cfg refuses to reload…

same with 2.2.17-dd94a25, released 2021/09/07

f1outsourcing · October 16, 2021, 10:58pm

If I run this, the health check is ok

dovecot-health-check.sh | nc -l 192.168.10.46 5001

So I guess it is something with the passthrough from dovecot. Sorry for wasting your time on this.

Topic		Replies	Views
Connection reset seen every 2 sec haproxy Help!	4	5188	March 2, 2018
Layer7 "TCPCHK got an empty response at step 1", check duration: 0ms? Help!	0	930	May 7, 2021
gRPC + HAProxy and connection reset by peer Help!	3	2231	March 28, 2018
Lastchk showing high values Help!	6	995	September 29, 2018
Connection reset with "SH" termination code \| 502 Bad Gateway Help!	1	1090	March 22, 2022

Httpck on bash script results in Socket error

Related Topics