Logging error details found in "show errors" socket command

Help!
1

Hello, we have some cases where a backend server returns an invalid response, which HAProxy then returns as a 502. We can see more details using the “show errors” command against the stats socket.

This behavior makes sense, but we are trying to capture the details of these errors into a log file. Often by the time we are troubleshooting a specific issue, the messages we need are already gone from “show errors”. Is there a recommended way to log these? I cannot tell if this is an issue with our logging configuration, or that these details are not able to be sent to syslog. A 502 is logged via syslog to the log file, following the log format template, but it is lacking the details available in “show errors”.

Is it possible to persist these errors outside of the “show errors” buffer?
Thank you for your time.

Below is the logging portion of our config

log                127.0.0.1 local0
log                127.0.0.1 local1 notice

# defaults
log                 global
log-format "%ci:%cp || [%tr] || %ft || %b/%s || %TR/%Tw/%Tc/%Tr/%Ta/%Ti/%Th/%Tt || %ST || %B || %CC || %CS || %tsc || %ac/%fc/%bc/%sc/%rc || %sq/%bq || %hr || %hs || %{+Q}r"

option              dontlognull

# no error-log-format is set
# log-separate-errors is not set

syslog conf:

$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514

local0.* /data/log/haproxy/haproxy.log
local1.* /data/log/haproxy/haproxy-status.log

When we reproduce a similar error, nothing related to the error is logged to haproxy-status.log. in haproxy.log, we see the error with the PH flag, following our log format. I am not sure whether or not to expect a log entry using the default error log format in this 502 case. As far as I can tell, all log events emitted by haproxy are captured by this syslog configuration. I also don’t see any other variables available in the log templates that contain the details in show errors. Any insight into this would be appreciated!

2023-09-26T16:09:25+00:00 example.com haproxy[1250915]: <redacted>:52718 || [26/Sep/2023:16:09:25.500] || main_frontend~ || INTERNAL_EXAMPLE_HTTPS/EXAMPLE-A1 || 0/0/3/-1/8/0/3/11 || 502 || 209 || - || - || PH-- || 28/27/0/0/0 || 0/0 || || || "POST /responseheaderssubmit HTTP/1.1"

These are the details from “show errors” we would like to find a way to log to a file:

 backend INTERNAL_EXAMPLE_HTTPS (#196): invalid response
  frontend main_frontend (#5), server EXAMPLE-A1 (#1), event #2, src <redacted>:52718
  buffer starts at 0 (including 0 out), 64 free,
  len 16320, wraps at 16336, error at position 16320
  H1 connection flags 0x80002220, H1 stream flags 0x00004810
  H1 msg state MSG_HDR_VAL(22), H1 msg flags 0x00001414
  H1 chunk len 0 bytes, H1 body len 0 bytes :

  00000  HTTP/1.1 200 OK\r\n
  00017  Content-Type: text/html; charset=utf-8\r\n
  00057  Content-Encoding: gzip\r\n
  00081  Vary: Accept-Encoding\r\n
  00104  Server: Microsoft-IIS/10.0\r\n
  00132  x-long: alskdjfalsdkfjalskdfjalsdkfjalskdfjlsadkfjsalkdfjalskdjfalsdkf
  00202+ jalskdfjalsdkfjalskdfjlsadkfjsalkdfjalskdjfalsdkfjalskdfjalsdkfjalskdf
  00272+ jlsadkfjsalkdfjalskdjfalsdkfjalskdfjalsdkfjalskdfjlsadkfjsalkdfjalskdj
<truncated>
2

No, show errors is not saved or send anywhere. You cannot log “show errors”.

If you reload haproxy for example, it will clear because it is only in a local buffer.

You’d have to automate this externally. Monitoring log files for specific events and saving show errors output externally (with timestamps).

1 Like
3

Thanks for clarifying!