How can I remove the log_format prefix?


#1

Hello!

I’m trying to send my HAProxy logs to Elasticsearch via Fluentd, but at the volume I am working with, Fluentd is not able to parse the log lines fast enough using Ruby’s regex engine. So, I was hoping to ‘trick’ HAProxy into logging JSON-structured messages, using the approach outlined here: https://jablonskis.org/2014/haproxy-logging-to-syslog-in-json/index.html

To that end, I started writing my custom log_format:

log-format {"message":"%ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r","timestamp":%Ts,"frontend_name":"%ft","backend_name":"%b","server":"%s","time_request":%Tq,"time_queue":%Tq,"time_backend_connect":%Tc,"time_backend_response":%Tr,"time_duration":%Tt,"http_status_code":%ST}

So, this will log the full default log line in the message field, and then re-log each field in its own JSON field. I didn’t finish defining all of the fields yet, but you can get the idea.

However, it seems that the real message sent by HAProxy starts off like this:

May  4 00:30:06 0262e79d1cbb haproxy[11]: {"message":...

How can I suppress that ‘prefix’ so that the logged message is valid JSON?

Thanks!


#2

Ahh, I realized that these fields are the syslog HEADER field, so once I figured that out, I figured out how to just parse the MESSAGE part as JSON, and everything seems to work well now.