How can I remove the log_format prefix?

Help!
1

Hello!

I’m trying to send my HAProxy logs to Elasticsearch via Fluentd, but at the volume I am working with, Fluentd is not able to parse the log lines fast enough using Ruby’s regex engine. So, I was hoping to ‘trick’ HAProxy into logging JSON-structured messages, using the approach outlined here: https://jablonskis.org/2014/haproxy-logging-to-syslog-in-json/index.html

To that end, I started writing my custom log_format:

log-format {"message":"%ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r","timestamp":%Ts,"frontend_name":"%ft","backend_name":"%b","server":"%s","time_request":%Tq,"time_queue":%Tq,"time_backend_connect":%Tc,"time_backend_response":%Tr,"time_duration":%Tt,"http_status_code":%ST}

So, this will log the full default log line in the message field, and then re-log each field in its own JSON field. I didn’t finish defining all of the fields yet, but you can get the idea.

However, it seems that the real message sent by HAProxy starts off like this:

May  4 00:30:06 0262e79d1cbb haproxy[11]: {"message":...

How can I suppress that ‘prefix’ so that the logged message is valid JSON?

Thanks!

2

Ahh, I realized that these fields are the syslog HEADER field, so once I figured that out, I figured out how to just parse the MESSAGE part as JSON, and everything seems to work well now.

3

Hi, could you share how did you remove the syslog header?

4

By setting format raw in the log configuration under the global section, e.g.:

global
  log stdout format raw local0

you can make sure that no syslog header is added to the output. Setting the format raw setting does also not interfere with custom log format definitions.