Using a timestamp for Filebeat multiline config

Just a quick note today as I’m working through some Filebeat configuration.

We have a docker container that’s outputting logs across multiple lines. For example:

2019-04-15T14:18:02Z [ERROR] Log line example 1 
    Log line example 2

I’m using Filebeat to ingest these logs and ship them off to Logstash, but these are currently being processed as two separate messages.

In order to process this as one single log message, you need to use Filebeat multiline. This allows you to define a pattern that specifies a new log entry in the log file, so you can combine the logs that might be split across multiple lines in to one single entry!

Let’s have a look at the config required for grouping log lines using a timestamp as our pattern than signifies a new log entry:

- type: docker
  combine_partial: true
  containers:
    ids:
      - "*"
    stream: all
  fields:
    type: container-logs
  fields_under_root: true
  multiline:
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    negate: true
    match: after

With ‘multiline’ section in in place, filebeat will recognise a new log entry to begin with a timestamp that matches the ‘pattern’. It will concatenate any subsequent log entries until it recognises another timestamp. Useful!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.