What are multiline logs, and how can I ship them to Logz.io?

Multiline logs are logs that span multiple lines, and typical examples of multiline logs are stack traces for instance.

A common scenario we come across a lot is: 

• You have a Java stack trace, where the log will typically start with a timestamp, followed by an error message and then the stack trace.
• And the problem is, shipping those logs with a standard shipping method will cause every line within the log to create a new log entry within Kibana. Which will make it difficult to read, understand and troubleshoot the log.

How can I ship multiline logs and properly index them?

To have your multiline logs properly indexed into Elasticsearch and displayed properly within Kibana, you must ship the logs using a shipper that supports multiline logs.

What this means is that within the shipper, you must be able to configure when a multiline log begins and ends -- and then the shipper will send the full log to Logz.io as a single entry.  

What are my options to ship multiline logs to Logz.io?

Many log shippers support multiline logs and every shipper has its own unique way of handling it, some shipper examples are:

• Filebeat
• Fluentd
• rsyslog
• Appenders

Our recommendation: Filebeat for multiline log shipping, and here is why:

• Can be installed on Linux, Mac, and Windows
• Ships logs using TLS encryption
• Compress the logs for shipping
• And it handles multiline logs well

Plus, we can help!

• Our Filebeat Configuration Wizard gives you instruction on how to do it
• And our support team can help you writing your Regex if necessary

For more details: To configure Filebeat to ship multiline logs, add the multiline option to the relevant prospector within your Filebeat configuration file. Also, check here to see which regular expressions are supported.