Articles in this section

How do I use rsyslog to ship logs ?

Avatar
Joshua Schnitzer
  • Updated
Follow

Rsyslog is the default syslog package that is commonly found in current Linux distributions.

It usually consists of a main configuration file (typically /etc/rsyslog.conf) and a directory (typically /etc/rsyslog.d/) where you can store, in a structured form and separated by files, the different filters and templates for processing rsyslog rules.

The following section explains how to manually configure rsyslog to monitor and process log files from applications installed in the system and send it to logz.io via rsyslog.

Pre requests:

  • The setup assumes that you have a sudo access
  • Rsyslog version 5.8.0 and above
  • Allow outgoing TCP traffic to destination port 5000
  • A common linux distribution
You can install rsyslog in one of two ways, eider using our automated configuration script, or defining the configuration by hand.

Using the configuration script:

The configuration script comes with support for installation of few well know log types (for the update list and documentation please visit the following Github repository)
  • syslog (monitor from vireos system files and facilities)
  • apache (monitor apache access and error log files)
  • nginx (monitor nginx access and error log files)
  • mysql (monitor mysql general slow-log and error log files)
Additionally you can use the script to monitor any other file / directory (first level only) in your system.

Monitor plain text file:

curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz && tar xzf logzio-rsyslog.tar.gz && sudo rsyslog/install.sh -t "<TYPE>" -a "<TOKEN>" --filepath "<FILE>"

  

Monitor JSON file:

curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz && tar xzf logzio-rsyslog.tar.gz && sudo rsyslog/install.sh -t "<TYPE>" -a "<TOKEN>" --filepath "<FILE>" -c json

  

Parameters:

<FILE> - The full path to the file on your file system
<TOKEN> - Your Logz.io token (from the Settings page)
<TYPE> - The log type that you are sending (you can get it from the "Supported Log Types" category)

Manual configuration:

Here is an example of rsyslog template to process a log file:
vim /etc/rsyslog.d/21-logzio-filename.conf

  

#   -------------------------------------------------------
#        File Logging Directives for Logz.io
#   -------------------------------------------------------

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog

$InputFileName <FILE>
$InputFileTag <TAG>:
$InputFileStateFile unique-name
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
$template logzFormatTemplate,"[<TOKEN>] <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [type=<TYPE>] %msg%\n"
if $programname == '<TAG>' then @@listener.logz.io:5000;logzFormatTemplate
if $programname == '<TAG>' then ~

 
Description of the template main fields:

  • $WorkDirectory: must exist and the user running rsyslog must have permissions on it. 
  • $PrivDropToGroup: if a start rsyslog as root is required, you can use this config directive to specify a group and/or user (PrivDropToUser) that rsyslogd should drop to after initialization.
  • $InputFileName: File PATH where the log file that we want to process is stored
  • $InputFileTag: label used by Rsyslog to classify the log line and to filter it ($programname) in to the relevant output action.
  • $InputFileStateFile: must be a unique name and it shouldn’t match with other rsyslog configuration state files names. It’s used to save the state of the information that have been processed from the file, thus, in case of an rsyslog crash or reboot, it won’t resend data already processed, duplicating the information.
  • $InputFilePollInterval: specifies how often files are to be polled for new data.
  • $InputFilePersistStateInterval: after how many events/lines is the file processing state written to disk.
  • $InputRunFileMonitor: This activates the current monitor. It has no parameters. If you forget this directive, no file monitoring will take place.
  • $template: Rsyslog allows you to define a template for your syslog message. 

The Template Structure:

When logs.io will receives a log via port 5000, it will accepts to see the following template structure: 

$template logzFormatTemplate,"[<TOKEN>] <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [type=<TYPE>] %msg%\n"

 

the %msg% template property will carry out your actual message
 
The template structure allows you to define as additional fields,
One of the define fields is the TYPE field that is used in logz.io to classify the log, It’s important to use a type that makes sense to you. If that log is included in one of the well known logz.io types (you can get it from the "Supported Log Types" category), you should use it. 
 
Adding fields: note the additional: key1=value1 key2=value2  

Parameters:

<FILE> - The full path to the file on your file system

<TOKEN> - Your Logz.io token (from the Settings page)

<TAG> - label used by Rsyslog to classify the log line

<TYPE> - The log type that you are sending (you can get it from the "Supported Log Types" category)

Related articles

Comments

0 comments

Please sign in to leave a comment.