Rsyslog is the default syslog package that is commonly found in current Linux distributions.
It usually consists of a main configuration file (typically /etc/rsyslog.conf) and a directory (typically /etc/rsyslog.d/) where you can store, in a structured form and separated by files, the different filters and templates for processing rsyslog rules.
The following section explains how to manually configure rsyslog to monitor and process log files from applications installed in the system and send it to logz.io via rsyslog.
Pre requests:
- The setup assumes that you have a sudo access
- Rsyslog version 5.8.0 and above
- Allow outgoing TCP traffic to destination port 5000
- A common linux distribution
Using the configuration script:
- syslog (monitor from vireos system files and facilities)
- apache (monitor apache access and error log files)
- nginx (monitor nginx access and error log files)
- mysql (monitor mysql general slow-log and error log files)
Monitor plain text file:
curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz && tar xzf logzio-rsyslog.tar.gz && sudo rsyslog/install.sh -t "<TYPE>" -a "<TOKEN>" --filepath "<FILE>"
Monitor JSON file:
curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz && tar xzf logzio-rsyslog.tar.gz && sudo rsyslog/install.sh -t "<TYPE>" -a "<TOKEN>" --filepath "<FILE>" -c json
Parameters:
Manual configuration:
vim /etc/rsyslog.d/21-logzio-filename.conf
# ------------------------------------------------------- # File Logging Directives for Logz.io # ------------------------------------------------------- $ModLoad imfile $InputFilePollInterval 10 $PrivDropToGroup adm $WorkDirectory /var/spool/rsyslog $InputFileName <FILE> $InputFileTag <TAG>: $InputFileStateFile unique-name $InputFileSeverity info $InputFilePersistStateInterval 20000 $InputRunFileMonitor $template logzFormatTemplate,"[<TOKEN>] <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [type=<TYPE>] %msg%\n" if $programname == '<TAG>' then @@listener.logz.io:5000;logzFormatTemplate if $programname == '<TAG>' then ~
Description of the template main fields:
- $WorkDirectory: must exist and the user running rsyslog must have permissions on it.
- $PrivDropToGroup: if a start rsyslog as root is required, you can use this config directive to specify a group and/or user (PrivDropToUser) that rsyslogd should drop to after initialization.
- $InputFileName: File PATH where the log file that we want to process is stored
- $InputFileTag: label used by Rsyslog to classify the log line and to filter it ($programname) in to the relevant output action.
- $InputFileStateFile: must be a unique name and it shouldn’t match with other rsyslog configuration state files names. It’s used to save the state of the information that have been processed from the file, thus, in case of an rsyslog crash or reboot, it won’t resend data already processed, duplicating the information.
- $InputFilePollInterval: specifies how often files are to be polled for new data.
- $InputFilePersistStateInterval: after how many events/lines is the file processing state written to disk.
- $InputRunFileMonitor: This activates the current monitor. It has no parameters. If you forget this directive, no file monitoring will take place.
- $template: Rsyslog allows you to define a template for your syslog message.
The Template Structure:
When logs.io will receives a log via port 5000, it will accepts to see the following template structure:
$template logzFormatTemplate,"[<TOKEN>] <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [type=<TYPE>] %msg%\n"
One of the define fields is the TYPE field that is used in logz.io to classify the log, It’s important to use a type that makes sense to you. If that log is included in one of the well known logz.io types (you can get it from the "Supported Log Types" category), you should use it.
Parameters:
<FILE> - The full path to the file on your file system
<TOKEN> - Your Logz.io token (from the Settings page)
<TAG> - label used by Rsyslog to classify the log line
<TYPE> - The log type that you are sending (you can get it from the "Supported Log Types" category)
Comments
0 comments
Please sign in to leave a comment.