Troubleshooting Rsyslog - Failed to install

Use the following tips to troubleshoot problems with Rsyslog. Additionally, you can read the Rsyslog manual, try their support forum (they offer professional Rsyslog support), or check out our Rsyslog configuration documents.

First ... wait a few minutes

Wait a few minutes after sending your logs to give us time to index and make them available for search. It normally happens from within seconds to one minute, but sometimes it can take longer.

Check out our status page

If Logz.io isn’t seeing data, check our status page to make sure that we are indexing your data and that all systems are operational. You should see a green, horizontal bar with "All Systems Operational" inside.

If you have not used it, you should try the automated installation script

Using the automated installation script will ensure that all network connectivity checks have been performed and that your environment is configured properly. It also helps you to avoid most of the common problems and saves you a lot of headaches. The automated installation script is highly configurable and relevant to most use cases. You can read all about it in our Rsyslog configuration docs or our GitHub repo.

Check your Rsyslog configuration

Restart Rsyslog again to make sure that your recent changes will take effect:

$ sudo service rsyslog restart

 

Make sure you have Rsyslog installed with version 5.8 or higher:

$ sudo service rsyslog status

Or

$ ps -A | grep rsyslog

 

Check the Rsyslog configuration. If there are no errors listed, then it is OK:

$ rsyslogd -N1

 

Open your config file:

Check that all configured files exist on the file system.
Check that the files actually contain log lines.   

Make sure that the spool directory (the $WorkDirectory) exists:

$WorkDirectory must exist and the user that is running Rsyslog must have any needed permissions. In some distributions, this directory is not created. In case it is missing, make sure that the spool directory is created with the correct permissions.

Find out which user runs Rsyslog:

$ ps -ouser= $(pgrep rsyslogd)

 

If it is different than root (in some distributions, it runs by default as root; in others, a drop of privileges to syslog user is made) we recommend performing:

 

$ mkdir /var/spool/rsyslog
$ chown syslog:syslog /var/spool/rsyslog
$ chmod 770 /var/spool/rsyslog

 

Make sure that the spool files are created in the spool directory (example):

$ ls -al /var/spool/rsyslog

-rw------- 1 root root 209 Jan  6 14:19 my-stat-file1
-rw------- 1 root root 205 Jan  6 14:19 my-stat-file2

 

If files have not been created, first try to restart Rsyslog. Otherwise, the problem pertains to permissions. Make sure that Rsyslog has the proper permissions over the spool directory.

Validate the monitored file permissions

Check the permissions of Rsyslog and the file that you want to monitor to make sure that it can read the file. In case Rsyslog is not running as root, you will need to drop privileges in Rsyslog. This can be done by setting either of the following:

$PrivDropToUser - The name of the user Rsyslog should run under after startup. (Please note that this user is looked up in the system tables. If the lookup fails, privileges are NOT dropped.)

Or:

$PrivDropToGroup - The name of the group Rsyslog should run under after startup. (Please note that this user is looked up in the system tables. If the lookup fails, privileges are NOT dropped.)

Just set them up in your config file.

Old spool files / unsynced spool files can prevent Rsyslog from functioning

If you have swapped files in your config definition, or have deleted the monitored file on the file system, or if something else weird just happened, it is a good idea to clean up the spool directory files:

$ service rsyslog stop
$ rm -f /var/spool/rsyslog/*
$ service rsyslog restart

 

If spool files have not been re-created after restart, try to restart again. (Yes, Rsyslog can be nasty!) 

Check the Linux system log for Rsyslog errors. You should see a new log with no errors.

Some logs may also be in /var/log/syslog OR /var/log/messages. In Ubuntu, it will output to /var/log/rsyslog/rsyslog.log.

Make sure that the config file output logs to the correct host and port

Logz.io will listeners host name:: listener.logz.io

Logz.io Listener Syslog Port: 5000

Validate connectivity:

Option 1 (using netcat):

$ echo hello | listener.logz.io 5000

 

Good outcome: Getting back the prompt very quickly in < 2 seconds

Bad outcome (port closed): Takes more than 10 seconds

Option 2 (telnet):

$ telnet listener.logz.io 5000

 

Good outcome: 

Connected to listener-group.logz.io.
Escape character is '^]'.

 

(to get out of that, type Ctrl+] and type "quit")

Bad outcome:  

trying xxx.xxx.xxx.xxx....

 

Validate connection established (example):

$ sudo netstat -taupn | grep syslog

tcp        0      0 172.31.3.89:33820           54.210.40.1:5000            ESTABLISHED 23615/rsyslogd

 

If no output is been sent, something is wrong. Check your network connectivity again.

Manually put something in the shipped log file to see if it is sent:

echo hello >> /var/log/my_log_file.log