Log Shipping Instructions

Logz.io offers a variety of shippers and technologies for shipping logs our way, and choosing the "best" shipping method is key for your success.

So, what is the best method used for shipping anyway?

Well, it really depends on the technologies you are using, and whenever choosing a shipping method for your environment, you will most likely never find the one shipping method that works best. You will typically want to look for the combination of methods that will produce the results you want, finding a single method may not be ideal because there are so many types of logs in first place.

Now, let's break down the major ways you can ship logs to Logz.io:

Note: For individual instructions on how to ship logs, please visit our Log Shipping page within the Logz.io application.  

1. From the Code (Libraries):

If you have control over your code, you can ship directly from the code itself to Logz.io by adding one of our Appenders, and the advantages are:

• Support the stack trace (most appenders)
• More Parsed data out of the box
• Plus, wherever you deploy the code (with network access), the logs will be shipped to Logz.io, not extra needed configuration is needed

Note: We also have endpoints (by using HTTP or TCP) that you can send us your logs in JSON format.

2. Using a Shipper (Log Shippers):

If you do not have control over your code, or you if you have products/solutions that write to log files, you may use a log shipper that will monitor and send the logs to Logz.io, and the advantages are:

• Support multiple sources, log types, and patterns
• Our recommended Shipper is Filebeat, simply because:
  • We offer a configuration Wizard
  • Works in all environments
  • Single Port (5015) - plus, the whole Beats environment uses the same protocol 
  • Compress and secures your data
  • Easy to troubleshoot and configure
• So why would you use another Log Shippers?
  
  • Depending on your environment, you may run into some limitations when it comes to Filebeat itself, or maybe you are already familiar with another shipper, so go with what makes the most sense for you
  • Important: We get a lot of questions about shipping Network Device data. Please keep in mind that most devices are very minimal in configuration (host and port as output only), and because Logz.io requires a Token, you will need to setup a mini-intermediate server that listens on the port and collects the logs to be shipped to Logz.io. Currently the best options are: Rsyslogs or FluentD

3. Docker Shippers:

If you are using Docker, and you are containerizing the options above, you may use our Docker shippers as well, and the advantages are:

• Easily configurable in a Docker environment 
• Listen on the standard out of all of the Dockers on the host, and forward the logs to Logz.io

But its disadvantages are:

• It does not support Multiline, every line will come separately 
• Cannot filter what is sent to Logz.io
  • You may filter the whole container, but not what logs from individual containers

4. AWS S3:

If you are shipping logs to your own S3 bucket, and the advantages are:

• The bucket is yours, and you can keep it forever on your side
• Ideal for services that write consistently and in a consistent log format
  • Note: Cannot be used as a random bucket to "throw" your logs in
• Specially good for services offered by Amazon

Next Steps:

After choosing the ideal shipping method, let's make sure you are getting the most value from your logs... let's parse them:

1. Why should I parse my logs?
2. Which log types are preconfigured on the Logz.io platform?
3. How do I parse logs in a custom format?