How can I parse my logs?

You can parse your logs using our Data Parsing Wizard. To use this feature, click on the cogwheel in the top right corner, go to Tools --> Data Parsing

Note: Our Data Parsing is currently in BETA.

Before we go into details on how to use the Data Parsing feature, keep in mind that we offer Parsing as a Service to all of our users. Simply send us a request and we will handle all of your parsing needs. For more details visit this page: Parsing Request

In order for the Data Parsing Wizard to work, you should first send custom logs to your Logz.io account. Once that has been done, let's start parsing our logs:

1. Setup - Start by choosing your Data Source from the dropdown menu, then click Next
   
   
   
2. Parse - This step is where you define how you want your log to be broken up and parsed
   
   1. After entering your Pattern Name
   2. Click on Select to choose some Sample Log Line
      
   3. From there, select up to five sample logs out of the last five hundred that have been shipped to Logz.io to create your parsing
   4. Click on Select
   5. You will then be able to choose your Parse Method from the dropdown. As previously mentioned, Grok is supported with Delimiter, JSON and Key Value parsing will be introduced soon.
   6. Start entering your Grok Pattern
      Note: We recommend using both the Grok Debugger and this list of grok patterns as reference.
      
   7. Extra Pointer:
      1. As soon as you begin grokking, your log lines will begin to be parsed into separate fields in the Parse Results table below. The colors help you to match the fields in the log lines with your parsing results. Make sure you see the name you chose for the fields in the log.
      2. To omit data, simply do not name the fields.
      3. Using “message” as the field name overrides the existing message field in the log, and you can use it as many times as you want — the values for this field will be concatenated into a comma-separated message field. If you do not use this field name in the grok pattern, the default message field will be used.
      4. After entering your grok pattern, you can define a field type for each field that you parse.
      5. While we recommend leaving this default setting (“Automatic”), you have the option to define other types such as boolean, date, IP, and byte. For geo-enrichment for example, you will need to select  the “Geo-Enrichment” field type.
         
         
3. Enrich - You can apply some advanced parsing customizations
   
   
   1. If you selected to parse one of the IP fields as a Geo IP field in the previous step, you will now be able to decide with which geo fields to enrich the field.
   2. Under “Set Timestamp,” you will be able to configure all the timestamp fields that appear within your logs. For example, you can set which timestamp is the leading timestamp (a leading timestamp determines the sequential order in which the logs are displayed).
      
      
4. Validate - It’s now time to make sure that you are happy with the parsing results
   

Some points that are worth highlighting:

• Data Parsing feature can only be used on log types that do not have pre-defined parsing. If you would like to apply changes to those logs anyway, contact help@logz.io.
• Again, Grok is the only supported parsing method. Delimiter, JSON, and Key Value parsing will be introduced soon.
• Our system will also monitor and automatically identify parsing that isn't efficient, notifies our Support team, who will reach out to assist you making your parsing better for you and our system.
• Why do the heavy lifting? We can do it for you - Parsing Request