Defining AppTags for Faster Error Detecting

Make your searches for anomalies in your log4j logs more effective with AppTags

There are many tools and methods out there speeding up your search for errors, but have you tried adding AppTags to your logs? In this series of posts I am covering some of the ways you can benefit from XpoLog V6’s new features and enhancements. I will concentrate mainly on how to get the most valuable information from your log4j event logs.

Once your log4j logs have been transferred to and properly defined in XpoLog Center, you can troubleshoot your java application by running Analytic Search on your log4j data, measure your application performance, create your own Apps or use XpoLog’s Apps for better monitoring, and create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum analysis.

This post will show you how to add AppTags to your logs to perform more enhanced searches. To read all our posts together, see our online hands-on-guide. If you want to test out the software as you go along, you can download it for free.

Try Xpolog Now
Adding AppTags to your logs in XpoLog can make any search, simple or complex, extremely powerful; as you will see later on.

Adding AppTags to log4j logs

If you look at one of my previous posts, Log collection and Appender configuration, you will notice that when adding a log to XpoLog, you are given the option of tagging the log to a number of applications, and you can also create new AppTags.

log4j post 4 tag to app in add log

Once your data has been transferred into XpoLog and been properly defined with regards to pattern and AppTags, start searching. You can do a Simple Search or a Complex Search. A good and detailed pattern together with well thought-through AppTags for a log will make either search that much more effective. To show you the power of the AppTags, I will begin by drawing a sketch:

sketch log4j better version from Lior

Imagine you have 3 servers, Server1, Server2, and Server3, and plenty of log4j files in each server. There are times you want to search in all of the files in all of the servers, and times you want to search files in one or two of the servers.

Inside XpoLog, create a log for each server, call them log4j_server1, log4j_server2, and log4j_server3. Then create an AppTag for all three logs and call it Log4J. Create an AppTag for the first log called “Georgia” and an AppTag for the second and third logs called “Atlanta”. Imagine that Georgia and Atlanta are the locations of your servers.

Simple Search examples

Now let’s do a simple search where we utilize the AppTags in the sketch. Say we want to look for a word that appears in the log4j files in Server1. Inside XpoLog Center, go to Search, in the search field, type:

* in log.log4j server1

where * = anything.

The result will show you what anomalies appear in any file in Server1.

Now look for the word “ERROR” in all the files on all 3 servers. Inside XpoLog Center, go to Search, in the search field, type:

error in log.log4j server*

where * = anything. Hence, in this example, the * refers to the numbers 1, 2, and 3.
The result will show you where the word “ERROR” appears in any file in any of the 3 servers.

Log4j Post 4 ex1b search error in all log log4jservers fixed

Now look for the word “remote” in any file on any server that is situated in Atlanta. In our example this means Server2 and Server3. Inside XpoLog Center, go to Search, in the search field, type:

remote IN apptag Atlanta

The result will show you where the word “remote” appears in any file in any server that has been tagged with “Atlanta”.

Log4j Post 4 ex1c search word in apptag_ATLANTA fixed

The reason this AppTag is so useful is that should you add more servers in Atlanta, or Georgia, and you want to just continue looking for texts or abnormalities in these servers, the moment you give the new server the AppTag “Atlanta” or “Georgia”, XpoLog will continue its search and automatically include searching through any files placed on the new servers. The same goes for removing a server. Once a server is removed, XpoLog will automatically continue its search through all files on all servers that are still there. No further configuration is necessary.

Now let’s look at an example which takes the pattern into consideration. In the previous post we saw how editing the pattern adds columns to the Log records analysis result field. In the Pattern Editor, we added the priority. Let’s conduct a search where we look for all log4j logs where the priority is ERROR. Inside XpoLog Center, go to Search, in the search field, type:

priority=error in log.log4j server*

* = anything. Hence, in this example we are searching through all the files on all the servers.

Log4j Post 4 priority error in all log4j servers fixed

Complex Search example

XpoLog automatically detects errors in the search results and presents these as suggestions (tagged to low/medium/high severities) next to the search results. This technique is known as “Integrated Layers” and it boosts troubleshooting and exposes issues in the logs you may never have thought of and this in turn helps you find the source of various problems faster.

Similar to the Simple Search, running a Complex Search query results in a summary table, presented in a tabular format, and you can also create dashboards and other visualization gadgets for an easier, more natural view; something I will cover in my upcoming posts. XpoLog performs advanced complex operations and reporting on any log events according to the criteria you ask for.

As an example, let’s look for the word “ERROR” or “Exception” in all log4j files on all servers, but also ask XpoLog to count how many errors (or exceptions) were found in each class. Inside XpoLog Center, go to Search, in the search field, type:

error or exception in log.log4j server* | count | group by class

The result will show you a table with a list of all classes where errors were found, and how many errors in each.

Log4j Post 4 complex search error OR exception count group by class fixed

Complex Search provides the option to aggregate log data and to generate advanced statistics, trends, business intelligence, and transactions analysis on the log data. I will speak more about this in my next post. Stay tuned or check out our documentation.

Try Xpolog Now

Refined Data Parsing: Log4j Patterns

Defining and Editing log4j Patterns in SysLog for more Refined Data Parsing

XpoLog’s updated version will not overlook any piece of raw data, no matter how small or insignificant it may seem. In this series of posts I am covering some of the ways you can benefit from XpoLog V6’s new features and enhancements. I will concentrate mainly on how to get the most valuable information from your log4j event logs.

Try Xpolog Now

Once your log4j logs have been transferred to and properly defined in XpoLog Center, you can troubleshoot your java application by running Analytic Search on your log4j data, measure your application performance, create your own AppTags for better monitoring, and create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum analysis. For details, check out our hands-on-guide.

This post will show you how to define and edit your events and log patterns before and after they reach XpoLog Center, when sending them through SysLog. By creating the most readable data you will allow for XpoLog to perform the highest detailed analysis of your logs. To follow more easily as I go along you can download the software for free.

Since logs are written in free format, XpoLog has an advanced built-in mechanism to detect the structure, or pattern, of the incoming log. As a user, you can edit and fine-tune these patterns to suit your needs.

Defining Patterns in SysLog Appenders

When sending events to XpoLog through SysLog, be sure to create a detailed conversion pattern while configuring your log4j SysLog appender. Here is an example:

#Logger definition
log4j.logger.events=INFO, SYSLOG
#Appender data for syslog
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.syslogHost=127.0.0.1
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.conversionPattern=[%t] %c%m%n
log4j.appender.SYSLOG.Facility=LOCAL1

(t = thread, c = class, m = message, and n = new line)

The SysLog appender will write this event logger to the SysLog. Remember to define a SysLog Listener account inside XpoLog Center. See my previous blog post for the instructions on how to do that.

The events that arrive at XpoLog Center are written internally. Here is what they might look like when created by the XpoLog SysLog listener:

XPLG:[1436716542132] [local1] [INFO] [test-1] []: [http-30303-Processor24] audit – [Master] [-] [LOGIN] [login/logout] [SECURITY] [http-30303-Processor24] [-] [-] [-] [-] release user admin

XPLG:[1436716542140] [local1] [INFO] [test-1] []: [http-30303-Processor24] audit – [Master] [Admin] [LOGIN] [login/logout] [SECURITY] [http-30303-Processor24] [EDA6FECA79A7BBB4480BAFC0FFB911F1] [administrators] [127.0.0.1] [127.0.0.1] login with username admin ok

The text at the very beginning is the extra data added by the XpoLog Syslog listener. The other parts of the text in the SysLog file correspond to the layout you created in the log4j SysLog appender (follow the color scheme).

Once the data arrives into XpoLog, a log is created with the default SysLog pattern:

XPLG:[{timestamp:Timestamp,MM/dd/yyyy HH:mm:ss.SSS}] [{text:Facility}] [{priority:Level,DEBUG;INFO;WARN;ERROR;FATAL}] [{text:Source Device}] {block,start,emptiness=true} {string:Message}

Edit the log and set the pattern to reflect the layout you defined in the log4j configuration.

To edit the pattern in a log in XpoLog:

1. In XpoLog Center, go to Administration and find the log under Folders and Logs in the tree in the left margin. Right-click on the log and select Edit.

blog 3 log4j Automaticall generated log4j syslog log - logviewer before editing

The Edit Log screen opens.

2. Click Next to get to the Log Pattern section. The pattern can be edited in the Pattern1 field of the Pattern Editor, or you can add a new pattern in addition to the existing one by clicking the New tab.

Toggle between the Manual button (far right) and the Wizard button to see either version of the pattern.

You can add as many patterns as you want by clicking the New tab. XpoLog will save all these patterns as templates for forthcoming logs.

3. Click Save.

In the screen capture below you can see how to define the log data pattern. It is displayed in the Pattern1 field. The pattern for this log is the following:

XPLG:[{timestamp:Timestamp,MM/dd/yyyy HH:mm:ss.SSS}] [{text:Facility}] [{priority:Level,DEBUG;INFO;WARN;ERROR;FATAL}] [{text:Source Device}] {block,start,emptiness=true}{text:Application Name}[{text:Process Id}]: {block,end,emptiness=true}{string:Message}

LOG4J blog 2 thread class message format small

Most of the pattern, up to and including {block,end,emptiness=true}, is part of the SysLog protocol and functions as an prefix to the message – it contains the SysLog timestamp, facility, priority and the source device.

As mentioned previously, you can edit the pattern inside XpoLog Center after the event logs have been sent. If your messages all follow the same structure, we recommend further editing the pattern to include this structure, to receive a more refined parsing. Here is a more refined pattern of the log shown above:

XPLG:[{timestamp:Timestamp,MM/dd/yyyy HH:mm:ss.SSS}] [{text:Facility}] [{priority:Level,DEBUG;INFO;WARN;ERROR;FATAL}] [{text:Source Device}] {block,start,emptiness=true}{text:Application Name}[{text:Process Id}]: {block,end,emptiness=true}[{text:ServerIp}] [{text:User,User}] [{choice:Action Type,LOGIN;VIEW;CHANGE}] [{text:Action description,Action description}] [{choice:Context,LOGS;FOLDERS;VERIFIERS;CONFIGURATION;

SECURITY;REPROTS;TASKS;JOBS;NODES;SEARCH_ENGINE}] {string:Message}

The following screen capture shows the same log as above, after editing. You can see the original message has been split into the relevant columns.

LOG4J 2nd blog refined audit log format small

Note that by creating the most readable data, you will receive the most detailed analysis of your logs from XpoLog.

Try Xpolog Now

In my next post, I will discuss how to tag the logs with AppTags, for easier monitoring, troubleshooting, and search. Stay tuned or go directly to our hands-on-guide.