Developer Games: RegExp and log4j Parsing

Extracting Valuable Data from log4j logs with Virtual Fields

In our recent upgrade to XpoLog V6 we enhanced the features of log4j analysis. In this series of posts I am covering some of the ways you can benefit from XpoLog V6’s new features and enhancements. I will concentrate mainly on how to get the most valuable information from your log4j event logs.

Once your log4j logs have been transferred to and properly defined in the XpoLog Center, you can troubleshoot your java application by running Analytic Search on your log4j data, measure your application performance, create your own AppTags for better monitoring, and create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum log analysis.

Try Xpolog Now

This post will show you how to define log4j logs in XpoLog, to create the most readable data and thus allow for XpoLog to perform highly detailed analysis of your logs. I will also show you an example of how you can virtually extract specific data from your message using Regular Expression to allow for XpoLog to perform a more refined parsing of your data. If you prefer to read our whole manual in one go, you can find it here.

Defining Patterns in XpoLog Center

If you are letting XpoLog access and pull data from your files, define the logger with a name, pattern and data pattern, and then define the log patterns in XpoLog Center.

For example:

#Logger definition

#Appender data for mylog

log4j.appender.mylog.layout.ConversionPattern=[%d] [%t] [%p] [%c] [%l] %m%n

(d = date, t = thread, p = priority, c = class, l = method, m = message, and n = new line)

Defining the log pattern in XpoLog Center:

  1. In XpoLog Center, add a new log. (See my instructions in the previous blog.) Once you have filled in the details, click Next to get to the Log Pattern screen.
  1. In the Wizard of the Pattern Editor, define the log pattern.

Log4J edit log4 wizard toggle for blog 2

Click Manual in the Pattern Editor and edit the XpoLog data pattern to comply with the log4j layout:

a. [%d] = [{date:Date,locale=en,yyyy-MM-dd HH:mm:ss,SSS}]
b. [%t] = [{text:Thread}]
c. [%p]= [{priority:Priority,DEBUG;INFO;WARNING;ERROR;FATAL}]
d. [%c]= [{string:Class}]
e. [%l]= [{string:Method}({text:Source}:{number:LineNumber})]
f. %m = {string:Message}
g. %n = new line

Log4J edit log5 manual toggle for blog 2

The XpoLog pattern in our example will be:

[{date:Date,locale=en,yyyy-MM-dd HH:mm:ss,SSS}] [{text:Thread}] [{priority:Priority,DEBUG;INFO;WARNING;ERROR;FATAL}] [{string:Class}] [{string:Method}({text:Source}:{number:LineNumber})] {string:Message}

  1. Click Save.

You can also edit the pattern after you have added the log, of which I will speak more of in my next post.

Virtually Extract Specific Data from your Message

XpoLog can also extract data from within the message if you use Regular Expression prior to the data transfer.

This is what the message might look like in the Log Pattern section of the Add Log screen without using Regular Expression:

In the Pattern Editor, all you see is {string:Message}.

log4j admin 1 message before regular expression

If you use Regular Expression to extract any word that appears after the word “Manager”, the Log Pattern section of the Add Log screen would look as follows:

In the Pattern Editor, you will now see:

{regexp:HTMLManager state,refName=Message,HTMLManager: (\w+)}{string:Message}

In the Log records analysis result section below XpoLog has added the column HTMLManager state for the data you wished to extract.

log4j admin 2 message after regular expression

In the Manager Interface of XpoLog Center, where you view your logs, you will also see this extra column, HTMLManager state, for the extracted data:

log4j log viewer after regular expression

By extracting the HTMLManager state into a new virtual field we can now measure and monitor the HTMLManager state performance and activity.

Try Xpolog Now

In the next post, I will show how to define and edit the log4j patterns when sending log events and log messages to XpoLog through SysLog. Stay tuned, or go directly to our hands-on-guide.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s