Beware of what you wish for…

Auto detecting new errors, exceptions and bugs in log4j logs

XpoLog’s Analytic Search has been around for quite a while, but our latest version brings many new gadgets and such a friendly user interface that you had better be careful what you wish for, because XpoLog’s unique search engine not only finds what you are looking for, but also all the things you never even thought to look for in the first place, with a complete analysis, and dishes it all out to you on a silver platter.

Try Xpolog Now

In this series of posts I am covering some of the ways you can benefit from XpoLog 6’s new features and enhancements and especially on how to get the most valuable information from your log4j event logs.

By running Analytic Search on your log4j data, you can measure your application performance and thread activity, create your own Apps for better monitoring, measure code activity with class and method analytics on log4j, build security analysis, and create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum analysis.

In this post I will show you the basics of XpoLog Analytics, how to auto detect new errors, exceptions and bugs in log4j logs, and discover unknown messages.

From Search to Analytics

In my previous post I discussed Simple and Complex Search using XpoLog’s powerful search engine. Already at this stage, XpoLog suggests analytic insights you may be interested in investigating further.

To put it simply, while XpoLog’s Search Engine gives you everything you asked for, XpoLog Analytics gives you everything else.

If you are searching for a string, a thread, an error, etc. in one or any number of logs, folders, applications, or servers, within a given time frame, XpoLog’s search engine will find and display all cases/events of the search request within all logs. But XpoLog will also open the door for any other abnormality that may occur in these logs within this time frame, and this brings us to Analytics. In other words, as soon as you conduct a search, XpoLog will already automatically present you with all other issues that you did not even know existed, be it errors, exceptions, unknown messages, or any other anomaly.

As an example, look at the simple search we did (in my previous post) looking for all log4j logs where the priority was ERROR. Quick recap: Inside XpoLog Center, on the Search page, in the search field, we typed:

priority=error in log.log4j server*

The result looked like this:

Log4j Post 5 taken from post 4

Below the graph, XpoLog displays all the events where the priority=ERROR.  But in the side bar (see red rectangle), XpoLog has already suggested Analytic Insights, such as ERROR,, … and the list goes on. These Insights may not necessarily appear in any of the events where priority=ERROR, but they do appear somewhere in these logs, and hence, they may indicate that something went wrong somewhere.

So already, at this stage when all you are doing is searching for something, XpoLog is already several steps ahead, analyzing, and inviting you to dig deeper to find the root of the issue.  From the Analytics Insight list, we can select one or more insights and either add them to the search, or use them to replace the search. We can then investigate the matter further, in XpoLog Analytics.

Inside XpoLog Analytics

The screen capture below shows the Analytics page. The top section has a graph showing you the data distribution and the maximum severity of the events over the selected time-span. Below the graph is a table showing the logs and folders in which these events were found. Below this section is another table showing the 10 most severe errors that were found in any of these logs and folders.

Analytics - apptag view

When listing the logs, XpoLog lists the logs containing the problems with the high severity first (red), then all those with medium priority problems (orange), and lastly, those containing low priority problems (green). XpoLog decides the severity level according to the highest severity anomaly found in the event. You may be searching for an anomaly with a medium severity, but if, in the same event, another anomaly with high severity is found, the event as a whole will be marked as high priority.

search failed to initialize hudson small

In the screen capture above, a search was conducted for Failed to initialize hudson, which has medium priority, but within the same event, XpoLog found a hidden message, java.lang.OutOfMemoryError, which has high priority, thus bringing the entire event up to high priority.

Drilling down

From the initial Analytics page, which by default shows the total summary of all anomalies in all logs of your search, you can drill down for more specific details. For the sake of our example, let’s drill down into log4j:

Analytics - apptag log4j drilldown with magnify

The Analytics page has now drilled down to the log4j level (see screen capture below). You can see the number of anomalies has been reduced, as has the amount of data being depicted. The first table below the graph now contains only folders of the log4j applications and the second table shows the most severe log4j problems found.

Analytics - apptag log4j after drilldown

Log4j Use Cases

Let’s have a look at a potential use case. Inside log4j is a tomcat folder.

A user is complaining that tomcat will not start. We don’t know what the problem is, so the easiest way to find out is to do a general search for anything abnormal going on in tomcat in the given time frame when the user was unable to start it.

Inside XpoLog Search, in the search field, we type the following query:

* IN folder tomcat 8

In addition to the requested search results, XpoLog suggests many more analytic insights. There could be many reasons why tomcat did not start, so from the Analytics Insight list, we will select and replaced the * search with this query:

“” IN folder tomcat 8 

The screen capture below shows how to replace the existing search query: right-click on the insight and then select Replace search:

Search all - replace bind exception

The search result for “” IN folder tomcat 8 will look as follows:

Search bind exception

We now have completely new events in the list and we see that for the insight “” that the “address is already in use”. This is why the user was unable to start tomcat.


In another example, we created an App called hudson. Hudson worked for a while, but at some point it stopped. We did not know why, so we did a search for hudson. In the search field, we typed:

hudson IN folder tomcat 8

We got the following results:

hudson with analytics div fixed

Here we can see in the Analytics Insight list that there is a high priority error: Java.lang.OutOf MemoryError.  In this particular example, this error occurs in the first event on our list of events containing “hudson”. We now already know why hudson stopped working. By zooming in and hovering over the graph, you also get a presentation of the high, middle, and low priority errors per moment in time.

Bottom line…

So we can see from these examples and use cases that while users are simply searching for known anomalies, or even searching for any anomaly without pre-knowledge, XpoLog is already several steps ahead analyzing everything else about the logs in the search that the user never thought of. In the fraction of a second it takes XpoLog to do the requested search, a complete list of analytic insights are also created, waiting for the users and inviting them to dig deeper into their logs, folders, apps or servers, to get to the root of whatever is causing them trouble.

So by now you should have figured out why we call our searches “Analytic Search”…

In my next post I will take XpoLog’s Analytic Search a step further and show you how to check your application’s performance and availability. Stay tuned; or go directly to “Getting the Maximum from your log4j logs”.

Try Xpolog Now

Defining AppTags for Faster Error Detecting

Make your searches for anomalies in your log4j logs more effective with AppTags

There are many tools and methods out there speeding up your search for errors, but have you tried adding AppTags to your logs? In this series of posts I am covering some of the ways you can benefit from XpoLog V6’s new features and enhancements. I will concentrate mainly on how to get the most valuable information from your log4j event logs.

Once your log4j logs have been transferred to and properly defined in XpoLog Center, you can troubleshoot your java application by running Analytic Search on your log4j data, measure your application performance, create your own Apps or use XpoLog’s Apps for better monitoring, and create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum analysis.

This post will show you how to add AppTags to your logs to perform more enhanced searches. To read all our posts together, see our online hands-on-guide. If you want to test out the software as you go along, you can download it for free.

Try Xpolog Now
Adding AppTags to your logs in XpoLog can make any search, simple or complex, extremely powerful; as you will see later on.

Adding AppTags to log4j logs

If you look at one of my previous posts, Log collection and Appender configuration, you will notice that when adding a log to XpoLog, you are given the option of tagging the log to a number of applications, and you can also create new AppTags.

log4j post 4 tag to app in add log

Once your data has been transferred into XpoLog and been properly defined with regards to pattern and AppTags, start searching. You can do a Simple Search or a Complex Search. A good and detailed pattern together with well thought-through AppTags for a log will make either search that much more effective. To show you the power of the AppTags, I will begin by drawing a sketch:

sketch log4j better version from Lior

Imagine you have 3 servers, Server1, Server2, and Server3, and plenty of log4j files in each server. There are times you want to search in all of the files in all of the servers, and times you want to search files in one or two of the servers.

Inside XpoLog, create a log for each server, call them log4j_server1, log4j_server2, and log4j_server3. Then create an AppTag for all three logs and call it Log4J. Create an AppTag for the first log called “Georgia” and an AppTag for the second and third logs called “Atlanta”. Imagine that Georgia and Atlanta are the locations of your servers.

Simple Search examples

Now let’s do a simple search where we utilize the AppTags in the sketch. Say we want to look for a word that appears in the log4j files in Server1. Inside XpoLog Center, go to Search, in the search field, type:

* in log.log4j server1

where * = anything.

The result will show you what anomalies appear in any file in Server1.

Now look for the word “ERROR” in all the files on all 3 servers. Inside XpoLog Center, go to Search, in the search field, type:

error in log.log4j server*

where * = anything. Hence, in this example, the * refers to the numbers 1, 2, and 3.
The result will show you where the word “ERROR” appears in any file in any of the 3 servers.

Log4j Post 4 ex1b search error in all log log4jservers fixed

Now look for the word “remote” in any file on any server that is situated in Atlanta. In our example this means Server2 and Server3. Inside XpoLog Center, go to Search, in the search field, type:

remote IN apptag Atlanta

The result will show you where the word “remote” appears in any file in any server that has been tagged with “Atlanta”.

Log4j Post 4 ex1c search word in apptag_ATLANTA fixed

The reason this AppTag is so useful is that should you add more servers in Atlanta, or Georgia, and you want to just continue looking for texts or abnormalities in these servers, the moment you give the new server the AppTag “Atlanta” or “Georgia”, XpoLog will continue its search and automatically include searching through any files placed on the new servers. The same goes for removing a server. Once a server is removed, XpoLog will automatically continue its search through all files on all servers that are still there. No further configuration is necessary.

Now let’s look at an example which takes the pattern into consideration. In the previous post we saw how editing the pattern adds columns to the Log records analysis result field. In the Pattern Editor, we added the priority. Let’s conduct a search where we look for all log4j logs where the priority is ERROR. Inside XpoLog Center, go to Search, in the search field, type:

priority=error in log.log4j server*

* = anything. Hence, in this example we are searching through all the files on all the servers.

Log4j Post 4 priority error in all log4j servers fixed

Complex Search example

XpoLog automatically detects errors in the search results and presents these as suggestions (tagged to low/medium/high severities) next to the search results. This technique is known as “Integrated Layers” and it boosts troubleshooting and exposes issues in the logs you may never have thought of and this in turn helps you find the source of various problems faster.

Similar to the Simple Search, running a Complex Search query results in a summary table, presented in a tabular format, and you can also create dashboards and other visualization gadgets for an easier, more natural view; something I will cover in my upcoming posts. XpoLog performs advanced complex operations and reporting on any log events according to the criteria you ask for.

As an example, let’s look for the word “ERROR” or “Exception” in all log4j files on all servers, but also ask XpoLog to count how many errors (or exceptions) were found in each class. Inside XpoLog Center, go to Search, in the search field, type:

error or exception in log.log4j server* | count | group by class

The result will show you a table with a list of all classes where errors were found, and how many errors in each.

Log4j Post 4 complex search error OR exception count group by class fixed

Complex Search provides the option to aggregate log data and to generate advanced statistics, trends, business intelligence, and transactions analysis on the log data. I will speak more about this in my next post. Stay tuned or check out our documentation.

Try Xpolog Now

Refined Data Parsing: Log4j Patterns

Defining and Editing log4j Patterns in SysLog for more Refined Data Parsing

XpoLog’s updated version will not overlook any piece of raw data, no matter how small or insignificant it may seem. In this series of posts I am covering some of the ways you can benefit from XpoLog V6’s new features and enhancements. I will concentrate mainly on how to get the most valuable information from your log4j event logs.

Try Xpolog Now

Once your log4j logs have been transferred to and properly defined in XpoLog Center, you can troubleshoot your java application by running Analytic Search on your log4j data, measure your application performance, create your own AppTags for better monitoring, and create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum analysis. For details, check out our hands-on-guide.

This post will show you how to define and edit your events and log patterns before and after they reach XpoLog Center, when sending them through SysLog. By creating the most readable data you will allow for XpoLog to perform the highest detailed analysis of your logs. To follow more easily as I go along you can download the software for free.

Since logs are written in free format, XpoLog has an advanced built-in mechanism to detect the structure, or pattern, of the incoming log. As a user, you can edit and fine-tune these patterns to suit your needs.

Defining Patterns in SysLog Appenders

When sending events to XpoLog through SysLog, be sure to create a detailed conversion pattern while configuring your log4j SysLog appender. Here is an example:

#Logger definition, SYSLOG
#Appender data for syslog
log4j.appender.SYSLOG.layout.conversionPattern=[%t] %c%m%n

(t = thread, c = class, m = message, and n = new line)

The SysLog appender will write this event logger to the SysLog. Remember to define a SysLog Listener account inside XpoLog Center. See my previous blog post for the instructions on how to do that.

The events that arrive at XpoLog Center are written internally. Here is what they might look like when created by the XpoLog SysLog listener:

XPLG:[1436716542132] [local1] [INFO] [test-1] []: [http-30303-Processor24] audit – [Master] [-] [LOGIN] [login/logout] [SECURITY] [http-30303-Processor24] [-] [-] [-] [-] release user admin

XPLG:[1436716542140] [local1] [INFO] [test-1] []: [http-30303-Processor24] audit – [Master] [Admin] [LOGIN] [login/logout] [SECURITY] [http-30303-Processor24] [EDA6FECA79A7BBB4480BAFC0FFB911F1] [administrators] [] [] login with username admin ok

The text at the very beginning is the extra data added by the XpoLog Syslog listener. The other parts of the text in the SysLog file correspond to the layout you created in the log4j SysLog appender (follow the color scheme).

Once the data arrives into XpoLog, a log is created with the default SysLog pattern:

XPLG:[{timestamp:Timestamp,MM/dd/yyyy HH:mm:ss.SSS}] [{text:Facility}] [{priority:Level,DEBUG;INFO;WARN;ERROR;FATAL}] [{text:Source Device}] {block,start,emptiness=true} {string:Message}

Edit the log and set the pattern to reflect the layout you defined in the log4j configuration.

To edit the pattern in a log in XpoLog:

1. In XpoLog Center, go to Administration and find the log under Folders and Logs in the tree in the left margin. Right-click on the log and select Edit.

blog 3 log4j Automaticall generated log4j syslog log - logviewer before editing

The Edit Log screen opens.

2. Click Next to get to the Log Pattern section. The pattern can be edited in the Pattern1 field of the Pattern Editor, or you can add a new pattern in addition to the existing one by clicking the New tab.

Toggle between the Manual button (far right) and the Wizard button to see either version of the pattern.

You can add as many patterns as you want by clicking the New tab. XpoLog will save all these patterns as templates for forthcoming logs.

3. Click Save.

In the screen capture below you can see how to define the log data pattern. It is displayed in the Pattern1 field. The pattern for this log is the following:

XPLG:[{timestamp:Timestamp,MM/dd/yyyy HH:mm:ss.SSS}] [{text:Facility}] [{priority:Level,DEBUG;INFO;WARN;ERROR;FATAL}] [{text:Source Device}] {block,start,emptiness=true}{text:Application Name}[{text:Process Id}]: {block,end,emptiness=true}{string:Message}

LOG4J blog 2 thread class message format small

Most of the pattern, up to and including {block,end,emptiness=true}, is part of the SysLog protocol and functions as an prefix to the message – it contains the SysLog timestamp, facility, priority and the source device.

As mentioned previously, you can edit the pattern inside XpoLog Center after the event logs have been sent. If your messages all follow the same structure, we recommend further editing the pattern to include this structure, to receive a more refined parsing. Here is a more refined pattern of the log shown above:

XPLG:[{timestamp:Timestamp,MM/dd/yyyy HH:mm:ss.SSS}] [{text:Facility}] [{priority:Level,DEBUG;INFO;WARN;ERROR;FATAL}] [{text:Source Device}] {block,start,emptiness=true}{text:Application Name}[{text:Process Id}]: {block,end,emptiness=true}[{text:ServerIp}] [{text:User,User}] [{choice:Action Type,LOGIN;VIEW;CHANGE}] [{text:Action description,Action description}] [{choice:Context,LOGS;FOLDERS;VERIFIERS;CONFIGURATION;


The following screen capture shows the same log as above, after editing. You can see the original message has been split into the relevant columns.

LOG4J 2nd blog refined audit log format small

Note that by creating the most readable data, you will receive the most detailed analysis of your logs from XpoLog.

Try Xpolog Now

In my next post, I will discuss how to tag the logs with AppTags, for easier monitoring, troubleshooting, and search. Stay tuned or go directly to our hands-on-guide.

Developer Games: RegExp and log4j Parsing

Extracting Valuable Data from log4j logs with Virtual Fields

In our recent upgrade to XpoLog V6 we enhanced the features of log4j analysis. In this series of posts I am covering some of the ways you can benefit from XpoLog V6’s new features and enhancements. I will concentrate mainly on how to get the most valuable information from your log4j event logs.

Once your log4j logs have been transferred to and properly defined in the XpoLog Center, you can troubleshoot your java application by running Analytic Search on your log4j data, measure your application performance, create your own AppTags for better monitoring, and create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum log analysis.

Try Xpolog Now

This post will show you how to define log4j logs in XpoLog, to create the most readable data and thus allow for XpoLog to perform highly detailed analysis of your logs. I will also show you an example of how you can virtually extract specific data from your message using Regular Expression to allow for XpoLog to perform a more refined parsing of your data. If you prefer to read our whole manual in one go, you can find it here.

Defining Patterns in XpoLog Center

If you are letting XpoLog access and pull data from your files, define the logger with a name, pattern and data pattern, and then define the log patterns in XpoLog Center.

For example:

#Logger definition

#Appender data for mylog

log4j.appender.mylog.layout.ConversionPattern=[%d] [%t] [%p] [%c] [%l] %m%n

(d = date, t = thread, p = priority, c = class, l = method, m = message, and n = new line)

Defining the log pattern in XpoLog Center:

  1. In XpoLog Center, add a new log. (See my instructions in the previous blog.) Once you have filled in the details, click Next to get to the Log Pattern screen.
  1. In the Wizard of the Pattern Editor, define the log pattern.

Log4J edit log4 wizard toggle for blog 2

Click Manual in the Pattern Editor and edit the XpoLog data pattern to comply with the log4j layout:

a. [%d] = [{date:Date,locale=en,yyyy-MM-dd HH:mm:ss,SSS}]
b. [%t] = [{text:Thread}]
c. [%p]= [{priority:Priority,DEBUG;INFO;WARNING;ERROR;FATAL}]
d. [%c]= [{string:Class}]
e. [%l]= [{string:Method}({text:Source}:{number:LineNumber})]
f. %m = {string:Message}
g. %n = new line

Log4J edit log5 manual toggle for blog 2

The XpoLog pattern in our example will be:

[{date:Date,locale=en,yyyy-MM-dd HH:mm:ss,SSS}] [{text:Thread}] [{priority:Priority,DEBUG;INFO;WARNING;ERROR;FATAL}] [{string:Class}] [{string:Method}({text:Source}:{number:LineNumber})] {string:Message}

  1. Click Save.

You can also edit the pattern after you have added the log, of which I will speak more of in my next post.

Virtually Extract Specific Data from your Message

XpoLog can also extract data from within the message if you use Regular Expression prior to the data transfer.

This is what the message might look like in the Log Pattern section of the Add Log screen without using Regular Expression:

In the Pattern Editor, all you see is {string:Message}.

log4j admin 1 message before regular expression

If you use Regular Expression to extract any word that appears after the word “Manager”, the Log Pattern section of the Add Log screen would look as follows:

In the Pattern Editor, you will now see:

{regexp:HTMLManager state,refName=Message,HTMLManager: (\w+)}{string:Message}

In the Log records analysis result section below XpoLog has added the column HTMLManager state for the data you wished to extract.

log4j admin 2 message after regular expression

In the Manager Interface of XpoLog Center, where you view your logs, you will also see this extra column, HTMLManager state, for the extracted data:

log4j log viewer after regular expression

By extracting the HTMLManager state into a new virtual field we can now measure and monitor the HTMLManager state performance and activity.

Try Xpolog Now

In the next post, I will show how to define and edit the log4j patterns when sending log events and log messages to XpoLog through SysLog. Stay tuned, or go directly to our hands-on-guide.

Log collection and Appender configuration for log4j to XpoLog

From log4j to XpoLog

XpoLog V.6 is here and already taking on an Exabyte-sized storm of logs as I write this. In this series of posts I will cover some of the ways you can use and benefit from its new features and enhancements. I will concentrate mainly on how to get the maximum amount of information from your log4j event logs. If you don’t want to wait for our continuation next week, you can look at our full tutorial right away.

Once your log4j logs have been transferred to and properly defined in XpoLog Center, you can troubleshoot your java application by running Analytic Search on your log4j data, measure your application performance, create your own Apps or use XpoLog’s Apps for better monitoring, create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum analysis.

You can download XpoLog for free if you want to follow as you go along. It only takes a few minutes.

Try Xpolog Now

In this post I will show you how to transfer your log events to XpoLog using log4j. There are two ways of doing this. The first method is to allow XpoLog direct access to your files. The other method is by defining a SysLog appender and sending your events and messages to XpoLog. XpoLog supports both methods.

Allowing XpoLog access to your files (PULL)

Assuming you are already using log4j to write your log events to files, to allow for XpoLog to perform analysis on your log data, you need to give XpoLog access to these files. Define the name, pattern, and data pattern so that XpoLog can read these files, collect and index the data, and start analyzing.

Using direct access (Local or Remote)

XpoLog can access a local log file, i.e. a log file that resides on the same server as XpoLog. XpoLog can also access a log file on a remote server to which it has been provided direct access, as long as XpoLog is provided with the UNC path (\\hostname\dirname) to the log files on the remote server.

Using SSH (Secured Shell)

XpoLog can access log files on remote servers over SSH agent-less, provided that XpoLog has an account with a username and password or private/public key for connecting to the SSH server where the log files are situated.

Note that XpoLog requires Read permissions for any log it reads, regardless of the source of the log file.

To allow for XpoLog to pull (data from) the files, define the logger and give XpoLog access to the remote server where the logger is defined; then add the log to XpoLog.

For example:

#Logger definition

#Appender data for mylog


log4j.appender.mylog.layout.ConversionPattern=[%d] [%t] [%p] [%c] [%l] %m%n

Adding a log to XpoLog:

  1. Inside XpoLog Center, go to Manager > Administration > Add Log. The Add Log screen opens.
  2. Give the log a name and a parent folder, and select an AppTag (Tag to Application(s)) from the drop-down list or create a new AppTag. You can select and create any number of AppTags for the same log. You do not have to tag the log at all, but in my forthcoming posts you will see how useful these AppTags can be. If you cannot wait for my next post, have a look at our “spoiler”.
  3. Select the log type to be Local and give a path (the screen capture below shows the example given).

Add Log mylog for 1st blog post

  1. Click Next to view the sample text from the log, the conversion pattern in the Pattern Editor field, and the log records analysis results; or just click Save. (In my next post I will give details regarding editing the pattern in the Pattern Editor  field.)

Sending your log4j log events to XpoLog (PUSH)

To send log events and log messages to XpoLog through SysLog, define a SysLog appender that uses the XpoLog server as the SysLog host. From inside XpoLog Center, define a TCP or a UDP SysLog Listener account and make sure the port (usually 1468 for TCP or 514 for UDP) is open on XpoLog’s machine. We recommend using TCP.

Defining a TCP SysLog Listener account:

  1. Inside XpoLog Center, go to Manager > Administration > Listeners. A Listeners accounts console opens and presents all the configured listeners available.
  2. Click Syslog TCP. The Syslog TCP Account window opens.

SysLog TCP account window small

  1. Add a descriptive name for the Listener account, click Advanced Settings and continue. Note that for General Information > Enabled you make sure the account is enabled.
  1. Click Save. The data received from the Syslog listener account will be placed under the configured parent folder you selected.

Configuring log4j

Now all you need to do is to make sure the SysLog events from your java application are sent to XpoLog. Configure log4j to use a SysLog appender. Here is an example configuration:

log4j.rootLogger=INFO, SYSLOG
log4j.appender.SYSLOG.layout.conversionPattern=%d{ISO8601} %-5p [%t] %c{2} %x – %m%n

After your logs reach XpoLog

Once your log events have been pushed to or pulled by XpoLog, XpoLog can start collecting, parsing, monitoring, and analyzing all your log data. XpoLog V.6 has enhanced its Analytic Search, added over 20 new visualization gadgets to its Apps, and also gives you the opportunity to create your own Apps and Dashboards, making performance monitoring, analysis, and visualization naturally fast and easy.

In my next post I will cover how properly define and edit your java log patterns; thus paving the way for receiving the highest possible value from the XpoLog analysis. Stay tuned, or go directly to our “spoiler” hands-on-guide.

Try Xpolog Now

Gear Up! XpoLog 6 is here: Imagine and Build Your Log Data Apps

by Haim Koschitzky, XpoLog CEO

XpoLog 6 is finally here. In previous posts I presented certain features of this new version in detail. We worked very hard to rebuild the product to make it a solution for our users to better manage their various complex systems. Now that we have the building blocks to build awesome apps for your log data, let us take a look at how this is done.

Add Logs

XpoLog provides highly functional tools to collect and parse log data. We provide some unique capabilities that we built into our log management platform that help manage logs in a very smart, secure, and efficient way. I also recommend checking out the virtual data engine and log parsing service that helps normalize log data automatically.

In recent years we witnessed growing availability of open source projects for log collection and log shipping, some of them being logstash, fluentd and others. Those tools help ship data and create log repositories.

If you are already using tools for log management we have great news for you, XpoLog can be integrated with most of them. Deploy XpoLog to collect or process log data in order to add leading analytics services on the work you already did.

If you are deploying a new log analysis platform, simply add the data to XpoLog using our SysLog, wizards, or other agents.

Organize Log Data and AppTags

Log data is organized as nodes in XpoLog; these data nodes are very powerful. You can change access permission to nodes, move nodes, duplicate nodes with different credentials, and even apply numerous patterns on the same log using multiple data nodes. When nodes are organized in logical structures, such as folders or Apps, it is easy to perform group operations and search queries on this new abstraction layer.

In the AppTags console, tag nodes to the applications, this will allow you to run queries on AppTags, and Apps. In a dynamic world you can add more data nodes to AppTags and your queries will still work for elastic and hybrid environments.

Search and Complex Search of Logs

If you are new to search, this part can look complicated but we have great tools that build search queries for you. XpoLog 6 Analytic Search was enhanced to automatically build millions of queries automatically according to the content of the log data. You can select queries or build them from the analytics services. Use complex search syntax in order to correlate log data and run complex statistical searches.

Become an expert in your application log data, and use the XpoLog 6 Analytic search query builder to discover more insight on your new or existing data repositories.

post for blog xpolog 6Build Log Data Apps

XpoLog 6 brings a refreshing approach to Apps, we created an amazingly simple (AngularJS based) UI work space in which you create apps. Within each App we recommend defining dashboards like: “Availability of service”, “Performance”, “Security”, “Statistics”, “Top Errors” etc. Within each dashboard, define visual gadgets that visualize data in the right context. For example, on the Performance dashboard, create “avg. time” between two steps executed in the same code. As you see the “avg. time” growing, you can conclude the machine is performing slower. On the Security dashboard, you can visualize “avg. failed logins per user” compared to “users that failed to login more times than the avg. user”.

One killer dashboard for your apps can use our new analytics visualization gadget that will summarize all unknown errors in logs sorted according to severity level. This additional Analytic service can be very useful to DevOps or Testing by presenting new errors in the last hour that were logged during a test cycle or release task.

Deploy Apps Across IT

Once Apps and Dashboards are created you can duplicate, export, and import those Apps. But the most important part is that by using AppTags and Apps sources, it is extremely easy to move apps between pre-production and production and also between different data centers, servers, and locations. Check out the Apps configuration to change the context of all gadgets and dashboards.

This can help you switch security context from one service to another, from one data center to another, or simply duplicate Apps for different end users.

Now Imagine and Build New Apps

XpoLog 6 brings a new level of freedom to data analysis, instead of focusing on log collection, parsing and manual search, you can now focus on advanced analytics, application building, and further developing the business innovation.

Recently Gartner named XpoLog as “Cool Vendor” in the “IT Operations Analytics 2015”. Download this new version to find out why.

Gear Up and Deploy XpoLog on your Log Data. XpoLog 6 is here!

XpoLog Named a “Cool Vendor” in Gartner Report

Gartner Cool Vender logo




Tel Aviv, Israel, and New York City, N.Y. , April 16th 2015XpoLog, a leading provider of Log Management and Analysis solutions for IT, Security and Business, announced it has been included in the list of “Cool Vendors” in Gartner’s April 11, 2015, “Cool Vendors in IT Operations Analytics, 2015” report by Will Capelli and Colin Fletcher . Gartner Inc. is a world leading IT technology research and advisory company. Vendors selected for the Gartner “Cool Vendor” report are innovative, impactful and intriguing.

“We are very happy to be included in the Cool Vendors report by Gartner, and we consider this yet another confirmation that our focus on advanced analytics and search for IT data will help our customers turn silos of unstructured data into meaningful intelligence and actions” said Haim Koschitzky, CEO of XpoLog. “With the upcoming launch of our latest version, XpoLog 6, we believe our product will revolutionize the speed and analysis of big IT data.”

XpoLog combines highly functional log management with a super-fast Analytic Search engine powered by several analytics technologies that analyze logs to establish the meaning and importance of the various log event messages. XpoLog’s unique technology, and specifically its Analytical Search engine, is designed to effectively deal with any log or machine data including home-grown applications.

Unlike others, XpoLog provides an Analytic Search that layers automated intelligence in the context of user searches, accelerating time to insights. Analytic Search proactively scans log data and correlates analytics layers to the result sets of user search. Our customers leverage the technology in IT Operations, DevOps, APM, Software development, Software testing, and Security Log management.

About XpoLog

XpoLog creates software that understands data and unlocks its hidden value, whether in your local storage, or in the cloud. XpoLog helps its customers troubleshoot, search, find, report, and visualize mission critical information on demand, on time.

Their product is ideal for organizations dealing with vast amounts of Log Data on a regular basis, such as Service providers, High Tech companies, Security and governmental institutions, eCommerce, telecom and financial institutes.

Contact XpoLog:


Gartner Disclaimer:

Gartner does not endorse any vendor, product or service depicted in our research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

XpoLog 6: Building NOC, Operations, DevOps, Security Rooms Views

by Haim Koschitzky, XpoLog CEO

XpoLog 6 is coming soon. In this series of posts I am covering the new primary features and enhancements. On my last post I discussed our visualization strategy, we are about to add more than 20 new visualization gadgets and there will be new ways to present information.

This short post will present one of the new cool features of XpoLog 6, the Ops View.

Different Perspective

We understand that different needs require different tools, and sometimes different situations require different views. The teams at the NOC, Operations, DevOps, SOC or High availability control rooms need to have ongoing status screens of managed environment. The need for an ongoing streaming of status views is very different from a single dashboard view or a search console view.

Turn Log Data to a Slideshow

In previous posts I wrote about Apps, AppTags, dashboards, and visualization gadgets. With XpoLog 6 you will be able to select multiple dashboards and run them as slideshows. Free your creativity and build great visual dashboards for security, performance, errors, business statistics, etc.

Rooms with A View

With XpoLog 6 you can build Availability and Business slideshow Views for application and business owners. Build R&D, Development, and Testing dashboards and have them slide in the meeting rooms. Operations, Security, and DevOps will be able to build multiple dashboards and Apps that will provide continuous feedback on systems and applications. Making a wealth of information and insights visibly accessible and dynamic will drive better actions.

image for 4th blog

Night Mode – Log Data can be Romantic

Check out the Themes options in the dashboard view that will invert the colors of the entire visualization views.

Create Smarter NOC and OPS Views

Release your creativity and build Apps and dashboards for any data, and correlate information from databases with logs. Visualize business Apps data from Hadoop with access logs stored on your CDN provider.

XpoLog 6 is coming soon along with some exciting news. Stay Tuned.

XpoLog 6: Log Data Visualization Components and Strategies

by Haim Koschitzky, XpoLog CEO

XpoLog 6 is coming soon. In this series of posts I am covering the new primary features and enhancements. This post will dive into our new visualization gadgets and the ideas guiding us in our long term visualization development road map.

Even though we see many log data analysis deployments, we still identify many challenges users are facing regarding IT log data visualization, analysis, and insights.


Although stating the obvious, before investing expensive efforts and resources into analyzing data, it is crucial to define your expectations and requirements. While in the past, merely collecting all log data and making it available for search was good enough, this is no-longer the case.

In order to ask the right questions, determine what the most important use cases your log data has shown you and what role you want your log data to play in your future ongoing work. To do this, you must monitor system availability, software quality, continuous deployment, application performance, and business insights, troubleshoot, analyze security incidents, compliance audit etc.

There are specific use cases for the application life cycle: architect, developer, tester, DevOps, APM, operations, and production support all have specific uses cases and requirements. Giving the right answer to the right question makes a big impact and will drive smart actions.


Once the requirements and expectations are well defined, add data to XpoLog. When doing so, organize data in Apps logical structures and AppTags, as was discussed in my previous post, XpoLog 6 Virtual Applications Structures, AppTags, and IT Visualization Strategies. Create an App that will contain a collection of dashboards; we recommend creating a dashboard per topic or use case, and providing each one with a meaningful name (“performance”, “errors”, “user audit”). Now follow the steps of creating search queries, or use out of the box gadgets for analytics.

With XpoLog 6 you will find example Apps that you will be able to use as examples of best use cases for log analysis data visualization.

In the new version we added more than 20 new gadgets including 3D graphs, as we witnessed a growing demand for better visualization tools. Once you’ve created search queries to analyze data and generate proper result sets, you will need to select the visualization gadget that best reads these result sets and visualizes it in the most effective way.

Let’s look at a result set that aggregated and computed the avg. memory consumption and total memory usage of two application servers. Take a look at the figure below. On gadget 1 you can see the totals over 24 hr aggregated memory consumption at 1 hr intervals. This gadget tells the story of both servers. Gadgets 2 and 3 represent the same data but for each of the individual servers. Once we split the data for each server we discover that each of the servers had a very different memory consumption pattern.

An hourly aggregation for memory is far from being accurate; memory changes at a much faster rate. On the upper row of gadgets we see the totals for both servers (gadget 4) and two additional gadgets, 5 and 6, representing each server in 1 min intervals.

3rd blog post screen capture


We were looking to monitor our application server memory consumption to avoid spikes that might crash one of our clusters. Choosing the right visualization tools, and in this case, intervals, makes a big difference.


Optimize your dashboards and visualization gadgets by verifying they deliver the insights you’re after in the right resolution. In the example above, analyzing memory for the entire cluster did not provide a clear status image of the memory consumption, but grouping by server and later reducing the time interval resolution to minutes gave a clear understanding of which cluster spiked.


Once your Apps and Dashboards provide clear views and visualization, it will become easy to identify problems, trends, and insights on your IT and applications. Now you will be able to monitor or view the dashboards live. Leverage the visibility and you will now be able to take actions that will make you applications more agile, secure, and optimized for the business.


Again, go to the first step. This is an ongoing process. Data changes every day. The content of logs and other data types is being updated by IT, developers, and vendors every day. In order to stay ahead, keep asking questions and never stop looking for the answers.

We will publish a more comprehensive use case on how to create, optimize, and use the new Apps module. In my next post I will present our new Operations and DevOps screens with more visual examples.

XpoLog 6 is coming soon. Stay tuned.

XpoLog 6 Virtual Applications Structures, AppTags, and IT Visualization Strategies

by Haim Koschitzky, XpoLog CEO

In my previous post, XpoLog 6 Log Management: Listening, Single Page App, AngularJS, and UX/UI, I summarized the key topics we focused on for XpoLog 6. This post will focus on methods we used to manage unstructured IT log data and visualization using smart tagging techniques.

Dynamic Spaghetti

I am sure, unless you are reading this post by accident :-), you have had the opportunity in the past to see Visio style IT diagrams, CMDB dependency flow charts, architecture schemas of complex applications, and/or security/network architecture charts. One thing all these diagrams have in common is that they all look like organized spaghetti. With the current complexity and dynamic nature of virtual infrastructure (private or public cloud), these “spaghetti diagrams” are no longer static in nature, but highly dynamic.

This means that infrastructure diagrams are no longer functioning as solid navigation maps for IT issues. The number of servers and their names, application components and so on are constantly changing according to system constraints. New APM strategies support this with the notion of smart tagging, transaction marking combined with high level application flow awareness based on advanced correlations. The ITOA and Log analysis should embrace those techniques to be able to present insights around common IT structures.

So how do we organize our data in a meaningful way that will not only make sense, but also be practical, usable, visible, and accessible quickly; in addition to being organized to support DevOps and APM insights?

Data Virtualization for IT Visualization

We decided to embrace the same approach we used with logical data structures. While adding additional data sources to XpoLog it is possible to organize these new data sources in a virtual structure of folders and log nodes. Later users can search data and browse automated analytics in the context of these virtual structures. This approach makes it easier to make bulk configuration changes, manage security policies and so on. One very cool option is to define a new virtual log source on another log data source but with a different pattern and different rules. This new log source can be used to filter, hide, or manipulate data for indexing.

In order to organize log data nodes and visualization views in a logical structure, we invented two new virtualization types: AppTags and Virtual Application Structures (Apps).


AppTags are tags that indicate the relevance of each log source to an Application. This means that a single Access log from a single sign-on (SSO) service that is serving 3 different apps, can be tagged with 3 AppTags: AppA, AppB, AppC. Further, in order to better analyze the SSO service, an additional SSO AppTag can be attached.

Why should you care?

Because after AppTags are attached, you can switch the log analysis context in the search, dashboards, visualization, automated log analytics etc. simply by filtering the view according to the Application context relevant to your tasks. This option will unleash use cases for Production/Non-Production tags, DevOps tags, build number tags, and much more.

This powerful concept now allows you to create views, searches, and apps without stating the actual data sources, but rather referring only to the AppTags. Thus dynamically added logs that have the relevant AppTags will automatically be analyzed in multi-dimensional contexts.

This level of abstraction is great for data and log management. The next step is to build actual logical Apps that will provide us with the capability to manage visualization in a robust, scalable, and highly dynamic way.

Virtual Application Structures (Apps)

XpoLog 6 is going to have an Apps console in which users and groups can organize visualization dashboards in the context of an App. For example, build an App (e.g. TomcatApp) for all Apache Tomcat application servers; in it you can create different dashboards – Performance, Availability, Errors and Exceptions, Threads and Memory, Security and User statistics, etc. Each dashboard will contain visualization gadgets presenting charts, graphs, and maps that will aggregate and summarize information in the context of the dashboard.

When building the App we recommend binding the queries to the AppTag Tomcat. This means all Tomcat logs relevant to the queries will be included automatically. A more advanced way is to zoom in on and focus only on Tomcat logs associated with AppA, visualizing data for a specific business application.

Once you get familiar with the concept of building visualization Apps, you can do this for any business application or IT solution you use. A more advanced capability will allow you to duplicate apps and change the context to use the source for different AppTags. This means you will be able to reuse your Apps in different environments and other Apps.

ITOA – IT Data Visualization Strategy

By defining AppTags and Apps, a new abstraction layer will help solve the exhausting configuration and maintenance involved in generating value from large amounts of data. The new structure will help organize insights, data, and visual components in both business and IT context.

In my next post I will expand the concept of visualization strategy and component.

XpoLog 6 is coming soon, so stay tuned…