Developer Games: RegExp and log4j Parsing

Extracting Valuable Data from log4j logs with Virtual Fields

In our recent upgrade to XpoLog V6 we enhanced the features of log4j analysis. In this series of posts I am covering some of the ways you can benefit from XpoLog V6’s new features and enhancements. I will concentrate mainly on how to get the most valuable information from your log4j event logs.

Once your log4j logs have been transferred to and properly defined in the XpoLog Center, you can troubleshoot your java application by running Analytic Search on your log4j data, measure your application performance, create your own AppTags for better monitoring, and create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum log analysis.

Try Xpolog Now

This post will show you how to define log4j logs in XpoLog, to create the most readable data and thus allow for XpoLog to perform highly detailed analysis of your logs. I will also show you an example of how you can virtually extract specific data from your message using Regular Expression to allow for XpoLog to perform a more refined parsing of your data. If you prefer to read our whole manual in one go, you can find it here.

Defining Patterns in XpoLog Center

If you are letting XpoLog access and pull data from your files, define the logger with a name, pattern and data pattern, and then define the log patterns in XpoLog Center.

For example:

#Logger definition

#Appender data for mylog

log4j.appender.mylog.layout.ConversionPattern=[%d] [%t] [%p] [%c] [%l] %m%n

(d = date, t = thread, p = priority, c = class, l = method, m = message, and n = new line)

Defining the log pattern in XpoLog Center:

  1. In XpoLog Center, add a new log. (See my instructions in the previous blog.) Once you have filled in the details, click Next to get to the Log Pattern screen.
  1. In the Wizard of the Pattern Editor, define the log pattern.

Log4J edit log4 wizard toggle for blog 2

Click Manual in the Pattern Editor and edit the XpoLog data pattern to comply with the log4j layout:

a. [%d] = [{date:Date,locale=en,yyyy-MM-dd HH:mm:ss,SSS}]
b. [%t] = [{text:Thread}]
c. [%p]= [{priority:Priority,DEBUG;INFO;WARNING;ERROR;FATAL}]
d. [%c]= [{string:Class}]
e. [%l]= [{string:Method}({text:Source}:{number:LineNumber})]
f. %m = {string:Message}
g. %n = new line

Log4J edit log5 manual toggle for blog 2

The XpoLog pattern in our example will be:

[{date:Date,locale=en,yyyy-MM-dd HH:mm:ss,SSS}] [{text:Thread}] [{priority:Priority,DEBUG;INFO;WARNING;ERROR;FATAL}] [{string:Class}] [{string:Method}({text:Source}:{number:LineNumber})] {string:Message}

  1. Click Save.

You can also edit the pattern after you have added the log, of which I will speak more of in my next post.

Virtually Extract Specific Data from your Message

XpoLog can also extract data from within the message if you use Regular Expression prior to the data transfer.

This is what the message might look like in the Log Pattern section of the Add Log screen without using Regular Expression:

In the Pattern Editor, all you see is {string:Message}.

log4j admin 1 message before regular expression

If you use Regular Expression to extract any word that appears after the word “Manager”, the Log Pattern section of the Add Log screen would look as follows:

In the Pattern Editor, you will now see:

{regexp:HTMLManager state,refName=Message,HTMLManager: (\w+)}{string:Message}

In the Log records analysis result section below XpoLog has added the column HTMLManager state for the data you wished to extract.

log4j admin 2 message after regular expression

In the Manager Interface of XpoLog Center, where you view your logs, you will also see this extra column, HTMLManager state, for the extracted data:

log4j log viewer after regular expression

By extracting the HTMLManager state into a new virtual field we can now measure and monitor the HTMLManager state performance and activity.

Try Xpolog Now

In the next post, I will show how to define and edit the log4j patterns when sending log events and log messages to XpoLog through SysLog. Stay tuned, or go directly to our hands-on-guide.

Log collection and Appender configuration for log4j to XpoLog

From log4j to XpoLog

XpoLog V.6 is here and already taking on an Exabyte-sized storm of logs as I write this. In this series of posts I will cover some of the ways you can use and benefit from its new features and enhancements. I will concentrate mainly on how to get the maximum amount of information from your log4j event logs. If you don’t want to wait for our continuation next week, you can look at our full tutorial right away.

Once your log4j logs have been transferred to and properly defined in XpoLog Center, you can troubleshoot your java application by running Analytic Search on your log4j data, measure your application performance, create your own Apps or use XpoLog’s Apps for better monitoring, create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum analysis.

You can download XpoLog for free if you want to follow as you go along. It only takes a few minutes.

Try Xpolog Now

In this post I will show you how to transfer your log events to XpoLog using log4j. There are two ways of doing this. The first method is to allow XpoLog direct access to your files. The other method is by defining a SysLog appender and sending your events and messages to XpoLog. XpoLog supports both methods.

Allowing XpoLog access to your files (PULL)

Assuming you are already using log4j to write your log events to files, to allow for XpoLog to perform analysis on your log data, you need to give XpoLog access to these files. Define the name, pattern, and data pattern so that XpoLog can read these files, collect and index the data, and start analyzing.

Using direct access (Local or Remote)

XpoLog can access a local log file, i.e. a log file that resides on the same server as XpoLog. XpoLog can also access a log file on a remote server to which it has been provided direct access, as long as XpoLog is provided with the UNC path (\\hostname\dirname) to the log files on the remote server.

Using SSH (Secured Shell)

XpoLog can access log files on remote servers over SSH agent-less, provided that XpoLog has an account with a username and password or private/public key for connecting to the SSH server where the log files are situated.

Note that XpoLog requires Read permissions for any log it reads, regardless of the source of the log file.

To allow for XpoLog to pull (data from) the files, define the logger and give XpoLog access to the remote server where the logger is defined; then add the log to XpoLog.

For example:

#Logger definition

#Appender data for mylog


log4j.appender.mylog.layout.ConversionPattern=[%d] [%t] [%p] [%c] [%l] %m%n

Adding a log to XpoLog:

  1. Inside XpoLog Center, go to Manager > Administration > Add Log. The Add Log screen opens.
  2. Give the log a name and a parent folder, and select an AppTag (Tag to Application(s)) from the drop-down list or create a new AppTag. You can select and create any number of AppTags for the same log. You do not have to tag the log at all, but in my forthcoming posts you will see how useful these AppTags can be. If you cannot wait for my next post, have a look at our “spoiler”.
  3. Select the log type to be Local and give a path (the screen capture below shows the example given).

Add Log mylog for 1st blog post

  1. Click Next to view the sample text from the log, the conversion pattern in the Pattern Editor field, and the log records analysis results; or just click Save. (In my next post I will give details regarding editing the pattern in the Pattern Editor  field.)

Sending your log4j log events to XpoLog (PUSH)

To send log events and log messages to XpoLog through SysLog, define a SysLog appender that uses the XpoLog server as the SysLog host. From inside XpoLog Center, define a TCP or a UDP SysLog Listener account and make sure the port (usually 1468 for TCP or 514 for UDP) is open on XpoLog’s machine. We recommend using TCP.

Defining a TCP SysLog Listener account:

  1. Inside XpoLog Center, go to Manager > Administration > Listeners. A Listeners accounts console opens and presents all the configured listeners available.
  2. Click Syslog TCP. The Syslog TCP Account window opens.

SysLog TCP account window small

  1. Add a descriptive name for the Listener account, click Advanced Settings and continue. Note that for General Information > Enabled you make sure the account is enabled.
  1. Click Save. The data received from the Syslog listener account will be placed under the configured parent folder you selected.

Configuring log4j

Now all you need to do is to make sure the SysLog events from your java application are sent to XpoLog. Configure log4j to use a SysLog appender. Here is an example configuration:

log4j.rootLogger=INFO, SYSLOG
log4j.appender.SYSLOG.layout.conversionPattern=%d{ISO8601} %-5p [%t] %c{2} %x – %m%n

After your logs reach XpoLog

Once your log events have been pushed to or pulled by XpoLog, XpoLog can start collecting, parsing, monitoring, and analyzing all your log data. XpoLog V.6 has enhanced its Analytic Search, added over 20 new visualization gadgets to its Apps, and also gives you the opportunity to create your own Apps and Dashboards, making performance monitoring, analysis, and visualization naturally fast and easy.

In my next post I will cover how properly define and edit your java log patterns; thus paving the way for receiving the highest possible value from the XpoLog analysis. Stay tuned, or go directly to our “spoiler” hands-on-guide.

Try Xpolog Now

Gear Up! XpoLog 6 is here: Imagine and Build Your Log Data Apps

by Haim Koschitzky, XpoLog CEO

XpoLog 6 is finally here. In previous posts I presented certain features of this new version in detail. We worked very hard to rebuild the product to make it a solution for our users to better manage their various complex systems. Now that we have the building blocks to build awesome apps for your log data, let us take a look at how this is done.

Add Logs

XpoLog provides highly functional tools to collect and parse log data. We provide some unique capabilities that we built into our log management platform that help manage logs in a very smart, secure, and efficient way. I also recommend checking out the virtual data engine and log parsing service that helps normalize log data automatically.

In recent years we witnessed growing availability of open source projects for log collection and log shipping, some of them being logstash, fluentd and others. Those tools help ship data and create log repositories.

If you are already using tools for log management we have great news for you, XpoLog can be integrated with most of them. Deploy XpoLog to collect or process log data in order to add leading analytics services on the work you already did.

If you are deploying a new log analysis platform, simply add the data to XpoLog using our SysLog, wizards, or other agents.

Organize Log Data and AppTags

Log data is organized as nodes in XpoLog; these data nodes are very powerful. You can change access permission to nodes, move nodes, duplicate nodes with different credentials, and even apply numerous patterns on the same log using multiple data nodes. When nodes are organized in logical structures, such as folders or Apps, it is easy to perform group operations and search queries on this new abstraction layer.

In the AppTags console, tag nodes to the applications, this will allow you to run queries on AppTags, and Apps. In a dynamic world you can add more data nodes to AppTags and your queries will still work for elastic and hybrid environments.

Search and Complex Search of Logs

If you are new to search, this part can look complicated but we have great tools that build search queries for you. XpoLog 6 Analytic Search was enhanced to automatically build millions of queries automatically according to the content of the log data. You can select queries or build them from the analytics services. Use complex search syntax in order to correlate log data and run complex statistical searches.

Become an expert in your application log data, and use the XpoLog 6 Analytic search query builder to discover more insight on your new or existing data repositories.

post for blog xpolog 6Build Log Data Apps

XpoLog 6 brings a refreshing approach to Apps, we created an amazingly simple (AngularJS based) UI work space in which you create apps. Within each App we recommend defining dashboards like: “Availability of service”, “Performance”, “Security”, “Statistics”, “Top Errors” etc. Within each dashboard, define visual gadgets that visualize data in the right context. For example, on the Performance dashboard, create “avg. time” between two steps executed in the same code. As you see the “avg. time” growing, you can conclude the machine is performing slower. On the Security dashboard, you can visualize “avg. failed logins per user” compared to “users that failed to login more times than the avg. user”.

One killer dashboard for your apps can use our new analytics visualization gadget that will summarize all unknown errors in logs sorted according to severity level. This additional Analytic service can be very useful to DevOps or Testing by presenting new errors in the last hour that were logged during a test cycle or release task.

Deploy Apps Across IT

Once Apps and Dashboards are created you can duplicate, export, and import those Apps. But the most important part is that by using AppTags and Apps sources, it is extremely easy to move apps between pre-production and production and also between different data centers, servers, and locations. Check out the Apps configuration to change the context of all gadgets and dashboards.

This can help you switch security context from one service to another, from one data center to another, or simply duplicate Apps for different end users.

Now Imagine and Build New Apps

XpoLog 6 brings a new level of freedom to data analysis, instead of focusing on log collection, parsing and manual search, you can now focus on advanced analytics, application building, and further developing the business innovation.

Recently Gartner named XpoLog as “Cool Vendor” in the “IT Operations Analytics 2015”. Download this new version to find out why.

Gear Up and Deploy XpoLog on your Log Data. XpoLog 6 is here!

XpoLog Named a “Cool Vendor” in Gartner Report

Gartner Cool Vender logo




Tel Aviv, Israel, and New York City, N.Y. , April 16th 2015XpoLog, a leading provider of Log Management and Analysis solutions for IT, Security and Business, announced it has been included in the list of “Cool Vendors” in Gartner’s April 11, 2015, “Cool Vendors in IT Operations Analytics, 2015” report by Will Capelli and Colin Fletcher . Gartner Inc. is a world leading IT technology research and advisory company. Vendors selected for the Gartner “Cool Vendor” report are innovative, impactful and intriguing.

“We are very happy to be included in the Cool Vendors report by Gartner, and we consider this yet another confirmation that our focus on advanced analytics and search for IT data will help our customers turn silos of unstructured data into meaningful intelligence and actions” said Haim Koschitzky, CEO of XpoLog. “With the upcoming launch of our latest version, XpoLog 6, we believe our product will revolutionize the speed and analysis of big IT data.”

XpoLog combines highly functional log management with a super-fast Analytic Search engine powered by several analytics technologies that analyze logs to establish the meaning and importance of the various log event messages. XpoLog’s unique technology, and specifically its Analytical Search engine, is designed to effectively deal with any log or machine data including home-grown applications.

Unlike others, XpoLog provides an Analytic Search that layers automated intelligence in the context of user searches, accelerating time to insights. Analytic Search proactively scans log data and correlates analytics layers to the result sets of user search. Our customers leverage the technology in IT Operations, DevOps, APM, Software development, Software testing, and Security Log management.

About XpoLog

XpoLog creates software that understands data and unlocks its hidden value, whether in your local storage, or in the cloud. XpoLog helps its customers troubleshoot, search, find, report, and visualize mission critical information on demand, on time.

Their product is ideal for organizations dealing with vast amounts of Log Data on a regular basis, such as Service providers, High Tech companies, Security and governmental institutions, eCommerce, telecom and financial institutes.

Contact XpoLog:


Gartner Disclaimer:

Gartner does not endorse any vendor, product or service depicted in our research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

XpoLog 6: Building NOC, Operations, DevOps, Security Rooms Views

by Haim Koschitzky, XpoLog CEO

XpoLog 6 is coming soon. In this series of posts I am covering the new primary features and enhancements. On my last post I discussed our visualization strategy, we are about to add more than 20 new visualization gadgets and there will be new ways to present information.

This short post will present one of the new cool features of XpoLog 6, the Ops View.

Different Perspective

We understand that different needs require different tools, and sometimes different situations require different views. The teams at the NOC, Operations, DevOps, SOC or High availability control rooms need to have ongoing status screens of managed environment. The need for an ongoing streaming of status views is very different from a single dashboard view or a search console view.

Turn Log Data to a Slideshow

In previous posts I wrote about Apps, AppTags, dashboards, and visualization gadgets. With XpoLog 6 you will be able to select multiple dashboards and run them as slideshows. Free your creativity and build great visual dashboards for security, performance, errors, business statistics, etc.

Rooms with A View

With XpoLog 6 you can build Availability and Business slideshow Views for application and business owners. Build R&D, Development, and Testing dashboards and have them slide in the meeting rooms. Operations, Security, and DevOps will be able to build multiple dashboards and Apps that will provide continuous feedback on systems and applications. Making a wealth of information and insights visibly accessible and dynamic will drive better actions.

image for 4th blog

Night Mode – Log Data can be Romantic

Check out the Themes options in the dashboard view that will invert the colors of the entire visualization views.

Create Smarter NOC and OPS Views

Release your creativity and build Apps and dashboards for any data, and correlate information from databases with logs. Visualize business Apps data from Hadoop with access logs stored on your CDN provider.

XpoLog 6 is coming soon along with some exciting news. Stay Tuned.

XpoLog 6: Log Data Visualization Components and Strategies

by Haim Koschitzky, XpoLog CEO

XpoLog 6 is coming soon. In this series of posts I am covering the new primary features and enhancements. This post will dive into our new visualization gadgets and the ideas guiding us in our long term visualization development road map.

Even though we see many log data analysis deployments, we still identify many challenges users are facing regarding IT log data visualization, analysis, and insights.


Although stating the obvious, before investing expensive efforts and resources into analyzing data, it is crucial to define your expectations and requirements. While in the past, merely collecting all log data and making it available for search was good enough, this is no-longer the case.

In order to ask the right questions, determine what the most important use cases your log data has shown you and what role you want your log data to play in your future ongoing work. To do this, you must monitor system availability, software quality, continuous deployment, application performance, and business insights, troubleshoot, analyze security incidents, compliance audit etc.

There are specific use cases for the application life cycle: architect, developer, tester, DevOps, APM, operations, and production support all have specific uses cases and requirements. Giving the right answer to the right question makes a big impact and will drive smart actions.


Once the requirements and expectations are well defined, add data to XpoLog. When doing so, organize data in Apps logical structures and AppTags, as was discussed in my previous post, XpoLog 6 Virtual Applications Structures, AppTags, and IT Visualization Strategies. Create an App that will contain a collection of dashboards; we recommend creating a dashboard per topic or use case, and providing each one with a meaningful name (“performance”, “errors”, “user audit”). Now follow the steps of creating search queries, or use out of the box gadgets for analytics.

With XpoLog 6 you will find example Apps that you will be able to use as examples of best use cases for log analysis data visualization.

In the new version we added more than 20 new gadgets including 3D graphs, as we witnessed a growing demand for better visualization tools. Once you’ve created search queries to analyze data and generate proper result sets, you will need to select the visualization gadget that best reads these result sets and visualizes it in the most effective way.

Let’s look at a result set that aggregated and computed the avg. memory consumption and total memory usage of two application servers. Take a look at the figure below. On gadget 1 you can see the totals over 24 hr aggregated memory consumption at 1 hr intervals. This gadget tells the story of both servers. Gadgets 2 and 3 represent the same data but for each of the individual servers. Once we split the data for each server we discover that each of the servers had a very different memory consumption pattern.

An hourly aggregation for memory is far from being accurate; memory changes at a much faster rate. On the upper row of gadgets we see the totals for both servers (gadget 4) and two additional gadgets, 5 and 6, representing each server in 1 min intervals.

3rd blog post screen capture


We were looking to monitor our application server memory consumption to avoid spikes that might crash one of our clusters. Choosing the right visualization tools, and in this case, intervals, makes a big difference.


Optimize your dashboards and visualization gadgets by verifying they deliver the insights you’re after in the right resolution. In the example above, analyzing memory for the entire cluster did not provide a clear status image of the memory consumption, but grouping by server and later reducing the time interval resolution to minutes gave a clear understanding of which cluster spiked.


Once your Apps and Dashboards provide clear views and visualization, it will become easy to identify problems, trends, and insights on your IT and applications. Now you will be able to monitor or view the dashboards live. Leverage the visibility and you will now be able to take actions that will make you applications more agile, secure, and optimized for the business.


Again, go to the first step. This is an ongoing process. Data changes every day. The content of logs and other data types is being updated by IT, developers, and vendors every day. In order to stay ahead, keep asking questions and never stop looking for the answers.

We will publish a more comprehensive use case on how to create, optimize, and use the new Apps module. In my next post I will present our new Operations and DevOps screens with more visual examples.

XpoLog 6 is coming soon. Stay tuned.

XpoLog 6 Virtual Applications Structures, AppTags, and IT Visualization Strategies

by Haim Koschitzky, XpoLog CEO

In my previous post, XpoLog 6 Log Management: Listening, Single Page App, AngularJS, and UX/UI, I summarized the key topics we focused on for XpoLog 6. This post will focus on methods we used to manage unstructured IT log data and visualization using smart tagging techniques.

Dynamic Spaghetti

I am sure, unless you are reading this post by accident :-), you have had the opportunity in the past to see Visio style IT diagrams, CMDB dependency flow charts, architecture schemas of complex applications, and/or security/network architecture charts. One thing all these diagrams have in common is that they all look like organized spaghetti. With the current complexity and dynamic nature of virtual infrastructure (private or public cloud), these “spaghetti diagrams” are no longer static in nature, but highly dynamic.

This means that infrastructure diagrams are no longer functioning as solid navigation maps for IT issues. The number of servers and their names, application components and so on are constantly changing according to system constraints. New APM strategies support this with the notion of smart tagging, transaction marking combined with high level application flow awareness based on advanced correlations. The ITOA and Log analysis should embrace those techniques to be able to present insights around common IT structures.

So how do we organize our data in a meaningful way that will not only make sense, but also be practical, usable, visible, and accessible quickly; in addition to being organized to support DevOps and APM insights?

Data Virtualization for IT Visualization

We decided to embrace the same approach we used with logical data structures. While adding additional data sources to XpoLog it is possible to organize these new data sources in a virtual structure of folders and log nodes. Later users can search data and browse automated analytics in the context of these virtual structures. This approach makes it easier to make bulk configuration changes, manage security policies and so on. One very cool option is to define a new virtual log source on another log data source but with a different pattern and different rules. This new log source can be used to filter, hide, or manipulate data for indexing.

In order to organize log data nodes and visualization views in a logical structure, we invented two new virtualization types: AppTags and Virtual Application Structures (Apps).


AppTags are tags that indicate the relevance of each log source to an Application. This means that a single Access log from a single sign-on (SSO) service that is serving 3 different apps, can be tagged with 3 AppTags: AppA, AppB, AppC. Further, in order to better analyze the SSO service, an additional SSO AppTag can be attached.

Why should you care?

Because after AppTags are attached, you can switch the log analysis context in the search, dashboards, visualization, automated log analytics etc. simply by filtering the view according to the Application context relevant to your tasks. This option will unleash use cases for Production/Non-Production tags, DevOps tags, build number tags, and much more.

This powerful concept now allows you to create views, searches, and apps without stating the actual data sources, but rather referring only to the AppTags. Thus dynamically added logs that have the relevant AppTags will automatically be analyzed in multi-dimensional contexts.

This level of abstraction is great for data and log management. The next step is to build actual logical Apps that will provide us with the capability to manage visualization in a robust, scalable, and highly dynamic way.

Virtual Application Structures (Apps)

XpoLog 6 is going to have an Apps console in which users and groups can organize visualization dashboards in the context of an App. For example, build an App (e.g. TomcatApp) for all Apache Tomcat application servers; in it you can create different dashboards – Performance, Availability, Errors and Exceptions, Threads and Memory, Security and User statistics, etc. Each dashboard will contain visualization gadgets presenting charts, graphs, and maps that will aggregate and summarize information in the context of the dashboard.

When building the App we recommend binding the queries to the AppTag Tomcat. This means all Tomcat logs relevant to the queries will be included automatically. A more advanced way is to zoom in on and focus only on Tomcat logs associated with AppA, visualizing data for a specific business application.

Once you get familiar with the concept of building visualization Apps, you can do this for any business application or IT solution you use. A more advanced capability will allow you to duplicate apps and change the context to use the source for different AppTags. This means you will be able to reuse your Apps in different environments and other Apps.

ITOA – IT Data Visualization Strategy

By defining AppTags and Apps, a new abstraction layer will help solve the exhausting configuration and maintenance involved in generating value from large amounts of data. The new structure will help organize insights, data, and visual components in both business and IT context.

In my next post I will expand the concept of visualization strategy and component.

XpoLog 6 is coming soon, so stay tuned…

XpoLog 6 Log Management: Listening, Single Page App, AngularJS, and UX/UI

by Haim Koschitzky, XpoLog CEO

The latest version of XpoLog Log Management will contain a huge face lift with regards to the UX/UI.


In the last couple of years we have received a lot of feedback from everyone who has used and/or evaluated XpoLog. For those of you who are not familiar with our numbers, we provide a Log analysis platform with approximately ten thousand installations worldwide, some of them are based on our free version, others on our OEM customers, data centers etc.

Of all the great feedback we received, there were two prominent points that were constantly repeated, the first one was something along the lines of “great technology”, “very smart analytics”, “rich functionality” etc. The second one was that the UI doesn’t look good, and the UX is not “cool enough”. I’ll tell you, it’s easier to listen to the first one and ignore the second. Human nature tends to prefer compliments, rather than criticism. But it is important to remember that constructive criticism can actually be helpful to us in the long run.

Last year I was speaking to analysts Colin Fletcher and Jonah Kowall from a leading research and analyst firm, Jonah later joined AppDynamics. At the end of that briefing, they summarized 3 primary points of feedback:

  • great technology and road map
  • bad UI/UX
  • more marketing needed

I left the meeting having a clear and focused plan for our next version; the message got through, I hope.

Single Page Apps and AngularJS

We did have plans to work on the UI/UX but we did not plan on working on it immediately, and only after processing all the feedback did we decide that doing only cosmetic changes was not going to cut it.

We wanted XpoLog to take a different direction when it came to UI/UX experience. When building a long term roadmap and strategy, if you lay the right foundations, something great can be built on top of it.

We selected AngularJS and Single-page app as we believed these are the front end technologies that will help us build a great user experience. There are a couple of technologies out there that help you build great UI and UX experience, in our case, fast and responsive user experience help us build smarter log data analytic tools.

Our creative team designed our new UI, only to redesign it again and again. We had users trying mockups, we had long arguments about buttons and flows, we built and rebuilt, we changed and we tested. We added more than 20 new data visualization types including 3D types. We are planning on adding many more.

We designed and implemented new ways of creating logical applications and visualization structures. We enabled data manipulation from visual components. We created operations and DevOps room screens with unique functions. We changed the data visualization gadgets and application structures and dashboard design. We built new UI/UX concepts for data analytics. We are proud of it all and we hope you guys will like it too.

It’s time to say thanks to our users, customers, friends, partners, critics and more critics ;). You guys really push us forward in our journey of building great products. I am sure our work just started but now it’s time to try the first release. Later this year we will redesign and reinvent the search console and the analytics console.

My next post will dive deep into logical application management and structure, smart tagging, and how to build a visualization strategy across IT.

XpoLog 6 will be released soon; stay tuned.

Hands On: XpoLog Augmented Search™ 5.0. with Hadoop and HDFS Integration

The new version of XpoLog’s Augmented Search Log analysis platform features integrates with Hadoop and HDFS. In this blog post we will look at how to actually perform this integration.

Adding intelligence to your Hadoop/HDFS logs

XpoLog Augmented Search 5.0 collects log data from Hadoop Distributed File System (HDFS), and layers new intelligent analytics on top of Hadoop. Those insights feed into augmented searches. XpoLog adds intelligence to log file search context with semantic analysis, and pattern and anomaly detection (to uncover insights and trends into application problems, systems, and user behavior). This helps users analyze problems within the Hadoop infrastructure and applications that run on the platform. It offers visibility into the distributed architecture, automatically triaging issues and errors for severity, and presenting results in a dashboard interface.

By combining XpoLog with Hadoop and HDFS you will be able to:

  • Quickly gain automated insights into Hadoop deployment logs and events
  • Troubleshoot and investigate Hadoop application logs
  • Access, collect and analyze any data that is stored on the HDFS

With XpoLog you can quickly detect node connection problems, any type of exception, data store and data block errors and much more.


Connecting XpoLog to your existing Hadoop/HDFS deployment

  1. Log into XpoLog and go to Manager>Administration>Cloud.
  2. Add a new Hadoop account and fill in the Name, Description, Host and Port fields.
  3. Go to Administration>Add Log, or Add entire logs directory path to XpoLog.
  4. Add the Hadoop logs and repeat this task as needed to get all your logs into XpoLog.
  5. Make sure that both the logs of the Hadoop apps and Hadoop infrastructure were added

That’s it!

Using XpoLog to investigate Hadoop/HDFS events

Go the Search console and run searches on both Hadoop logs and Hadoop HDFS stored logs

If the logs were tagged to specific apps you can simple search “* in app.hadoop” for example.


Check out the Analytics console for automated insights layer of problems and errors


Try the Augmented Search options to combine the power of automated intelligence in the context of the search in order to discover new intelligence layers and errors in the context of your search.

You can now go to the dashboards and add visualization gadgets/dashboards based on your search queries and the log analytics insights layers.

Get started with the full featured XpoLog Platform free license – click here


Analytics and machine learning add intelligence to Hadoop application and server log analysis


NEW YORKMay 21, 2014 /PRNewswire/ — XpoLog Ltd, the company that invented Augmented Search for IT log analysis, has introduced XpoLog Augmented Search™ 5.0. The new version brings XpoLog’s troubleshooting capabilities to the Hadoop platform to help DevOps teams rapidly diagnose and solve Hadoop deployment and application problems.

Testing applications on Hadoop, a large-scale, distributed data processing platform, isn’t a trivial task, and it’s made even more difficult without tools that accelerate DevOps activities. XpoLog Augmented Search 5.0 helps in two ways: it collects log data from Hadoop Distributed File System (HDFS), and by layering new intelligent analytics on top of Hadoop. Those insights feed into augmented searches.

XpoLog adds intelligence to log file search context with semantic analysis, and pattern and anomaly detection (to uncover insights and trends into application problems, systems, and user behavior). This helps users analyze problems within the Hadoop infrastructure and applications that run on the platform. It offers visibility into the distributed architecture, automatically triaging issues and errors for severity, and presenting results in a dashboard interface.

“With our latest version, organizations large and small can use augmented search to gain vital insights into Hadoop deployments,” said XpoLog VP Solutions Omry Koschitzky. “Our solution is particularly useful for Hadoop distributions across a large number of servers and data centers.”

Customers using XpoLog’s Augmented Search for DevOps have reported drastically reduced time-to-resolution throughout key verticals. There is also no barrier to entry, because the XpoLog platform is free for processing up to 1 gigabyte of log data per day. XpoLog is also the only automated log data analytics solution that supports thelogstash data infrastructure, which is also supported by the 5.0 release.

The full product brochure, an online demo, additional pricing information, and a free trial of the Log Analysis Platform, are all available through XpoLog’s Web site.

About XpoLog Ltd
XpoLog Ltd is an IT operations analytics software company based in the Israel and the U.S. that invented augmented search, a breakthrough which unlocks the hidden value of log data. The platform drastically reduces time to resolution and provides a wealth of intelligence, trends, and insights into enterprise IT environments. XpoLog is a trusted source for DevOps analytics tools with over 10 years of experience in the IT industry. Trial versions of the XpoLog Platform can be downloaded as a standalone server or J2EE application through the company’s official website. []