About XpoLog

XpoLog makes a Log Analysis Platform for applications, servers and cloud applications. XpoLog Center provide log management, log viewer, log analysis, reports, problem analysis, log collection, correlation and many other features that help application groups, operations and administrators to quickly investigate and monitor applications. XpoLog help to optimize business applications.

XpoLog 6: Log Data Visualization Components and Strategies

by Haim Koschitzky, XpoLog CEO

XpoLog 6 is coming soon. In this series of posts I am covering the new primary features and enhancements. This post will dive into our new visualization gadgets and the ideas guiding us in our long term visualization development road map.

Even though we see many log data analysis deployments, we still identify many challenges users are facing regarding IT log data visualization, analysis, and insights.


Although stating the obvious, before investing expensive efforts and resources into analyzing data, it is crucial to define your expectations and requirements. While in the past, merely collecting all log data and making it available for search was good enough, this is no-longer the case.

In order to ask the right questions, determine what the most important use cases your log data has shown you and what role you want your log data to play in your future ongoing work. To do this, you must monitor system availability, software quality, continuous deployment, application performance, and business insights, troubleshoot, analyze security incidents, compliance audit etc.

There are specific use cases for the application life cycle: architect, developer, tester, DevOps, APM, operations, and production support all have specific uses cases and requirements. Giving the right answer to the right question makes a big impact and will drive smart actions.


Once the requirements and expectations are well defined, add data to XpoLog. When doing so, organize data in Apps logical structures and AppTags, as was discussed in my previous post, XpoLog 6 Virtual Applications Structures, AppTags, and IT Visualization Strategies. Create an App that will contain a collection of dashboards; we recommend creating a dashboard per topic or use case, and providing each one with a meaningful name (“performance”, “errors”, “user audit”). Now follow the steps of creating search queries, or use out of the box gadgets for analytics.

With XpoLog 6 you will find example Apps that you will be able to use as examples of best use cases for log analysis data visualization.


In the new version we added more than 20 new gadgets including 3D graphs, as we witnessed a growing demand for better visualization tools. Once you’ve created search queries to analyze data and generate proper result sets, you will need to select the visualization gadget that best reads these result sets and visualizes it in the most effective way.

Let’s look at a result set that aggregated and computed the avg. memory consumption and total memory usage of two application servers. Take a look at the figure below. On gadget 1 you can see the totals over 24 hr aggregated memory consumption at 1 hr intervals. This gadget tells the story of both servers. Gadgets 2 and 3 represent the same data but for each of the individual servers. Once we split the data for each server we discover that each of the servers had a very different memory consumption pattern.

An hourly aggregation for memory is far from being accurate; memory changes at a much faster rate. On the upper row of gadgets we see the totals for both servers (gadget 4) and two additional gadgets, 5 and 6, representing each server in 1 min intervals.

3rd blog post screen capture


We were looking to monitor our application server memory consumption to avoid spikes that might crash one of our clusters. Choosing the right visualization tools, and in this case, intervals, makes a big difference.


Optimize your dashboards and visualization gadgets by verifying they deliver the insights you’re after in the right resolution. In the example above, analyzing memory for the entire cluster did not provide a clear status image of the memory consumption, but grouping by server and later reducing the time interval resolution to minutes gave a clear understanding of which cluster spiked.


Once your Apps and Dashboards provide clear views and visualization, it will become easy to identify problems, trends, and insights on your IT and applications. Now you will be able to monitor or view the dashboards live. Leverage the visibility and you will now be able to take actions that will make you applications more agile, secure, and optimized for the business.


Again, go to the first step. This is an ongoing process. Data changes every day. The content of logs and other data types is being updated by IT, developers, and vendors every day. In order to stay ahead, keep asking questions and never stop looking for the answers.

We will publish a more comprehensive use case on how to create, optimize, and use the new Apps module. In my next post I will present our new Operations and DevOps screens with more visual examples.

XpoLog 6 is coming soon. Stay tuned.

XpoLog 6 Virtual Applications Structures, AppTags, and IT Visualization Strategies

by Haim Koschitzky, XpoLog CEO

In my previous post, XpoLog 6 Log Management: Listening, Single Page App, AngularJS, and UX/UI, I summarized the key topics we focused on for XpoLog 6. This post will focus on methods we used to manage unstructured IT log data and visualization using smart tagging techniques.

Dynamic Spaghetti

I am sure, unless you are reading this post by accident :-), you have had the opportunity in the past to see Visio style IT diagrams, CMDB dependency flow charts, architecture schemas of complex applications, and/or security/network architecture charts. One thing all these diagrams have in common is that they all look like organized spaghetti. With the current complexity and dynamic nature of virtual infrastructure (private or public cloud), these “spaghetti diagrams” are no longer static in nature, but highly dynamic.

This means that infrastructure diagrams are no longer functioning as solid navigation maps for IT issues. The number of servers and their names, application components and so on are constantly changing according to system constraints. New APM strategies support this with the notion of smart tagging, transaction marking combined with high level application flow awareness based on advanced correlations. The ITOA and Log analysis should embrace those techniques to be able to present insights around common IT structures.

So how do we organize our data in a meaningful way that will not only make sense, but also be practical, usable, visible, and accessible quickly; in addition to being organized to support DevOps and APM insights?

Data Virtualization for IT Visualization

We decided to embrace the same approach we used with logical data structures. While adding additional data sources to XpoLog it is possible to organize these new data sources in a virtual structure of folders and log nodes. Later users can search data and browse automated analytics in the context of these virtual structures. This approach makes it easier to make bulk configuration changes, manage security policies and so on. One very cool option is to define a new virtual log source on another log data source but with a different pattern and different rules. This new log source can be used to filter, hide, or manipulate data for indexing.

In order to organize log data nodes and visualization views in a logical structure, we invented two new virtualization types: AppTags and Virtual Application Structures (Apps).


AppTags are tags that indicate the relevance of each log source to an Application. This means that a single Access log from a single sign-on (SSO) service that is serving 3 different apps, can be tagged with 3 AppTags: AppA, AppB, AppC. Further, in order to better analyze the SSO service, an additional SSO AppTag can be attached.

Why should you care?

Because after AppTags are attached, you can switch the log analysis context in the search, dashboards, visualization, automated log analytics etc. simply by filtering the view according to the Application context relevant to your tasks. This option will unleash use cases for Production/Non-Production tags, DevOps tags, build number tags, and much more.

This powerful concept now allows you to create views, searches, and apps without stating the actual data sources, but rather referring only to the AppTags. Thus dynamically added logs that have the relevant AppTags will automatically be analyzed in multi-dimensional contexts.

This level of abstraction is great for data and log management. The next step is to build actual logical Apps that will provide us with the capability to manage visualization in a robust, scalable, and highly dynamic way.

Virtual Application Structures (Apps)

XpoLog 6 is going to have an Apps console in which users and groups can organize visualization dashboards in the context of an App. For example, build an App (e.g. TomcatApp) for all Apache Tomcat application servers; in it you can create different dashboards – Performance, Availability, Errors and Exceptions, Threads and Memory, Security and User statistics, etc. Each dashboard will contain visualization gadgets presenting charts, graphs, and maps that will aggregate and summarize information in the context of the dashboard.

When building the App we recommend binding the queries to the AppTag Tomcat. This means all Tomcat logs relevant to the queries will be included automatically. A more advanced way is to zoom in on and focus only on Tomcat logs associated with AppA, visualizing data for a specific business application.

Once you get familiar with the concept of building visualization Apps, you can do this for any business application or IT solution you use. A more advanced capability will allow you to duplicate apps and change the context to use the source for different AppTags. This means you will be able to reuse your Apps in different environments and other Apps.

ITOA – IT Data Visualization Strategy

By defining AppTags and Apps, a new abstraction layer will help solve the exhausting configuration and maintenance involved in generating value from large amounts of data. The new structure will help organize insights, data, and visual components in both business and IT context.

In my next post I will expand the concept of visualization strategy and component.

XpoLog 6 is coming soon, so stay tuned…

XpoLog 6 Log Management: Listening, Single Page App, AngularJS, and UX/UI

by Haim Koschitzky, XpoLog CEO

The latest version of XpoLog Log Management will contain a huge face lift with regards to the UX/UI.


In the last couple of years we have received a lot of feedback from everyone who has used and/or evaluated XpoLog. For those of you who are not familiar with our numbers, we provide a Log analysis platform with approximately ten thousand installations worldwide, some of them are based on our free version, others on our OEM customers, data centers etc.

Of all the great feedback we received, there were two prominent points that were constantly repeated, the first one was something along the lines of “great technology”, “very smart analytics”, “rich functionality” etc. The second one was that the UI doesn’t look good, and the UX is not “cool enough”. I’ll tell you, it’s easier to listen to the first one and ignore the second. Human nature tends to prefer compliments, rather than criticism. But it is important to remember that constructive criticism can actually be helpful to us in the long run.

Last year I was speaking to analysts Colin Fletcher and Jonah Kowall from a leading research and analyst firm, Jonah later joined AppDynamics. At the end of that briefing, they summarized 3 primary points of feedback:

  • great technology and road map
  • bad UI/UX
  • more marketing needed

I left the meeting having a clear and focused plan for our next version; the message got through, I hope.

Single Page Apps and AngularJS

We did have plans to work on the UI/UX but we did not plan on working on it immediately, and only after processing all the feedback did we decide that doing only cosmetic changes was not going to cut it.

We wanted XpoLog to take a different direction when it came to UI/UX experience. When building a long term roadmap and strategy, if you lay the right foundations, something great can be built on top of it.

We selected AngularJS and Single-page app as we believed these are the front end technologies that will help us build a great user experience. There are a couple of technologies out there that help you build great UI and UX experience, in our case, fast and responsive user experience help us build smarter log data analytic tools.

Our creative team designed our new UI, only to redesign it again and again. We had users trying mockups, we had long arguments about buttons and flows, we built and rebuilt, we changed and we tested. We added more than 20 new data visualization types including 3D types. We are planning on adding many more.

We designed and implemented new ways of creating logical applications and visualization structures. We enabled data manipulation from visual components. We created operations and DevOps room screens with unique functions. We changed the data visualization gadgets and application structures and dashboard design. We built new UI/UX concepts for data analytics. We are proud of it all and we hope you guys will like it too.

It’s time to say thanks to our users, customers, friends, partners, critics and more critics ;). You guys really push us forward in our journey of building great products. I am sure our work just started but now it’s time to try the first release. Later this year we will redesign and reinvent the search console and the analytics console.

My next post will dive deep into logical application management and structure, smart tagging, and how to build a visualization strategy across IT.

XpoLog 6 will be released soon; stay tuned.

Hands On: XpoLog Augmented Search™ 5.0. with Hadoop and HDFS Integration

The new version of XpoLog’s Augmented Search Log analysis platform features integrates with Hadoop and HDFS. In this blog post we will look at how to actually perform this integration.

Adding intelligence to your Hadoop/HDFS logs

XpoLog Augmented Search 5.0 collects log data from Hadoop Distributed File System (HDFS), and layers new intelligent analytics on top of Hadoop. Those insights feed into augmented searches. XpoLog adds intelligence to log file search context with semantic analysis, and pattern and anomaly detection (to uncover insights and trends into application problems, systems, and user behavior). This helps users analyze problems within the Hadoop infrastructure and applications that run on the platform. It offers visibility into the distributed architecture, automatically triaging issues and errors for severity, and presenting results in a dashboard interface.

By combining XpoLog with Hadoop and HDFS you will be able to:

  • Quickly gain automated insights into Hadoop deployment logs and events
  • Troubleshoot and investigate Hadoop application logs
  • Access, collect and analyze any data that is stored on the HDFS

With XpoLog you can quickly detect node connection problems, any type of exception, data store and data block errors and much more.


Connecting XpoLog to your existing Hadoop/HDFS deployment

  1. Log into XpoLog and go to Manager>Administration>Cloud.
  2. Add a new Hadoop account and fill in the Name, Description, Host and Port fields.
  3. Go to Administration>Add Log, or Add entire logs directory path to XpoLog.
  4. Add the Hadoop logs and repeat this task as needed to get all your logs into XpoLog.
  5. Make sure that both the logs of the Hadoop apps and Hadoop infrastructure were added

That’s it!

Using XpoLog to investigate Hadoop/HDFS events

Go the Search console and run searches on both Hadoop logs and Hadoop HDFS stored logs

If the logs were tagged to specific apps you can simple search “* in app.hadoop” for example.


Check out the Analytics console for automated insights layer of problems and errors


Try the Augmented Search options to combine the power of automated intelligence in the context of the search in order to discover new intelligence layers and errors in the context of your search.

You can now go to the dashboards and add visualization gadgets/dashboards based on your search queries and the log analytics insights layers.

Get started with the full featured XpoLog Platform free license – click here


Analytics and machine learning add intelligence to Hadoop application and server log analysis


NEW YORKMay 21, 2014 /PRNewswire/ — XpoLog Ltd, the company that invented Augmented Search for IT log analysis, has introduced XpoLog Augmented Search™ 5.0. The new version brings XpoLog’s troubleshooting capabilities to the Hadoop platform to help DevOps teams rapidly diagnose and solve Hadoop deployment and application problems.

Testing applications on Hadoop, a large-scale, distributed data processing platform, isn’t a trivial task, and it’s made even more difficult without tools that accelerate DevOps activities. XpoLog Augmented Search 5.0 helps in two ways: it collects log data from Hadoop Distributed File System (HDFS), and by layering new intelligent analytics on top of Hadoop. Those insights feed into augmented searches.

XpoLog adds intelligence to log file search context with semantic analysis, and pattern and anomaly detection (to uncover insights and trends into application problems, systems, and user behavior). This helps users analyze problems within the Hadoop infrastructure and applications that run on the platform. It offers visibility into the distributed architecture, automatically triaging issues and errors for severity, and presenting results in a dashboard interface.

“With our latest version, organizations large and small can use augmented search to gain vital insights into Hadoop deployments,” said XpoLog VP Solutions Omry Koschitzky. “Our solution is particularly useful for Hadoop distributions across a large number of servers and data centers.”

Customers using XpoLog’s Augmented Search for DevOps have reported drastically reduced time-to-resolution throughout key verticals. There is also no barrier to entry, because the XpoLog platform is free for processing up to 1 gigabyte of log data per day. XpoLog is also the only automated log data analytics solution that supports thelogstash data infrastructure, which is also supported by the 5.0 release.

The full product brochure, an online demo, additional pricing information, and a free trial of the Log Analysis Platform, are all available through XpoLog’s Web site.

About XpoLog Ltd
XpoLog Ltd is an IT operations analytics software company based in the Israel and the U.S. that invented augmented search, a breakthrough which unlocks the hidden value of log data. The platform drastically reduces time to resolution and provides a wealth of intelligence, trends, and insights into enterprise IT environments. XpoLog is a trusted source for DevOps analytics tools with over 10 years of experience in the IT industry. Trial versions of the XpoLog Platform can be downloaded as a standalone server or J2EE application through the company’s official website. [http://www.xpolog.com]


XpoLog enhances logstash with Augmented Search

IT departments that have adopted the open source solution can add on the world’s most advanced log analysis platformImage

NEW YORK CITY – May. 10, 2014 – XpoLog Ltd, the company that invented Augmented Search for IT log analysis, is integrating its industry leading technologies with, logstash, a popular open source log-processing tool. Organizations benefit by leveraging their existing infrastructure with XpoLog’s robust log management features, automated triaging of IT problems, and advanced reporting and visualization – with pricing that ranges from free to affordable.


Elasticsearch.org’s logstash project centralizes logging with user-defined rules; however, it lacks enterprise level features. While it may be used in combination with several separate open source projects, only XpoLog offers a production-proven, holistic solution to maximize its full potential.


XpoLog enables logstash users to create Web access log statistics, application log error summaries, performance charts, and other reports, quickly and efficiently. Statistical queries instantly visualize log data, which can be used to build charts, dashboards, geomaps, and much more.


XpoLog is also the only automated log data analytics solution that supports the logstash data infrastructure. Users can avoid time-consuming manual searches. Customers using XpoLog’s Augmented Search for DevOps have reported drastically reduced time-to-resolution across major vertical industries.


“We believe that it’s important to support the data infrastructures that our customers want to use,” said XpoLog CEO Haim Koschitzky. “It’s also vital for organizations that have already invested in a data engine to have access to the most advance data analytics technologies available on the market.”


XpoLog’s solution uniquely layers intelligence on top of search context using semantic analysis, as well as pattern and anomaly detection to uncover insights and trends into application problems, systems, and user behavior. Those capabilities allow it to capture a wealth of intelligence from within IT environments, including the analysis of proprietary log data. There is also no barrier to entry, because the XpoLog platform is free for processing up to 1 gigabyte of log data per day.


The full product brochure, an online demo, additional pricing information, and a free trial of the Log Analysis Platform, are all available through XpoLog’s Web site.



About XpoLog Ltd
XpoLog Ltd is an IT operations analytics software company based in the Israel and the U.S. that invented augmented search, a breakthrough which unlocks the hidden value of log data. The platform drastically reduces time to resolution and provides a wealth of intelligence, trends, and insights into enterprise IT environments. XpoLog is a trusted source for DevOps analytics tools with over 10 years of experience in the IT industry. Trial versions of the XpoLog Platform can be downloaded as a standalone server or J2EE application through the company’s official website. [http://www.xpolog.com]

DevOps and Log Management Tools Drive Agile IT

By Omry Koschitzky, VP Solutions of XpoLog

Complex IT environments have made troubleshooting increasingly difficult, but remedies including the DevOps trend and IT log analysis tools have emerged in response. These approaches can be a powerful combination for faster resolution.

barx2It’s never good when mission critical applications fail; it’s even worse when this happens after normal business hours. Application uptime matters – a lot, because users want constant connectivity and IT is critical to many business operations.

An application has many stakeholders from DevOps to compliance, security teams and, of course, the sponsor. The motivations for rapid troubleshooting might differ. However, time-to-resolution is a unifying factor that dissolves old divisions. But how do you determine where the problem lies? The cause can be elusive when users complain about transaction failure. There may be an integration issue, bad code, and problems with infrastructure, security, load, and so on.

Deployments that have several components running in private and public clouds, and/or highly virtualized hybrid deployments make the triage process a complicated task, and the adoption of cloud services is only increasing. A recent Gartner survey of 651 organizations found that only 38% were using cloud services, but 80% will be within the next 12 months. That’s a recipe for greater complexity.

DevOps to the rescue

A holistic approach is needed to find exactly where something went wrong. DevOps provides a number of benefits including closer collaboration between IT and developers. These departments didn’t always see eye-to-eye, and developers were sometimes thought of as second-class citizens. DevOps has changed that by making everyone a stakeholder working for a common purpose, and can result in much faster code deployments (30%) and remarkably fewer failures (50%), according to a recent Puppet Labs survey.

“Why is DevOps reshaping enterprise IT? Quite simply, because it works. Because IT operations and development are better in collaboration than in competition,” said Matt Asay, vice president of business development and corporate strategy at MongoDB. “63% of organizations have adopted DevOps practices,” he noted.

Teamwork is essential to the DevOps triage process, which can untangle complex systems to determine a root cause. Otherwise, it would be difficult to learn whether a transaction failed across an entire tier (e.g., all app servers), if only one server is causing the problem, or whether bad code or a 3rd party or cloud service is to blame. It’s almost impossible to use old style methodologies in such dynamic environments, but adopting DevOps practices is a good starting point.

In fact, DevOps yields superior results over time. The Puppet Labs survey found that failure rates and time-to-resolution continue to fall the longer DevOps practices are followed. Puppet Labs commercializes open source IT automation software, but it’s part of a booming market to support DevOps. Other companies include IBM and Opscode, the company behind Chef, an open source tool. Gartner has rated several tool companies as cool vendors for DevOps. Tools establish a framework for collaboration with a range of capabilities that can scale up to large enterprises.

Finding the needle in a needles factory

However, collaboration is only one aspect of the triage process. Teams also require tools that must be flexible enough to support dynamic data center architectures and that have the power to provide real time analysis of the data. That’s where IT log analysis comes into play. Application and server logs can exist in many places and formats, so analysis tools have evolved to centralize log management and search.

Some tools even utilize correlation and machine learning to help users triage their application problems by filtering out unrelated events. For instance, it’s now possible for software to determine that 25 log entries out of thousands related to database connectivity are urgent. These tools can all help to resolve the aforementioned database problem, but can differ in time- to-resolution depending on the level of automation that the project leaders or vendor has provided.

Dark Reading has said that while log analysis can sometimes be a daunting task, prior to the introduction of augmented search results: “When done right, however, it is a process that can improve response time for both operational and security staff.”

Log analysis tools can vary considerably in utility, but there are many mature options available, both commercial as well as free software. Every organization has its own requirements and can determine what tool is best for its DevOps teams. Just using log mining makes DevOps even more effective and helps with triage of problems – which is important for increasingly complex environments.

Organizations should consider implementing DevOps practices alongside IT log analysis tools. Receiving an alert about a transaction failure at 10pm could be disaster, but the right organizational resources can swiftly turn a would-be late night crisis into an easily resolved incident and extra pillow time.

Free Log Management XpoLog 1GB/Day Full-Featured Version

by Haim Koschitzky, CEO

Untitled-34Great news folks, today we have unveiled our new pricing plan, which includes a unique free offering for those who want to start using XpoLog’s log management and analysis system with Augmented Search technology. The free plan includes all features and is limited to 1GB/Day. The license has no limitations on the number of users or devices that send log data. XpoLog can be installed on-premise as well as on the client’s private and public cloud environment.

We are very excited to be first in the market to offer a fully-featured, leading technology solution for free. Continue reading

Application Log Management Vs. Security Log Management / SIEM

base22In the past several years log analysis technologies have matured, becoming a mainstream solution for troubleshooting a variety of problems across the various IT layers (infrastructure elements as well as applications). Although at first sight these technologies seem to do the same thing, i.e. enable the analysis of log events, different technologies have evolved to deal with different use cases. One of the main differentiation is the use of log analysis for data security vs. the use of log analysis for troubleshooting applications. In this post I will explain the main differences between these two use cases and the different technologies that are most appropriate for each one. Continue reading

Search, big data and log analysis: a coming of age story

Search, big data and log analysis: a coming of age story

by Gal Berg, CTO of XpoLog

The evolution of search is a fascinating story that can tell us a lot about how to solve other big data challenges such as log management and log analysis in IT environments. After all, in all these cases the basic idea is finding the “needle in a haystack” as quickly as possible. Continue reading