Hands On: XpoLog Augmented Search™ 5.0. with Hadoop and HDFS Integration

The new version of XpoLog’s Augmented Search Log analysis platform features integrates with Hadoop and HDFS. In this blog post we will look at how to actually perform this integration.

Adding intelligence to your Hadoop/HDFS logs

XpoLog Augmented Search 5.0 collects log data from Hadoop Distributed File System (HDFS), and layers new intelligent analytics on top of Hadoop. Those insights feed into augmented searches. XpoLog adds intelligence to log file search context with semantic analysis, and pattern and anomaly detection (to uncover insights and trends into application problems, systems, and user behavior). This helps users analyze problems within the Hadoop infrastructure and applications that run on the platform. It offers visibility into the distributed architecture, automatically triaging issues and errors for severity, and presenting results in a dashboard interface.

By combining XpoLog with Hadoop and HDFS you will be able to:

  • Quickly gain automated insights into Hadoop deployment logs and events
  • Troubleshoot and investigate Hadoop application logs
  • Access, collect and analyze any data that is stored on the HDFS

With XpoLog you can quickly detect node connection problems, any type of exception, data store and data block errors and much more.


Connecting XpoLog to your existing Hadoop/HDFS deployment

  1. Log into XpoLog and go to Manager>Administration>Cloud.
  2. Add a new Hadoop account and fill in the Name, Description, Host and Port fields.
  3. Go to Administration>Add Log, or Add entire logs directory path to XpoLog.
  4. Add the Hadoop logs and repeat this task as needed to get all your logs into XpoLog.
  5. Make sure that both the logs of the Hadoop apps and Hadoop infrastructure were added

That’s it!

Using XpoLog to investigate Hadoop/HDFS events

Go the Search console and run searches on both Hadoop logs and Hadoop HDFS stored logs

If the logs were tagged to specific apps you can simple search “* in app.hadoop” for example.


Check out the Analytics console for automated insights layer of problems and errors


Try the Augmented Search options to combine the power of automated intelligence in the context of the search in order to discover new intelligence layers and errors in the context of your search.

You can now go to the dashboards and add visualization gadgets/dashboards based on your search queries and the log analytics insights layers.

Get started with the full featured XpoLog Platform free license – click here

XpoLog enhances logstash with Augmented Search

IT departments that have adopted the open source solution can add on the world’s most advanced log analysis platformImage

NEW YORK CITY – May. 10, 2014 – XpoLog Ltd, the company that invented Augmented Search for IT log analysis, is integrating its industry leading technologies with, logstash, a popular open source log-processing tool. Organizations benefit by leveraging their existing infrastructure with XpoLog’s robust log management features, automated triaging of IT problems, and advanced reporting and visualization – with pricing that ranges from free to affordable.


Elasticsearch.org’s logstash project centralizes logging with user-defined rules; however, it lacks enterprise level features. While it may be used in combination with several separate open source projects, only XpoLog offers a production-proven, holistic solution to maximize its full potential.


XpoLog enables logstash users to create Web access log statistics, application log error summaries, performance charts, and other reports, quickly and efficiently. Statistical queries instantly visualize log data, which can be used to build charts, dashboards, geomaps, and much more.


XpoLog is also the only automated log data analytics solution that supports the logstash data infrastructure. Users can avoid time-consuming manual searches. Customers using XpoLog’s Augmented Search for DevOps have reported drastically reduced time-to-resolution across major vertical industries.


“We believe that it’s important to support the data infrastructures that our customers want to use,” said XpoLog CEO Haim Koschitzky. “It’s also vital for organizations that have already invested in a data engine to have access to the most advance data analytics technologies available on the market.”


XpoLog’s solution uniquely layers intelligence on top of search context using semantic analysis, as well as pattern and anomaly detection to uncover insights and trends into application problems, systems, and user behavior. Those capabilities allow it to capture a wealth of intelligence from within IT environments, including the analysis of proprietary log data. There is also no barrier to entry, because the XpoLog platform is free for processing up to 1 gigabyte of log data per day.


The full product brochure, an online demo, additional pricing information, and a free trial of the Log Analysis Platform, are all available through XpoLog’s Web site.



About XpoLog Ltd
XpoLog Ltd is an IT operations analytics software company based in the Israel and the U.S. that invented augmented search, a breakthrough which unlocks the hidden value of log data. The platform drastically reduces time to resolution and provides a wealth of intelligence, trends, and insights into enterprise IT environments. XpoLog is a trusted source for DevOps analytics tools with over 10 years of experience in the IT industry. Trial versions of the XpoLog Platform can be downloaded as a standalone server or J2EE application through the company’s official website. [http://www.xpolog.com]