The new version of XpoLog’s Augmented Search Log analysis platform features integrates with Hadoop and HDFS. In this blog post we will look at how to actually perform this integration.
Adding intelligence to your Hadoop/HDFS logs
XpoLog Augmented Search 5.0 collects log data from Hadoop Distributed File System (HDFS), and layers new intelligent analytics on top of Hadoop. Those insights feed into augmented searches. XpoLog adds intelligence to log file search context with semantic analysis, and pattern and anomaly detection (to uncover insights and trends into application problems, systems, and user behavior). This helps users analyze problems within the Hadoop infrastructure and applications that run on the platform. It offers visibility into the distributed architecture, automatically triaging issues and errors for severity, and presenting results in a dashboard interface.
By combining XpoLog with Hadoop and HDFS you will be able to:
- Quickly gain automated insights into Hadoop deployment logs and events
- Troubleshoot and investigate Hadoop application logs
- Access, collect and analyze any data that is stored on the HDFS
With XpoLog you can quickly detect node connection problems, any type of exception, data store and data block errors and much more.
Connecting XpoLog to your existing Hadoop/HDFS deployment
- Log into XpoLog and go to Manager>Administration>Cloud.
- Add a new Hadoop account and fill in the Name, Description, Host and Port fields.
- Go to Administration>Add Log, or Add entire logs directory path to XpoLog.
- Add the Hadoop logs and repeat this task as needed to get all your logs into XpoLog.
- Make sure that both the logs of the Hadoop apps and Hadoop infrastructure were added
Using XpoLog to investigate Hadoop/HDFS events
Go the Search console and run searches on both Hadoop logs and Hadoop HDFS stored logs
If the logs were tagged to specific apps you can simple search “* in app.hadoop” for example.
Check out the Analytics console for automated insights layer of problems and errors
Try the Augmented Search options to combine the power of automated intelligence in the context of the search in order to discover new intelligence layers and errors in the context of your search.
You can now go to the dashboards and add visualization gadgets/dashboards based on your search queries and the log analytics insights layers.