Hands On: XpoLog Augmented Search™ 5.0. with Hadoop and HDFS Integration

The new version of XpoLog’s Augmented Search Log analysis platform features integrates with Hadoop and HDFS. In this blog post we will look at how to actually perform this integration.

Adding intelligence to your Hadoop/HDFS logs

XpoLog Augmented Search 5.0 collects log data from Hadoop Distributed File System (HDFS), and layers new intelligent analytics on top of Hadoop. Those insights feed into augmented searches. XpoLog adds intelligence to log file search context with semantic analysis, and pattern and anomaly detection (to uncover insights and trends into application problems, systems, and user behavior). This helps users analyze problems within the Hadoop infrastructure and applications that run on the platform. It offers visibility into the distributed architecture, automatically triaging issues and errors for severity, and presenting results in a dashboard interface.

By combining XpoLog with Hadoop and HDFS you will be able to:

  • Quickly gain automated insights into Hadoop deployment logs and events
  • Troubleshoot and investigate Hadoop application logs
  • Access, collect and analyze any data that is stored on the HDFS

With XpoLog you can quickly detect node connection problems, any type of exception, data store and data block errors and much more.

 

Connecting XpoLog to your existing Hadoop/HDFS deployment

  1. Log into XpoLog and go to Manager>Administration>Cloud.
  2. Add a new Hadoop account and fill in the Name, Description, Host and Port fields.
    Image
  3. Go to Administration>Add Log, or Add entire logs directory path to XpoLog.
  4. Add the Hadoop logs and repeat this task as needed to get all your logs into XpoLog.
    Image
  5. Make sure that both the logs of the Hadoop apps and Hadoop infrastructure were added
    Image

That’s it!

Using XpoLog to investigate Hadoop/HDFS events

Go the Search console and run searches on both Hadoop logs and Hadoop HDFS stored logs

If the logs were tagged to specific apps you can simple search “* in app.hadoop” for example.

 Image

Check out the Analytics console for automated insights layer of problems and errors

 Image

Try the Augmented Search options to combine the power of automated intelligence in the context of the search in order to discover new intelligence layers and errors in the context of your search.

You can now go to the dashboards and add visualization gadgets/dashboards based on your search queries and the log analytics insights layers.

Get started with the full featured XpoLog Platform free license – click here

XPOLOG LOG AUGMENTED SEARCH™ 5.0 is HERE!

Analytics and machine learning add intelligence to Hadoop application and server log analysis

Image

NEW YORKMay 21, 2014 /PRNewswire/ – XpoLog Ltd, the company that invented Augmented Search for IT log analysis, has introduced XpoLog Augmented Search™ 5.0. The new version brings XpoLog’s troubleshooting capabilities to the Hadoop platform to help DevOps teams rapidly diagnose and solve Hadoop deployment and application problems.

Testing applications on Hadoop, a large-scale, distributed data processing platform, isn’t a trivial task, and it’s made even more difficult without tools that accelerate DevOps activities. XpoLog Augmented Search 5.0 helps in two ways: it collects log data from Hadoop Distributed File System (HDFS), and by layering new intelligent analytics on top of Hadoop. Those insights feed into augmented searches.

XpoLog adds intelligence to log file search context with semantic analysis, and pattern and anomaly detection (to uncover insights and trends into application problems, systems, and user behavior). This helps users analyze problems within the Hadoop infrastructure and applications that run on the platform. It offers visibility into the distributed architecture, automatically triaging issues and errors for severity, and presenting results in a dashboard interface.

“With our latest version, organizations large and small can use augmented search to gain vital insights into Hadoop deployments,” said XpoLog VP Solutions Omry Koschitzky. “Our solution is particularly useful for Hadoop distributions across a large number of servers and data centers.”

Customers using XpoLog’s Augmented Search for DevOps have reported drastically reduced time-to-resolution throughout key verticals. There is also no barrier to entry, because the XpoLog platform is free for processing up to 1 gigabyte of log data per day. XpoLog is also the only automated log data analytics solution that supports thelogstash data infrastructure, which is also supported by the 5.0 release.

The full product brochure, an online demo, additional pricing information, and a free trial of the Log Analysis Platform, are all available through XpoLog’s Web site.

About XpoLog Ltd
XpoLog Ltd is an IT operations analytics software company based in the Israel and the U.S. that invented augmented search, a breakthrough which unlocks the hidden value of log data. The platform drastically reduces time to resolution and provides a wealth of intelligence, trends, and insights into enterprise IT environments. XpoLog is a trusted source for DevOps analytics tools with over 10 years of experience in the IT industry. Trial versions of the XpoLog Platform can be downloaded as a standalone server or J2EE application through the company’s official website. [http://www.xpolog.com]

 

XpoLog enhances logstash with Augmented Search

IT departments that have adopted the open source solution can add on the world’s most advanced log analysis platformImage

NEW YORK CITY – May. 10, 2014 – XpoLog Ltd, the company that invented Augmented Search for IT log analysis, is integrating its industry leading technologies with, logstash, a popular open source log-processing tool. Organizations benefit by leveraging their existing infrastructure with XpoLog’s robust log management features, automated triaging of IT problems, and advanced reporting and visualization – with pricing that ranges from free to affordable.

 

Elasticsearch.org’s logstash project centralizes logging with user-defined rules; however, it lacks enterprise level features. While it may be used in combination with several separate open source projects, only XpoLog offers a production-proven, holistic solution to maximize its full potential.

 

XpoLog enables logstash users to create Web access log statistics, application log error summaries, performance charts, and other reports, quickly and efficiently. Statistical queries instantly visualize log data, which can be used to build charts, dashboards, geomaps, and much more.

 

XpoLog is also the only automated log data analytics solution that supports the logstash data infrastructure. Users can avoid time-consuming manual searches. Customers using XpoLog’s Augmented Search for DevOps have reported drastically reduced time-to-resolution across major vertical industries.

 

“We believe that it’s important to support the data infrastructures that our customers want to use,” said XpoLog CEO Haim Koschitzky. “It’s also vital for organizations that have already invested in a data engine to have access to the most advance data analytics technologies available on the market.”

 

XpoLog’s solution uniquely layers intelligence on top of search context using semantic analysis, as well as pattern and anomaly detection to uncover insights and trends into application problems, systems, and user behavior. Those capabilities allow it to capture a wealth of intelligence from within IT environments, including the analysis of proprietary log data. There is also no barrier to entry, because the XpoLog platform is free for processing up to 1 gigabyte of log data per day.

 

The full product brochure, an online demo, additional pricing information, and a free trial of the Log Analysis Platform, are all available through XpoLog’s Web site.

 

 

About XpoLog Ltd
XpoLog Ltd is an IT operations analytics software company based in the Israel and the U.S. that invented augmented search, a breakthrough which unlocks the hidden value of log data. The platform drastically reduces time to resolution and provides a wealth of intelligence, trends, and insights into enterprise IT environments. XpoLog is a trusted source for DevOps analytics tools with over 10 years of experience in the IT industry. Trial versions of the XpoLog Platform can be downloaded as a standalone server or J2EE application through the company’s official website. [http://www.xpolog.com]

DevOps and Log Management Tools Drive Agile IT

By Omry Koschitzky, VP Solutions of XpoLog

Complex IT environments have made troubleshooting increasingly difficult, but remedies including the DevOps trend and IT log analysis tools have emerged in response. These approaches can be a powerful combination for faster resolution.

barx2It’s never good when mission critical applications fail; it’s even worse when this happens after normal business hours. Application uptime matters – a lot, because users want constant connectivity and IT is critical to many business operations.

An application has many stakeholders from DevOps to compliance, security teams and, of course, the sponsor. The motivations for rapid troubleshooting might differ. However, time-to-resolution is a unifying factor that dissolves old divisions. But how do you determine where the problem lies? The cause can be elusive when users complain about transaction failure. There may be an integration issue, bad code, and problems with infrastructure, security, load, and so on.

Deployments that have several components running in private and public clouds, and/or highly virtualized hybrid deployments make the triage process a complicated task, and the adoption of cloud services is only increasing. A recent Gartner survey of 651 organizations found that only 38% were using cloud services, but 80% will be within the next 12 months. That’s a recipe for greater complexity.

DevOps to the rescue

A holistic approach is needed to find exactly where something went wrong. DevOps provides a number of benefits including closer collaboration between IT and developers. These departments didn’t always see eye-to-eye, and developers were sometimes thought of as second-class citizens. DevOps has changed that by making everyone a stakeholder working for a common purpose, and can result in much faster code deployments (30%) and remarkably fewer failures (50%), according to a recent Puppet Labs survey.

“Why is DevOps reshaping enterprise IT? Quite simply, because it works. Because IT operations and development are better in collaboration than in competition,” said Matt Asay, vice president of business development and corporate strategy at MongoDB. “63% of organizations have adopted DevOps practices,” he noted.

Teamwork is essential to the DevOps triage process, which can untangle complex systems to determine a root cause. Otherwise, it would be difficult to learn whether a transaction failed across an entire tier (e.g., all app servers), if only one server is causing the problem, or whether bad code or a 3rd party or cloud service is to blame. It’s almost impossible to use old style methodologies in such dynamic environments, but adopting DevOps practices is a good starting point.

In fact, DevOps yields superior results over time. The Puppet Labs survey found that failure rates and time-to-resolution continue to fall the longer DevOps practices are followed. Puppet Labs commercializes open source IT automation software, but it’s part of a booming market to support DevOps. Other companies include IBM and Opscode, the company behind Chef, an open source tool. Gartner has rated several tool companies as cool vendors for DevOps. Tools establish a framework for collaboration with a range of capabilities that can scale up to large enterprises.

Finding the needle in a needles factory

However, collaboration is only one aspect of the triage process. Teams also require tools that must be flexible enough to support dynamic data center architectures and that have the power to provide real time analysis of the data. That’s where IT log analysis comes into play. Application and server logs can exist in many places and formats, so analysis tools have evolved to centralize log management and search.

Some tools even utilize correlation and machine learning to help users triage their application problems by filtering out unrelated events. For instance, it’s now possible for software to determine that 25 log entries out of thousands related to database connectivity are urgent. These tools can all help to resolve the aforementioned database problem, but can differ in time- to-resolution depending on the level of automation that the project leaders or vendor has provided.

Dark Reading has said that while log analysis can sometimes be a daunting task, prior to the introduction of augmented search results: “When done right, however, it is a process that can improve response time for both operational and security staff.”

Log analysis tools can vary considerably in utility, but there are many mature options available, both commercial as well as free software. Every organization has its own requirements and can determine what tool is best for its DevOps teams. Just using log mining makes DevOps even more effective and helps with triage of problems – which is important for increasingly complex environments.

Organizations should consider implementing DevOps practices alongside IT log analysis tools. Receiving an alert about a transaction failure at 10pm could be disaster, but the right organizational resources can swiftly turn a would-be late night crisis into an easily resolved incident and extra pillow time.

Free Log Management XpoLog 1GB/Day Full-Featured Version

by Haim Koschitzky, CEO

Untitled-34Great news folks, today we have unveiled our new pricing plan, which includes a unique free offering for those who want to start using XpoLog’s log management and analysis system with Augmented Search technology. The free plan includes all features and is limited to 1GB/Day. The license has no limitations on the number of users or devices that send log data. XpoLog can be installed on-premise as well as on the client’s private and public cloud environment.

We are very excited to be first in the market to offer a fully-featured, leading technology solution for free. Continue reading

Application Log Management Vs. Security Log Management / SIEM

base22In the past several years log analysis technologies have matured, becoming a mainstream solution for troubleshooting a variety of problems across the various IT layers (infrastructure elements as well as applications). Although at first sight these technologies seem to do the same thing, i.e. enable the analysis of log events, different technologies have evolved to deal with different use cases. One of the main differentiation is the use of log analysis for data security vs. the use of log analysis for troubleshooting applications. In this post I will explain the main differences between these two use cases and the different technologies that are most appropriate for each one. Continue reading

Search, big data and log analysis: a coming of age story

Search, big data and log analysis: a coming of age story

by Gal Berg, CTO of XpoLog

The evolution of search is a fascinating story that can tell us a lot about how to solve other big data challenges such as log management and log analysis in IT environments. After all, in all these cases the basic idea is finding the “needle in a haystack” as quickly as possible. Continue reading