by Haim Koschitzky, XpoLog CEO
In my previous post, XpoLog 6 Log Management: Listening, Single Page App, AngularJS, and UX/UI, I summarized the key topics we focused on for XpoLog 6. This post will focus on methods we used to manage unstructured IT log data and visualization using smart tagging techniques.
I am sure, unless you are reading this post by accident :-), you have had the opportunity in the past to see Visio style IT diagrams, CMDB dependency flow charts, architecture schemas of complex applications, and/or security/network architecture charts. One thing all these diagrams have in common is that they all look like organized spaghetti. With the current complexity and dynamic nature of virtual infrastructure (private or public cloud), these “spaghetti diagrams” are no longer static in nature, but highly dynamic.
This means that infrastructure diagrams are no longer functioning as solid navigation maps for IT issues. The number of servers and their names, application components and so on are constantly changing according to system constraints. New APM strategies support this with the notion of smart tagging, transaction marking combined with high level application flow awareness based on advanced correlations. The ITOA and Log analysis should embrace those techniques to be able to present insights around common IT structures.
So how do we organize our data in a meaningful way that will not only make sense, but also be practical, usable, visible, and accessible quickly; in addition to being organized to support DevOps and APM insights?
Data Virtualization for IT Visualization
We decided to embrace the same approach we used with logical data structures. While adding additional data sources to XpoLog it is possible to organize these new data sources in a virtual structure of folders and log nodes. Later users can search data and browse automated analytics in the context of these virtual structures. This approach makes it easier to make bulk configuration changes, manage security policies and so on. One very cool option is to define a new virtual log source on another log data source but with a different pattern and different rules. This new log source can be used to filter, hide, or manipulate data for indexing.
In order to organize log data nodes and visualization views in a logical structure, we invented two new virtualization types: AppTags and Virtual Application Structures (Apps).
AppTags are tags that indicate the relevance of each log source to an Application. This means that a single Access log from a single sign-on (SSO) service that is serving 3 different apps, can be tagged with 3 AppTags: AppA, AppB, AppC. Further, in order to better analyze the SSO service, an additional SSO AppTag can be attached.
Why should you care?
Because after AppTags are attached, you can switch the log analysis context in the search, dashboards, visualization, automated log analytics etc. simply by filtering the view according to the Application context relevant to your tasks. This option will unleash use cases for Production/Non-Production tags, DevOps tags, build number tags, and much more.
This powerful concept now allows you to create views, searches, and apps without stating the actual data sources, but rather referring only to the AppTags. Thus dynamically added logs that have the relevant AppTags will automatically be analyzed in multi-dimensional contexts.
This level of abstraction is great for data and log management. The next step is to build actual logical Apps that will provide us with the capability to manage visualization in a robust, scalable, and highly dynamic way.
Virtual Application Structures (Apps)
XpoLog 6 is going to have an Apps console in which users and groups can organize visualization dashboards in the context of an App. For example, build an App (e.g. TomcatApp) for all Apache Tomcat application servers; in it you can create different dashboards – Performance, Availability, Errors and Exceptions, Threads and Memory, Security and User statistics, etc. Each dashboard will contain visualization gadgets presenting charts, graphs, and maps that will aggregate and summarize information in the context of the dashboard.
When building the App we recommend binding the queries to the AppTag Tomcat. This means all Tomcat logs relevant to the queries will be included automatically. A more advanced way is to zoom in on and focus only on Tomcat logs associated with AppA, visualizing data for a specific business application.
Once you get familiar with the concept of building visualization Apps, you can do this for any business application or IT solution you use. A more advanced capability will allow you to duplicate apps and change the context to use the source for different AppTags. This means you will be able to reuse your Apps in different environments and other Apps.
ITOA – IT Data Visualization Strategy
By defining AppTags and Apps, a new abstraction layer will help solve the exhausting configuration and maintenance involved in generating value from large amounts of data. The new structure will help organize insights, data, and visual components in both business and IT context.
In my next post I will expand the concept of visualization strategy and component.