XpoLog 6: Building NOC, Operations, DevOps, Security Rooms Views

by Haim Koschitzky, XpoLog CEO

XpoLog 6 is coming soon. In this series of posts I am covering the new primary features and enhancements. On my last post I discussed our visualization strategy, we are about to add more than 20 new visualization gadgets and there will be new ways to present information.

This short post will present one of the new cool features of XpoLog 6, the Ops View.

Different Perspective

We understand that different needs require different tools, and sometimes different situations require different views. The teams at the NOC, Operations, DevOps, SOC or High availability control rooms need to have ongoing status screens of managed environment. The need for an ongoing streaming of status views is very different from a single dashboard view or a search console view.

Turn Log Data to a Slideshow

In previous posts I wrote about Apps, AppTags, dashboards, and visualization gadgets. With XpoLog 6 you will be able to select multiple dashboards and run them as slideshows. Free your creativity and build great visual dashboards for security, performance, errors, business statistics, etc.

Rooms with A View

With XpoLog 6 you can build Availability and Business slideshow Views for application and business owners. Build R&D, Development, and Testing dashboards and have them slide in the meeting rooms. Operations, Security, and DevOps will be able to build multiple dashboards and Apps that will provide continuous feedback on systems and applications. Making a wealth of information and insights visibly accessible and dynamic will drive better actions.

image for 4th blog

Night Mode – Log Data can be Romantic

Check out the Themes options in the dashboard view that will invert the colors of the entire visualization views.

Create Smarter NOC and OPS Views

Release your creativity and build Apps and dashboards for any data, and correlate information from databases with logs. Visualize business Apps data from Hadoop with access logs stored on your CDN provider.

XpoLog 6 is coming soon along with some exciting news. Stay Tuned.

XpoLog 6: Log Data Visualization Components and Strategies

by Haim Koschitzky, XpoLog CEO

XpoLog 6 is coming soon. In this series of posts I am covering the new primary features and enhancements. This post will dive into our new visualization gadgets and the ideas guiding us in our long term visualization development road map.

Even though we see many log data analysis deployments, we still identify many challenges users are facing regarding IT log data visualization, analysis, and insights.

first.ask.questions.

Although stating the obvious, before investing expensive efforts and resources into analyzing data, it is crucial to define your expectations and requirements. While in the past, merely collecting all log data and making it available for search was good enough, this is no-longer the case.

In order to ask the right questions, determine what the most important use cases your log data has shown you and what role you want your log data to play in your future ongoing work. To do this, you must monitor system availability, software quality, continuous deployment, application performance, and business insights, troubleshoot, analyze security incidents, compliance audit etc.

There are specific use cases for the application life cycle: architect, developer, tester, DevOps, APM, operations, and production support all have specific uses cases and requirements. Giving the right answer to the right question makes a big impact and will drive smart actions.

then.visualize.

Once the requirements and expectations are well defined, add data to XpoLog. When doing so, organize data in Apps logical structures and AppTags, as was discussed in my previous post, XpoLog 6 Virtual Applications Structures, AppTags, and IT Visualization Strategies. Create an App that will contain a collection of dashboards; we recommend creating a dashboard per topic or use case, and providing each one with a meaningful name (“performance”, “errors”, “user audit”). Now follow the steps of creating search queries, or use out of the box gadgets for analytics.

With XpoLog 6 you will find example Apps that you will be able to use as examples of best use cases for log analysis data visualization.

how.to.visualize.

In the new version we added more than 20 new gadgets including 3D graphs, as we witnessed a growing demand for better visualization tools. Once you’ve created search queries to analyze data and generate proper result sets, you will need to select the visualization gadget that best reads these result sets and visualizes it in the most effective way.

Let’s look at a result set that aggregated and computed the avg. memory consumption and total memory usage of two application servers. Take a look at the figure below. On gadget 1 you can see the totals over 24 hr aggregated memory consumption at 1 hr intervals. This gadget tells the story of both servers. Gadgets 2 and 3 represent the same data but for each of the individual servers. Once we split the data for each server we discover that each of the servers had a very different memory consumption pattern.

An hourly aggregation for memory is far from being accurate; memory changes at a much faster rate. On the upper row of gadgets we see the totals for both servers (gadget 4) and two additional gadgets, 5 and 6, representing each server in 1 min intervals.

3rd blog post screen capture

 

We were looking to monitor our application server memory consumption to avoid spikes that might crash one of our clusters. Choosing the right visualization tools, and in this case, intervals, makes a big difference.

optimize.insights.

Optimize your dashboards and visualization gadgets by verifying they deliver the insights you’re after in the right resolution. In the example above, analyzing memory for the entire cluster did not provide a clear status image of the memory consumption, but grouping by server and later reducing the time interval resolution to minutes gave a clear understanding of which cluster spiked.

actions

Once your Apps and Dashboards provide clear views and visualization, it will become easy to identify problems, trends, and insights on your IT and applications. Now you will be able to monitor or view the dashboards live. Leverage the visibility and you will now be able to take actions that will make you applications more agile, secure, and optimized for the business.

ask.more.questions.

Again, go to the first step. This is an ongoing process. Data changes every day. The content of logs and other data types is being updated by IT, developers, and vendors every day. In order to stay ahead, keep asking questions and never stop looking for the answers.

We will publish a more comprehensive use case on how to create, optimize, and use the new Apps module. In my next post I will present our new Operations and DevOps screens with more visual examples.

XpoLog 6 is coming soon. Stay tuned.

XpoLog 6 Virtual Applications Structures, AppTags, and IT Visualization Strategies

by Haim Koschitzky, XpoLog CEO

In my previous post, XpoLog 6 Log Management: Listening, Single Page App, AngularJS, and UX/UI, I summarized the key topics we focused on for XpoLog 6. This post will focus on methods we used to manage unstructured IT log data and visualization using smart tagging techniques.

Dynamic Spaghetti

I am sure, unless you are reading this post by accident :-), you have had the opportunity in the past to see Visio style IT diagrams, CMDB dependency flow charts, architecture schemas of complex applications, and/or security/network architecture charts. One thing all these diagrams have in common is that they all look like organized spaghetti. With the current complexity and dynamic nature of virtual infrastructure (private or public cloud), these “spaghetti diagrams” are no longer static in nature, but highly dynamic.

This means that infrastructure diagrams are no longer functioning as solid navigation maps for IT issues. The number of servers and their names, application components and so on are constantly changing according to system constraints. New APM strategies support this with the notion of smart tagging, transaction marking combined with high level application flow awareness based on advanced correlations. The ITOA and Log analysis should embrace those techniques to be able to present insights around common IT structures.

So how do we organize our data in a meaningful way that will not only make sense, but also be practical, usable, visible, and accessible quickly; in addition to being organized to support DevOps and APM insights?

Data Virtualization for IT Visualization

We decided to embrace the same approach we used with logical data structures. While adding additional data sources to XpoLog it is possible to organize these new data sources in a virtual structure of folders and log nodes. Later users can search data and browse automated analytics in the context of these virtual structures. This approach makes it easier to make bulk configuration changes, manage security policies and so on. One very cool option is to define a new virtual log source on another log data source but with a different pattern and different rules. This new log source can be used to filter, hide, or manipulate data for indexing.

In order to organize log data nodes and visualization views in a logical structure, we invented two new virtualization types: AppTags and Virtual Application Structures (Apps).

AppTags

AppTags are tags that indicate the relevance of each log source to an Application. This means that a single Access log from a single sign-on (SSO) service that is serving 3 different apps, can be tagged with 3 AppTags: AppA, AppB, AppC. Further, in order to better analyze the SSO service, an additional SSO AppTag can be attached.

Why should you care?

Because after AppTags are attached, you can switch the log analysis context in the search, dashboards, visualization, automated log analytics etc. simply by filtering the view according to the Application context relevant to your tasks. This option will unleash use cases for Production/Non-Production tags, DevOps tags, build number tags, and much more.

This powerful concept now allows you to create views, searches, and apps without stating the actual data sources, but rather referring only to the AppTags. Thus dynamically added logs that have the relevant AppTags will automatically be analyzed in multi-dimensional contexts.

This level of abstraction is great for data and log management. The next step is to build actual logical Apps that will provide us with the capability to manage visualization in a robust, scalable, and highly dynamic way.

Virtual Application Structures (Apps)

XpoLog 6 is going to have an Apps console in which users and groups can organize visualization dashboards in the context of an App. For example, build an App (e.g. TomcatApp) for all Apache Tomcat application servers; in it you can create different dashboards – Performance, Availability, Errors and Exceptions, Threads and Memory, Security and User statistics, etc. Each dashboard will contain visualization gadgets presenting charts, graphs, and maps that will aggregate and summarize information in the context of the dashboard.

When building the App we recommend binding the queries to the AppTag Tomcat. This means all Tomcat logs relevant to the queries will be included automatically. A more advanced way is to zoom in on and focus only on Tomcat logs associated with AppA, visualizing data for a specific business application.

Once you get familiar with the concept of building visualization Apps, you can do this for any business application or IT solution you use. A more advanced capability will allow you to duplicate apps and change the context to use the source for different AppTags. This means you will be able to reuse your Apps in different environments and other Apps.

ITOA – IT Data Visualization Strategy

By defining AppTags and Apps, a new abstraction layer will help solve the exhausting configuration and maintenance involved in generating value from large amounts of data. The new structure will help organize insights, data, and visual components in both business and IT context.

In my next post I will expand the concept of visualization strategy and component.

XpoLog 6 is coming soon, so stay tuned…

XpoLog 6 Log Management: Listening, Single Page App, AngularJS, and UX/UI

by Haim Koschitzky, XpoLog CEO

The latest version of XpoLog Log Management will contain a huge face lift with regards to the UX/UI.

Listening

In the last couple of years we have received a lot of feedback from everyone who has used and/or evaluated XpoLog. For those of you who are not familiar with our numbers, we provide a Log analysis platform with approximately ten thousand installations worldwide, some of them are based on our free version, others on our OEM customers, data centers etc.

Of all the great feedback we received, there were two prominent points that were constantly repeated, the first one was something along the lines of “great technology”, “very smart analytics”, “rich functionality” etc. The second one was that the UI doesn’t look good, and the UX is not “cool enough”. I’ll tell you, it’s easier to listen to the first one and ignore the second. Human nature tends to prefer compliments, rather than criticism. But it is important to remember that constructive criticism can actually be helpful to us in the long run.

Last year I was speaking to analysts Colin Fletcher and Jonah Kowall from a leading research and analyst firm, Jonah later joined AppDynamics. At the end of that briefing, they summarized 3 primary points of feedback:

  • great technology and road map
  • bad UI/UX
  • more marketing needed

I left the meeting having a clear and focused plan for our next version; the message got through, I hope.

Single Page Apps and AngularJS

We did have plans to work on the UI/UX but we did not plan on working on it immediately, and only after processing all the feedback did we decide that doing only cosmetic changes was not going to cut it.

We wanted XpoLog to take a different direction when it came to UI/UX experience. When building a long term roadmap and strategy, if you lay the right foundations, something great can be built on top of it.

We selected AngularJS and Single-page app as we believed these are the front end technologies that will help us build a great user experience. There are a couple of technologies out there that help you build great UI and UX experience, in our case, fast and responsive user experience help us build smarter log data analytic tools.

Our creative team designed our new UI, only to redesign it again and again. We had users trying mockups, we had long arguments about buttons and flows, we built and rebuilt, we changed and we tested. We added more than 20 new data visualization types including 3D types. We are planning on adding many more.

We designed and implemented new ways of creating logical applications and visualization structures. We enabled data manipulation from visual components. We created operations and DevOps room screens with unique functions. We changed the data visualization gadgets and application structures and dashboard design. We built new UI/UX concepts for data analytics. We are proud of it all and we hope you guys will like it too.

It’s time to say thanks to our users, customers, friends, partners, critics and more critics ;). You guys really push us forward in our journey of building great products. I am sure our work just started but now it’s time to try the first release. Later this year we will redesign and reinvent the search console and the analytics console.

My next post will dive deep into logical application management and structure, smart tagging, and how to build a visualization strategy across IT.

XpoLog 6 will be released soon; stay tuned.