How to Move Beyond “Mean-Time-to-Innocence”

June 10, 2014 | By Matt Weaver | 1 Comment
MTTI decision tree 350px

Click on the image to see a larger version of the MTTI decision tree. ExtraHop helps to abolish MTTI with correlated, cross-tier visibility.

You may have heard of the tongue-in-cheek term mean-time-to-innocence (MTTI), or the time it takes to prove that the network, virtualization, database, storage, or the application is not to blame for an application performance issue. One of the primary reasons why IT organizations choose the ExtraHop platform is to abolish MTTI.

If Everything’s Green, Why Is the Application Still Slow?

All of us in IT have seen this scenario: IT gets notified of an issue, and the serial troubleshooting process begins. Deciding where to begin is haphazard. Tribal knowledge leads someone to declare that the issue “feels” like a network/database/application/whatever problem. Perhaps the party guilty for the last issue receives the honor of proving their innocence first.

The group first accused either finds the issue (not typical) or reports that their silo-specific toolset shows clean and green (MTTI), and then it’s the next group’s turn to prove their innocence. This process continues as each group uses their own set of tools and data sources. Adding pressure to the situation are constant demands from business stakeholders for status updates and an ETA on resolution.

Packets Don’t Lie, But You Have to Make Them Talk

When the process has run its course and the issue remains elusive, there is only one thing left to do: A packet capture that hopefully records the issue. That may have been okay in the days of <1 gigabit wire speeds, provided the requisite network engineering expertise is on hand to interpret the packet data. Even then, the engineer typically also needs to possess a deep understanding of how the systems and applications should work. Nowadays with 10, 20, 40, and even 100 gigabit wire speeds, traditional packet capture is no longer feasible. There is simply too much data to comb through even for the most talented engineer. But the fact that we have always resorted to packet captures proves that we have always known that the truth is on the wire. Mining this rich, objective source of data to produce easily understandable information is the trick.

An Objective Source of Truth: Real-Time Wire Data

The ExtraHop platform enables IT organizations to skip the blamestorming process and to tap into the wealth of information found in wire data. It does this at a sustained 20Gbps while providing digestible L2-L7 metrics in real-time. ExtraHop provides the context required to quickly identify where performance issues reside, eliminating unnecessary MTTI exercises and greatly reducing mean-time-to-resolution (MTTR). With correlated visibility into all layers of the application delivery chain, IT teams can bypass the guesswork of where the issue may hide. Moreover, this is all possible using data that you already have. There is no need to stand up yet another database to store and query performance data.

Related case studies
MedSolutions provides real-time IT insights to all IT groups
Claris Networks speeds up troubleshooting for hosted applications

Real-World Story of Network Team vs. Application Team

I recently worked with an IT organization where ExtraHop proved its worth as the objective source of truth. After the networking team had installed new load balancers, the application team immediately reported an increase of 12 seconds in response time for one of their applications. The old load balancers were redeployed and, sure enough, the issue went away.

Over the next 18 months, each team used their own set of tools to try to uncover what actually was causing the performance hit with the new equipment. The network team was sure it was not the load balancers, but the application team would not allow the new ones to be installed until the root cause was uncovered. As the standoff continued, the old load balancers were approaching end-of-life, and one of them actually failed altogether, creating a dangerous single point of failure.

The organization contacted ExtraHop to see if we could assist. Within an hour of analyzing the traffic and with zero custom configuration, ExtraHop was able to identify the problem. A flaw in an SQL query statement that did not affect the old load balancers was exposed with the new load balancers. Armed with this information off the wire, the application team made a simple code change that sped up the application on the new load balancers so that it performed better than it ever had with the old equipment.

ExtraHop provides real-time IT insights for all IT groups so that they can bypass the MTTI exercise. DBAs, systems engineers, network architects, InfoSec analysts, application developers—they can all benefit from a view into wire data, all L2-L7 communications between systems. The truth is on the wire, and ExtraHop helps IT teams tap into it. Check it out for yourself with the ExtraHop Discovery Edition, a free virtual appliance.

0

Filed in: ExtraHop Analysis, ExtraHop Blog, Good Reads, Stories from the IT Trenches | Tags: , , ,

About the Author (Author Profile)

Matt Weaver is a Systems Engineer with ExtraHop based in Denver. After a couple of decades in IT operations focused on a variety of technologies such as virtualization, storage, servers, and networking, he made the jump to the vendor side. His past experience serves as a nice backdrop for the application of ExtraHop's wire data analytics platform. Contact Matt if you want to hear all about ExtraHop while riding lifts at one of Colorado's many awesome resorts!

Comments (1)

  1. Bill Storey

    Interested in learning more about the product.

Leave a Reply

Trackback URL | RSS Feed for This Entry