The IT organization at Seattle Children's Hospital supports more than 100 applications for over 8,500 users across 25 different physical locations. "We've got an environment with just about every technology you could think of," says Tim Holt, Senior Director of Enterprise Architecture. "And consequently, it's very, very difficult to troubleshoot performance from an end-user perspective."
Seeking greater visibility, Seattle Children's turned to ExtraHop for something none of their traditional monitoring products offered: the ability to tap into a wealth of wire data flowing through their environments. Wire data is all L2- L7 communications between systems, including full bidirectional transactional payloads.
Seeing How Apps Really Work
For the first time, Seattle Children's had the cross-tier visibility to put troubleshooting issues in context.
"I've never seen anything comparable to ExtraHop," says Bruce Fulton, Senior Systems Infrastructure Team Engineer. "It's our way to see how a transaction flows from start to finish through these various applications. We simply couldn't get that end-to-end perspective with any of our previous technologies."
"We're familiar with network sniffers and so forth," adds Holt. "But those things only give you raw data, not the contextual associations across tiers as you have with ExtraHop."
No More Blind Troubleshooting
Before ExtraHop, the IT infrastructure team was forced to follow a time-consuming, "guilty until proven innocent" approach to troubleshooting, spending 20% or more of their time trying to diagnose non-obvious problems. ExtraHop cut that in half.
"We always found ourselves trying to prove that a problem wasn't coming from a particular technology silo," says Holt. "We'd start with the network team, who would burn a whole bunch of time proving the network was operating as expected, and then you'd move on to the next level of the stack. 'Well, it's not here, must be somewhere else.'"
With ExtraHop, IT teams at Seattle Children's significantly improved end-user experience for key applications and gained critical insight needed to do their jobs more efficiently.
Speedy Citrix Logins
Seattle Children's relies heavily on virtual desktop infrastructure (VDI), delivering nearly 3,000 Windows 7 desktops through a Citrix environment. Troubleshooting with ExtraHop has helped to dramatically improve Citrix logins and application launches, both key metrics for measuring end-user experience.
Seattle Children's IT teams had spent weeks trying to isolate the cause of extremely slow Citrix XenDesktop logins every morning around 8:30 a.m. Inexplicably, login times would increase from 12 seconds to nearly a minute on average. Shortly after installing ExtraHop, the problem was identified as severe contention at the storage tier, caused by an end-user unintentionally loading 2GB of photos in his My Pictures folder every time he logged in. With this insight, Seattle Children's restricted use of the My Pictures folder and made other optimizations that earned goodwill from users and paved the way for an expansion of the VDI deployment.
"Imagine a physician doing morning rounds, going workstation to workstation, pulling up their desktop," says Holt. "Connects and disconnects need to be as quick as possible—and even 30 seconds might not seem like much, but multiply that over 30 or 40 incidences and you have a lot of wasted time."
Greater Third-Party Accountability
Before implementing ExtraHop, IT teams troubleshooting performance issues on SaaS and third-party hosted applications often didn't have enough visibility to determine root causes or to verify claims from vendors and service providers. "We have a hosted Lawson ERP application, and now the Lawson team can look at the metrics for all communications between our site and the external Lawson site, broken down by server processing time and round-trip time, to see where there might be an issue," says Fulton.
A More Empowered IT Staff
"All you really need to use ExtraHop is an understanding of the OSI model," says Fulton. "Once you explain the layers to them, it's really easy for any IT staff member to apply ExtraHop to their work. Our developers and application support teams have been really excited about ExtraHop because it provides them with new insights into how efficiently their applications are working."
ExtraHop has helped IT staff at Seattle Children's understand how complex applications really work. "Before, I would ask people, can you map out what's really happening here—for example, with logging in to a Cerner application—and almost no one could map that end-to- end," says Holt. "Now with ExtraHop, we have at least 15 staff who can map that out in a heartbeat, and that number is growing."
Faster troubleshooting = $354,500 annual savings
Seattle Children's IT infrastructure team used to spend 20% or more of their time diagnosing non-obvious problems. ExtraHop has cut that in half for those trained on the platform.
With an infrastructure team of 40 and an average salary of $88,625 for an IT Operations Engineer in the Seattle area, that amounts to approximately $29,500 in potential savings in just the first month or $354,500 annually.
The Hidden Cost of Slow Logins
Clinicians and nurses log into and out of their virtual Windows 7 desktops as they move around the hospital. A single nurse might log in to more than 40 different devices during a 12-hour shift, so when ExtraHop helped reduce connect and disconnect times, that added up to a significant positive impact on productivity and quality of care.