A large research hospital underwent a major virtual desktop infrastructure (VDI) expansion to support 4,000 users and more than 100 applications. Early on, boot-up and application launch times took about 10 to 12 seconds. Over the next several weeks, however, users began to complain about increasingly slow login times. Logins could take as long as 45 seconds, which was completely unacceptable. The slowdowns threatened to derail the VDI initiative.
Internal teams set up to investigate the issue were unsuccessful, even after several months. The tools they used would only confirm that Citrix was indeed experiencing a slowdown. The mystery issue became known as the "ghost in the machine." The team realized that Citrix is a complex environment with many interdependencies in the background, but they were unable to see the entire delivery chain.
- Reduce Citrix login times to 10 to 12 seconds.
- Gain visibility into Citrix delivery chain from remote access to XenDesktop servers to authentication and storage.
- Restore user confidence in the VDI rollout.
A large research hospital underwent a major VDI expansion that supported 4,000 users and more than 100 applications … Over the next several weeks, however, users began to complain about increasingly slow login times. Logins could take as long as 45 seconds, which was completely unacceptable.
The IT team decided to deploy the ExtraHop platform to try and troubleshoot the issue. The next time users reported slow logins, the team investigated the Citrix servers. The IT team identified the particular server associated with the slow login times, then drilled down to see that one user experienced consistently slow load times while most of the other users had load times ranging from five seconds to 30 seconds.
To investigate what was causing this behavior, the team next looked at which other L7 protocols that specific Citrix server used. A gigantic spike in CIFS traffic coincided with the long load time. Focusing on the actual files being transferred, they noticed a significant number of JPEG files in the user's roaming profile. This user stored photos in his Citrix profile folder. When this particular user logged in, the XenDesktop server would load his profile including the JPEG files, causing storage I/O contention and slowing all concurrent user logins. The IT team fixed this problem by removing the My Pictures folder from user's profiles.
To proactively monitor this and prevent performance issues in the future, the team built a dashboard to track all of the components of the Citrix XenDesktop delivery chain, including the network, authentication, applications, and storage. This dashboard displayed correlated information on users, files, and client IPs, allowing for quick identification of undesirable user behavior.
After reconfiguring the user's roaming profile and reducing its size, load times returned to the acceptable launch time of 10 to 12 seconds. The elusive "ghost in the machine" had been found, and user concerns about VDI performance mollified.
With ExtraHop's ability to mine the wire for information the customer saved over $400,000 spent for specialist consulting and many hours of troubleshooting. The Citrix administrators now had a way to look across the Citrix environment and the supporting infrastructure.