For Concur, optimizing the efficiency of its application infrastructure means building a competitive advantage. The largest SaaS provider in the ERP market, Concur is able to deliver a better product at lower prices because of its relentless pursuit of performance and scale.*
"Concur processes more than $50 billion in travel and expense reports each year— roughly 10 percent of the worldwide total," says Drew Garner, Director of Architecture Services at Concur. "As a software-as-a-service product, our pricing is directly tied to how much it costs to process each expense report. We have to be able to serve a transaction tomorrow with fewer resources than today. If we don't do that, we'll get beaten by the competition because they'll figure out how to do it first."
Seeking greater scalability and speed, Concur sought to replace its homegrown caching system with memcache. To identify the best candidates for migration to memcache, the R&D Operations team at Concur needed to analyze SQL query performance across thousands of databases. The team also needed to be able to monitor memcache performance and correlate that performance to activity at other tiers of the application infrastructure.
To optimize the database and caching tiers of its application, Concur worked with ExtraHop to implement memcache analysis. The ExtraHop system provides real-time transaction analysis at wire speed—up to a sustained 10Gbps—that covers the network, web, database, and storage tiers of the application.
"Concur stores 52 million items in 1.4 terabytes of memcache with sub-millisecond access and response times, but there is no way to query the system to find a particular key without dramatically impacting performance," explains Garner. "ExtraHop provides this visibility by passively analyzing transactions as they pass over the network."
In one case, the R&D Operations team used the ExtraHop system to find specific memcache keys that were not stored because they exceeded the default 1MB limit. "ExtraHop was the only product that could detect the problem and find the offending keys. With this specific information, we could apply compression in the application to fix the problem," says Garner. "Usually, people monitor memcache with server-side and client-side metrics, but there is a lot of activity in the middle that is crucial. With ExtraHop, we can monitor our memcache implementation from end to end."
Besides memcache, Concur uses the ExtraHop system to monitor and optimize the performance and scalability of its database tier. "We have thousands of database servers running two billion SQL queries and pushing out 10 terabytes of SQL data each day. This is the 'brain' of our application, and our DBAs use ExtraHop to make sure it is operating as efficiently as possible," says Garner.
Specifically, the R&D Operations team uses ExtraHop to pinpoint critical SQL queries and stored procedures that are performing poorly. "We customized analysis in the ExtraHop system to determine the total weight of SQL queries by calculating the number of times a query is run multiplied by the time required to return the data," explains Garner. "ExtraHop is the only solution that can provide this analysis continuously and in real time for our entire database infrastructure."
The alternative approach to measuring SQL performance would be to routinely run SQL traces on each database with a profiler— hardly feasible given the scale of Concur's infrastructure. "Our DBAs could either run a trace on each database, which would be like shining a spotlight on a small section of a highway, or they could use the ExtraHop system, which is like lighting the entire highway," says Garner.
Flexible Real-Time Analysis
One reason Concur has seen such great success with ExtraHop is the use of the Application Inspection Triggers technology, a framework for real-time analysis based on scriptable event processing at the application-protocol level. "We use Application Inspection Triggers to target specific real-time metrics that we want to investigate," says Garner. "For instance, we had an abnormally high rate of HTTP aborts for a pool of 60 front-end webservers that host three different sites. We have so much traffic to this pool that it was extremely difficult to isolate the problem server using our user-experience monitoring tool. By customizing the analysis in the ExtraHop system, we could identify a server that was configured to debugging mode. We turned debugging off and immediately saw the HTTP aborts fall by 95 percent in ExtraHop."
Concur uses the ExtraHop system to run its applications faster and more efficiently than competitors while minimizing the time and effort required to achieve its performance goals.
Sustained Competitive Advantage
By tuning the performance of its database and storage infrastructure, Concur minimizes IT costs even as usage increases. The ExtraHop system plays a key role in this optimization, helping Concur to easily identify SQL workloads that should be migrated to memcache and tuning the memcache deployment.
"Service providers like Concur need to continually seek ways to efficiently scale their infrastructure or risk falling behind," says Garner. "In part due to our focus on scalability, Concur was able to grow revenue while minimizing operating costs. Scale is even more important going forward as we expand internationally and grow platform services such as the Concur Connect Platform. ExtraHop gives us the intelligence we need to continually increase efficiency and sustain a competitive advantage."
Improved Customer Experiences
Concur uses the ExtraHop system to detect subtle issues that affect the speed and reliability of its services. "The ExtraHop system enables us to drill down and solve subtle, intermittent issues that are not easily identifiable in other tools and affect only some of our customers, some of the time," says Garner. "Even though these issues might not breach our service-level agreements, it is very important to us that we know about them and solve them as soon as possible."
Members of Concur's R&D Operations team can accomplish more with the ExtraHop system, which provides comprehensive visibility across all tiers of the application coupled with focused, actionable information. "The Application Inspection Triggers functionality in the ExtraHop system enables us to quickly get the targeted information that we need to make decisions and take action," says Garner. "In many cases, the alternative to using the ExtraHop system is so time-consuming and laborious as to be impractical. But ExtraHop empowers our lean staff to manage a growing infrastructure that includes thousands of databases and an expanding memcache deployment."