BlogIntroduction
One of our customers manages the worldwide distribution as well as the activation of over 1000 different kinds of prepaid gift cards cards on their platform. If you've ever purchased or used a Nordstrom, iTunes, Google Play or Starbucks gift card, you've used this customer's technology.
According to the National Retail Federation, prepaid gift cards are the #1 requested gift item for the holidays and this growing popularity means more revenue for our customer but also puts more technical demands on their backend systems to be able to handle this seasonal spike in card activations.
The Gift Card Activation Process
Most gift cards require the account number linked to a card to be activated before making the card functional for purchases. Although activation requires an extra step on the part of the gift card purchaser, this process allows our customer to run fraud prevention checks such as making sure the card is valid, and provides a chance to cancel the card if it has been reported lost or stolen.
The typical card activation process takes a few seconds but the seasonal spike of card activations around the holidays puts a heavier than usual load on the backend servers which can lead to network timeouts, conditional card approvals (without the fraud prevention steps - like most credit card processors, if it takes too long, approval happens and ALL the risk is on the processor), or the purchaser abandoning the sale. Each of those outcomes are not desirable for our customer as it leads to lost revenue and/or affects customer satisfaction.
What the Customer Wanted
Our main advocate runs the Infrastructure team and is accountable to upper management for performance, especially during Q4. It was his first full year on the job and he knew that management expected improvement: he needed to document what he was doing and how it was improving performance. He realized that he needed to get visibility into how their backend card activation process was performing as well as to identify the bottlenecks in the path from the store to those servers. In many countries the customer uses Alliance Partners that interface with the customer's system, acting as gateways for transactions from the individual vendor stores. There are specific SLAs in place for the time taken by these Alliance Partners to send activation requests to be processed, and the customer wanted to measure those times as well.
While the customer did have a monitoring solution in place to solve issues in the activation process, it had proven to be difficult to deploy and use and the customer did not feel that it was providing value in relation to its high price tag. Some of the network team used and loved this incumbent tool but our proponent has an application background and he could not get his required data from the network team or by himself. In addition a new chain of corner retail stores in Japan was about to be signed on as a vendor and the customer wanted to have a more capable monitoring tool in place to ensure that the rollout went smoothly. Enter Extrahop!
The ExtraHop Solution
The customer installed the complete ExtraHop performance monitoring solution in both their primary and backup data centers. Traffic was tapped and aggregated by using existing and new Gigamon taps and fed into ExtraHop by using multiple 10 Gbps data feed connections. One of the first configurations we did was to group the servers responsible for the gift card activation servers into a distinct Application Group so that their performance could be analyzed separately from the rest of the infrastructure. This first step gave the customer important high level insights into traffic volumes and network round trip times as well as processing times for their backend servers.
Deeper Analysis
Next the customer wanted to take a deeper look into the card activation process to identify the specific locations/stores where the requests were coming from and the number and types of successful and failed transactions from each store. A successful transaction can be a new card activation, a reload of an existing card, or a refund, while a failed activation can be a 'soft' decline due to a network issue or non-responsive server or a 'hard' decline due to an expired, lost or stolen card.
All of this information is contained in the proprietary XML-based protocol used between the stores and the customers servers. The ExtraHop services team was able to parse and decode the protocol (using a trigger) and extract the relevant pieces of 'message type' information about the status of each transaction. Custom metrics were used on dashboards to visualize the number of gift card activation successes vs. failures, soft vs. hard declines over time as well as deep dive metrics about the types of gift cards and locations of the stores where they were being activated from. Custom EXA records were also created that allowed for the searching of transactions based on location, store, and message type.
Using ExtraHop = Faster Issue Resolution
The customer was able to quickly apply these dashboards to look at card activation performance from a new vendor in Japan that was starting to sell gift cards from retail stores across the country. They could see in real time, the volume and rate of activations, successful and failed activations as well as the network round trip times from the stores to the partner gateway and from there to the customer's backend servers. They were also able to quickly identify stores that were not set up with the correct Point of Sale configurations to authorize the gift cards by correlating individual stores with the error codes displayed in ExtraHop.
This level of visibility was very useful to the customer since instead of opening a generic service ticket with the partner, they were instead able to provide specific guidance on the nature of the problems seen in the stores and how to resolve them. This has resulted in fewer service tickets while getting the issues resolved faster. Due to the overseas location of this new vendor and the time difference, having these definitive insights from ExtraHop has ensured this new rollout to go smoothly for the customer.
Additional Value
When the ExtraHop services team was on-site, the customer ran into an unexpected issue where many of their vendor SSL connections were failing because of a planned phaseout of insecure cipher suites. These vendors had either missed the prior communication from the customer about this planned change or were not able to change to the upgraded ciphers in time. ExtraHop was quickly able to show the customer the list of vendors that were failing to connect as well as the list of old ciphers that they were trying to use.
The customer temporarily enabled those ciphers as they continued to watch the progress of the connections in ExtraHop. Though this was not one of the reasons that the customer purchased ExtraHop, they were pleasantly surprised that they were able to easily use the product to solve this unexpected SSL issue.
