Transaction Tracing for Web Gateways, Load Balancers, and Other Proxies

As with the Perl language, ExtraHop makes easy things easy and hard things possible with Application Inspection Triggers, a programmatic interface to the Context and Correlation Engine.

Larry Wall, the creator of the Perl programming language, championed the idea of "making easy things easy and hard things possible." This month's Performance Metric of the Month highlights how ExtraHop, with its programmatic interface to its parsing engine, makes hard things possible. In this case, a major web security firm used ExtraHop to pinpoint the cause of extreme latency experienced by a fraction of its users for its web gateway/proxy SaaS offering.

Statistical Averages and Performance Outliers

averagethe performance of outliers

Challenge: Stitching Together Transactions Broken Up by NATed Proxies

Context and Correlation Engine

The web security company needed to measure latency for each stage of the full transaction, but did not have control over the ingress and egress points, only the web gateway proxy.

To build a unique identifier for the request flow (1-7 in the diagram above), the team built an Application Inspection Trigger that would recognize the URI, UserAgent, and the client IP contained in the X-Forwarded-For HTTP header field. For the response flow (4-11 in the diagram above), the team built the unique identifier from the URI, proxy IP address, ETag header, Set-Cookie header, and Expires header. Together, these identifiers comprised a unique fingerprint for each transaction that did not depend on instrumentation of the application code or inserted tags. It's worth noting that this type of agentless recognize-and-trace transaction tracing is even simpler in scenarios without a proxy. In those cases, IT teams can use a existing unique identifier such as a customer ID, Object ID, or the embedded tags and session IDs inserted by JSP, PHP, and Microsoft ASP. For example, ExtraHop offers a solution bundle that recognizes and traces multi-hop web-to-database transactions for SharePoint. The point of this web gateway illustration is to show the extensibility of ExtraHop wire data analysis to handle the worst-case scenario, or to put it in terms that Larry Wall would appreciate, to make easy things easy and hard things possible.

Solving Complex Performance Problems

• What was the latency for the complete transaction (1-11 in the diagram above)?
• What was the latency for the request across the proxy (2-6 in the diagram)?
• What was the latency for retrieving content from the destination (7-8 in the diagram)?
• What was the latency for the response across the proxy (9-10 in the diagram)?

Median and percentile bands provide a good picture of overall performance, but can hide important statistical outliers

In ExtraHop, the IT Operations team could see the median latency and 25th to 75th percentile spread for each leg of the complete transaction, shown in the graph above. These averages showed that performance was good. Viewing the ExtraHop metrics in a heatmap, however, revealed 95 percentile outliers all the way to the 9-second mark on responses traversing the web gateway proxy. This indicated that the proxy itself was introducing the delay. With the help of the development team, the IT Operations team found a DNS reverse-lookup process that was timing out. Fixing this process eliminated the unusual latency (as much as 2 minutes!) experienced by some users.

Extensibility Is Important

Watch the video below to learn about the importance of investigating anomalies and outliers in your datasets. Learn more about how ExtraHop's visualizations preserve meaning when aggregating large sets of wire data.