Network Address Translation and PAWS Dropped SYNs
When we do an ExtraHop “Show and Tell” to a prospective customer, it’s always like opening up a black box and shining a light on the extremely complex inner-workings of their networks. It’s often hard to say which key metrics will “shine” in that situation and help pinpoint key performance problems. We’ve seen RTO basking in that spotlight multiple times, as has HTTP and Database Errors. This week saw a new contender for the title: PAWS Dropped SYNs.
First, an overview of the metric. PAWS, or Protection against Wrapped Sequence Numbers is TCP’s way of protecting against old duplicate segments that may corrupt an open TCP connection. On a fast network connection, TCP sequence numbers can wrap during the transfer of a long data stream. This sometimes causes the receiving end to reset the connection and/or discard the data. With PAWS, the timestamp of the packet is also checked to make sure that even when there is a seemingly wrapped sequence number, the packet is in fact new and not a duplicate. The ExtraHop metric PAWS Dropped SYNs counts up the number of times SYNs were dropped due to the PAWS mechanism.
Now, onto the application of this metric. In this particular customer’s example, they saw some weird connection behavior between their office network and the co-located data center. The speed at which a new connection can be established deterioated over time, and sometimes failed all together. In looking at the ExtraHop system, the one number that jumped out immediately was the high PAWS Dropped SYN count. A live test confirmed the direct correlation we saw in the Dropped SYN numbers with the failing connection attempts.We found the culprit.
What happened was that the office network in question was behind a NATted firewall. NAT, or Network Address Translation made it possible for multiple IP addresses to live behind a single IP address, thus creatively solving the problem of our impending exhaustion of IPv4 address space. However, now you have multiple servers with mis-matched internal clocks sending out packets with timestamps that most likely don’t fall into sequence. Then PAWS kicks in, resulting in these TCP connections failing to initiate because the current device interpreted the SYN as belonging to a previous connection.
The solution? Increasing the connection linger time on the NAT device or decreasing connection linger time on the receiving device to help mitigate this problem.
- Helen
Tags: ipv4, nat, network address translation, network troubleshooting, paws, paws dropped syn, tcp, tcp connection problem
March 25th, 2009 at 2:03 pm
[...] “We surveyed 100 people, and the top answers are on the board.” Ok, so the Internet Society (ISOC) didn’t really say that, but they did survey their members regarding IPv6 and found that “specific business-case drivers did not yet exist” for IPv6. But what if you run out of address spaces? Aren’t you worried about that? (An event that’s widely projected to happen in 2012.) Well that’s ok, we’ll just use NATs more! (Great, leading to other possible management challenges.) [...]