TCP Resets (RST): Protect against DoS Attacks & Command and Control

An attacker has compromised a host on your network. Maybe they used a phishing attack to get a user to download malware or snuck it in through a software update. They've established a command and control (C2) server and are ready to use it to send commands to that compromised host.

How do you stop them before they make their next move?

How to Block C2 Command and Control Traffic

C2 traffic helps the attacker maintain persistent communication with a compromised host. After the connection is established between the host and the C2 server, C2 traffic—containing commands, additional malware, or exfiltrated data—is exchanged between the compromised host and the attacker.

You know this is happening because you can see the C2 traffic with your network detection and response (NDR) solution.

Your NDR sees the attempted C2 connection as it happens. There's a brief window of time to stop the communication, preventing attackers from expanding their foothold or moving toward their end goal.

There are four primary methods for blocking traffic in a modern network, listed below in order of ease of implementation—and from least to most effective. Unfortunately the easiest to engineer is also the least effective.

1. TCP resets
2. Access control list (ACL) router
3. Firewalls
4. Intrusion protection systems (IPS)

What Are TCP Resets?

A TCP reset (RST) closes a connection between a sender device and recipient device, and informs the sender to create another connection and resend the traffic. TCP is a protocol that defines connections between hosts over the network at the transport layer (L4) of the network OSI model, enabling traffic between applications (talking over protocols such as HTTP or FTP) on separate devices. TCP was designed to prevent unreliable packet delivery, lost or duplicated packets, and network congestion.

A TCP reset is like a panic button that alerts the sender that something went irreversibly wrong with the packet delivery. TCP resets are also useful when a device crashes in the middle of a transmission. For example, if a laptop crashes (becoming unresponsive while sending packets to the recipient), the recipient sends a TCP RST packet to restart the disrupted connection after the laptop reboots.

A TCP reset tells the recipient to close the connection without finishing the conversation. It's a little out of character for TCP, as guaranteed delivery is one of TCP's core attributes. So if a laptop sends something to a server, the connection stays open until the server acknowledges it got the message. But if that laptop sends a reset, it's telling the server to close the connection and the laptop will never know if the message was received.

Resets are part of how TCP guarantees delivery. If communication gets somehow garbled and that laptop gets a message it doesn't understand, it can tell the server to close the connection (instead of re-sending the confusing message). The idea here is generally that the server can then start over at the beginning.

TCP Resets and Security

What Are TCP Reset Attacks?

An attacker can cause a denial of service (DoS) by flooding a device with TCP packets. In the case of TCP reset, the attacker spoofs TCP RST packets that aren't associated with real TCP connections. As the victim receives the TCP RST packets, it spends valuable resources searching in vain for the connections related to the fake TCP RST packets. As a result, the victim's processing time slows down or the victim becomes unavailable.

TCP RST and Automated Containment

TCP resets are used by some NDR products as a remediation technique for closing suspicious connections. Unfortunately, attackers can punch holes through perimeter defenses to establish connections with a victim device. While closing established connections with TCP resets—in a way unintended by the TCP protocol specifications—can work, it can also be problematic.

Network switches (designed to prevent DoS attacks) might block any TCP RST packets, considering those packets to be part of a flood attack. If malicious C2 traffic is indirectly routed to the C2 server (for example, through a proxy server), the TCP reset might close the incorrect connection. Finally, a TCP reset might only delay the C2 traffic, instead of blocking it completely. For example, the compromised host receiving the TCP RST packet will likely restart the connection with the C2 server and resume the transmission.

Attackers Can Easily Circumvent TCP Resets

Attackers can choose from many C2 techniques (such as tunneling, beaconing, or external connections) that don't rely on TCP. For example, UDP is another transport-level protocol for establishing connections. Domain name system (DNS) queries are submitted over UDP by default. TCP resets are unlikely to affect DNS tunneling, which is a technique for disguising C2 traffic. ICMP tunnels are completely immune to TCP resets, because ICMP messages can transmit payloads between devices without the requirement of an established connection.

Furthermore, the impact of TCP resets might only be temporary. Instead of closing a connection that might be restarted, firewalls permanently block connections associated with known C&C servers. ACL routers and firewalls rely on rules that block traffic to unauthorized or malicious endpoints. Intrusion detection systems (IDS) and intrusion prevention systems (IPS) are also effective at blocking connections based on malicious domains, IP address, ports, and other factors.

Integration with Tools like Firewalls and EDR

The most effective way to close malicious connections is to use network insights to trigger solutions that are actually designed for containment instead of using a less effective but convenient function of TCP. Integrating NDR with firewall and EDR tools for containment closes all the gaps discussed above, using these to their best advantage.