BlogDetails in this story have been anonymized to protect the privacy of the individuals and businesses involved.
I spent a day training some very smart technologists on how to use ExtraHop. They work for a company that handles valuable media properties before they're released to the public. This is a juicy target for hackers.
At the end of a long day of training we were looking at DNS because the customer wanted to do a victory dance. They had optimized their DNS to the point where they were only getting 0.02% errors on a quarter million DNS lookups. That is shockingly good.
While we were looking at their phenomenal DNS deployment, we noticed a machine in their network that was using Google for DNS. In production, you typically do not have servers reach directly to Google for DNS.
Within ExtraHop, we pivoted from performance metrics to DNS records with a click...and immediately found a server in the customer's environment doing DNS lookups to a very suspicious looking IP address in China.
Nerd time: When you get an IP address at home, you have a temporary hold on that IP address. Your IP address can change over time. There are ways to tell the internet, "even if my IP address changes, you can find me here." This is called "dynamic DNS." The structure of the DNS query looked A LOT like a server in customer production IP space was looking up a home DSL modem in China.
That is totally, completely, not a good thing. In fact, it's what security types would call an "indicator of compromise."
Bad guys are quite fond of compromised home internet connections because the connections are fast and are an awesome place to park stolen goods in transport.
I asked what the machine that had made the weird DNS requests was supposed to be doing.
The machine's purpose was to prepare content from major entertainment companies for publication.
While this was happening, the class was taking screenshots, making notes, and opening up multi-human Slack channels. Many F-bombs were dropped in this process.
Me: "Ok, what else is this server doing?"
Click.
We could see that ExtraHop had identified that this machine was also acting as an SSH server, meaning it was receiving inbound SSH traffic.
Me: "Hmm. SSH server? It's supposed to be an SSH server?"
Customer: [delivered with a bit of relief] "Yeah, our customers send us files via SFTP."
Me: "Ok, click SSH, then click Clients. Hmm."
It looked like someone had opened up a hole in their firewall for a small batch of IP addresses to be able to send in data.
Me: "Hmm, click that 'Geomap' button."
On the ExtraHop Geomap, we could see LOTS of connections from China, some from Taiwan, and just a few connections from the U.S.
This was not an airtight confirmation that something was wrong… but it wasn't good. Another indicator of compromise.
We then looked at an overview of traffic in and out of their system by L7 protocol over the past three days.
We see a steady stream of inbound SSH traffic from IP addresses in China. Not bursty or sporadic, but a constant stream of traffic. Traffic is around 5-15 Kb/s. You'd expect an upload to happen much, much faster. This looked more like human activity.
The customer looked at the logs for the machine in question and saw many, many instances of "login failed for root."
Those errors had come to a total stop recently, which meant the bad guys either stopped trying (not likely) or got in (likely).
That was enough. The customer completely decommissioned the machine in question and kept investigating.
It turned out that rather than allowing inbound connections from a narrow band of IP addresses, somebody had configured the firewall to allow inbound traffic from anywhere on the internet. Oops.
It took me longer to write (and proofread) this story than it did for the story to play out in real life.
For real threat hunting workflows you can implement yourself, download our Threat Visibility for Cyber Hunters white paper. To learn how to detect anomalous behavior and accelerate investigations using ExtraHop Reveal(x), visit the product page.
