DevOps Hearts Race While CISO Looks for Heartbleed

I was in a meeting with a Chief Information Security Officer (CISO) of a large online company when the topic of Heartbleed came up.

"How are you doing with this?" I asked him.

The question obviously got his attention given the recent announcement of the popular OpenSSL cryptographic library's vulnerability to steal protected information. He perked up, and the first thing he told us was his DevOps guys were pushing back because he was running scans in his network looking for affected versions of the software.

"Why would they be concerned?" I asked.

"They're nervous that I'm going to crash their systems doing a port scan," he replied.

Wow, I thought. Here I am sitting with the CISO of a very large company that has thousands, if not hundreds of thousands, of IP addresses that move millions of dollars around. Not only does he have to do a port scan of all his systems looking for vulnerable OpenSSL software, but his own DevOps guys are nervous that a port scan is going to bring down the systems. That's why ExtraHop exists, to help IT organizations run IT better, I thought to myself.

Read more: Detect Heartbleed Exploits with ExtraHop's Free Download

"I can't believe an internal port scan makes them nervous about system stability. Imagine what would happen if we had a real attack on our systems!" he said.

I wondered how he felt about the effort to get a handle on the scope of their vulnerability.

"So how's the scan going?" I asked.

"It's going great. We've been scanning all our systems and I expect an answer anytime now. So far, none of the systems have gone down because of the scan," he replied.

That was a silver lining, at least. But here we were, roughly 48 hours after the initial announcement from Codenomicon, and they still did not know the impact on their environment because of the limitations of their legacy security monitoring tools.

"May I show you something?" I asked him.

Although I had only met the CISO briefly, I could tell he was a very smart man. We had presented the ExtraHop solution to him some months back and I recall him instantly "getting it." As we spoke again, I could tell he already knew how wire data analytics was going to extend to information security and threat detection in his environment.

"Sure" he said with a humble but confident tone.

ExtraHop Reveals Heartbleed Exploits

Earlier that day, I had pulled up a report for our Atlas Connect service for the last 24 hours. During this timeframe, ExtraHop appliances connected to our Atlas Connect service had recorded the following SSL metrics:

• 32,000+ SSL servers under management, with 71,000+ clients connecting
• Approximately 471 million SSL sessions
• A detailed breakdown of each session by version, cipher suite, and content type
• Over 16 billion records by content type (a majority being application data with a small fraction being "Other")
• 14,601 "Other" records by content type. The "Other" records almost exclusively consist of heartbeats. (We did not include them in our recorded content types because they are an extension and not part of core TLS. View the TLS content type registry here.) Drilling down reveals which servers are sending and receiving OpenSSL heartbeats.

"Wow," he said. "I feel left out now."

We wrapped up our conversation shortly after that, but the message delivered was well received. ExtraHop is a platform that provides visibility into what's going on in the environment right now, and presents an abundance of answers to both known and unknown questions. The ExtraHop platform was neither built to look for OpenSSL vulnerabilities specifically, nor to look for heartbeats as part of the OpenSSL protocol. It simply offered an unbiased observation of the traffic moving across the wire, and presented the 14,601 "Other" records as part of its out-of-the-box SSL traffic analysis.

While we didn't dig deeper into the records any further in the meeting, the details showed that organizations connected to the ExtraHop Atlas Connect service had a handful of servers that were communicating using OpenSSL's heartbeat and potentially being exploited. For hundreds of customers, ExtraHop revealed in two mouse-clicks what had taken a single organization 48 hours to find.

Technical Notes

The Heartbleed bug affects the heartbeat records as part of the OpenSSL protocol. The payload length is not checked against the record length, which allows an attacking client to control the heartbeat size and structure to be larger than expected. Sending a malicious packet to an affected OpenSSL server would cause the server to respond with up to 64k of memory allocation outside the bounds of what the heartbeat should be able to access, thus compromising sensitive information. In theory, if repeated enough times, an attacker could get the private key from the session.

For more details on Heartbleed, refer to the following links:

• http://heartbleed.com/
• http://www.troyhunt.com/2014/04/everything-you-need-to-know-about.html
• https://twitter.com/WarrenGuy/status/453510021930680320/photo/1

Please contact us for more information on how ExtraHop can help with Heartbleed and much more in your organization.