[from IOActive] Hiding in the Noise

Greetings! I’m Corey Thuen. I spent a number of years at Idaho National Laboratory, Digital Bond, and IOActive (where we affectionately refer to ourselves as pirates, hence the sticker). At these places, my job was to find 0-day vulnerabilities on the offensive side of things.

Now, I am a founder of Gravwell, a data analytics platform for security logs, machine, and network data. It’s my background in offensive security that informs my new life on the defensive side of the house. I believe that defense involves more than looking for how threat actor XYZ operates, it requires an understanding of the environment and maximizing the primary advantage that defense has—this is your turf and no one knows it like you do. One of my favorite quotes about security comes from Dr. Eugene Spafford at Purdue (affectionately known in the cybersecurity community as “Spaf”) who said, “A system is good if it does what it’s supposed to do, and secure if it doesn’t do anything else.” We help our customers use data to make their systems good and secure, but for this post let’s talk bad guys, 0-days, and hiding in the noise.

A very important part of cybersecurity is threat actor analysis and IOC generation. Security practitioners benefit from knowledge about how adversary groups and their toolkits behave. Having an IOC for a given vulnerability is a strong signal, useful for threat detection and mitigation. But what about for one-off actors or 0-day vulnerabilities? How do we sort through the million trillion events coming out of our systems to find and mitigate attacks?

IOActive has a lot of great folks in the pirate crew, but at this point I want to highlight a pre-pandemic talk from Jason Larsen about actor inflation. His thesis and discussion are both interesting and hilarious; the talk is certainly worth a watch. But to tl;dr it for you, APT groups are not the only attackers interested in, or capable of, compromising your systems. Not by a long shot. 

When I was at IOActive (also Digital Bond and Idaho National Laboratory), it was my job as a vulnerability researcher to find 0-days and provide detailed technical write-ups for clients so vulnerabilities could be discovered and remediated. It’s this work that gives me a little different perspective when it comes to event collection and correlation for security purposes. I have a great appreciation for weak signals. As an attacker, I want my signals to the defenders to be as weak as possible. Ideally, they blend into the noise of the target environment. As a defender, I want to filter out the noise and increase the fidelity of attacker signals.

Hiding in the Data: Weak Signals

What is a weak signal? Let’s talk about an example vulnerability in some ICS equipment. Exploiting this particular vulnerability required sending a series of payloads to the equipment until exploitation was possible. Actually exploiting the vulnerability did not cause the device to crash (which often happens with ICS gear), nor did it cause any other obvious functionality issues. However, the equipment would terminate the network communication with an RST packet after each exploit attempt. Thus, one might say an IOC would be an “unusual number of RST packets.” 

Now, any of you readers who have actually been in a SOC are probably painfully aware of the problems that occur when you treat weak signals as strong signals. Computers do weird shit sometimes. Throw in users and weird shit happens a lot of the time. If you were to set up alerts on RST packet indicators, you would quickly be inundated; that alert is getting switched off immediately. This is one area where AI/ML can actually be pretty helpful, but that’s a topic for another post.

The fact that a given cyber-physical asset has an increased number of RST packets is a weak signal. Monitoring RST packet frequency itself is not that helpful and, if managed poorly, can actually cause decreased visibility. This brings us to the meat of this post: multiple disparate weak signals can be fused into a strong signal.

Let’s add in a fictitious attacker who has a Metasploit module to exploit the vulnerability I just described. She also has the desire to participate in some DDoS, because she doesn’t actually realize that this piece of equipment is managing really expensive and critical industrial processes (such a stretch, I know, but let’s exercise our imaginations). Once exploited, the device attempts to communicate with an IP address to which it has never communicated previously—another weak signal. Network whitelisting can be effective in certain environments, but an alert every time a whitelist is violated is going to be way way too many alerts. You should still collect them for retrospective analysis (get yourself an analytics platform that doesn’t charge for every single event you put in), but every network whitelist change isn’t going to warrant action by a human.

As a final step in post-exploitation, the compromised device initiates multiple network sockets to a slack server, above the “normal” threshold for connection counts coming out of this device on a given day. Another weak signal.

So what has the attacker given a defender? We have an increased RST packet count, a post-exploitation download from a new IP address, and then a large uptick in outgoing connections. Unless these IP addresses trigger on some threat list, they could easily slide by as normal network activity hidden in the noise.

An analyst who has visibility into NetFlow and packet data can piece these weak signals together into a strong signal that actually warrants getting a human involved; this is now a threat hunt. This type of detection can be automated directly in an analytics platform or conducted using an advanced IDS that doesn’t rely exclusively on IOCs. When it comes to cybersecurity, the onus is on the defender to fuse these weak signals in their environment. Out-of-the-box security solutions are going to fail in this department because, by nature, they are built for very specific situations. No two organizations have exactly the same network architecture or exactly the same vendor choices for VPNs, endpoints, collaboration software, etc. The ability to fuse disparate data together to create meaningful decisions for a given organization is crucial to successful defense and successful operations...