Welcome to our webinar. Today we’re going to talk about looking for threats through statistical outliers and statistical analysis. So a few housekeeping items off the top as some of you might have heard already.  Number one, if you lose your connection or you need to step away from this webinar, it will be recorded and you’re going to receive a copy of that recording at the end. Number two, you’re all on mute so don’t worry you’re not going to make any accidental mistakes and talk into your microphone so that the whole world hears it, but that also means that you can use that question and answers box or the chat box.

Watch a video version instead

I prefer question-and-answer box if you need to ask anything or if you’re having trouble hearing or anything and I can address that, so please feel free to use that both throughout the webinar and towards the end when we get more into the questions and answers. We are only going to run this for about 15 to 30 minutes usually again depending on how much coffee I’ve had this morning really depends on how long the webinars will run and more than anything we really want to thank you for taking the time to come today. We know you’re all busy people you all have things to do so we really appreciate your time and we certainly intend to make the most out of it.

As I mentioned earlier we’re going to be talking about doing some statistical analysis using the SNOW platform from Arc4dia. Now how we do this is actually through this interface that you all should be seeing now which is called SNOWboard. SNOWboard is our window into all of the data that our SNOW agents are collecting every minute of every day across our customer networks and we even monitor our own networks and of course we monitor our test network that you’re going to see today.

For those of you who are new I’m going to walk you through a little bit about how SNOWboard works and kind of how the data comes in and how we look at it because that’s important but also going to show you how we move into more statistical analysis and how we can use it as well to look at the network at a higher level and also look at how infections might be spreading.

Hopefully you all can see my screen at this point. If not, we have major technical problems but you should be seeing is this kind of list of investigations and this is exactly what myself and the other hunters on Arc4dia do every single day as well as some of our customers is we get into SNOWboard and we take a look at what’s going on on our customer networks and each one of these is what’s called an investigation that’s pretty easy to understand.

If we take a look at just picking one here, what this investigation gets triggered by something that our agent has identified that looks suspicious. So this could be any number of things. In this case there’s process hollowing, which we covered in a previous webinar encourage you to go back and watch that. We also see some suspicious threads.

This investigation is going to embody a lot of information and it also includes a lot of tools that are useful for both hunting as an individual and hunting as a team. One of the things will often do is we’ll use the logging mechanism where we can actually say “Hey this is Justin, you know I pulled down a binary for analysis and Ida”. That’s going to tell my other co-workers “Hey, you know there’s no point be pulling on this binary, Justin is taking a look at it i’ll put my notes back in and we can have a conversation about what’s going on in this particular investigation.

This is also useful so that customers can actually collaborate with us so they can see what we’re doing maybe even provide input. Lots of our customers are actually really technical people who understand a lot of what we’re talking about. They might also be seeing things on other hosts that they’ll pop in and say “Hey, you know, are you guys noticing this, or I saw this on another host, or I saw something else they’re all alert. This is really useful.

The other thing that’s really powerful when working in this environment, is the notion of clustering. We have the ability as a team to look at a particular investigation and look at some of the data points which we’re going to get into and say well how can we actually save ourselves some time in the future by writing these powerful clustering rules that will automatically say okay if we see this particular process at this particular time on this host in the future we want to pull that into this investigation so that this investigation can kind of become the main point for investigating similar pieces of information. This is really powerful, this is so that we’re not duplicating work continually investigating the same things over and over. This is exactly what this clustering tab is for and again we’re probably going to do a webinar just on clustering at some point in the future.

The commands panel here you’re gonna see a bit of it in action later but we have the ability to speak to any one of our agents in the network and get them to perform tasks for us. This could be anything from retrieving a file, to retrieving memories, or performing other forensic operations and this is really powerful because it allows us to kind of expand our intelligence collecting and say okay if we have something going on this host we want to pull back some evidence, we want to collect some things. We also have the ability to trigger some of that stuff automatically, which you’re going to see very shortly.

The next thing down is the leads list. Just like if a police officer was investigating a homicide, you have an investigation which is the actual murder itself and then you have the leads that are coming in which are all the pieces of information and people that contribute to that investigation. As much the same thing and SNOWboard, that is kind of logically how we think is humans and we’ve applied that here. We say, okay we have these events that have happened around this time on this host, let’s pull them all into this investigation so we have a deeper view of what’s going on on this particular machine around this time.

We have the ability to look at these leads individually and we can see that each one of them has slightly different information. We see process following on the first three or four leads and then we see a suspicious thread found at the bottom.  By clicking on either one of these, what were able to do is actually see highly detailed technical information about what is going on and why that lead was included in the investigation. What this means is that we have looked at “okay what is going on here and what is relevant” and so we also have the ability then to capture this information and begin to say well what else can we look at around this time or what else can we look at from other tools.

If we look at the top here we actually have the ability to also do some automated forensic collection. This means that when this particular lead was brought in it automatically triggered some follow-up friends of work to happen and this means reading some process memory to extract this particular thread that was injected. So we’ve actually extracted the code of memory so that we can now do analysis on it.

We’ve also triggered a process memory map, which is gonna tell us here is what the memory layout look like for this process at the time that this was triggered. This is really useful for us because it gives us that information, and as you know malware has the ability to clear memory or kill itself and do things very quickly and so we want to collect as much evidence and as much intelligence as we can, as fast as we can, so we have all the information we need in order to do some analytical work because once it’s destroyed it’s destroyed, and so that is really key. All of this is rolled into this lead.

One of the other powerful things we have the ability to do is say well if we have this particular event that’s occurred now, what’s happened around the time of this event. By clicking on these little arrows right here we actually pull up a really neat thing called the event stream. This is all of the events that agent has collected around the time that that particular lead came in.

You can see that we’ve triggered some there’s some TCP connections, we’ve seen the thread injection here. We see some more TCP connections. We’re also looking at what is the behavior both before and after this lead has come in. That’s gonna tell us potentially the infection vector so, are we seeing Internet Explorer being run and then we’re seeing some Russian malware domain being accessed and then we see all of a sudden now we have a thread injection and a bunch of suspicious stuff happening after that. It helps us to work backwards to figure out “okay where did this happen, what occurred, are we seeing a known vulnerability that’s being used one against one of our customers or did they download a binary that they shouldn’t have?” That really helps us to figure this stuff out and this is a really powerful and useful feature again we could probably do an entire webinar just on the event stream but we live and die going through our investigations looking at this stream to figure out what’s going on around the time of that infection or the time of that suspicious behavior.

This is kind of the main investigation view, again this will give you kind of the basis for where a lot of our investigations start, but today of course, we are looking at how do we look at the network or look at host at a larger level and say what is a statistical outlier or can we look at things that are suspicious just based on the fact that their reputation count is low, meaning we haven’t seen them around a lot, we haven’t seen a particular binary get executed a lot around the network.

How we do that is through the host view and I already have a host few loaded up here and so this is an actual host view for any one of our agents.

We have the ability to pull this information up for any one of our agents across the entire network. This is really useful because when we’re looking at infections or potential outbreaks, we also going to stand back and look at what are they targeting a particular windows version. In this case it’s a windows 7 machine, are they targeting a particular subnet inside of the network, is there a way that it can be contained that way. This host information although granular, meaning we can drive down into one host, it also allows us to kind of look at that 30,000 foot view and say are we seeing commonalities or patterns or are we seeing an exploit being used against only a certain version of windows or a certain set of users and this is really useful.

The other thing to note, is that our customers actually have the ability, when our agents are deployed to segments networks based on utility. So inside of our SNOW system, we have the ability to say anything in this particular subnet is actually the executive team, so it’s the CEO, the CFO, the COO and we can segment those away from the development teams, away from the servers, and so that also allows us to look at what’s going on in a particular subnet, what’s going on with a particular group of people, and those logical divisions are really important when you’re looking at kind of higher-level statistical analysis of threats.

If we start to take a look at what’s available inside the host view, one of the cool things is we have all the latest events from that host. It is literally a running tally, a running audit trail. Here’s all the things that’s been going on in that particular host, starting with the newest stuff at the top, going oldest down. We can see all the events and that means you know unique DNS requests, we can see that we ran some commands, or we can see a code identity event, which means that our agent is actually looked at a binary and extracted its code signing certificate to take a look at it. This is really useful for us because we can see kind of a high level view of what’s going on that particular machine.

We can look at the event stream again, which is what we looked at more specifically to an investigation. We can do the same for a host, or we can begin to drive into the host objects. Inside of the SNOW platform, one of the coolest features is the fact we have this object database. It’s this big database of all the processes, the files, everything we see across an entire network, all gets rolled into this database so that we can actually do analytical work on it and that also means so as you’ll see later, we can query for very specific certain pieces of information so that we can dig into them. This object database is extremely useful and again is something that’s right hunters find incredibly useful when they’re digging into a particular process and they want to say you know where else have we seen this or they want to cross-reference a particular binary against virustotal but you’re also going to see that some of the statistical analysis here is already done for us.

Watch a video version instead

You can already see this very handy outliers link. I have loaded it up in another tab and we’re going to click on that. What the outliers is, if you never heard the term before is basically anything that is statistically outside of the normal. If you’ve ever looked at those graphs where there’s a whole bunch of little thoughts following a line on a graph, and then there’s that one dot up all lonely to the left in the top left of a graph, that’s an outlier, that’s a typical out like Google for a graph outlier that’s typically what you’ll see. It’s stuff that doesn’t fit the normal baseline which is of course where a lot of us live and die in the security world. We want to look at what is the baseline, what are things that we see all the time, and what are things that don’t fit within that baseline.

The beautiful thing about SNOWboard is actually boils down into something that’s really easy for you to look at and understand. We can look at something as simple as the number of locations that we’ve seen that particular binary or dll and say, hey if we have a thousand machines that are all be actively monitor in this particular subnet and we only ever see this file once on all of those machines, we probably want to go take a look at it because this is statistically unusual.

How we know that is we can actually reverse sort this and begin to take a look at what are things that are common. We can see common DLLs that are being used, we can see common processes the further we scroll down so things like svchost which is a critical windows process that runs on every machine, we can see hey there’s you know over 1,300 examples of this particular process running on the subnet that’s not that suspicious. This is really critical and this is often where we will start when we’re looking at what’s going on a host from an outliers perspective. Is anything that’s got a low number of locations in it, is a potential hit for a piece of malware or other threat that could be analyzed.

We’re looking at things like, you know while the chrome updater, this was only executed twice could it be for example that this is just a brand-new version of Chrome that’s being rolled out well.Very easily have the ability to query the database as you will see very shortly to actually look and say well what is this binary, where we’ve seen it before and what’s relevant about it. This is one of the very first things in the fundamental things we do when we’re looking at statistical analysis is the outliers and this is really critical.

What else can we actually look at? Even if we identify that one of these things look suspicious or hey this low number of execution counts. For example, we see malware demo windows, if you ever see malware demo windows running on one of your machines you typically want to go look at it, unless you’re doing one of these webinars. What else can we take a look at? Another thing we can look at are the autoruns. Anything that is going to do some form of persistence is usually most commonly going to target the autoruns keys in the registry and say okay every time this machine boots up or every time this user logs in, we want to make sure that our malware continues to run, that’s really critical.

When we’re looking at autoruns again, we see scrolling down we see things that are really common, we see things that are starting up all the time, and then looking at this particular host we see that there’s some things that look pretty strange. When there’s only for executions out of a network of a thousand-plus machines, that’s pretty strange to see something in the autoruns that is only available on four machines.  Again, if you have some logical separation your network, maybe it’s not that abnormal if you only have four machines because then it wouldn’t mean that that statistically irrelevant, it’s a hundred percent of them. In this case we have over a thousand machines, it is absolutely something that stands out.

What can we actually do to say okay what is this thing and what’s going on? The cool thing is we can just highlight this path and you’ll see this little fly out come up down here, we can click on the bullseye, and say search and objects.

I’ll close our little thing here. When this is done is actually query the database and said okay we want to know more about this particular path.

We actually have a couple of things that have come back from the database. One is a file path, so it’s saying hey we recognize this file path from somewhere. We can look at it and say okay there was four locations, interesting, we don’t have any associated parents of children for this particular file, but then we can actually see the hosts where this particular path showed up. This immediately is going to for us hunters, we’re gonna look at this and say uh oh, there’s four other machines here as it showed the location, we should probably include these roll these into the investigation because they too may have some suspicious behavior going on or they might already be in our investigation list we want to roll the stuff all together into one larger investigation. This might mean that we have a bit of an outbreak on our hands, or again this was for executives could also mean that there’s a spearfishing campaign that has been successful against all four executives and we want to go take a look at it. This is really useful that were able to query this database, take a look at it and say, well what is this thing, where we finding it and that also gives us the ability to go and take a look at what else we potentially need to do to contain the threat.

Same thing with autoruns, being able to click on that object in the database, look at the autoruns information and say, okay where else is it the auto runs?  We see it’s the same for hosts, so not only are we seeing the same file path showing up, we’re also seeing that it’s also persisted, so it’s also inserted itself into the auto runs on these four hosts. We probably want to go take a look at those four machines double hard now because this is really really suspicious. Once we’ve reached this kind of consensus that something very strange is going on, how do we actually begin to react to this? What do we actually do for next steps?

As I mentioned before one of the great things that we can actually highlight this path and we can use it and just say, okay we want to go and retrieve this file just by using our commands box down here, I’m not going to do that today. We reach out and can grab that file back, we can meet of course, depending on the customer, we can begin to work with them to say. hey I think there’s an incident here let’s start talking about implementing your incident response plan, assuming you have one, hopefully everyone listening here today has one, or a number of other things. We can begin reverse-engineering, taking a look at that particular file, we can also begin to look at other things that are going on that host.

This is where we will also look at okay well what have we seen any weird domains coming up in the last while or at all time that looks suspicious or don’t seem to fit, yet there’s some weird stuff we’re seeing here what is going on? Often will also see stuff like people trying to access tor onion, like hidden services through a regular web browser and were often alerted like that’s really weird what is this, what is this person up, when they’re it’s nine o’clock in the morning at their day job. Stuff like that that were able to actually look at and say, well what other things are actually contributing to the overall kind of threat or the overall reason why we would raise kind of the alert level for this particular organization or these sets of hosts.

This is the great thing is that we can dive down into any one of these things on a host, begin to do some statistical analysis on it and then bring that back into more of the investigative viewpoint, so we can always start from an investigative point and work our way up. or we can start from a statistical standpoint and work our way down to an actual investigation. Often what we’ll find is that customers tell us, you know here are critical machines or they might even put them into that logical subnet and we’re able to say, okay you know let’s go out and just start looking at outliers and this is something myself and other hunters on the team will do and we’ll look at it and say, okay well here’s some interesting outliers on these machines, you know what’s interesting or not? We’ll hear this is a weird powershell execution that we don’t see anywhere else, let’s take a look at powershell script and see what it is, ok it’s a homegrown system men you know who’s a written a thing to kind of purge a mail queue or whatever it is. We have the ability to immediately look at those things and start that investigation from 30,000 feet and begin to drive downwards into okay how do we actually acquire some evidence here begin some reverse engineering or looking at you know what other what else is going on that machine.

That’s kind of the beauty of having this data at your fingertips is that you can work from both ends something has been triggered an investigation, work your way up to let’s look at the statistical analysis of this to see if it’s spread or to see if there’s other things that are causing it. You can also start from the other side and work down towards opening an investigation begin rolling some leads together to figure out what’s actually going on.

Watch a video version instead