Today the topic is responding to incidents, which of course could be a 10 day class or longer, but we are going to try and distill it down into about 30 minutes. What I’m going to talk to you a bit about us hunters at Arc4dia approach investigations, how we use some of the tools that we have that Arc4dia built to respond to these incidents to investigate them and work with our customers to try to contain the incident.

SNOWboard is the interface to our agents that are deployed in our customer’s networks. This is where we, the hunters, spend all of our time looking at what’s going on in our customer networks, what investigations are happening, what kinds of stuff we see going on, and of course anything that looks interesting. 

Watch a video version instead

Now today what we’re going to do is focus on a particular host that has had some interesting stuff going on. I’m going to walk you through why we would determine this to be interesting of course this is a piece of malware that we custom build and deploy that actually helps us to train people and helps us to make sure we know exactly how to respond to these things. That’s also what you get with the Arc4dia team, not only defensive mind of people, but offensive mind of people. People who have deep experience working on offensive operations and looking at these things from the other side because it’s absolutely critical to understand how an attacker thinks or how they’re going to write malware and even to determine potential next steps they would take.

We see here this is our investigation list. So, every morning this is exactly what an Arc4dia hunter is going to be looking at, although much larger – this is a demo network. We are going to start looking at things that look potentially suspicious or things that look out of place.

Looking at this host here, and when clicking on this investigation is going to open up this particular screen and this screen is actually our investigation view.

An investigation is made of leads. So just when you see a detective on TV investigating a murder, that murder case is going to have leads that come together to give context for entire investigation. This is exactly how we structure SNOWboard to display information to us that we have leads that make up this investigation.

One of the cool things we have the ability to do is collaborate with one another via log. We can actually work together to write messages, say it’s time to reverse engineer this thing we found, we can leave notes for one another and our customers. This allows us to all work collaboratively as a team to investigate these incidents.

Further down into the lead list. Starting at the bottom, take a look at what we’re seeing. We have some information on the left that’s saying at this particular time on this host we had one suspicious thread that was found. This is on its own, could be potentially malicious, it could be something that actually completely benign, but the important thing is that we want to dive into the technical information. So, we are not interested in looking at fancy pie graphs or other information that doesn’t give us context. We’re interested in the raw technical data that’s going to allow us to make judgement calls based on our experience as responders and as attackers to think about what does this actually mean.

We can take a look at the information on the right hand side and say Ok, there’s a suspicious thread in vmtoolsd.exe.

VM tools is a process that is used to help make your virtual machines run a little nicer when you’re using them. This is suspicious that we see something being injected into vmtoolsd.exe.

We can see some pretty technical information here about what was injected, how big the memory was of the thread, what process it was, the threat ID, and where this thread was actually started – the base address and start address of that particular thread.

What we’re also able to do is, which is critical in an incident, is look at the context around when this thread injection happened. We can do this by clicking the fancy arrows that will bring up what’s called the event stream.

The event stream allows us to take a look at what’s going on before this event and what’s going on afterwards. You might see someone spawn Internet Explorer, then shortly thereafter there’s another process that gets executed and then you see a chain of events happening after that. This is really important because without context it’s extremely difficult to respond to an incident because you are kind of operating in the dark. This little tool here allows us to very quickly scroll through and see what’s going on. This is a very detailed view.

If we want only wanted to, for example, see all of the new processes created, around this time, we can apply a filter very easily to show us the new processes and we can filter it out and see all the new processes we have in this time range before and after. We can also expand the time range or contract it and adjust it based on how we want to investigate this particular incident.

Scrolling through, we’re building up a picture of what’s actually going on and it allows us to really become familiar on what is happening on this machine at this particular time. Often if you don’t have this information it’s very difficult to make judgement calls because sometimes you will see things that might look suspicious initially, but after you dig into them and look through things that are going on around it, it’s really not that suspicious. There’s a lot of genuinely non-invasive software out there that does sometimes behave a little suspiciously.

Scrolling through and some point we see this THREAD_INJECTION occur.

We also see some other events like a READPROCESSMEMORY, and will get into that. That’s part of SNOW’s automatic forensics capabilities that’s going to automatically a hunter or an analyst by collecting information for us.

We can see the injection happens here

We can see that we trigger a SUSPICIOUS_FILE_COLLECTION that we’ve pulled down this particular file again.

We see some HOLLOWING after that.

If you were on our previous webinar, we talked about process hollowing where there’s a piece of malware that actually hollows out a chunk of memory in a process and shoves code in it and executes it.

We can see some other stuff like a TCP_CONNECTION

We’re starting to build a picture around this time that something definitely looks suspicious, something is definitely up with this machine.

When we’re talking about a machine, we’re talking about hosts in SNOWboard. What we would also want to do in this case is say, what else can we learn about this host to help aid in this investigation.

Clicking in the box, the host view will pop up.

Any host in the network that we have the SNOW agent deployed on, were able to dive in to see what’s installed on that machine and what investigations that machine has triggered. We can also take a look at things that are more detailed like what processes we see running, unique modules (dlls or binaries that are actually executing on that machine). We can see the installed services, which can be really useful when you see something that is using a Window service for persistence.

Often we’ll also see that there are things being installed in the autoruns of that particular machine.

This is where malware will use registry keys or other techniques to gain persistence so when you reboot the machine the malware is going to get restarted so that it can continue to speak back to its command and control or it can continue to suck up keyboard keystrokes or whatever it’s designed to do.

Clicking on that view actually has got a lot of data that might not be apparent at first. The first thing we always look at is the column on the left hand side which is the number of locations that we’ve seen this particular file being executed. When we see this and we see a very low number like four out of hundreds of thousands of deployed agents, that tells us that something is definitely weird with this that we haven’t seen this very often and we’re seeing this now on this particular machine.

As you scroll down, the reputation of these binaries is actually going up. That’s exactly what are number of locations is. It’s effectively a reputational score telling you that the more times you’ve seen this thing, the more likely it’s OK or that we can easily explain where it comes from. The lower the number the worse it is for that particular host.

Immediately we’re looking at this and we’re saying, so we have this strange thing in program data. Program data is not a standard place that we see VMware installing things so this looks a little suspicious.

What we are able to do is then query our own database to see what we know about this particular process and can we learn from other hosts.

Watch a video version instead

One of the cool things we have is the ability to highlight any binaries we want, in the context menu saying search inobjects. It allows us to search this vast database we have of processes, modules, and file names. It will come back and tell us if we have some records for that particular object. We’ve only seen this process a couple of times on a couple of hosts. We can actually look specifically at the file name. This is also telling us that there’s more than this one host that may have been potentially infected or have these suspicious processes running. This is something that will tell us there’s more context here. We would probably want to make a note in this investigation to add these hosts also have this exact same processes, maybe we need to hash the files and see what’s going on.

This object database is extremely powerful because it allows us to easily drill down into what’s happening on that machine and look at exactly every binary that’s involved in this investigation, every process and say where else is it located and have we seen some of these things before.

If we were looking at this particular file we saw it was pretty rare and it wasn’t located on a lot of other machines. We can also see part of the investigation was vmtoolsd.exe as we saw in the leads. We can see it was executed in 252 different locations. This is also telling us some information that potentially this piece of malware is injecting threads directly into vmtoolsd.exe and then is attempting to hide itself as a vmware product in a different directory and in the autoruns of that machine.

What can we do from a response perspective? One of the things we can do is to reach out to that machine and say we’re a little suspicious of this, what we want to do is grab that file and make sure we have a copy of it. By highlighting the file path and then selecting the hand, this gives us a small list of commands that we can run. We have a larger list of commands that I will show you shortly.

This allows us to very simply in a couple of clicks, grab this file from this host. The investigation here is blank because I’m doing it from the host view, if we were doing this from the investigation view it automatically populates this in.

What we want this to do, is when we task this agent and bring us back this file, we want this file to automatically be assigned to the investigation we are working on so we can keep all the information in one place and so  hunters can being doing tasks and sharing the information with each other.

Click apply and that’s going to immediately send that command off to that agent. That agent is now going to work on pulling that file and sending it back to us and assigning it to that investigation. Often this is now where as hunters we’re also reverse engineers, so we’re going to take that file were going to pull it down and run it through some of the tools we’ve developed as well as traditional reverse engineer tools such as debuggers, IDA Pro, or hex editors, so we can begin to determine is this something we’ve seen before, is this one of our customer’s being explicitly targeted, is this a piece of malware designed explicitly for this customer, or is this something that’s part of larger group of ransomware family or some other piece of malware that has been repurposed so that the hash is different. This is also another value added thing you are going to get. As hunters, we’re also skilled at being able to pull apart this code and analyze it to see what’s going on behind the scenes.

We’ve done a little bit of investigation into the host, the other thing we can do is now go back and take a look a bit more information in surrounding what else is going in this particular lead that we can gain some information from. We’ve only really looked at the one lead and there’s other ones we can look at.

One cool thing that really assists hunters and our customers, is the ability for SNOW to automatically collect information when something weird has happened. Very easily we can expand the key at top, and we can see that a few things are already available for us as analysts or hunters.

We see a process memory map, this gives us a memory map of the process in question telling us who owns what chunk of memory, are there any pieces of memory that are flagged that are suspicious or look a little weird to us. We’re able to grab segments of memory with the click of a mouse. If we wanted to pull it back and analyze that memory and do a bit of forensics work on it and see what’s going on with it. This is all done automatically, we didn’t have to interact with anything, this is part of the intelligence of the SNOW system that is automatically pulling this information back for us. It’s saving us time and giving us highly contextual information. As malware executes, if you get there much later and haven’t had an opportunity to run tools or something else has impacted your ability to analyze it, having some very contextualized information at the time that there was something suspicious detected, is really important.

We see the process memory map, another thing that it did was automatically collect the little chunk of memory where the thread is executing. Down here where we see this thread executing, it’s actually reached out and grabbed that chunk of memory back so we that we can reverse engineer it. This is really critical because as you know memory is not persistent, when something writes something to disk there’s a level of persistence that you are able to retrieve. If something writes something into memory and that process exits at some point, that memory is gone and you have no ability to perform forensics on it or do any analysis. Whereas with this system, we able to get that memory right away so that we can come back much later if needed and retrieve it and still look at what’s going on in that chunk of memory. This is really important and important when dealing with incidents are able to do that.

This is critical stuff that allows us to build a much broader and detailed investigation. Working with our customers we are able to decide how do you want us to proceed. Sometimes it’s decided in advance or customers are very pro-active want to be involved.

Now that we’ve identified something weird going on, how can we actually work to stop what it’s doing for now, completely terminate it, or what do we actually have to do to get rid of this. This is going to vary depending on customer’s operating policies and is also going to depend on the particular piece of malware we’re dealing with.

What we can do is we actually have a list of very detailed commands that allows us to interact with that machine. I’ve already pulled up this list of commands – this is accessible through our dashboard by clicking on commands – and have populated the terminate thread command already with the thread ID that we identified as well as the host.

This is actually a real subtle but powerful technique. For example, if we identify that there is a suspicious thread running in a critical process, what we shown and done in the past is we can actually isolate that thread and suspend just that thread and that allows the critical process to continue executing. Sometimes malware will actually corrupt the process and it will blow up and die, other times if they’ve done it right, they will actually have the thread there sitting in memory executing and we’re able to come along and just suspend that thread and continue executing without destroying that critical process or destroying any evidence. In this case, if I wanted to completely terminate the thread, I can signal to our agent that the thread ID I’ve identified is being kind of malicious, we want to terminate that thread completely and get rid of it so that’s it’s not executing anymore.

This is actually a really cool command that we have the ability to do and do this to any process on the machine and we send that off for tasking.

Some of the other ones we demonstrated already such as getting files, we can get some information about the file. Also, if we’ve identified the vmware update file that was in the autoruns, if we identify that that’s a bad file and we want to get rid of it, what we can simply do is “delete file” and we can tell the remote agent to delete the file. That allows us to just completely remove that file off of that machine and we already have a copy of it because we reached out the machine and pulled it back and we already have a copy of it locally that we can operate off of.

In some cases, you will do this if it’s a known piece of malware. Other cases you might not want to do that, you might want to do some network isolation or some other things. This is really driven by the customer, we don’t make decisions on our own, we work very closely with our customers when there are incidents to make sure that they are as much as part of the team as anyone else.

Those are some of the file operations. Some of the traditional stuff like you can suspend a process so we can tell it to hang there while we investigate other pieces on the machine, we don’t want to kill the process, but we want to hang it and continue investigating around it and look around on the machine. We might even want to resume the malware so we can identify where it’s command and control is, or we want to completely suspend it and then decide to kill this process all together and have it completely dead and perform clean-up on that machine.

This is also one of those things that there are many different best practices and no two incidents are the same, but having strong policies helps teams approach these things in a systematic way. Sometimes we’re able to clean up a Stage 0 infection very easily all on our own with a little input from the customer. Other times there’s much more detailed things that have to happen, and we work to write reports that say all the things we’ve discovered about it, and the reverse engineering report surrounding what happened with this particular piece of malware. This varies from incident to incident, but is a very collaborative approach between Arc4dia and our customers and that’s exactly how we handle these things.

Watch a video version instead