Houston, we’ve lost contact

The first step when trying to figure out why contact was lost is to examine all possibilities. Here are the likely scenarios:

1. The agent process is attacked:
   1. it is killed.
   2. it is suspended.
2. The agent process crashed.
3. The machine shut down
   1. The machine was properly turned off (using OS-specific tools).
   2. The machine has had software, hardware or power failure that brought it down.
   3. The machine was virtual, and was interrupted.
4. Network failure…
   1. … affecting only this one host or subnet.
   2. … affecting the whole organization being monitored, or the Internet route between the organization and the remote analytics database where events are monitored from.

A few simple measures can help distinguish between most of these scenarios. Let’s start from the bottom of the list and go up.

Communication failures

Scenario 4b is easy to distinguish from all others — connectivity is thence lost for all other machines within the same organization. A phone call to the IT staff is in order, but sometimes the Internet is just that tough to deal with. So, it comes down to diagnosing a single machine going incommunicado.

The way Arcadia’s endpoint agents will handle this case is by having them in contact with one another across an ad hoc peer-to-peer network deployed on the customer’s network. In other words, each SNOW-defended endpoint within a LAN maintains a TCP connection with a subgroup of others (the size of this subgroup depending on the machine’s purpose and resource constraints). Should any of these TCP connections fail, its peers know it instantly and can report that fact back to the central analytics cloud. This set of peer-to-peer communication features are under development as this is written.

In addition, the agent is aware of its host’s connectivity problems. When the machine regains connectivity, it reports on how long it was down, so that we can corroborate this information with what was reported by its peers. This way, if an agent disablement is being disguised by the attacker as a network failure, there will be a gap in the offline time interval during which the agent was not alive. This is enough information to warrant deeper a investigation, hence to raise an alert.

Machine failures

Obviously, machine failures get detected as communications failures. However, many such failure modes can be documented using supplemental clues. With scenario 3a, the agent gets a message that the machine is being shut down, and is typically given time to stop of its own volition. It can take advantage of that moment to report a final telemetry beacon indicating the situation.

In scenario 3b, the agent is forcibly restarted when the machine goes down. Additionally, the uptime counter for the machine gets reset. Therefore, the restart of the agent reports the uptime counter: if it’s low enough, the diagnostic is complete. Alternatively, virtual machines (scenario 3c) being interrupted look like network failures, until they are resumed. When this happens, there is a gap in the telemetry stream that corresponds to the duration of the interruption. So when the agent beacons again, without any indication to being restarted and with no telemetry buffered within the realtime interval, we understand the machine was virtual and was interrupted. Further indications that the machine is virtual (such as the presence of VMware Tools or access to AWS local queries) confirm that the machine is virtual, further supporting the diagnostic.

Agent crashes and attack scenarios

The SNOW endpoint agent is programmed by mere humans, so yes, certain situations get it to crash. There! I said it. Two measures facilitate the detection and remediation of crashes.

First, agent processes are registered with the operating system so as to be restarted as soon as they go down. This way, following a crash, the restarted agent reports immediately to the central analytics cloud, carrying the forensics information it accumulated before the crash event.

Second, the endpoint agent is actually composed of two processes (both registered to the OS, as described above) tracking each other’s life cycle. If any of these two processes goes down, the other reports the fact over to the cloud, and repeatedly attempts to bring it back up. Therefore, effectively disabling the agent requires to bring down these two processes. For any one of these processes to crash is odd, but possible; for both of them to crash simultaneously is highly unlikely, enough to raise an alert regarding the possibility of deliberate termination.

Attack scenarios

The last point makes it clear that scenario 1a cannot go undetected: either the agent is restarted right away by the OS or its watchdog, so that either the malware actions get recorded despite the attacker’s intent, or the attacker generates so much noise that he raises the attention of hunters. Scenario 1b still is much more pernicious, as its fingerprint is very similar to that of scenario 3c. That said, some heuristics can work in our favor:

• Non-virtual machines cannot be interrupted and resumed (a machine being put to sleep sends a message when it goes down and when it wakes up).
• When a VM is interrupted, it is typically for a rather long period of time. If the attacker suspends the agent processes during mere seconds, this should raise an alert as an abnormal VM interruption pattern.
• Peers within the local network, as discussed for mitigating communications failures, send ping-pongs at randomized regular intervals, and keep statistics of how quick the response comes back. A suspended agent process would not respond within the expected sub-second delay, unless the local network is extremely busy (an atypical condition).

A failure to check any of these heuristics should raise an alert to conduct a full investigation on the target host.

Authentication of agents

The last frontier in agent disablement attack is full agent communications spoofing: an attacker reverses the communications protocol between the agent, peers and central analytics cloud, then responds correctly to neighbor requests and plays dumb with the cloud. This is not an easy attack to pull off, as it takes lots of resources to perform this reverse engineering, and then communicate without tripping any behavior normality heuristic. However, it underscores a fundamental weakness of endpoint protection: up to now, it has been assumed that agents would communicate without any external injection of trust. Therefore, agents can authenticate the cloud, but agents cannot authenticate each other, nor can the cloud authenticate an agent. In other words, the cloud is never sure it is really speaking to an agent, or to a dog. We Arcadians are hard at work on this problem as this is written… stay tuned.

What is process hollowing?

Process hollowing is a trick for malware to hide its running computation. In a nutshell, it involves the replacement of the code of a process with that of the malware. Consider its implementation on the Windows OS, where Arc4dia’s hunters typically observe it. A bootstrap program will first start a process in the suspended state. The OS sets up the process structure and loads the program’s code. Then, the bootstrap uses debugging API functions in order to open the suspended process and replace the content of one of its code segments with its malware payload. It then tweaks the hollowed process’ entry point to start through the malware payload. The latter may then fork a thread to run the malware computation, leaving the main program’s thread to run its usual entry point, or replace that main function outright. When done against a little-used system process, this trick can very effectively hide a malware computation.

So, how does a hunter detect hollowed-out processes? On Windows, program code is loaded from files stored on disk; it is difficult to modify or destroy these files while the program runs. Hence, one can scan through the code segments of all processes, comparing the loaded code to that expected to have been loaded from file. Because of dynamic relocation, the code in memory never matches exactly its stored origin, but strong positive correlation is always expected — except when the code in memory has been replaced.

Pitfalls of process hollowing detection

The astute reader will have noticed that the approach described above actually detects modified processes, not specifically hollowed-out processes. The latter is a subset of the former, and program self-modification is unfortunately a very common phenomenon. Many legitimate programs leverage so-called packer systems, such as BoxedApp. Some of these packers decrypt and/or uncompress the actual program code within the same code segment. Such approaches help reduce the programs’ storage footprint, or to conceal their functionality from reverse engineers (who are not fooled by the scheme, by the way). Many online update or self-updating programs exhibit similar code replacement behavior. And all of them trip the detection technique described above.

Therefore, any attempt at detecting process hollowing is bound to generate multiple false positives. As suggested, process code modification is infrequent enough to take interest when it happens, but also common enough not to be a surefire malware identifier. Process hollowing alerts must thus be analyzed **in context** of other events reported from and other alerts raised by the concerned host. Some context elements are of particular interest:

1. Malware that runs from a hollowed-out process requires a bootstrap program to deploy their code. This program is likely to be the one that provides persistence to the malware. Therefore, taking a close look at programs that start out at machine boot or with user sessions may raise concerns.
2. Furthermore, the malware may leverage process hollowing on its own bootstrap in order to hamper a reverse engineer’s efforts to unravel its features. Taking a look at the original code that has been replaced may yield surprises. Typical packers have a signature that reverse engineers are good at identifying.
3. One may assume that malware will either execute from ever-running programs (such as a user’s base Explorer process, or as a service) or from programs run very often by the system or user (web browser, word processor for office machines, etc.). Process hollowing detected on an infrequent program, or on a program with a self-update feature, is likely to be a false positive.
4. Malware does not leverage process hollowing just for show: this code should be run by a thread within the process. Further detection heuristics that exhibit thread injection should trip, as possibly MZ-headered modules unaccounted for in the process’ module index.

Case study #1: SC2_x64.exe

Let us first consider a process hollowing alert generated by looking at a process loaded from the SC2_x64.exe program (namely, the main executable for the popular video game StarCraft II):

At first glance, this looks rather good. Video games often download updated code from the Internet: that it would replace the code within the same process is quite 1337 and surprising, but not worrisome. Furthermore, the tactical interest of hollowing a video game for malware purposes is low, unless the user plays with regularity and dedication — features that this games elicits, by the way, so it may become interesting to ask the user about his habits. We dig in SNOW’s object database to see how often we’ve see this module:

So we’ve seen this executable only on this host, which is perplexing. However, a quick check of its hash on VirusTotal shows that this is the real StarCraft II client, a well-known signed executable. Therefore, it is very unlikely to be a self-hollowing bootstrap; the hypothesis of a self-modifying game code is reinforced. We then look at the list of autoruns (programs made to run when the host boots up), which reveals nothing out of the ordinary. Finally, we look at other alerts raised for the host under investigation:

The process hollowing alert was raised on December 27th; the previous alert for this box goes back two days before (simply an object never seen before on other systems), and nothing more has happened since (at the time of writing). There is thus no context clue for worrying about an attack. Conclusion: false positive.

Case study #2: vmtoolsd.exe

Let us now look in depth at how the malware examined in Justin Seitz’s webinar stacks up. The initial process hollowing alert shows up like this:

The target here is a VMware Tools daemon used by the VMware hypervisor for various data sharing features between the host and the virtual machine (e.g. clipboard, automatic resolution adjustment, etc.). As we are investigating a virtual machine, this makes sense. This daemon runs all the time and boots with the machine. Thus, from a tactical point of view, if would be a very good place for malware to hide. Like we did with the Starcraft II case, let us check that this vmtoolsd.exe program is legit.
This time, we have seen many avatars of file vmtoolsd.exe across all the networks we defend. The hash observed on the host under investigation is this one:

It checks out good on VirusTotal, so if there is a bootstrap, it is elsewhere.

At this stage, I start to look out for further context on this alert. Other alerts generated on this host:

The initial process hollowing alert corresponds to the next to bottom-most entry in this list. So we see that over the following few hours, the process hollowing alert repeated, side by side with thread injection and hidden module alerts targetting the same vmtoolsd.exe process. This raises the alert to orange level — an execution thread has its entry point within the modified code segment.

The hidden module alert:

The base address (0x7ff693880000) indicated for this hidden module (e.g. identified MZ-headered code segment unaccounted for within the process’ list of modules) is exactly the same as that of the hollowed-out code segment. The alert level has now risen to scarlet. Hunters would team up to get a dump of this segment and start reverse engineering it, while somebody else looks for the persistent bootstrap. The first place to look is the autoruns list, for which we see here the beginning:

It takes close examination, but a good look at entry 3 shows something fishy: the vmware_update.exe made to run is in C:\ProgramData. This directory is normally hidden to the Explorer on most computers, so this is a pretty good place to put files better concealed… A look at this file’s hash shows it is unknown by VirusTotal, which is very uncommon for files related to VMware. This is a likely candidate for the bootstrap! Reverse engineering of this file (obtained through SNOW) and of the dumped module quickly show definitive malware features. Conclusion: very malware.

Summary

So, process hollowing detection is a nice feat of live memory forensics, but it has its caveats. The detection test actually identifies process code modification, which is a phenomenon too common to systematically label such events as malware. However, taken in proper context, process hollowing alerts do facilitate and accelerate the detection of an ongoing attack, enabling a quicker response and a more effective defense.

Obviously, sorting out all the false positives for process hollowing detection remains tedious. However, automation planned for implementation in SNOW can eliminate many common false positive cases. In particular, packed executables (e.g. executables set up using packer software, as described above) are easy to identify using tools such as PEiD. Automating this analysis, as well as other heuristics for flagging classes of legitimate process code modifications, should help the SNOW platform to scale better.

Thank you to Justin Seitz for his advice on contextual analysis and Marc Théberge for info on packers.

For the majority of startups, small and medium businesses, the risks mostly relate to the phones and computers. Being in this situation myself with a background in IT Security, I thought I would share the easy low costs tips and tricks of securing your IT devices.

Before taking action, we must first understand how we are exposed to hackers and thieves who could access our computers.

To make it easy, let’s divide the attack surface in 3 categories easy to remember:

• The targeted malware, coming after you
• The physical hardware loss or theft
• The commodity malware, attacking the masses

First, let’s clear the targeted malware.

The chance of being hacked directly early on in the business is unlikely. However, targeted malware vector of attack grows as your business matures and starts disturbing competitor market.  The main reasons for this attack vector are:

• Cyber Criminals see you as profitable target.

Example:

A group of criminal could be targeting the finance department to gather all the information necessary to wire money out of your accounts. Many medium size business fall victim of such attacks and suffer important losses hurting their cash flow.

• Cyber Criminals are funded by your competitors to get information or to sabotage your business.
• Your competitors might also get state sponsored hacking support in parallel without them even knowing about it. Corporate espionage is much more popular than most people expect.

Example:

A popular attack from competitors is to get copies of quotes from your business to under bid you. Many CEOs and Co-founders come to me and explain me how they were targeted and hacked. Some are at risks because of international contract negotiation. When business owners share their story with us, it’s typically because they lost a contract unexpectedly and wondered what happened, looking for internal traitors, when in fact, there were none, they were simply hacked. Their privacy had been breached.

Second, let’s discuss commodity malware.

Lately, commodity malware such as ransomware has picked up momentum. As you may already know, ransomware encrypts all existing files on your device and asks you for bitcoins in exchange to decrypt these files. This type of attack is targeting the masses to infect as many hosts as possible. An attack such as this significantly impacts business operations and adds financial implications in order to recover encrypted files. The rising numbers of ransomware attacks motivate businesses to harden their IT security posture.  Even large enterprises are suffering from ransomware attacks.

So, what is the simple way to improve your IT security posture? Backups!

• Test your backups
• Keep more than one backup
• Don’t forget to encrypt your backups

Another popular commodity malware is used to build and control botnets. Botnets are swarm of infected computers or internet connected devices. They are used to perform many tasks. The most popular: Distributed Denial of Service attacks (DDoS). These type of commodity attacks are constantly raising in size and capacity. The most recent attack on Dyn is certainly a proof of the potential damage that can be caused by DDoS.

Let’s cover the loss or theft of devices.

As more and more of our information is centralizing to our devices such as our phones, computers, the loss of them can create irreparable loss of personal, business data. Some IT security services offer features to wipe out lost devices remotely. My advice would be not to rely on such features.  From all devices that have been lost or stolen by Arc4dia’s staff and clients, none have ever connected to the Internet that would have allowed to send the remote wipe (erase all information).

In my opinion, the only practical protection here is to have a good password and full disk (device) encryption. With the latter, you can expect that your device will have to be reset for usability, which will erase your documents, before it ends up being sold on the black market.

To sum up the protection against theft, good password + full disk (device) encryption.

Note

On an iPhone, the disk encryption is always on. Simply put a strong pin or password combined with the wipe on too many failures option.

On Android, in most cases, you have to turn on device encryption in the settings. It also lacks the wipe on too many failures feature. You will need to install an App called Locker for this or equivalent.

For the Mac, the disk encryption is easily turned on in the security settings.

https://support.apple.com/en-us/HT204837

On a Windows computer, the disk encryption can be turned on in the security settings. However, it does require the Pro or higher version to enable Bitlocker. 

https://support.microsoft.com/en-us/instantanswers/e7d75dd2-29c2-16ac-f03d-20cfdf54202f/turn-on-device-encryption

There a few basic tactics that can be considered and implemented with minimal or no additional cost.

• Be different and avoid software used by the masses
• Be disciplined and aware

What does this mean in practice? It depends on your situation, but there are many generic choices applicable to most of us.

Start with avoiding the top 4 cyberattack vectors:

• Flash
• PDF
• Microsoft Office Documents
• Javascript

By avoiding or limiting these 4 cyberattack vectors, you will most likely prevent 99.9% of attacks against startup, small and even medium size businesses or personally.

Example:

Amazon asked to remove any Antivirus software installed from every employee’s computer!  Why? Recently, Antivirus engines have been easier to hack then the systems themselves, putting Antivirus users arguably at more risk than without it.

Let’s evaluate the most critical software, the web browser.

I cannot stress enough to avoid using Internet Explorer, Edge and Safari, only use it where it is necessary due to legacy reasons.

I recommend using Chrome browser instead. Why? Chrome is built by a team of experts who greatly care about security and minimizing its attack surface. It is not to say that it is perfect, but, in my opinion, it is more secure than the browsers mentioned above.

Chrome has been offering click to play on Flash for much longer than any other browsers, a feature you must make sure is enabled. Click to play on Flash feature prevents hidden Flash module code from running and leaving only the Flash you see to run. Flash is really cycling out of the internet, less and less websites require Flash now to run properly.

To make sure this feature is on, go to Chrome settings –> Show Advanced Settings –> Content Settings –> in Plug-ins, select “Let me choose when to run plug-in content”

https://support.google.com/chrome/answer/142064?hl=en

In addition, Chrome sandboxes your PDF opening it in a process of its own with limited access to your system. This is a great default feature and no action is required to benefit from it.

You can also enable Javascript whitelisting.

What does this mean? Almost every website runs Javascript code in your browser by today’s standard. Almost every non-document attack requires Javascript execution to work. No Javascript, no successful exploitation.

Now, the big problem is that we require Javascript to enjoy the browsing experience, making online payments, listening to music and videos, etc. One possible solution is to use a feature similar to Click to Play on Plugins such as Flash. We can enable Whitelisting on Javascript. However, this will break many websites dynamic and rendering. It takes a little bit of time to manage Javascript whitelisting which can be frustrating. On the up side, you will break a lot of attacks by making them impossible to work against you. Being a user of Javascript whitelisting myself, I’ve come up with a few tricks up my sleeve to avoid some annoyances.

First, here’s how to enable this feature in Chrome.

Go to Settings –> Show Advanced Settings –> Content Settings –> In Javascript section, select “Do not allow any site to run Javascript”

Don’t worry, you can still let Javascript run easily wherever you would like to. Here’s how

1. Go to www.youtube.com or www.gmail.com or any site you use daily.
2. The first time you will load the chosen website, you will see this icon “<>” with an x at the end of the URL bar (where you typed in the website). To enable Javascript for this website, click on this icon, from the menu that appears, select “Always allow Javascript on <website>”.

3. Then re-enter the website by either re-typing the website or simply selecting it again and pressing ‘enter’. This time, the website will load with Javascript and the “<>” icon will not appear.

Voilà! You are now blocking all external and indirect Javascript, saving yourself from many bad Javascript out there.

The most annoying problem that you will encounter is when shopping online. Sometimes you need to whitelist 4 different websites to allow one single transaction to go through. Most of the time, by the time you allow all the websites to work, your purchase will fail and you have to do it all over again. Thankfully, there is a simple way to work it. In such cases, I simply use another browser, such as Firefox, that runs Javascript by default.

Now that we’ve limited our exposure through the browser, we can get back to discuss how to safely use documents.

Documents are the attack vector of choice of commodity malware because exploitation leveraging documentts is cheaper and easier to develop. They are also used in targeted attacks.

How can anyone prevent or avoid document attacks? The logic is not to open the document, or in certain cases, not to “look” at it. However, this is not a very practical solution.

Before opening the document, it is important to assess if the document is actually something for you from someone you know. It’s far from a perfect solution, but you will block certain incoming attacks.

Example

You received an invoice via email, is it really an invoice you were expecting? If not, it’s likely a document with an exploit in it. Simply delete the email, don’t open it. You can even tag it as SPAM or phishing in Gmail.

Simple and easy solution to avoid document exploits is to Let Google Doc open it for you.

1. Login to Google Drive (Free) and
2. Upload the Word document, PDF, Excel, pretty much anything to it.
3. Then, you can open the document as a preview or even convert it to Google’s native online Docs.

This doc opening technique is a strong protection mechanism because in the process of conversion, Google will filter out any suspicious part of it by scanning document for potential malware. If it cannot convert it, it could be because of exploitation attempt.

In theory, we can use Microsoft One Drive with the same results.

We lose some privacy in the process to open docs via Google or Microsoft, but most of the day to day files are not that sensitive and keeping your computer and phone clean is more important.

Another recommendation is to avoid using Adobe Flash on your computer. Simply don’t install it or uninstall it. Chrome will run Flash more securely as it is running in a sandbox.

Adobe Reader and Preview on the Mac are also frequent vector of attacks. Choose another software to read your PDFs, arguably less secure but not targeted and remember, you can safely read PDFs on your Google drive.

Anti-Virus or not, Sandboxing or not, these techniques will keep you safe from many attacks. It has worked for me for several years keeping the number of successful attack to minimal levels. In a following blog, we will discuss what we can do to detect attacks that will make it through.

My main use case for making C programs is to study the system and facilitate development and testing using some scripting tools. Also, you know, an Android phone is a Linux computer (if you look at it from far away). So let’s treat it with the respect it deserves and write C for it.

Building C programs

The Android NDK offers a complete C/C++ toolchain, which the SDK and Android Studio uses to build shared libraries. These libraries provide hooks to be loaded into Android applications, thereby offering services implemented as native code. Since making command-line programs is not a goal pursued through the publication of the NDK, the default link script does not bring in the C runtime initialization, so that shared libraries the program depend on are not loaded implicitly. Furthermore, the build here is a cross-compilation: the target system for the executable program is distinct from the one where the program is built. Therefore, we cannot rely on trivial system dependancies: the dependancies on user-mode GNU/Linux subsystems must be explicitly stated. Therefore, if one uses the NDK tools naively, such as

$ $NDKTOOL/gcc -o mytool mytool.c

the resulting mytool executable is valid, but crashes when it is run on the device.

In order to properly incorporate the libc preamble and conclusion for command-line execution, we must subvert the default link script. Thus, the next two commands compile, then link mytool:

$ $NDKTOOL/gcc -c -I$NDKPLAT/usr/include -o mytool.o mytool.c
$ $NDKTOOL/gcc -o mytool -L$NDKPLAT/usr/lib \
    -dynamic-linker=/system/lib/linker \
    $NDKPLAT/usr/lib/crtbegin_dynamic.o \
    mytool.o \
    $NDKPLAT/usr/lib/crtend_android.o -lc -lgcc

Let us go over the details of these various commands.

Environment variables

The toolchain programs from the Android NDK must be explicitly chosen; including this toolchain into the shell’s PATH runs the possibility of confusing between the NDK toolchain and the system’s native toolchain. To avoid this, we may set the following environment variables:

$ export NDK=/home/benoit/android-ndk-r11c
$ export NDKVER=android-5
$ export NDKPLAT=$NDK/platforms/$NDKVER/arch-arm
$ export NDKTOOL=$NDK/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64/arm-linux-androideabi/bin

The first environment variable should be modified to reflect the path obtained from the extraction of the NDK archive. The second variable introduces an interesting aspect of the NDK: it recapitulates (and, therefore, contains) the most recent Android platform, as well as all previous ones. For instance, platform version android-5 selected here corresponds to the application binary interface (ABI) of Android 2.2 (Froyo). As far as my experience has gone, recent platforms are still back-compatible with android-5, so although I have left behind anything prior to Android 5.0, I still build using the Froyo ABI. Variables 3 and 4 point to the platform (header files and libraries) and build tool directory, respectively.

Link editor parameters

While the compiler invocation is easily understood (at least by seasoned C programmers), the link editor command line is certainly more complex. Let us break down the parameter list:

• -o hello_world — Generated program will be called hello_world.
• -L$NDKPLAT/usr/lib — Link against the libraries of the Android platform.
• -dynamic-linker=/system/bin/linker — Find the dynamic linker of the target (Android) system at path /system/bin/linker. This is important, as typically, the link editor for Linux systems is /lib/ld-linux.so.2.
• $NDKPLAT/usr/lib/crtbegin_dynamic.o — This is the preamble for self-contained executables that make use of shared objects (including the shared version of libc).
• mytool.o — Object code for the program to build.
• $NDKPLAT/usr/lib/crtend_android.o — Common C runtime self-contained executable postamble.

Running the program

That one is easy. We simply use the Android Debug Bridge (ADB) to upload the program on an Android device (or emulator) and then run it from a shell started from ADB. In a nutshell, upload:

$ adb push mytool /data/local/tmp
$ adb shell
root@generic:/ $ cd /data/local/tmp
root@generic:/data/local/tmp $ chmod 755 mytool
root@generic:/data/local/tmp $ ./mytool

/data/local/tmp is a temporary file directory exactly for this kind of experimental purpose.

Debugging the program

In addition to the build toolchain, the NDK comes with a corresponding version of the GNU Debugger, GDB. We use GDB all the time to debug on Linux without much issue (beyond the fact that it’s a command-line debugger), so this should not pose any problem on Android?

Well, there are some difficulties to take into account due to the fact that we’re debugging from a computer host remotely, over an ADB/USB link. In addition, the target computer (the Android device) has a different system organization compared to the host system (the computer from which we build and debug). Thus, by default, GDB does not find the files in the directory structure reported by the debugging server (which runs on the device), and fails to build coherent stack backtraces. This makes the investigation of crashes overwhelmingly difficult.

Duplicating the runtime environment

The solution to this problem is to rebuild a sufficient part of the Android system our program runs in on the host, so that GDB can find it and properly account for stack organization. We start from a shell on the device:

root@generic:/data/local/tmp $ mkdir -p system/bin
root@generic:/data/local/tmp $ cp /system/bin/linker system/bin
root@generic:/data/local/tmp $ cp -R /system/lib system/
root@generic:/data/local/tmp $ chmod -R 666 system
root@generic:/data/local/tmp $ ls system/lib > alllibs
root@generic:/data/local/tmp $ chmod 666 alllibs

We thus created a copy of the dynamic linker and all basic dynamic libraries on the Android device; in my experience, this has proven sufficient to debug my programs. Unfortunately, the ADB file transfer tools do not handle file bundles: this is why we created the listing of the system/lib directory and stored it in alllibs. We will use it to download each file in turn.

$ adb pull /data/local/tmp/alllibs alllibs
$ mkdir -p system/bin
$ mkdir -p system/lib
$ adb pull /data/local/tmp/system/bin/linker system/bin
$ for L in `cat alllibs`; do adb pull -p /data/local/tmp/system/lib/$L system/lib/$L; done
$ rm alllibs

We can now delete the transferred partial snapshot that lives on the Android device:

$ adb shell rm -rf /data/local/tmp/system

Running the debugging session

We are now ready to debug the program remotely, using GDB. The term remotely suggests that the debugger interface runs on a distinct system than the debugged program. This means that the actual debugger runs on the device, and communicates with the client over some link. In this case, the link is a TCP/IP connection. It can be done using wi-fi networking between the host and the device. However, Android emulators do not expose a wi-fi interface to the network where the host lives; as for the host itself, it might be a virtual machine that shares connection with the host system, therefore standing behind NAT. To alleviate all these problems and provide a more private and reliable link, ADB can forward local TCP ports of the host over to TCP ports on the device, through a USB link. We will use this feature.

The first step consists in uploading the GDB server program over to the device, using ADB:

$ adb push $NDK/prebuilt/android-arm/gdbserver/gdbserver /data/local/tmp
$ adb shell chmod 777 /data/local/tmp/gdbserver

We now initiate the forwarding of a TCP port on which GDB will serve hooks for the debugger interface. Let’s use port 10000; it could be any other non-privileged port.

$ adb forward tcp:10000 tcp:10000

Let us now start a shell from which we will run the GDB server. We will debug program mytool with parameters asdf and 5432, running in directory /data/local/tmp.

$ adb shell
root@generic:/ $ cd /data/local/tmp
root@generic:/data/local/tmp $ ./gdbserver :10000 ./mytool asdf 5432

The Android shell then gets output indicating that the server is ready to initiate the debugging session. The program does not start before the client says so. From another terminal, let we start the client and connect it to the server. Please mind that the host system I run it from is a Linux host; for Windows or Mac OS X, the path would be slightly different. From the GDB prompt, we connect to the debugging server:

$ $NDK/prebuilt/linux-x86_64/bin/gdb
[GDB banner and license info]
(gdb) target remote localhost:10000
[Output indicating connection succeeds.]

GDB now complains that it cannot find all the dynamic libraries it needs to assess how the program interacts with the system. To help it locate all this, we need to set the sysroot and solib-search-path GDB variables.

(gdb) set sysroot .  # Where we downloaded the system/ directory.
(gdb) set solib-search-path system/lib

At this stage, GDB accounts for all the dynamic libraries used by our Android program. We can make sure by running

(gdb) info sharedlibrary

We may finally set breakpoints and resume program execution that was suspended on start by the debugging server:

(gdb) c

From then on, GDB will be able to track call stacks properly, and report accurate stack backtraces with command bt.

As with anything GDB, all these settings (from starting the debugging server to resuming program execution) have to be redone every time we initiate a debugging session. Fortunately, all client-side commands can be written to a file, which may then be sourced in GDB on startup (using the source GDB command).

Bonus! Analysing an Android core dump

Core dumps are snapshots of a program state as it is terminated by the system because of an exception (e.g. memory access violation). They are tremendously useful when used in conjunction with programs that bear debugging information, because they contain the stack trace of all running threads at the moment of the crash. These stack traces may be observed using GDB, provided it has been set up as described above.

The first step to using core dumps for debugging is to enable their generation and capture. In the Android shell where the program is meant to run:

root@generic:/ $ ulimit -c unlimited
root@generic:/ $ echo '/data/local/tmp/core_pid%p_user%u_group%g_sig%s_time%t' > /proc/sys/kernel/core_pattern

The latter command enables core generation for all programs, but the former only affects the programs forked from the shell it’s run in. Once you have this, run your buggy program until it crashes. Once this crash happens, you will find the core dump in /data/local/tmp. To analyze it, download it to the host system, then open the GDB client. From there:

(gdb) file mytool
(gdb) core-file core_[...]

The first command indicates that whatever we do next concerns program mytool; the second loads the core dump we downloaded from the Android device. At this moment, GDB complains that it cannot find all necessary dynamic libraries, so run the setup steps from the last section that follow the target remote command.

Much has been written and spoken about on the topic of cyber threat, and yet within government and industry today the prevailing leadership habit is one of risk acceptance without understanding or even appreciating the risk that is being accepted. Tragically, this risk acceptance is often perceived be these leaders as ‘not taking unnecessary risk’ and characterized by inaction in the face of significant actual risk. They simply do not appreciate their organization’s exposure to cyber threat or the impact to their operations it could have.

So it begs the question: why?

At its fundamental level cyber defense is a contest between humans, with each party attempting to gain advantage by imposing their will upon the other. The attacker looks to exploit vulnerability while the defender arrays himself to both raise the level of difficulty for the attacker and position himself to respond when his defenses are overcome. This reality shines a light on the need to look beyond pure technology solutions and calls for adopting an approach to cyber defense that considers the human element as a critical factor.

Recognizing and then accepting that all defenses can and eventually will be overcome is vital to forming an effective cyber defense strategy.

So why does the prevailing position remain one of inaction?

I believe that the reticence to take decisive cyber defense action that characterizes the leadership of most organizations today stems from a lack of understanding of the environment and a lack of awareness of actual threat activity.

This is a highly technical and rapidly evolving domain. Networks were created for openness and sharing and we weren’t designing networks or computers with security in mind until very recently. Formal education quickly becomes overtaken by technological advances and if knowledge is not reinforced by experience and continually developed, skill fade has a very real impact on the ability to competently describe the situation and often shakes the confidence of advisors to the point where they are reluctant to advise. Because of the technical complexity and the historically poor ability of technical staffs to effectively translate technical impact into operational implications, the leadership echelons in public and private institutions typically have not been presented with decision-quality information.

Awareness of actual cyber threat activity is overwhelmingly kept sequestered behind closed doors and classified beyond usefulness by governments and is therefore not shared with the public. Those responsible for security within organizations are often embarrassed by what they fear will be perceived as a failure to do their job, and so are less likely to proactively expose breaches or intrusions to their leadership for fear of the consequences and being seen as failing.

There are two distinct but compounding challenges: education and awareness. Often the one is confused for the other, and when highly educated and intelligent people are presented with cyber threat information they were not aware of they dismiss it as untrue or exaggerated fear mongering.

As a result, we’ve grown a leadership culture that assumes away risk without understanding, and a public that is not empowered to make wise choices for themselves or their organizations because they quite simply don’t know they need to – a very dangerous reality. I believe that leaders want and the public deserve to know what is happening on their networks so that they can understand the actual risks of both their actions and their decision not to act. We must empower technical staffs with the education, training and the tools that make it possible for them to support decision-making and effectively defend their networks.

So why do I wish I had a Counter-Advanced Persistent Threat (APT) Hunt capability when I was directing cyber operations in the military?

A Hunt capability pairs sophisticated detection and response technology with expert operators to actively seek out APT presence within a supported network.

With the fidelity that telemetry from a host-based covert agent provides, I can detect those APTs that my traditional defensive technologies miss; I can know with precision where a threat on my network is; and I can take immediate action remotely through the agent to render that threat ineffective across my network with no impact to ongoing operations. This means I gain time and save costs directly by not having to deploy teams physically to remote locations to address cyber security breaches – often with significant impact to operations.

With expert human operators and analysts manning this over watch capability and synthesizing adversary activity information for me, I can now quickly and confidently present to my leadership: my considered subjective judgement on what risks they should accept or reject; a detailed understanding of the costs of both the threat and the cure; and I can relay it to them in terms of their ongoing operation.

In short: I can regain the initiative and preserve my freedom of action because I now know exactly what is happening on every host and can respond surgically to threats within minutes.

I’ll say it again: recognizing and then accepting that all defenses can and eventually will be overcome is vital to forming an effective cyber defense strategy. Proper usage policies, architectures, configuration and monitoring are each important and should be designed, established and maintained for best effect. That said, without a dedicated manned capability positioned to monitor and then rapidly respond when defenses are overcome, your resilience to cyber attack is limited to patching schedules and update priorities that are beyond your control and simply too slow to keep pace with advanced cyber attackers’ tools and methods.

In this volatile and complex environment, a Counter-APT Hunt capability gives leaders the ability to assure the integrity of their networks at the speed of attack. Freedom of action within your own network is a pre-condition for all operations and business activities, and this freedom starts with quality situational information and an understanding of the actual risk you face.