Note that attempting to interfere with operations of a profitable botnet can be dangerous, as your actions may cause attackers to retaliate. Therefore, consider these steps as informational thoughts, rather than an encouragement to follow FireEye’s footsteps.

1. Obtain a copy of the bot through forensic analysis of a compromised system. It helps to get hands on several instances of the malicious program, in case multiple variants possess meaningful behavioral differences.
2. Understand the bot’s command and control mechanism. How does the attacker control the botnet? Reverse-engineer the malicious program to understand the C&C protocol and to get a sense for the commands the botnet understands. You may find a way to authenticate to the botnet and, posing as the attacker, commandeer it. (Warning: As Andre posted in the comments, “Logging on to network that is not your own, and issuing commands to take it over could potentially be considered illegal access.”)
3. Identify which systems, if taken off line, could disrupt the botnet. To accomplish this, look for weaknesses in the command and control implementation, such as the reliance on a small set of servers to distribute commands or weakness in the C&C servers’ IP or domain names generation algorithm. (You may recall how researchers at UC-Santa Barbara gained control over an instance of the Torpig botnet.)
4. Contact ISPs hosting suspected C&C servers. In your correspondence with them, present documentation that supports your claim that the systems they are hosting are being misused. Be specific about which IPs violate the ISP’s policy by acting maliciously and should be disabled.
5. Contact registrars of C&C domains. In your correspondence with them, present documentation that supports your claim that the domains they are hosting are being misused.  Be specific about which domains violate the registrar’s policy by being used for malicious purposes and should be disabled.
6. Consider registering unused domains that the botnet’s C&C mechanism may attempt to use later. This can be expensive, depending on the number of domain names associated with the botnet’s C&C implementation.

Botnets come in different shapes, sizes, and flavors. The steps above don’t apply to all of them, but they should give you a sense for how defenders can take action against traditional botnets. For an example of these steps in the context of a specific botnet, see the “Smashing the Mega-d/Ozdok botnet in 24 hours” write-up by FireEye.

Have you taken steps to disrupt a botnet? Share your thoughts and experiences in the comments below.

– Lenny

Lenny Zeltser teaches the Reverse-Engineering Malware course at SANS Institute. You can find him on Twitter as @lennyzeltser.

• Examining the websites that may be associated with the incident, often because they are suspected in hosting exploits that acted as the infection vector
• Obtaining reputational data about IP addresses of systems involved in the incident, often because they are suspected of hosting malicious files that were dropped on the system, or acting as the command and control server for the attacker
• Looking up IP addresses associated with the infected organization in blocklists, to determine whether additional systems may have been performing malicious activities and may have gotten compromised
• Performing automated behavioral analysis of malware involved in the incident, to get a general sense for its characteristics to plan subsequent manual reverse-engineering tasks

Each of the following pages lists 10 or so freely-available on-line tools for helping to perform the tasks outlined above:

• On-Line Tools for Malicious Website Lookups
• Blocklists of Suspected Malicious IPs and URLs
• Automated Malware Analysis Services

What other on-line tools help understand the context of the infection? Tell us in comments below.

– Lenny

Lenny Zeltser teaches the Reverse-Engineering Malware course at SANS Institute. You can find him on Twitter as @lennyzeltser.

“SpyKing Vista Spy secrectly logs all keystrokes, web sites, emails, chats & IMs: MSN Messenger, Windows Live Messenger, ICQ, AOL Messenger, AIM, Yahoo! Messenger, Windows Messenger and Skype. Takes screen snapshots at every X seconds like a surveillance camera. Displays exact activities, like MySpace, Facebook, PC games, online searches & shopping, file transfers and webmails. You can receive reports remotely via emails or ftp”.

As you can see from the image below, the site has been reported as a known attack site with a number of malicious scripts being located on their system.

There is a trial and a commercial version of the software available. For this exercise, I have used the paid commercial version in order to gain the complete set of utilities and have all the features. This way there is little chance that the software will be located due to a trial feature that is removed in the commercial product.

A good number of the windows tools are either listed with the source or are from Sysinternals (http://live.sysinternals.com). In either case, these are free tools. I shall concentrate on the process instead of the results in this post as this will enable you to do your own analysis of other programs (and not to just rely on the work of another).

For this analysis, I have configured a Windows XP VM on my RHEL host. This is a clean host with no updates as yet. At present there is not a great deal going on in the system. From a networking perspective we can see a number of basic Windows ports listening.

Next, I am creating an initial snapshot of the “AutoRuns”. These are the settings, programs, codecs etc that are loaded when Windows boots or when a number of other events occur (such as opening Internet Explorer).

By saving the complete list, we can take snapshots (before, during and after) of the installation process. In this way, we get a list of the changes that have occurred on the system. We can isolate these and then associate them with the effect. To do this effectively, we need to capture a complete set of changes to the system. In Windows, this means the registry (below we are using the SysInternals Registry Monitor tool to capture all registry activity) and many other areas of the system.

In addition, RegShot can be used to take before and after snapshots of the system as well as to create a comparison of the changes.

We start with a before snapshot on our pristine system clicking “1^st shot”.

Later, following the install, we take another shot and at each shot, save the capture.

Following the installation, RegShot will also allow us to directly compare the changes to the system.

In addition to the registry, it is essential to monitor the file-system. From the image included below, we can see data being written to the “C:\Program Files\SKPCS\data” directory. This is the location where Spyking is saving data (more on this when we have covered the installation process).

At the same time, we also monitor system processes. To capture the network information, we setup a capture using tcpdump with a host filter on the underlying linux system (that our VM’s are running on).

Installing the software

Now that we have setup the monitoring tools, we will want to install the software and capture what occurs in this process.

Start with the registered version of the software

We can see from process explorer that Spyking spawns a separate process (is-S3N8.tmp).

In this case we use the default folder. This is configurable and should only be used as an indication, not a definitive signature.

As an exercise, I also attempted to reinstall Spyking over a running version of the software.

Note that the folder may be hidden, but you still receive error messages if you attempt to write over it.

So now back to the install. Here we have selected the default install folder.

And we have installed the program successfully. Next comes the unlock section. Here we enter the details of our license. Without this, Spyking runs in demo mode and leaves a visible sign of being installed.

Once we have unlocked it, we are taken to the setup wizard.

Here we will monitor all activity. In this configuration, SpyKing is far more verbose and far easier to recover. The longer it is run and the more that it logs equals the easier it is to find information.

In the second step of the wizard we set the ‘hotkey’. This is used to ‘unhide’ the program and make it available.

Finally, we setup the location of the logging. Setting a spoofed host is simple (to act as an email server) and we can record the activity of the program. The information in these emails can be used as a signature for network detection. This would have to be validated against multiple versions of the software before relying on this and it will also do little against other spyware programs. The emails and logs are clear text however. This does make network based detection relatively simple.

And we are ready to roll.

Incidentally, when we setup the program, the licensed version uses an online activation.

In the setup, it must be noted that the installation program sets up a UDP listener.

This is bound to the localhost and no traffic was monitored to or from this port from the outside. More research should be made on what exactly this process does.

Well let’s log into the software.

Installed and Running

Now that we have logged into the program, we are taken to the admin screen. Note that this is a registered version – this however still provides the option of purchasing more licenses online.

This interface allows us to set individual actions for each of the monitoring sub-systems. We shall accept these options and look at a few options. First, there is an option to run the program as Administrator. This is where the program is most effective.

Then as another example, we have the advanced admin section. We see that the hotkey is ALWAYS a combination of “Ctrl + Alt + *“ where * is a key of the users choice. This is not a function key.

Hence, a user has a means of checking for the program. On top of this, a simple scanner hooking into the input function of the system could scan for all possible combinations in seconds.

The list is a drop-down selection of 10 numerals and the 26 alpha keys. This is a total keyspace of 36 characters. The shift key does not come into this and detection for a home user is as simple as hitting 36 key combinations. In fact, the reality is that this is a keyspace of les than 36 characters as some combinations are already seledcted and used by other system functions.

Next, with the program running in stealth mode I installed and ran the rootkit revealer program. This was used with the complete options selected:

Here we have a couple strange entries, but nothing serious.

Basically, the spyware program does not embed itself that deep into the system and kernel that it is detected as unusual.

But why a VMWare image?

There are several reasons for conducting analysis in a VM, one of which is it is simple to capture network traffic. Next is that you can setup a host once and use snapshots to gain several images and even reverse any mistakes you may make.

One strange occurrence that will require further investigation is the discovery of the Linux TcpDump command strings used on the host system being discovered in the PageFile of the system being monitored. My understanding was that this should not occur. Once we have this data, we can take the pcap network trace that we saved using tcpdump and run it through other tools. In this case, I used the following tools to analyse what was occurring:

• NTop (Produces a graphical summary of traffic and destinations)
• DNStop (Summarises the domains and name lookups found in the network capture file)
• Wireshark (provides a detailed graphical view of the data after the fact)
• TCPReplay (Allows for the reconstruction of files from the network capture)

Below we see a snapshot of our ‘chatty’ spy program.

With a series of network captures, we see the emails, ftp and other traffic that is leaking the information from our host.

Analysing the Running processes

We see from “Process Explorer” that the ‘symserv.exe’ is listing on PID 1592. This process ID does vary, but it is possible to locate the processes and threads used by SpyKing as it is running.

The PE Header information of this program makes a simple signature (far more effective than the presence of the default directory). With the Hex data from the PE Header, you can search the used and unused space on the drive image and discover this program (if it is installed).

What about when we uninstall the program?

Of course in attempting to remove the program we do not find that it is in the “Add / Remove Programs” list.

We instead have to use the uninstall provided with the software.

Clicking this takes us to the removal process.

And we are sure.

So it is now removed.

At least from a normal user perspective it is removed.

With snapshots of the program installed and also with it removed, we now proceed to imaging the various systems.

Lastly, the drive images

In this case, the drive images are simple to analyse. Some programs hide themselves in “non-standard” structures, SpyKing is not one of these. Using the Helix CD image, dd for capture and the Autopsy forensic browser, the recovery of the program was simple.

In the image above, we see the deleted “C:\Program Files\SKPCS” directory for the system we had uninstalled the program from. The program, sysserv.exe which forms a part of the running SpyKing program is no longer in the pagefile, but a number of strings related to this program can still be found a day later (subsequent to removal and a single reboot).

Below we see the image and analysis of the system that had SpyKing running (this was not yet removed).

The program directory (although hidden when in Windows) is simple to find. On top of this, there are copious amounts of data related to the SpyKing program in the pagefile.

For a spyware program, this is a really large footprint.

What was most unusual (and this can be seen in the image above) was the inclusion of the command that was run on the Linux host being uncovered in the Windows VMWare client. The linux memory and commands have been incorporated into the Windows VM host pagefile. This is so far something I have only been able to replicate on these hosts and is something that will require further research.

We have little information from the Autoruns program in this instance, but there is a voluminous trail of access information from the registry, process and file monitoring programs.

The result is that the best indication is to capture data at the network choke points. Where this is not feasible (or the analysis is after the fact), the review of file signatures is the next best option. This requires a binary search. The entire file of each of the binaries can be hashed and added to a known bad list, or alternatively, the PE header including the program optional headers can be used. The best programs to detect include:

• eventsys.exe
• symserv.exe

The sub-folders of the program should also be recoverable to see what has been leaking:

• data
• logs
• scrshot

There are a number of programs that use the “symserv.exe” executable as a simple web search will demonstrate. There is a good likelihood that the person installing this software could also lose control of it creating a RAT on the system. As a consequence, this is not even a good option for the monitoring of your own system, let alone the issues connect to monitoring the systems of other people.

Conclusion

For all of the hype, SpyKing is simple to find. The program leaves a large system footprint for a ‘spyware’ system. It does not clean up after itself and has no covert network capability. Traffic is not encrypted or even XOR’d, so it is simple to set network based filters for this traffic. A BPF with TCPDump could be created to monitor for this without effort and a simple filter could easily be implemented on a pf or IPTables firewall to stop this connection and hence the leak.

Worst of all (or best depending on your opinion and goals), the software is simple to find in the registry and from a drive image – both when installed and after it has been removed.

Craig Wright is a Director with Information Defense in Australia. He holds both the GSE-Malware and GSE-Compliance certifications from GIAC (and the GSE as well). He is a perpetual student with numerous post graduate degrees including an LLM specializing in international commercial law and ecommerce law as well as working on his 4^th IT focused Masters degree (Masters in System Development) from Charles Stuart University where he is helping to launch a Masters degree in digital forensics. He is involved with his second doctorate, a PhD on the quantification of information system risk at CSU.

In this Webcast, Richard Bejtlich, Director of Incident Response for General Electric, and Ken Bradley, Information Security Incident Handler for the General Electric Computer Incident Response Team, will discuss professional incident detection. Richard will interview Ken to explore his thoughts on topics like the following:

1. How does one become a professional incident detector?
2. What are the differences between working as a consultant or as a member of a company CIRT?
3. How have the incident detection and response processes changed over the last decade?
4. What challenges make it difficult to identify intruders, and how can security staff overcome these obstacles?

Richard will lead this event and conduct it more like a podcast, so the audio will be the important part. This is a short-notice event, but it will be cool. Please join us. Thank you!

If you have training funds available for 2009 there are still several SANS forensics classes scheduled.   As an added bonus, classes tend to be smaller this time of year, allowing for even more individual attention.

Most of the upcoming events for all the Digital Forensic Courses and training that SANS offers can be found at the upcoming events page of the Computer Forensics Website.

Vancouver
http://www.sans.org/vancouver09/description.php?tid=3667

Colorado Springs
http://www.sans.org/coloradosprings09_cs/description.php?tid=3667

Tucson
http://www.sans.org/tucson09_cs/description.php?tid=3667

Washington D.C.
http://www.sans.org/cyber-defense-initiative-2009/category.php?c=SEC&pcs=2

Do not want to travel?

SEC408, Computer Forensic Essentials is being taught by Rob Lee via vLive starting on Nov 30, 2009.

http://www.sans.org/vlive/details.php?nid=20023

First of all, all the SANS Digital Forensic Courses have already included up-to-date material fully covering Windows 7 and Vista unlike anyone has done before.  In fact, our challenge for SEC408, Computer Forensic Essentials is strictly based off of a Windows Vista case. We have details in SEC408, Computer Forensic Essentials, that are not as covered in peer courses.

Here is just a few things we have helped document regarding Windows 7.

User Profiles:

With the release of Vista/Win7, Microsoft significantly changed the folder structure and mechanisms used by the operating system for user profiles. One of these changes was to make roaming profiles more explicit. Roaming profiles allow users to log onto other systems in the domain and have their profile information follow them. They have been around for many years, and in Vista/Win7, Microsoft decided to make what follows a user, and what doesn’t follow a user much more explicit. Hence within a user profile in Vista/Win7, there are now two different set of folders: Roaming and Local. For our purposes, we want to be able to determine where our browser artifacts will be located in this new file structure. Traditionally Microsoft has included cookies in a roaming profile and excluded cache and history files by default. Thus, cookies are now found under the Roaming folder and history and cache can be found within the Local folder.

Internet Explorer:

The major change within Vista/Win7 that affects us when performing browser forensics is the newly implemented “Protected Mode”. The idea is that if malicious code is run in the browser, it will not have the necessary privileges to cause harm to the operating system. Since not all activities using the browser will be unprivileged, a duplicate set of directories were necessary to store files from unprivileged use, called Low folders. An example of what this looks like in the file system is:

%userprofile%\AppData\Local\Microsoft\Windows\History\Low\History.IE5 (for the IE history files)

“Protected Mode” conducts web browsing as an unprivileged user

• A new set of locations were added: low folders
• Most browser evidence will be in low folders
• Local file usage is stored in the standard history folder (because it is not performed with restricted permissions)
• If Protected Mode is turned off, low folders will not be utilized
• If User Access Control (UAC) is turned off, low folders will not be utilized (it is required for Protected Mode to operate)
• If the instance of IE is run with Administrator permissions, the low folders are also not used

USB Key Analysis:

We discussed full Windows 7 USB Key Analysis in this post:  http://blogs.sans.org/computer-forensics/2009/09/09/updated-computer-forensic-guide-to-profiling-usb-thumbdrives-on-win7-vista-and-xp/

USB Drive Enclosure Analysis:

We discussed how to perform Windows 7 USB Drive Enclosure Analysis in this post:  http://blogs.sans.org/computer-forensics/2009/09/09/usb-key-analysis-vs-usb-drive-enclosure-analysis/

Defrag Analysis:

Chad Tilbury discussed detecting defrag analysis here: http://blogs.sans.org/computer-forensics/2009/08/17/de-mystifying-defrag-identifying-when-the-windows-defragmenter-has-been-used-for-anti-forensics-part-2-vista/

Timeline Analysis:

Kristinn Guðjónsson developed and released a full scope timeline creation tool called log2timeline that is able to parse many Windows Vista and Windows 7 artifacts in a single simple tool.

• Prefetch directory (reads the content of the directory and parses files found inside)
• UserAssist key info (reads the NTUSER.DAT user registry file to parse the content of UserAssist keys)
• Squid access logs (with emulate_httpd_log off)
• Restore points (reads the content of the directory and parses rp.log file inside each restore point)
• Windows shortcut files (LNK)
• Firefox 3 history file (places.sqlite)
• Windows Recycle Bin (INFO2)
• Windows IIS W3C log files
• OpenXML Metadata (for metadata inside Office 2007 documents)
• ISA Server text export from queries (saved to clipboard and from there to a text file)
• TLN (Timeline) body file
• Mactime body file (so it can be output in a different format)

http://blogs.sans.org/computer-forensics/2009/08/13/artifact-timeline-creation-and-analysis-tool-release-log2timeline/

Shadow Copy Forensics

Troy Larson from Microsoft has done a wonderful job continuing to discuss the Shadow Volume Copy and ways you can examine them in an investigation.  We posted back in 2008 on many of his techniques.

http://blogs.sans.org/computer-forensics/2008/10/10/shadow-forensics/

The work continues:  There are many artifacts yet to be uncovered and more work is being done.  Keep your eyes peeled on this site and additional sites like Harlan Carvey’s http://windowsir.blogspot.com/ as he is publishing many details as well.

If you have any sites that contains Windows 7 artifact information please post them in comments and Ill update the post as we move forward.

Rob Lee is a Director  for MANDIANT, a leading provider of information security consulting services and software to Fortune 500 organizations and the U.S. Government. Rob has over 13 years experience in computer forensics, vulnerability discovery, intrusion detection and incident response. Rob is the lead course author and faculty fellow for the computer forensic courses at the SANS Institute and lead author for SEC408 Computer Forensic Essentials and SEC508 Computer Forensics, Investigation, and Response.

The cemail.vol file is a proprietary Microsoft format and there are limited tools for parsing this format directly. In some situations, viewing this file using a hex viewer will reveal deleted messages and other items that are not acquired using common forensic tools. Although XACT from Microsystemation has the ability to interpret cemail.vol databases automatically, forensic practitioners with limited budgets are seeking lower cost solutions.

One effective approach to interpreting this type of database using freely available software is to mount a copy of the acquired cemail.vol file into a Windows Mobile Emulator and use the itsutils package to navigate the database and extract the desired items. The pdblist utility in the itsutils package can dump many databases on a Windows Mobile device.

To illustrate, consider the following message “I have your package” in an acquired cemail.vol file viewed with a hex viewer.

Mounting the Acquired File in Window Mobile Emulator

First, it is necessary to mount the acquired cemail.vol file in a Windows Mobile Emulator. Although it is not necessary to use an Emulator that exactly matches the evidentiary device, some similarity is recommended. There are a number of emulators included in Visual Studio. Additional emulators can be downloaded from the Microsoft Web site.

Once a suitable Windows Mobile Emulator has been selected, it is necessary to configure it to access the folder on the examination computer where the acquired cemail.vol file is stored. The following screenshot shows the shared folder being configured to point to C:\Documents and Settings\Administrator\Desktop\WindowsMobile, which is then accessible under the volume named “Storage Card” within the Emulator.

After launching and configuring the desired Windows Mobile Emulator, it is necessary to create a conduit that itstutils uses to send commands to the Emulator by establishing an ActiveSync connection. You achieve this by opening the Device Emulator Manager in Visual Studio (under the Tools menu), then right-clicking the selected Emulator and selecting Cradle. In addition, within ActiveSync connection settings it is necessary to allow DMA connections.

Useful Commands

After an ActiveSync connection has been established with the Emulator, you can access its contents using components of the itsutils package. For our purposes, the pdblist utility can list accessible volumes, including the virtual “Storage Card” that contains the cemail.vol file to be examined as shown here:

C:\Tools\itsutils>pdblist -v
volume {00000000-0000-0000-0000-000000000000} \Documents and Settings\default.vol
volume {40684a00-994b-f835-7742-f7f435ba8d2b} \ReplStorVol
volume {15005d00-12f3-a6e9-76e8-595b9d742cc8} \mxip_notify.vol
volume {65ca7a00-7d53-6505-5671-0b1908d7e6eb} \cemail.vol
volume {225c1b00-e193-8a1a-785f-68f818cf3dd0} \Storage Card\cemail.vol
volume {c479de00-e4b7-9037-1352-dced359be0ad} \mxip_system.vol
volume {d071d100-fb8f-1505-782c-e71b23e00165} \mxip_lang.vol

 
More importantly from a forensic examination perspective, pdblist can list components of databases that are accessible via the emulator as shown here:

C:\Tools\itsutils>pdblist -D
volume {225c1b00-e193-8a1a-785f-68f818cf3dd0} \Storage Card\cemail.vol
oid310000c0: dbase F00000017 T00000000    0    356 ... 'fldr31000095'
   ORDERING: 0e060040:00000000 0c1a001f:00000002 0037001f:00000002 001a0013:00000000
[cut for brevity]
oid38000079: dbase F00000017 T00000000    1    484 ... 'fldr31000028'
   ORDERING: 0e060040:00000000 0c1a001f:00000002 0037001f:00000002 001a0013:00000000
oid32000087: dbase F00000017 T00000000    0    356 ... 'pmailAttachs'
   ORDERING: 81000013:00000000
oid37000081: dbase F00000017 T00000000    0    356 ... 'fldr32000023'
   ORDERING: 0e060040:00000000 0c1a001f:00000002 0037001f:00000002 001a0013:00000000
oid34000071: dbase F00000017 T00000000    3    800 ... 'fldr31000026'
   ORDERING: 0e060040:00000000 0c1a001f:00000002 0037001f:00000002 001a0013:00000000

[cut for brevity]
oid33000029: dbase F00000017 T00000000    0    356 ... 'pmailVolumes'
oid3b000017: dbase F00000017 T00000000   53   3768 ... 'pmailNamedProps'
   ORDERING: 8300001f:00000000 83010013:00000000
oid30000009: dbase F00000017 T00000000   12   1020 ... 'pmailMsgClasses'
   ORDERING: 8300001f:00000000 83010013:00000000
oid30000007: dbase F00000017 T00000000    0    356 ... 'pmailOldTables'
oid30000003: dbase F00000017 T00000000    6   1824 ... 'pmailMsgs'
   ORDERING: 800c001f:00000000 0e090013:00000000 00150040:00000000
oid30000001: dbase F00000017 T00000000   21   3052 ... 'pmailFolders'
   ORDERING: 0e090013:00000000
[cut for brevity]

 
The same utility can be used to dump a particular object by name. Working through the objects listed in the above pdblist output, the same text message shown earlier in a hex viewer is revealed in fldr31000026 as shown below using the pdblist command in this manner. Additional details like the date-time stamp associated with the message are also displayed along with other text messages.
 

C:\Tools\itsutils>pdblist -d fldr31000026
3f000089 (  284 12      2)
        8005 T13 L0000 F0000 UI4 838860938
        8011 T13 L0000 F0000 UI4 3
        001a T13 L0001 F0000 UI4 822083599
        003d T1f L0000 F0000 STR [00169898]( 0) ''
        0037 T1f L0000 F0000 STR [0016989c](19) 'I have your package'
        0e17 T13 L1ebe F0000 UI4 262144
        0e06 T40 L0000 F0000 FT  2009-04-22 21:01:47.000
        0e07 T13 L0004 F0000 UI4 33
        0c1f T1f L0000 F0000 STR [001698c4](11) '14438509426'
        0c1a T1f L0000 F0000 STR [001698dc](11) '14438509426'
        8001 T13 L0001 F0000 UI4 1056964745
        3008 T40 L9b35 F0000 FT  2009-04-22 21:01:47.000

3000008e (  284 11     78)
        8005 T13 L0000 F0000 UI4 973078668
        8011 T13 L0000 F0000 UI4 5
        0e17 T13 L0001 F0000 UI4 0
        001a T13 L0000 F0000 UI4 822083597
        003d T1f L0000 F0000 STR [00169888]( 0) ''
        0037 T1f L1ebe F0000 STR [0016988c](13) 'meeting place'
        0e08 T13 L0000 F0000 UI4 9284
        0e06 T40 L0004 F0000 FT  2009-04-22 21:05:45.000
        8001 T13 L0000 F0000 UI4 805306510
        0e07 T13 L0000 F0000 UI4 268501033
        3008 T40 L0001 F0000 FT  2009-04-22 21:05:45.000

3e0000a1 (  284 12     72)
        8005 T13 L0000 F0000 UI4 855638176
        8011 T13 L0000 F0000 UI4 7
        0e1b T13 L0001 F0000 UI4 0
        8012 T13 L0000 F0000 UI4 0
        001a T13 L0000 F0000 UI4 822083597
        003d T1f L1ebe F0000 STR [00169898]( 0) ''
        0037 T1f L0000 F0000 STR [0016989c]( 8) 'codeword'
        0e08 T13 L0004 F0000 UI4 17015
        0e06 T40 L0000 F0000 FT  2009-04-22 23:56:46.000
        8001 T13 L0000 F0000 UI4 1040187553
        0e07 T13 L0001 F0000 UI4 268501033
        3008 T40 L006d F0000 FT  2009-04-22 23:56:47.000

Additional Evidence

Be aware that Windows Mobile creates temporary files in various locations where you may find useful information depending on what you are seeking (e.g., e-mail, MMS). We cover Windows Mobile in the SANS Mobile Device Forensics course, and we delve into cemail.vol and other useful data sources on these devices. The next course is January 11 – 15, 2010 in New Orleans.

Eoghan Casey is founding partner of cmdLabs (http://www.cmdlabs.com/) , author of the foundational book Digital Evidence and Computer Crime, and coauthor of Malware Forensics. He has been involved in a wide range of digital investigations, including network intrusions, fraud, violent crimes, identity theft, and on-line criminal activity. He has testified in civil and criminal cases, and has submitted expert reports and prepared trial exhibits for computer forensic and cyber-crime cases.

In this segment, we will introduce the attack progression (aka “kill chain”) and briefly descibe its intersection with indicators.  The next segment will go into more detail about how to use the attack progression model for more effective analysis and defense, including a few contrived examples based on real attacks.

On Indicators

Just like you or I, adversaries have various computer resources at their disposal.  They have favorite computers, applications, techniques, websites, etc.  It is these fundamentally human tendencies and technical limitations that we exploit by collecting information on our adversaries.  No person acts truly random, and no person has truly infinite resources at their disposal.  Thus, it behooves us in CND to record, track, and group information on our sophisticated adversaries to develop profiles.  With these profiles, we can draw inferences, and with those inferences, we can be more adaptive and effectively defend our data.  After all, that’s what intelligence-driven response is all about: defending data that sophisticated adversaries want.  It’s not about the computers.  It’s not about the networks.  It’s about the data.  We have it, and they want it.

Indicators can be classified a number of ways.  Over the years, I and my colleagues have wrestled with the most effective way to break them down.  Currently, I am of the mind that indicators fall into one of three types: atomic, computed, and behavioral (or TTP’s)

Atomic indicators are pieces of data that are indicators of adversary activity on their own.  Examples include IP addresses, email addresses, a static string in a Covert Command-and-control (C2) channel, or fully-qualified domain names (FQDN’s).  Atomic indicators can be problematic, as they may or may not exclusively represent activity by an adversary.  For instance, an IP address from whence an attack is launched could very likely be an otherwise-legitimate site.  Atomic indicators often need vetting through analysis of available historical data to determine whether they exclusively represent hostile intent.

Computed indicators are those which are, well, computed.  The most common amongst these indicators are hashes of malicious files, but can also include specific data in decoded custom C2 protocols, etc.  Your more complicated IDS signatures may fall into this category.

Behavioral indicators are those which combine other indicators – including other behaviors – to form a profile.  Here is an example: ‘Bad guy 1 likes to use IP addresses in West Hackistan to relay email through East Hackistan and target our sales folks with trojaned word documents that discuss our upcoming benefits enrollment, which drops backdoors that communicate to A.B.C.D.’  Here we see a combination of computed indicators (Geolocation of IP addresses, MS Word attachments determined by magic number, base64 encoded in email attachments) , behaviors (targets sales force), and atomic indicators (A.B.C.D C2).  To borrow some parlance, these are also referred to as Tactics, Techniques, and Procedures (TTP’s).  Already you can probably see where we’re going with intelligence-driven response… what if we can detect, or at least investigate, behavior that matches that which I describe above?

One likes to think of indicators as conceptually straightforward, but the truth is that proper classification and storage has been elusive.  I’ll  save the intricacies of indicator difficulties for a later discussion.

Adversary Behavior

The behavioral aspect of indicators deserves its own section.  Indeed, most of what we discuss in this installment centers on understanding behavior.  The best way to behaviorally describe an adversary is by how he or she does his job – after all, this is the only discoverable part for an organization that is strictly CND (some of our friends in the USG likely have better ways of understanding adversaries).  That “job” is compromising data, and therefore we describe our attacker in terms of the anatomy of their attacks.

Ideally, if we could attach a human being to each and every observed activity on our network and hosts, we could easily identify our attackers, and respond appropriately every time.  At this point in history, that sort of capability passes beyond ‘pipe dream’ into ‘ludicrous.’   However mad this goal is, it provides a target for our analysis: we need to push our detection “closer” to the adversary.  If all we know is the forged email address an adversary tends to use in delivering hostile email, assuming this is uniquely linked to malicious behavior, we have a mutable and temporal indicator upon which to detect.  Sure, we can easily discover when it’s used in the future, and we are obliged to do so as part of our due diligence.  The problem is this can be changed at any time, on a whim.  If, however, the adversary has found an open mail relay that no one else uses, then we have found an indicator “closer” to the adversary.  It’s much more difficult (though, in the scheme of things, still somewhat easy) to find a new open mail relay to use than it is to change the forged sending address.  Thus, we have pushed our detection “closer” to the adversary.  Atomic, computed, and behavioral indicators can describe more or less mutable/temporal indicators in a hierarchy.  We as analysts seek the most static of all indicators, at the top of this list, but often must settle for indicators further from the adversary until those key elements reveal themselves.  The figure below shows some common indicators of an attack, and where we’ve seen them fall in terms of proximity to the adversary, variability, and inversely mutability and temporality.

Fig 1: Indicator Hierarchy

That this analysis begins with the adversary and then dovetails into defense makes it very much a security intelligence technique as we’ve defined the term.  Following a sophisticated actor over time is analogous to watching someone’s shadow.  Many factors influence what you see, such as the time of day, angle of sun, etc.  After you account for these variables, you begin to notice nuances in how the person moves, observations that make the shadow distinct from others.  Eventually, you know so much about how the person moves that you can pick them out of a crowd of shadows.  However, you never know for sure if you’re looking at the same person.  At that point, for our purposes, it doesn’t matter.  If it looks like a duck, and sounds like a duck… it hacks like a duck.  Whether the same person (or even group) is truly at the other end of behavior every time is immaterial if the profile you build facilitates predicting future activity and detecting it.

Attack Progression, aka the “Kill Chain”

We have found that the phases of an attack can be described by 6 sequential stages.  Once again loosely borrowing vernacular, the phases of an operation can be described as a “kill chain.”  The importance here is not that this is a linear flow – some phases may occur in parallel, and the order of earlier phases can be interchanged – but rather how far along an adversary has progressed in his or her attack, the corresponding damage, and investigation that must be performed.

Fig. 2: The Attack Progression

Reconnaissance

The reconnaissance phase is straightforward.  However, in security intelligence, often times this is manifested not in portscans, system enumeration, or the like.  It is the data equivalent: browsing websites, pulling down PDF’s, learning the internal structure of the target organization.  A few years ago I never would’ve believed that people went to this level of effort to target an organization, but after witnessing it happen, I can say with confidence that it does.  The problem with activity in this phase is that it is often indistinguishable from normal activity.  There are precious few cases where one can collect information here and find associated behavior in the delivery phase matching an adversary’s behavioral profile with high confidence and a low false positive rate.  These cases are truly gems – when they can be identified, they link what is often two normal-looking events in a way that greatly enhances detection.

Weaponization

The weaponization phase may or may not happen after reconnaissance; it is placed here merely for convenience.  This is the one phase that the victim doesn’t see happen, but can very much detect.  Weaponizaiton is the act of placing malicious payload into a delivery vehicle.  It’s the difference in how a Soviet warhead is wired to the detonator versus how a US warhead is wired in.  For us, it is the technique used to obfuscate shellcode, the way an executable is packed into a trojaned document, etc.  Detection of this is not always possible, nor is it always predictable, but when it can be done it is a highly effective technique.  Only by reverse engineering of delivered payloads is an understanding of an adversary’s weaponization achieved.  This is distinctly separate and often persistent across the subsequent stages.

Delivery

Delivery is rather straightforward.  Whether it is an HTTP request containing SQL injection code or an email with a hyperlink to a compromised website, this is the critical phase where the payload is delivered to its target.  I heard a term just the other day that I really like: “warheads on foreheads” (courtesy US Army).

Compromise / Exploit

The compromise phase will possibly have elements of a software vulnerability, a human vulnerability aka “social engineering,” or a hardware vulnerability.  While the latter are quite rare by comparison, I include hardware vulnerabilities for the sake of completeness.

The compromise of the target may itself be multi-phase, or more straightforward.  As a result, we sometimes have the tendency to pull apart this phase into separate sub-phases, or peel out “Compromise” and “Exploit” as wholly separate.  For simplicity’s sake, we’ll keep this as a single phase.  A single-phase exploit results in the compromised host behaving according to the attacker’s wishes directly as a result of the successful execution of the delivered payload.  For example, if an attacker coaxes a user into running an EXE attachment to an email which contained the desired backdoor code.  A multi-phase exploit typically will involve delivery of shellcode whose sole function is to pull down and execute more capable code upon execution.  Shellcode often needs to be portable for a variety of reasons, necessitating such an approach.  We have seen other cases where, possibly through sheer laziness, adversaries end up delivering exploits whose downloaders download other downloaders before finally installing the desired code.  As you can imagine, the more phases involved, the lower an adversary’s probability for success.

This is the pivotal phase of the attack.  If this phase completes successfully, what we as security analysts have classically called “incident response” is initiated: code is present on a machine that should not be there.  However, as will be discussed later, the notion of “incident response” is so different in intelligence-driven response (and the classic model so inapplicable) that we have started to move away from using the term altogether.  The better term for security intelligence is “compromise response,” as it removes ambiguity from the term “incident.”

C2

The command-and-control phase of the attack represents the period after which adversaries leverage the exploit of a system.  A compromise does not necessarily mean C2, just as C2 doesn’t necessarily mean exfiltration.  In fact, we will discuss how this can be exploited in CND, but recognize that successful communications back to the adversary often must be made before any potential for impact to data can be realized.  This can be prevented intentionally by identifying C2 in unsuccessful past attacks by the same adversary resulting in network mitigations, or fortuitously when adversaries drop malware that is somehow incompatible with your network infrastructure, to give but two examples.

In addition to the phone call going through, someone has to be present at the other end to receive it.  Your adversaries take time off, too… but not all of them.  In fact, a few groups have been observed to be so responsive that it suggests a mature organization with shifts and procedures behind the attack more refined than that of many incident response organizations.

We will also lump lateral movement with compromised credentials, file system enumeration, and additional tool dropping by adversaries broadly into this phase of the attack.  While an argument can be made that situational awareness of the compromised environment is technically “exfiltration,” the intention of the next phase is somewhat different.

Exfiltration

The exfiltration phase is conceptually very simple: this is when the data, which has been the ultimate target all along, is taken.  Previously I mentioned that gathering information about the environment of the compromised machine doesn’t fall into the exfiltration phase.  The reason for this is that such data is being gathered to serve but one purpose, either immediately or longer-term: facilitate gathering of sensitive information.  The source code for the new O/S.  The new widget that cost billions to develop.  Access to the credit cards, or PII.

Analytical Approach

As we analyze attacks, we begin to see that different indicators map to the phases above.  While an adversary may attempt to use the exploit du jour to compromise target systems, the backdoor (C2) may be the same as past attacks by the same actor.  Different proxy IP addresses may be used to relay an attack, but the weaponization may not change between them.  These immutable, or infrequently-changing properties of attacks by an adversary make up his/her/their behavioral profile as we discussed in moving detection closer to the adversary.  It’s capturing, knowing, and detecting this modus operandi that facilitates our discovery of other attacks by the same adversary, even if many other aspects of the attack change.

This need for the accumulation of indicators for detection means that analysis of unsuccessful attacks is important, to the extent that the attack is believed to be related to an APT adversary.  A detection of malware in email by perimeter anti-virus, for instance, is only the beginning when the weaponization is one commonly used by a persistent adversary.  The backdoor that would have been dropped may contain a new C2 location, or even a whole new backdoor altogether.  Learning this detail, and adjusting sensors accordingly, can permit future detection when that tool or infrastructure is reused, even if detection at the attack phase fails.  Discovery of new indicators also means historical searches may reveal past undetected attacks, possibly more successful than the latest one.

Analysis of attacks quickly becomes complicated, and will be further explored in future entries culminating with a new model for incident response.

The Indicator Lifecycle

As a derivative (literary, not mathematical) of the analysis of attack progression, we have the indicator lifecycle.  The indicator lifecycle is cyclical, with the discovery of known indicators begetting the revelation of new ones.  This lifecycle further emphasizes why the analysis of attacks that never progress past the compromise phase are important.

Fig. 3: The Indicator Lifecycle State Diagram

Analysis // Revelation

The revelation of indicators comes from many places – internal investigations, intelligence passed on by partners, etc.  This represents the moment that an indicator is revealed to be significant and related to a known-hostile actor.

Search & Tune // Maturation

This is the point where the correct way to leverage the indicator is identified.  Sensors are updated, signatures written, detection tools put in the correct place, development of a new tool makes observation of the indicator possible, etc.

Discovery // Utility

This is the point at which the indicator’s potential is realized: when hostile activity at some point of the kill chain is detected thanks to knowledge of the indicator and correct tuning of detection devices, or data mining/trend analysis revealing a behavioral indicator, for example.  And of course, this detection and the subsequent analysis likely reveals more indicators.  Lather, rinse, repeat.

In the next section, I will walk through a few examples and illustrate how following the attack progression forward and backward leads to a complete picture of the attack, as well as how attacks can be represented graphically.  Following that will be our new model of network defense which brings all of these ideas together.  You can expect amplifying entries thereafter to further enhance detection using security intelligence principles, starting with user modeling.

Michael is a senior member of Lockheed Martin’s Computer Incident Response Team.  He has lectured for various audiences including SANS, IEEE, and the annual DC3 CyberCrime Convention, and teaches an introductory class on cryptography.  His current work consists of security intelligence analysis and development of new tools and techniques for incident response. Michael holds a BS in computer engineering, has earned GCIA (#592) and GCFA (#711) gold certifications alongside various others, and is a professional member of ACM and IEEE.

Why certify at all?

Certifications are not intended to ensure that someone is awesome at their job, but that they pass the minimal qualifications for someone in the field.  Much like basic training teaches you the basics to fight in combat, but hardly makes you an Army Ranger.

For the sake of the profession, something similar to the bar or medical exams has to ensure that a basic set of knowledge exists for an entry level individual.  CPAs, doctors, lawyers, all need to pass a test.  However, the best professionals in those fields have the most experience.  However, in order to even begin the first day in those professions, they have to prove that they at least know enough not to make a critical error on day 1.

I know many smart lawyers or doctors.  However, none of them cannot do their jobs unless they passed their tests.  Their IQ does not matter.  You cannot fly a plane without passing tests.  In fact, you cannot drive a car without a license.  I know many people that can drive a car without it, but the test is geared to show you understand the basics of road safety and vehicle control.

That is the point of certification.

Professionalization for Digital Forensics

Unfortunately, licensing will be barreling down on our profession faster than you think for everyone in both information security and computer forensics.   There are bills in congress as well as legislative actions that are taking place in many states.

We live in a society where you need to be licensed to cut hair, be a plumber (Joe the unlicensed plumber) or babysit (Michigan).  Do we really believe that we will not need a license of any sort to do the job we love?

Good certifications are needed as a counter to that.  The organizational efforts of the CDFS are a part of that solution as well, but the states want educational/testable proof that someone doing the job has jumped through a couple of hoops so they are not snake oil salesmen.

For the profession overall to be recognized, certifications are needed.  Personally, I respect many certifications.  EnCE, CCE, the potential of the DFCA /DFCP , and the CFCE.  Last year I sent out a Common Body of Knowledge to over 80 practioners, the CBK comment process outline which skills are needed and which skills are “nice to have.” I received much feedback, but we need more people that we can reach out and involve in these discussions.

As a profession, we will need to become tested to perform our work.  It is not a matter of “If”, but “when”.

Your call on how we should get that license.  Leave it to biased industry groups such as the PI lobby or have digital forensic professionals (you and I) to decide together what the minimal qualifications are.

How many professions that have been around for a while do not have at least an entry-level test?

I personally am not advocating any specific certifications.  There are many good ones out there that are recognized, but professionals should consider certifications in their profession of choice.  Get certified to show we are a true profession.

Do we need to back only one certification now?

In my opinion no.  If we back one too soon, creativity and ingenuity will begin to languish.  We need the certifications to continue to evolve and become better.  Competition will do that for us.  However, having said that, I think all the certifications should understand that it is in our best interest to cross promote all the certifications.  We are in this together, that is the mantra of the CDFS.  For example, SANS , HTCIA, and ISFCE have routinely worked together.  The SANS digital forensic courses are certified as CCE Bootcamps even though we offer a competing certification?  Why?  The CCE certification might be more useful in your specific industry such as Law Enforcement vs Information Security. We respect their certification objectives and as a friend in the industry.

The key is understanding that the current discussion is not “Which Certification?”  The battle is “Should we certify at all?”  This is why I am adamant about pushing individuals to certify in a respected certification.  There are many I realize.    Get certified that will help you in your specific career in Law Enforcement, Litigation Support, or Information Security.

We need your help

Help us decide what the qualifications are needed for a minimally qualified professional in digital forensics we do not think we have the best idea, but we need to come together and help professionalize digital forensics.   Any additional ideas on how to foster professionalization in this community?  Send comments back to me at rlee@sans.org and Ill share thoughts periodically.

Help mold the future of your profession.

Rob Lee

__________________________________________________________________________

Rob Lee is a Director  for MANDIANT, a leading provider of information security consulting services and software to Fortune 500 organizations and the U.S. Government. Rob has over 13 years experience in computer forensics, vulnerability discovery, intrusion detection and incident response. Rob is the lead course author and faculty fellow for the computer forensic courses at the SANS Institute.

Several months ago, I blogged about using alternate superblocks to fake out the ext3 drivers so you could mount file system images read-only, even if they were needing journal recovery.  However, due to recent changes in the ext file system driver the method I describe in my posting is no longer sufficient.  Happily, there’s a quick work-around.

Let’s try the solution from the end of my previous posting under a more recent Linux kernel:

# mount -o loop,ro,sb=131072 dev_sda2.dd /mnt
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
 missing codepage or helper program, or other error
 In some cases useful info is found in syslog - try
 dmesg | tail  or so

This looks like the original error output we got when using the primary superblock.  Looking at the relevant dmesg output we see something different, however:

[163135.527484] JBD: Ignoring recovery information on journal
[163135.795917] Buffer I/O error on device loop0, logical block 0
[163135.795931] lost page write due to I/O error on loop0
[163135.795944] Buffer I/O error on device loop0, logical block 1
[163135.795949] lost page write due to I/O error on loop0
[163135.795958] Buffer I/O error on device loop0, logical block 2
[163135.795963] lost page write due to I/O error on loop0
[163135.795973] Buffer I/O error on device loop0, logical block 3
[163135.795977] lost page write due to I/O error on loop0
[163135.795986] Buffer I/O error on device loop0, logical block 18
[163135.795991] lost page write due to I/O error on loop0
[163135.795999] Buffer I/O error on device loop0, logical block 32
[163135.796034] lost page write due to I/O error on loop0
[163135.796232] Buffer I/O error on device loop0, logical block 73
[163135.796238] lost page write due to I/O error on loop0
[163135.796248] Buffer I/O error on device loop0, logical block 74
[163135.796253] lost page write due to I/O error on loop0
[163135.796261] Buffer I/O error on device loop0, logical block 94
[163135.796267] lost page write due to I/O error on loop0
[163135.796275] Buffer I/O error on device loop0, logical block 96
[163135.796280] lost page write due to I/O error on loop0
[163135.796516] JBD: recovery failed
[163135.796520] EXT3-fs: error loading journal.

It would appear that even though we’re using an alternate superblock that’s marked as not requiring journal recovery, the ext file system driver is still finding the uncompleted journal entries and trying to apply them.  This is arguably “more correct” behavior than the old driver used, but it doesn’t help us very much.

The simple work-around is to tell the ext file system driver to ignore the journal by forcing the file system to be mounted as ext2:

# mount -t ext2 -o loop,ro,sb=131072 dev_sda2.dd /mnt
# ls /mnt
bin   dev  home    lib         mnt  proc  sbin  usr
boot  etc  initrd  lost+found  opt  root  tmp   var

Excellent!  With this small modification our trick is working again.  Hurrah!

You might well wonder what happens if you just try to mount our image as ext2 without using the alternate superblock.  Unfortunately, simply mounting as ext2 is not sufficient because the primary superblock is still marked as needing journal recovery.  Though I wonder why this flag should be relevant to an ext2 file system, it’s enough to prevent the mount from happening.  So the result is that you need to both mount using an alternate superblock and (at least on modern Linux kernels) mount the file system as ext2 to stop the file system driver from looking at the journal.