Push the Red Button

What I Did on My Summer Vacation

2011-09-06T12:34:00.000-05:00

Over the summer I worked at Microsoft Research, which has a fantastically smart bunch of people working on really cool and interesting problems. I just noticed that they've posted the video of my end-of-internship talk, Monitoring Untrusted Modern Applications with Collective Record and Replay. Please take a look if you're curious about what it might look like to try and monitor mobile apps in the wild with low overhead!

Paper and Slides Available for "Virtuoso: Narrowing the Semantic Gap in Virtual Machine Introspection"

2011-05-28T02:34:00.004-05:00

I've recently returned from Oakland, CA, where the 2⁵ IEEE Symposium on Security and Privacy was held. There were a lot of excellent talks, and it was great to catch up with others in the security community. Now that the conference is over, I'm happy to release the paper and slides of our work, "Virtuoso: Narrowing the Semantic Gap in Virtual Machine Introspection", which I have described in an earlier post.

The slides contain some animations, and so I've made them available in three formats:

You can also get a copy of the full paper here. I'm also hoping to have the source ready for release soon; when it is available, you'll be able to find it on Google Code under the name Virtuoso.

Once again, thanks to my most excellent co-authors at MIT Lincoln Labs and Georgia Tech for helping me see this project through!

Applying Forensic Tools to Virtual Machine Introspection

2011-04-06T16:36:00.002-05:00

I've just released a technical report summarizing some work I did a couple years ago that explores how forensic memory analysis and virtual machine introspection are closely linked.

Abstract: Virtual machine introspection (VMI) has formed the basis of a number of novel approaches to security in recent years. Although the isolation provided by a virtualized environment provides improved security, software that makes use of VMI must overcome the semantic gap, reconstructing high-level state information from low-level data sources such as physical memory. The digital forensics community has likewise grappled with semantic gap problems in the field of forensic memory analysis (FMA), which seeks to extract forensically relevant information from dumps of physical memory. In this paper, we will show that work done by the forensic community is directly applicable to the VMI problem, and that by providing an interface between the two worlds, the difficulty of developing new virtualization security solutions can be significantly reduced.

You can read the full paper on SMARTech. Hopefully this will encourage others to start using great memory analysis tools like Volatility for live analysis of virtual machines!

Automatically Generating Memory Forensic Tools

2011-03-15T15:36:00.008-05:00

Now that the IEEE Symposium on Security and Privacy program has finally been posted, I can describe some research I've been working on for the past year and a half related to virtual machine introspection (VMI) and memory forensics.

A well-known problem with VMI and memory forensics is the semantic gap -- basically, the kind of information you want out of a memory image or a running VM is high level information (what processes are running, what files are open, and so on) but what you get is a big bunch of uninterpreted bytes (i.e., a view of physical memory). Bridging this gap is what tools like Volatility were built to do, and they do it well.

However, building a tool like Volatility takes a lot of work and a lot of knowledge about the internals of the operating system you're trying to examine. With operating systems like Windows, which are closed source, this kind of knowledge comes from things like the Windows Internals book, blog posts, and good old fashioned reverse engineering. This takes a lot of time, and the process has to be repeated every time there's a new version of Windows or a new operating system you want to support. Volatility's next release will support Vista and Windows 7, but it hasn't been easy – the networking code, for example, was rewritten for Vista, which required some reverse engineering by MHL and a new plugin.

Is there an easier way? What we want, in an ideal world, is some way that we can generate some of these tools automatically, for any OS or version. That's the problem that we set out to solve, and it's one that I think we made some good progress on -- though as with any academic work, there's still lots of room for improvement :)

The basic idea is that many of the tools we want to run on a memory image could be easily coded if we had access to the native APIs on the system – for example, we could easily write something similar to pslist if we had access to the Windows API by doing something like:

Our system, which we call Virtuoso, takes advantage of this fact. We take small programs like the one shown above and run them inside a virtual machine that logs every instruction they execute, both in user-mode and in the kernel. From these logs, we can then automatically generate Volatility plugins that do the same thing. Of course, I'm omitting a lot of technical detail here – there's a lot of work that needs to be done to clean up the logs, cut out irrelevant parts of the computation, and reconstitute the logs back into something that resembles a program – but that's the core idea.

In our paper, we show off our technique by automatically generating 6 different programs on Linux, Windows, and Haiku. These programs do things like list the PIDs of currently running processes, enumerate loaded kernel modules, and retrieve the executable name for a given PID, and didn't require any special knowledge to create: we just looked up the API functions that did what we wanted and wrote small programs like the one shown above, then let Virtuoso do the hard work of creating a Volatility plugin.

In future posts, I'll go deeper into the technical methods used to achieve this. I'll also post the paper itself once the conference happens (after all, I have to give people some reason to come and see the talk ;) ). And finally, I'm hoping to release the code itself, once I get approval from the people that funded the research. For now, I'm going to employ a tactic known as "proof by screenshot", showing the steps involved in creating a plugin to list the PIDs of running proceses under Haiku. (Click any of the screenshots to see a larger version.)

First we write a program that uses the Haiku API to get a list of running processes. We annotate the program with some markers that tell our logging engine where to start and stop the trace, and what the inputs and outputs are (the calls to vm_mark_buf_{in,out}):

We now compile and run that program inside a virtual machine running Haiku, and log what computation it does:

Next, we run our analyzer on it, which does its magic and produces a plugin for Volatility:

Finally, we can run that plugin within Volatility to analyze a Haiku memory image:

To wrap things, up, I want to thank my co-authors Tim Leek, Michael Zhivich, Jonathon Giffin, and Wenke Lee. It's been a long road, but I'm hoping this research will make it a lot easier to build exciting new security tools for VMI and memory forensics!

GDI Utilities: Taking Screenshots of Memory Dumps

2010-07-15T18:30:00.006-05:00

I've posted about this before (twice!), but somehow never gotten around to releasing functioning code. Here (click), for your downloading pleasure, is a set of plugins designed to extract information about on-screen (graphical) windows from Windows XP SP2/3 memory images. This includes:

window_list - give a text listing of the window hierarchy, with each window's on-screen coordinates, current style, and its class (Button, Window, etc.). Here's some example output to whet your appetite.
screenshot - save a wireframe "screenshot" of the on-screen windows in a memory image. See later in this post for some examples. Requires PIL.
wndmon - continuously monitor a memory image and provide an updating view of the on-screen windows. Works best in a live environment, e.g. with XenAccess and PyXa. Requires PyGame. (This is what I used for the video demo).

All three plugins require the distorm disassembly library to work. I had a bit of trouble getting it to work under Linux, so here's the steps required so you don't have to go through the pain:

Get distorm3 from its Google Code site.
Go into build/linux and type "make".
Copy the resulting libdistorm3.so into the Python directory.
Rename the Python directory to "distorm" and move it somewhere in your Python path (I use Debian, and found that /usr/local/lib/python2.6/dist-packages/ worked well).

Hopefully this is a bit simpler under Windows, but I don't have a Windows box handy so I can't test that at the moment.

Have fun with the code. If you go exploring in the source, you may find some interesting things -- there's more functionality there than is exposed through the plugins, including some functions and data structures that can extract HTML content from IE in memory... ;)

Anyway, to wrap things up, here's an example of the output from the screenshot plugin, running on the two NIST memory images:

From the 6/25 image:

From the 7/4 image:

And that, my friends, is the power of memory analysis.

Plugin Post: Robust Process Scanner

2010-07-06T13:17:00.012-05:00

It's pretty well known, in memory forensics circles, that there are two common ways of finding processes in memory images: list-walking, which traverses the kernel's linked list of process data structures, and scanning, which does a sweep over memory, looking for byte patterns that match the data found in a process data structure.

Having two different ways of finding processes can be very handy, especially when we suspect that someone may be trying to hide processes. One common way of hiding processes in Windows is called DKOM (Direct Kernel Object Manipulation); this technique works by just unlinking the process you want to hide from the kernel's list, like so:

This makes it invisible from programs such as the task manager, as well as memory forensic tools that use list-walking (including Volatility's pslist). However, such hidden processes can still be found by scanning memory using a signature for the process data structure; this is what psscan2 does.

Unfortunately, it's been known since at least 2007 (as mentioned in AAron Walters and Nick Petroni's Blackhat DC talk, and more recently in a presentation by Jesse Kornblum) that even signature scans can be evaded by crafty attackers. Signatures typically rely on "magic" values found in the process data structure. For example, in Windows XP, process data structures always begin with "\x03\x00\x1b\x00", which makes it pretty easy to find them in memory images.

But is that magic value really essential to the correct functioning of a process in Windows? What if an attacker just overwrites those four bytes with zeroes? As it turns out, Windows will be perfectly happy to keep running the process! At the same time, it will be completely hidden from existing forensic tools. What's more, as I demonstrated in my paper for CCS 2009 (Robust Signatures for Kernel Data Structures), around 51 fields in the process data structure can be manipulated by attackers in this way – including nearly all of the fields currently used to find processes.

So what's a forensic analyst to do? Luckily, there are some parts of the process data structure that are hard for an attacker to mess with without causing one of these:

So if we can build a signature based on these fields, we can find processes that existing signature scanners might miss.

And that's just what I've done. Here, for your consideration and consumption, is the creatively-named psscan3 (just drop it into the memory_plugins directory of Volatility 1.3.2). It uses a only fields that have been identified as "robust" to locate processes in Windows memory. It's a bit slower than the existing scanners, right now, because it's checking for more things.

If you want to try it out, you might also want to download this sample memory image, which has a hidden process at offset 0x01a4bc20. In Volatility, pslist, psscan, and psscan2 all miss the process, but psscan3 detects it, as shown in this exciting screenshot (click to enlarge; the windows show, from left to right, psscan, psscan2, and psscan3) [EDIT: Blogger is for some reason refusing to link to the larger size; click here to view it]:

If you'd like a copy of the rootkit that hid this process (which is based on the FU Rootkit), send me an e-mail (but be warned that I probably won't be able to dig up the source until this fall).

So that's it! If you want to find out more about what went into this plugin, you're encouraged to check out my paper, or browse the slides from the talk at CCS 2009.

SANS Forensic Summit: Thoughts and Slides

2009-07-11T21:28:00.010-05:00

This past Tuesday I attended the 2009 SANS Forensic Summit. In part, I was there to give a talk on combining volatile memory analysis with forensic analysis (see below for the slides from that), but I was also pretty excited about getting to hang out with the bright lights of the forensics community like Harlan Carvey, Chris Pogue, Richard Bejtlich, and many more.

Unfortunately, I was only able to attend the first day, which consisted primarily of technical talks on various aspects of forensics, incident response, and live forensics. All the talks were really excellent; Rob Lee and the folks at SANS should be commended for their great work in putting everything together. In this post I'm going to just describe the talks, rather than the panels; unfortunately I forgot to take notes during the panels and so I don't have as much to say about them, other than that they were fun and highly informative.

On to the talks! The first talk of the morning was Richard Bejtlich's keynote, which gave a really great analysis of the current state of the industry and the challenges faced by investigators today. He drew heavily from the Verizon Data Breach Investigations Report, which gave his assertions a nice feel of solidity to them; for example, when he says that we're in bad shape (getting compromised left and right), he can back that up with statistics showing that most intrusions are discovered only through third party notifications. If you're not already reading Richard's posts over at TaoSecurity, I highly encourage it.

After the keynote, Kris Harms got up to talk about live response. He gave a lot of cool tips on how to use some standard tools that most people should be familiar with (pslist, handles, etc.) to quickly triage a system and make a determination on whether it needs deeper analysis. I have to admit that I don't usually think a lot about live analysis--from a standpoint of simply collecting volatile data, I think that memory forensics offers a much better solution. However, from a triage perspective, live analysis makes a lot of sense; you can get a lot of leads very quickly by just knowing how to poke around on the live system.

Nevertheless, I did have one quibble with this talk. It seemed like a lot of the techniques presented, while cool, were a little haphazard. That is, "poking around" isn't necessarily repeatable, which means that as an investigator you could end up missing data by performing a different set of actions on different cases. After all, we're only human, and sometimes we forget things. I personally prefer to make sure that anything I'm going to do more than once is scripted. This allows one to codify an investigative procedure so that it's consistent and repeatable -- think of it as an executable checklist.

For example, in the presentation, Kris Harms described finding hidden processes by using handle.exe and pulling out the PIDs of each handle table it finds (Harlan Carvey now has a nice perl script that automates this). However, there are several rootkit detectors (such as IceSword) that will do this handle table vs. process list cross-view for you. I think we should definitely learn about these techniques and how they work, but I don't see the point in trying to keep them all in your head and do them by hand each time -- put it in a script and let the computer do the work.

After lunch, Harlan Carvey got up to talk about Windows registry analysis, a field that he did a lot of pioneering work in and essentially dominates. Things got a little hectic at the end, as he raced through some information-dense slides on specific kinds of forensic information you could get out of the registry, but overall I really found the talk engaging and illuminating. It also served as a really great motivator for my own talk: he spent a while near the beginning talking about volatile registry data and some of the reasons it's important. This set me up very nicely, since my own presentation was all about extracting registry data from memory. And I didn't even have to bribe him (much)!

Ending out the day (for presentations, at least) was a combined, hour and a half long session on memory analysis with Jamie Butler, Peter Silberman, and me. Peter and Jamie gave a great talk on Memoryze, which is Mandiant's free (as in beer) tool for analyzing volatile memory. Although most of the stuff presented was nothing new if you've been following memory analysis research, it was nice to see their software in action. They also announced the release of a new version of Memoryze, which supports Vista more fully, including the reworked networking code. Peter and Jamie are both very smart, and while I personally prefer Volatility for my own work, I'm glad that people have great options like Memoryze and Volatility to choose from.

Finally, after Jamie and Peter, I gave my own talk on combining registry analysis with memory forensics. There wasn't much new research presented in the talk, but I think it serves as a nice introduction to the toolset for people that haven't seen it before. The slides are available at the bottom of this post (assuming I can get this embedding thing to work), and I'll let them speak for themselves. :)

Once again, a huge thanks to Rob Lee and everyone who organized and attended the SANS Forensics Summit 2009! If you missed it this year, I hope this post has given you a taste of some of the great stuff that goes on there, and will encourage you to go next time!

SANS Forensics 2009 - Memory Forensics and Registry Analysis

View more documents from mooyix.

Odds and Ends

2009-07-01T08:50:00.004-05:00

I've been too busy to do any longer entries recently, but I wanted to note a couple things quickly.

First up, Andreas Schuster has just released a wonderful set of slides on using Volatility to do memory forensics. The slides include:

Great background material on the how, what, and why of memory acquisition and forensics.
A refresher on some OS basics you need to really understand memory analysis.
An amazing and comprehensive walkthrough on how to use a number of Volatility modules plugins in an investigation (including a few of my own tools, like ssdt.py and VolReg).
Great information on the internals of Volatility, including a tutorial on creating your own plugins.

This is really awesome stuff, and I highly recommend it to anyone looking to learn more about Volatility or even start contributing to the community with new plugins! Many thanks to Andreas!

Second, I wanted to let everyone know once again that I'm going to be speaking at the SANS WhatWorks Summit in Forensics and Incident Response in Washington, DC on how to combine registry analysis and memory forensics for more effective incident response. I'm really looking forward to this event, as it promises to bring together a lot of luminaries from the forensics community, such as Harlan Carvey, Jesse Kornblum, and Chris Pogue, as well some people with a lot of knowledge and experience with offensive techniques like Jamie Butler and Peter Silberman.

If you're planning on attending, or are in the DC area, drop me a note and perhaps we can meet up at the summit!

VolReg 0.6, now with BIG_DATA

2009-06-08T08:16:00.003-05:00

If you follow Matthieu Suiche's blog (and if you don't, you really should!), you probably saw his post about an undocumented type of registry value -- CM_BIG_DATA. This is a registry optimization introduced by Microsoft with Windows XP that allows for more efficient storage of large amounts of data in the registry.

You can read more about the details of this new way of storing large values in his post, but I wanted to announce the release of a new version of VolReg with experimental support for BIG_DATA values. As always, you can get the latest version of VolReg from my Volatility plugin page. This release also fixes a bug found by AAron Walters where an exception would be raised if the data returned for a value is less than the required amount for that type (e.g., only two bytes being available for a REG_DWORD).

I also updated Volshell, fixing a regression found by J. Hewlett that broke the ability to use the "dt" command to examine a data structure without overlaying it on memory. I also added the ability to pass the "db" and "dd" commands an address space, so that you can now get a hexdump or dword-dump of things like physical memory and registry hive spaces. The syntax for this is "dd(address, space=[addr_space])". More details are available in the online help, which you can read by doing "hh(dd)" or "hh(db)".

Windows 7 Registry Hives

2009-06-06T12:28:00.002-05:00

Tim Morgan was looking for some reference Windows 7 registry hives the other day to test reglookup, and it occurred to me that others might find them useful as well. So, without further ado, here's a link to download some registry hives I took from a fresh Windows 7 VM.

Also, in case you were thinking of being clever, the VM password was "password" ;)

Comprehensive New Resource on the Windows Registry

2009-05-21T10:59:00.000-05:00

Peter Norris was kind enough to point me at his Masters Thesis, The Internal Structure of the Windows Registry. Much of the information was previously available from a variety of sources (including this blog), but Peter's work also goes into a lot of unexplored territory, and doesn't shy away from the more esoteric aspects of the registry -- like how and when the Configuration Manager decides to update the on-disk copy of the registry with changes from memory.

Anyone who works with registry data for forensics or creates tools to work with the Windows registry would do well to give this a thorough read-through. Thanks to Peter for this excellent contribution!

Using Volatility for Introspection

2009-03-23T15:59:00.006-05:00

This post could also be titled "Teaser", part 2 :)

As part of my research at GT, I've been looking at using Volatility to examine the state of running virtual machines. Using PyXa, a wrapper around Bryan Payne's XenAccess library (available in the tools directory of the latest XenAccess release), you can get access to the memory of Xen guest VMs in Python. From there, it's just a small step to create a new address space that Volatility can use to examine virtual machines just as if they were any other memory image.

One application of this is using introspection to find out the state of windows on screen. This has advanced significantly since the last time I mentioned it, and it's now possible to track windows, including their z-order and some on-screen text, in near-real time. To demo this I used Volatility to examine the internal data structures of Win32k.sys and extract the locations and sizes of all visible windows, and then used PyGame to draw them on screen. The script keeps looping and re-drawing, allowing a near real-time view of what's going on inside the guest VM.

Here's a video of it in action. Notice how the reconstructed view updates to match what's actually going on on screen:

Cool, huh? :D

Quick Response to Guidance

2009-03-12T13:50:00.005-05:00

Everyone's been (rightfully) beating up on Guidance for their recent FUD-laden e-mail about F-Response; you should check out the excellent rebuttals by Harlan Carvey and hogfly. I'm not going to address most of the e-mail, but I thought I should tackle some things that are simply factually wrong about Volatility. I'm hardly an unbiased party (being a developer on Volatility), but unlike Guidance, I'm going to provide verifiable information to back up my claims.

From the Guidance mail:

While these utilities [included in the Volatility Framework] can identify running processes, open files and registry handles for running processes and open network sockets and connections, they cannot identify hidden processes, injected DLLs and NIC information.

There are three factual inaccuracies in this statement:

Volatility cannot identify hidden processes.
In fact, Volatility can list processes using multiple methods. The pslist module walks the list of active processess, and will find any processes that have not been hidden with DKOM. The psscan2 module scans memory looking for signatures of process data structures; it will find even those processes that have been hidden using DKOM-like techniques. Together, these can be used to find any hidden processes on the machine.
Volatility cannot find injected DLLs.
Michael Hale Ligh recently released a plugin called malfind that automates the work of doing exactly this. In fact, it is advanced enough that it can detect many types of generic code injection as well as simple DLL injection. I'm not familiar with Guidance's offering in this area, but I strongly suspect that they do not handle as many types of code injection.
Volatility cannot get NIC information.
Again, this is false. My own registry tools can examine the registry keys that describe the network interfaces on the machine. If this is too much work for an investigator, the recent integration with RegRipper will make this task even easier. In the example RegRipper log I posted, you can see the network card information has been gathered from the memory image.

So, the moral of the story is, don't believe everything you read, even when it comes from a large security vendor.

Updates and a New Home for Plugins

2009-03-03T17:14:00.008-05:00

As I've now released a number of plugins for Volatility, and some have gone through a couple revisions, I thought I'd put them all up on a single page, which can point to the latest versions and act as a sort of one-stop shop.

I've also updated the registry tools yet again, to fix some bugs and add new functionality, and also made some enhancements to volshell. You can read about the changes below:

Changes to VolReg:

New command hivedump: dump keys and timestamps (and optionally value data) from all hives to a CSV file.
Many improvements to robustness and error handling when reading key and value data.
When checking the registry hive names, catch exceptions and try to continue anyway (reported by chris).

Changes to volshell:

A new command, dis, is available. If distorm is installed, it will disassemble bytes from a given memory address as x86 code.
db no longer rounds length to a multiple of 4.
Use a single profile object throughout all commands (speed improvement)
dt can now overlay an arbitrary structure at an address, for example: dt('_EPROCESS', 0x81234567)

Enjoy!

RegRipper and Volatility Prototype

2009-03-01T16:40:00.039-05:00

When I first released the registry tools for Volatility, I discussed the possibility of interoperating with Harlan Carvey's excellent RegRipper. Now, thanks to Inline::Python and a bit of hackery, you can now run RegRipper against a memory image! Unfortunately, since Inline::Python only seems to work on Linux, you'll need to have a working Linux box around to use this (if anyone knows of a cross-platform way to use Python code from Perl, please let me know!).

I'll get to the details of how this works later, but for now let's talk about how you actually use this stuff. First of all, since we depend on Inline::Python to manage the unholy union of Perl and Python, you'll need to get it from CPAN or your distribution's package manager. No need to install Parse::Win32Registry; I've replaced it with my own registry code that will run against memory.

Next, you should download the latest version of the registry tools [tarball, zip] (side note: I updated them yet again, fixing a couple bugs and adding some basic checking to make sure you've got the right hives when using the credential dumping plugins), as well as the VolRip package [tarball, zip]. Unpack both into your Volatility directory.

Finally, you'll need a list of the virtual addresses of each registry hive in the image you want to examine. You can generate this list by first running the hivescan plugin to find the physical address of any hive, and then passing that to the hivelist plugin. You can see the results for the xp-laptop-2005-07-04 image here. For this example, we'll be using the SYSTEM hive, which is at address 0xe1035b60 in that image.

Now, let's run RegRipper! We're going to run the system plugins against the xp-laptop-2005-07-04 image, like so:

perl rip.pl -r xp-laptop-2005-07-04-1430.img@0xe1035b60 -f system

You can see the results here, if you can't follow along directly at home. If you're familiar with RegRipper, you'll notice that where we usually give the path to the registry hive, we've instead given the path to the memory image, and the virtual address of the hive we want to examine (here, the SYSTEM hive).

That's really about all there is to it! You use rip.pl just as you always would, but instead of giving it the hive filename, you instead pass the image and address of the hive, separated by an "@" sign. You can still run individual plugins using -p, use the "hive guessing" feature with -g, or run a whole batch of plugins using -f. Now, if you want to know more about how this works, read on...

The basic idea was to create a set of Python objects that emulated the interface provided by James Macfarlane's Parse::Win32Registry as closely as possible. This is what regwrap.py does: provides a translation layer between my own registry API and the one exposed by Parse::Win32Registry. Once we've done that, we can drop this emulated object into rip.pl, and the Perl code will run happily, thinking it's operating on normal hive files.

I had to make the following changes to rip.pl to get this to work:

Instead of importing Parse::Win32Registry, use Inline::Python to import my own wrapper class and bind it to the name "Parse::Win32Registry".
Comment out the lines that check to make sure the hive file exists -- since we're smushing the memory image and hive address into a single "filename", those checks will now fail.
Minor changes so that rip.pl works on Linux.

As you can see, not many changes were required. With just these modifications, most RegRipper plugins run perfectly.

However, there was still one small wart that I couldn't get rid of just by emulating the library. See, in Perl, you can call a function in two contexts: "scalar" and "array". This basically means, "is the function result being stored in an array, or in a scalar variable"--and functions can decide to do different things depending on what context they're called from. An example of this is the get_data method of Value objects in Parse::Win32Registry.

The get_data method, when called on a REG_MULTI_SZ value, will return an array of strings if called from array context, but returns a string with all the strings separated by spaces if called in a scalar context. Unfortuantely, the Python code has no way of telling what Perl context it's being called from, so my wrapper always returns a list. This means that a couple of plugins (shares.pl and nic_mst2.pl) would print "ARRAY(0x12345)" instead of the actual data in some cases.

Instead of trying to come up with a convoluted way around this, I opted to just change the two plugins in question to remove any ambiguity as to how the function was called. I'm still not sure I've identified all the plugins that do this (I tried to run them all to check, but may have missed something), but the results look good to me now.

So, to summarize: it's now possible to run RegRipper against memory images! The modifications required were pretty minimal, so it should be easy to keep things updated as new versions of RegRipper come out. In particular, aside from the two plugins I mentioned, upgrading the plugins should be as easy as just dropping them into the rrplugins directory.

Enjoy, and as always, let me know if you find any bugs :)

Teaser

2009-02-12T17:37:00.004-05:00

I don't have time for a full post right now, but I thought I'd offer a fun product of some things I've been working on recently. The work involves getting information about the windows on screen at the time a memory image was taken. One of the things you can extract is the position and size of each widget (called a "window", though I find this terminology a little confusing).

Since I don't have time to go into the data structures and so on involved, I thought I'd give you all two "screenshots" that I reconstructed from the NIST XP SP2 memory images. Basically it's a white canvas as large as the screen resolution, with rectangles drawn on for each window on the screen. Without further ado:

From the 6/25 image:

From the 7/4 image:

More details to come :)

Registry Code Updates

2009-01-23T12:17:00.004-05:00

I've found a couple bugs in the registry code I released recently, and at least one is significant enough that a new release is warranted. Teach me to release code I wrote in a couple hours on a plane ;)

The list of fixes is:

Fix a bug that prevented any volatile subkeys from appearing when using the subkeys() function.
Add a check for None when using lsadump (reported by Paul Bobby, thanks!)
Add appropriate license statements at the top of each file (thanks AAron!). For the record, the license is the GNU General Public License (GPL).

You can download the new version as a zip or tarball, and install it exactly as the previous version, by extracting it into your Volatility directory. If you have a previous version installed, this should just overwrite it (though you may have to tell your unzip program that's okay). As before, PyCrypto is required for the credential extraction modules.

One final note: I have seen some crashes when people attempt to use the hash extraction code, but pass the wrong address for the hive in memory. I'd like to fix this, but I don't yet have a good way of checking to make sure that a given hive is the SYSTEM or SAM or SECURITY hive. I'll try to find something that works, though, and release it with the next update.

Memory Registry Tools!

2009-01-16T14:44:00.009-05:00

Hi everyone! I know it's a bit late, but I made you all a Christmas present: tools for accessing registry data in Windows memory dumps. This the work that I presented at DFRWS 2008; it took a while to release because I had to find time to port it to Volatility 1.3.

To use them, grab either the zip or the tarball and extract it to your Volatility directory. You'll get the following new plugins (along with some supporting files):

hivescan: finds the physical address of CMHIVE structures, which represent a registry hives in memory
hivelist: takes a physical address of one CMHIVE, returns the virtual address of all hives, and their names
printkey: takes a virtual address of a hive and a key name (e.g., 'ControlSet001\Control'), and display the key's timestamp, values and subkeys
hashdump: dump the LanMan and NT hashes from the registry (deobfuscated). See this post for more details on how this is accomplished.
lsadump: dump the LSA secrets (decrypted) from the registry. See this post for more information.
cachedump: dump any cached domain password hashes from the registry. This will obviously only work if the memory image comes from a machine that was part of a domain. See this post for more details.

In general, when you start working with an image you should:

Use hivescan to find the physical address of a hive.
Pass one of those (any of them should do) to hivelist using the -o option. This will give you the virtual address of each of the hives in memory.
With this list in hand, you can now examine individual keys within a hive using printkey, or use hashdump, lsadump, or cachedump to extract credentials from the memory image. The latter three need the virtual addresses of specific hives (SYSTEM and SAM for hashdump, SYSTEM and SECURITY for lsadump and cachedump).

The full list of options available for each command can be obtained with --help. Note that hashdump, lsadump, and cachedump require PyCrypto in order to de-obfuscate the various credentials!

If you're curious how registry access works under the hood, you can read my DFRWS paper or presentation, or check out the blog posts I wrote while doing that research: Enumerating Registry Hives, Reading Open Keys, and Cell Index Translation. And, of course, you can always just look at the source code :)

Some final, closing thoughts. Harlan Carvey's RegRipper has been a huge sucess, and has shown a lot of people how much forensic information is stored in the registry. It would be awesome if this great tool could be applied to memory forensics, but unfortunately I have not yet figured out a way to integrate the two. However, this perl module looks promising; perhaps someone could write a Python wrapper around my registry code that would expose an interface compatible with Parse::Win32Registry, allowing RegRipper to run against a memory image with only slight modifications.

Of course, since RegRipper plugins are all written in Perl, you could also just see what they're doing and reimplement it as a Volatility plugin! To make that easier, here's a quick guide to using the new registry API in Volatility:

Start by opening a hive and assigning it to an address space. The HiveAddressSpace represents a hive in memory, and takes a virtual address space and offset to a hive as a base. Also, because the registry types are not part of Volatility's standard types, you need to add them in.
```
from forensics.win32.hive2 import HiveAddressSpace
from forensics.win32.regtypes import regtypes
types.update(regtypes)
syshive = HiveAddressSpace(addr_space, types, syshive_address)
```

Next, use the get_root() function to find the root key of the hive. You need to pass in the current profile.

from forensics.win32.rawreg import *
from forensics.object2 import Profile
theProfile = Profile()
root = get_root(syshive, theProfile)

To open a key, use open_key():

key = open_key(root, ["ControlSet001", "Control", "Lsa"])
print key.Name

To get a list of subkeys of a key, use subkeys():
```
for sk in subkeys(key):
    print sk.Name
```
To get the values of a key, use values():
```
for v in values(key):
    print v.Name
```

To parse the value data and return it in a more meaningful representation, use value_data():

for v in values(key):
   val_type, val_data = value_data(v)
   print v.Name, val_type, val_data

That's pretty much all you need to know! You can access other attributes of keys and values as members of the Python objects returned. The attributes available can be found in regtypes.py; keys are _CM_KEY_NODE objects and values are _CM_KEY_VALUE objects.

I don't have time to write any more, unfortunately, but I hope these will be useful to people! I'll try to post an example of a RegRipper plugin converted to a Volatility one soon so there's a concrete example to work with. Until then, enjoy and feel free to e-mail me (or leave comments) with any questions!

Plugin Post: Moddump

2009-01-06T00:45:00.000-05:00

By now, you all probably know that you can dump running programs from memory using the procdump module in Volatility. But not all malware runs as a user-mode process. What about malicious kernel modules?

As it turns out, dumping these is also quite straightforward, and it's easy to write a plugin to do it. In fact, it's downright trivial -- kernel modules are just PE files mapped into kernel memory (in exactly the same way as normal programs are PE files mapped into user memory). So to dump a particular kernel module, we can use Volatility's built-in PE dumper (the source is in forensics/win32/executable.py, and point it at the memory address of a kernel module.

Naturally, I've made a plugin that implements this: grab moddump.py and put it in your memory_plugins directory, and you'll be good to go. Here's what it looks like in action:

$ python volatility moddump --help
Usage: moddump [options] (see --help)

Options:
  -h, --help            show this help message and exit
  -f FILENAME, --file=FILENAME
                        (required) XP SP2 Image file
  -b BASE, --base=BASE  (optional, otherwise best guess is made)
                        Physical offset (in hex) of directory
                        table base
  -t TYPE, --type=TYPE  (optional, default="auto") Identify the
                        image type (pae, nopae, auto)
  -m MODE, --mode=MODE  strategy to use when saving executable.
                        Use "disk" to save using disk-based section
                        sizes, "mem" for memory-based sections.
                        (default: "mem")
  -u, --unsafe          do not perform sanity checks on sections
                        when dumping
  -o OFFSET, --offset=OFFSET
                        dump module whose base address is OFFSET (hex)
  -p REGEX, --pattern=REGEX
                        dump modules matching REGEX
  -i, --ignore-case     ignore case in pattern match

The "mode" and "unsafe" options do the same thing as in the procdump module. The other options are new though, so I'll go through them briefly here.

The -p option allows you to give a regular expression to specify which modules to dump. The -i option allows you to make this match case insensitive. If you don't give any way of specifying a module, it will simply dump all modules in the kernel's loaded module list.

Of course, if a driver is being stealthy, it could unlink itself from the kernel's list of loaded modules (just as processes can be hidden by removing from the systemwide process list). Often, these hidden drivers can be found by scanning memory for the data structure that represents a kernel module; Volatility's modscan and modscan2 are good for this. Once you've found the hidden module, you can pass its base address to moddump using the -o option.

Regardless of how you choose the modules to dump, they will be saved in a file called driver.BASE_ADDRESS.sys, where BASE_ADDRESS is the module's address im memory.

Anyway, I hope you'll get some good use out of this. It's also possible to create a similar plugin that dumps DLLs loaded in a process, but I haven't gotten around to writing it yet, so if anyone's looking to get their feet wet with writing plugins for Volatility it could be a fun first project.

Cheers and Happy New Year until next time!

Window Messages as a Forensic Resource

2008-09-21T13:17:00.073-05:00

In which window messages are explored, a new plugin is created, and we learn the importance of reading messages sent to you regularly.

In Windows, the GUI system is event-driven–actions occur in response to various events generated by the system. When you press a key on the keyboard, Windows generates a window message (specifically, WM_KEYDOWN) and sends it to the thread that owns the window that's in focus. That thread then call's the window's event handling procedure (the so-called WindowProc) to process the message. There are many such messages, covering input events such as keyboard and mouse actions, system-level events such as notification of a time change or a change in the system's power state, and events related to the windowing system, such as resizing or moving a window.

How can these be forensically relevant? Well, as it turns out, some threads in buggy applications aren't always good at processing their messages, and messages they receive pile up in the queue. This means that by looking at the message queues on a system, we can get some information about its past state. To make this more concrete, let's look at the message queue for a thread belonging to a certain full-disk encryption vendor on one of the images in my collection:

0:03:42.812,WM_WTSSESSION_CHANGE,WTS_CONSOLE_CONNECT
0:03:42,WM_WTSSESSION_CHANGE,WTS_SESSION_LOGON
1:45:05,WM_WTSSESSION_CHANGE,WTS_CONSOLE_DISCONNECT
1:45:06,WM_WTSSESSION_CHANGE,WTS_REMOTE_CONNECT
1:45:34,WM_WTSSESSION_CHANGE,WTS_REMOTE_DISCONNECT
1:46:19,WM_WTSSESSION_CHANGE,WTS_CONSOLE_CONNECT
2:28:50,WM_WTSSESSION_CHANGE,WTS_SESSION_LOCK
18:18:07,WM_WTSSESSION_CHANGE,WTS_SESSION_UNLOCK
22:05:51,WM_WTSSESSION_CHANGE,WTS_SESSION_LOCK
22:23:47,WM_WTSSESSION_CHANGE,WTS_SESSION_UNLOCK
23:32:34,WM_WTSSESSION_CHANGE,WTS_SESSION_LOCK
23:34:22,WM_WTSSESSION_CHANGE,WTS_SESSION_UNLOCK
1 day,  0:07:54.468,WM_WTSSESSION_CHANGE,WTS_SESSION_LOCK
[...]
8 days, 0:18:13.046,WM_WTSSESSION_CHANGE,WTS_SESSION_UNLOCK
8 days, 0:32:02.640,WM_WTSSESSION_CHANGE,WTS_SESSION_LOCK
8 days, 0:36:01.500,WM_WTSSESSION_CHANGE,WTS_SESSION_UNLOCK

(I've omitted many of the details here to save space. Window messages also include things like the handle of the window the message is for and cursor position at the time the message was sent.)

If we look up WM_WTSSESSION_CHANGE on MSDN, we find that it's a message related to fast user switching; one of these is sent whenever someone logs in, locks the screen, or connects remotely (i.e. via remote desktop). The message is sent to all applications that have called WTSRegisterSessionNotification. However, in this case, despite the fact that the application asked to be notified of such changes, it didn't bother to process the messages it received! This means that we can now look through its queue and find out when the user logged in and out, when the screen was locked, and so on. (The times given are relative to the time the system was booted, and are only good up to 49.7 days–this is because the timestamp comes from the GetTickCount function). It should be clear why such information might be useful in a forensic investigation.

I want to stress that we simply got lucky in this case by finding such a wealth of information in the message queue. It would be unwise to rely on every system having a misbehaving application like the one in this example; on the NIST images, for example, and on my own (clean) test VMs, there were no messages found on the system at all–the applications involved had processed them, and so they were no longer queued. Still, on real-world systems, buggy applications may be more common, so this trick could come in handy.

To look at the message queues in a memory dump, you can use a small plugin for Volatility that I've created. It enumerates all threads on the system and lists any messages it finds. It has an internal table mapping numeric IDs to names for a large number of standard window messages; however, it is very likely that the list is not complete. In addition, applications can define their own message types; in these cases, interpreting the message is impossible without analyzing the program. Caveats aside, I'm happy to offer threadqueues.py. As usual, drop it into your "memory_plugins/" directory, and then run it with:

$ python volatility thread_queues -f [IMAGE]

As with the SSDT plugin, thread_queues requires my list-walking library, which should be placed in the "forensics/win32/" directory.

If you want to know how the plugin works internally, read on!

Each thread in Windows (represented by the _ETHREAD strucutre) has a field in its Thread Control Block (or Tcb, which is a _KTHREAD) called Win32Thread. This field points to a data structure, _W32THREAD, which is defined in the kernel-mode portion of the Windows graphical subsystem, win32k.sys. You can actually examine the _W32THREAD structure by issuing "dt win32k!_W32THREAD" in WinDbg; however, if you start reverse engineering win32k.sys, you'll quickly find that the information given there is far from complete. In fact, _W32THREAD is a much larger data structure, which includes information about the current desktop, keyboard layout, installed window hooks, and, most importantly for us, the input message queue. You can read more about the internals of win32k in Alex Ionescu's BlackHat talk from this year; however, most of the details are still unknown, and the best way to find out about them is to reverse engineer win32k.sys.

In my case, I found the answers I wanted in the PostMessage() function. To make a (very) long story short, PostMessage is defined as:

PostMessage(PWND window, ulong msg, ulong wParam, ulong lParam)

When called, it eventually allocates a new entry in the thread queue using AllocQEntry, places the message information into it with StoreQMessage (which also adds the timestamp and current cursor position), and finally tells the thread that it should wake up because there's a message pending (using SetWakeBit). In Windows XP SP2, the message queue is found at offset 0xD0 of _W32THREAD, and looks like:

typedef struct _MSG_QUEUE {
PMSG_QUEUE_ENTRY Head;
PMSG_QUEUE_ENTRY Tail;
unsigned long NumberOfMessages;
} MSG_QUEUE;

Each queue entry, in turn, looks like:

typedef struct _MSG_QUEUE_ENTRY {
PMSG_QUEUE_ENTRY pNext;
PMSG_QUEUE_ENTRY pPrev;
struct _MSG {
 unsigned long hWnd;
 unsigned long ulMsg;
 unsigned long wParam;
 unsigned long lParam;
 unsigned long time; // In milliseconds
 POINT pt; // Mouse cursor position
 unsigned long dwUnknown;
 unsigned long dwExtraInfo;
 } Msg;
} MSG_QUEUE_ENTRY, * PMSG_QUEUE_ENTRY;

So for each thread, we can just start at the head of the queue, and keep following pNext until we get to the end, at which point pNext will be NULL.

One quick note of warning, though: when you're exploring these structures, you may initially think that all of the information is paged out, because none of the memory addresses seem valid. As it turns out, although Win32Thread and its friends all live in kernel space, this portion of kernel space is not visible from all threads. This flies in the face of a very common assumption in Windows memory analysis–that the kernel portion of memory looks the same to every process. In fact, the portions related to the GUI subsystem are only visible from processes that have at least one GUI thread! So, in particular, the System process, which is what's most commonly used in Volatility to access kernel memory, can't see any of the structures we're interested in. In my plugin, I make sure to use the address space of the process that owns the threads we're examining, so that the GUI structures will be accessible if the thread is a GUI thread.

There's a lot more cool stuff you can do once you start digging into the kernel mode portion of the Windows graphics system. In a future post, I'll describe how we can enumerate all the windows on the system, and list their titles and position, and (if I can figure out how) possibly even how to extract a screenshot at the time the memory was captured. Though this last bit doesn't have much forensic value, it would be pretty slick to be able to see exactly what was on the desktop for public images like the DFRWS challenge images or the NIST reference images.

Auditing the System Call Table

2008-08-20T17:48:00.009-05:00

When malicious, kernel-level code is installed on the system, one action it may take is to hook various system services. What this means is that it takes some standard piece of operating system functionality and replaces it with its own code, allowing it to alter the way all other programs use the OS. For example, it may hook functions involved in opening registry keys, and modify their output so as to hide registry keys the rootkit uses. As system calls are the primary interface between user and kernel mode, the system call table is a popular place to do such hooking.

It's worth noting that many security products also make heavy use of hooking. One common example is antivirus software; among the many functions it hooks is NtCreateProcess (used, as the name suggests, to start a new process) so that it can do its on-demand scanning of any newly launched programs. For this reason, it's not safe to assume that any hooking of system calls is malicious; in fact, some of the most suspicious-looking things initially often turn out to be security software.

Still, it may be quite useful to be able to examine the system call table of a memory image during an investigation, in order to detect any hooks that shouldn't be there. To do this, we'll first look at how system calls work in Windows and lay out the data structures that are involved. I'll then describe a Volatility plugin that examines each entry in the system call table, gives its symbolic name, and then tells what kernel module owns the function it points to. If you want to skip the learning experience and get straight to the plugin, you can download it here and place it in your memory_plugins directory. You'll also need to get my library for list walking and place it in "forensics/win32".

If you look at any of the native API functions, like ZwCreateFile, you'll notice that they all look almost identical:

lkd> u nt!ZwCreateFile
nt!ZwCreateFile:
804fd724 b825000000      mov     eax,25h
804fd729 8d542404        lea     edx,[esp+4]
804fd72d 9c              pushfd
804fd72e 6a08            push    8
804fd730 e83cf10300      call    nt!KiSystemService (8053c871)
804fd735 c22c00          ret     2Ch

We see that the function just places the value 0x25 into eax, points edx at the stack, and calls nt!KiSystemService. It turns out that this value, 0x25, is the system call number that corresponds to the CreateFile function.

Without going into too much detail about how KiSystemService works, the function essentially takes the value in the eax register, and then looks up that entry in a global system call table. The table contains function pointers to the actual kernel-land functions that implement that system call.

But, of course, the situation isn't quite as simple as that. In fact, Windows is designed to allow third party developers to add their own system calls. To support this, each _KTHREAD contains a member named ServiceTable which is a pointer to a data structure that looks like this:

typedef struct _SERVICE_DESCRIPTOR_TABLE {
SERVICE_DESCRIPTOR_ENTRY Descriptors[4];
} SERVICE_DESCRIPTOR_TABLE;

typedef struct _SERVICE_DESCRIPTOR_ENTRY {
PVOID KiServiceTable;
PULONG CounterBaseTable;
LONG ServiceLimit;
PUCHAR ArgumentTable;
} SERVICE_DESCRIPTOR_ENTRY;

As you can see, we can actually have up to four separate system service tables per thread! In practice, however, we only see the first two entries in this array filled in: the first one points to nt!KiServiceTable, which contains the functions that deal with standard OS functionality, and the second points to win32k!W32pServiceTable, which contains the functions for the GDI subsystem (managing windows, basic graphics functions, and so on). For system call numbers up to 0x1000, the first table is used, while for the range 0x1000-0x2000 the second table is consulted (this may generalize for 0x2000-0x3000 and 0x3000-0x4000, but I haven't tested it).

To take a look at the contents of these two tables, we can use the dps command in WinDbg, which takes a memory address and then attempts to look up the symbolic name of each DWORD starting at that address. To examine the full table, you should pass dps the number of DWORDS you want to examine -- the exact number will be the value found in the ServiceLimit member for the table you're interested in. For example:

lkd> dps nt!KiServiceTable L11c
805011fc  80598746 nt!NtAcceptConnectPort
80501200  805e5914 nt!NtAccessCheck
80501204  805e915a nt!NtAccessCheckAndAuditAlarm
80501208  805e5946 nt!NtAccessCheckByType
[...]
8050128c  8060be48 nt!NtCreateEventPair
80501290  8056d3ca nt!NtCreateFile
80501294  8056bc5c nt!NtCreateIoCompletion
[...]

Note that NtCreateFile is the 0x25th entry in the table, as we expected. On a system with no hooks installed, all functions in nt!KiServiceTable will point into the kernel (ntoskrnl.exe), and all functions in win32k!W32pServiceTable will be be inside win32k.sys. If they don't, it means the function has been hooked.

The plugin for Volatility, then, works as follows. First, we go over each thread in each process, and gather up all distinct pointers to service tables. We examine all of them in case one thread has had its ServiceTable changed while the others remain untouched. Then we display each entry in each (unique) table, along with the name it usually has (in an unhooked installation), and what driver the function belongs to. Here's some sample output:

$ python volatility ssdt -f xp-laptop-2005-07-04-1430.img
Gathering all referenced SSDTs from KTHREADs...
Finding appropriate address space for tables...
SSDT[0] at 804e26a8 with 284 entries
Entry 0x0000: 0x805862de (NtAcceptConnectPort) owned by ntoskrnl.exe
Entry 0x0001: 0x8056fded (NtAccessCheck) owned by ntoskrnl.exe
Entry 0x0002: 0x8058945b (NtAccessCheckAndAuditAlarm) owned by ntoskrnl.exe
[...]
Entry 0x0035: 0xf87436f0 (NtCreateThread) owned by wpsdrvnt.sys
[...]
SSDT[1] at bf997780 with 667 entries
Entry 0x1000: 0xbf93517d (NtGdiAbortDoc) owned by win32k.sys
Entry 0x1001: 0xbf946c1f (NtGdiAbortPath) owned by win32k.sys
[...]

Here we can see that the NtCreateThread function has been hooked by wpsdrvnt.sys. A little Googling shows that this driver is a part of Sygate Personal Firewall -- as mentioned before, security products are the most common non-malicious software that hooks kernel functions.

In closing, I should mention one caveat to using this tool: at the moment, the names of the system calls are hardcoded with the values derived from WinDbg on Windows XP SP2. As demonstrated by the Metasploit System Call Table page, the order and number of entries in the system call table change between different versions of Windows, so make sure that you only analyze SP2 images with this plugin! As always, patches are welcome if you want to adapt this to deal with other versions of Windows.

Now go forth, and catch those rootkits!

Introducing Volshell

2008-08-17T16:48:00.004-05:00

This one's for all the command line lovers out there: I'm happy to release volshell, an interactive shell built on Python and designed with memory analysis research in mind. I gave a demo of this at my OMFW talk, "Interactive Memory Exploration with Volatility"; since it was more of a live demo, I don't have slides from that, but you can find my notes here. You should be able to follow the notes as a sort of walkthrough that will get you up and running with volshell, and introduce some of the more advanced features.

Briefly, here are some of the features of volshell:

Shell is a full Python interpreter, so all the power of Python can be leveraged.
Uses Volatility 1.3 object model for easy access to data structures in memory.
Can use iPython for the underlying shell if available, which enables some nice features.
Commands modelled after WinDbg.
Works with any memory image format that Volatility supports (dd, crash, vmem, hibernation file)

To use it, just download volshell.py and drop it in your memory_plugins directory in Volatility 1.3. Then start the shell with:

$ python volatility volshell -f $IMAGE

Enjoy!

Linking Processes to Users

2008-08-16T17:20:00.000-05:00

In the course of an investigation, it may be critical to be able to link up a process that's running to a particular user account. Particularly in a multi-user environment such as Windows Terminal Server, this isn't always as easy as checking who was logged in at the time.

Luckily, each process in Windows has an associated token, a chunk of metadata that describes what Security Identifier (SID) owns the process and what privileges have been granted to it. As Larry Osterman explains, A SID is essentially a unique ID that is assigned to a user or group, and is broken into several parts: the revision (currently always set to 1), the identifier authority (describing what authority created the SID, and hence how to interpret the subauthoriries), and finally a list of subauthorities.

In general, when users see SIDs (which they rarely do), they are in what's called the Security Descriptor Definition Language (SDDL) form. This is a string that looks like:

S-1-5-21-1957994488-484763869-854245398-513

Here, "1" is the revision, "5" is the identifier authority, and the remaining portions are the subauthorities. The exact data structure for a SID is:

typedef struct _SID {
   BYTE  Revision;
   BYTE  SubAuthorityCount;
   SID_IDENTIFIER_AUTHORITY IdentifierAuthority;
   DWORD SubAuthority[ANYSIZE_ARRAY];
} SID, *PISID;

The SID_IDENTIFIER_AUTHORITY here is actually an array of 6 characters. However, at the moment only the final character is used. Osterman's article enumerates all the possible identifier authorities; for our purposes we will be focusing on the NT authority, which is {0,0,0,0,0,5}. This is the authority which describes accounts managed by the NT security subsystem.

So how can we get this information from a memory image? We start, as usual, by looking at the _EPROCESS structure. Inside it, we find the Token member at offset 0xc8. However, the member doesn't directly point to an object of type _TOKEN, as we might expect. Instead, it is described as an _EX_FAST_REF. These types of objects are basically an optimization used by Windows to store both a pointer to and object and the object's reference count, all in a single DWORD. In an _EX_FAST_REF, the last 3 bits are co-opted to encode the reference count of the object. To get the actual pointer, you can mask off the last 3 bits, like so:

token_address = proc.Token.Value & ~0x7

Now, on to the token itself. Each token contains a list of user and group SIDs. The relevant members of the _TOKEN structure (for our immediate purpose) are UserAndGroupCount (unsigned long) and UserAndGroups (pointer to array of _SID_AND_ATTRIBUTES), at offsets 0x4c and 0x68, respectively. _SID_AND_ATTRIBUES, in turn, contains a pointer to the SID itself and a DWORD of flags giving the SID's attributes (the meaning of which are dependent on the type of SID; for group SIDs, the flags can be found in winnt.h).

Unfortunately, just having the SIDs by themselves may not be so meaningful to you. Actual account names would be better; luckily, these can be found by looking in the registry. The key HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList contains all the local user account SIDs on the machine, along with the location of their profile on disk. The username can usually be inferred from this; e.g. a profile directory of %SystemDrive%\Documents and Settings\Bob would imply that the username is Bob.

Aside from the individual account SIDs, there are also a number of well-known group SIDs. These are SIDs that Microsoft has set aside for specific purposes, and will be the same on any Windows machine. A full list is available in KB243330, "Well-known security identifiers in Windows operating systems".

As a reward for sitting through all that dry description, here's a nice juicy tool that you can use to get started looking at the SIDs associated with a process in a memory image, written as a plugin for the just-released Volatility 1.3. You can download getsids.py here. To use it, just drop it in your plugins directory (memory_plugins) and run:

$ python volatility getsids -f xp-laptop-2005-07-04-1430.img
[...]
dd.exe (3300): S-1-5-21-1957994488-484763869-854245398-1006
dd.exe (3300): S-1-5-21-1957994488-484763869-854245398-513 (Domain Users)
dd.exe (3300): S-1-1-0 (Everyone)
dd.exe (3300): S-1-5-32-544 (Administrators)
dd.exe (3300): S-1-5-32-545 (Users)
dd.exe (3300): S-1-5-4 (Interactive)
dd.exe (3300): S-1-5-11 (Authenticated Users)
dd.exe (3300): S-1-5-5-0-49673 (Logon Session)
dd.exe (3300): S-1-2-0 (Users with the ability to log in locally)

Volatility 1.3 is out!

2008-08-16T15:50:00.002-05:00

After tons of hard work by a lot of people (including me), Volatility 1.3 has been released to the world at large. AAron has a blog post up with all the juicy details, including a list of new features. For my part, I want to take this opportunity focus on a couple new things that I think are really cool. They're mostly developer-focused, so if you have no interest in adding new capabilities to Volatility, you can skip the rest of this entry and just head over to AAron's post to see all the new modules and functionality.

The first feature I want to point out is the new plugin system. Basically, rather than creating a new module and then editing vmodules.py to add new commands to Volatility, you can now just create a class extending forensics.commands.command with the code you want to run, drop it into a file, and put that file into the memory_plugins directory, and Volatility will pick it up and see it as a new command that can be run. This means that anyone can just give out a single file and allow anyone to use it in Volatility with minimal effort.

In addition, it removes the need to manually integrate new modules from contributors into the source tree; instead, plugins can be developed and distributed independently, without relying on the Volatility devs at all. I'm hoping that this capability will allow a cool little cottage industry of plugin development to form around Volatility, in much the same way that users of EnCase currently trade EnScripts.

Second, I want to describe the new object model. Volatility 1.3 contains a new way of working with data structures in memory dumps. Each data structure found in vtypes.py can now be instantiated at a given memory address as a full-fledged Python object, and the data inside it can be accessed using standard Python syntax. No need to use read_obj again! For example, to print the size of a process located at address 0x823c87c0, we can do:

eprocess = Object('_EPROCESS', 0x823c87c0,
addr_space, None, profile=Profile())
print eprocess.VirtualSize

In addition, each structure can be given object-specific behaviors by subclassing the main Object class. For example, the _UNICODE_STRING type's Buffer member is a pointer to an unsigned short. By creating a specific _UNICODE_STRING class (as is done in memory_objects/Windows/xp_sp2.py), we can cause the Buffer member to be returned to the user as a Python string with the correct string data, automatically translated from Unicode.

Hopefully, with these new features, developing cool stuff for Volatility will be easier than ever. I know that for myself, I've found that it's now orders of magnitude faster to go from "Wouldn't it be neat if ..." to a full, working plugin. In the coming weeks, I'll be writing some posts introducing the new development features in more depth, so that as many people as possible can get involved. Remember, the power of Volatility is in its community!

Sorry for the Hiatus!

2008-08-15T16:30:00.008-05:00

It's been quite a while since I wrote any new blog posts. This isn't entirely because I've been lazy; rather, I've picked up and relocated to sunny (and often hot and humid) Atlanta, Georgia to start the PhD program at Georgia Tech. I'm going to be working on lots of cool stuff with the Georgia Tech Information Security Center.

Now that I'm here and starting to get settled in, you can expect the blog posts to start up again. Particularly with the forthcoming release of Volatility 1.3, I'm going to have a lot of new plugins and functionality to blog about.

As a teaser, here are some of the things I've got in the works:

getsids.py -- get the SID (kind of like a user ID in unix) that owns each process

moddump.py -- extract loaded kernel modules from memory

unloaded_modules.py -- list recently unloaded kernel modules

ssdt.py -- show the System Service Descriptor Table, along with the kernel module that owns the memory. This can be used to detect hooking, legitimate and otherwise.

volshell.py -- an interactive shell designed for exploration of memory images (presented at OMFW; note that this is aimed mainly at memory forensics researchers)

windowlist.py -- extracts a list of window handles and titles by using some reverse-engineered GDI structures in the kernel

I'm planning on accompanying these with posts describing the technical details of how they work. Also, as soon as I get the code ported to 1.3, I'll be releasing the code I wrote to extract registry information (as presented in my DFRWS paper).

I also spoke with Michael Cohen, the creator of PyFlag at DFRWS, and it sounds like he's interested in integrating in-memory registry support into PyFlag through Volatility. This will let users access the registry data in a memory dump through the PyFlag VFS, and perform queries and correlation on the registry data. This will be, I believe, "wicked awesome" (technical term).

Hopefully this has given you a taste of things to come, and gotten you good and excited about 1.3 (the amazing features of which I'll also be writing about soon). Stay tuned!