This article is an introduction to understanding Intel’s assembly language while debugging a computer program.

Debugging in assembly can be a daunting task and not every developer likes to make sense of the mnemonics that appear on the screen.

While debugging, one is normally used to viewing local variables, parameters  and syntax highlighted code in the most user friendly manner.  This is the primary reason why many developers dislike debugging in assembly initially.

Why Debug In Assembly?

Debugging in assembly is not an optional skill to have.   Every developer encounters a situation where there is no other alternative other than cracking open the assembly code.  Here are a few reasons why a developer should get their hands dirty with this skill -

1. You have a binary and its corresponding code but the build environment is not functional.  This could be because of missing tools, unavailable compilers or the lack of understanding of complex scripts and steps needed to recreate the binary.
2. You are dealing with a release build only bug and the debug symbols are unavailable or you are dealing with a third party library and don’t have access to its source code.
3. You like to have fun

What You Need?

1. A debugger that can display assembly language (Visual Studio, WinDbg, gdb, XCode are capable debuggers).
2. A ready reference to the Intel instruction set.  Google works just fine.
3. Patience and maybe a link to this article.  :-)

The key to debugging in assembly is to understand how functions are called, parameters are passed, local variables accessed.  The code flow and logic can be understood by looking up the Intel instruction set .

The diagram below (click for larger image)  is roughly how a stack should look like when a function is called.  Mapping the assembly code to this diagram should simplify the debugging experience.  Note that the stack grows from top to bottom.

Passing Parameters To A Function

Parameters are the first to be pushed onto the stack.  The caller of the function pushes them from right to left.

Cleanup Of Parameters

Parameter cleanup may happen in the function that was called (e.g. stdcall calling convention) or by the caller of the function (e.g. cdecl).  Various kinds of calling conventions of the x86 architecture are explained in detail in this wiki article.   Stack cleanup code right after the function call gives a good idea of  the calling convention being used.

Take note that if cleanup happens in the function being called, variable number of arguments cannot be passed to it.  In contrast, variable number of arguments can be implemented if the parameter cleanup happens outside the function.  This essentially is the main difference between stdcall and cdelc calling conventions on a x86 architecture.

The Function Call

Before control is passed to the function, the return address (where the caller is supposed to resume after the function completes execution) is pushed on the stack.

To recap, parameters are pushed first, followed by the return address.  Later depending on the calling convention of the function, you may see cleanup happening right after the function call or within the function itself.

In the stack diagram above, as the stack grows downwards, the parameters are at the top followed by the return address.

Example

Take a look at the snippet and the equivalent assembly below :

[1] int a = 4;
[2] mov         dword ptr [a],4
[3]
[4] char c = 0;
[5] mov         byte ptr [c],0
[6]
[7] c = f(a, 22);
[8] push        16h
[9]   mov         eax,dword ptr [a]
[10]  push        eax
[11] call        f (4111D1h)
[12] add         esp,8
[13] mov         byte ptr [c],al

• In line [8], the constant value 22 (16 hex) is being pushed on the stack.
• In line [9-10] the variable ‘a’ is pushed.
• Line [11] is the function call to f() which implicitly pushes the return address.
• The return value of the function is passed in a register and copied in variable ‘c’ (line[13]).
• In line [12] one can see that 8 bytes are being discarded from the stack as f() uses the cdecl calling convention and cleanup needs to happen in the caller function.

The Function

The function usually starts with a prolog and ends with an epilog.

The prolog is either a simple ENTER instruction or more commonly, it saves the ebp register and copies the stack pointer value in it so as to use it as frame pointer.

push        ebp
mov         ebp,esp

The epilog just reverses what the prolog had done.  It is implented by a LEAVE instruction or is implemented as follows

mov         esp,ebp
pop         ebp

The ret instruction passes control from the function to the return address stored in the stack.

Compilers provide a switch that allows one to omit the frame pointer (fpo or frame pointer omission) which effectively removes the prolog and epilog and uses the ebp register for other optimizations.  It is easy to demarcate function boundaries where the frame pointer is present but one should be aware that the entry and exit points may be absent.

Allocation Of Local Variables

Local variables are allocated on the stack.  The total size of the local variables is computed at compile time and at runtime those many bytes are reserved on the stack.

004113A0  push        ebp
004113A1  mov         ebp,esp
004113A3 sub          esp,0E8h

Note that in the example above, 232 (0xE8) bytes are being reserved for local variables.

The above code will also help in understanding why local variable allocation is much faster than requesting memory from heap.   Allocating local variables requires moving the stack pointer whereas memory heap management is much more complex.

As the return address and local variable area are very close to each other, buffer overflows can be caused if data is written past the the local variable area which can then overwrite the the return address.   When this return address if forced to point to shell code (or to some other code placed intentionally by a hacker), such a buffer overflow is termed as an exploit.

Local Variables And Parameters In Assembly

The most important part in assembly is to be able to identify access to locals and parameters.  The frame pointer that is set in the prolog, acts as the reference pointer using which all variables can be accessed.  If you add to the frame pointer (Remember : P for Plus and P for Parameters), you will be able to access parameters.  If you subtract from the frame pointer, you will be able to access local variables.

mov         byte ptr [ebp-20h],3
mov         byte ptr [ebp+20h],5

The first line above accesses a local variable whereas the second accesses a parameter.

In the case of frame pointer omission, everything is calculated with respect to the stack pointer.  Therefore [esp + 20h] might refer to a local or a parameter depending on where the stack pointer currently points to.  And if say a register is pushed on the stack, the same variable will now be referred using [esp + 24h].  Debugging functions that have optimized out the frame pointer is not that easy as the changes made to the stack pointer need to be constantly tracked.

Conclusion

Debugging in assembly is not only fun but a useful tool to debug difficult problems.  Different debuggers provide different interfaces to interact with the assembly code but under the hood, all programs work alike.   Understanding this is the key to debugging in assembly with ease.

I have compiled a list of my favorite features in a debugger and how to invoke them on different debuggers (Visual Studio, XCode, gdb and Windbg). 

This is not a substitute for the debugger’s documentation but helpful for quickly switching to an unfamiliar debugging environment.  Click the image below for viewing the table or download the PDF version as a ready reference.

Debugger Cheat Sheet (Download PDF)

Please note :

• The commands (especially for gdb) are not necessarily complete and the debugger’s help should be consulted for detailed usage.
• The list is not comprehensive and I have only put in my favorite commands that I use while debugging.  
• A square bracket [ ] denotes a keyboard shortcut.

Most programmer’s restrict the definition of data breakpoints to breakpoints that help halting the execution of code in the debugger when memory is written to. This is the kind of breakpoint that helps in catching most of the corruption bugs.

GDB provides data breakpoints (at least on Intel platforms) that do slightly more than that. As data breakpoints are implemented through hardware assistance, it is the hardware platform that provides the different kinds of data breakpoints and the debuggers provide the required interface.

watch

watch is gdb’s way of setting data breakpoints which will halt the execution of a program if memory changes at the specified location.

watch breakpoints can either be set on the variable name or any address location.

watch my_variable
watch *0×12345678
where 0×12345678 is a valid address.

Usually a crash because of heap corruption or invalid outcome due to buffer overruns shows up in the debugger when it is too late to figure out what went wrong. The watch or write data breakpoints can be used to find when a memory location has changed. The debugger at that very instant shows the reason for the inadvertent change.

The cause usually is double deletion of memory, writing to deleted memory, writing past the buffer boundary, etc. In order to fix such issues, it is more important to know when the corruption happens than to know what happens when the corruption has taken place.

Another interesting use of write data breakpoints is to find out the cause of memory leaks in reference counted objects by monitoring the increase and decrease in the reference count of the objects.

rwatch

rwatch (read-watch) breakpoints break the execution of code when the program tries to read from a variable or memory location.

rwatch iWasAccessed
rwatch *0×12345678
where 0×12345678 is a valid address.

For example, say you are new to your project and would like to figure out where exactly are your encryption routines in code. As a start, you can first search your codebase but you may hit a few false positives. If you know the memory location of your password and you set your rwatch breakpoints correctly, it would not be long before the debugger breaks execution right in your encryption algorithms which have to read the password in order to perform their function. Getting evil ideas already?

Another use of read data breakpoints is finding out the code that is reading from memory that has already been deleted and is using this corrupt information later.

awatch

awatch or access watches break execution of the program if a variable or memory location is written to or read from. In summary, awatches are watches and rwatches all in one. It is a handy way of creating one breakpoint than two separate ones.

awatch *0×12345678
where 0×12345678 is a valid address.

The problem with data breakpoints like any other breakpoint is that they can be triggered far too many times and the programmer may lose track of the problem being debugged. To ensure the breakpoints are hit the least number of times, the data being worked upon should be minimal and the breakpoint should be set as late as possible.

It is very useful to be able to break execution of code at a line number of your choice.  Breakpoints are provided in debuggers to do exactly that.  It is fun getting to the root of the problem by setting breakpoints in a debug session.  It is even more fun to know how do breakpoints work in the first place.

Software breakpoints work by  inserting a special instruction in the program being debugged.  This special instruction on the Intel platform is “int 3″.  When executed it calls the debugger’s exception handler.

Example

Let us look at a very simple example that inserts a breakpoint in a program at compile time and not through a debugger.  The code uses the Intel instruction “int 3″ and you may need to figure out the equivalent instruction for a non-Intel platform.

// The code below works well with Visual Studio.
int main()
{
__asm int 3;
printf("Hello World\n");
return 0;
}

// The code below works well with gcc + gdb
int main()
{
asm("int $3");
printf("Hello World\n");
return 0;
}

If you run this program in Visual Studio, you get a dialog saying “helloworld.exe has triggered a breakpoint“.

In gdb you get the message “Program received signal SIGTRAP, Trace/breakpoint trap.”

In the example above, a call to “int 3″ invokes the debugger’s exception handler.

It is also interesting to note the assembly instructions generated for the program above.

In Visual Studio, right click on the code and click on “Show Disassembly”. Also ensure that “Show Code Bytes” is on in the same context menu.

In gdb type disassemble at the gdb command.

0×0040107a <main+42>:   int3

Now obtain the opcode of the int3 instruction using the x (examine memory) command

(gdb) x/x main+42

0×40107a <main+42>:     0xcc

As seen above, the breakpoint opcode we inserted during compilation is 0xCC .

How Do Debuggers Insert Breakpoints?

For practical reasons, it is unwise to ask for a recompilation whenever a breakpoint is added or deleted.  Debuggers change the loaded image of the executable in memory and insert the “int 3″ instruction at runtime.  The common steps a debugger performs to provide the functionality of a line breakpoint to a user are as follows -

1. When a user inserts a breakpoint in a line of code, the debugger saves the opcode at that given location and replaces it with 0xCC (int 3).
2. When the program is run and it executes the “int 3″ instruction, control is passed to the debugger’s exception handler.
3. The debugger notifies the user that a breakpoint has been hit. Say that the user instructs the debugger to resume execution of the program.
4. The debugger replaces the opcode 0xCC with the one it had saved earlier.  This is done to restore the instructions to their original state.
5. The debugger then single steps the program.
6. It then resaves the original instruction and re-inserts the opcode 0xCC.  If this step were not done, the breakpoint would have been lost.  Temporary breakpoints on the other hand skip this step.
7. The debugger then resumes execution of the program.

Hardware breakpoints are limited in number but debuggers are able to provide unlimited breakpoints by implementing them through software.

Knowing what goes behind the scenes makes debugging a bit easier.  A debugger may defer setting a breakpoint if the module is not loaded in memory yet.  It needs to replace some opcode with 0xCC and that can happen only when the module is in memory.  Likewise, a mismatch between a binary, its sources and its debug symbols (or the lack of it) may cause breakpoints to be hit at unexpected locations because the debugger is not able to correctly map the source line to the opcode that it needs to replace with 0xCC.  At times debuggers complain about the mismatch and refuse to set the breakpoints.

Many of the setup issues with breakpoints become obvious once we know how they work internally.  And when all else fails and release build breakpoints  adamantly refuse to work, you always have the option of compiling an “int 3″ breakpoint right into your code.

The good thing about a program that is thrashing a system is that it tends to be in the slow portion of the code for a long period of time.  So when it is bringing down the system, it is mostly executing the code that is responsible for the situation and all we want is to take a peek at the call stack at that very moment.  An obvious choice is to use a debugger or a profiler but given the low memory condition of the system, one may get little help from such tools.  When debugging or profiling become painfully slow, people may get evil ideas of reformatting their system or may start comparing their state of the art machines with ones they had five years back .  This article describes a light weight profiling trick where one can get the call stack of the unresponsive program without really loading the system further.

Procexp (Process Explorer) is a tool from sysinternals (now Microsoft) and both the download and documentation is available at the link here (Microsoft’s site).

The tool allows one to view the call stack of any running program on the system.  Below are the steps needed to display a call stack of any running program.

1. Launch procexp.
2. In the process tree, find your process that is thrashing the system.
3. Right click on “Properties” and select the “Threads” tab.    
       

    

    

    

    

   procexp threads tab

4. Sort the “Cycles Delta” (on Vista) or “CSwitch Data” (on Windows XP) column in descending order and select the topmost thread.  For some programs there might be just one thread.
5. For the selected thread click on the stack button to see the current call stack of the program.  Do note that this is a snapshot of the call stack and does not change dynamically.      
       

    

    

    

    

   procexp call stack

This call stack can provide a good insight of the area of the code that is causing the system to stall.  In the example above functiondothis() is where the thread is spending the most time.  Take more than one sample to reconfirm the findings.  This is a very unintrusive and light weight method of getting a call stack of a running program.  The same trick can be used to debug a hang but there a debugger works equally well. 

Sometimes you don’t need heavy debugging tools and sometimes you just can’t use them.  Procexp is a nifty debugging utility (in addition to being a process explorer) that a developer should download and keep handy for times when nothing else works.

Sample Usage

strace ./my_crashing_binary

strace ./my_crashing_binary 2>out.txt

strace -p 12345 2>out.txt

The strace Output

A sample output of strace is given below -

<snip>
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fe1000
mprotect(0x645000, 4096, PROT_READ) = 0
mprotect(0x51a000, 4096, PROT_READ) = 0
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7fe16c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
munmap(0xb7fe3000, 115297) = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 5), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fff000
write(1, "Hello World\n", 12) = 12
brk(0) = 0x940e000
brk(0x942f000) = 0x942f000
open("/myfile", O_RDONLY) = -1 ENOENT (No such file or directory)
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV (core dumped) +++


In the above example each line of the strace output has the system call, the parameters passed to it and the return value.

Take note that just before the above program crashed, it tried to access /myfile. With experience it is not hard to guess that the file is being accessed even though it was not found.  Either this file is absolutely necessary for the program to run or the required error checks are missing that could have averted the crash.

When Is strace Useful?

strace is very useful while debugging crashes.  Some scenarios where strace can be helpful are -

1. Debugging why an installation crashes on a machine.
2. Detecting file contention issues such as a file name clash while creating temporary files.
3. Debugging random crashes that are most probably due to the program running out of memory or due to it requesting an arbitrarily large chunk of memory.
4. Finding out how the program interacts with the file system.
5. Debugging crashes reproducibly only on one machine.
6. Debugging crashes in unfamilar code or in cases when sources are unavailable.

More Information

This article is meant as a pointer to the strace utility but it is not a substitute for the man page.  Go to the man page to explore all options strace provides.

man strace

strace is not a substitute for a debugger but a lightweight tool which can be used to generate logs, diagnose common problems and study the usage of system calls in a program.  Like any other program in a programmer’s toolset, it can save a lot of debugging time if used at the right time.

A useful feature in a debugger is a temporary breakpoint that automagically gets deleted when hit thereby reducing the clutter of unnecessary breakpoints.  These breakpoints are useful when you wish to stop at a code location only once and do not require the execution to stop at that location ever again. 

For example, say you are trying to determine whether a particular test scenario invokes a specific line of code or not, in that case a temporary breakpoint can be used as the breakpoint is not useful once it has been hit alteast once.

Below are steps on how to set temporary breakpoints in various debuggers.

gdb

Use the tb command to set a temporary breakpoint in gdb.  It is similar to the break command but the breakpoint will automatically be deleted when hit.

(gdb)help tb
Set a temporary breakpoint.
Like “break” except the breakpoint is only temporary,
so it will be deleted when hit.  Equivalent to “break” followed
by using “enable delete” on the breakpoint number.
    

Windbg

In Windbg, breakpoints set in the Command window using the bl /1 command can be used to create temporary breakpoints.  The /1 tells Windbg that the breakpoint should be deleted when hit.

In Windbg temporary breakpoints are also known as “one shot breakpoints”.

Visual Studio

I found it a bit painful to create temporary breakpoints in Visual Studio.  The only way I could create one was by setting a breakpoint and then setting the hit count for the breakpoint to be equal to 1.   The  article here explains how to set a hit count in Visual Studio.

The amount of work involved to do this sometimes doesn’t make temporary breakpoints worthwhile to set.  Moreover the breakpoint lingers on and doesn’t actually get deleted when hit.

A normal debugging session involving assembly language consists of the following steps -

1. Launch the program in gdb and set the required breakpoint(s).
2. When the breakpoint is hit, use the disassemble command to view the assembly language of the current frame.
3. Use the frame command and figure out the program counter and thus your location in the assembly that gdb produced in the last step.
4. Use the si and ni commands to step in and step over the instructions in assembly language.

The problem is that this may require a bit of scrolling to do if the disassembled function is very long.  The display command can be used to solve this problem.

The display command in gdb allows a user to configure the variables that should show up each time the program is suspended.

(gdb) help display
Print value of expression EXP each time the program stops.
/FMT may be used before EXP as in the “print” command.
/FMT “i” or “s” or including a size-letter is allowed,
as in the “x” command, and then EXP is used to get the address to examine
and examining is done as in the “x” command.

With no argument, display all currently requested auto-display expressions.
Use “undisplay” to cancel display requests previously made.

One can use the display command to view the next instruction that is about to be executed.  The pointer to the instruction yet to be executed is stored in the computer’s program counter (also known as the instruction pointer).   The following command will show the next 3 instructions that are pointed to by the program counter.

display /3i $pc

The /3i above causes three instructions to be printed and $pc references the program counter register.  A sample output is shown below.

1: x/3i $pc
0×401175 <main+37>:     call   0×4103f0 <__main>
0×40117a <main+42>:     lea    0xffffffe8(%ebp),%eax
0×40117d <main+45>:     mov    %eax,(%esp)
(gdb)

I then explored the Command Window in more depth and found out that it has some real nice commands which save you a lot of mouse clicks and time.

The Command Window is available through the shortcut CTRL + A or through the Menu ( View -> Other Windows -> Command Window). 

Visual Studio 2005 Command Window

Instead of the open command you could also have used the File.OpenFile command (analogous to Menu->File->Open File).  However the Command Window is not just an alternate way of accessing the menu. 

The Command Window provides the ability to use aliases and to create new ones.  This is what makes it very powerful and useful.  open in the above example is an alias for File.OpenFile.  It is easy to remember and type in. 

Visual Studio has a lot of predefined aliases and below are some that I found very useful.

alias     : lists all existing alias or creates a new alias.
cls       : clears the text from the Command Window.
disasm    : switches to disassembly view.
code      : switches to the code view.
bp        : toggle an existing breakpoint.
? <var>   : prints the value of the variable.
?? <var>  : displays the variable in the quickwatch window.
open file : opens the file and supports auto-complete.
nf        : new file.
closeall  : close all open windows.
|         : list processes.
~         : list threads.
r         : list registers.
g         : run program.
q         : stop debugging.

New aliases can be set by using the alias command and the auto-complete facilitates this.  They continue to persist even after you have exited Visual Studio.  For example, the following command will create an alias only  that will close all files except for the active one.

alias only File.CloseAllButThis

Many aliases in the Command Window do the task that can also be performed with keyboard shortcuts.  Aliases are not meant to replace keyboard shortcuts or menu items but provides an alternative way of debugging in Visual Studio which is intuitive to those who have worked on command line interfaces.  Some commands such as only  (created above) are more easily used through the Command Window whereas others might be more accessible through keyboard shortcuts. 

One mistake developers often make is to jumpstart the debugging process sooner than needed.  They fire up their favorite debugger, set the breakpoints and then start debugging when actually they should have first reduced the data set required for reproducing the bug.

The rule of debugging is very simple  -

“Every developer should strive to reproduce the bug by hitting the required set of breakpoints the least number of times. This allows faster and efficient debugging sessions.”

So if you are hitting a breakpoint hundred times more than you should have, then you are either not using the debugger efficiently or you have not worked enough to reduce the data required to reproduce the bug.

The latter problem is more common as sometimes developers put in little effort to reduce the input data and start debugging prematurely.   This leads to longer debugging cycles where a lot of time is wasted on investigating code that is not even relevant to the problem being solved.

For example, if a there is 1MB text file that is processed by your program and crashes it, then one should first try minimizing the text file such that if it is reduced any further, the crash goes away.  The text file obtained in such a manner is the smallest input data set required to reproduce the crash.  Once this goal is achieved, the program will now be processing lesser data and effectively less code will be executed which results in important breakpoints being hit fewer times.

Well how does the developer go about reducing the input data set?  There is no single method but the one that commonly works is to run a binary search on the input data.  For example in the above case, the text file should be split into two.  Two text files of half MB each would be obtained and one should test to see if the crash reproduces with either one of the files.   If you still see the crash, you have halved the data set.  Then the smaller file should be again split into  two and the process should be repeated till a very small text file is obtained that still causes the crash to occur.  Depending on what your program does, you can even reduce the 1 MB text file to a single character file.  Debugging your program with a single character file is much simpler than using the initial 1MB file.

The Caveat

Sometimes a large data set may have multiple issues.  By reducing it as described above, one may solve a partial problem but other problems may go unnoticed in the reduced set.  Therefore once a bug has been solved on a reduced input data set, it should be tested against the one provided with the original bug.  This ensures that no other issue that should have been fixed got ignored in a bid to made debugging more efficient.

Final Note

Reducing the input data is essential before starting the debugging process and a great productivity aid too.  If possible, this should be a part of the bug reporting process for quality engineers or customers who often log the issue.    Reducing the data may not always be possible but it is certainly worth an attempt.