EurekaLog's blog

What my application is doing? (introducing Threads Snapshot Tool)

2012-05-21T17:02:00.002+01:00

Have you ever wondered what your application might do in this very moment? Have you tried to troubleshoot hangs and deadlocks?

Attaching debugger

Well, you can connect a Delphi debugger to your application. Just launch Delphi, open your project and use "Run" / "Attach to process" command. Select your process and click OK. Now you can use threads window (View / Debug windows / Threads) and call stack window (View / Debug windows / Call stack) to analyze state of your application. You may check what it's doing, check variables values, etc.

But what if debugger is not available? Sure, you can try to install and use Delphi remote debugger. What if this happens on customer's machine? Yeah, you can ask customer if he can allow you to connect to his machine, install remote debugger and analyze the case. That sounds not very comfortable...

Threads Snapshot Tool

EurekaLog 7 as well as standalone EurekaLog Tools Pack have a Threads Snapshot Tool, which can help you to peek inside running applications. Basically, it's a simple GUI exe with 1 helper DLL. You can pack it into .zip archive and send it to the customer, asking to run it on his machine. Then customer may send you log files back, so you can analyze them locally.

Here is how it looks:

UI of Threads Snapshot Tool

All you need to do is to select your running application and click on the button. This will ask you where to save log files and will take a snapshot of all threads inside selected application.

Log file was created

Here is a sample log from the hang inside some simple application:

Process [ 0E8C / 3724 ]: C:\Users\Александр\Documents\RAD Studio\Projects\RichEdit\richeditdemo.exe (2012.05.21 19:15:27)

[ 2CA8 / 11432 ] Priority: 8

EIP:    7727F8B1  EFlags: 00000246
EBP:    0018FB5C  ESP:    0018FAF0
EAX:    00000000  EBX:    00000000  ECX:    00000000  EDX:    00000000
ESI:    000006BC  EDI:    00000000
SegCS:  00000023  CegSS:  0000002B  SegGS:  0000002B  SegFS:  00000053  SegES:  0000002B  SegDS:  0000002B

DeadLock: True
Blocked waiting for thread [ 27DC / 10204 ]
thread -> thread: [ 27DC / 10204 ] is blocked -> SendMessage -> thread: [ 2CA8 / 11432 ] is blocked

-------------------------------------------------------------------------------------------------------------------------------------
|Methods |Details|Stack   |Address |Module          |Offset  |Unit        |Class           |Procedure/Method               |Line    |
-------------------------------------------------------------------------------------------------------------------------------------
|Running Thread: ID=11432; Parent=0; Priority=0                                                                                     |
|Class=; Name=                                                                                                                      |
|Comment=                                                                                                                           |
|-----------------------------------------------------------------------------------------------------------------------------------|
|7FFFFFFE|03     |00000000|7727F8B1|ntdll.dll       |0001F8B1|ntdll       |                |ZwWaitForSingleObject          |        |
|00000020|03     |0018FB60|764A118F|kernel32.dll    |0001118F|kernel32    |                |WaitForSingleObjectEx          |        |
|00000020|03     |0018FB78|764A1143|kernel32.dll    |00011143|kernel32    |                |WaitForSingleObject            |        |
|00000020|04     |0018FB8C|004E69A9|richeditdemo.exe|000E69A9|remain      |TMainForm       |FileOpen                       |351[5]  |
|00000020|03     |0018FBB4|0044AFAB|richeditdemo.exe|0004AFAB|Classes     |TBasicAction    |Execute                        |        |
|00000020|03     |0018FBBC|004658FD|richeditdemo.exe|000658FD|ActnList    |TContainedAction|Execute                        |        |
|00000020|03     |0018FBCC|004666B8|richeditdemo.exe|000666B8|ActnList    |TCustomAction   |Execute                        |        |
|00000020|03     |0018FBE0|00495F00|richeditdemo.exe|00095F00|Menus       |TMenuItem       |Click                          |        |
|00000020|03     |0018FBEC|00497413|richeditdemo.exe|00097413|Menus       |TMenu           |DispatchCommand                |        |
|00000020|03     |0018FBF4|004DB9EB|richeditdemo.exe|000DB9EB|Forms       |TCustomForm     |WMCommand                      |        |
|00000020|03     |0018FC00|00476D64|richeditdemo.exe|00076D64|Controls    |TControl        |WndProc                        |        |
|00000020|03     |0018FC18|0047B628|richeditdemo.exe|0007B628|Controls    |TWinControl     |WndProc                        |        |
|00000020|03     |0018FD7C|004D88FC|richeditdemo.exe|000D88FC|Forms       |TCustomForm     |WndProc                        |        |
|00000020|03     |0018FDA8|0047ACC8|richeditdemo.exe|0007ACC8|Controls    |TWinControl     |MainWndProc                    |        |
|00000020|03     |0018FDD8|0044B650|richeditdemo.exe|0004B650|Classes     |                |StdWndProc                     |        |
|00000020|03     |0018FE1C|750B6D35|USER32.dll      |00016D35|USER32      |                |(possible GetThreadDesktop+210)|        |
|00000020|03     |0018FE94|750B77BF|USER32.dll      |000177BF|USER32      |                |(possible CharPrevW+307)       |        |
|00000020|03     |0018FEF4|750B7885|USER32.dll      |00017885|USER32      |                |DispatchMessageW               |        |
|00000020|03     |0018FF04|004E182F|richeditdemo.exe|000E182F|Forms       |TApplication    |ProcessMessage                 |        |
|00000020|03     |0018FF20|004E1872|richeditdemo.exe|000E1872|Forms       |TApplication    |HandleMessage                  |        |
|00000020|03     |0018FF44|004E1B9D|richeditdemo.exe|000E1B9D|Forms       |TApplication    |Run                            |        |
|00000020|04     |0018FF74|004EDCE8|richeditdemo.exe|000EDCE8|richeditdemo|                |Initialization                 |15[4]   |
|00000020|03     |0018FF8C|764A3398|kernel32.dll    |00013398|kernel32    |                |BaseThreadInitThunk            |        |
-------------------------------------------------------------------------------------------------------------------------------------

[ 27DC / 10204 ] Priority: 8

EIP:    750B72B9  EFlags: 00000246
EBP:    02D0FEF8  ESP:    02D0FEB8
EAX:    00D59FB0  EBX:    02090F60  ECX:    00000000  EDX:    00000000
ESI:    00D59FB0  EDI:    00000000
SegCS:  00000023  CegSS:  0000002B  SegGS:  0000002B  SegFS:  00000053  SegES:  0000002B  SegDS:  0000002B

DeadLock: True
Blocked waiting for SendMessage owned by thread [ 2CA8 / 11432 ]
thread -> SendMessage -> thread: [ 2CA8 / 11432 ] is blocked -> thread: [ 27DC / 10204 ] is blocked

-------------------------------------------------------------------------------------------------------------------
|Methods |Details|Stack   |Address |Module          |Offset  |Unit    |Class      |Procedure/Method       |Line   |
-------------------------------------------------------------------------------------------------------------------
|Running Thread: ID=10204; Parent=0; Priority=0                                                                   |
|Class=; Name=                                                                                                    |
|Comment=                                                                                                         |
|-----------------------------------------------------------------------------------------------------------------|
|7FFFFFFE|03     |00000000|750B72B9|USER32.dll      |000172B9|USER32  |           |(possible GetPropW+146)|       |
|00000020|03     |02D0FEFC|750B96C0|USER32.dll      |000196C0|USER32  |           |SendMessageW           |       |
|00000020|04     |02D0FF20|004E6918|richeditdemo.exe|000E6918|remain  |TLoadThread|UpdateStatus           |336[1] |
|00000020|04     |02D0FF3C|004E68F6|richeditdemo.exe|000E68F6|remain  |TLoadThread|Execute                |331[1] |
|00000020|03     |02D0FF48|00448FDA|richeditdemo.exe|00048FDA|Classes |           |ThreadProc             |       |
|00000020|03     |02D0FF78|00406828|richeditdemo.exe|00006828|System  |           |ThreadWrapper          |       |
|00000020|03     |02D0FF8C|764A3398|kernel32.dll    |00013398|kernel32|           |BaseThreadInitThunk    |       |
-------------------------------------------------------------------------------------------------------------------

When you have such log file - can you determinate why the application is not responding?

Yes, now you can do it very easily. First, note that there are two threads, both of each is in a deadlock state (note "Deadlock: True"). Two deadlocks mean that these two threads are waiting for each other. Why they do it? You can determinate this either by looking at call stack or at wait chains.

Analyzing call stacks

Call stack for first thread indicate that main thread has called WaitForSingleObject function to wait for something inside it's TMainForm.FileOpen method. You can determinate what main thread waits by looking at source code:

WaitForSingleObject(LoadThread.Handle, INFINITE); // remain.pas, line 351

Second thread is a worker thread. It calls SendMessage from its UpdateStatus method. You can determinate which window will receive this message by looking at source code:

SendMessage(MainForm.Handle, WM_DoSomething, 0, 0); // remain.pas, line 336

As you can see, main thread created a worker thread to load file in the background and is waiting for its completion. And worker thread tried to update status of main window by sending a window message to it. Thus, you get a hang, because main thead is waiting for worker thread and it can not process message, which background thread has sent to it. And this is the reason for the hang.

Analyzing wait chains

And you can confirm this guess by looking at wait chains for each thread. Wait chain for main thread is:

Blocked waiting for thread [ 27DC / 10204 ]
thread -> thread: [ 27DC / 10204 ] is blocked -> SendMessage -> thread: [ 2CA8 / 11432 ] is blocked

Which means: "main thread (2CA8) is blocked by waiting for worker thread (27DC), which is blocked by SendMessage, which is waiting for thread 2CA8 (which is the main thead)".

Wait chain for worker thread is:

Blocked waiting for SendMessage owned by thread [ 2CA8 / 11432 ]
thread -> SendMessage -> thread: [ 2CA8 / 11432 ] is blocked -> thread: [ 27DC / 10204 ] is blocked

Which means: "worker thread (27DC) is blocked by SendMessage, which is waiting for main thread (2CA8), which is waiting for thread 27DC (which is the worker thread)".

So, you immediately get information about what both threads are waiting for! Any of two wait chains allows you to fully reconstruct the problem scenario.

Final notes

Threads Snapshot Tool requres debug information for your application to display human-readable call stacks. It supports all usual sources of debug information for EurekaLog 7: EurekaLog itself, map files, TD32/TDS, DLL export, JclDebug, madExcept, Microsoft formats. Therefore, if you compile your application with EurekaLog (any version) - then you do not need any additional actions. If not - then you need to supply debug information for your application. One simple way to do this - set "Map file" = "Detailed" option ("Linker" tab in project options inside IDE) and deploy .map files along with your .exe and .dll files.

Wait Chains are created only on Windows Vista and above. For Windows XP and below - only call stacks and CPU states are provided.

Threads Snapshot Tool is available in both x86-32 and x86-64 versions. You need to use application of corresponding bitness. Attempt to use 32-bit Threads Snapshot Tool to capture state of 64-bit application (and visa versa) will result in error message.

32-bit version of Threads Snapshot Tool is installed under C:\Program Files\EurekaLab\EurekaLog 7\Bin\ folder by default. 64-bit version is installed only on 64-bit editions of Windows. It's placed in C:\Program Files (x86)\EurekaLab\EurekaLog 7\Bin64\ by default.

Threads Snapshot Tool consists of ThreadsSnapshotGUI.exe and ThreadsSnapshotWorkerDLL.dll files. Those are the files which you can copy to another machine. ThreadsSnapshot.exe is a console equivalent of ThreadsSnapshotGUI.exe file. You can use it to capture state from the console by calling it with arguments (launch it without agruments to get help on using).

What does this address mean? (introducing Address Lookup Tool)

2012-03-19T16:38:00.001Z

Did you ever encounter cryptic error messages like "Access violation at address 00515974 in module 'Project1.exe'. Write of address 00000056"? Have you wondered what source code line is mentioned in this error message? Well, look no further...

Exception tracer solution?

Well, basically, if you use EurekaLog or any other exception tracer tool - you usually don't get just "Access Violation at address..." error message, but get a full bug report instead - including call stack and CPU state. However, if:

you're working with application without exception tracer
application handles the error by itself without passing it to exception tracer

then you will have a simple error message only, without any advanced/detailed information. If you can reproduce this case again - good. You can catch it under the debugger and analyze the situation. But if you for some reason is unable to do that, but urgently need to restore original source code point - you're stuck with numbers like "00515974".

Address Lookup Tool

EurekaLog 7 as well as standalone EurekaLog Tools Pack have a Address Lookup Tool, which can help you to analyze addresses in the application and provide more information than just RAW address. You can launch this tool via Start menu (Programs\EurekaLog 7\Tools\Address Lookup) or via Tools\EurekaLog\Address Lookup IDE command (for EurekaLog 7 only, not applicable to Tools Pack).

Note: there are 32-bit and 64-bit versions available. Use the tool of corresponding bitness. 32-bit Address Lookup will fail for 64-bit executable files and visa versa. 64-bit Address Lookup will only be installed on Win64 machine.

Address Lookup main window

As you can see - the interface is simple. First, you need to specify your application. It can be .exe file, DLL or BPL (package). Use the file which has raised the error in question. If it's not mentioned in error message - you can try to find it from address by using other tools (see more detailed explanation below). Of course, executable file must match the file which has raised the error. I.e. you can't use version 2 of your application to resolve addresses from version 1 - that will most probably don't work, since addresses may be changed.

Next, select source of debug information to lookup. Usually, you should leave this option in its default position - "Auto". But you may want to use specific source, if you want to.

Note: support for madExcept information is experimental. It may not always work properly.

So, basically, you'll need at least one source of debug information available. If you use EurekaLog or other exception tracer solution - you already have one. If not - then you should supply it manually. The easiest way to do this - enable Map file = Detailed option on Linker page from project options in IDE (however, you'll need to rebuild your application).

Last thing to do - is to enter address itself. There is a catch though. Usually you get the full absolute address like "00515974" in the example above. This address can not be resolved into source code line. What you need is to have an offset from the start of the exe/DLL/BPL file. If you have .exe file - this task is easy: just substract $400000 from your address value and you're done:

Looking for source line from address (click on image to enlarge)

$00515974 - $00400000 = $00115974 - and (as you can see) this is line number 128 in Unit99.pas, which belongs to TForm99.Button1Click method. Location found.

The Address Lookup Tool also supports include files:

Finding address inside .inc file (click to enlarge)

A quick explanation for the information description:

Provider - is the name of a EurekaLog debug information provider class, which was able to extract location information from your executable. If you leave "Auto" position in "Debug information source" option - then this can be any class among supported. If you select specific source - then this will be specific class only which corresponds to your choice.
Location - a short description of found source code location. The same information is provided below this line with more detailed view:
- Unit - name of the unit with the code. This is the name as it appears into "unit UnitName;" statement.
- Source - name of source file. Well, usually this is the same as unit name + .pas extension, but it is not necessary so. Think about .dpr file and .inc files. This is the file which you need to open, if you want to see source code location.
- Class - name of the class to which belongs method. If your address belongs to simple function - class name will be empty.
- Routine - name of the routine (procedure, function or method).
- Line - absolute line number inside "source" file.
The values which are not present in the below detalization are: offset ("(00114974)"), binary file name ("{Project102.exe}") and absolute address ("[0B735974]"). These values do not tell you any additional information and are provided only for sanity checks. Please note that .exe file will be loaded at the different base address than $00400000 (because $00400000 is already busy by AddressLookup.exe), so calculated absolute address (0B735974) will not match your data.
Routine offset (bytes) - the difference (offset) between current address and routine's start in bytes.
Routine offset (lines) - the difference (offset) between current line and first line in routine (in lines). This valus is useless if you use include files inside routine.
Line offset (bytes) - the offset of code from the start of the current line in bytes. This is applicable for complex expressions in single line.

The last 3 pieces of information can help you to track down location if source code was changed. For example, suppose that you have a error in version 1 of your application, and you search for source code location using .exe file from version 1; of course, you'll get location in source code for version 1. But what if you don't source code for version 1 of your application? Then you still can try to find it if source code wasn't changed much around your address. Just find routine by name in source code of version 2 and then shift down from the start of the routine to the specified amound of lines (or bytes) - and you'll get to the desired location.

Working with addresses and offsets

Each executable module (exe or DLL) can be loaded into memory on different addresses, which are called base addresses. As you can guess, the same code will have different address if it was loaded on different base addresses. Thus, absolute address of code location is usually useless - because base address of executable on analyzing stage may be different than it was when the error was raised.

For example, suppose there was an error in your DLL at address $6E775974 and DLL itself was loaded at address $6E660000 (which is base address for that DLL). Now, you want to find source code location for that DLL from address $6E775974. So, you load DLL and try to look for this address, but on your machine DLL is now loaded at address $642A0000 (not $6E660000 - as it was when error was raised). This means that address $6E775974 is useless - it will point to something else.

This is similar to the process address space isolation, when two processes can have different data on the same address. If you need to exchange data between processes you can't pass pointers (addresses) - you need to use relative offsets.

The similar approach is applied here too: find an offset of your code from the start of the module. In our imaginable example this is $6E775974 - $6E660000 = $00115974. And now - when you have offset - you can find exact the same location at any time. Just retrieve the base address of DLL (you can use Process Explorer tool to find it) and add our offset to it: $642A0000 + $00115974 = $643B5974 - and that is the final address of your code when you loaded DLL.

So that is how you work with addresses and offsets. To find source code location you need either:

relative offset (1 value)
absolute address and base address (2 values)

Just absolute address alone is useless.

However, there is a quick tip: exe files are loaded at the same address in 99.99% cases. So you can blindly assume the base address of $00400000 for .exe files.

Manual search

If you can't use Address Lookup Tool (for example: you miss debug information source), then you still can get useful information from code, but only if you have source code for exact the same version of your application. To do that:

load your application into IDE (restore it to previous version, if needed);
launch it via Run/Run (F9) command;
put your application on pause (Run/Pause);
use Search/Go to address command;
enter absolute address (don't forget to add '$' prefix) into dialog;
click OK;

Searching for address location manually (click to enlarge)

And you'll be moved to the source code location. If there is debug information available for that location - you will see source code in IDE code editor; otherwise you'll see a CPU debugger positioned to your location.

You can download freeware EurekaLog Tools Pack here.

What does this error code mean? (introducing Error Lookup Tool)

2012-03-19T13:46:00.001Z

Did you ever encounter cryptic error messages like "Error -1073741819" or "Run-time error 216"? Have you wondered what this could be and how to find more meaningfull description of the error? Well, look no further...

Error Lookup Tool

EurekaLog 7 as well as standalone EurekaLog Tools Pack have a Error Lookup Tool, which can help you to analyze different error codes and provide more information than contains in the code itself. You can launch this tool via Start menu (Programs\EurekaLog 7\Tools\Error Lookup) or via Tools\EurekaLog\Error Lookup IDE command (for EurekaLog 7 only, not applicable to Tools Pack).

Error Lookup main window

As you can see - the interface is very simple. Just enter your mystic error code value into edit (don't forget to add '$' prefix for hex-codes) and hit "Lookup..." button:

Error Lookup resolving -1073741819 error code (click on image to enlarge)

Error Lookup resolving 216 error code (click to enlarge)

So now you know that -1073741819 - is actually a system error code for access violation exception, and 216 is the same access violation, but now expressed via Delphi RTL run-time codes.

Error Lookup supports many sources of error codes, for example:

Illustrating more error sources (click to enlarge)

As you can see for some error codes there are multiply possible descriptions available. The correct interpretation depens on the code which generated that error code. Error Lookup can not know that, so it just displays all possible cases - and you'll need to pick the one matching the context of the error.

The opposite case can also be true. You can get an error code, which is unrecognized by Error Lookup Tool. This can either be a custom error code or new standard error code. For example:

Custom error code of HRESULT kind is displayed in Error Lookup

The correct interpretation of custom error codes depends on the component which returns that code. The same code means different things for different components. When you encounter such code you should refer/search documentation for the called code. Custom codes are applicable for Win32, NTSTATUS and HRESULT values. RTL, I/O, WinInet and other kinds of codes do not support custom codes extensions.

Another case of unknown error code:

Unknown error code do not display any additional information

If there is a new error code which is not known to Error Lookup Tool - it will show you no information at all. In this case we ask you to send us a notification about error code which is not covered by us, so we can improve our tools.

You can also try to force-display all interpretation of any error code by marking "Show all" option before clicking on "Lookup..." button:

Showing all possible interpretations of error code

This will display all interpretations of entered error code to you - regardless of if it's meaningfull at all.

You can download freeware EurekaLog Tools Pack here.

EurekaLog RC 3 (public Win64 test release)

2012-03-13T19:51:00.001Z

We've just released EurekaLog 7 RC 3.

We are pleased to announce the availability of the new EurekaLog 7.0.0.194 version (RC 3). The main addition for RC 3 is first public access to Win64 support in EurekaLog. You can download and install EurekaLog 7 RC 3 to try EurekaLog for Win64. For more information about installing a beta - see here. For installing both EurekaLog 6 and EurekaLog 7 on the same machine - see here.

Note: this is a beta version. Each application compiled with beta will expired after 30 days. Do not use it for production.

EurekaLog for Win64 quick walktrough

Create new project and right-click on "Platforms" node in project manager:

Platforms context menu

Click on "Add platform..." command and you'll see adding platform dialog:

Adding new platform to project

Select "64-bit Windows", check "Make the selected platform active" option and click OK.

Project configured for Win64

Now to go Project/EurekaLog project options:

EurekaLog project options under Project menu (image is clickable)

Check "Activate EurekaLog" option and select type of your application (usually - VCL Forms):

Enabling EurekaLog for the project (click to enlarge/zoom in)

Now, build your project. If you have enabled visual feedback - you should see EurekaLog-processing stage at the end of the build process:

EurekaLog processes your project

Run your project and try to do something bad. In my example I'll just do:

raise Exception.Create('Error Message');

in the OnClick event of the button.

So, when I click on the button - I get usual EurekaLog error dialog:

Win64 MS Classic Error Dialog

You can click on "click here" link to see detailed report:

64-bit call stack (click to zoom in/enlarge)

64-bit CPU view (click to enlarge/zoom in)

So, that's it!

Win64 support status

Basically, EurekaLog for Win64 is very similar to EurekaLog for Win32 with few exceptions:

Not all options are applicable for Win64 (for example, stack tracing method option is ignored).
Win64 disassembler is not ready.
EurekaLog for Win32 and EurekaLog for Win64 may behave slightly different - due to difference in exception processing on Win32/Win64.

It's also highly recommended to use the latest Delphi XE2 update pack for EurekaLog 7 RC 3.

EurekaLog Tools Pack

EurekaLog Tools Pack was also updated with Win64 counter-equivalents of some tools which depends on bitness (module analyzer, threads snapshot, etc) - while most tools still has only 32-bit version. 64-bit EurekaLog Tools are also included into EurekaLog 7 RC 3.

Please note that 64-bit executables will only be installed on 64-bit system. On the other hand, Win64 support in EurekaLog can be installed on any system - 32-bit and 64-bit.

Conclusion

We're waiting on your feedback. Tell us what you think!

EurekaLog and FireMonkey / Win64 / OSX

2012-01-25T00:26:00.001Z

Clarification about EurekaLog and latest Delphi versions...

Delphi XE2 has introduced a new visual framework - FireMonkey, as well as support for more platforms, including Win64 and OSX.

There are constant income of questions about EurekaLog's future and possible support for these new technlogies. So I've decided to publish general explanation for everyone.

EurekaLog and FireMonkey

The answer is simple - EurekaLog will... never support FireMonkey, because there is nothing to support! FireMonkey is visual framework. But EurekaLog just don't use any visual frameworks - it don't use VCL and it certainly don't need to use FireMonkey. EurekaLog uses API for all visual UI.

That's why you already can use EurekaLog with both VCL and FireMonkey applications.

Of course, if you want to create your own custom error dialogs for EurekaLog or something like that - fine, use API, VCL, FireMonkey or whatever technology else. There is no limitations. However, EurekaLog is not related to VCL or FireMonkey. It's just exception tracer.

EurekaLog and Win64

Sorry, but EurekaLog 6 will never support Win64. It was written without considering other platforms in mind.

However, there are good news: EurekaLog 7 will support Win64. Right now we're working on it, even though it dind't appear in any public releases. Support for Win64 is our current top priority!

It's expected that Win64 support will be released (as beta) after we ship stable release of EurekaLog 7. You can help this to happen sooner by testing EurekaLog.

Preleminary estimate is that Win64 support will be basic for first few releases and then we'll add advanced features (like disassembler view).

Another good news is that we want to make migration process as painless as possible. Ideally you should be able to install EurekaLog 7, open your old project, recompile it - and it should work.

EurekaLog and OSX

The case with OSX support is very similar to Win64. EurekaLog 6 is out of question, but support for OSX is planned for EurekaLog 7. This is our second top-priority.

We want to finish Win64 first and then start going for OSX. Currently, there was no work done for OSX, but all current work is done with future OSX target in mind.

EurekaLog and... Lazarus/MSVS C++

Yes, there is slight possibility for appearing EurekaLog for other IDEs like Lazarus or Microsoft Visual Studio (C++). However, currently this is considered as low-priority in far-far away future.

Note: this information is highly preliminary and is suject to change. Also, there is no estimates available.

EurekaLog 7 RC 2

2012-01-25T00:07:00.001Z

We've just released EurekaLog 7 RC 2.

We fixed all found issues (thanks to all beta-testers!) and make some improvements and additions. A new beta is available for download from our web-site.

EurekaLog 7 Tools Pack was also updated to the latest version (one additional tool was added).

We recommend to manually uninstall any previous beta or previews of EurekaLog 7 or EurekaLog Tools Pack before installing EurekaLog 7 RC 2 or updated EurekaLog Tools Pack. After installation - you can re-activate EurekaLog 6 or EurekaLog 7 via Start/Programs/EurekaLog 7/Manage menu item.

EurekaLog 6.1.04 is out

2012-01-19T17:35:00.001Z

We are pleased to announce the availability of the new 6.1.04 version.

EurekaLog is a tool for Delphi and C++Builder that gives your application the power to catch every exception and memory leak, generating a detailed log of the call stack (with unit, procedure and line number), optionally sending you a copy of each log entry via email or the Web via the most used Web Bug-Tracking tools (as Mantis, BugZilla and FogBugz).

EurekaLog represents the most advanced exception and memory leaks logger technology available for Delphi/C++Builder developers.

To learn more about EurekaLog, please visit our website at:
http://www.eurekalog.com

To download a demo, please visit:
http://www.eurekalog.com/downloads.php

If you would like to purchase a new license, please visit:
http://www.eurekalog.com/buy.php

Upgrade policy:

If you would like to purchase an upgrade, this is our upgrade discount policy:

1)...upgrade from any 4.x or 5.x to any 6.x - 50% discount;
2)...upgrade from a 6.x license type to another (ex: from a "Single Professional" to a "Corporate Professional") - the price difference.

NOTE: the upgrade from an old 6.x version to the last released version is full free.

Buy your upgrade or download your full version here:
http://www.eurekalog.com/login.php

Changes:

Added: support for MainSource in dproj files
Fixed: compatibility issues with RAD Studio XE2 Update 3
Fixed: unexpected dialog behaviour in unattended enviroments
Fixed: some complex C++ Builder's idents were not recornized by Viewer

EurekaLog Users Survey

2011-12-19T20:42:00.000Z

We would like to invite you to participate into EurekaLog Customers Survey.

This short survey will take only about 10 minutes of your valuable time and will greatly help us to define EurekaLog's future. This survey is for all (actual and potential) EurekaLog users. Don't feel shy to share your opinion!

Take a survey.

Announcing public EurekaLog 7 beta

2011-12-19T20:35:00.000Z

Get a free EurekaLog 7 license! ;)

We are pleased to announce the availability of the first EurekaLog 7 public beta.

You can download it from "Delphi downloads" on our web-site.

We also would like to invite you to participate into EurekaLog User Survey, which you can take here.

How to try EurekaLog 7

EurekaLog 7 can be installed with EurekaLog 6. You can switch between EurekaLog's versions by using new "Manage" menu item in Start Menu/Programs.

You can find more information here.

Differences with EurekaLog 6

EurekaLog 7 is similar to EurekaLog 6. The only important difference for quick start - is that you need to select your application type after activating EurekaLog in your projects.

Please, look though Quick Start Tutorials guide to quickly learn about changes.

EurekaLog 7 and exising EurekaLog 6-projects

EurekaLog 7 changes some behaviour and introduces new features. In order to keep backward compatibility we decided to add so-called "EurekaLog 6 Compatibility Mode". You can access it in EurekaLog project options on "General" tab.

The idea is when you open old project - it will be opened (imported) in "EurekaLog 6 Compatibility Mode". When you create new project - it will be created with "EurekaLog 6 Compatibility Mode" turned off. When "EurekaLog 6 Compatibility Mode" is turned on - the EurekaLog behaviour will be the same as EurekaLog 6, so you would not need to change anything in your old existing projects!

Where to get information

EurekaLog 7 has much more detailed help than EurekaLog 6. However, we're still writing it. Help can be accessed localy (via Start Menu or via F1 key in EurekaLog's windows) or Online.

Code reference is under construction, but you can find header files in \Source sub-folder of your EurekaLog 7 installation (it's C:\Program Files (x86)\EurekaLab\EurekaLog 7\Source\ by default). Full source code will be available only for Enterprise edition of EurekaLog, but any edition has partial source code available!

We also plan to start publising blog articles about new and extended EurekaLog 7 features and typical use cases. Stay tuned for more info!

You can find list of changes in EurekaLog 7 here (list is not complete yet!).

Participate in beta - win EurekaLog license!

We would like to reward you for participating in EurekaLog 7 beta testing. Most active beta testers will receive a free EurekaLog 7 Enterprise license! (and possibly other editions - as consolation prize) This applies to both existing customers and even non-EurekaLog users.

What you need to do? Just install EurekaLog 7 and try to use it. Report any bug you found. Or just share opinion and feedback. Or make a code contribution. The more information you submit - the higher are chances to win licenses.

How to submit an information? You can either use build-in feedback sending methods or use tech-support form. Be sure to use single e-mail address for all of your submissions. If you win - your EurekaLog license will be registered for that e-mail address. Please, don't use forums for that. Don't send your messages directly to the staff. Basically, if you got auto-responce message from our support system for your message - then it is registered for this offer.

Winners will be announced after releasing EurekaLog 7 stable. Amount of winners is not fixed and will be decided based on actual number of active beta-testers. The winners will be determinated by most useful contribution into EurekaLog's developing. The exact measure of usefulness is left for us to decide. One good detailed report can be more useful than few short reports (however, this doesn't mean that you need to write multi-page-so-long-to-read-boring-reports). We retain rights to change conditions of this offer.

If you will not win the license - then you still will be able to upgrade your existing EurekaLog license with 50% discount (or even for free, if you have buyed the license in the last 90 days before the release date of EurekaLog 7 stable).

Note: EurekaLog 7 release date is not set. EurekaLog 7 will be released, when it'll be ready. You can help us to release it faster by testing it.

Note: the only EurekaLog 7 edition available now is Trial (which you can download from our web-site). Other editions (like Professional and Enterprise) will only be accessible after releasing EurekaLog 7 stable.

Warning: this is first beta of EurekaLog. We strongly do not recomend to use it for real-life production projects.

Using EurekaLog 6 and EurekaLog 7 on the same machine and/or in the same IDE version

2011-12-19T20:00:00.002Z

If you want to try EurekaLog 7, but can't abandon EurekaLog 6 yet (because you use it in your projects) - you can install them both.

Summary

To put this short:

Install EurekaLog 6
Install EurekaLog 7
Use "Manage" Start menu item to switch between EurekaLog 6 and 7

Warning: only EurekaLog 6.1.x is supported for parallel installation with EurekaLog 7. Old EurekaLog 6.0.x will not work.

Detailed explanation

If you want to install two versions of EurekaLog on the same machine, you need to install them one after one, starting with older version. So, first you need to install EurekaLog 6 (well, right now you probably already have it installed).

In the next step you should install EurekaLog 7. Since you have EurekaLog 6 installed, you will get such warning:

EurekaLog 7 has detected installed EurekaLog 6

You should answer "No", if you want to have both EurekaLog's versions installed (i.e. EurekaLog 6 and 7). If you answer "Yes", then EurekaLog 6 will be removed and EurekaLog 7 will be installed. This message will not appear, if you don't have EurekaLog 6 installed.

The rest of the installation process is similar to EurekaLog 6 with few exceptions:

EurekaLog 6 will refuse to install, if there is already another version of EurekaLog installed. That's why you must install EurekaLog 6 first.
EurekaLog 7 will install itself into another folder. By default, the installation path is:
- EurekaLog 6: C:\Program Files (x86)\EurekaLab\EurekaLog 6\
- EurekaLog 7: C:\Program Files (x86)\EurekaLab\EurekaLog 7\
("(x86)" is removed for 32-bit Windows).
EurekaLog 7 doesn't support Delphi 3 for technical reasons.
EurekaLog 6 installer comes with English language only. EurekaLog 7 installer has different languages available.
EurekaLog 7 installer offers more install options. Notable, EurekaLog 7 includes more tools as well as partial source code (for non-source code editions).
EurekaLog 7 offers a separate "Register EurekaLog into IDE" option.

Basically, you should see a set of checkboxes for each of your installed IDE (like "Delphi 7" or "C++ Builder XE2"):

IDE selection in EurekaLog 6 installer

Each of such checkboxes installs support files for selected IDE. If you want to use EurekaLog for some IDE - mark the checkbox, otherwise - clear it.

Additionally, EurekaLog 7 has "Register EurekaLog into IDE" option:

IDE selection in EurekaLog 7 installer

If you set this checkbox - EurekaLog 7 will be registered into IDE (much like as EurekaLog 6 installer does). If you clear this checkbox - EurekaLog's files will be installed (copied), but EurekaLog will not be registered in IDE.

If you install EurekaLog 7 when EurekaLog 6 is already installed, this option will not be visible (and it will be considered cleared). That's why you can safely install EurekaLog 7 - your EurekaLog 6 installation will not be affected.

You can find both EurekaLog versions in Start Menu after installation:

EurekaLog 6 and 7 in Start Menu/Programs

As we've already mentioned - EurekaLog 6 is active and EurekaLog 7 was installed without registration.

Now, you can alter EurekaLog configuration in your own desired way.

There may be different patterns of using EurekaLog. For example, if you have Delphi 7 and Delphi XE2 installed - you may want to use EurekaLog 6 for Delphi 7 and EurekaLog 7 for Delphi XE2. Other pattern: you have only one IDE installed (like Delphi XE2) and you want to use EurekaLog 6 for some of your projects and EurekaLog 7 for other projects. First usage case can be archieved by single configuration. However, since you can't have the two EurekaLog versions in one IDE at the same time (unit's files will conflict) - you can't archive case two with single one-time configuration. You need to manually switch between EurekaLog versions. You need to do this each time, when you want to use different EurekaLog version.

Anyways, both cases are archived though EurekaLog manager. You can launch it from Start Menu/Programs/EurekaLog 7.0.0 Trial/Manage menu item:

EurekaLog IDE configuration manager
(indicating that EurekaLog 6 is active for Delphi 7)

Top panel contains information about installed EurekaLog's versions. You should see information for EurekaLog 6 and 7 here. Bottom part contains list of installed IDEs. The list is collapsed by default. Click on any IDE to expand it and you'll see manage options (on the screenshot above: the "Delphi 7" IDE is expanded).

Each IDE has few information labels, indicated current state of EurekaLog, and options to remove EurekaLog ("No EurekaLog" button) or install a specific EurekaLog version ("EurekaLog 6 with IDE expert" or "EurekaLog 7 with IDE expert" buttons). There are also additional options to install EurekaLog in search path without installing IDE expert. These options aren't used in most cases, they are only used in custom cases.

In order to function, EurekaLog has to specify search path for it's units ("Paths" item). If you want to compile your projects from IDE, EurekaLog also needs to install EurekaLog compiler ("Compiler" item) and IDE expert ("Expert" item). Those items are installed into \Bin folder of your IDE.

Just expand your IDE group and click on the desired option - and you'll switch to that configuration.

Note: be sure to restart your IDE for changes to take effect.

Detecting installed EurekaLog version

When you have two EurekaLog's versions installed - it's very important not to mix the settings and files, otherwise results will be catastrophical. If you install and use EurekaLog IDE expert - then you can quickly determinate currently active EurekaLog version by looking into IDE menu:

EurekaLog 6 adds menu item into root IDE's menu

EurekaLog 7 adds menu item under "Tools" menu

Alternatively, you can use different "About" windows to get exact EurekaLog version:

EurekaLog in Help/About menu

EurekaLog in "About EurekaLog" menu

If you don't have IDE expert installed - then the best way is to use "Manage" item in Start Menu/Programs. It will show you the current status of EurekaLog.

Alternatively, you can check IDE search path settings. Go to Tools/Options/Library and check "Library path". It should point to EurekaLog 6 path (if EurekaLog 6 is active), EurekaLog 7 path (if EurekaLog 7 is active) or doesn't contain any EurekaLog-related path at all (if EurekaLog is not registered in IDE at all). Default EurekaLog path is:

EurekaLog 6: C:\Program Files (x86)\EurekaLab\EurekaLog 6\
EurekaLog 7: C:\Program Files (x86)\EurekaLab\EurekaLog 7\

Manage EurekaLog configuration for non-administrator user accounts

"Manage" Start Menu item also allows you to manage EurekaLog configuration for different operating system user accounts. By default, EurekaLog installs and registers itself (if enabled during installation) only for single user account: the one, which launched installation. This must be an administrator account (in order to have permission access rights to write into Program Files folder). But you're not limited to using EurekaLog under administrator account only.

To use EurekaLog under any other account:

Log in as administrator.
Run "Manage" menu item from the Start Menu.
Make sure that EurekaLog is installed into your IDE. If not - click on "EurekaLog 7 with IDE expert".
Log off from administrator account.
Log in to your target user account. It may be limited account.
Run "Manage" menu item from the Start Menu.
Click on "EurekaLog 7 with IDE expert".
Run your IDE.

Actions 1-4 are optional. They are only needed, if you didn't register EurekaLog 7 into IDE during installation process. Here steps 1-4 are done to copy EurekaLog files into Program Files folder (only administrator can do this) and steps 5-8 are done to register EurekaLog under separate user account.

Uninstalling

You can uninstall EurekaLog via Start Menu or "Uninstall a program" Control Panel applet. If you have two versions of EurekaLog installed and you want to remove one of them - you'll need to reinstall another version.

If for some reason you want to installe old EurekaLog version (which is not available on the web-site anymore) - please, contact our support on that matter (use "Techincal support for Delphi/C++ Builder" category).

Troubleshooting

If you encounter any issues with installation/uninstallation, please refer to our troubleshooting guides:

If you're unable to resolve your problem with these detailed instructions - please, contact our support (use "Techincal support for Delphi/C++ Builder" category).

EurekaLog 7 Tools Pack

EurekaLog 7 Tools Pack can be installed to a machine with EurekaLog 6 without any additional issues, because it doesn't requires registration. It also can be uninstalled without interferring with EurekaLog 6.

Note: you can't install both EurekaLog 7 and EurekaLog 7 Tools Pack on the same machine, because EurekaLog 7 already contains EurekaLog 7 Tools Pack. Just install EurekaLog 7 only, no need to install EurekaLog 7 Tools Pack.

EurekaLog and XE2 update 3

2011-12-19T17:57:00.001Z

Update 3 for RAD Studio XE2 (Delphi and C++ Builder) has broken working of EurekaLog.

The new EurekaLog 6.1.04 RC 2 should be able to address this issue. You may download it from our web-site: area for the registered customers.

Note: EurekaLog 6.1.04 RC 2 is not compatible with Delphi XE2/C++ Builder XE2 update 1 or update 2. You must have update 3 installed. If you don't have update 3 installed - you'll have a warning during installation.

EurekaLog 6.1.03 is out - now with XE2 and FireMonkey support

2011-09-28T19:33:00.000+01:00

We are pleased to announce the availability of the new 6.1.03 version.

EurekaLog is a tool for Delphi and C++Builder that gives your application the power to catch every exception and memory leak, generating a detailed log of the call stack (with unit, procedure and line number), optionally sending you a copy of each log entry via email or the Web via the most used Web Bug-Tracking tools (as Mantis, BugZilla and FogBugz).

Added: support for RAD Studio XE2 (Win32 platform only)
Added: support for FireMonkey framework (Win32 platform only)
Improved: new leaks for "Hide Borland leaks" option
Fixed: Escape key could cancel process termination
Fixed: variables didn't worked for exception filters
Fixed: rare application hang up at shutdown
Fixed: encoding issues for run-time option switching

EurekaLog 6.5.12 for .NET is out

2011-09-15T11:04:00.000+01:00

We are pleased to announce the availability of the new 6.5.12 version.

EurekaLog is a tool for Delphi and C++Builder that gives your application the power to catch every exception and memory leak, generating a detailed log of the call stack (with unit, procedure and line number), optionally sending you a copy of each log entry via email or the Web via the most used Web Bug-Tracking tools (as Mantis, BugZilla and FogBugz).

Improved: uninstall procedure to clean up any leftover files that are no longer needed
Improved: e-mail validation
Improved: Mantis compatibility
Improved: thread safety
Fixed: bug that in some cases causes Visual Studio to hang on load
Fixed: bug that mistakenly reported Windows 7 as Windows Vista
Fixed: problem with detail dialog, not showing threaded and inner exceptions' details in all cases
Fixed: problem causing VS to open a lot of files unnecessarily while building the solution

EurekaLog 6.1.02 is out

2011-07-22T18:19:00.000+01:00

We are pleased to announce the availability of the new 6.1.02 version.

EurekaLog is a tool for Delphi and C++Builder that gives your application the power to catch every exception and memory leak, generating a detailed log of the call stack (with unit, procedure and line number), optionally sending you a copy of each log entry via email or the Web via the most used Web Bug-Tracking tools (as Mantis, BugZilla and FogBugz).

Added: BugZilla 4.x support
Added: (Mantis) category is now optional
Fixed: address space leak due to multi-thread concurensy
Fixed: Sometimes EurekaLog unable to get thread's call stack
Fixed: EurekaLog fails to show information about last routine in unit
Fixed: memory leaks detection has trimmed call stack (some specific Delphi versions)
Fixed: Viewer configuration issue on limited user
Fixed: emake sometime doesn't work
Fixed: crash, if module info wasn't available
Fixed: wrong detection of handled exceptions for certain environments
Fixed: problems with thread locking for specific components
Fixed: error in fallback code in DllFullPath
Fixed: Installer creates unnecessary files
Fixed: nag screen is not DPI awared
Fixed: wrong handling of WM_QUIT - dialog enters endless cycle
Fixed: .eof file can be not recognized as valid .eof file on some rare conditions
Fixed: changed date-time is not updated when overwriting .eof file
Fixed: increased stability for getting memory dumps

Introducing EurekaLog 6.1

2010-12-30T17:07:00.003Z

We made a present for our customers at New Year 2010-2011 by releasing EurekaLog 6.1. So... what's new?

Q: What is new in EurekaLog 6.1?
A: EurekaLog 6.1 is exactly the same as EurekaLog 6.0 (6.0.25 is the latest available version), except one "little" detail: we've added support for new compiler from EurekaLog 7.0 (which is in development right now).

Q: Why is it 6.1? Why not 6.0.26?
A: We do only bug fixes in 6.x branch, so we get 6.0.10, 6.0.20, 6.0.25, etc. Each new version (like 6.0.xx) contains bug fixes, but no new features. Since we've added a major new feature - we decided to increase a version number, so new version became 6.1 - indicating addition of a major new feature.

Q: Is it safe to upgrade? Am I affected?
A: Don't worry, 6.1.xx behaves as 6.0.xx by default. You can safely upgrade and no new changes will affect you (except the usual bug fixes). New compiler have no effect, until explicty enabled.

Q: What's so great about new compiler?
A: In recent 3 months there were many reports about problems with EurekaLog and debug information (i.e. mapping between code and human-readable names of units, methods, etc). After investigation we've discovered that there appears linker bugs (usually on large projects), which prevents EurekaLog from proper functioning. Also, there are some internal limitations of EurekaLog for large projects¹.

What does it mean? It means that for some large projects EurekaLog can't show you a proper call stack because of wrong debug information, which is usually caused by linker's bugs.

Well, I'm a bit surprised that no one is ever reported such issues before (at least not while I was working for EurekaLab s.a.s.). So these last months were really a hit on this.

OK, so the new compiler from EurekaLog 7 have a new architecture and a new code (it's re-written almost on 80%) and it is able to solve these problems (along with help of new format of debug information).

Another advantages are:

Increased speed. New compiler should be (theoretically) faster than old, since we've reduced amount of work with strings.
Reduced memory consumption. New compiler should be (theoretically) more conservative about memory usage, since we're working on since mapped file, instead of parsing it into separate strings.
Increased stability. Any issues with compiler itself does not affect IDE, since compiler is now run into separate process.
Increased detalization. New compiler is able to store routines without line numbers. Old compiler can't do this. Usually this doesn't matter anyway, since we always recommend to enable "Use Debug DCUs" option.

Items 2 and 3 allows you to use EurekaLog for very large projects, which wasn't possible before (because of shared virtual address space with IDE and, thus, hitting a memory limits).

Word "theoretically" means that we think so, but we don't have any exact estimates yet. So, if you're owning a large project (say, 10+ MB .exe file or 15+ MB .map file) - please, send us (even very approximate) details about how fast (or slow!) compilation is with old and new compiler.

Q: Why add compiler from EurekaLog 7 to EurekaLog 6? Why not just fix issues in EurekaLog 6? Or just wait for EurekaLog 7 release?
A: Because it's impossible to fix these issues with old logic of compiler (there are much technical details here, I won't going to discuss them). To solve these issues in EurekaLog 6, one needs to almost re-write the compiler - pretty much the same job as was already completed in EurekaLog 7 (compiler upgrade in EurekaLog 7 wasn't because of these issues - we just improved code. Solving these issues was a side-effect).

So why just not port code from EurekaLog 7 to EurekaLog 6? Well, we tried this, but it's not easy too, since there are a lot of changes (also remember that we dropped support for Delphi 3, so we used new features a lot).

Why not just leave this as is, claiming that customers should wait for EurekaLog 7? I'm not saying that EurekaLog 7 have a long road before release, but I'm saying that we care about our customers, so we've decided to release intermediate EurekaLog 6.1, which solves some major problems of our customers.

Thus, the only solution left is to include new compiler as an option to EurekaLog 6.

Q: When I should use new compiler?
A: If you have small/medium project and never suffer from weird call stacks - then you can just ignore it. Don't use new compiler (which is default, BTW). Otherwise - try it. New compiler may help you to solve your issues.

Q: How to use it?
A: Very simple. Just go to Project / EurekaLog options, select Build options tab and enable Use EurekaLog 7 compiler checkbox:

Q: What about command line compilation and build automation?
A: That is possible too. You can find new compiler in your EurekaLog folder under sub-folder which corresponds to your IDE. For example:

C:\Program Files\EurekaLab\EurekaLog 6\Delphi15\el7c.exe

The same .exe file can be used for both Delphi and C++ Builder - just be sure that you pick a correct version (in case if you have several different versions of IDE installed).

You can use this file exactly as the old ecc32.exe/emake.exe. Except there are few additional options:

--el_output=el6 - instructs compiler to produce debug information in EurekaLog 6 format. Mandatory. You must add this switch.
--el_mode=delphi or --el_mode=builder - switches between Delphi and C++ Builder project. Mandatory. You must add this switch.
--el_path=folderpath - folderpath is full absolute path to EurekaLog folder. This is optional switch, but we strongly recommend to specify it. For example:
```
"--el_path=C:\Program Files\EurekaLab\EurekaLog 6"
```
--el_gui_error - show errors in message boxes instead of writing them to the console. Optional. Usually you want to add this, if you run compiler from IDE or other GUI tool. You probably don't want to specify it, if you run compiler from console or automated build script.
--el_target=filename - specifies output file name. Optional. By default EurekaLog tries to get this from project's options (which is not always possible). It's not needed for --el_alter_exe switch, since output file name can be specified inside --el_alter_exe switch itself.

Q: I don't want to use experimental compiler from a EurekaLog 7, which doesn't even have a stable release!
A: First I want to highlight that EurekaLog's code remain the same. New compiler doesn't affect the code. It's still old good and tested 6.0.25. New compiler affects only additional debug information. So, if you have problems with debug information, new compiler may solve it. I.e. it's not worse than it's already is. Surely, you may not use new compiler, if you don't need to.

Yes, new compiler comes from unstable version of EurekaLog 7. However, we've tested a new compiler quite strict. Sure we may miss something, but it's quite stable now.

Of course, you shouldn't just enable this option and throw your application to your customers. You need to do a few simple tests:

Try just to compile your application with new compiler - will it be successfull? New compiler contains additional asserts, which trigger on unsupported cases.
Take a known issue with bug report and check this issue against new compiler - will a new bug report be better than old?
You can place a button, which raises exception somewhere deep inside your code. Will a bug report be detailed enough?

If these tests will pass - then you can safely use new compiler. If not - please, contact us, so we can improve our product!

Thank you for all your valuable feedback and I wish you a happy New Year!

Remarks:
¹ - for example, EurekaLog uses signed word (2 bytes) as offset between code from neighbour lines. Usually this is not a problem, since machine code goes in the same order as source (indeed, many people even rely on this undocumented behavior), so you rarely see values larger than 128 - not even saying about 32768!

However, we've found that in some (rare) cases code may go in really wild order. So if you wave a unit larger than 32768 lines AND such thing happens (I don't know if this is another linker bug or planned behavior) - EurekaLog won't be able to handle this. New debug format of EurekaLog 7 uses up to 4 bytes for offsets, so these cases are covered. However, old debug format is still limited to 2 bytes offsets.

There may be 1 or 2 other but similar issues - I don't really remember them now.

ITDevCon sessions from EurekaLab s.a.s.

2010-12-01T21:55:00.000Z

There was a ITDevCon conference recently - 17-19th November 2010. I (Alex) took two sessions on this event: "Advanced features of Delphi's debugger" and "How to debug application's crash".

I had prepared two presentations for these sessions in OpenOffice Impress and some accompanied text in OpenOffice Writer for this event. And I thought that you may be interested in those materials, so I decided to publish them.

You can download both sessions in one archive here.

EurekaLog 6.0.25 is out

2010-11-22T14:02:00.017Z

We are pleased to announce the availability of the new 6.0.25 version.

EurekaLog is a tool for Delphi and C++Builder that gives your application the power to catch every exception and memory leak, generating a detailed log of the call stack (with unit, procedure and line number), optionally sending you a copy of each log entry via email or the Web via the most used Web Bug-Tracking tools (as Mantis, BugZilla and FogBugz).

Added: support for Delphi XE and C++Builder XE;
Added: SafeMM memory manager support;
Fixed: a little Call-stack tracing improvment;
Fixed: EurekaLog catched exceptions also when disabled;
Fixed: installer's bug that didn't activate EurekaLog by default if this option was enabled;
Fixed: wrong decode of some assembles views in the EurekaLog Viewer window;
Changed: EurekaLog IDE integration is now set to OFF by default (to activate it use the "EurekaLog/IDE Integration" menu);

EurekaLog 6.0.24 is out

2010-07-27T13:43:00.007+01:00

We are pleased to announce the availability of the new 6.0.24 version.

EurekaLog is a tool for Delphi and C++Builder that gives your application the power to catch every exception and memory leak, generating a detailed log of the call stack (with unit, procedure and line number), optionally sending you a copy of each log entry via email or the Web via the most used Web Bug-Tracking tools (as Mantis, BugZilla and FogBugz).

Added: ModuleInfoByAddr and ModuleInfoByHandle routines to improve EurekaLog usage with packages;
Fixed: unable to save empty multi-line properties;
Fixed: Mantis 1.2.1 login issue;
Fixed: EurekaLog wasn't working with IntraWeb and C++ Builder 2010;
Fixed: rare AV in BDS 2010;
Fixed: changes date-time format while EurekaLog application running could break it;
Fixed: splash screen is not large DPI aware;
Fixed: fixed issue with missing wsock32.dll wrapper;
Fixed: wrong encoding for saving some files in BDS 2009 and above;
Fixed: Mantis encoding issues;

SDN sessions from EurekaLab s.a.s.

2010-06-08T17:06:00.004+01:00

There was a SDN event recently - 18th May 2010. Fabio Dell'Aria took two sessions on this event: "How to find the exception source line of a just delivered software" and "How to prevent source code bugs and speed-up the testing phase".

I had prepared two presentations for these sessions in OpenOffice Impress and some accompanied text in OpenOffice Writer for this event. And I thought that you may be interested in those materials, so I decided to publish them.

You can download both sessions in one archive here.

How to debug application's hang?

2010-06-08T16:25:00.057+01:00

Today’s article will cover different approaches to debugging of application’s hangs.

It contains six parts:

Preparations – common actions for preparing for all debugging cases.
Delphi – debugging hang with Delphi.
EurekaLog – debugging hang with EurekaLog.
Process Explorer – debugging hang with Process Explorer.
Threads Snapshot – debugging hang with Threads Snapshot.
A practical example – example with artificial hang.

Preparations

Well, before you start actual debugging – you need to do some tasks first. First thing is: you need to add debug information to your project and make a build (not just compile). Different tools requires/supports different formats of debugging info, so just enable everything – so you don’t need to think about it :) This step is required, if you want to deal with human-readable sources and not some raw machine code and offsets.

Okay, so you need to turn on these options on “Linking” page (Project/Options/Linking): "MAP file" - Detailed, "Debug Information" (it’s called "Include TD32 debug info" in old Delphi versions) - True, "Include remote debug symbols" - True:

Apart from debug info, you should also enable "Stack Frames" and "Use Debug DCUs" options on "Compiling" page:

Please, allow me not describe again, what these options are doing.

Don’t forget to make Build after changes. Obviously, if you have several projects (DLL, BPL, etc) – then you need to do the same thing for each project. Also note, if you’re building with run-time packages, you should disable them for debugging session, if it’s possible. Packages increase complexity of debugging significantly.

Delphi

You could think: "but I can’t reproduce this hang under debugger!" or "I do not have Delphi installed on that machine!". Wait a moment, don’t go to the next article’s item.

First: you don’t need to run your application under Delphi. You can run your program as usual (without any debugger attached), work as much, as you need (few days?) – and wait, until it hang. Now you can attach Delphi’s debugger to hanged process and start debugging. Second: if you don’t have Delphi installed – you can use remote debugger. I won’t go into details here, as we have a lot things to do today, but it’s not hard task – refer to Delphi’s help.

So, you’ve run your application, it worked for some time and hanged. Now open Delphi, open your project and do Run/Attach to process command:

Delphi’s debugger will connect to selected process and will set it on pause – by hitting breakpoint in special debugger’s service thread:

You can ignore that debugger’s thread – just open Threads window and select any of your threads for debugging. You can view call stacks, walk call stack, switch threads, analyze variables, do stepping and running, etc, etc – in other words: you may debug your application as usual. If there is debug information available – you can close CPU window and debug by using source code.

Note, that you should use Windows Vista or above for debugging, if this is possible – that’s because these OS support new feature called “Wait Chain Traversal”. Delphi’s debugger is capable of using it, starting with Delphi 2009. So, if you’re using Delphi 2009 or above and Windows Vista or above – then you can see wait chain status in Threads window:

EurekaLog

EurekaLog have anti-freeze feature. Well, I’ve already covered it in much details a long time ago in a separate article. For this reason, I’ll not discuss it in details here, as there is a lot of other material to discuss.

Short summary: this is a very convenient feature, if you can use it in your application. You can’t use it, if your main thread is not pumping messages (like console or service applications). The result is usual EurekaLog report with call stack and accompanied information, focusing on hanged thread. This feature is handy for letting you know about problems on client machines. It will report you about any hang on the field. But if you want to debug a reproducible hang – it’s better to use Delphi’s debugger, as shown above.

Process Explorer

If you can’t use Delphi for some reason – then the only thing left is manual debugging.

You may use a Process Explorer tool for that. But before doing actual debugging, you need to do two steps. Both steps are optional, but highly recommended.

Step one – setup system debug information in Process Explorer. Windows ships with raw binaries, which have their debug information stripped. We just enabled debug info for our project (see first article’s item), but how can we do the same thing for the Windows itself? Well, Microsoft is aware of that issue and it can give you debug info separately, if you ask for this. You can download this debug info and get you pretty human-readable call stacks.

First you need to download and install Windows Debugging Tools. Next, you need to make a choice: either you download all debug info at once in one package or you can download it by request for each executable. You may try the first approach, if you want, but I prefer the second way.

You need to create a writable folder for the second way. And setup a Process Explorer of course (Options/Configure symbols):

First edit specifies path to DbgHelp.dll – pick the one from installed Windows Debugging Tools.

Second edit contains two things: first – your writable folder (it’s C:\ProgramData\DebugSymbols for me, but I’ve added a “write” access to it), where you want to store debug info; second – a debug info server, which will serve your requests for new debug info (I’m using default Microsoft’s server: http://msdl.microsoft.com/download/symbols).

After setting these things, Process Explorer will start to use debug information for system binaries. It will ask server for appropriate version of debug info and cache (store) it in the specified folder. So, you may see a "Loading symbols for ABC.exe+0xXYZ..." text while watching threads or call stacks.

Step two – you need to convert your map-file to some other format. The reason: Delphi generates several debug info formats, but all of them are Borland-related. Process Explorer, being a Microsoft’s tool, can accept a range of Microsoft’s formats, but not Borland’s ones. You can do this by using map2dbg tool. This is a simple console utility. Download, unpack, open cmd and write:

map2dbg Project1.exe

This tool will use Project1.exe and Project1.map to create a Project1.dbg, which can be used by Microsoft’s tools.

Well, I’ve just said a lot of things. Now I need to show you, what you’ll gain from all this stuff. So, here are 3 screenshots below. Walk from top to bottom: call stack without both steps (no system’s and no project’s debug info), call stack with first step only (system’s debug info present, project’s debug info is not), and call stack with both steps completed (both system’s and project’s debug info are available):

So, as you can see, enabling debug info gives you 3 things:

More readable call stack (you got module!routine+offset instead of just module+offset)
More full call stack (tracing heuristic may filter out calls without accessible debug info for validating entry)
More meaningful call stack (analyzer may show you the wrong function name without debug info – if code calls internal function, which have no name, but there is a public function with name nearby)

Okat, looks like we’re done. As you can see, it’s a loot better with debug info than without it :)

Now, what exactly should you do with Process Explorer? Well, you should run your application and let it hang. Now, you run Process Explorer, select your process, right click on it –> Properties, and select Threads tab in properties window:

Here you can watch your threads, what they are doing, who eats CPU, who is waiting for something, etc. You can select a thread and hit "Stack" button to view this thread’s call stack (see three screenshots above for example).

Of course, you can’t view variables or something like that – only states and execution points of threads. So you need to use your psychic powers to figure out what’s going on. It’s harder, than debugging in Delphi, but possible.

Threads Snapshot

Well, using Process Explorer though not that hard, but sure doesn’t look as piece-a-cake for beginner (and especially for customer!). There is a lot of work and a lot of new concepts. Honestly saying, all this mess with debug info is not very convenient, isn’t it? So I decided to write a small and simple tool, which allows you to select a process and it will dump all information about threads to text file. This information includes:

Basic info: ID, priority, etc.
Call stack. Tool supports the following sources: Borland-related, JCL, EurekaLog and madExcept. I’ll add support for Microsoft and download-at-request (as Process Explorer does) a bit later.
Wait Chain Traversal information (on Windows Vista and above only).
Thread’s context – CPU’s registers and flags.

So, you run your application and let it hang again. Now you run Threads snapshot tool and pick your hanged process. The tool will capture information and dump it into text log, which you can analyze.

User interface:

Resulting log file:

Process [ 147C / 5244 ]: H:\Test\Project88.exe (2010.06.08 16:31:07)
[ 18C4 / 6340 ] Priority: 8
Wait Chain Count = 1; Deadlock: False
Type: Thread; status: Blocked; process: [ 147C / 5244 ]; thread: [ 18C4 / 6340 ]; wait time: 701985; context switches: 4040
EIP:    772E438D  EFlags: 00000202
EBP:    0018FF1C  ESP:    0018FEEC
EAX:    000100B8  EBX:    0008E301  ECX:    00000000  EDX:    00000000ESI:    00000000  EDI:    0018FEE4
SegCS:  00000023  CegSS:  0000002B  SegGS:  0000002B  SegFS:  00000053  SegES:  0000002B  SegDS:  0000002B
----------------------------------------------------------------------------------------------------------------------------------------------
|Methods |Details|Stack   |Address |Module       |Unit     |Class    |Procedure/Method                                             |Line     |
----------------------------------------------------------------------------------------------------------------------------------------------
|Calling Thread: ID=6340; Priority=??; Class=                                                                                                |
|--------------------------------------------------------------------------------------------------------------------------------------------|
|00000008|03     |00000000|772E438D|user32.dll   |USER32   |         |WaitMessage                                                  |         |
|000000F2|04     |0018FF20|0049F5CC|Project88.exe|Forms    |Forms    |TApplication.Idle                                            |10358[0] |
|000000F2|04     |0018FF20|0049E8FF|Project88.exe|Forms    |Forms    |TApplication.HandleMessage                                   |9814[23] |
|000000F2|04     |0018FF44|0049E8E8|Project88.exe|Forms    |Forms    |TApplication.HandleMessage                                   |9813[0]  |
|000000F2|04     |0018FF44|0049EC1D|Project88.exe|Forms    |Forms    |TApplication.Run                                             |9951[201]|
|000000F2|04     |0018FF74|0049EB54|Project88.exe|Forms    |Forms    |TApplication.Run                                             |9925[0]  |
|000000F2|04     |0018FF74|004A6C91|Project88.exe|Project88|Project88|Project88                                                    |13[73]   |
|000000F2|03     |0018FF8C|770D3675|kernel32.dll |kernel32 |         |BaseThreadInitThunk                                          |         |
|000000F2|03     |0018FF98|77B29D70|ntdll.dll    |ntdll    |         |Unknown function at 77B29D70 near RtlInitializeExceptionChain|         |
|000000F2|03     |0018FFD8|77B29D4B|ntdll.dll    |ntdll    |         |Unknown function at 77B29D4B near RtlInitializeExceptionChain|         |
|000000F2|03     |0018FFD8|77B29D40|ntdll.dll    |ntdll    |         |Unknown function at 77B29D40 near RtlInitializeExceptionChain|         |
----------------------------------------------------------------------------------------------------------------------------------------------

[ 1834 / 6196 ] Priority: 8
Wait Chain Count = 1; Deadlock: False
Type: Thread; status: Blocked; process: [ 147C / 5244 ]; thread: [ 1834 / 6196 ]; wait time: 720156; context switches: 2
EIP:    77B100FD  EFlags: 00000202
EBP:    02F7FF88  ESP:    02F7FDF4
EAX:    77B51C7F  EBX:    77B51C20  ECX:    00000000  EDX:    00000000ESI:    006B7C60  EDI:    00000000
SegCS:  00000023  CegSS:  0000002B  SegGS:  0000002B  SegFS:  00000053  SegES:  0000002B  SegDS:  0000002B
-----------------------------------------------------------------------------------------------------------------------------------
|Methods |Details|Stack   |Address |Module      |Unit    |Class|Procedure/Method                                             |Line|
-----------------------------------------------------------------------------------------------------------------------------------
|Calling Thread: ID=6196; Priority=??; Class=                                                                                     |
|---------------------------------------------------------------------------------------------------------------------------------|
|00000008|03     |00000000|77B100FD|ntdll.dll   |ntdll   |     |ZwWaitForMultipleObjects                                     |    |
|000000F2|03     |02F7FF8C|770D3675|kernel32.dll|kernel32|     |BaseThreadInitThunk                                          |    |
|000000F2|03     |02F7FF98|77B29D70|ntdll.dll   |ntdll   |     |Unknown function at 77B29D70 near RtlInitializeExceptionChain|    |
|000000F2|03     |02F7FFD8|77B29D4B|ntdll.dll   |ntdll   |     |Unknown function at 77B29D4B near RtlInitializeExceptionChain|    |
|000000F2|03     |02F7FFD8|77B29D40|ntdll.dll   |ntdll   |     |Unknown function at 77B29D40 near RtlInitializeExceptionChain|    |
-----------------------------------------------------------------------------------------------------------------------------------

As you can see, this is a tiny and convenient alternative to using Process Explorer tool. You can even ask your customer to capture snapshot for you. Just don’t forget to transfer all necessary files (like map-files).

I’ve just wrote this tools for a two days. So it’s a bit untested. I think that fixed version of this tool may appear in EurekaLog v7.

A practical example

I’ve also wrote a simple test application with two buttons. You can use it as training example. First button creates several threads, which deadlock due to circular waiting. Second button creates a memory corruption, which leads to hang. Run this example, hit any button and try to figure out a reason for hang. See if you can debug these cases.

Using Delphi: if you have WCT support – than it’s piece-a-cake to debug first case: just run application, hit button, put application on pause and look on Threads window:

You’ll see a problem immediately. You have 3 threads (ignore last thread – it’s debugger’s thread, not yours). The main thread (a first one) waits for worker thread (a second one) to complete. Second thread issues a SendMessage call, which targets last thread (third). Third thread handles the message, but it needs capture critical section, which is owner by second thread. Here is a deadlock: second thread waits third thread and third thread waits second thread.

If you don’t have WCT available – then you need to do this analysis manually. You need to switch between threads and see their call stacks:

Right after switching to thread – there will be CPU window with current machine instruction. You can ignore it and click on some call stack items to view source code instead.

You can reconstruct situation by analyzing call stacks of all three threads.

How the same situation looks in Process Explorer and Threads snapshot tools? Well, you can run Process Explorer and see call stacks for threads:

You don’t see line numbers here – but at least you see routine names and their sequence. For example, note Thread1Func, dispatch-like, CriticalSection.Aquire, etc routines in the screenshot above. And again: you look at call stacks of all threads and trying to reconstruct the situation. It will be harder than using Delphi (you don’t see variables and line number), but (using a certain amount of psychic powers) possible.

Considering Threads snapshot – it produces such log (I cut some unimportant details to minimize size):

Process [ 1A1C / 6684 ]: H:\Test\HangDemo.exe (2010.06.08 17:54:33)
[ 1BD8 / 7128 ] Priority: 8
Wait Chain Count = 7; Deadlock: TrueType: Thread; status: Blocked; process: [ 1A1C / 6684 ]; thread: [ 1BD8 / 7128 ]; wait time: 1038023; context switches: 762
Type: Unknown; status: Owned; name: " "; timeout: 0; alertable: 0
Type: Unknown; status: Unknown; name: " "; timeout: 0; alertable: 0
Type: Unknown; status: Unknown; name: ""; timeout: 0; alertable: 0
Type: Unknown; status: Unknown; name: ""; timeout: 0; alertable: 0
Type: Unknown; status: Unknown; name: ""; timeout: 0; alertable: 0
Type: Unknown; status: Unknown; name: ""; timeout: 0; alertable: 0
EIP:    77B0F871  EFlags: 00000202
EBP:    0018F438  ESP:    0018F3CCEAX:    00000000  EBX:    00000000  ECX:    00000000  EDX:    00000000ESI:    00000120  EDI:    00000000
SegCS:  00000023  CegSS:  0000002B  SegGS:  0000002B  SegFS:  00000053  SegES:  0000002B  SegDS:  0000002B
----------------------------------------------------------------------------------------------------------------------------------------------
|Methods |Details|Stack   |Address |Module        |Unit      |Class   |Procedure/Method                                           |Line      |
----------------------------------------------------------------------------------------------------------------------------------------------
|Calling Thread: ID=7128; Priority=??; Class=                                                                                                |
|--------------------------------------------------------------------------------------------------------------------------------------------|
|00000008|03     |00000000|77B0F871|ntdll.dll     |ntdll     |        |ZwWaitForSingleObject                                      |          |
|000000F2|03     |0018F43C|7757077E|KERNELBASE.dll|KERNELBASE|        |WaitForSingleObjectEx                                      |          |
|000000F2|03     |0018F43C|770D117F|kernel32.dll  |kernel32  |        |WaitForSingleObjectEx                                      |          |
|000000F2|03     |0018F454|770D1141|kernel32.dll  |kernel32  |        |WaitForSingleObjectEx                                      |          |
|000000F2|03     |0018F454|770D1133|kernel32.dll  |kernel32  |        |WaitForSingleObject                                        |          |
|000000F2|03     |0018F468|770D1126|kernel32.dll  |kernel32  |        |WaitForSingleObject                                        |          |
|000000F2|04     |0018F468|004B3580|HangDemo.exe  |UnitMain  |UnitMain|TfmMain.Button1Click                                       |101[68]   |
|000000F2|04     |0018F47C|0048331F|HangDemo.exe  |Controls  |Controls|TControl.Click                                             |7178[111] |
----------------------------------------------------------------------------------------------------------------------------------------------

[ 065C / 1628 ] Priority: 8
Wait Chain Count = 5; Deadlock: TrueType: Thread; status: Blocked; process: [ 1A1C / 6684 ]; thread: [ 065C / 1628 ]; wait time: 1038024; context switches: 34
Type: Unknown; status: No access; name: " "; timeout: 0; alertable: 0
Type: Unknown; status: Unknown; name: " "; timeout: 0; alertable: 0
Type: Unknown; status: Unknown; name: ""; timeout: 0; alertable: 0
Type: Unknown; status: Unknown; name: ""; timeout: 0; alertable: 0
EIP:    77B0F871  EFlags: 00000202
EBP:    0237FD40  ESP:    0237FCDC
EAX:    00000000  EBX:    00000000  ECX:    00000000  EDX:    00000000ESI:    006293C4  EDI:    00000000
SegCS:  00000023  CegSS:  0000002B  SegGS:  0000002B  SegFS:  00000053  SegES:  0000002B  SegDS:  0000002B
-------------------------------------------------------------------------------------------------------------------------------------------
|Methods |Details|Stack   |Address |Module      |Unit    |Class   |Procedure/Method                                             |Line     |
-------------------------------------------------------------------------------------------------------------------------------------------
|Calling Thread: ID=1628; Priority=??; Class=                                                                                             |
|-----------------------------------------------------------------------------------------------------------------------------------------|
|00000008|03     |00000000|77B0F871|ntdll.dll   |ntdll   |        |ZwWaitForSingleObject                                        |         |
|000000F2|03     |0237FD44|77B28B9A|ntdll.dll   |ntdll   |        |Unknown function at 77B28B9A near RtlIntegerToUnicodeString  |         |
|000000F2|03     |0237FD44|77B28B43|ntdll.dll   |ntdll   |        |Unknown function at 77B28B43 near RtlIntegerToUnicodeString  |         |
|000000F2|03     |0237FD6C|77B12260|ntdll.dll   |ntdll   |        |RtlEnterCriticalSection                                      |         |
|000000F2|04     |0237FD6C|00441408|HangDemo.exe|SyncObjs|SyncObjs|TCriticalSection.Acquire                                     |546[4]   |
|000000F2|04     |0237FD74|004B3354|HangDemo.exe|UnitMain|UnitMain|WndProc                                                      |43[20]   |
|000000F2|03     |0237FDA8|772D6215|user32.dll  |USER32  |        |Unknown function at 772D6215 near gapfnScSendMessage         |         |
|000000F2|03     |0237FDA8|772D68E5|user32.dll  |USER32  |        |Unknown function at 772D68E5 near gapfnScSendMessage         |         |
|000000F2|03     |0237FDEC|772D6893|user32.dll  |USER32  |        |Unknown function at 772D6893 near gapfnScSendMessage         |         |
|000000F2|03     |0237FE20|772D682D|user32.dll  |USER32  |        |Unknown function at 772D682D near gapfnScSendMessage         |         |
|000000F2|03     |0237FE20|772D7172|user32.dll  |USER32  |        |Unknown function at 772D7172 near GetWindowLongW             |         |
|000000F2|03     |0237FEA8|772D682D|user32.dll  |USER32  |        |Unknown function at 772D682D near gapfnScSendMessage         |         |
|000000F2|03     |0237FEA8|772D7D2C|user32.dll  |USER32  |        |Unknown function at 772D7D2C near LoadStringW                |         |
|000000F2|03     |0237FEF0|772D7E2C|user32.dll  |USER32  |        |DispatchMessageW                                             |         |
|000000F2|03     |0237FEF0|772D7EB8|user32.dll  |USER32  |        |GetMessageW                                                  |         |
|000000F2|03     |0237FF0C|772D7E92|user32.dll  |USER32  |        |GetMessageW                                                  |         |
|000000F2|04     |0237FF0C|004B3420|HangDemo.exe|UnitMain|UnitMain|Thread1Func                                                  |71[132]  |
|000000F2|04     |0237FF78|00405FE4|HangDemo.exe|System  |System  |ThreadWrapper                                                |13579[40]|
|000000F2|03     |0237FF8C|770D3675|kernel32.dll|kernel32|        |BaseThreadInitThunk                                          |         |
|000000F2|03     |0237FF98|77B29D70|ntdll.dll   |ntdll   |        |Unknown function at 77B29D70 near RtlInitializeExceptionChain|         |
|000000F2|03     |0237FFD8|77B29D4B|ntdll.dll   |ntdll   |        |Unknown function at 77B29D4B near RtlInitializeExceptionChain|         |
|000000F2|03     |0237FFD8|77B29D40|ntdll.dll   |ntdll   |        |Unknown function at 77B29D40 near RtlInitializeExceptionChain|         |
-------------------------------------------------------------------------------------------------------------------------------------------

[ 1504 / 5380 ] Priority: 8
Wait Chain Count = 5; Deadlock: TrueType: Thread; status: Blocked; process: [ 1A1C / 6684 ]; thread: [ 1504 / 5380 ]; wait time: 1038024; context switches: 8
Type: Unknown; status: Running; name: " "; timeout: 0; alertable: 0
Type: Unknown; status: Unknown; name: " "; timeout: 0; alertable: 0
Type: Unknown; status: Unknown; name: ""; timeout: 0; alertable: 0
Type: Unknown; status: Unknown; name: ""; timeout: 0; alertable: 0
EIP:    772D723B  EFlags: 00000202
EBP:    03C1FF28  ESP:    03C1FEE8
EAX:    00BD2C30  EBX:    005E0EC0  ECX:    00000000  EDX:    00000000ESI:    00BD2C30  EDI:    00000401
SegCS:  00000023  CegSS:  0000002B  SegGS:  0000002B  SegFS:  00000053  SegES:  0000002B  SegDS:  0000002B
-------------------------------------------------------------------------------------------------------------------------------------------
|Methods |Details|Stack   |Address |Module      |Unit    |Class   |Procedure/Method                                             |Line     |
-------------------------------------------------------------------------------------------------------------------------------------------
|Calling Thread: ID=5380; Priority=??; Class=                                                                                             |
|-----------------------------------------------------------------------------------------------------------------------------------------||00000008|03     |00000000|772D723B|user32.dll  |USER32  |        |Unknown function at 772D723B near GetPropW                   |         |
|000000F2|03     |03C1FF2C|772DCC02|user32.dll  |USER32  |        |Unknown function at 772DCC02 near GetWindow                  |         |
|000000F2|03     |03C1FF2C|772DCD7C|user32.dll  |USER32  |        |SendMessageW                                                 |         |
|000000F2|03     |03C1FF50|772DCD35|user32.dll  |USER32  |        |SendMessageW                                                 |         |
|000000F2|04     |03C1FF50|004B350F|HangDemo.exe|UnitMain|UnitMain|Thread2Func                                                  |85[67]   |
|000000F2|04     |03C1FF78|00405FE4|HangDemo.exe|System  |System  |ThreadWrapper                                                |13579[40]|
|000000F2|03     |03C1FF8C|770D3675|kernel32.dll|kernel32|        |BaseThreadInitThunk                                          |         |
|000000F2|03     |03C1FF98|77B29D70|ntdll.dll   |ntdll   |        |Unknown function at 77B29D70 near RtlInitializeExceptionChain|         |
|000000F2|03     |03C1FFD8|77B29D4B|ntdll.dll   |ntdll   |        |Unknown function at 77B29D4B near RtlInitializeExceptionChain|         |
|000000F2|03     |03C1FFD8|77B29D40|ntdll.dll   |ntdll   |        |Unknown function at 77B29D40 near RtlInitializeExceptionChain|         |
-------------------------------------------------------------------------------------------------------------------------------------------

Unfortunately, WCT output is not very well formatted yet (that’s my home-work! :) ), but you still can track wait chains. Besides, there are call stacks for threads (without system debug info – that’s my second home-task!), which strongly hint what is happening.

Nothing extremely hard here.

The second case is significantly harder. Because hang is not an error here – it’s just a consequence of another problem.

Okay, so you run application in Delphi and it hangs, so we put it on pause. We have only one thread, so WCT won’t give you any useful hints, even if it’s available. So we switch to main thread and analyze call stack:

And again: by clicking on call stack’s items we can see source code.

But analyzing call stack only is not enough this time. It’s unclear what happened. And application seems even not hanging – it’s doing something. That’s why we start doing single-step debugging to see, what application is doing. After walking a bunch of lines, we discover that application is busy doing some cycle with Sleep inside. This cycle constantly checking some flag. We can see that this code belongs to FastMM. And by reading comments, we realize that FastMM is trying to acquire some kind of lock. The flag is “busy/free” switch. Since FastMM is checking this flag for an hour - at this point it becomes clear that something isn’t right. Besides, we have only one thread in application – there is nobody else, who can be waited for. In other words, the state of flag is no longer corresponds to reality. I.e. it’s corrupted.

Unfortunately, resolving memory corruption is not that easy work and it’s another a big and separate topic, which I’ve finished discussing not so long ago.

So, the task of this case is to find the reason. We’ve found it: it’s a memory corruption problem. So, our hang analysis is done, but the problem is not identified yet and isn’t solved. So, the next step is analyzing for memory problems.

Actually, this second case looks the same in all tools: we get the same call stack, which can vary from time to time, but it will point to Sleep in 99% of cases. So we just check it few times, see the same values and suppose endless cycle here. We analyze, why there can be endless cycle and make an educated guess about memory problems. Here is example of the same call stack in Process Explorer:

Okay, I’m done for today. Hope that you’ve enjoyed today’s lesson.

P.S. If you want – there is an additional bit of information: how to get source line from pointer/address – see "How to find the exception source line" session here.

New Exception class in Delphi 2009 and above

2010-05-05T16:25:00.032+01:00

Exception class from SysUtils unit is a base class for all exceptions in your application. It wasn't changed since Delphi 2, if I remember correctly. But now, it got new properties and, thus, new features - starting with Delphi 2009 and above. So, I decided to give an overview of new capabilities of Exception class.

Okay, in Delphi 2007 and below, the Exception class looks like this:

type
  Exception = class(TObject)
  // ...
  public
    // All constructors below are just different variants of filling Message and HelpContext properties
    constructor Create(const Msg: string);
    // ...
    property HelpContext: Integer read FHelpContext write FHelpContext;
    property Message: string read FMessage write FMessage;
  end;

Actually, we have only a textual description of the problem here - Message property (HelpContext is barely used on practice). Of course, we can declare a new custom class, where we can add anything we like, but don't you think that base Exception class could be a more useful? How about information about previous exception?

Well, nevertheless, the Exception class has changed in Delphi 2009 (doh, finally!) by acquiring new properties and methods:

type
  Exception = class(TObject)
  // ...
  protected
    procedure SetInnerException;
    procedure SetStackInfo(AStackInfo: Pointer);
    function GetStackTrace: string;
    procedure RaisingException(P: PExceptionRecord); virtual;
  public
    constructor Create(const Msg: string);
    // ...
    function GetBaseException: Exception; virtual;
    function ToString: string; override;
    property BaseException: Exception read GetBaseException;
    property HelpContext: Integer read FHelpContext write FHelpContext;
    property InnerException: Exception read FInnerException;
    property Message: string read FMessage write FMessage;
    property StackTrace: string read GetStackTrace;
    property StackInfo: Pointer read FStackInfo;
  class var
    GetExceptionStackInfoProc: function (P: PExceptionRecord): Pointer;
    GetStackInfoStringProc: function (Info: Pointer): string;
    CleanUpStackInfoProc: procedure (Info: Pointer);
    class procedure RaiseOuterException(E: Exception); static;
    class procedure ThrowOuterException(E: Exception); static;
  end;

All new things belongs to one of the following categories:

Nested exceptions support
Exception tracing support

Let's deal with them individually.

Nested exceptions

Nested exception (sometimes called "chained exception") appears in situations, when new exception is raised, while your code processes another exception - i.e. if there is exception in exception handler (finally or except block). If you don't have support for nested exceptions - then old exception will be lost. Sometimes, it's what you want anyway, but sometimes it's not.

Two examples. A first one:

procedure TSomeClass.SaveToStream(const AStream: Stream);
begin
  try
    // ... saving instance to stream here
  except
    raise ESomeClassSaveError.Create('Error saving data to stream');
  end;
end;

I think that this example is quite clear. We raise high-level error (ESomeClassSaveError) from low-level error (it can be plain EStreamError due to out of disk space or something more tricky like range error or access violation due to damage of internal instance's state). Anyway, the user will get his description - and that was our target. However, note that information about original issue is lost, as high-level exception masked low-level one.

In this example, we've raised exception by ourself, as expected one. In the next example, this will be unexpected event.

Example two:

  SomeClass := TSomeClass.Create;
  try
    // ... suppose that there is some exception here
  finally
    FreeAndNil(SomeClass); // suppose that SomeClass's destructor raises exception too
  end;

In this example, we didn't expect exception to happen. We got unexpected exception in destructor of SomeClass instance. It may be some access violation, because we wasn't prepared for something. Similar to previous example: the new exception hides previous one, but we don't want it in this case - we want original exception, so we can fix it. We can only fix the second one without this information, which won't solve original problem.

Anyway, in both examples we leak important information: an information about previous exception, which can give us hints to the problem. And it is when nested exceptions appears.

The new exception class (whoa, that sure took a lot of time, but I should introduce newbies to nested exceptions after all) have InnerException and BaseException properties. Both properties are set (managed) by SysUtils unit automatically, you don't need to fill them manually. So, you can read and use them. InnerException property stores the previous exception. BaseException stores exception with most nested level - i.e. the very first exception, which started this exception chain. If there are only two exceptions in the chain (old and new, as in examples above), then InnerException will be equal to BaseException. If there is only one exception (i.e. there is no nested exception at all) - then both properties will be equal to nil.

Nested exceptions, however, aren't saved by default. In order to save nested exception, you need to raise new exceptions via Exception.RaiseOuterException (Delphi's style) or Exception.ThrowOuterException (C++ Builder's style). For example:

procedure TSomeClass.SaveToStream(const AStream: Stream);
begin
  try
    // ... saving instance to stream here
  except
    Exception.RaiseOuterException(ESomeClassSaveError.Create('Error saving data to stream'));
  end;
end;

After executing this code, we will get exception of ESomeClassSaveError class, which will have non-nil InnerException property (and BaseException too), which, in turn, will contain original exception (EStreamError or whatever it was).

In the second example (the one with exception in destructor) we still get InnerException = nil, cause RaiseOuterException isn't used anywhere.

How this support of nested exceptions affects error messages? Well, Message property didn't changed - it's a message of current exception only. So, any code, which isn't awared about nested exceptions, will show only message of the last exception. You can use ToString method of Exception class to show all messages from all exceptions - each exception will be on new line (obviously, ToString will be equal to Message property in case of single exception in the chain).

On the other side, Application.ShowException method looks a bit strange: this method shows message from BaseException - I suppose that it's not what you want (see our first example). That's why I think that you may want to make your own Application.OnException event handler to change this behaviour. For example:

procedure TForm1.ApplicationEvents1Exception(Sender: TObject; E: Exception);
var
  Msg: String;
begin
  Msg := E.Message; // or E.ToString
  Application.MessageBox(PChar(Msg), PChar(Application.Title), MB_OK + MB_ICONSTOP);
end;

The next thing: I really don't understand, why there is no way to enable capturing of nested exceptions in each case, without need to use special raising routines. If you want this feature - you may add the following unit to your uses clause (attention: this is a hack, not documented way):

unit ChainedExceptionsAlways;

interface

implementation

uses
  SysUtils;

var
  OldRaiseExceptObject: Pointer;

type
  EExceptionHack = class
  public
    FMessage: string;
    FHelpContext: Integer;
    FInnerException: Exception;
    FStackInfo: Pointer;
    FAcquireInnerException: Boolean;
  end;

procedure RaiseExceptObject(P: PExceptionRecord);
type
  TRaiseExceptObjectProc = procedure(P: PExceptionRecord);
begin
  if TObject(P^.ExceptObject) is Exception then
    EExceptionHack(P^.ExceptObject).FAcquireInnerException := True;

  if Assigned(OldRaiseExceptObject) then
    TRaiseExceptObjectProc(OldRaiseExceptObject)(P);
end;

initialization
  OldRaiseExceptObject := RaiseExceptObjProc;
  RaiseExceptObjProc := @RaiseExceptObject;
end.

After you add this unit (the sooner - the better) - our second example will start to collect previous exceptions too, and you can use either Exception.RaiseOuterException or just raise in the first example - the behaviour will be the same.

Exception tracing

Well, if you read our blog from time to time or if you're using EurekaLog - you're already familiar with exception tracers, so I'll skip introduction here, but I still want to briefly describe architecture of exception tracers to show how new properties of new Exception class fit in this picture.

Application is set of machine's codes, which are just numbers in bytes. By default, there is no reference to source code in the compiled application. That's why you can't built a human-readable call stack. So, you need to use some solution, which do the following:

Embeds debug information (reference between raw code and sources) into compiled application in public or proprietary format.
Installs hook on exception's raising (i.e. hook on any function, which is called when exception occurs). This includes patching of something (import table or code's section).
Builds a call stack in the hook handler by using any stack tracing method.

There is no problem with first and third item - they are documented and acceptable solutions. Item two is different, as it includes using of dirty hacks. That's why, Delphi's developers have added support for exception tracers in Exception class to simplify things here (it's a bit late, but late is still better, than never). All that this support do - it allows you to call your routine at exception raise, removing need to hook/patch binary. Nothing more, nothing less. In other words, there is no way to get exception tracing in Delphi out of the box.

One more time: this new feature is designed for developers of exception tracers.

Since all already written exception tracers already works somehow - they do not use this new feature, that's why, if you want to get exception's call stack, then you need to call appropriate function of exception tracer. For example, it's GetLastExceptionCallStack for EurekaLog and JclLastExceptStackList for JCL.

However, you can integrate any existing exception tracer into this new architecture - and this task is quite simple. If your exception tracer consists of distinct modules, you can turn off that part, which is responsible for hooking, leaving only stack tracing and debug information parts.

Okay, so if you've decided that you want it, here is how you can do it.

First thing is that the mentioned support includes System and SysUtils units. All base functionality is included into System unit - similar to old features of exceptions. SysUtils unit is just a convenient "wrapper" for System unit's functionality. System unit provides certain events (ExceptProc, ErrorProc, ExceptClsProc, ExceptObjProc, RaiseExceptionProc, RTLUnwindProc, RaiseExceptObjProc, ExceptionAcquired, ExceptionClass, SafeCallErrorProc, AssertErrorProc and AbstractErrorProc), which SysUtils unit uses. You should not use these events, until you throw out SysUtils unit. Instead, use SysUtils's services.

So, what SysUtils unit can offer us? It offers a new Exception class, which have new events like GetExceptionStackInfoProc, GetStackInfoStringProc and CleanUpStackInfoProc. They aren't used by anyone by default - as I said, it's impossible to implement them, as default application doesn't have information for that.

This means that it's we, who need to implement these handlers. Since we don't write exception tracer, but use already existing one - we only need to write a wrapper. Wrapper will just call appropriate function from exception tracer, that's all. For example:

unit ExceptionEurekaLogSupport;

interface

implementation

uses
  SysUtils, Classes, ExceptionLog;

function GetExceptionStackInfoEurekaLog(P: PExceptionRecord): Pointer;
const
  cDelphiException = $0EEDFADE;
var
  Stack: TEurekaStackList;
  Str: TStringList;
  Trace: String;
  Sz: Integer;
  DI: PEurekaDebugInfo;
begin
  Stack := GetCurrentCallStack;
  try
    New(DI);
    DI^.ModuleInfo := ModuleInfoByAddr(Cardinal(P^.ExceptAddr));
    if P^.ExceptionCode = cDelphiException then
      GetSourceInfoByAddr(Cardinal(P^.ExceptAddr), DI)
    else
      GetSourceInfoByAddr(Cardinal(P^.ExceptionAddress), DI);
    Stack.Insert(0, DI);

    Str := TStringList.Create;
    try
      CallStackToStrings(Stack, Str);
      Trace := Str.Text;
    finally
      FreeAndNil(Str);
    end;
  finally
    FreeAndNil(Stack);
  end;

  if Trace <> '' then
  begin
    Sz := (Length(Trace) + 1) * SizeOf(Char);
    GetMem(Result, Sz);
    Move(Pointer(Trace)^, Result^, Sz);
  end
  else
    Result := nil;
end;

function GetStackInfoStringEurekaLog(Info: Pointer): string;
begin
  Result := PChar(Info);
end;

procedure CleanUpStackInfoEurekaLog(Info: Pointer);
begin
  FreeMem(Info);
end;

initialization
  Exception.GetExceptionStackInfoProc := GetExceptionStackInfoEurekaLog;
  Exception.GetStackInfoStringProc := GetStackInfoStringEurekaLog;
  Exception.CleanUpStackInfoProc := CleanUpStackInfoEurekaLog;
end.

And here is variant for JCL:

unit ExceptionJCLSupport;

interface

implementation

uses
  SysUtils, Classes, JclDebug;

function GetExceptionStackInfoJCL(P: PExceptionRecord): Pointer;
const
  cDelphiException = $0EEDFADE;
var
  Stack: TJclStackInfoList;
  Str: TStringList;
  Trace: String;
  Sz: Integer;
begin
  if P^.ExceptionCode = cDelphiException then
    Stack := JclCreateStackList(False, 3, P^.ExceptAddr)
  else
    Stack := JclCreateStackList(False, 3, P^.ExceptionAddress);
  try
    Str := TStringList.Create;
    try
      Stack.AddToStrings(Str, True, True, True, True);
      Trace := Str.Text;
    finally
      FreeAndNil(Str);
    end;
  finally
    FreeAndNil(Stack);
  end;

  if Trace <> '' then
  begin
    Sz := (Length(Trace) + 1) * SizeOf(Char);
    GetMem(Result, Sz);
    Move(Pointer(Trace)^, Result^, Sz);
  end
  else
    Result := nil;
end;

function GetStackInfoStringJCL(Info: Pointer): string;
begin
  Result := PChar(Info);
end;

procedure CleanUpStackInfoJCL(Info: Pointer);
begin
  FreeMem(Info);
end;

initialization
  Exception.GetExceptionStackInfoProc := GetExceptionStackInfoJCL;
  Exception.GetStackInfoStringProc := GetStackInfoStringJCL;
  Exception.CleanUpStackInfoProc := CleanUpStackInfoJCL;
end.

Depending on the exception tracer that you use - you need to include either one of these units into uses clause (the sooner - the better) and you'll magically have your stack traces.

Use the following example of OnException handler to verify that it's working (note that you need to turn on RTL handlers calling and turn off EurekaLog's dialog in EurekaLog options):

procedure TForm1.ApplicationEvents1Exception(Sender: TObject; E: Exception);
var
  Msg, Stack: String;
  Inner: Exception;
begin
  Inner := E;
  Msg := '';
  while Inner <> nil do
  begin
    if Msg <> '' then
      Msg := Msg + sLineBreak;
    Msg := Msg + Inner.Message;
    if (Msg <> '') and (Msg[Length(Msg)] > '.') then
      Msg := Msg + '.';

    Stack := Inner.StackTrace;
    if Stack <> '' then
    begin
      if Msg <> '' then
        Msg := Msg + sLineBreak + sLineBreak;
      Msg := Msg + Stack + sLineBreak;
    end;

    Inner := Inner.InnerException;
  end;
  Application.MessageBox(PChar(Msg), PChar(Application.Title), MB_OK + MB_ICONSTOP);
end;

Let's see example. The following code:

procedure TForm1.Button1Click(Sender: TObject);
begin
  try
    PInteger(nil)^ := 0;
  except
    Exception.RaiseOuterException(Exception.Create('Error occured'));
  end;
end;

will produce the following message box:

Error occured.

(000A7D0F){Project68.exe} [004A8D0F] Unit1.TForm1.Button1Click (Line 61, "Unit1.pas" + 4) + $16
(00004901){Project68.exe} [00405901] System.@RaiseExcept (Line 12194, "System.pas" + 47) + $0
(00076473){Project68.exe} [00477473] Controls.TControl.Click (Line 7178, "Controls.pas" + 9) + $8
(0007A7CC){Project68.exe} [0047B7CC] Controls.TWinControl.WndProc (Line 9819, "Controls.pas" + 144) + $6
(00057BE4){Project68.exe} [00458BE4] StdCtrls.TButtonControl.WndProc (Line 4377, "StdCtrls.pas" + 13) + $4
(0007A91F){Project68.exe} [0047B91F] Controls.DoControlMsg (Line 9888, "Controls.pas" + 12) + $11
(0007A7CC){Project68.exe} [0047B7CC] Controls.TWinControl.WndProc (Line 9819, "Controls.pas" + 144) + $6
(0009AC94){Project68.exe} [0049BC94] Forms.TCustomForm.WndProc (Line 3960, "Forms.pas" + 191) + $5
(00079E6C){Project68.exe} [0047AE6C] Controls.TWinControl.MainWndProc (Line 9540, "Controls.pas" + 3) + $6
(0003DC58){Project68.exe} [0043EC58] Classes.StdWndProc (Line 13014, "Classes.pas" + 8) + $0
(0007A8CC){Project68.exe} [0047B8CC] Controls.TWinControl.DefaultHandler (Line 9860, "Controls.pas" + 30) + $17
(0007A7CC){Project68.exe} [0047B7CC] Controls.TWinControl.WndProc (Line 9819, "Controls.pas" + 144) + $6
(00057BE4){Project68.exe} [00458BE4] StdCtrls.TButtonControl.WndProc (Line 4377, "StdCtrls.pas" + 13) + $4
(0003DC58){Project68.exe} [0043EC58] Classes.StdWndProc (Line 13014, "Classes.pas" + 8) + $0

Access violation at address 004A8CE8 in module 'Project1.exe'. Write of address 00000000.

(000A7CE8){Project68.exe} [004A8CE8] Unit1.TForm1.Button1Click (Line 59, "Unit1.pas" + 2) + $4
(0000453F){Project68.exe} [0040553F] System.@HandleAnyException (Line 11245, "System.pas" + 13) + $0
(00076473){Project68.exe} [00477473] Controls.TControl.Click (Line 7178, "Controls.pas" + 9) + $8
(0007A7CC){Project68.exe} [0047B7CC] Controls.TWinControl.WndProc (Line 9819, "Controls.pas" + 144) + $6
(00057BE4){Project68.exe} [00458BE4] StdCtrls.TButtonControl.WndProc (Line 4377, "StdCtrls.pas" + 13) + $4
(0007A91F){Project68.exe} [0047B91F] Controls.DoControlMsg (Line 9888, "Controls.pas" + 12) + $11
(0007A7CC){Project68.exe} [0047B7CC] Controls.TWinControl.WndProc (Line 9819, "Controls.pas" + 144) + $6
(0009AC94){Project68.exe} [0049BC94] Forms.TCustomForm.WndProc (Line 3960, "Forms.pas" + 191) + $5
(00079E6C){Project68.exe} [0047AE6C] Controls.TWinControl.MainWndProc (Line 9540, "Controls.pas" + 3) + $6
(0003DC58){Project68.exe} [0043EC58] Classes.StdWndProc (Line 13014, "Classes.pas" + 8) + $0
(0007A8CC){Project68.exe} [0047B8CC] Controls.TWinControl.DefaultHandler (Line 9860, "Controls.pas" + 30) + $17
(0007A7CC){Project68.exe} [0047B7CC] Controls.TWinControl.WndProc (Line 9819, "Controls.pas" + 144) + $6
(00057BE4){Project68.exe} [00458BE4] StdCtrls.TButtonControl.WndProc (Line 4377, "StdCtrls.pas" + 13) + $4
(0003DC58){Project68.exe} [0043EC58] Classes.StdWndProc (Line 13014, "Classes.pas" + 8) + $0

Also note, that (in case of JCL) we do not use hooking code (which is located in JclHookExcept unit), since new Exception class takes care of that for us. We only use JCL's features on stack tracing and debug information (which are located in JclDebug unit).

Memory problems in Delphi apps - final article

2010-03-30T16:26:00.059+01:00

I've published several articles already, which cover different aspects of memory's issues. I'm going to give a short review of these articles first and then finish this series with current article. If you didn't read my previous articles or if you're a newbie - I recommend to read them from first to last before you'll start reading this one.

Okay, back to our series, here comes it's

1. Introducing article about pointers (and here is a separate article about strings). Well, these are not my articles, but it's great and well-written ones, so I'll just use a link to them. These article discuss pointers, reference types and many other things, which you should know about before you'll start debugging your memory problems.

2. Preparations. Tools for diagnosing memory issues require certain environment and conditions. You can improve (or make worse) their results by adjusting the options of your projects. That's why, before advancing any further, this article explains various debugging options and their affect on your application. There are also listed options sets for typical use cases (note, those are only recommendations). Well, this article is optional, if you already have a memory issue, but do not suffer from low detalization of your bug report (assuming you have it at all).

3. EAccessViolation exception. This exception (as, probably, most common one in Delphi) may have various reasons, but all reasons have something in common: they all are the memory problems. This article explains what this exception is, examines examples of its reasons, and studies how can you diagnose and fix it - either by debugging manually or by using debugging tools. As it becomes clear to the end of article, the EAccessViolation exception is actually a blessing - because it allows you to spot (and react on) the problem immediately. The article provides an example of how the same code can lead (depending on external conditions) either to exception ("good"), to the application's crash/data corruption ("bad"), or even to normal operation, although the code still has a bug ("very bad"). The article discusses only the first case - an obvious exception, and leaves other cases for later (this article).

4. Memory leaks. Though I'm talking about memory leaks in this article, my real target is to introduce memory managers and their debug mode. I've mentioned this at the beginning of the article - namely, that leak searching is actually close related to debugging memory corruptions. I've concentrated on leaks only in this article, saving other interesting questions for later (this article again).

5. (off-topic 1) How to read bug reports. This article is not related to this series about memory issues, but I think it's worth to mention it. Apparently, many beginners have difficulties with reading/understanding bug reports, which are generated by different debugging tools, mentioned in previous articles. So, this article tries to explain, how you should read reports and to discuss typical mistakes in interpreting reports.

6. (off-topic 2) Add-on to my article about memory leaks. Well, it looks like many people have used my first article about memory leaks for its most obvious purpose: catching mem-leaks ;) Apparently, original article wasn't good enough for this purpose, because there were many additional questions from people about it. Well, may be it is not that surprising, if you recall that original purpose was to introduce debugging memory managers. That's why I've decided to write an add-on, which covers missed topics. This add-on wasn't commented well, but we're receiving much less questions about mem-leaks, so I think that it hit the spot.

7. This article. I've only mentioned most scary problems in the previous articles, dealing with simple cases first and leaving hard cases for later. Now it's time to deal with them, therefore finishing the series. To put it short: if you have strange issues with crashes, empty reports (apart most obvious case of low-detailed debug info) or hangs - this article is for you (note: sometimes hang can be caused by memory corruption too: example).

Now, I've made a review of this series, so we can get back to this article itself:

When EurekaLog can not help you (or: what we will talk about)

Note: when I say "EurekaLog" - I actually mean any similar tool, like JclHookExcept, madExcept, WER, etc.

EurekaLog is a tool for debugging your application. It is not a "silver bullet", which magically solves any of your problem. Like every tool, EurekaLog have the purpose, the scope and the limitations. That's why there are cases, when this great tool can not help you.

The problem is that each tool requires some basics to function. You break this basis - you broke the tool. And there is no way a tool can escape that. Well, usually it's hard thing to do. But it is possible and it even may be not so hard, if you mis-use pointers and memory usage. I've already gave examples of simple code that can crash your application and nobody will do a thing about it (see, for example, the end of "Looking for the Access Violation’s reason by analyzing the code" block in this article).

So, all those scary problems, which make your debugging tools useless, are all memory issues. Namely: memory corruption. Any tool need to store its data somewhere. If a bug in your application damages or even erase this data - the game is over. Even more: the code can trash not only tool's data, but app's critical data too - like return addresses or saved pointers to exception handlers. You may already noticed that most hard cases includes stack corruption. That's because critical data (which you can affect at all from user-mode) is stored in application's stack.

Well, some of memory corruption problems (apparently, not so hardest ones) can be even detected by EurekaLog. EurekaLog tries to create an simplified version of the bug report in that case and to display the following message:

As you can see, EurekaLog thinks that this is its fault, though most common case is it's somebody else's fault. EurekaLog says so, because it have encounter a problem during its work on exception handling. But this problem may be caused either a bug in EurekaLog or a bug in other code (may be yours, may be not).

EurekaLog suggests you to send this report to EurekaLog's developers. If you agree, it will open your default e-mail client with such report (example):

Version   : 6.0.23
Date      : Sat, 27 Mar 2010 13:09:00 +0300
OS        : Microsoft Windows 7 (64 bit)
RAD       : BDS 7.0
Dump      : $89 $10 $8B $45 $94 $8B $40 $08 $48 $85 $C0 $7C $7B $40 $89 $85 $30 $FF $FF $FF $C7 $85 $38 $FF $FF $FF $00 $00 $00 $00 $8B $95
Section   : 16
LastExcept: Exception
Address   : $004A7FE6 - [00400000] Project38.exe - ExceptionLog.pas -  - InternalExceptNotify - 14540[484]
Exception : EAccessViolation
Message   : Access violation at address 004A7FE6 in module 'Project38.exe'. Write of address 00000000
Call Stack: 00 $00419534 - [00400000] Project38.exe - SysUtils.pas - Exception - Create - 17419[0]
            01 $00515E63 - [00400000] Project38.exe - Unit39.pas - TForm39 - Button1Click - 29[1]
            02 $004A6E6C - [00400000] Project38.exe - ExceptionLog.pas -  - InternalExceptNotify - 14056[0]
            03 $004A879A - [00400000] Project38.exe - ExceptionLog.pas - TExceptionThread - Execute - 14689[2]
            04 $0043E7DA - [00400000] Project38.exe - Classes.pas -  - ThreadProc - 11018[8]
            05 $0040667C - [00400000] Project38.exe - System.pas -  - ThreadWrapper - 13579[33]
            06 $76EF3675 - [76EE0000] kernel32.dll
            07 $77C89D70 - [77C50000] ntdll.dll
            08 $77C89D4B - [77C50000] ntdll.dll
            09 $77C89D40 - [77C50000] ntdll.dll

This is a short version of bug report for fatal errors. Sure, you should send us such reports - however, if you see that the problem in question is access violation or invalid pointer exceptions - this means memory corruption problem in your application. Which means that you better check your application very carefully, as in most cases such reports indicate problem in your or 3rd party code and not in EurekaLog. Either way, if you think that this is definitely a problem with EurekaLog - be prepare for long conversation with developers. That's because memory corruption problem can't be fixed from one report. Report can only indicate it. To fix it - additional work is required.

It's worth to note, that EurekaLog integrates itself into Delphi's IDE, hooking all IDE exceptions too. I.e. EurekaLog catches not only errors in your applications, but in Delphi too. This is useful feature for old Delphi's version, which do not have similar feature. New Delphis come with their own tool for automatic reporting - so you may want to disable this EurekaLog behaviour. In any case - you can turn off or on this behaviour in "EurekaLog"/"EurekaLog IDE Options" menu:

"IDE Integration" check-box enables exception hooking inside IDE. Here is an example of IDE problem.

Why I'm telling this stuff here? Well, apparently, not all people realize, that EurekaLog hooks IDE too. Sometimes they confuse errors with their applications and errors with IDE. If you're in doubt - turn off this option.

Okay, let's get back to our apps. An report about internal problem is just one case of how memory corruption can manifest itself. Other (the harder ones) cases include application's crash or hang. You will not get any message or report in those cases.

How to diagnose and fix memory problem

If you have memory corruption issue and you gor a report for it - this report will be a simple indication that you have a problem. You won't be able to fix the problem by using this report. Why? Because any such report is a note that the problem had occurred somewhere and some time ago. It's somehow similar to memory leaks - and we've already discussed it earlier. The problem is that nobody can scan each CPU instruction and ask: "is this command going to corrupt my memory?" That's why all checks are performed from time to time at certain checkpoints. Besides, only special data can be validated automatically. For example, if we take mem-leaks case - the checkpoints are calls of memory manager's routines and verified data are internal structures and freed memory. But even in that simplest case memory manager does not scan the entire memory pool on each request, limiting check to one memory block in question only. This is a usual trade-off between speed and functionality.

Okay, so, having a report, you will know that there is a problem. But you don't know where is it. You have a chance to locate it in the case of memory leaks, but not in the case of memory corruption. That's because you have some references to code for leaks, but references to code for mem corruptions are off-topic. The real culprit-code can sit a million instructions away in space and time from the code, which crashed because of it, and there is no any references to it. That's why the very first thing, that you should try to do (wherever you have a report or just crash/hang) is to try to reproduce the problem. Sometimes you can do it easily; sometimes it is possible, but hard to do; and often it is just not possible at all.

If you've managed to reproduce the problem - then it is a very simple case. Just debug your application as much as you want. I suspect that the most useful tools here will be memory breakpoints. General strategy is simple: you need to find a moment, when memory is committed, but is not corrupted yet. You place a break-point on the memory (yes, Delphi's debugger can do it; I'll not discuss it here - please, refer to other resources or Delphi's help) and you just run your application. As soon as this break-point fire - you'll find the culprit for memory corruption. Make yourself at home and take your time: analyze the call stack, variables, etc, etc - the situation is under your control.

So, to put it short: the main question here is to locate the problem (assuming you can reproduce it at all). I'll discuss the different methods below, which you can use to locate the problem. Some of them you can use always - both in debug and release version. Some of them are only applicable to debug version.

If you aren't able to solve the problem (either you can't reproduce it or you can reproduce, but can't locate it) – then the only options is to use passive methods. I.e. things, which aren't directed to your particular issue, but rather helps you to improve your code - that way after improvements you'll be able to diagnose the problem or it may be that the problem will go away without doing anything specific. For example, if your code is chaotic mix of totally unrelated routines calls without slightest sign of logic (okay, I'm just joking - I do not think that you're that bad :-D ) - you can spend half a year looking for the reason (and still not solve it). Or you can spend few months to refactor your code, to improve it - and then hunt down and fix not only this problem, but other issues too, which you've spotted because your code becomes much clearer.

I also think that we may publish in our blog examples of debugging particular problems. For example, I want to publish a demo how to debug a hang in your application. But this is for the next time. Just be sure to scan and read us periodically ;)

Problem's locating (active methods)

First at all, you should analyze, what can be your problem. There are two main cases here: dynamic memory (heap) or the stack. Depending on the answer you may use methods for the heap or for the stack. For example, using debugging memory manager can help you with the memory corruptions in the heap, but it can do nothing about stack corruptions. If you aren't sure about it - just use all methods ;)

1. Using debugging memory manager (heap). Debugging memory manager is any memory manager, which provides additional features for debugging memory problems. We meet some examples (like EurekaLog or FastMM) in articles about memory leaks. That's because searching for memory leaks and searching for memory corruption bugs use the very similar approach. EurekaLog's case: additional checks are enabled with "Catch leaks exceptions" option. FastMM's case: see "CheckHeapForCorruption" and "CatchUseOfFreedInterfaces" options (together with "FullDebugMode"). Other options may affect the results too, but these ones are primary options for memory corruption checks. We've already discussed an installation and use of debugging memory managers in previous articles, so I won't go into details here: it's the same for memory corruption problems as it's for mem-leaks – just don't forget to enable additional options and run your application, until memory manager will catch a problem.

Apart from already discussed memory managers - I want to mention a SafeMM memory manager. This is a debugging memory manager too, but it's a bit different from already discussed ones. You can download it here. And here is an example of use (a video) – the first part is about profiling and second part (starting from 22-nd minute) is about memory problems.

2. Enabling debugging options (stack and heap). We mentioned this before too. The main option here is "Range check errors", which allows you to catch out of range errors in array-based data structures (note, that this option have a bug in old Delphi's versions). Besides this option, you may want to disable inlining and optimization (to simplify debugging and to avoid bugs like this). Unfortunately, Delphi's compiler do not have a more generic option for checking stack's state like others compilers have. I've created a request for it – feel free to vote for it. If such feature exist - this would greatly simplify hunting for stack corruption problems. Well, without compiler support - we have only manual workarounds.

3. Forced checkpoints (stack and heap). As I've already said, any report about memory problem reports only about moment of detection, not about the problem itself. You must locate the problem. But how can you do it? Obviously, you need to find a point, when problem is not occurred yet (memory is not corrupted); find the point, when memory is corrupted. Therefore, the problem will sit somewhere between those two points. Each of these moments will be a checkpoint. Moving (or creating) checkpoints - you can reduce code's area with problem until you locate it. Sometimes, those checkpoints are created automatically. For example, debugging memory manager validates memory block each time its routine is called for this block. For the stack: it can be routine leave. Since you successfully leave the routine - this means that return address wasn't damaged, so there was no stack corruption (at least some type of it). There may be other examples, but it's not important now. I just want to say that if those checkpoints aren't suffice to locate the problem (or they aren't created at all) - then you need to create them manually.

An very good way for stack would be an option proposed just in previous item. Well, we don't have it, so there are no good checkpoints for stack - you need to do everything by hands: by validating local variables manually from time to time - see the next item.

We have an option to force check manually for the heap. You can call CheckMemoryOverrun routine for EurekaLog's case and ScanMemoryPoolForCorruptions routine for FastMM's case. You force memory manager to scan the entire memory pool for corruptions by calling one of these routines (obviously, only consistence of internal info/headers can be validated, not the data inside memory blocks). By putting calls to these functions around the code - you put explicit checkpoints. Start with calling them periodically. Once you found a problem between two calls - move them closer to each other, until you locate the problem. Additionally, FastMM allows you to enable full scanning on each memory operation. To do that - you need to set global variable FullDebugModeScanMemoryPoolBeforeEveryOperation to True. However, you need to understand, that doing so will bring your application's performance to its knees. Never turn it on for large amount of time. First, try to locate the problem with ScanMemoryPoolForCorruptions's calls as much as you can. And only if you aren't able to move further - enable FullDebugModeScanMemoryPoolBeforeEveryOperation, but only for the code between two calls to ScanMemoryPoolForCorruptions. Well, probably, in many cases you would like to use SafeMM to track the problem.

4. Debug checks (stack and heap). It's not always possible to use or set checkpoints as discussed in previous item. For example, no one can check consistency of your information, all automated tools can check only their info, not yours. That's why you may need to validate your info manually. Well, it's simple: just place as many checks as you want around your code. Put Assert's call everywhere. Check every thing, that you're able to check. Once you found a problem between two Assert's call - move them closer to each other, just like in checkpoint's case. As soon as you reduce a gap enough to acquire the address of corrupted memory - you're done. Just run your application until the moment before problem and put a memory-break-point on this address (see also below).

5. Avoiding local variables (stack). Since we don't have much tools for the stack - you can move the problem elsewhere by avoiding local variables: try to use global variables (just for test, of course) or (better yet) put all local variables into record, which you allocate dynamically in the heap. This will move the problem to another area, where we have some handy tools (your favourite debugging memory manager).

6. Problem with threads (heap). Multi-threading usually does not affect stacks, but it can be a reason for many hard-to-detect problems with global or heap's data (well, not multi-threading by itself, but rather synchronization errors). Debugging of multi-threaded application is large and complex thing, so I won't go into details here - see other resources.

7. Memory breakpoints (stack and heap). If you'll found an specific address for memory, which was corrupted, things will become much easier. All you need to do now is to use memory breakpoints. Memory breakpoints is handy ability of Delphi's debugger, which allows you to put break-point on memory, just like you do this for code. A memory break-point triggers, when some code accesses memory. I won't go into details on how to use them - use Delphi's help. Ok, back to our problem. So, you have a memory's address. Run your application until the moment, when this memory will be available (allocated). It should be in the valid state at this moment. Place a memory break-point on it. And run your application. When break-point fires - check the code, which caused it. You'll find the culprit eventually.

So, if you wasn't able to solve your problem with the above methods - then the only thing left is:

Prevention of problems with memory (passive methods)

1. Avoid low-level code. It's simple: scan all your code, looking for calls of low-level routines (which aren't type-safe, therefore have a high chance of corrupting memory). Double-check all usage cases. Replace low-level code with high-level counterpart, if you can do it. It's better to do it slow and safe/correct than do it fast, but incorrect.

2. Use wrappers. I've mentioned this before, when we talked about WinAPI calls and "other resources" in add-on for mem-leaks article. Separate all such code into separate unit/class, which you can validate as single entity. You'll reduce searching area and simplify code by placing suspicious/potential troublesome code in the same place.

3. Code review by other developer. It's well-know fact, that your eyes see only things, which you brains want to see. That's why it's good thing to give your code to colleague - sometimes he/she can spot obvious problem, which you can't solve for few hours/days.

4. Actually, this section is endless. There are many books, which tells you how to write a good quality code. And they do this in more details, than I can do it here. That's why I won't list anything further - just give you some advice: read "smart" books. Consider the text above only as short example. You can improve yourself and your code by reading books and blogs. Many problems will be easier to spot or they can disappear eventually.

To put it short: read, read and read again (including us :) ).

Well, that's it. The end of the series. I hope you'll find it useful.

EurekaLog 6.0.23 is out

2010-03-02T09:00:00.006Z

We are pleased to announce the availability of the new 6.0.23 version.

EurekaLog is a tool for Delphi and C++Builder that gives your application the power to catch every exception and memory leak, generating a detailed log of the call stack (with unit, procedure and line number), optionally sending you a copy of each log entry via email or the Web via the most used Web Bug-Tracking tools (as Mantis, BugZilla and FogBugz).

Added support for BugZilla 3.4.4;
Fixed a memory corruption in TLogFile.LoadFromStream (BDS 2009 and above);
Fixed a fatal crash in Base64 (usually appears in SMTP sending);
Fixed of the log validity checks;
Fixed a garbage instead of very long idents in call stack;
Fixed a rare ecc32 crash on certain conditions;
Fixed a possible buffer overflow in PNG encoding routines;
Fixed a random crash at application's shutdown;

Catching memory leaks, redux

2010-02-16T16:26:00.056Z

I tried to discuss things, that I've missed last time.

Reading bug reports (or: how can I resolve mem-leak?)

Well, we already talked about this before. Thought I've already said all things, but I think that it's worth to mention them one more time - more focused on mem-leaks this time. That's because many people seems to miss the whole point of mem-leak reports.

Okay, so the first error of newbies is immediate (without any side-thoughts or doubts) opening the code, on which call stack points to - and just sitting here, starring at the line, trying to figure out, what go wrong with this code. The problem is: memory leak report does not point to the problem.

For example, report about leaking of String type will (probably) point to some trivial code like this:

S := IntToStr(I);

And what do you expect to find here? :)

And it is not some kind of tool's limitation - it's just that these tools can not read your mind (well, not yet, you know? :D ). Let's think for a second: what is a leak? Leak is... well, it is when we allocate something and do not release/free it. So, mem-leak report can (and, actually, will) contain that "something" - a resource; and it contains "allocation" - i.e. call stack to line of code, which allocates resource. But where is our problem? An actual problem is sitting at "release/free" moment! A tool can not know: where did you (your code) planned to release resource? That's why report contain only information about allocation. There is no direct information on the problem in the report.

What does it mean, "the problem is in release"? It means, that either we lost pointer to resource or we do have a pointer, but our release routine wasn't called for some reason. And those are points, which you should look at.

So, what should you do with mem-leak report? Well, you first need to follow call stack and find code. But the next thing is different (comparing to exception bug report): you don't need to analyze this line. You need:

Note, what resource was allocated here (object, string, array, memory block, etc...).
Find, where this resource should be released "by plan" (call do destructor, out of scope, explicit free call, etc...).
Found reason, why resource wasn't release at founded location.

As I already said, there can be 2 reasons for item 3: we lost reference or we missed the call.

For some reason, newbies like to post their mem-leak reports on forums, saying: "Help me! I can't understand, what's wrong here!". Hope, now you're seeing, how strange this looks: nobody, except the author of the code, can know that! That's because only author can tell, where did he planned to release resources. This looks especially stunning if there was no code posted, except allocation line (which, as we already figured out, have absolute nothing to do with the leak problem) - I think they suppose, that everybody have some kind of telepathic powers :)

That's why it will be more correct to post at least your code along with the report. Better yet - study situation by yourself before posting (we're discussing it now). Or just ask a different question. For example: ask, what can be possible reasons for leak in this code? There may be few guesses answered - and you can check, if they are valid for your case, thus, solving the leak.

Well, it is (more or less) clear with real pointers: you forgot to call release func, you re-assign pointer without releasing old value, etc, etc. But the leaks with auto-finalized types (strings, interfaces) often stun people: how is it possible at all? Isn't automatic management supposed to solve these kind of problems?

Well, certainly, yes - but this doesn't mean, that you can't mess up here ;)

Here goes few examples (note: all examples below are special crafted; nevertheless, they illustrate real-life problems, which you can encounter in your practice; other note: different Delphi versions can generate different machine code, so you may need to adjust examples a little; examples below use Delphi 14.0.3513.24210).

Example #1: memory corruption.
You have absolutely no problems with resource management in this example. But you have another problem somewhere else. You overwrite memory (a dreaded buffer overflow error!) - possible, by using low-level routines - thus, erasing pointer to resource, which leads to forgotten resource and, therefore, a leak:

// Warning: BAD code ahead

{$O-} // disable optimization to force compiler to accept stupid code 

procedure TForm1.FormCreate(Sender: TObject);

  procedure Test;
  const
    ArrLen = 5;
  var
    A: array[1..ArrLen - 1] of Integer;
    S: String;
  begin
    // Some actions here
    S := IntToStr(5); // <- mem-leak here!
    // Some actions here
    FillChar(A, ArrLen * SizeOf(Integer), 0);
    // Some actions here
  end; 

begin
  Test;
end; 

initialization
  ReportMemoryLeaksOnShutdown := True;
end.

Code in this example will produce a memory leak. Call stack will point to marked line, which (obviously) does not have any problem. Even more: all work with S does not have any problem either! The real reason lies within work with A array: a FillChar routine clears one element more, than array contains (quite a real problem, BTW - here it happened due to unusual array's range). Because, there is a S variable right after array - FillChar will clear it. Ooops! We just lost our pointer to string!

Example #2: stack corruption.
Well, strictly saying, it is memory corruption too. But first example can occur with any memory - not necessary the stack. This example is specific for stack only:

// Warning: BAD code ahead

{$O-} // disable optimization 

procedure SomeProc(I1: Pointer; I2: Integer); stdcall;
begin
  // Some actions here
end; 

procedure TForm1.FormCreate(Sender: TObject);

  procedure Test;
  var
    P: procedure(I: Pointer); stdcall;
    D: Pointer;
    S: Pointer;
  begin
    P := @SomeProc;
    D := nil;
    // Some actions here
    GetMem(S, 512); // <- mem-leak here!
    P(S);
    SomeProc(nil, 0);
    FreeMem(S);
    // Some actions here
  end; 

begin
  Test;
end; 

initialization
  ReportMemoryLeaksOnShutdown := True;
end.

The problem in this example is mismatch of prototypes (declarations) of P and SomeProc. So, calling P (actually - SomeProc) will pop one more item from stack, resulting in "shifted" stack data. So, any further work (call of SomeProc right after P's call) will damage (in our example - erase) S pointer. How can application survive a stack corruption, you say? Well, that happens. So, we lose the pointer in this example too.

How to fight with such problems - in the next time.

Leak of other resources' types

You can catch memory leaks and it is memory, that was allocated through Delphi's memory manager. I mean, that you can do this by using methods and tools, explained in previous article. You can't monitor any other types of resources by using this approach (replacing/using Delphi's memory manager). Sometimes, people forgot that their tool is not "all-mighty-seeing-eye", and when their apps start growing, but tool doesn't report anything - they just don't know what to think.

For example, GDI resources - they are allocated and deleted by using corresponding WinAPI calls, which (obviously) have nothing to do with Delphi's memory manager - therefore, they aren't counted in your logic of memory leaks. It's worth to note, that often all work with low-level resource is wrapped into class. For example, Delphi applications usually uses TBitmap class to manage HBITMAP objects. TBitmap is a Delphi's wrapper for system resource HBITMAP. You have one-to-one correspondence in that case (between HBITMAP's handle and its class wrapper - TBitmap). This means, that leaking one kind of resource will automatically mean leak of other resource's type and visa verse. In that case - you can catch resource leak by catching leak of corresponding memory. Well, it is not direct catch, but still a very good choice. Sometimes, there are exceptions though. In the example with TBitmap: it have ReleaseHandle method, which allows you to "release" handle HBITMAP, dis-assigning it from wrapper object - thus, breaking the one-to-one correspondence.

Second example: not resources, but memory. But this time - memory, allocated through different allocator, than Delphi's memory manager. This usually includes memory allocations for WinAPI or 3rd party DLL calls. For example, if you change String type to WideString in the example #1 above - there will be no leak reported. Why? Well, the leak is still here, but it is hidden now: we have leak of different kind. WideString is wrapper for BSTR system type, which have a certain requirement: all memory requests must be completed by specific system memory manager. This means that Delphi's memory manager will be not called - and so we have no chances to find a leak.

Note, that you can analyze these operations, but this is a more complex task, than catching memory leaks in Delphi's memory manager. That's because there is no one central management here: just a bunch of different functions (besides, there is no official way to add your code to these routines - only by using hacking hooks). I have already mentioned AQ tools as example of tools, which can analyze these situations. But you can use them at testing/developer stage on developer machine only (like FastMM), you can't deploy them to use at client machines (like EurekaLog).

In fact, these examples shows importance of class wrappers. Don't use low-level functions directly all over your code - just write a class wrapper. You can test it and then use in your code. You can significantly simplify code (and, therefore, searching for a leak) by gathering all related code in one place: inside class wrapper.

How can I find mem-leak without leaving application?

Any mem-leak report can only be created at application's shutdown - you can't create it "on demand". This is not a limitation of some method or tool - just common sense.

What is a leak? Leak is when you allocate resource (memory) and don't release it. How can you find it? You need to find all un-released resources, of course. And how can you know, if this particular resource was/will be un-released? All resources, which exists at application's exit, are (definitely) un-released. If they were released - then they didn't be here in the first place.

But how can you do the same logic while program still running? You still need to know: will/was this resource be released or not. How can you know that? You can't. There is no way you may know how many references are there on this resource, and (even if they are) will there be a release call for any reference or not.

Simple example:

function GetWorkFilePath: String;
begin
  // a lot of action
end;

You want to know: does GetWorkFilePath have a leak? So, you write something like this:

// Warning: BAD code ahead

function GetWorkFilePath: String;
begin
  MemState := RememberAllAllocatedMemory;
  try
    // a lot of action
  finally
    CreateMemLeakReport(MemState);
  end;
end;

Here: RememberAllAllocatedMemory routine makes a snapshot of all memory allocations, and CreateMemLeakReport routine compares current snapshot with saved (passed as argument) one. CreateMemLeakReport creates a mem-leak report for each mis-match in these two snapshots.

Your idea is: I have a solid block of code; all memory should be released, right? So "memory before" and "memory after" should be the same - so any difference will indicate a leak.

Unfortunately, this is not that simple.

In the example above: GetWorkFilePath routine allocates and return a string. This string will be a leak (judging by your logic) - because it is still here, after function's return. But this happens not because string is a leak, but because function pass string's ownership to caller, making him responsible for string's deletion.

Even more: strings (and not only strings) have a reference counter. Which means, that you can't apply any logic, which is based on memory manager's calls match. That's because most operations with these kind of resources will not call memory manager at all, cause all work will just inc/dec reference counter! The example above could allocate and could notallocate memory for the result. The last case holds when routine just return already allocated string (some global variable, for example) or even const.

Searching for hidden leaks

As a matter of fact, when somebody asks question, like in previous point - he didn't want an answer for it. What he's really asking is: how can he find hidden mem-leaks.

What is a hidden mem-leak? Take this example: your code creates many objects during your application's run. You want trade memory for speed, so you decided to re-use created objects, putting them to some kind of cache: TObjectList with ownership for inserted objects. When you need the same object again - you don't recreate it, but just picks from this cache.

And suppose that your code have a bug: one specific kind of object is never deleted from the cache. I.e. your application is growing and growing as the time passes - those are usual "visual signs" of memory leak. In normal conditions this grow is restricted by maximum size of a cache. But since some of your objects are never deleted from the cache - cache grow is unlimited. However: this is not a strict mem-leak. That's because you didn't lose reference to these "lost" objects: they are still here, in the cache. And when application shutdown - cache will be deleted and, thus, all objects in cache (including those forgotten objects) will be deleted too. Strictly saying: you have no mem-leak.

Here is what I mean by "hidden mem-leak". And people ask different questions, trying to find these leaks.

In fact, you don't need some "mem-leak of demand" here. There is one trick for this: run your application and let it work for few hours. Let him eat as much memory as it wants to. Now: pick any random address in your process. What will you see here? A hidden-leak.

Why is that so? Suppose that real working data for your app costs 50 Mb of memory. And suppose that your process eats 60 Mb of memory. Among these 60 Mb: 50 Mb are real working data and 10 Mb - "leaked" objects in your cache. Now, let your process eats 2 Gb of memory. Among these 2 Gb: 50 Мб — are still your real data (okay, may be 70 Mb, but this doesn't matter). All other space (2 Gb - 50 Mb) is for your forgotten objects. So, if you pick any random address - you have a (2048 — 50) / 2048 = 98% chance, that you hit you forgotten object. All you need to do is get its call stack (you run your app in debug mode, right?) and proceed as with usual mem-leak. The more memory process eats - the more garbage be there, and, therefore, the higher chances will be. Even 50% chance is not bad (garbage amount is the same as actual data: 50 + 50 = 100 Mb) - you only need to check few locations, not one.

For these tasks you can use FastMM and its LogAllocatedBlocksToFile routine. May be you will also find useful other routines (like FastGetHeapStatus, GetMemoryManagerState, GetMemoryManagerUsageSummary and GetMemoryMap) — just walk through interface section of FastMM4.pas unit. And if you didn't find suitable routine - there always is an option to write a simple stub memory manager for your case.

Why mem-leaks are bad and do I always need to release all memory?

Generally speaking, often mem-leak does not mean any visible problem to a user: application still works. Mem-leaks? So what? Program still do all tasks, that I need from it. This is especially true for client applications: cause they work for limited amount of time. So mem-leak is not scary - since all memory will be reclaimed at application's exit (refer to Jeffrey Richter's book on native code for more info) and so all leaks will be removed too. No, I don't mean that you don't need to fight mem-leaks here: mem-leak is always bad. It is just that mem-leaks aren't that fatal. Of course, this is not applicable for server applications. Server applications work for long period of time, so even minor leak will be deadly.

Some other question, which is close related to above, is: "if all memory is reclaimed upon app's shutdown - can I skip cleanup for global variables? They still will be deleted by automatic cleanup from OS!"

Well, the formal answer is: "you can do it". This is correct and you really can do it. But "can" does not mean "should". Obviously, there will be no technical problems with that approach. So, why is this bad?

Because you can't find a real mem-leaks, if you do like this. If you don't pedantically clean all of your resources - you will get a bunch of mem-leak reports. Well, leaks "by design": technically it's a leak (since resource wasn't released), but it is not a logical leak. Since you know, that those aren't reports about real mem-leaks - you will ignore them. And the problem is that if there will be a report about real leak - you may just miss it.

That's why it is a common "good rule" to always clean your resources. Unfortunately, there can be cases, when you can't do that. Those are very rare cases, but it can happen. But general rule is: always clean your resources, if you can do it. Don't rely on system's cleanup to throw out garbage for you. This will greatly simplify your life in the future.

Delphi's bugs

Well, I guess we can't ignore them :)

Before starting doing anything - make sure, that the problem really exists: run your application in wild run without debugger. This will eliminate any possible false mem-leaks like this.

Aside from IDE's bugs, there can be bugs in RTL/VCL too: example. It can be direct bugs (and there is change for their fix in next Delphi version - example), or things that just can't be fixed. Anyway, both cases introduce a mem-leaks in your application and your code has nothing to do with it.

So what can you do here? Putting patching apart - the only thing you can do is to ignore them (since you can't fix them). You should use capabilities of your tool: see if it have some routine like RegisterExpectedMemoryLeak - you can call this routine at application's start and pass a pointer. When tool will build a mem-leak report it will ignore any leak, that match one of such "registered" pointers. Yes, this is a workaround. You don't fix a problem - you just hide it, so you can concentrate on problems, which you can fix. The main danger here is overuse of such routines: do NOT add all mem-leaks as "registered" - don't forget that this will not fix the problem!

EurekaLog 6.0.22 is out

2009-12-15T14:07:00.003Z

We are pleased to announce the availability of the new 6.0.22 version.

EurekaLog is a tool for Delphi and C++Builder that gives your application the power to catch every exception and memory leak, generating a detailed log of the call stack (with unit, procedure and line number), optionally sending you a copy of each log entry via email or the Web via the most used Web Bug-Tracking tools (as Mantis, BugZilla and FogBugz).

Fixed: call stack tracing improvements;
Fixed: call stack of main thread is now captured;
Added: support for Mantis 1.2;
Added: new command-line compiler "--el_verbose" option to get detailed output;
Added: new command-line compiler "--el_path" option (to can specify a different EurekaLog's path);
Added: support for multiply projects compilation (Delphi 2010);
Added: "CustomSmtpHeloCommand" routine to customize the SMTP HELO command;
Fixed: Improved the Viewer ability to check the line breaks at the end of the file;
Fixed: Some Help improvements;
Fixed: Improved/fixed the Zip routines support (improving the MSVCRT-library checks);
Fixed: Viewer bug that imported some invalid logs;
Fixed: Viewer bug that was unable to decode some names in assembler view;
Fixed: Viewer bug unable to recover after "Wrong file format" error;
Fixed: Base64 implementation that corrupted memory;
Fixed: bug that could change working directory when sending email;
Fixed: bug that showed Cyrillic texts as ???? (from Delphi 2009);
Fixed: bug that could create malformed bug reports;
Fixed: bug that caused invalid parsing of map files;
Fixed: wrong support for C++Builder 2010;

EurekaLog's blog

What my application is doing? (introducing Threads Snapshot Tool)

Attaching debugger

Threads Snapshot Tool

Analyzing call stacks

Analyzing wait chains

Final notes

What does this address mean? (introducing Address Lookup Tool)

Exception tracer solution?

Address Lookup Tool

Working with addresses and offsets

Manual search

What does this error code mean? (introducing Error Lookup Tool)

Error Lookup Tool

EurekaLog RC 3 (public Win64 test release)

EurekaLog for Win64 quick walktrough

Win64 support status

EurekaLog Tools Pack

Conclusion

EurekaLog and FireMonkey / Win64 / OSX

EurekaLog and FireMonkey

EurekaLog and Win64

EurekaLog and OSX

EurekaLog and... Lazarus/MSVS C++

EurekaLog 7 RC 2

EurekaLog 6.1.04 is out

EurekaLog Users Survey

Announcing public EurekaLog 7 beta

How to try EurekaLog 7

Differences with EurekaLog 6

EurekaLog 7 and exising EurekaLog 6-projects

Where to get information

Participate in beta - win EurekaLog license!

Using EurekaLog 6 and EurekaLog 7 on the same machine and/or in the same IDE version

Summary

Detailed explanation

Detecting installed EurekaLog version

Manage EurekaLog configuration for non-administrator user accounts

Uninstalling

Troubleshooting

EurekaLog 7 Tools Pack

EurekaLog and XE2 update 3

EurekaLog 6.1.03 is out - now with XE2 and FireMonkey support

EurekaLog 6.5.12 for .NET is out

EurekaLog 6.1.02 is out

Introducing EurekaLog 6.1

ITDevCon sessions from EurekaLab s.a.s.

EurekaLog 6.0.25 is out

EurekaLog 6.0.24 is out

SDN sessions from EurekaLab s.a.s.

How to debug application's hang?

Preparations

Delphi

EurekaLog

Process Explorer

Threads Snapshot

A practical example

New Exception class in Delphi 2009 and above

Nested exceptions

Exception tracing

Memory problems in Delphi apps - final article

Contents

When EurekaLog can not help you (or: what we will talk about)

How to diagnose and fix memory problem

Problem's locating (active methods)

Prevention of problems with memory (passive methods)

EurekaLog 6.0.23 is out

Catching memory leaks, redux

Reading bug reports (or: how can I resolve mem-leak?)

Leak of other resources' types

How can I find mem-leak without leaving application?

Searching for hidden leaks

Why mem-leaks are bad and do I always need to release all memory?

Delphi's bugs

EurekaLog 6.0.22 is out