A new package repository has been deployed to production at https://nuget.smbsrc.net and contains all packages from https://www.symbolsource.org/Public/Metadata/NuGet (around 50 thousand of them!). In other words this is the new complementary service to https://www.nuget.org, running on a codebase rewritten from scratch – something we needed to do to solve all of the performance issues you have been facing with SymbolSource in the past.

What about the SymbolSource repository for MyGet, or the other company instances, you might ask? Those will be migrated to the new service too, and I will report as soon as we will have done that.

For now let’s see how the new service operates. You can try all of these commands yourself, and update all of your scripts and debuggers accordingly.

Publishing a symbol package

To push a symbol package run this command:

> nuget push SymbolSource.DemoLibrary-1.0.0.0.nupkg a0f4e24d-851f-4327-a9ec-181d335e7e95 -source https://nuget.smbsrc.net
Pushing SymbolSource.DemoLibrary 1.0.0.0 to 'https://nuget.smbsrc.net'...
Your package was pushed.

The command will succeed immediately without any timeouts, it just sends data to Azure storage. Packages are queued and processed asynchronously. That means that you will need to wait a bit before the symbols and sources become available, but we’ll be monitoring the queue length to make sure it never takes more than a few minutes.

Waiting and refreshing isn’t much fun, so we included a simple notification service, which you can use directly from the command prompt. Just append your Twitter username to the API key after a slash, like  this:

> nuget push SymbolSource.DemoLibrary-1.0.0.0.nupkg a0f4e24d-851f-4327-a9ec-181d335e7e95/@TripleEmcoder -source https://nuget.smbsrc.net
Pushing SymbolSource.DemoLibrary 1.0.0.0 to 'https://nuget.smbsrc.net'...
Your package was pushed.

You’ll get tweeted by our friendly SymbolSource Bot (@smbsrc) when the package starts processing and when it finishes. The tweet will also tell you if anything went wrong, in which case @SymbolSource will also be mentioned to alert us about the issue.

One thing you’ll notice immediately is that we’ve added a suffix to the version number. Each upload to SymbolSource is identified separately. We then do a form of reference counting on all of the symbols and sources, so there is no cost to that. But in case you upload different builds with the same version number, all of them will be available to clients. Unless of course you decide to delete a package. Which is now a fully supported self-service operation!

Deleting a symbol package

If you know the SymbolSource version string of your package, deleting is as simple as running this command:

> nuget delete SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 a0f4e24d-851f-4327-a9ec-181d335e7e95/@TripleEmcoder -source https://nuget.smbsrc.net
Deleting SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 from the 'https://nuget.smbsrc.net'.
SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 was deleted successfully.

Once again NuGet will lie a bit, as this is an asynchronous operation. But you will get the same friendly tweets if you include your username with the API key, as I did above.

But what if you didn’t push with Twitter notifications enabled, and are now at a loss as to what the version string is? No worries, you can query SymbolSource as any other package source and find out what the version is.

Before we talk about that, however, if you really want to remove your content from SymbolSource, run this too:

> nuget delete SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 a0f4e24d-851f-4327-a9ec-181d335e7e95 -source https://nuget.smbsrc.net/,all,original
Deleting SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 from the 'https://nuget.smbsrc.net'.
SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 was deleted successfully.

SymbolSource saves a copy of all uploaded packages, in case there is a change in the indexing algorithm and we need to reprocess. They are hosted in subfeed called original – see below for an explanation of what that is.

Listing symbol packages

By now you probably have already guessed the command to list packages from SymbolSource:

> nuget list SymbolSource.DemoLibrary -source https://nuget.smbsrc.net
No packages found.

Well, that result might seem unexpected. But remember that we added an additional version string and therefore promoted all packages to semantic versioning. NuGet treats those as prerelease packages. A quick fix:

> nuget list SymbolSource.DemoLibrary -prerelease -source https://nuget.smbsrc.net
SymbolSource.DemoLibrary 1.0.0.0-at-0E8GM1

There it is! Just remember about that NuGet will only show the most recent package, but you can show all of the uploads:

> nuget list SymbolSource.DemoLibrary -prerelease -allversions -source https://nuget.smbsrc.net
SymbolSource.DemoLibrary 1.0.0.0-at-0E2NAN
SymbolSource.DemoLibrary 1.0.0.0-at-0E8G1E
SymbolSource.DemoLibrary 1.0.0.0-at-0E8GM1

All of this works in NuGet Package Explorer too! As an exercise try downloading one of those packages. What you will get is a status package, useful in determining what symbols and source actually have been indexed by SymbolSource. More on this next time.

And in case you’re wondering… That version string is just a timestamp, encoded into an alphabet-based number system to save length. Unfortunately NuGet imposes a 20-character limit on all semantic versions.

Advanced listing by status and ownership

In my last post I also mentioned subfeeds, which let you query by status and ownership. To recap here are all of the statuses – they’ve slightly changed since our original spec:

• new – not yet opened, random package name and unknown version,
• original – a copy of the uploaded package, without any processing, useful if we improve the indexer and need to resubmit packages without any users’ actions,
• indexingqueued – indexing will soon start asynchronously,
• indexing – processing is in progress as you are looking at the list,
• succeeded – the package has been indexed without any issues,
• deletingqueued – deleting with soon start asynchronously,
• deleting – deleting is in progress as you are looking at the list,
• deleted – the package has been completely removed from the index,
• partial – something went wrong, either during indexing or deleting, and some symbols and sources might be available, but some may not,
• damagednew – package could not be read with a standard NuGet library,
• damagedindexing, damageddeleting – these will probably indicate an error in SymbolSource.

For example, if you wanted to confirm that a package has been deleted, you would need to run this command:

> nuget list SymbolSource.DemoLibrary -prerelease -allversions -source https://nuget.smbsrc.net/,all,deleted
SymbolSource.DemoLibrary 1.0.0.0-at-0D9I8Z
SymbolSource.DemoLibrary 1.0.0.0-at-0D9I90
SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51

As you can see, the list contains a package we did delete a few paragraph earlier – version 1.0.0.0-at-0E8H51. There are however also some old versions, which I deleted before writing this post. I can now delete them again, sort of, to simply remove the status package. Note that at this point symbols and sources have already been deleted, this is just similar to cleaning log files:

> nuget delete SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 a0f4e24d-851f-4327-a9ec-181d335e7e95 -source https://nuget.smbsrc.net/
Deleting SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 from the 'https://nuget.smbsrc.net/'.
Failed to process request. 'Not Found'.
The remote server returned an error: (404) Not Found..

A 404 error is a bit odd, isn’t it? Well not exactly. Remember, we have already deleted this package. It doesn’t show in the feed. To remove the status package we need to address the subfeed specifically.

> nuget delete SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 a0f4e24d-851f-4327-a9ec-181d335e7e95 -source https://nuget.smbsrc.net/,all,deleted
Deleting SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 from the 'https://nuget.smbsrc.net/,all,deleted'.
SymbolSource.DemoLibrary 1.0.0.0-at-0E8H51 was deleted successfully.

If you don’t specify a subfeed at all, SymbolSource defaults to succeeded state and all packages (as opposed to own packages). The first comma separates the feed name, which in case of nuget.org is empty. Things will be different for myget.org, but more on that in a future post.

So far we have listed all packages, regardless of who uploaded them. Of course pushing and deleting is permitted according to nuget.org package ownership. But you can also limit the packages listed by a query, using the following command:

nuget list -prerelease -allversions -source https://nuget.smbsrc.net/,own,succeded
Please provide credentials for: https://nuget.smbsrc.net/,own,succeded
UserName: @TripleEmcoder
Password: ************************************
SymbolSource.DemoLibrary 1.0.0.0-at-0E2NAN
SymbolSource.DemoLibrary 1.0.0.0-at-0E8G1E
SymbolSource.DemoLibrary 1.0.0.0-at-0E8GM1

SymbolSource will ask for credentials this time, as it needs to known who the user is, to list only the right packages. You can pass anything as the username, but if you give us something meaningfull like a Twitter handle, we will be able to get back to you in case of problems. The password is your NuGet API key.

Debugging in Visual Studio

And last, but not least, the new URL for Visual Studio is…

https://nuget.smbsrc.net

Just replace the old symbolsource.org URL in your Visual Studio options and you’re done. Happy (and fast) debugging

Wrapping it up

In this post I described the core features of our new SymbolSource engine. I hope you will find the new service much faster and much more stable than the previous one. We will be gradually upgrading other parts of the service over the next weeks and months. If you have any questions or feedback please leave them in the comment section.

It’s been a long time since I have written here, and it’s been an equally long time since we have made any visible improvements to SymbolSource. Part of the reason is that Kamil and I have both been involved in building an EHR (electronic health record) webapp for the Polish outpatient care market. Well, involved doesn’t really cut it, that startup has been more or less our entire life for three years. But now I can say, not without a lot of pride, that we have successfully led the project out of the startup phase. If you’re interested in what we have managed to build, visit www.mediporta.pl. (The website only has a Polish version at the moment.)

We also might not have seemed very responsive when contacted with issues, but we have never stopped reading and noting all the feedback that we received. It has been very hard to reply, when the current SymbolSource architecture has not given us many options of implementing improvements. When we started the project, there was no Azure, no NuGet (yes!) and normalized SQL databases ruled the Earth. That resulted in quite a few mediocre decisions, looking back today.

Those times have passed, however, and it’s time to loose the excuses and make things better.

Before we begin

If you are not familiar with what purpose SymbolSource serves, or how it works, first have a look at the official Wiki:

• http://docs.nuget.org/create/creating-and-publishing-a-symbol-package
• http://www.symbolsource.org/Public/Wiki/Index

Back to drawing board

A few months ago we have started talking about a total rewrite of SymbolSource, and even with the limited time we could have spared, we have made good progress in designing a new, scalable architecture for the service. It will enable us to support many different scenarios with the best possible performance:

• a public symbol repository for nuget.org,
• public and private feeds integrated with myget.org,
• hosted instances deployed in our Azure subscription,
• instances deployed in private Azure subscriptions,
• on-premise instances integrated with Active Directory.

But today I don’t want to talk about architecture, as it is more or less irrelevant when a service performs well. Today I’d like to share with you how we see the most basic form of interaction with the new SymbolSource – pushing and managing packages.

Consider this post a spec which, although already implemented, isn’t yet available publicly for testing. We are looking forward to any questions and comments!

Pushing a package

Pushing to the public repository:

nuget.exe push NHibernate.4.0.3.4000.symbols.nupkg 8ac00d48-a8e8-48e4-bb40-4fc92f18e15c -source http://nuget.smbsrc.net

Pushing to a named feed integrated with MyGet:

nuget.exe push EntityFramework.6.1.4-alpha1-40301.symbols.nupkg 60b1845b-116f-4eb0-8086-f96acaae46d7 -source http://myget.smbsrc.net/aspnetwebstacknightly

As a part of our effort to improve SymbolSource performance, we have decided to make all package operations asynchronous, which means that a successful push will only acknowledge that the package was received correctly. Read on to see how to determine the true package status.

Listing packages in various states

If all is well, you should see the package listed in the feed:

nuget.exe list -source http://nuget.smbsrc.net -allversions -prerelease

The list of packages will be similar to the following:

• NHibernate 4.0.3.4000-smbsrc150302193927

Note that SymbolSource has automatically added a SemVer compatible version suffix. It uniquely identifies each package upload with a timestamp. There is no guarantee that two packages with identical versions don’t have different symbol files inside. That’s why we process them independently.

You can probably see why the extra options to nuget.exe are needed:

• -prerelease shows packages with the smbsrc suffix,
• -allversions disables skipping earlier uploads.

What if a previous upload was accidental? Read on for instructions on how to delete a package.

Deleting a package

Removing a package – and all of its symbols and sources! – will be as simple as issueing:

nuget.exe delete NHibernate 4.0.3.4000-smbsrc150302193927 8ac00d48-a8e8-48e4-bb40-4fc92f18e15c -source http://nuget.smbsrc.net

Again, this operation is asynchronous. A success message from nuget.exe will only tell you that the package has been found and correctly queued for deletion.

Determining other states

The new SymbolSource will also introduce a concept of subfeeds, which will let users list packages in various states. The list command showed earlier targets the default subfeed, which is own,succeded. You will be able to explicitly target:

• ownership – currently own or all (subject to permissions),
• state – one of success, partial, indexing and so on.

Here’s an example of listing all failed packages in the default feed as an administrator of the Caliper company instance:

nuget.exe list -source http://caliper.smbsrc.net/,all,partial -allversions -prerelease

The meaning of the various states is as follows:

• new – not yet opened, random package name and unknown version,
• original – a copy of the uploaded package, without any processing, useful if we improve the indexer and need to resubmit packages without any users’ actions,
• damaged – package could not be read with a standard NuGet library,
• indexingqueued – indexing will soon start asynchronously,
• indexing – processing is in progress as you are looking at the list,
• succeeded – the package has been indexed without any issues,
• deletingqueued – deleting with soon start asynchronously,
• deleting – deleting is in progress as you are looking at the list,
• deleted – the package has been completely removed from the index,
• partial – something went wrong, either during indexing or deleting, and some symbols and sources might be available, but some may not.

If listing is possible… is downloading too?

Yes! Just as for any other NuGet package:

nuget.exe install NHibernate -version 4.0.3.4000-smbsrc150302193927 -source http://nuget.smbsrc.net

But you might be surprised by the results. Remember that when no subfeed is specified, you target the succeeded state. Packages in that state have no content, but status files instead! I will blog more about this in a future post. At the moment these are JSON files that specify what symbols and sources where detected and whether they have been uploaded to permanent storage. If there were any problems, a package will be listed in the partial state, and the status files will provide error messages.

By the way, since the JSON status files have complete information about indexed symbols and sources, we can delete a package entirely based only on those, without hitting any database at all. A big win performance-wise!

Time for some feedback

What do you think about the scheme that we designed? Please share your thoughts.

There’s a couple of things that Orchard solves for you, that make it great as a general framework:

• database connectivity (with NHibernate), including out-of-the-box support for local, automatic SqlCe databases,
• data migrations, for upgrading table schemas with new releases of your web applications or modules,
• theme and module management, with the use of NuGet packages and feeds,
• dynamic compilation, which lets you do in place changes to your web site or application, even through FTP, and see changes instantenously.

Orchard is distributed as a ZIP file that you can deploy as an application in IIS or open as a web application in Visual Studio. Module and theme packages are only managed through Orchard’s web UI, which gets them from a special Orchard Gallery Feed URL: http://packages.orchardproject.net/FeedService.svc. If you open a module from that feed you’ll see that it contains all of its sources (as in *.cs files), for Orchard’s dynamic compilation to take care of. Open-source at its best. Or is it?

As much as we love having access to sources at Caliper, we love having them on-demand in our debugging sessions even more – instead of moving thousands of them around on disk for no reason. And we all have SymbolSource to handle that for free

So we set out to try and make Orchard and its modules more vanilla-nuggety and symbol/source server friendly.

Step 1: Packaging Orchard

First, we decided to try putting Orchard itself into packages. The end-result experience should speak for itself:

1. Open Visual Studio and create an ASP.NET Empty Web Application (yes, no MVC, no Web Forms, no nothing).
2. Add http://nuget.gw.symbolsource.org/Public/Orchard/FeedService.mvc as a new package source to Visual Studio (yes, SymbolSource can host and serve NuGet packages as a package repository)
3. Install the Orchard package from this feed in the empty project.
4. Run the project with F5. You now have the Orchard setup page ready for action.

Things to note, as this should be considered work-in-progress still:

• All standard modules and themes are downloaded as dependencies. We haven’t tried yet to determine what subset is needed for setup, and which of them could be installed an on as-needed basis later with NuGet in Visual Studio or Orchard through the web UI.
• External dependencies are taken from the official NuGet feed only if they were taken by the Orchard team as-is, without modification or recompilation. All the rest is in Orchard.External.
• SqlCe native binaries are not included yet. But you should be good to go on your development machine, as you probably have in installed globally in your system.
• We haven’t yet figured out how to do versioning, so all packages have strict dependencies (==, i.e. [x.y.z] in NuGet).
• You can easily create your own binary module packages that take other Orchard modules as NuGet dependencies and get compiled with the right references.

Now for the best part:

• No *.cs files anywhere! Less files is more speed, less waiting, less transfers, less comparing on updates, less temptation to modify modules in place.
• Debuggable – with SymbolSource! All symbols (PDB files) and sources are indexed and accessible with Visual Studio using the regular, public SymbolSource URL. This applies only to Orchard.Framework, Orchard.Core and Orchard.Web projects at the moment, but all Orchard modules will follow soon.
• No dynamic compilation! No more screwed up assembly names and WCF services unable to load. Yes, you can disable dynamic compilation in configuration, but then you’re still required to compile modules and cleanup sources yourself.
• All views are included in the main project as content files, so it’s easy to find, edit or copy them to your theme.

Step 2: Repackaging modules

Since a lot of the standard Orchard distribution is built using modules, we had to figure out how to transform them form source form into binary form. We have a small application that interfaces with NuGet and MSBuild to get the modules, guess their interdependencies from project references, then compile and repackage them. All of the standard modules have been uploaded in repackaged, binary form to the SymbolSource Orchard repository.

So far we haven’t touched any contributed modules from the Orchard Gallery, but our plan is to either create a SymbolSource gateway that would do that compilation on the fly (module authors would decide to push to it additionally), or monitor the gallery feed and repackage new modules into our Orchard repository automatically.

What do you think?

All the tooling that we created for this is available on GitHub: http://github.com/SymbolSource/Orchard. To run it, just clone, open a Visual Studio prompt and do msbuild Orchard.proj. The output will be in pkg (main Orchard packages) and pkgbin (module packages).

I also mentioned in that post that you can push any PDB files and sources to SymbolSource, including those of entire applications. Because NuGet is still the easiest way of pushing symbol packages, what we would need is a NuGet-based method of distributing programs. Fortunately such tool exists already and is called Chocolatey. If you haven’t heard of it before, here’s how it’s described by its authors:

Chocolatey NuGet is a Machine Package Manager, somewhat like apt-get, but built with windows in mind.

In this post I will show you how Chocolatey and SymbolSource can be used to publish open-source applications with full debugging support, allowing users with programming skills to analyze problems through code and provide much more useful bug reports. Or even patches and pull requests!

The end-user experience

First let’s try installing a sample application and see how a user can get into its source when something goes wrong.

1. Install Chocolatey if you haven’t done so yet:
   @powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString('http://bit.ly/psChocInstall'))" && SET PATH=%PATH%;%systemdrive%\chocolatey\bin
2. Install our sample application:
   cinst SymbolSource.DemoApplication
3. Chocolatey automatically makes executables included in a package’s lib folder visible in PATH, so can now start our example simply as:
   SymbolSource.DemoApplication
4. As printed out, you can now simulate a crash by passing an argument to the program:
   SymbolSource.DemoApplication CrashMePlease
5. This will give you a stack trace:
   Unhandled Exception: System.Exception: Exception of type 'System.Exception' was thrown.
at SymbolSource.DemoApplication.Program.Main(String[] args)
   And a Windows Error Reporting window:
   
6. Wait it out and click the Debug button. Then choose New instance of Microsoft Visual Studio.
   
7. You might also get this windows instead (does anybody know why?):
   
8. Make sure that the Visual Studio instance that just opened is configured for SymbolSource. Load symbols for SymbolSource.DemoApplication from the call stack and double click on the call to Program.Main to see where and why the program failed.
9. Oops, the symbols loaded fine, but the source didn’t! It turns out, that because Chocolatey installs packages in C:\Chocolatety, Visual Studio is having trouble downloading files from the source server. If you enable Print source server diagnostic messages to the Output window in Tools -> Options -> Debugging -> General and try again, you’ll notice this hint in the log: SRCSRV: Source server cannot retrieve the source code (…) Access is denied.
10. My solution was to start again from step 4 in an elevated prompt. Visual Studio will warn you that it needs to elevate itself too, to debug an elevated process:
    
11. You should see the source now, loaded by Visual Studio on-demand from SymbolSource.

When looking at the source, notice however that the debugger is stopped at the exit of the scope where the exception occured.

This is because we are actually doing post-mortem debugging here, which was triggered by the absence of any exception handler along the way. Have a look at my previous post for more on this matter: Writing an automatic debugger in 15 minutes – section Choosing the right place and time. It would help, if we set up a breakpoint at the beginning of the scope, and stepped into the exception manually. Unfortunately, for a simple application like this, there’s no way of attaching the debugger fast enough, but you can always take Chocolatey’s package install location and start the program from within the debugger. Except that Visual Studio doesn’t offer this possibility directly, so you’ll need to create a bogus project (e.g. ConsoleApplication or ClassLibrary) and go to Properties -> Debug -> Start action to set Start external program to the actual culprit. Visual Studio will then allow you to start debugging as usual, believing that at some point this external application will call into the current project. It won’t, but we just need Visual Studio to catch the exception that we’ve seen before.

Unfortunately, at this point Visual Studio will still break only at the end of the scope, if you have chosen to load symbols manually at Tools -> Options -> Debugging -> Symbols. Add SymbolSource.DemoApplication.exe to the module list or choose automatic loading for all modules. After rerunning the bogus project, Visual Studio will now stop at the exception throw site:

Building a debug-enabled Chocolatey package

Now let’s see how easy it is to create a Chocolatey package and how even easier it is to enable symbols and sources publishing for it.

Chocolatey provides a lot of helpers to wrap external setups or ZIP files in packages, but it also allows distributing applications as-is. Like I mentioned before, any executable that you put in the package’s lib folder will be included on your PATH for convenience.

So creating a Chocolatey package for a simple console program like SymbolSource.DemoApplication (you’ve seen all of its source already) is as easy as packaging a .nuspec file like the following (I prefer building packages with <files> instead of from projects):

With the application built and ready for packaging, all you need to do is register at chocolatey.org to obtain a NuGet API key and run commands similar to:

1. nuget.exe pack SymbolSource.DemoApplication.nuspec -symbols
2. nuget.exe setapikey <key> -source http://chocolatey.org
3. nuget.exe setapikey <key> -source http://nuget.gw.symbolsource.org/Public/Chocolatey
4. nuget.exe push SymbolSource.DemoApplication.nupkg -source http://chocolatey.org
5. nuget.exe.push SymbolSource.DemoApplication.symbols.nupkg -source http://nuget.gw.symbolsource.org/Public/Chocolatey

Let me know if you’re aware of any open-source applications that have chosen Chocolatey as their native distribution method. They’re the first to benefit from all this, the easiest to enable symbol and source publishing for.

And our regular motto to finish off this post:

Let’s create not only an open- but also debuggable-source world 

But what if the reason still remains a mystery after reading the code? Fortunately then you have the most powerful tool in your development toolbox still left – the debugger. Let’s recap what it usually takes to start debugging third-party code?

Libraries

1. Figure out the branch and tag. If you’re lucky, the owners maintain a great source repository. If you’re not, it’s plain sorcery.
2. Get the code. SVN, Git, Mercurial – piece of cake.
3. Figure out the build system. MSBuild, NAnt, Rake, missing Visual Studio extensions – starting to get a headache.
4. Rebuild the library. Usually straightforward after step 3.
5. Replace library in the actual application. Often hard, especially when the build script failed to produce a matching version number or there was no signing key.
6. Rebuild application with replaced library. To work around missing signatures or mismatched version numbers.
7. Now you have all the binaries, PDBs and sources to dive deep into happy debugging.

Plugins

A plugin is essentially a library, so the steps are similar, but replacing it is much, much easier. Usually it’s just a matter of dropping it into to the right folder.

Applications

We’ve already covered getting and building the source. Replacing applications with your debug-enabled copies might, however, prove especially problematic. What if the application is installed in a location requiring elevation to overwrite? What if it’s part of a production deployment, or consists of many moving parts that would need to be replaced simultaneously?

Fortunately, we can avoid all this with a symbol and source server like SymbolSource, which will provide all the needed PDB files and sources on-demand, when you start a debugging session – regardless if the debugee is a library, a plugin or an entire application. Just configure Visual Studio in a few simple steps.

Personally I’m really happy that so many people publish to SymbolSource nowadays. It’s really easy, and most of the time requires close to no additional work when releasing a library to NuGet. Have a look at these posts to see how easy it is:

• The easy way to publish NuGet packages with sources by David Ebbo,
• The new integrated NuGet workflow by yours truly,
• Why everyone should be using a symbol server by Xavier Decoster.

If you missed it, you can also apply this to private MyGet feeds and be sure your symbols and sources are secured:

• Publishing symbol packages for a MyGet feed by Maarten Balliauw,
• How SymbolSource integrates with NuGet and MyGet and ASP.NET Web Stack nightlies now also on SymbolSource by yours truly.

By pushing symbols and sources to SymbolSource you’ll help all your users avoid the pain of rebuilding code as detailed above.

We’re really proud at SymbolSource that this has caught on! So there was and will be no Part 1 of the series, as there is enough information and adoption on the web already

Debuggable plugins

Staying in the NuGet universe, there is a tool – NuGet Package Explorer – that has a really cool way of distributing plugins – well… umm… NuGet.  There is a special MyGet feed where anyone is welcome to push plugin packages to, and NPE uses it to provide a list of plugins to its users, download selected plugins to a local cache, and then load all MEF exports from included assemblies.

Creating an NPE plugin is as easy as following these steps:

1. How to: Write plugins for NuGet Package Explorer by Luan Nguyen,
2. nuget.exe pack MyPlugin.csproj
3. nuget.exe push MyPlugin.nupkg -source http://www.myget.org/F/npe

All you need to do let users debug an NPE plugin is to change those steps to:

2. nuget.exe pack MyPlugin.csproj -symbols
3. nuget.exe push MyPlugin.nupkg -source http://www.myget.org/F/npe
4. nuget.exe push MyPlugin.symbols.nupkg -source http://nuget.gw.symbolsource.org/MyGet/npe

You can try the result yourself with our SymbolSource plugin that has symbols published already:

1. Install NuGet Package Explorer from CodePlex if you don’t have it yet.
2. Install our SymbolSource plugin in NuGet Package Explorer.
   
3. Open any package. If there’s nothing named Any on the NuGet feed, you can try one of the ASP.NET Web Stack Nightlies from http://www.myget.org/F/aspnetwebstacknightly.
4. Run Tools -> Analyze package. Our plugin provides additional package validation rules, which are very useful when creating your own symbol packages. In this case it will just warn you that it cannot verify the package filename, because it was opened from a feed.
5. Now rename System.Web.Mvc.xml to System.Web.Mvc.pdb to trick our plugin into loading a malformed symbols file, which we did not anticipate in our code.
6. Run Tools -> Analyze package again to crash the app. Select Debug the program and New instance of Microsoft Visual Studio to launch a debugging session.
7. You should now see Visual Studio detailing a NullReferenceException without any source code to back it up.
8. If you haven’t done so yet, configure http://srv.symbolsource.org/pdb/MyGet as one of your symbol servers at Tools -> Options -> Debugging -> Symbols.
9. Open Debug -> Windows -> Modules and make sure SymbolSource.Integration.NuGet.PackageExplorer.dll has symbols loaded. Right-click on its row and select Load Symbols From -> Symbol Path to load them manually.
10. Find SymbolSource.Intergration.NuGet.PackageExplorer.dll on the call stack and double-click to see where our plugin failed.
11. But that’s not the real reason, there’s a lot more methods on the call stack. Explore it by right-clicking those calls and loading symbols. You’ll need to have http://srv.symbolsource.org/pdb/Public configured as a symbol server for this to work, as all the other assemblies are available as packages on the NuGet Gallery and have symbols published to the regular SymbolSource repository, not as part of the MyGet plugin feed.
12. You should be able to see that something went horribly wrong while reading the PDB file’s hash, included in its header. That’s because we have an XML file serving as a corrupted PDB file.

That’s it. In Part 3 I will talk about debuggable applications and how you can publish them with NuGet, Chocolatey and SymbolSource, so stay tuned.

If you know about any other programs that use NuGet to distribute plugins, don’t hesitate to leave a comment. Also, remember that NuGet is just a means of transport for SymbolSource, and you can upload symbols and sources for just about anything, including tools published with ZIP files, MSI packages or regular setup files.

Let’s create not only an open- but also debuggable-source world

Let’s start by analyzing how PostSharp aspects are architected. Then we’ll move to implementing the actual IL transformation required to temporarily store the exception about be thrown, pass it to the aspect code, and then actually throw it expected.

A look at the FastTrace sample

Although the PostSharp SDK is unsupported and undocumented, Gael Fraiteur (the creator of PostSharp) was kind enough to point me to a sample on the SharpCrafters website that showed me how to solve the most important part of the problem: registering with the PostSharp engine. The sample is FastTrace (Download -> All downloads -> Samples -> PostSharp 2.1 – Sample Code -> FastTrace), which basically achieves the same thing as the standard and supported OnMethodBoundaryAspect, but shows how it can be done manually. The actual PostSharp implementation of this aspect, as far as I can tell, is far more complicated and robust, but for a little experiment like mine this is more than enough.

Registering PostSharp plugins

The FastTrace sample consists of 3 projects:

1. FastTrace – defines the aspect attribute, and is shared by both the client project (FastTrace.Test) and the plugin project (FastTrace.Weaver);
2. FastTrace.Test – contains test code that has the aspect attribute attached, we expect that aspect code will run along regular code once the project is built and processed by PostSharp;
3. FastTrace.Weaver – contains transformation code (the so-called weaver) that runs inside PostSharp, when it is invoked at the end of the build process – the engine detects the aspect attribute and calls into the weaver code passing the elements the attribute was applied to, so that tracing code can be inserted.

The solution has an additional manual dependency (apart from project references) added between FastTrace.Test and FastTrace.Weaver. This ensures that when FastTrace.Test is built, the FastTrace.Weaver binary is available. Without it, PostSharp would report an error about a missing plugin (thanks to the RequirePostSharp attribute on the aspect class). Where do plugins come from? The PostSharp installation directory has Plugins subfolder, but a project property can also override the plugin search path. The FastTrace.Test.csproj file contains the following code:

You could also specify this property on the msbuild command-line. Speaking of msbuild, the absolutely irreplacable switch is /p:PostSharpAttachDebugger=True, which will let you debug the aspect transformation code during compile-time. See Debugging Compile-Time Logic for more information. Also, what can save you from losing a lot of hair, is to always stop the PostSharp tray server and do a full rebuild of all projects (attribute, weaver, and target) each time you change the weaver code.

To be recognized as a PostSharp plugin, an assembly must have a corresponding .psplugin file. For FastTrace, it looks as follows:

The file is copied to the output bin\$(Configuration) directory when FastTrace.Weaver is built. It registers the FastTracePlugIn class as task to be run by PostSharp. In that class you can see how the transformation code is associated with a given aspect attribute:

The FastTraceAspectWeaver class contains mainly plumbing code, but also one thing that will be of interest to us. See line 6 (line 17 in the original file):

This property tells PostSharp whether an instance of the aspect attribute class should be made available to the weaver. In case of FastTrace, this is not necessary, as the attribute it is only used to mark pieces of code for transformation. But later on, when we proceed to implement the minidump aspect, it will be needed to be able to call custom code implemented in the aspect attribute.

Transforming assemblies with aspect weavers

Let’s move to FastTraceTransformation for the really cool part of AOP. This class implements the actual IL rewriting, although it cheats a bit by using the pre-defined MethodBodyWrappingImplementation class. Generally speaking, the starting point here is the MethodBodyTransformation base class, that specifies that rewriting will be performed on the method level.

First, references to other code elements are obtained, in this case to Console.WriteLine, so that the injected code can write out traces to the console:

For performance reasons this is cached – remember that an aspect can easily be attached to all elements of a large assembly with multicasting.

Processing starts with a MethodBodyTransformationContext object in Instance.Implement. It provides a reference to a method that was annotated with the FastTrace attribute, and is called for each one of them. Then, the actual rewriting is done in EmitMessage:

Implementing the OnThrowAspect

Now for the even cooler part, where we implement our own custom transformation. As discussed before, for the richest minidumps, they need to be created right before an exception is thrown. It is is possible to capture that moment with a debugger (see Writing an automatic debugger in 15 minutes). This applies to all exceptions, including those originating from system code. Because running under a debugger has its implications, let’s try actually injecting the minidump code before all throw instructions. Of course with approach we we will only be able to hijack exceptions thrown in our own code, where an aspect attribute can be applied.

All further code will refer to Padre.PostSharp (more on that later), which is a direct result of trial-and-error modifications to FastTrace.

Finding throw instructions

Since the transformation will process method bodies, and not wrap around them, we can safely delete all code run by Instance.Implement. Instead we need to look into the IL instruction sequences and find all throws. Luckily, PostSharp  allows that with the visitor pattern:

Before I show the final piece of code and explain what it does, let’s go through all the various helpers that we need. First, we want to be able to implement custom aspects that will be invoked before throwing exceptions, so let’s prepare a base attribute class that supports that:

The minidump generation code will just implement this aspect. You can of course do anything else, like logging the exception, in the following way:

To be able to call into the aspect code, we need to set RequiresRuntimeInstance to true, and gather the required assets:

In the AspectWeaverInstance we can call our MethodBodyVisitor to scan for throw instructions:

Now, finally, the actual IL transformation for all throw instructions found:

Here’s a quick walkthrough of what the code does:

• lines 7-8: create a temporary variable (unnamed) to store the exception, after making sure that the method initializes stack space for local variables – if we are introducing the first one, it may not be the case;
• lines 10-12: split the instruction sequence of the current block, so that we can insert code right before the throw instruction – first sequence will reference everything before the throw, and the second – after (after may be null if throw is the last instruction of the block – this will be true most of the time, as any code after throw is unreachable anyway, and will probably be optimized way, if ever actually written), then create a new instruction sequence between before and throw (attaching directly to before would overwrite existing code);
• lines 14-15: create an InstructionWriter, which is a convenient factory of opcodes and their arguments, and attach it to the new instruction sequence;
• line 16: store the exception object, which is currently on the stack ready to be thrown, into the temporary variable;
• line 17: emit code that will read the deserialized aspect attribute instance (luckily PostSharp does this for us) – we will now have the this value ready for calling OnThrowAspectAttribute.OnThrow;
• line 18: push the exception object back to the stack – this will serve as the first argument to OnThrow;
• line 19: execute OnThrow as a virtual function;
• line 20: push the exception object again to the stack – for the original throw instruction that will immediately follow.

It took about 3 hours to get to that 20 line piece of code, but that’s probably still less than it took to write this blog post, and absolutely worth every minute. PostSharp SDK provides a really convenient way of rewriting IL code, and very robust engine that seamlessly integrates with msbuild and VisualStudio. It’s a real shame that SharpCrafters don’t have the resources to market and support it. I imagine a lot of really cool aspects could be developed using it – after all IL manipulation isn’t that hard. And most of the time peverify.exe will tell you exactly what is wrong with the code you produced.

Trying out the OnThrowAspect

The final code is available on GitHub as part of the PADRE repository (as it serves the same purpose as a the PADRE project partly will – to generate information-rich minidumps automatically).

Would anyone like to help and create a NuGet package for it? I suppose it would require writing a custom install script to insert PostSharpSearchPath modifications into .csproj files.

What we’re trying to achieve

Let’s recap the most important point about minidumps: they can’t be created in the usual catch-all exception handlers of ASP.NET (in Global.asax or an HttpModule), WinForms (Application.ThreadException) or the general AppDomain.UnhandledException, because when they execute, the stack frames of calls leading up to the exception (along with variable values) are already all lost. The only information available is a stack trace in the Exception object, but’s that’s a normal runtime feature (used by Elmah, for example) that doesn’t provide any value for minidumps. Therefore, we need a way to insert minidump creation code as close to exception throw instructions as possible.

There are at least a few ways to achieve this:

• attach a debugger and respond to exception thrown events – covered in previous posts, this method has the additional benefit of applying to exceptions originating in external code, including the .NET Framework itself;
• insert minidump calls before every throw instruction of your code – I think I don’t need to tell you how big of a maintenance nightmare that would be;
• insert minidump calls into every catch handler of your code – there will probably be less of those than throw instructions, bit it’s still a very weak solution, and it will cost you some context information (stack frames lost between the actual throw and executing catch;
• use PostSharp or a different aspect framework to inject code automatically – in this post I’ll show you where and how calls can be injected with PostSharp, to create the information-rich minidumps possible.

PostSharp and exceptions

Intercepting exceptions is the first part of the PostSharp tutorial: PostSharp Principles: Day 1 – OnExceptionAspect. It introduces the OnExceptionAspect, which we could extend the following way to generate minidumps:

There is a convenient way of attaching this aspect to all methods in an assembly, described in PostSharp Principles: Day 3 – Applying Aspects with Multicasting Part 2:

Here’s what an OnExceptionAspect does:

The OnException method is called when an unhandled exception occurs in the target method. PostSharp will wrap the target method in a try/catch where the code we provided will be placed in the catch.

Indeed, PostSharp wraps each and every method in an exception handler:

Decompiling with ILSpy shows what the Main method is rewritten to:

Personally, I don’t like the fact that every method is now wrapped in a try-catch, but since wisemen say that it doesn’t have a negative performance impact (one proof of that is here), we could go along with it. But there a different problem. Take a look at this method (also after rewriting by PostSharp):

This code exhibits the same problem that I described earlier, when talking about generic catch-all exception handlers, although on a smaller scale – when the exception aspect is run, variables from local blocks are lost – in this case j (which is serves as an extremely hard to calculate value, which we simply must have minidumped). Here is a screenshot form a post-mortem debugging session opened from a minidump generated by the throw instruction on line 10:

Intercepting the throw instruction

Ideally, we could insert minidump code right before every throw instruction. The simplest way to do this is to always use custom exception types and invoke ClrDump.Dump in their constructors. Below is a similar post-mortem debugging session, this time with a minidump created in an exception constructor:

A variation of this would be to apply an OnMethodBoundaryAspect aspect to constructors of all exceptions defined in your assemblies. Using system exceptions, however, would require MSIL rewriting – to insert aspect code right before the throw instructions. As far as I know, there is no out-of-the-box support for this in PostSharp. Also, as far as I can tell, PostSharp authors are going to great lengths to discourage writing custom aspects like the one we need for minidumps. I get why the PostSharp SDK is unsupported and undocumented. But it would be great if there was at least a null aspect sample to show how to configure PostSharp to load a plugin. With a fair knowledge of MSIL and a class browser, the rest is doable.

I would love to hear from someone with bigger experience in using PostSharp!

Timing method

For timing of all the samples, I used this method inside the debugee:

private static void Time(Action action, int count)
{
    var sum = 0L;

    for (var i = 0; i < count; i++)
    {
        var s = new Stopwatch();
        s.Start();
        action();
        s.Stop();
        sum += s.ElapsedMilliseconds;
        Thread.Sleep(1000);
    }
    Console.WriteLine(sum/count);
}

To minimize the impact of random system events, I run all of the samples 10 times, with a 1 second pause between each invocation, and then took an arithmetic average of the execution times.

CPU-bound code

private static void CPU(int count)
{
    Array.Sort(Enumerable.Range(1, count).Reverse().ToArray());
}

Conclusion: Running under a debugger has no impact on CPU-bound code. All measurements are within the margin of error.

IO-bound code

Base on the above findings, and a personal hunch, I decided to skip this one.

Exception-bound code

This test should prove most interesting, as it’s the only that has code really interfacing with the debugger.

The sample code throws exceptions in a loop:

private static void Exception(int count)
{
    for (int i = 0; i < count; i++)
    {
        try
        {
            throw new Exception();
        }
        catch
        {
        }
    }
}

And the debugger inspects them and does nothing:

process.PostDebugEvent +=
    (sender, e) =>
        {
            if (e.CallbackType == ManagedCallbackType.OnBreakpoint)
                process.Go();

            if (e.CallbackType == ManagedCallbackType.OnException2)
            {
                var e2 = (CorException2EventArgs)e.CallbackArgs;
            }

            if (e.CallbackType == ManagedCallbackType.OnProcessExit)
                stop.Set();
        };

Conclusion: Going back and forth between the debugger and its debuggee, causes a performance drop of 2 orders of magnitude. I guess Visual Studio is more because of the Output window, which logs all exceptions.

The question is: how many exceptions does code generate on average? Instead of guesstimating, I decided to guessmeasure. All of the code indexed by SymbolSource, including only the most recent version of each project, consists of more than 1.5 million lines of code (that’s where I stopped counting). Of the lines used for this measurement, there were 12 thousand lines that contained the word “throw”. Therefore, I have guessmeasured, that all code is 0.8% exception on average.

Final conclusion: running under MDbgEngine causes a 24.8% drop in performance.

Would you be willing to take that, for the promise of gaining access to detailed call stacks and being able to generate minidumps for really hard to catch bugs?

Personally, I’ll shout a loud yes for all development and integration, and for occasional production too.

Using MDbgEngine, you can do much, much more – and all with a very high-level API.Have a look at how you can print a call stack (with argument values!):

process.PostDebugEvent +=
    (sender, e) =>
    {
        log.Log(e.CallbackType.ToString());

        if (e.CallbackType == ManagedCallbackType.OnException2)
        {
            var ce = (CorException2EventArgs)e.CallbackArgs;

            if (ce.EventType == CorDebugExceptionCallbackType.DEBUG_EXCEPTION_FIRST_CHANCE)
            {
                var thread = process.Threads.Lookup(ce.Thread);

                foreach (var frame in thread.Frames)
                {
                    if (!frame.IsManaged || frame.Function.FullName.StartsWith("System."))
                        break;

                    log.Log("{0}({1})", frame.Function.FullName, string.Join(", ", frame.Function.GetArguments(frame).Select(
                        arg => string.Format("{0} = {1}", arg.Name, arg.GetStringValue(false)))));
                }
            }
        }
    };

This code, when attached to an example process, printed logged the following information:

OnException2
AspNet.Default.B(i = 3, t = "test 2")
AspNet.Default.A(i = 1, t = "test")

PADRE – Pluggable Automatic Debugging and Reporting Engine

In the previous blog post, and on Twitter, I’ve already mentioned the idea of turning this into a project. The above sample code is now available on GitHub, along with some boilerplate code in MEF, to make it truly pluggable and a sample ASP.NET application throwing exceptions.

Just open the solution, run it and go to the ASP.NET application to click Button 2. You should get a stack trace in the console window like above.

I’d love to hear some feedback!

You may also want to check out:

• Accessing stack traces with MDbgEngine and PADRE
• Performance impact of running under MDbgEngine

Bugs in production

Remember all those reports coming back form clients saying “hey, your program crashed” or “hey, your is site showing these ugly yellow pages at random moments”? So do I. Unfortunately, there isn’t much that can be done to diagnose problems in running software, basically we’re at the mercy of our clients’ reports (which vary in quality, most of the time tending towards useless) or more or less verbose logging. Having those, we can reproduce issues during a debugging session to see what code causes them. If that fails,  we can go to the extremes of attaching a debugger in a production environment. Not feeling enough pressure today? Attach to a live site and try to setup breakpoints the exact way needed to only catch the error, and not stop a milion people from doing their daily work. At first try. Logging, on the other hand is very safe, it but will only tell you as much as you predicted that would be needed.

Now what if we could have something in the middle? More than logging, but less than an interactive debugging session?

Windows Error Reporting

In the ancient times, when not all software run as web applications, and especially in the even darker times when programs were being written mostly in native code, you could often see hard crashes that literally made entire applications disappear. Then Microsoft invented Windows Error Reporting, that filled that post-crash emptiness with a nice-looking window that supposedly gathered some data and sent it back to the author. This feature is present in all recent versions of Windows, and is available not only for Microsoft applications, but also for third party programs. Provided that their author gets a signing certificate (authenticode), signs all components of the software and registers on a deeply hidden Microsoft site. If all these conditions are met, Windows will automatically create a so called minidump of the process when it crashes (with important parts of memory, including stack traces of all threads) and  ask the user to allow sending it to Microsoft.

Since a minidump contains stack state of all threads at the moment of the crash, it allows you to open a post-mortem debugging session, and feel almost as if you were attached to the original process when the failure occured. WinDbg has supported this since… probably always, but Visual Studio 2010 does so too – although through a slightly hidden feature of just opening (File -> Open -> File…) minidumps, i.e. *.dmp files.

Just having access to precise call stacks used to be a killer feature at the time, but does it provide any value to the .NET developer, used to getting stack traces with every exception?

You guessed it: yes, it does.

With a minidump, you’ll be able to see parameter and variable values for all functions on the call stack.

Crashes in a .NET world

Before we go deeper, we need to clarify one thing. What exactly is a crash? What causes Windows Error Reporting to kick in? It’s unhandled exceptions. Native unhandled exceptions. Normally, the last exception handler registered in a running process is a system function that reports program failure, which in turn causes a default (defined in the registry) system just-in-time debugger to launch – which will be Windows Error Reporting a.k.a Dr. Watson, Visual Studio, or something else depending on your setup. Some programs also register their own unhandled exception handlers to report errors. Firefox does that, Winamp too. Perhaps you’ve already seen their crash windows in action. By registering their own handlers, these programs can create minidumps of their own liking (there are various types to choose from) and send them wherever they want.

What about .NET apps then? Do they crash at all? Of course they do, but unfortunately not in the sense described so far. You can have an unhandled managed exception in your code, but it will always be caught by the runtime – which in turn prints a stack trace (in a console application), shows a pop-up window (in a Windows Forms application) or displays the Yellow Screen of Death (in a web application). No minidump will ever be created automatically for a .NET application.

Creating minidumps manually

A minidump is created by calling MiniDumpWriteDump from dbghelp.dll. Mind though, that it needs to be a version from Debugging Tools for Windows, not the Windows system folder. There are various tools that call it, like SysInternals’ ProcDump. MiniDumpWriteDump and ProcDump have been described in an article in the recent edition of MSDN Magazine: Writing a Plug-in for Sysinternals ProcDump v4.0.

One of those tools is also ClrDump, which is available both as a very simple command-line tool and a convenient library. Read this blog post, and you’ll know everything about how to create minidumps from .NET applications: Creating and analyzing minidumps in .NET production applications.

Choosing the right place and time

The last article I linked to has one thing wrong though. It triggers minidump generation from AppDomain.UnhandledException and Application.ThreadException. And it would be equally wrong to call it from an error handler in Global.asax or to register one with an HttpModule, like Elmah does. The reason is that those handlers are called from a generic exception handler at the base of your application. At this point all of the context (call stack, variables) available where the exception was thrown is lost. To really benefit from minidumps, you need to trigger them as deep in your call hierarchy as you can. Also, often you’ll face exceptions that are caught by your own code, but you’ll still want minidumps generated, to better explain why those exceptions occured.

And let’s not forget generating minidumps like this is similar to debugging with printf – you need to modify code and redeploy. We need something smarter. And external to the inspected application.

Writing a managed debugger in C#

The CLR exposes debugging capabilities through COM, so it isn’t impossible for a mere mortal to implement a debugger. But thanks to the CLR Managed Debugger Sample anyone can do it, really. The sample mainly consists of a console tool, mdbg.exe, that you can use as a poor-man’s replacement for windbg.exe or cdb.exe, and – more importantly – a shared debuging engine library, written in C#. The latter is now also available on NuGet (courtesy of yours truly) as Microsoft.Samples.Debugging.MdbgEngine.

Let’s consider this trivial console test application:

namespace TestApplication
{
    class Program
    {
        static void Main(string[] args)
        {
            A(1, new { property = 2 });
        }

        private static void A(int i, object state)
        {
            B(i + 4, state);
        }

        private static void B(int i, object state)
        {
            try
            {
                C(i);
            }
            catch (Exception exception)
            {
                Console.WriteLine(exception.Message + ": " + i);
            }
        }

        private static void C(int i)
        {
            throw new InvalidOperationException("Expected failure");
        }
    }
}

Here is an only slighly longer console application that runs a debugger and creates minidumps on each exception (as you can see, also those caught in code):

using Microsoft.Samples.Debugging.CorDebug;
using Microsoft.Samples.Debugging.MdbgEngine;

namespace DebuggerApplication
{
    class Program
    {
        static void Main(string[] args)
        {
            var stop = new ManualResetEvent(false);
            var engine = new MDbgEngine();
            var process = engine.CreateProcess("TestApplication.exe", "", DebugModeFlag.Default, null);
            process.Go();

            process.PostDebugEvent +=
                (sender, e) =>
                    {
                        if (e.CallbackType == ManagedCallbackType.OnBreakpoint)
                            process.Go();

                        if (e.CallbackType == ManagedCallbackType.OnException2)
                        {
                            ClrDump.CreateDump(process.CorProcess.Id, @"C:\temp.dmp", (int)MINIDUMP_TYPE.MiniDumpWithFullMemory, 0, IntPtr.Zero);
                        }

                        if (e.CallbackType == ManagedCallbackType.OnProcessExit)
                            stop.Set();
                    };

            stop.WaitOne();
        }
    }
}

Go ahead and try it. If you open test.dmp in Visual Studio, you’ll be able to see not only the call stack at the moment when the exception occurred, not only values of variables that are easy to log (like i), but you’ll be also able to inspect complex and dynamic structures (like state).

I hope you’re having a big “holy shit” moment. I definitely did.

Where can we go from here?

The code above “logs” all exceptions. To implement this idea in the real world, we’d need quite a bit more – load symbols (you know where to get those from – SymbolSource), set breakpoints at specific locations, catch only specified exceptions.

Catch me on Twitter if you like the idea. This has potential for a great and very useful open source project.

We also have some very cool ideas how to integrate this with SymbolSource. Imagine setting breakpoints on the website and choosing exceptions to monitor, then seeing automatically collected reports from all your deployments. Call stacks with variable values, failure statistics, minidumps for download and offline analysis.

Interested?