0:173> k
RetAddr               Call Site
00000000`557c5820     <Unloaded_LibUtils_CloudNs_3.dll>+0x265fe
00000000`00000008     <Unloaded_LibUtils_CloudNs_3.dll>+0x2b5820
00000000`0000000e     0x8
00000000`00000008     0xe
00000000`557c8c18     0x8
ffffffff`fffffffe     <Unloaded_LibUtils_CloudNs_3.dll>+0x2b8c18
00000000`00000000     0xffffffff`fffffffe

There isn’t much on the stack at all.

0:173> dps @rsp
00000000`1248f920  00000000`557c5820 <Unloaded_LibUtils_CloudNs_3.dll>+0x2b5820
00000000`1248f928  00000000`00000008
00000000`1248f930  00000000`0000000e
00000000`1248f938  00000000`00000008
00000000`1248f940  00000000`557c8c18 <Unloaded_LibUtils_CloudNs_3.dll>+0x2b8c18
00000000`1248f948  ffffffff`fffffffe
00000000`1248f950  00000000`00000000
00000000`1248f958  00000000`00000000
00000000`1248f960  00000000`00000000
00000000`1248f968  00000000`00000000
00000000`1248f970  00000000`00000000
00000000`1248f978  00000000`00000000
00000000`1248f980  00000000`00000000
00000000`1248f988  00007ff9`a2117344 kernel32!BaseThreadInitThunk+0x14
00000000`1248f990  00000000`00000000
00000000`1248f998  00000000`00000000

This is just a worker thread the operates entirely inside LibDB.CloudNs.3.dll. It doesn’t have a very deep stack, so I suspect that it’s idle and is waiting for work to do.

For these types of investigations, there usually isn’t much to see directly in the crashing thread. That thread is the victim. You have to do additional research to figure out who unloaded the DLL prematurely.

Some snooping around found another stack that involves this unloaded DLL:

0:159> k
RetAddr               Call Site
00007ff9`9fdbbea0     ntdll!ZwWaitForMultipleObjects+0x14
00007ff9`9fdbbd9e     KERNELBASE!WaitForMultipleObjectsEx+0xf0
00000000`554d65fe     KERNELBASE!WaitForMultipleObjects+0xe
00000000`55765820     <Unloaded_LibDB_CloudNs_3.dll>+0x965fe
00000000`00000003     <Unloaded_LibUtils_JsonNs_3.dll>+0x255820
00000000`00000004     0x3
00000000`00000008     0x4
00000000`55768c18     0x8
ffffffff`fffffffe     <Unloaded_LibUtils_CloudNs_3.dll>+0x258c18
00000000`00000000     0xffffffff`fffffffe

The most recently unloaded DLLs are

00007ff9`6d7c0000 00007ff9`6d80a000   FabrikamContextMenu.dll
00007ff9`115e0000 00007ff9`1172f000   LitWareSync.dll
00007ff9`643d0000 00007ff9`64681000   CcNamespace.dll
00000000`55440000 00000000`5550b000   LibDB_CloudNs_3.dll
00000000`55860000 00000000`55998000   LibNet_CloudNs_3.dll
00000000`557f0000 00000000`5585b000   LibJson_CloudNs_3.dll
00000000`55510000 00000000`557e7000   LibUtils_CloudNs_3.dll
00000000`561a0000 00000000`56238000   MSVCP100.dll
00000000`56240000 00000000`56312000   MSVCR100.dll
00007ff9`85130000 00007ff9`85167000   EhStorShell.dll
00007ff9`3cac0000 00007ff9`3cb61000   wpdshext.dll
00007ff9`78a00000 00007ff9`78a26000   EhStorAPI.dll
00007ff9`686f0000 00007ff9`68754000   PlayToDevice.dll
00007ff9`67110000 00007ff9`6718d000   provsvc.dll

So the LibDB.CloudNs.3.dll that got unloaded is just part of an entire ecosystem of Lib*.CloudNs.3.dll dynamic libraries that all got unloaded together.

The ringleader of this operation appears to be CcNamespace.dll, which looks like the Contoso namespace extension that adds a “Contoso” node under My Computer This PC that gives you a view into all your Contoso things stored in the Contoso cloud service. All the other DLLs are helpers that the main CcNamespace.dll uses to accomplish its tasks.

The main CcNamespace.dll was loaded by Explorer as a shell extension, and its Dll­Can­Unload­Now function was returning S_OK when there were no active references to objects in CcNamespace.dll. Unfortunately, when it said “Sure, it’s safe to unload me”, that linchpin DLL unloaded all its minions, unaware that one of the minions (the utility library) had spun up some worker threads.

You might think that the fix is to update the utility library’s Dll­Can­Unload­Now to return S_FALSE if there are still busy background threads.¹ But that doesn’t work because the utility library is probably not a COM DLL in the first place. It’s just a traditional DLL that CcNamespace.dll uses, and it is CcNamespace.dll that is the COM DLL.

The Dll­Can­Unload­Now in CcNamespace.dll could warn LibUtils.CloudNs.3.dll that it should start winding down, but you’re basically in a tricky spot because the DLL_PROCESS_ATTACH cannot wait for the worker thread to exit.

I think the way to go is for the worker thread to increment the DLL reference count when it starts its worker thread, and to use Free­Library­And­Exit­Thread to exit the worker thread. Alternatively, it could make its worker thread a threadpool thread and use Free­Library­When­Callback­Returns to request that the system decrement the DLL reference count when it finishes.

This is probably something the utility library should have done anyway. I suspect that the worker thread is not something that clients of the utility library are even aware of. It is just an implementation detail of the utility library, created without the knowledge of the main DLL.

Fortunately, the application compatibility team has a copy of Contoso Cloud in their library, so even though we couldn’t reproduce the crash, we were still able to confirm that CcNamespace.dll is indeed the shell extension DLL whose unloading triggers the unloading of all the dependent DLLs.

We were about to contact Contoso with our conclusions and suggestions for improvement, but we discovered that it would be pointless because Contoso discontinued that namespace extension years ago. They replaced it with a different way of integrating their cloud content into Windows; the only people using the namespace extension are those who still using an old version, either because they don’t want to pay for the upgrade, or because they are actively avoiding the upgrade because they like the old way.

Those customers are using a product that has gone out of support. Contoso doesn’t care about those old customers any more. Windows will have to fix it without Contoso’s help.

The Explorer team added an application compatibility flag for the Contoso Cloud namespace extension to say “When you load this shell extension, do a Get­Module­Handle­Ex with the GET_MODULE_HANDLE_EX_FLAG_PIN flag so the DLL never unloads.” That way, even if the DLL says “Sure, go ahead and unload me, it’s totally safe, trust me,” and COM does a FreeLibrary, the DLL doesn’t actually unload.

¹ Even if you manage to get return Dll­Can­Unload­Now to return S_FALSE, it doesn’t help if COM is being uninitialized. In that case, CoUninitalize will ask a DLL if it is okay to unload now, but the answer is a foregone conclusion: If COM is shutting down, COM is going to unload all the DLLs that it loaded. It asks you if you are okay with it, not because it cares what your answer is, but to give you a chance to do cleanup outside of DllMain.

The post The case of the thread executing from an unloaded third-party DLL appeared first on The Old New Thing.

An attacker can bypass security policies by modifying the following registry keys to disable ⟦security feature 1⟧ and ⟦security feature 2⟧.

The statement is true, but what they don’t mention is that administrator privileges are required to modify those keys. This is like saying that a door lock is insecure because you can open the door from the inside. If you are inside, then you have already gotten past the door!

Indeed, the purpose of those keys is to define the security policy in the first place! So it boils down to “It’s a security vulnerability that an administrator can change a security policy.”

What the security researcher found was that if your system has been compromised, the first guy who gets into your inner sanctum can make your system even more vulnerable.¹ If you assume that the attacker has full control, then it’s not surprising that they control everything.

¹ Isn’t this the plot to half of the sci-fi movies ever made? The plucky hero sneaks behind enemy lines in order to disable the bad guys’ shields long enough to let the rest of the team in. This isn’t a security flaw in the shields. It’s a security flaw in whatever was supposed to protect the switch that turns off the shields.

The sci-fi movie analogy would be “If we can get to the switch that turns off the shields, then we can turn them off!”

Well, yeah. The hard part is getting into the room that has the switch.

It rather involved being on the other side of this airtight hatchway.

Bonus chatter: This is a repeat of It rather involved being on the other side of the airtight hatchway: Disabling a security feature as an administrator, but this type of bogus vulnerability report happens so much, I wrote it up again before I realized that it was a duplicate.

The post It rather involved being on the other side of this airtight hatchway: Changing administrative settings appeared first on The Old New Thing.

• The hardest working font in Manhattan: Marcin Wichary digs up the history of the font named Gorton. I grew up with this font, seeing it not only on punch card keyboards but also in a Leroy lettering kit that we used in sixth grade graphics class when we studied mechanical drawing.
• RollerCoaster Tycoon’s Overengineered Puking System. Learn more than you ever needed to know about the algorithms in the game RollerCoaster Tycoon that determine when a park guest pukes. No really.
• The Spaghetti Policy for All MLB Teams by the appropriately named Sickos Committee breaks down the outside food policies for all of teams in Major League Baseball to determine which ones would let you bring in a gallon of spaghetti. The first part is fairly straightforward, but it gets far more weird (and sicko) as they look at the stadiums where a gallon of spaghetti might not be permitted. (Background information: Sports fans in Philadelphia have a reputation for poor behavior. These are the people who booed Santa Claus and pelted him with snowballs.)
• We have reached the second generation now. “Microsoft invented “escrow builds” to launch functional apps – that’s internal ‘Microspeak’ jargon for quality control” is a rehash of my article on the history of the term “escrow”, but the article appears to just be a suspicious mishmash of fragments from the article. For example, in the sentence “He described it as unhelpful because the blog essentially described a metaphor using another metaphor” the antecedents of “it” and “the blog” are not present in the article, leading the reader to think that “it” is the explanation of what escrow means and that “the blog” is my blog. On top of that, the title of the article introduces the phrase “launch functional apps” (whatever that means), even though none of those three words appear anywhere in the original article, nor in the dodgy rehash. Undaunted, a different AI content farm picked up on that article and somehow expanded it into an even longer article that conveys even less information.

The post 2026 mid-year link clearance appeared first on The Old New Thing.

One thing I’ve learned is that anywhere it’s possible to hide a pointer, people will hide a pointer there. This is true even for small integers.

As I was digging up the history of the extra bytes, I saw a special note in the 16-bit code for Set­Class­Word: It says that there’s an app that expects to be able to modify the value of GWW_CB­CLS­EXTRA.

Now, modifying this value has no practical effect because the memory for the class was allocated when you called Register­Class. You can’t go back in time and change the allocation size.

But one program realized that it could use this value as a place to store some private data, so they did. Sure, that’s not the purpose of the GWW_CB­CLS­EXTRA, but that never stopped them.

For compatibility, Windows lets 16-bit programs modify GWW_CB­CLS­EXTRA. But at least it blocks it for 32-bit and 64-bit programs. One loophole closed. Countless more to go.

The post A compatibility note on the abuse of Windows window class extra bytes appeared first on The Old New Thing.

We’re going to look at the system-defined offsets.

In 16-bit Windows, these were the available extra bytes and the function you used to read them:

There is clearly a naming pattern here for class and window bytes.

The first letter G stands for Get. The second letter C or W stands for Class or Window. And the third letter W or L stands for Word or Long.¹

For window bytes that apply only to dialog windows, the first letter changes to D for “dialog”. These values are zero or positive, since they are really just extra bytes registered to the standard dialog class.

Now, in 16-bit Windows, handles were 16-bit values, but in 32-bit Windows, they expand to 32-bit values, so 32-bit Windows changed the functions from Get­Something­Word to Get­Something­Long, and the prefixes correspondingly changed from W to from L. So our table now looks like this:

Name	16-bit prefix/size	32-bit prefix/size
MENUNAME	GCW_ int16_t	GCL_ int32_t ◱
HBRBACKGROUND	GCW_ int16_t	GCL_ int32_t ◱
HCURSOR	GCW_ int16_t	GCL_ int32_t ◱
HICON	GCW_ int16_t	GCL_ int32_t ◱
HMODULE	GCW_ int16_t	GCL_ int32_t ◱
CBWNDEXTRA	GCW_ int16_t	GCL_ int32_t ◱
CBCLSEXTRA	GCW_ int16_t	GCL_ int32_t ◱
WNDPROC	GCL_ int32_t	GCL_ int32_t ◱
STYLE	GCW_ int16_t	GCL_ int32_t ◱
ATOM	GCW_ int16_t	GCW_ int16_t
HICONSM		GCL_ int32_t
WNDPROC	GWL_ int32_t	GWL_ int32_t ◱
HWNDPARENT	GWW_ int16_t	GWL_ int32_t ◱
ID	GWW_ int16_t	GWL_ int32_t ◱
STYLE	GWL_ int32_t	GWL_ int32_t
EXSTYLE	GWL_ int32_t	GWL_ int32_t
USERDATA		GWL_ int32_t
MSGRESULT	DWL_ int32_t	DWL_ int32_t
DLGPROC	DWL_ int32_t	DWL_ int32_t
USER	DWL_ int32_t	DWL_ int32_t

The ◱ symbol represents a value that got bigger, and the symbol represents values that did not exist in 16-bit Windows.

Even though control IDs are typically small integers, the space for them was expanded from a 16-bit value to a 32-bit value because some people were using it to hold pointers or handles. (One way to create a process-wide unique number is to allocate memory and use its address.)

The next step in the evolution of extra bytes is the conversion from 32-bit to 64-bit Windows. Pointers and handles expand to 64-bit values on 64-bit Windows, so all of the extra bytes that are used to (or could be used to) hold a handle or pointer were expanded to a 64-bit version.

To make it possible to write code that targets both 32-bit and 64-bit Windows, the design of 64-bit Windows didn’t make the hard break that 32-bit Windows did from 16-bit Windows. Instead, they introduced new functions that accept pointer-sized integers, which are 32-bit values on 32-bit Windows and 64-bit values on 64-bit Windows. That way, you just use those new functions everywhere, and they will expand on 64-bit systems and remain the same on 32-bit systems.

The new functions have names like Get­Window­Long­Ptr, and the corresponding prefixes were changed to GWLP_ and so on.

Name	16-bit prefix/size	32-bit prefix/size	32/64-bit prefix/size
MENUNAME	GCW_ int16_t	GCL_ int32_t ◱	GCLP_ intptr_t ◱
HBRBACKGROUND	GCW_ int16_t	GCL_ int32_t ◱	GCLP_ intptr_t ◱
HCURSOR	GCW_ int16_t	GCL_ int32_t ◱	GCLP_ intptr_t ◱
HICON	GCW_ int16_t	GCL_ int32_t ◱	GCLP_ intptr_t ◱
HMODULE	GCW_ int16_t	GCL_ int32_t ◱	GCLP_ intptr_t ◱
CBWNDEXTRA	GCW_ int16_t	GCL_ int32_t ◱	GCL_ int32_t
CBCLSEXTRA	GCW_ int16_t	GCL_ int32_t ◱	GCL_ int32_t
WNDPROC	GCL_ int32_t	GCL_ int32_t ◱	GCLP_ intptr_t ◱
STYLE	GCW_ int16_t	GCL_ int32_t ◱	GCL_ int32_t
ATOM	GCW_ int16_t	GCW_ int16_t	GCW_ int16_t
HICONSM		GCL_ int32_t	GCLP_ intptr_t ◱
WNDPROC	GWL_ int32_t	GWL_ int32_t ◱	GWLP_ intptr_t ◱
HWNDPARENT	GWW_ int16_t	GWL_ int32_t ◱	GWLP_ intptr_t ◱
ID	GWW_ int16_t	GWL_ int32_t ◱	GWLP_ intptr_t ◱
STYLE	GWL_ int32_t	GWL_ int32_t	GWL_ int32_t
EXSTYLE	GWL_ int32_t	GWL_ int32_t	GWL_ int32_t
USERDATA		GWL_ int32_t	GWLP_ intptr_t ◱
MSGRESULT	DWL_ int32_t	DWL_ int32_t	DWLP_ intptr_t ◱
DLGPROC	DWL_ int32_t	DWL_ int32_t	DWLP_ intptr_t ◱
USER	DWL_ int32_t	DWL_ int32_t	DWLP_ intptr_t ◱

From the prefix on the name of the extra bytes, you can read off which function it is meant to be used with.

Prefix	Function
GCW_ GetClassWord	GWW_ GetWindowWord
GCL_ GetClassLong	GWL_ GetWindowLong
GCLP_ GetClassLongPtr	GWLP_ GetWindowLongPtr

The weirdo is DWLP_ because it needs to encode both the type of window that it can be used with (D = dialog) as well as the function name it goes with (WindowLongPtr).

As a concession, Windows lets you pass GCL_ and GWL_ values to Get­Class­Long­Ptr and Get­Window­Long­Ptr (respectively) even though they are intended to be used with Get­Class­Long and Get­Window­Long (respectively). If you do that, you get the corresponding 32-bit value zero-extended if necessary to be the size of a pointer.² This is seen primarily in the case of GWL_ID because most people don’t use the full range of IDs, so if you’re willing to live within the 32-bit subset, you can just pretend that the values are not pointer-sized.³

“Why bother changing all the prefixes? Doesn’t that just create a lot of busy work for people porting from 32-bit code to 64-bit code?”

Yes, but it’s good busy work. The point is to force build breaks at places where you need to make fixes, because you have to call the function that accesses a pointer-sized integer rather than a 32-bit integer; otherwise you suffer from integer truncation bugs.

¹ This is a common prefixing convention for classic Win32. For example, the operation parameter to Show­Window is prefixed SW_; the flags to Set­Window­Pos are prefixed SWP_; and the relationship parameter for Get­Window is prefixed GW_.

² The use of the GWL_ values with Set­Window­Long­Ptr is a bit more problematic. It looks like you’re storing a pointer-sized integer, but only the bottom 32 bits are honored.

³ The ID is unusual in that it is defined both as GWL_ID and GWLP_ID. All of the other values are defined with only one prefix.

The post The evolution of window and class extra bytes in Windows appeared first on The Old New Thing.

A colleague of mine who was looking at other crashes coming from this process found that most of those other crashes were also of the form “a data structure was corrupted because somebody wrote the single byte 01 into it.” That piece of information made everything fall into place for my side of the investigation.

We saw earlier that the bottom bit of the HMODULE is set for datafile module handles. Therefore, if one of these stray 01 bytes happens to overwrite the bottom byte of an existing HMODULE handle, that turns it into a (fake) datafile module handle. And then, during process destruction, a component dutifully cleans up the DLLs they loaded by freeing them (say because they were stored in an RAII type like wil::unique_hmodule), the code will pass this (fake) datafile module handle to Free­Library. The Free­Library function sees the bottom bit set and says, “Oh, this must be the handle to a module that was loaded via LOAD_LIBRARY_AS_DATAFILE,” so it frees it as a datafile.

Freeing a datafile module means undoing the steps that were taken when the module was loaded as a datafile: Unmapping the DLL from memory. In particular, loading a module as a datafile does not add the DLL to the list of DLLs that were loaded as code; therefore, unloading a datafile module doesn’t remove it from that list. As far as the DLL list is concerned, the DLL is still in memory.

A one-bit error caused the code to lie and attempt to free a module handle that did not correspond to a Load­Library call, resulting in mass havoc.

The “DLL unmapped from memory” crash is just an alternate manifestation of the “somebody is writing 01 bytes to places they shouldn’t” bug. The original bug had a larger bucket spray than we initially thought.

The good news is that all of the crashes have funneled down to a single bug. The bad news is that you now have to debug this one memory corruption bug.

Unfortunately, at the time of this writing, the root memory corruption bug in the third party program has yet to be identified. We don’t know whether it’s coming from an operating system component or from the program itself. Though the fact that it appears to occur only in one process, where it sprays across multiple modules, suggests that it’s a problem with that program, or that there’s something peculiar about how this specific process uses the system.

If you look at the original stack trace, you can see that the problem is occurring at process termination. That’s probably why the problem has lurked for so long: Crashes at exit often go unnoticed because there is no end-user loss of functionality. The user was finished with the program anyway. Whether it exits cleanly or with a crash doesn’t affect the user much.

Sorry. Not all stories have a happy ending.

The post The case of the DLL that was not present in memory despite not being formally unloaded, part 2 appeared first on The Old New Thing.

The post Raymond’s hot take on Hainanese chicken appeared first on The Old New Thing.

 # Child-SP          RetAddr           Call Site
00 000000ba`92851098 00007ff9`fed521c1 ntdll!_chkstk+0x37
01 000000ba`928510b0 00007ff9`feea5ace ntdll!RtlDispatchException+0x2d1
02 000000ba`92851300 00007ff9`fed4e02d ntdll!KiUserExceptionDispatch+0x2e
03 000000ba`92852060 00007ff9`fed5222f ntdll!RtlLookupFunctionEntry+0x8d
04 000000ba`928520b0 00007ff9`feea5ace ntdll!RtlDispatchException+0x33f
05 000000ba`92852800 00007ff9`fed4e02d ntdll!KiUserExceptionDispatch+0x2e
06 000000ba`92853560 00007ff9`fed5222f ntdll!RtlLookupFunctionEntry+0x8d
07 000000ba`928535b0 00007ff9`feea5ace ntdll!RtlDispatchException+0x33f
08 000000ba`92853d00 00007ff9`fed4e02d ntdll!KiUserExceptionDispatch+0x2e
09 000000ba`92854a60 00007ff9`fed5222f ntdll!RtlLookupFunctionEntry+0x8d 
0a 000000ba`92854ab0 00007ff9`feea5ace ntdll!RtlDispatchException+0x33f  
0b 000000ba`92855200 00007ff9`fed51f29 ntdll!KiUserExceptionDispatch+0x2e
0c 000000ba`92855f70 00007ff9`feea5ace ntdll!RtlLookupFunctionEntry+0x8d
0d 000000ba`928561c0 00007ff9`fed4e02d ntdll!RtlDispatchException+0x33f
...

The highlighted block of stack frames (from Rtl­Lookup­Function­Entry to Ki­User­Exception­Dispatch) repeated for a very long time.

We are clearly in some sort of recursive exception handling death spiral. An exception occurred, and the kernel has decided that it is not something that kernel mode can handle,¹ so it reflected the exception back into user mode for further processing (Ki­User­Exception­Dispatch). While trying to figure out which exception handler to call, (Rtl­Lookup­Function­Entry), we took an exception, which restarted the exception loop.

Eventually, all of these recursive exceptions exhausted the stack, and we take a stack overflow exception that terminates the process.

The bug was assigned to shell32 because it looked like shell32 was the source of the original exception. If you walk all the way back to the bottom of the stack, you get something like this:

23f 000000ba`9294c620 00007ff9`fed5222f ntdll!RtlLookupFunctionEntry+0x8d
240 000000ba`9294c670 00007ff9`feea5ace ntdll!RtlDispatchException+0x33f
241 000000ba`9294cdc0 00007ff9`fed4e02d ntdll!KiUserExceptionDispatch+0x2e
242 000000ba`9294db20 00007ff9`fed5222f ntdll!RtlLookupFunctionEntry+0x8d
243 000000ba`9294db70 00007ff9`feea5ace ntdll!RtlDispatchException+0x33f
244 000000ba`9294e2c0 00007ff9`fcba0af0 ntdll!KiUserExceptionDispatch+0x2e
245 000000ba`9294f018 00007ff9`fde2ad13 combase!CoTaskMemFree
246 000000ba`9294f020 00007ff9`fc7abc75 shell32!wil::details::string_maker::~string_maker+0x13
247 000000ba`9294f050 00007ff9`fc7ab897 ucrtbase!<lambda_f03950bc5685219e0bcd2087efbe011e>::operator()+0xa5
248 000000ba`9294f0a0 00007ff9`fc7ab84d ucrtbase!__crt_seh_guarded_call<int>::operator()+0x3b
249 000000ba`9294f0d0 00007ff9`fc7d2f0c ucrtbase!execute_onexit_table+0x3d
24a 000000ba`9294f110 00007ff9`fdff4645 ucrtbase!__crt_state_management::wrapped_invoke+0x2c
24b 000000ba`9294f140 00007ff9`fdff476e shell32!dllmain_crt_process_detach+0x45
24c 000000ba`9294f180 00007ff9`fee9f6fe shell32!dllmain_dispatch+0xe6
24d 000000ba`9294f1e0 00007ff9`fed4bcae ntdll!LdrpCallInitRoutineInternal+0x22
24e 000000ba`9294f210 00007ff9`fedcd37f ntdll!LdrpCallInitRoutine+0x10e
24f 000000ba`9294f280 00007ff9`fedcc54e ntdll!LdrShutdownProcess+0x17f
250 000000ba`9294f390 00007ff9`fdcb18ab ntdll!RtlExitUserProcess+0x9e
251 000000ba`9294f3c0 00007ff9`e754882e kernel32!ExitProcessImplementation+0xb
252 000000ba`9294f3f0 00007ff9`e754f344 mscoreei!RuntimeDesc::ShutdownAllActiveRuntimes+0x2fa
253 000000ba`9294f6d0 00007ff9`e66f464b mscoreei!CLRRuntimeHostInternalImpl::ShutdownAllRuntimesThenExit+0x14
254 000000ba`9294f700 00007ff9`e66f44c9 clr!EEPolicy::ExitProcessViaShim+0x8b
255 000000ba`9294f760 00007ff9`e66f441e clr!SafeExitProcess+0x9d
256 000000ba`9294f9e0 00007ff9`e66f3f44 clr!HandleExitProcessHelper+0x3e
257 000000ba`9294fa10 00007ff9`e66f3e24 clr!_CorExeMainInternal+0xf8
258 000000ba`9294faa0 00007ff9`e753d6da clr!CorExeMain+0x14
259 000000ba`9294fae0 00007ff9`e75d785b mscoreei!CorExeMain+0xfa
25a 000000ba`9294fb40 00007ff9`fdc9e8d7 mscoree!CorExeMain_Exported+0xb
25b 000000ba`9294fb70 00007ff9`fedcc40c kernel32!BaseThreadInitThunk+0x17
25c 000000ba`9294fba0 00000000`00000000 ntdll!RtlUserThreadStart+0x2c

The repeating block stops at the source of the first exception: combase!Co­Task­Mem­Free.

We can look for the exception record to see what the original problem was.

The exception record and context record are probably passed to Rtl­Dispatch­Exception, so we can see what Ki­User­Exception­Dispatch passes.

  # Child-SP          RetAddr           Call Site
243 000000ba`9294db70 00007ff9`feea5ace ntdll!RtlDispatchException+0x33f
244 000000ba`9294e2c0 00007ff9`fcba0af0 ntdll!KiUserExceptionDispatch+0x2e

0:000> u ntdll!KiUserExceptionDispatch 00007ff9`feea5ace 
ntdll!KiUserExceptionDispatch:
00007ff9`feea5aa0 cld
00007ff9`feea5aa1 mov     rax,qword ptr [ntdll!Wow64PrepareForException (00007ff9`fef272f0)]
00007ff9`feea5aa8 test    rax,rax
00007ff9`feea5aab je      ntdll!KiUserExceptionDispatch+0x1c (00007ff9`feea5abc)
00007ff9`feea5aad mov     rcx,rsp
00007ff9`feea5ab0 add     rcx,4F0h
00007ff9`feea5ab7 mov     rdx,rsp
00007ff9`feea5aba call    rax
00007ff9`feea5abc mov     rcx,rsp 
00007ff9`feea5abf add     rcx,4F0h
00007ff9`feea5ac6 mov     rdx,rsp 
00007ff9`feea5ac9 call    ntdll!RtlDispatchException (00007ff9`fed51ef0)
00007ff9`feea5ace test    al,al

We see that the two parameters passed to Rtl­Dispatch­Exception are at rsp+4f0h and rsp. I’m guessing that the exception record comes first, followed by the context record, since that’s the order that those pointers appear in the EXCEPTION_POINTERS.

  # Child-SP          RetAddr           Call Site
244 000000ba`9294e2c0 00007ff9`fcba0af0 ntdll!KiUserExceptionDispatch+0x2e

00007ff9`feea5ace test    al,al
0:000> dps 000000ba`9294e2c0+4f0
000000ba`9294e7b0  00000000`c0000005 ← STATUS_ACCESS_VIOLATION
000000ba`9294e7b8  00000000`00000000
000000ba`9294e7c0  00007ff9`fcba0af0 combase!CoTaskMemFree
000000ba`9294e7c8  00000000`00000002
000000ba`9294e7d0  00000000`00000008

Yup, that looks like an exception record. It starts with the exception code, and is shortly after followed by the code address where the exception was taken.

0:000> .exr 000000ba`9294e2c0+4f0
ExceptionAddress: 00007ff9fcba0af0 (combase!CoTaskMemFree)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000008
   Parameter[1]: 00007ff9fcba0af0
Attempt to execute non-executable address 00007ff9fcba0af0

Okay, so we attempted to execute a non-executable address, and the address is combase!Co­Task­Mem­Free.

Just for fun, let’s confirm that the second parameter really is a context record:

0:000> .cxr 000000ba`9294e2c0
rax=00007ff9fe3a9850 rbx=000001bbebd12388 rcx=000001bbebd63140
rdx=00007ff9fe4e99e0 rsi=000001bbebd12828 rdi=000001bbebd12310
rip=00007ff9fcba0af0 rsp=000000ba9294f018 rbp=0000df1c60b20569
 r8=000001bbebd12310  r9=0000df1c60b20569 r10=d94b3944a87271f0
r11=000000000000000b r12=0000000000000001 r13=00007ff9fdff47c0
r14=000000ba9294f128 r15=000001bbebd12310
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202
combase!CoTaskMemFree:
00007ff9`fcba0af0 sub     rsp,28h

Yup, looks like a context record.

But wait, the exception claims that combase!Co­Task­Mem­Free isn’t executable. First, let’s see if the debugger agrees with this assessment.

0:000> !address  00007ff9`fcba0af0

Usage:                  Image
Base Address:           00007ff9`fcb20000
End Address:            00007ff9`fcea6000
Region Size:            00000000`00386000 (   3.523 MB)
State:                  00010000          MEM_FREE
Protect:                00000001          PAGE_NOACCESS
Type:                   <info not present at the target>
Image Path:             C:\Windows\System32\combase.dll
Module Name:            combase
Loaded Image Name:      combase.dll
Mapped Image Name:      C:\symbols\combase.dll
More info:              lmv m combase
More info:              !lmi combase
More info:              ln 0x7ff9fcba0af0
More info:              !dh 0x7ff9fcb20000

Content source: 2 (mapped), length: 1eb510

The memory that contains the Co­Task­Mem­Free function has been freed!

In fact, if you look at the base address and region size, you see that the entirety of combase.dll has been unloaded from memory.

On the other hand, if you ask the loader what it thinks about that address, it says “Oh, that’s code inside combase.dll.”

0:000> !dlls -c 00007ff9`fcba0af0

0x1bbeb111020: C:\WINDOWS\System32\combase.dll
      Base   0x7ff9fcb20000  EntryPoint  0x7ff9fcc9a9d0  Size        0x00386000    DdagNode     0x1bbeb114380
      Flags  0x0028a2cc  TlsIndex    0x00000000  LoadCount   0xffffffff    NodeRefCount 0x00000000
             <unknown>
             LDRP_LOAD_NOTIFICATIONS_SENT
             LDRP_IMAGE_DLL
             LDRP_PROCESS_ATTACH_CALLED

Okay, now that we’ve gathered evidence, let’s see what theory we can develop.

The combase.dll is still in the loader’s bookkeeping, and we see that its load count is 0xFFFFFFFF, which means that the DLL has been “pinned”, meaning that the loader will never unload it. These two pieces of information suggest that the DLL was not removed from memory by Free­Library, but rather by somebody explicitly freeing it, say by doing a Virtual­Free on the memory.

My guess is that a memory corruption bug somewhere caused some code to clean up the wrong memory blocks, and it unwittingly freed the memory occupied by combase.dll, say because somebody overwrote its “don’t forget to free this” variable with the address of combase.dll, or because there is an uninitialized variable bug, and the uninitialized value just happened to be a leftover copy of combase.dll‘s base address.

But either way, the problem is not with shell32. Shell32 is just another victim, being the first DLL to call into combase after it was forcibly removed from memory by some unknown component.

If this theory is true, then I should be able to find similar types of crashes where some other DLL is the victim of a DLL being forcibly removed from memory.

I asked for the 100 most recent crashes in that third party program and put them into a pivot table so I could see the distribution.

The shell32 bug is the second-from the bottom, responsible for 11% of the crashes. But there are 13 other stack overflow bugs. And there are also a bunch of access violations in “unknown”.

I spot checked those stack overflow and “unknown access violation” crashes, and I found that they were all the same form as the shell32 bug, but with different DLLs: While sending DLL_PROCESS_DETACH notifications, a DLL was found to have been forcible removed from memory, and whatever DLL was the next one to call into that force-unloaded DLL was blamed, even though it was the victim. (A bunch of these arrived as “unknown access violation” because the system saw the crash inside the exception dispatching code and was for some reason unable to walk the stack all the way to the start of the recursive crash loop.)

So a total of 46% of the crashes were due to this rogue force-unload of a DLL. This is a case of bucket spray, where a single underlying cause generates a large number of different types of crashes.

The good news for the shell32 team is that they are off the hook; they are the victim. The bad news is that we don’t know who the culprit is.

Next time, we’ll learn some more about these crashes, and that will help confirm some theories about this specific one and may even discredit other theories.

¹ Things that kernel mode can handle are things like guard page exceptions (by expanding the stack) or page faults in paged-out memory (by paging it back in).

The post The case of the DLL that was not present in memory despite not being formally unloaded, part 1 appeared first on The Old New Thing.

For the purpose of this discussion, I will collectively call these “asynchronous activities”.

One of the things you can do with asynchronous activities is cancel them, by calling the Cancel method. This method submits a request to cancel, but it does not wait for the operation to acknowledge the cancellation. If you want to wait for the operation to stop executing, you have to wait for it to call the completion callback.²

Asynchronous cancellation is important for avoiding deadlocks.

Most of the time, the scenarios involve cross-thread synchronous calls, but here’s an extremely obvious way it can happen.

Suppose that you have registered a progress callback on your asynchronous activity with progress.

// C#
async Task DoSomethingWithTimeoutAsync()
{
    var op = DoSomethingAsync();
    op.Progress = (sender, p) => {
        UpdateProgress(p);
        if (p >= 0.5) {
            sender.Cancel();
        }
    };
    try {
        await op;
    } catch (TaskCanceledException) {
        // ignore cancellation
    }
}

// C++/WinRT
winrt::fire_and_forget Widget::DoSomethingWithTimeoutAsync()
{
    auto op = DoSomethingAsync();
    op.Progress([&](auto&& sender, auto p) {
        this->UpdateProgress(p);
        if (p >= 0.5) {
            sender.Cancel();
        }
    });

    try {
        co_await op;
    } catch (winrt::hresult_canceled const&) {
        // ignore cancellation
    }
    co_return;
}

The code calls DoSomethingAsync() and attaches a progress callback which cancels the operation once the progress reaches 50%. If the Cancel() method waited for outstanding progress callbacks to completed, you have a deadlock: The Cancel() is waiting for the progress callback to complete. But the progress callback is itself calling Cancel().¹

To avoid deadlocks when cancellation occurs while a progress callback is in progress, the cancellation method doesn’t wait for an acknowledgment. If you want to know when the activity is finished, wait for it to complete. If you want to ignore progress reports that arrive after you cancel, you can do that yourself.

// C#

async Task DoSomethingWithTimeoutAsync()
{
    var op = DoSomethingAsync();
    bool canceled = false;
    op.Progress = (sender, p) => {
        if (!canceled) {
            UpdateProgress(p);
            if (p >= 0.5) {
                canceled = true;
                sender.Cancel();
            }
        }
    };
    try {
        await op;
    } catch (TaskCanceledException) {
        // ignore cancellation
    }
}

// C++/WinRT

winrt::fire_and_forget Widget::DoSomethingWithTimeoutAsync()
{
    auto op = DoSomethingAsync();
    bool canceled = false;
    op.Progress([&](auto&& sender, auto p) {
        if (!canceled) {
            this->UpdateProgress(p);
            if (p >= 0.5) {
                canceled = true;
                sender.Cancel();
            }
        }
    });

    try {
        co_await op;
    } catch (winrt::hresult_canceled const&) {
        // ignore cancellation
    }
    co_return;
}

(The canceled variable doesn’t need to be atomic because progress callbacks do not overlap.)

Notice in the C++/winRT version that even after we call Cancel(), we wait for the co_await op to report completion before we return. Otherwise, the Progress callback will access an already-destroyed canceled variable.

¹ This is also the cancellation model for I/O and RPC: The cancellation method submits a cancellation request and returns immediately, and the underlying operation indicates that it has stopped executing by reporting some sort of completion.

² You might try to solve this by saying “Cancellation is asynchronous if the Cancel is issued from the same thread as the progress event”, but that doesn’t help in this case, which is more realistic:

// C#
async void CancelAfter(IAsyncInfo op, TimeSpan delay)
{
    co_await Task.Delay(delay);
    op.Cancel();
}

async Task DoSomethingWithTimeoutAsync()
{
    var op = DoSomethingAsync();
    op.Progress = (sender, p) => {
        Invoke(() => UpdateProgress(p));
    };
    CancelAfter(op, TimeSpan.FromSeconds(5));
    try {
        await op;
    } catch (TaskCanceledException) {
        // ignore cancellation
    }
}

Suppose the Progress event is raised on a background thread at 4.9999 seconds. Before the lambda can call Invoke(), the Cancel­After­Delay timeout elapses, and the UI thread calls Cancel(). Now you have a deadlock because the Progress event is waiting for the lambda, the lambda is waiting for the Invoke, the Invoke is waiting for the UI thread, the UI thread is waiting for the Cancel, and the Cancel is waiting for the Progress event.

The post Cancellation of Windows Runtime activities is asynchronous appeared first on The Old New Thing.

Some people wondered, “Isn’t that just a release candidate? Why do you Microsoft people have to make up new names for things that already have perfectly good names?”

Yes, the Microspeak term escrow corresponds to what most people call a release candidate, but we don’t call it a release candidate because that name is used for some other purpose.

I wrote about this quite some time ago, but it was for the now-defunct TechNet Magazine, not for the blog, which means that it doesn’t show up in a blog search.

Here’s the final draft of that column. Now that I’ve put it on the blog, people can find it more easily.

Bonus chatter: You can compare this submitted version against the version that was published to see what was trimmed to fit the page. And a sign that this is an older document is its use of em-dashes, which are shunned nowadays due to their association with AI-generated text.

The post Microspeak elaborated: Isn’t escrow just a release candidate by another name? appeared first on The Old New Thing.