What’s so haunted about it is that the default behavior of the WM_INPUT­LANG­CHANGE­REQUEST message is to change input language!

For historical (and therefore now compatibility) reasons, when a hotkey-initiated input language change request is accepted, the system applies the change to all threads of that process. This means that all UI threads of the process need to be pumping messages so that they can receive the notification that their keyboard state has changed.

In this case, the customer had a background thread that created a window but was not pumping messages. That prevented the language change from completing and caused the main UI thread to hang.

The customer wanted to know if there was a way to configure their program so that hotkey-initiated input language changes don’t require all threads to be pumping messages. But that’s trying to solve too narrow a problem. If your thread has created a window, then it must pump messages. Today it’s causing trouble with input language changes. Tomorrow, it’s going to cause problems with DDE, and the day after tomorrow, it’s going to cause problems with theme changes.

Even if you had a way to change the way language changes work, that’s just one of the problems that your non-responding thread is causing. You should fix the root cause: Either pump messages or destroy the window so that it is no longer a UI thread and is no longer obligated to pump messages.

The post The case of the hang when the user changed keyboard layouts appeared first on The Old New Thing.

A collegue of mine pointed out that you still have the reverse problem: Since handles must be marked as inheritable for them to participate in PROC_THREAD_ATTRIBUTE_HANDLE_LIST, if another thread calls Create­Process with bInheritHandles = TRUE but without using PROC_THREAD_ATTRIBUTE_HANDLE_LIST, then they will accidentally inherit all of your handles.

This problem could have been avoided if the PROC_THREAD_ATTRIBUTE_HANDLE_LIST allowed you to include non-inheritable handles, in which case they would be non-inheritable by normal Create­Process but inheritable if explicitly opted back in. But alas, that’s not how it was designed.

Instead, you can create a helper process. All this helper process does is wait for the main process to exit, and then exit itself.

WaitForSingleObject(hMainProcess, INFINITE);
ExitProcess(0);

This process doesn’t sound like it’s doing anything useful, and it’s not. But what makes it useful is not what it’s doing but rather what is done to it.

The components in the main process create their handles as non-inheritable. When they wants to create a process with specific inherited handles, they duplicate the desired handles into the helper process (as inheritable), and then build a PROC_THREAD_ATTRIBUTE_HANDLE_LIST that lists those duplicates as handles to inherit. They also use the PROC_THREAD_ATTRIBUTE_PARENT_PROCESS to specify that the helper process is the parent process that the handles should be inherited from. Then they pass those thread attributes to Create­Process, and the new process will inherit exactly those handles. Then they clean up by closing the handles in the helper process with the help of Duplicate­Handle and DUPLICATE_CLOSE_SOURCE.

Notice that multiple threads can simultaneously be operating on the helper process in this way, so you need only one helper process to service all your handle-inheritance-control needs.

This avoids the accidental inheritance problem because the handles that belong to the components in the main process are still marked non-inheritable, so any other code in the main process that does a Create­Process will not inherit them.

The post Additional notes on controlling which handles are inherited by <CODE>Create­Process</CODE> appeared first on The Old New Thing.

Yes, you can do it by switching from Read­Directory­ChangesW. to Read­Directory­Changes­ExW and asking for Read­Directory­Notify­Extended­Information. This produces the FILE_NOTIFY_EXTENDED_INFORMATION structure, and that structure includes the FileId of the affected file. You can then match that up between the FILE_ACTION_RENAMED_OLD_FILE and FILE_ACTION_RENAMED_NEW_FILE to confirm that they are the two halves of the same rename operation.

The post Developing more confidence when tracking renames via <CODE>Read­Directory­ChangesW</CODE> appeared first on The Old New Thing.

After yet another incident of Visual Studio secretly changing the file encoding from 1252 to UTF-8 and breaking all non-ASCII strings, combined with Azure DevOps and Visual Studio simply ignoring encoding changes when showing diffs, a colleague decided to solve the problem once and for all by using explicit Unicode escapes \x#### to represent non-ASCII characters. That way, it doesn’t matter whether the file encoding is 1252 or UTF-8 because the two code pages agree on the common ASCII subset.

What used to be

IDS_AWESOME "That’s great!"

was changed to

IDS_AWESOME "That\x2019s great!"

Unfortunately, the resulting string that appeared on screen was

That 19s great!

What went wrong?

If you are encoding Unicode into your string, you have to put an L prefix on the quoted string. Otherwise, the \xABCD sequence is interpreted as an 8-bit \xAB escape sequence, followed by two literal characters CD. In this case, the \x2019 was interpreted as \x20 (which encodes a space) followed by the literal characters 19, resulting in the string That␣19s great!.

The correct conversion includes the L prefix.

IDS_AWESOME L"That\x2019s great!"

The post When you upgrade your resource strings to Unicode, don’t forget to specify the L prefix appeared first on The Old New Thing.

Commenter tbodt wrote,

This one seems easy enough to fix by Apple’s technique of giving the function the old behavior when the program is linked against the old SDK.

This sure sounds easy. If your program links with the newer SDK, then it gets the new behavior of accepting self-relative security descriptors. But if it links with the old SDK, then it gets the old behavior of requiring absolute security descriptors. If you want the new behavior, then you link with the new SDK.

This does create a subtlety that if you choose the wrong SDK to link against, everything still builds, but the results are different. Traditionally, Windows SDKs are forward-compatible: You can take an old program and link it against a newer SDK, and it will work exactly the same because the old program uses only the backward-compatible subset of the newer SDK. If you change behavior based on the SDK version that you link with, then it may not be obvious that the change in behavior you are experiencing is due to having upgraded the SDK libraries.

Also, what if a program is linked with one version of the SDK, but a DLL that it uses is linked with a different version of the SDK? Maybe you’re using a UI framework library that hasn’t seen any need to update to the newer SDK. Or maybe your program is the one using an old version of the SDK, but the UI framework library is using the newer one. Do you let the main program’s SDK version dictate the behavior of the function, even though the DLL is expecting different behavior? The poor DLL is going to call Co­Initialize­Security, and it won’t behave the way it expects.

Okay, so maybe you decide that the function changes its behavior not based on the program’s linked SDK version but rather the version of the calling DLL. But how does a function know which DLL called it? You might say, “Well, you can look at which DLL the return address belongs to.” But that doesn’t work in the case of tail call optimization.

// some function in a DLL
HRESULT InitializeWidgets(
    UINT maxWidgets,
    const WIDGET_ID* ownerId,
    PCWSTR ownerDescription,
    PCWSTR countainerName,
    PCWSTR containerDescription,
    COLORREF defaultColor,
    UINT defaultWidth,
    UINT defaultHeight,
    bool isRemoteAccessible,
    bool isPersistent)
{
    ⟦ various initialization steps ⟧

    static BYTE sd[] = { 0x01, ⟦ hard-coded values ⟧ };

    return CoInitializeSecurity(sd, -1, nullptr, nullptr,
                                RPC_C_AUTHN_LEVEL_DEFAULT,
                                RPC_C_IMP_LEVEL_IDENTIFY,
                                nullptr, EOAC_NONE, nullptr);
}

That final call to Co­Initialize­Security could be optimized into a tail call, in which case the subroutine call instruction changes to an unconditional branch, with the return address being the address of Initialize­Widget‘s caller. If Co­Initialize­Security snooped at its return address, it would be checking the SDK version of the wrong DLL.

Conversely, what if the function in the DLL is just a wrapper?

HRESULT CoInitializeSecuritywithLogging(
    _In_opt_ PSECURITY_DESCRIPTOR pSecDesc,
    _In_ LONG cAuthSvc,
    _In_reads_opt_(cAuthSvc) SOLE_AUTHENTICATION_SERVICE* asAuthSvc,
    _In_opt_ void* pReserved1,
    _In_ DWORD dwAuthnLevel,
    _In_ DWORD dwImpLevel,
    _In_opt_ void* pAuthList,
    _In_ DWORD dwCapabilities,
    _In_opt_ void* pReserved3)
{
    if (dwCapabilities & EOAC_APPID) {
        LogUuid("CoInitializeSecurity with APPID", (UUID*)pSecDesc);
    } else if (dwCapabilities & EOAC_ACCESS_CONTROL) {
        Log("CoInitializeSecurity with IAccessControl");
    } else {
        LogSecurityDescriptor("CoInitializeSecurity with security descriptor", pSecDesc);
    }
    HRESULT hr = CoInitializeSecurity(pSecDesc, cAuthSvc, asAuthSvc, pReserved1,
                        dwAuthnLevel, dwImpLevel, pAuthList, dwCapabilities, pReserved3);
    Log("CoInitializeSecurity returned", hr);
}

If you look at the return address, you will find the wrapper function and change your behavior to match the version that the wrapper function was built with, but that wrapper function is just passing through the parameters from its caller. It’s really the caller whose behavior we want to match, not the wrapper.

And what if the library is a static library rather than a DLL? It was written for one version of the SDK, but you link to another, and the behavior changes, and even if the function checks the return address, it will get the DLL’s address and see the DLL’s SDK version rather than the version the library wanted.

Changing behavior based on the SDK version you link to works only if programs are monolithic.

Bonus chatter: Changing to a newer SDK’s header files do create behavioral changes because, for example, structures with an explicit size member might get extended to contain additional fields, and the API uses the value of the size member to decide which version of the SDK the caller is using. But this is not dependent on the SDK that the caller links to, which is a good thing, because it lets you take static libraries which use different versions of the SDK header files and link them all together into a single program or DLL, and they will still work.

The post Why not have changes in API behavior depend on the SDK you link against? appeared first on The Old New Thing.

One of many points of mismatch was the organizational structure.

A colleague recalls that while he was assigned to the IBM offices in Boca Raton, Florida, there was a dispute over what key should be used to move from one field to another in dialog boxes. The folks at IBM were not happy with my colleague’s decision to use the TAB key, so they asked him to escalate the issue to his manager back in Redmond.

My colleague’s manager replied, “The reason you are in Boca is to make these decisions so I don’t have to be in Boca.”

My colleague rephrased this reply in a more corporate manner before passing it on to IBM: “Microsoft supports the use of the TAB key for this purpose.”

Unsatisfied, the IBM folks escalated the issue up their organizational chain for several levels, and replied that their VP (who was around seven levels of management above the programmers) was absolutely opposed to the use of the TAB for this purpose, and they wanted confirmation from the equivalent-level manager at Microsoft that Microsoft stands by the choice of the TAB key.

My colleague replied, “Bill Gates’s mother is not interested in the TAB key.”

This apparently ended the discussion, and the TAB key stayed.

Note: This upcoming Sunday is Mother’s Day in the United States. You probably shouldn’t ask her for her opinion on the TAB key.

¹ There was probably merit to both arguments.

The post A dispute over the <KBD>TAB</KBD> key highlights a mismatch between Microsoft and IBM organizational structures appeared first on The Old New Thing.

Windows doesn’t know whether your file is binary or text. As far as Windows is concerned, it’s just a bunch of bytes, and it’s up to you to interpret it. So in a sense, all files are binary files. If you want to insert carriage returns before linefeeds, you will have to do it yourself.

Now, it is often the case that you are using a higher level library, like the C runtime, in which case you can ask the library to do it for you, such as opening the file in "w" mode to indicate that the runtime should treat the file as a text file, or in "wb" to open as a binary file. But this work happens in the runtime library, not in Windows itself. The runtime library performs the necessary transformations and passes binary data to Windows. There are no further transformations once the data hits Write­File.

“But wait, there’s an old MS-DOS ioctl AH=4401h (Set device information) where you pass flags in DX, and bit 5 is the raw (binary) mode bit. So what’s the Windows version of this ioctl?”

If you look more closely, that MS-DOS ioctl applies only to character devices. If you try to use it on a disk file, you get ERROR_INVALID_FUNCTION.

ioctl_check_permissions:
        CMP     AL,2
        JAE     ioctl_control_string
        CMP     AL,0
        MOV     AL,BYTE PTR ES:[DI+sf_fcb+fcb_devid]
        JZ      ioctl_read              ; read the byte
        OR      DH,DH
        JZ      ioctl_check_device      ; can I set with this data?
        error   error_invalid_data      ; no DH <> 0

ioctl_check_device:
        TEST    AL,devid_ISDEV          ; can I set this handle?
        JZ      ioctl_bad_fun           ; no, it is a file.

...

ioctl_bad_fun:
        error   error_invalid_function

This IOCTL can be used to tell the console things like whether to perform line buffering on input. The Win32 equivalent is Set­Console­Mode, roughly corresponding to the Unix stty.

If you want to perform content transformations on files, you’ll have to do it yourself, or ask someone else (like the runtime library) to do it for you.

The post How do I inform Windows that I’m writing a binary file? appeared first on The Old New Thing.

That serious problem is abandonment.

Suppose a process crashes while it holds a shared or exclusive lock on our cross-process reader/writer lock. Semaphores don’t have owners, so if a thread terminates while in possession of a semaphore token, that token is lost forever. For our cross-process reader/writer lock, that means that the maximum number of shared acquirers goes down by one, and exclusive acquisitions will never succeed, since they will be waiting for that last token which will never be returned.

A synchronization object that does have the concept of ownership is the mutex, so we can build our reader/writer lock out of mutexes.

The idea here is that instead of claiming semaphore tokens, we claim mutexes. This means that we need one mutex for each potential shared acquisition, plus one more to avoid the starvation problem.

The outline is

• Shared acquisition: Claim any available token mutex.
• Shared release: Release the claimed token mutex.
• Exclusive acquisition: Claim all token mutexes.
• Exclusive release: Release all token mutexes.

HANDLE sharedMutex;
HANDLE tokenMutexes[MAX_SHARED];

struct TimeoutTracker
{
    explicit TimeoutTracker(DWORD timeout)
        : m_timeout(timeout) {}

    DWORD m_start = GetTickCount();

    DWORD Wait(HANDLE h)
    {
        DWORD elapsed = GetTickCount() - m_start;
        if (elapsed > m_timeout) return WAIT_TIMEOUT;
        return WaitForSingleObject(h, m_timeout - elapsed);
    }

    DWORD WaitMultiple(DWORD count, const HANDLE* handles, BOOL waitAll)            
    {                                                                               
        DWORD elapsed = GetTickCount() - m_start;                                   
        if (elapsed > m_timeout) return WAIT_TIMEOUT;                               
        return WaitForMultipleObjects(count, handles, waitAll, m_timeout - elapsed);
    }                                                                               
};

We change the return value of the Wait method so it returns the wait result rather than a success/failure. We also add a Wait­Multiple method for wrapping Wait­For­Multiple­Objects.

Next is a handy helper function.

int WaitResultToindex(DWORD result)
{
    auto index = result - WAIT_OBJECT_0;
    if (index < MAX_SHARED) return static_cast<int>(index);

    index = result - WAIT_ABANDONED_0;
    if (index < MAX_SHARED) return static_cast<int>(index);

    return -1;
}

The Wait­Result­To­Index function takes the wait result and returns the index of the acquired mutex, or -1 if no mutex was acquired.

Notice that this code treats the abandoned the state the same as the normal wait state. We are assuming that the code can recover from inconsistent data somehow. (For example, maybe the shared and exclusive accesses are to control access to a set of files, so the existing code already has to deal with file corruption.)

All that’s left is to implement the outline.

int AcquireShared()
{
    WaitForSingleObject(sharedMutex, INFINITE);

    auto result = WaitForMultipleObjects(MAX_SHARED, tokenMutexes, FALSE /* bWaitAll */, INFINITE);

    ReleaseMutex(sharedMutex);

    return WaitResultToIndex(result);
}

void ReleaseShared(int index)
{
    ReleaseMutex(tokenMutexes[index]);
}

int AcquireSharedWithTimeout(DWORD timeout)
{
    TimeoutTracker tracker(timeout);
    DWORD result = tracker.Wait(hSharedMutex);
    if (result != WAIT_OBJECT_0) return -1;
    result = tracker.WaitMultiple(MAX_SHARED, tokenMutexes, FALSE /* waitAll */);
    ReleaseMutex(sharedMutex);

    return WaitResultToIndex(result);
}

void AcquireExclusive()
{
    WaitForSingleObject(sharedMutex, INFINITE);

    auto result = WaitForMultipleObjects(MAX_SHARED, tokenMutexes, TRUE /* bWaitAll */, INFINITE);

    ReleaseMutex(sharedMutex);
}

void ReleaseExclusive()
{
    for (unsigned i = 0; i < MAX_SHARED; i++) {
        ReleaseMutex(tokenMutexes[i]);
    }
}

bool AcquireExclusiveWithTimeout(DWORD timeout)
{
    TimeoutTracker tracker(timeout);
    DWORD result = tracker.Wait(hSharedMutex);
    if (result != WAIT_OBJECT_0) return -1;
    result = tracker.WaitMultiple(MAX_SHARED, tokenMutexes, TRUE /* waitAll */);
    ReleaseMutex(sharedMutex);

    return result != WAIT_TIMEOUT;
}

The post Developing a cross-process reader/writer lock with limited readers, part 4: Abandonment appeared first on The Old New Thing.

The problem is that exclusive acquisitions are working to claim semaphore tokens one at a time, so it can lose out to shared acquisitions that are requested even after the exclusive acquisition had started, effectively letting shared acquisitions “jump ahead of the exclusive acquisition”, and thereby starving out exclusive acquisitions.

Let’s say that we have capped the number of shared acquisitions to five. In the above scenario, we have an exclusive acquiring thread and four shared acquiring threads. The first two shared acquiring threads (call them A and B) succeed at their shared acquisitions, and then the exclusive acquiring thread tries to enter exclusively. The exclusive acquiring thread needs five tokens, and it quickly gets three of them before blocking when it tries to get the fourth.

While the exclusive acquiring thread is waiting to get its fourth token, two other shared acquiring threads (call them C and D) try to enter in shared mode. They too block.

One of the original shared acquiring threads releases its shared lock, which release a token, and that token is quickly snapped up by the exclusive acquiring thread, thanks to the “mostly FIFO” policy for synchronization objects. (Let’s assume for the purpose of this discussion that none of the things that violate FIFO-ness has occurred.) The exclusive acquiring thread now waits to claim its fifth token.

When the second of the original shared acquiring threads releases its token, it is given to thread C, even though thread C started its shared acquisition after the exclusive acquiring thread tried to acquire exclusively.

And then when thread C releases its token, that token is given to thread D, since its request for the token precedes the exclusive thread’s request for the fifth token. The exclusive acquiring thread has once again gotten boxed out.

To fix this, we can make all acquisitions claim the shared mutex. The shared mutex then does the work of enforcing “mostly FIFO” acquisition behavior across all acquisitions.

Since we’re going to be doing combined timeouts, I’ll refactor the timeout management into a helper class.

struct TimeoutTracker
{
    explicit TimeoutTracker(DWORD timeout)
        : m_timeout(timeout) {}

    DWORD m_start = GetTickCount();

    bool Wait(HANDLE h)
    {
        DWORD elapsed = GetTickCount() - m_start;
        if (elapsed > m_timeout) return false;
        return WaitForSingleObject(h, m_timeout - elapsed)
                    == WAIT_OBJECT_0;
    }
};

We can use this helper class to manage our timeouts.

HANDLE sharedSemaphore;
HANDLE sharedMutex;

void AcquireShared()
{
    WaitForSingleObject(sharedMutex, INFINITE);

    WaitForSingleObject(sharedSemaphore, INFINITE);

    ReleaseMutex(sharedMutex);
}

bool AcquireSharedWithTimeout(DWORD timeout)
{
    TimeoutTracker tracker(timeout);         
    bool result = tracker.Wait(hSharedMutex);
    if (!result) return false;               
    result = tracker.Wait(sharedSemaphore);  
    ReleaseMutex(sharedMutex);               
    return result;                           
}

// no change to AcquireExclusive
void AcquireExclusive()
{
    WaitForSingleObject(sharedMutex, INFINITE);

    for (unsigned i = 0; i < MAX_SHARED; i++) {
        WaitForSingleObject(sharedSemaphore, INFINITE);
    }

    ReleaseMutex(sharedMutex);
}

// no functional change, but using the new helper class
bool AcquireExclusiveWithTimeout(DWORD timeout)
{
    TimeoutTracker tracker(timeout);        
    bool result = tracker.Wait(sharedMutex);
    if (!result) return false;              

    for (unsigned i = 0; i < MAX_SHARED; i++) {
        if (!tracker.Wait(sharedSemaphore)) {
            // Restore the tokens we already claimed.
            if (i > 0) {
                ReleaseSemaphore(sharedSemaphore, i, nullptr);
            }
            ReleaseMutex(sharedMutex);
            return false;
        }
    }
    ReleaseMutex(sharedMutex);
    return true;
}

(Yes, I’m not using RAII. I’ve made that choice for expository purposes, since it lets you see exactly when each synchronization object is acquired and released.)

Are we done?

No, we’re not done.

There is still a serious problem that needs to be fixed. We’ll look at it next time.

The post Developing a cross-process reader/writer lock with limited readers, part 3: Fairness appeared first on The Old New Thing.

The problem occurs when two threads both try to acquire the lock exclusively. In that case, both threads try to claim all the tokens. And the problem is that they can get into a stalemate, where one thread has half of the tokens, and the other thread has the other half, and neither side will back down, resulting in an impasse.

We can avoid this by serializing all the attempts to acquire exclusive locks. That way, there is at most one greedy thread at a time.

HANDLE sharedSemaphore;
HANDLE sharedMutex;

void AcquireExclusive()
{
    WaitForSingleObject(sharedMutex, INFINITE);

    for (unsigned i = 0; i < MAX_SHARED; i++) {
        WaitForSingleObject(sharedSemaphore, INFINITE);
    }

    ReleaseMutex(sharedMutex);
}

bool AcquireExclusiveWithTimeout(DWORD timeout)
{
    DWORD start = GetTickCount();
    WaitForSingleObject(sharedMutex, INFINITE);

    for (unsigned i = 0; i < MAX_SHARED; i++) {
        DWORD elapsed = GetTickCount() - start;
        if (elapsed > timeout ||
            WaitForSingleObject(sharedSemaphore, timeout - elapsed) == WAIT_TIMEOUT)) {
            // Restore the tokens we already claimed.
            if (i > 0) {
                ReleaseSemaphore(sharedSemaphore, i, nullptr);
            }
            ReleaseMutex(sharedMutex);
            return false;
        }
    }
    ReleaseMutex(sharedMutex);
    return true;
}

Okay, this avoids the problem of two exclusive acquisitions, but we still have a problem: Exclusive access throughput is poor. We’ll look at this next time.

The post Developing a cross-process reader/writer lock with limited readers, part 2: Taking turns when being grabby appeared first on The Old New Thing.