No, there is no way to tell Find­First­File to enumerate the files in date order. The files enumerated by Find­First­File are produced in whatever order the file system driver wants. For example, FAT typically enumerates them in the order the files appear in the directory listing, which could be in order of creation if the files were added sequentially, or some mishmash order if there were renames or deletions mixed in.

Since you can’t control the order in which the files are enumerated, you’ll have to do the sorting yourself. The naïve solution is to read in all the entries, sort them by last-modified date, and then delete all but the last ten. This is O(n) space and O(n log n) running time.

But you can do better.

This job calls for a priority queue. A priority queue is a data structure that supports these operations, where n is the number of items in the priority queue.

• Add sorted: O(log n)
• Find largest: O(1)
• Remove largest: O(log n)

The above description is for a max-priority queue. There is also a min-priority queue where the final two operations are “find smallest” and “remove smallest”. The two versions are equivalent because you can just use a reverse-sense comparison to switch from one to the other.

What we can do is enumerate all the files and add them one by one to a min-priority queue sorted by modified date. The priority queue holds the newest items. If the priority queue size exceeds 10, then we delete the file corresponding to the “smallest” (earliest) entry in the priority queue, and the remove that entry from the priority queue.

Since the priority queue size has a fixed cap, all of the operations run in O(1) time because the value of n is bounded by a predetermined constant. (Of course, the larger the cap, the larger the constant in O(1).) The overall algorithm then runs in O(n) times, where n is the number of files in the directory.

Here’s a sketch of a solution. To get a min-priority heap, we have to reverse the sense of the comparison in dateAscending.

constexpr int files_to_keep = 10;

auto dateAscending = [](const WIN32_FIND_DATA& a, const WIN32_FIND_DATA& b) {
    return CompareFileTime(&a.ftLastWriteTime, &b.ftLastWriteTime) > 0;
    };

std::priority_queue<WIN32_FIND_DATA,
        std::vector<WIN32_FIND_DATA>, decltype(dateAscending)>
        names(dateAscending);

WIN32_FIND_DATA wfd;
wil::unique_hfind findHandle( FindFirstFileW(L"*.*", &wfd));
if (findHandle.is_valid())
{
    do
    {
        if (wfd.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) {
            // Skip directories
            continue;
        }

        names.push(wfd);
        if (names.size() > files_to_keep) {
            DeleteFileW(names.top().cFileName);
            names.pop();
        }
    } while (FindNextFileW(findHandle.get(), &wfd));
}

It’s unfortunate that std::priority_queue doesn’t have a deduction guide that deduces the Comparator. We have to specify it explicitly, and since it comes after the Container, we have to write out the container type manually instead of allowing it to be deduced.

It’s also unfortunate that it’s hard to call reserve() on the vector hiding inside the priority_queue. This means that the names.push() could throw an exception. At least we use an RAII type (wil::unique_hfind) to ensure that the find handle is not leaked.

If you have access to std::inplace_vector, you could use a

std::priority_queue<WIN32_FIND_DATA,
        std::inplace_vector<WIN32_FIND_DATA, files_to_keep + 1>,
        decltype(dateAscending)> names(dateAscending);

to avoid memory allocations entirely. (It also makes it clearer that the algorithm is constant-space.)

This is an example of a so-called online algorithm, an algorithm that does its work incrementally rather than requiring all of the input before it can start working.

Exercise: What if the task was to delete the 10 oldest files?

The post A constant-space linear-time algorithm for deleting all but the 10 most recent files in a directory appeared first on The Old New Thing.

What’s so haunted about it is that the default behavior of the WM_INPUT­LANG­CHANGE­REQUEST message is to change input language!

For historical (and therefore now compatibility) reasons, when a hotkey-initiated input language change request is accepted, the system applies the change to all threads of that process. This means that all UI threads of the process need to be pumping messages so that they can receive the notification that their keyboard state has changed.

In this case, the customer had a background thread that created a window but was not pumping messages. That prevented the language change from completing and caused the main UI thread to hang.

The customer wanted to know if there was a way to configure their program so that hotkey-initiated input language changes don’t require all threads to be pumping messages. But that’s trying to solve too narrow a problem. If your thread has created a window, then it must pump messages. Today it’s causing trouble with input language changes. Tomorrow, it’s going to cause problems with DDE, and the day after tomorrow, it’s going to cause problems with theme changes.

Even if you had a way to change the way language changes work, that’s just one of the problems that your non-responding thread is causing. You should fix the root cause: Either pump messages or destroy the window so that it is no longer a UI thread and is no longer obligated to pump messages.

The post The case of the hang when the user changed keyboard layouts appeared first on The Old New Thing.

A reader asked, “What was the real reason client editions were prevented from using more than 4 GB of RAM?”

The use of the word “real” in the question implies that the reader believed that the official reason was a lie, and there was some nefarious evil reason for the limitation. It’s unclear what this nefarious reason would be. Maybe the reader thought the “real” reason was “To force users to buy copies of Windows Server, which is far more lucrative”, though that doesn’t make sense. The cheapest version of Windows Server 2003 32-bit edition that supported more than 4 GB of RAM was Enterprise Edition, which sold for $3,999.¹ This is an outrageous price for a consumer operating system.

The reason why consumer products don’t use RAM above 4 GB is explained in the documentation that accompanied the introduction of the feature under “Driver issues”.

Typically, device drivers must be modified in a number of small ways. Although the actual code changes may be small, they can be difficult. This is because when not using PAE memory addressing, it is possible for a device driver to assume that physical addresses and 32-bit virtual address limits are identical. PAE memory makes this assumption untrue.

…

[M]any device drivers designed for these systems may not have been tested on system configurations with PAE enabled. In order to limit the impact to device driver compatibility, changes to the hardware abstraction layer (HAL) were made to Windows XP SP2 and Windows Server 2003 SP1 Standard Edition to limit physical address space to 4 GB.

As explained above, memory above 4 GB was not enabled for compatibility reasons. Many drivers inadvertently assume that all physical address fit in 32 bits. (DMA drivers for example.) Those drivers would corrupt memory if memory above 4 GB were made available.

Memory above 4 GB is enabled on server because if you are a server administrator, you don’t install random drivers for that hand-held scanner you bought at Best Buy from the bargain bin for $10. Server administrators typically run only the plain vanilla drivers that come with Windows. (They don’t even install manufacturer video drivers.) All the drivers that come with Windows have been tested for addresses above 4 GB. That 2001 driver for the $10 handheld scanner has not, and there’s a good chance that it will truncate addresses above 4 GB and corrupt memory as a result.

The consumer market and the server market are very different in terms of usage pattern. Consumers will install practically anything. Server administrators install as little as possible. Consumers have no technical expertise. Server administrators have access to highly-skilled staff.

Of course, this is all now a historical oddity. Systems with only 4 GB of RAM are vanishingly rare, and Windows began discouraging the production of systems using 32-bit processors in 2020, finally ending the production of 32-bit editions entirely with Windows 11.

¹ The only other version that supported more than 4 GB of RAM was Datacenter Edition, and on the pricing sheet I found, they didn’t even bother listing the price. If you have to ask, you can’t afford it.

The post Why do Windows client editions on 32-bit x86 systems artificially limit RAM to 4 GB? appeared first on The Old New Thing.

A collegue of mine pointed out that you still have the reverse problem: Since handles must be marked as inheritable for them to participate in PROC_THREAD_ATTRIBUTE_HANDLE_LIST, if another thread calls Create­Process with bInheritHandles = TRUE but without using PROC_THREAD_ATTRIBUTE_HANDLE_LIST, then they will accidentally inherit all of your handles.

This problem could have been avoided if the PROC_THREAD_ATTRIBUTE_HANDLE_LIST allowed you to include non-inheritable handles, in which case they would be non-inheritable by normal Create­Process but inheritable if explicitly opted back in. But alas, that’s not how it was designed.

Instead, you can create a helper process. All this helper process does is wait for the main process to exit, and then exit itself.

WaitForSingleObject(hMainProcess, INFINITE);
ExitProcess(0);

This process doesn’t sound like it’s doing anything useful, and it’s not. But what makes it useful is not what it’s doing but rather what is done to it.

The components in the main process create their handles as non-inheritable. When they wants to create a process with specific inherited handles, they duplicate the desired handles into the helper process (as inheritable), and then build a PROC_THREAD_ATTRIBUTE_HANDLE_LIST that lists those duplicates as handles to inherit. They also use the PROC_THREAD_ATTRIBUTE_PARENT_PROCESS to specify that the helper process is the parent process that the handles should be inherited from. Then they pass those thread attributes to Create­Process, and the new process will inherit exactly those handles. Then they clean up by closing the handles in the helper process with the help of Duplicate­Handle and DUPLICATE_CLOSE_SOURCE.

Notice that multiple threads can simultaneously be operating on the helper process in this way, so you need only one helper process to service all your handle-inheritance-control needs.

This avoids the accidental inheritance problem because the handles that belong to the components in the main process are still marked non-inheritable, so any other code in the main process that does a Create­Process will not inherit them.

The post Additional notes on controlling which handles are inherited by <CODE>Create­Process</CODE> appeared first on The Old New Thing.

Yes, you can do it by switching from Read­Directory­ChangesW. to Read­Directory­Changes­ExW and asking for Read­Directory­Notify­Extended­Information. This produces the FILE_NOTIFY_EXTENDED_INFORMATION structure, and that structure includes the FileId of the affected file. You can then match that up between the FILE_ACTION_RENAMED_OLD_FILE and FILE_ACTION_RENAMED_NEW_FILE to confirm that they are the two halves of the same rename operation.

The post Developing more confidence when tracking renames via <CODE>Read­Directory­ChangesW</CODE> appeared first on The Old New Thing.

After yet another incident of Visual Studio secretly changing the file encoding from 1252 to UTF-8 and breaking all non-ASCII strings, combined with Azure DevOps and Visual Studio simply ignoring encoding changes when showing diffs, a colleague decided to solve the problem once and for all by using explicit Unicode escapes \x#### to represent non-ASCII characters. That way, it doesn’t matter whether the file encoding is 1252 or UTF-8 because the two code pages agree on the common ASCII subset.

What used to be

IDS_AWESOME "That’s great!"

was changed to

IDS_AWESOME "That\x2019s great!"

Unfortunately, the resulting string that appeared on screen was

That 19s great!

What went wrong?

If you are encoding Unicode into your string, you have to put an L prefix on the quoted string. Otherwise, the \xABCD sequence is interpreted as an 8-bit \xAB escape sequence, followed by two literal characters CD. In this case, the \x2019 was interpreted as \x20 (which encodes a space) followed by the literal characters 19, resulting in the string That␣19s great!.

The correct conversion includes the L prefix.

IDS_AWESOME L"That\x2019s great!"

The post When you upgrade your resource strings to Unicode, don’t forget to specify the L prefix appeared first on The Old New Thing.

Commenter tbodt wrote,

This one seems easy enough to fix by Apple’s technique of giving the function the old behavior when the program is linked against the old SDK.

This sure sounds easy. If your program links with the newer SDK, then it gets the new behavior of accepting self-relative security descriptors. But if it links with the old SDK, then it gets the old behavior of requiring absolute security descriptors. If you want the new behavior, then you link with the new SDK.

This does create a subtlety that if you choose the wrong SDK to link against, everything still builds, but the results are different. Traditionally, Windows SDKs are forward-compatible: You can take an old program and link it against a newer SDK, and it will work exactly the same because the old program uses only the backward-compatible subset of the newer SDK. If you change behavior based on the SDK version that you link with, then it may not be obvious that the change in behavior you are experiencing is due to having upgraded the SDK libraries.

Also, what if a program is linked with one version of the SDK, but a DLL that it uses is linked with a different version of the SDK? Maybe you’re using a UI framework library that hasn’t seen any need to update to the newer SDK. Or maybe your program is the one using an old version of the SDK, but the UI framework library is using the newer one. Do you let the main program’s SDK version dictate the behavior of the function, even though the DLL is expecting different behavior? The poor DLL is going to call Co­Initialize­Security, and it won’t behave the way it expects.

Okay, so maybe you decide that the function changes its behavior not based on the program’s linked SDK version but rather the version of the calling DLL. But how does a function know which DLL called it? You might say, “Well, you can look at which DLL the return address belongs to.” But that doesn’t work in the case of tail call optimization.

// some function in a DLL
HRESULT InitializeWidgets(
    UINT maxWidgets,
    const WIDGET_ID* ownerId,
    PCWSTR ownerDescription,
    PCWSTR countainerName,
    PCWSTR containerDescription,
    COLORREF defaultColor,
    UINT defaultWidth,
    UINT defaultHeight,
    bool isRemoteAccessible,
    bool isPersistent)
{
    ⟦ various initialization steps ⟧

    static BYTE sd[] = { 0x01, ⟦ hard-coded values ⟧ };

    return CoInitializeSecurity(sd, -1, nullptr, nullptr,
                                RPC_C_AUTHN_LEVEL_DEFAULT,
                                RPC_C_IMP_LEVEL_IDENTIFY,
                                nullptr, EOAC_NONE, nullptr);
}

That final call to Co­Initialize­Security could be optimized into a tail call, in which case the subroutine call instruction changes to an unconditional branch, with the return address being the address of Initialize­Widget‘s caller. If Co­Initialize­Security snooped at its return address, it would be checking the SDK version of the wrong DLL.

Conversely, what if the function in the DLL is just a wrapper?

HRESULT CoInitializeSecuritywithLogging(
    _In_opt_ PSECURITY_DESCRIPTOR pSecDesc,
    _In_ LONG cAuthSvc,
    _In_reads_opt_(cAuthSvc) SOLE_AUTHENTICATION_SERVICE* asAuthSvc,
    _In_opt_ void* pReserved1,
    _In_ DWORD dwAuthnLevel,
    _In_ DWORD dwImpLevel,
    _In_opt_ void* pAuthList,
    _In_ DWORD dwCapabilities,
    _In_opt_ void* pReserved3)
{
    if (dwCapabilities & EOAC_APPID) {
        LogUuid("CoInitializeSecurity with APPID", (UUID*)pSecDesc);
    } else if (dwCapabilities & EOAC_ACCESS_CONTROL) {
        Log("CoInitializeSecurity with IAccessControl");
    } else {
        LogSecurityDescriptor("CoInitializeSecurity with security descriptor", pSecDesc);
    }
    HRESULT hr = CoInitializeSecurity(pSecDesc, cAuthSvc, asAuthSvc, pReserved1,
                        dwAuthnLevel, dwImpLevel, pAuthList, dwCapabilities, pReserved3);
    Log("CoInitializeSecurity returned", hr);
}

If you look at the return address, you will find the wrapper function and change your behavior to match the version that the wrapper function was built with, but that wrapper function is just passing through the parameters from its caller. It’s really the caller whose behavior we want to match, not the wrapper.

And what if the library is a static library rather than a DLL? It was written for one version of the SDK, but you link to another, and the behavior changes, and even if the function checks the return address, it will get the DLL’s address and see the DLL’s SDK version rather than the version the library wanted.

Changing behavior based on the SDK version you link to works only if programs are monolithic.

Bonus chatter: Changing to a newer SDK’s header files do create behavioral changes because, for example, structures with an explicit size member might get extended to contain additional fields, and the API uses the value of the size member to decide which version of the SDK the caller is using. But this is not dependent on the SDK that the caller links to, which is a good thing, because it lets you take static libraries which use different versions of the SDK header files and link them all together into a single program or DLL, and they will still work.

The post Why not have changes in API behavior depend on the SDK you link against? appeared first on The Old New Thing.

One of many points of mismatch was the organizational structure.

A colleague recalls that while he was assigned to the IBM offices in Boca Raton, Florida, there was a dispute over what key should be used to move from one field to another in dialog boxes. The folks at IBM were not happy with my colleague’s decision to use the TAB key, so they asked him to escalate the issue to his manager back in Redmond.

My colleague’s manager replied, “The reason you are in Boca is to make these decisions so I don’t have to be in Boca.”

My colleague rephrased this reply in a more corporate manner before passing it on to IBM: “Microsoft supports the use of the TAB key for this purpose.”

Unsatisfied, the IBM folks escalated the issue up their organizational chain for several levels, and replied that their VP (who was around seven levels of management above the programmers) was absolutely opposed to the use of the TAB for this purpose, and they wanted confirmation from the equivalent-level manager at Microsoft that Microsoft stands by the choice of the TAB key.

My colleague replied, “Bill Gates’s mother is not interested in the TAB key.”

This apparently ended the discussion, and the TAB key stayed.

Note: This upcoming Sunday is Mother’s Day in the United States. You probably shouldn’t ask her for her opinion on the TAB key.

¹ There was probably merit to both arguments.

The post A dispute over the <KBD>TAB</KBD> key highlights a mismatch between Microsoft and IBM organizational structures appeared first on The Old New Thing.

Windows doesn’t know whether your file is binary or text. As far as Windows is concerned, it’s just a bunch of bytes, and it’s up to you to interpret it. So in a sense, all files are binary files. If you want to insert carriage returns before linefeeds, you will have to do it yourself.

Now, it is often the case that you are using a higher level library, like the C runtime, in which case you can ask the library to do it for you, such as opening the file in "w" mode to indicate that the runtime should treat the file as a text file, or in "wb" to open as a binary file. But this work happens in the runtime library, not in Windows itself. The runtime library performs the necessary transformations and passes binary data to Windows. There are no further transformations once the data hits Write­File.

“But wait, there’s an old MS-DOS ioctl AH=4401h (Set device information) where you pass flags in DX, and bit 5 is the raw (binary) mode bit. So what’s the Windows version of this ioctl?”

If you look more closely, that MS-DOS ioctl applies only to character devices. If you try to use it on a disk file, you get ERROR_INVALID_FUNCTION.

ioctl_check_permissions:
        CMP     AL,2
        JAE     ioctl_control_string
        CMP     AL,0
        MOV     AL,BYTE PTR ES:[DI+sf_fcb+fcb_devid]
        JZ      ioctl_read              ; read the byte
        OR      DH,DH
        JZ      ioctl_check_device      ; can I set with this data?
        error   error_invalid_data      ; no DH <> 0

ioctl_check_device:
        TEST    AL,devid_ISDEV          ; can I set this handle?
        JZ      ioctl_bad_fun           ; no, it is a file.

...

ioctl_bad_fun:
        error   error_invalid_function

This IOCTL can be used to tell the console things like whether to perform line buffering on input. The Win32 equivalent is Set­Console­Mode, roughly corresponding to the Unix stty.

If you want to perform content transformations on files, you’ll have to do it yourself, or ask someone else (like the runtime library) to do it for you.

The post How do I inform Windows that I’m writing a binary file? appeared first on The Old New Thing.

That serious problem is abandonment.

Suppose a process crashes while it holds a shared or exclusive lock on our cross-process reader/writer lock. Semaphores don’t have owners, so if a thread terminates while in possession of a semaphore token, that token is lost forever. For our cross-process reader/writer lock, that means that the maximum number of shared acquirers goes down by one, and exclusive acquisitions will never succeed, since they will be waiting for that last token which will never be returned.

A synchronization object that does have the concept of ownership is the mutex, so we can build our reader/writer lock out of mutexes.

The idea here is that instead of claiming semaphore tokens, we claim mutexes. This means that we need one mutex for each potential shared acquisition, plus one more to avoid the starvation problem.

The outline is

• Shared acquisition: Claim any available token mutex.
• Shared release: Release the claimed token mutex.
• Exclusive acquisition: Claim all token mutexes.
• Exclusive release: Release all token mutexes.

HANDLE sharedMutex;
HANDLE tokenMutexes[MAX_SHARED];

struct TimeoutTracker
{
    explicit TimeoutTracker(DWORD timeout)
        : m_timeout(timeout) {}

    DWORD m_start = GetTickCount();

    DWORD Wait(HANDLE h)
    {
        DWORD elapsed = GetTickCount() - m_start;
        if (elapsed > m_timeout) return WAIT_TIMEOUT;
        return WaitForSingleObject(h, m_timeout - elapsed);
    }

    DWORD WaitMultiple(DWORD count, const HANDLE* handles, BOOL waitAll)            
    {                                                                               
        DWORD elapsed = GetTickCount() - m_start;                                   
        if (elapsed > m_timeout) return WAIT_TIMEOUT;                               
        return WaitForMultipleObjects(count, handles, waitAll, m_timeout - elapsed);
    }                                                                               
};

We change the return value of the Wait method so it returns the wait result rather than a success/failure. We also add a Wait­Multiple method for wrapping Wait­For­Multiple­Objects.

Next is a handy helper function.

int WaitResultToindex(DWORD result)
{
    auto index = result - WAIT_OBJECT_0;
    if (index < MAX_SHARED) return static_cast<int>(index);

    index = result - WAIT_ABANDONED_0;
    if (index < MAX_SHARED) return static_cast<int>(index);

    return -1;
}

The Wait­Result­To­Index function takes the wait result and returns the index of the acquired mutex, or -1 if no mutex was acquired.

Notice that this code treats the abandoned the state the same as the normal wait state. We are assuming that the code can recover from inconsistent data somehow. (For example, maybe the shared and exclusive accesses are to control access to a set of files, so the existing code already has to deal with file corruption.)

All that’s left is to implement the outline.

int AcquireShared()
{
    WaitForSingleObject(sharedMutex, INFINITE);

    auto result = WaitForMultipleObjects(MAX_SHARED, tokenMutexes, FALSE /* bWaitAll */, INFINITE);

    ReleaseMutex(sharedMutex);

    return WaitResultToIndex(result);
}

void ReleaseShared(int index)
{
    ReleaseMutex(tokenMutexes[index]);
}

int AcquireSharedWithTimeout(DWORD timeout)
{
    TimeoutTracker tracker(timeout);
    DWORD result = tracker.Wait(hSharedMutex);
    if (result != WAIT_OBJECT_0) return -1;
    result = tracker.WaitMultiple(MAX_SHARED, tokenMutexes, FALSE /* waitAll */);
    ReleaseMutex(sharedMutex);

    return WaitResultToIndex(result);
}

void AcquireExclusive()
{
    WaitForSingleObject(sharedMutex, INFINITE);

    auto result = WaitForMultipleObjects(MAX_SHARED, tokenMutexes, TRUE /* bWaitAll */, INFINITE);

    ReleaseMutex(sharedMutex);
}

void ReleaseExclusive()
{
    for (unsigned i = 0; i < MAX_SHARED; i++) {
        ReleaseMutex(tokenMutexes[i]);
    }
}

bool AcquireExclusiveWithTimeout(DWORD timeout)
{
    TimeoutTracker tracker(timeout);
    DWORD result = tracker.Wait(hSharedMutex);
    if (result != WAIT_OBJECT_0) return -1;
    result = tracker.WaitMultiple(MAX_SHARED, tokenMutexes, TRUE /* waitAll */);
    ReleaseMutex(sharedMutex);

    return result != WAIT_TIMEOUT;
}

The post Developing a cross-process reader/writer lock with limited readers, part 4: Abandonment appeared first on The Old New Thing.