The customer’s investigations revealed that the problem occurred because their window was receiving message 0x0091, and the parameters are wrong. Who is sending this message with the wrong parameters?

Okay, first of all, how do you even know that the parameters are wrong? The message is not listed in winuser.h or in MSDN (as it was then called).

We explained that message 0x0091 is an internal message that they should just pass to Def­Window­Proc unchanged. What makes the customer think that the message is being received with the wrong parameters?

The customer said that their program was using that message as a custom message, and now, in addition to getting it when their program sends the message, they are also getting spurious copies of the message with WPARAM and LPARAM values that don’t correspond to any values that the program itself sent.

We informed them that they shouldn’t have been using that message for their own purposes. Those messages are in the system-defined range, which means that they are off-limits to applications. If they want to send a private message, use one in the application space.

It’s like finding an empty closet in an office building and using it to store your bicycle, but now, when you come to work, you find that the closet is filled with other stuff and there’s no room for your bicycle any more. “Why is there stuff in that closet?” Because it wasn’t your closet in the first place.

The liaison took our advice back to the customer, but mentioned that the customer probably won’t like that answer. The message 0x0091 was not the only message they were using. They also used other messages below WM_USER, and they were all causing problems; they just wanted to start their investigation with 0x0091.

Oh well. But I hope it’s as simple as just changing a macro definition from

#define WM_MYSECRETMESSAGE 0x0091

to

#define WM_MYSECRETMESSAGE (WM_APP + 1020) // or something

Pick a message in the range available to applications for custom use.

The post What’s up with window message <CODE>0x0091</CODE>? We’re getting it with unexpected parameters appeared first on The Old New Thing.

My psychic powers tell me that the thread didn’t actually exit.

What the customer is observing is probably that their thread procedure has returned, signaling the end of the thread. But a lot of stuff happens after the thread procedure exits. The system needs to send DLL_THREAD_DETACH notifications to all of the DLLs (unless the DLL has opted out via Disable­Thread­Library­Calls), and doing so requires the loader lock.

I would use the debugger to look for the thread you thought had exited and see what it’s doing. It might be blocked waiting for the loader lock because some other thread is hogging it. Or it could be running a DLL’s detach code, and that detach code has gotten stuck on a long-running operation.

I suspect it’s the latter: One of the DLLs is waiting for something in its detach code, and that something takes about a minute.

We didn’t hear back from the customer, which could mean that this was indeed the problem. Or it could mean that this didn’t help, but they decided that we weren’t being helpful and didn’t pursue the matter further. Unfortunately, in a lot of these customer debugging engagements, we never hear back whether our theory worked. (Another possibility is that the customer wrote back with a “thank you”, but the customer liaison didn’t forward it to the engineering team because they didn’t want to bother them any further.)

The post Why is there a long delay between a thread exiting and the <CODE>Wait­For­Single­Object</CODE> returning? appeared first on The Old New Thing.

What was this red telephone for? Was it a direct line to Bill Gates’s office? Or maybe it was a direct line to Security?

Nope.

It was just a plain telephone.

And that’s what made it special.

As is customary at large companies, the telephones on the Microsoft campus were part of a corporate PBX (private branch exchange). A PBX is a private telephone system within a company, and companies use them to save on telephone costs, as well as to provide auxiliary telephone services. For example, you could call another office by dialing just the extension, and the call would be routed entirely within the PBX without having to interact with the public telephone systems. Generally, most calls are typically from one office to another, so a PBX saves considerable money by reducing demand for outside communications services. Also, a PBX allows integration with other systems. For example, if somebody leaves you a voicemail, the system can email you a message.

But what if the PBX is down, and there is an emergency?

The red telephones are plain telephones with standard telephone service. They are not part of the PBX and therefore operate normally even if there is a PBX outage. If there is an emergency, the receptionist can use the red telephone to call emergency services. Presumably, each red telephone was registered in the telephone system with the address of its building, allowing emergency services to dispatch assistance quickly.

Bonus chatter: What was the “Quick Reference Guide”? It was a guide to emergency procedures. It makes sense that it was kept next to the emergency telephone.

Bonus bonus chatter: Bill Gates kept a red telephone in his own office as well. If the PBX went down, I guess it was technically true that the red telephones could be used to call Bill Gates’s office.

The post Why was there a red telephone at every receptionist desk? appeared first on The Old New Thing.

The backstory behind this puzzle is that my colleague had thought this problem was solvable in O(N log N), presumably by sorting the array and then scanning for the duplicate. They posed this as an interview question, and the interviewee found an even better linear-time algorithm!

My solution is to interpret the array as a linked list of 1-based indices, and borrow the sign bit of each integer as a flag to indicate that the slot has been visited. We start at index 1 and follow the indices until they either reach a value whose sign bit has already been set (which is our duplicate), or they return to index 1 (a cycle). If we find a cycle, then move to the next index which does not have the sign bit set, and repeat. At the end, you can restore the original values by clearing the sign bits.²

I figured that modifying the values was acceptable given that the O(N log N) solution also modifies the array. At least my version restores the original values when it’s done!

But it turns out the interview candidate found an even better O(N) algorithm, one that doesn’t modify the array at all.

Again, view the array values as indices. You are looking for two nodes that point to the same destination. You already know that no array entry has the value N, so the entry at index N cannot be part of a cycle. Therefore, the chain that starts at N must eventually join an existing cycle, and that join point is a duplicate. Start at index N and use Floyd’s cycle detector algorithm to find the start of the cycle in O(N) time.

¹ If you constrain the problem further to say that there is exactly one duplicate, then you can find the duplicate by summing all the values and then subtracting N(N−1)/2.

² I’m pulling a fast one. This is really O(N) space because I’m using the sign bit as a convenient “initially zero” flag bit.

The post Finding a duplicated item in an array of <VAR>N</VAR> integers in the range 1 to <VAR>N</VAR> − 1 appeared first on The Old New Thing.

What we can do is wait for the waiting thread to acknowledge the operation.

_Guarded_by_(desiredMutex) DWORD desiredCounter = 1;
DWORD activeCounter = 0;

void wait_until_active(DWORD value)
{
    DWORD current = activeCounter;
    while (static_cast<int>(current - value) < 0) {
        WaitOnAddress(&activeCounter, &current,
                      sizeof(activeCounter), INFINITE);
        current = activeCounter;
    }
}

The wait_until_active function waits until the value of active­Counter is at least as large as value. We do this by subtracting the two values, to avoid wraparound problems.¹ The comparison takes advantage of the guarantee in C++20 that conversion from an unsigned integer to a signed integer converts to the value that is numerically equal modulo 2ⁿ where n is the number of bits in the destination. (Prior to C++20, the result was implementation-defined, but in practice all modern implementations did what C++20 mandates.)²

You can also use std::atomic:

_Guarded_by_(desiredMutex) DWORD desiredCounter = 1;
std::atomic<DWORD> activeCounter;

void wait_until_active(DWORD value)
{
    DWORD current = activeCounter;
    while (static_cast<int>(current - value) < 0) {
        activeCounter.wait(current);
        current = activeCounter;
    }
}

As before, the background thread manipulates the desiredHandles and desiredActions, then signals the waiting thread to wake up and process the changes. But this time, the background thread blocks until the waiting thread acknowledges the changes.

// Warning: For expository purposes. Almost no error checking.
void waiting_thread()
{
    bool update = true;
    std::vector<wil::unique_handle> handles;
    std::vector<std::function<void()>> actions;

    while (true)
    {
        if (std::exchange(update, false)) {
            std::lock_guard guard(desiredMutex);

            handles.clear();
            handles.reserve(desiredHandles.size() + 1);
            std::transform(desiredHandles.begin(), desiredHandles.end(),
                std::back_inserter(handles),
                [](auto&& h) { return duplicate_handle(h.get()); });
            // Add the bonus "changed" handle
            handles.emplace_back(duplicate_handle(changed.get()));

            actions = desiredActions;

            if (activeCounter != desiredCounter) {
                activeCounter = desiredCounter;   
                WakeByAddressAll(&activeCounter); 
            }
        }

        auto count = static_cast<DWORD>(handles.size());
                        
        auto result = WaitForMultipleObjects(count,
                        handles.data()->get(), FALSE, INFINITE);
        auto index = result - WAIT_OBJECT_0;
        if (index == count - 1) {
            // the list changed. Loop back to update.
            update = true;
            continue;
        } else if (index < count - 1) {
            actions[index]();
        } else {
            // deal with unexpected result
        }
    }
}

void change_handle_list()
{
    DWORD value;
    {
        std::lock_guard guard(desiredMutex);
        ⟦ make changes to desiredHandles and desiredActions ⟧
        value = ++desiredCounter;
        SetEvent(changed.get());
    }
    wait_until_active(value);
}

The pattern is that after the background thread makes the desired changes, they increment the desiredCounter and signal the event. It’s okay if multiple threads make changes before the waiting thread wakes up. The changes simply accumulate, and the event just stays signaled. Each background thread then waits for the waiting thread to process the change.

On the waiting side, we process changes as usual, but we also publish our current change counter if it has changed, to let the background threads know that we made some progress. Eventually, we will make enough progress that all of the pending changes have been processed, and the last ackground thread will be released from wait_until_active.

¹ You’ll run into problems if the counter increments 2 billion times without the worker thread noticing. At a thousand increments per second, that’ll last you a month. I figure that if you have a worker thread that is unresponsible for that long, then you have bigger problems. But you can avoid even that problem by switching to a 64-bit integer, so that the overflow won’t happen before the sun is expected to turn into a red giant.

² The holdouts would be compilers for systems that are not two’s-complement.

The post How do you add or remove a handle from an active <CODE>Wait­For­Multiple­Objects</CODE>?, part 2 appeared first on The Old New Thing.

You can fake it by using an event which means “I want to change the list of handles.” The background thread can add that handle to its list, and if the “I want to change the list of handles” event is signaled, it updates its list.

One of the easier ways to represent the desired change is to maintain two lists, the “active” list (the one being waited on) and the “desired” list (the one you want to change it to). The background thread can make whatever changes to the “desired” list it wants, and then it signals the “changed” event. The waiting thread sees that the “changed” event is set and copies the “desired” list to the “active” list. This copying needs to be done with Duplicate­Handle because the background thread might close a handle in the “desired” list, and we can’t close a handle while it is being waited on.

wil::unique_handle duplicate_handle(HANDLE other)
{
    HANDLE result;
    THROW_IF_WIN32_BOOL_FALSE(
        DuplicateHandle(GetCurrentProcess(), other,
            GetCurrentProcess(), &result,
            0, FALSE, DUPLICATE_SAME_ACCESS));
    return wil::unique_handle(result);
}

This helper function duplicates a raw HANDLE and returns it in a wil::unique_handle. The duplicate handle has its own lifetime separate from the original. The waiting thread operates on a copy of the handles, so that it is unaffected by changes to the original handles.

std::mutex desiredMutex;
_Guarded_by_(desiredMutex) std::vector<wil::unique_handle> desiredHandles;
_Guarded_by_(desiredMutex) std::vector<std::function<void()>> desiredActions;

The desiredHandles is a vector of handles we want to be waiting for, and the The desiredActions is a parallel vector of things to do for each of those handles.

// auto-reset, initially unsignaled
wil::unique_handle changed(CreateEvent(nullptr, FALSE, FALSE, nullptr));

void waiting_thread()
{
    while (true)
    {
        std::vector<wil::unique_handle> handles;
        std::vector<std::function<void()>> actions;
        {
            std::lock_guard guard(desiredMutex);

            handles.reserve(desiredHandles.size() + 1);
            std::transform(desiredHandles.begin(), desiredHandles.end(),
                std::back_inserter(handles),
                [](auto&& h) { return duplicate_handle(h.get()); });
            // Add the bonus "changed" handle
            handles.emplace_back(duplicate_handle(changed.get()));

            actions = desiredActions;
        }

        auto count = static_cast<DWORD>(handles.size());
                        
        auto result = WaitForMultipleObjects(count,
                        handles.data()->addressof(), FALSE, INFINITE);
        auto index = result - WAIT_OBJECT_0;
        if (index == count - 1) {
            // the list changed. Loop back to update.
            continue;
        } else if (index < count - 1) {
            actions[index]();
        } else {
            // deal with unexpected result
            FAIL_FAST(); // (replace this with your favorite error recovery)
        }
    }
}

The waiting thread makes a copy of the desiredHandles and desiredActions, and adds the changed handle to the end so we will wake up if somebody changes the list. We operate on the copy so that any changes to desiredHandles and desiredActions that occur while we are waiting won’t affect us. Note that the copy in handles is done via Duplicate­Handle so that it operates on a separate set of handles. That way, if another thread closes a handle in desiredHandles, it won’t affect us.

void change_handle_list()
{
    std::lock_guard guard(desiredMutex);
    ⟦ make changes to desiredHandles and desiredActions ⟧
    SetEvent(changed.get());
}

Any time somebody wants to change the list of handles, they take the desiredMutex lock and can proceed to make whatever changes they want. These changes won’t affect the waiting thread because it is operating on duplicate handles. When finished, we set the changed event to wake up the waiting thread so it can pick up the new set of handles.

Right now, the purpose of the changed event is to wake up the blocking call, but we could also use it as a way to know whether we should update our captured handles. This allows us to reuse the handle array if there were no changes.

void waiting_thread()
{
    bool update = true;                        
    std::vector<wil::unique_handle> handles;   
    std::vector<std::function<void()>> actions;

    while (true)
    {
        if (std::exchange(update, false)) {
            std::lock_guard guard(desiredMutex);

            handles.clear();
            handles.reserve(desiredHandles.size() + 1);
            std::transform(desiredHandles.begin(), desiredHandles.end(),
                std::back_inserter(handles),
                [](auto&& h) { return duplicate_handle(h.get()); });
            // Add the bonus "changed" handle
            handles.emplace_back(duplicate_handle(changed.get()));

            actions = desiredActions;
        }

        auto count = static_cast<DWORD>(handles.size());
                        
        auto result = WaitForMultipleObjects(count,
                        handles.data()->get(), FALSE, INFINITE);
        auto index = result - WAIT_OBJECT_0;
        if (index == count - 1) {
            // the list changed. Loop back to update.
            update = true;
            continue;
        } else if (index < count - 1) {
            actions[index]();
        } else {
            // deal with unexpected result
            FAIL_FAST(); // (replace this with your favorite error recovery)
        }
    }
}

In this design, changes to the handle list are asynchronous. They don’t take effect immediately, because the waiting thread might be busy running an action. Instead, they take effect when the waiting thread gets around to making another copy of the desiredHandles vector and call Wait­For­Multiple­Objects again. This could be a problem: You ask to remove a handle, and then clean up the things that the handle depended on. But before the worker thread can process the removal, the handle is signaled. The result is that the worker thread calls your callback after you thought had told it to stop!

Next time, we’ll see what we can do to make the changes synchronous.

The post How do you add or remove a handle from an active <CODE>Wait­For­Multiple­Objects</CODE>? appeared first on The Old New Thing.

You can’t.

Even if you could, it meant that the return value of Msg­Wait­For­Multiple­Objects would not be useful. Suppose it returns to say “Handle number 2 was signaled.” You don’t know whether handle 2 was signaled before the handle list was updated, or whether it was signaled after. You don’t know whether it’s referring to the handle number 2 or the new one.

So maybe it’s not a bad thing that you can’t change the list of handles in an active Msg­Wait­For­Multiple­Objects, since the result wouldn’t be useful. But you can ask the thread to stop waiting, update its handle list, and then go back to waiting.

Since the thread is in a Msg­Wait­For­Multiple­Objects, it will wake if you send a message to a window that belongs to the thread. You can have a “handle management window” to receive these messages, say

#define WM_ADDHANDLE    (WM_USER+0) // wParam = index, lParam = handle
#define WM_REMOVEHANDLE (WM_USER+1) // wParam = index

The background thread could send one of these messages if it wanted to add or remove a handle, and the message procedure would perform the corresponding operation.

In reality, you probably need more information than just the handle; you also need to know what to do if that handle is signaled. The lParam is more likely to be a pointer to a structure containing the handle as well as instructions on what the handle means. Those instructions could take the form of a callback function, or it could just be a value from an enum. Pick the design that works for you.

Next time, we’ll look at the case where you don’t want to block the background thread, or if the waiting thread is waiting in Wait­For­Multiple­Objects so the message option is not available.

The post How do you add or remove a handle from an active <CODE>Msg­Wait­For­Multiple­Objects</CODE>? appeared first on The Old New Thing.

Yes, there were problems that were so bad that the program was effectively unfixable. The largest category of them were those which took various types of handles and figured out how to convert them to pointers, and then used those pointers to access (and sometimes modify!) the internal implementation details of those objects.¹ Not only did the implementation details change, but the mechanism they used to convert handles to pointers also stopped working because Windows 95 used a 32-bit heap for user interface and graphics objects, whereas Windows 3.1 used a 16-bit heap.

Sometimes the programs were kind enough to do strict version checks to confirm that they were running on a system whose internals they understood. Sometimes their version checks were themselves broken, like one program that assumed that if the Windows version is not 3.0, 3.1, or 2.1, then it must be 2.0!

There were other one-off problems, like programs which detoured APIs in a way that no longer worked, but the ones that dug into operating system internals were by far the most common.

¹ What’s particularly frustrating is the cases where the program did this to access internal implementation details, when the information they wanted was already exposed by a public and supported API.

The post Were there any Windows 3.1 programs that were so incompatible with Windows 95 that there was no point trying to patch them? appeared first on The Old New Thing.

    MIDL_INTERFACE("3127CA40-446E-11CE-8135-00AA004BB851")
    IErrorLog : public IUnknown
    {
    public:
        virtual HRESULT STDMETHODCALLTYPE AddError( // error here
            /* [in] */ __RPC__in LPCOLESTR pszPropName,
            /* [in] */ __RPC__in EXCEPINFO *pExcepInfo) = 0;
        
    };

The error message is

oaidl.h(5457,43): error C3927: '->': trailing return type is not allowed after a non-function declarator
oaidl.h(5457,43): error C3613: missing return type after '->' ('int' assumed)
oaidl.h(5457,43): error C3646: 'Log': unknown override specifier
oaidl.h(5457,43): error C2275: 'LPCOLESTR': expected an expression instead of a type
oaidl.h(5457,43): error C2146: syntax error: missing ')' before identifier 'pszPropName'
oaidl.h(5459,60): error C2238: unexpected token(s) preceding ';'

The compiler is seeing ghosts: It’s complaining about things that aren’t there, like -> and Log.

When you see the compiler reporting errors about things that aren’t in the code, you should suspect a macro, because macros can insert characters into code.

In this case, I suspected that there is a macro called AddError whose expansion includes the token ->.

The customer reported that they had no such macro.

I asked them to generate a preprocessor file for the code that isn’t compiling. That way, we can see what is being produced by the preprocessor before it goes into the part of the compiler that is complaining about the illegal use of ->. Is there really no -> there?

The customer reported back that, oops, they did indeed have a macro called AddError. Disabling the macro fixed the problem.

The compiler can at times be obtuse with its error messages, but as far as I know, it isn’t malicious. If it complains about a misused ->, then there is probably a -> that is being misused.

The post Learning to read C++ compiler errors: Illegal use of <TT>-></TT> when there is no <TT>-></TT> in sight appeared first on The Old New Thing.

Recall that Read­Directory­ChangesW and Find­First­Change­Notification are for detecting changes to information that would appear in a directory listing. Your program can perform a Find­First­File/Find­Next­File to cache a directory listing, and then use Read­Directory­ChangesW or Find­First­Change­Notification to be notified that the directory listing has changed, and you have to invalidate your cache.

But there are a lot of operations that don’t affect a directory listing.

For example, a program could open a file in the directory with last access time updates suppressed. (Or the volume might have last access time updates suppressed globally.) There is no change to the directory listing, so no event is signaled.

Functions like Read­Directory­ChangesW and Find­First­Change­Notification functions operate at the file system level, so the fundamental operations they see are things like “read” and “write”. They don’t know why somebody is reading or writing. All they know is that it’s happening.

If you are a video rental store, you can see that somebody rented a documentary about pigs. But you don’t know why they rented that movie. Maybe they’re doing a school report. Maybe they’re trying to make illegal copies of pig movies. Or maybe they simply like pigs.

If you are the file system, you see that somebody opened a file for reading and read the entire contents. Maybe they are loading the file into Notepad so they can edit it. Or maybe they are copying the file. You don’t know. Related: If you let people read a file, then they can copy it.

In theory, you could check, when a file is closed, whether all the write operations collectively combine to form file contents that match a collective set of read operations from another file. Or you could hash the file to see if it matches the hash of any other file.¹ But these extra steps would get expensive very quickly.

Indeed, we found during user research that a common way for users to copy files is to load them into an application, and then use Save As to save a copy somewhere else. In many cases, this “copy” is not byte-for-byte identical to the original, although it is functionally identical. (For example, it might have a different value for Total editing time.) Therefore, detecting copying by comparing file hashes is not always successful.²

If your goal is to detect files being “copied” (however you choose to define it), you’ll have to operate at another level. For example, you could use various data classification technologies to attach security labels to files and let the data classification software do the work of preventing files from crossing security levels. These technologies usually work best in conjunction with programs that have been updated to understand and enforce these data classification labels. (My guess is that they also use heuristics to detect and classify usage by legacy programs.)

¹ It would also generate false positives for files that are identical merely by coincidence. For example, every empty file would be flagged as a copy of every other empty file.

Windows 2000 Server had a feature called Single Instance Store which looked for identical files, but it operated only when the system was idle. It didn’t run during the copy operation. This feature was subsequently deprecated in favor of Data Deduplication, which looks both for identical files as well as identical blocks of files. Again, Data Deduplication runs during system idle time. It doesn’t run during the copy operation. The duplicate is detected only after the fact. (Note the terminology: It is a “duplicate” file, not a “copy”. Two files could be identical without one being a copy of the other.)

² And besides, even if the load-and-save method produces byte-for-byte identical files, somebody who wanted to avoid detection would just make a meaningless change to the document before saving it.

The post How can I use <CODE>Read­Directory­ChangesW</CODE> to know when someone is copying a file out of the directory? appeared first on The Old New Thing.