Bradley Grainger

How to use GitHub Pages on Windows

2011-09-07T00:00:00-07:00

Install

Install RubyInstaller. (I used version 1.9.2-p290.)
Install the RubyInstaller DevKit (be sure to customise the install path, e.g., to C:\RubyDevKit).
Install Jekyll: Run "Start Command Prompt with Ruby", then gem install jekyll
Install Python. (I used Python 2.7.2 Windows Installer.)
Install Python setuptools. (I used setuptools-0.6c11.win32-py2.7.)
Add C:\Python27\Scripts to your path.
Install pygments: easy_install pygments
Apply the patch (linked here) to prevent "Liquid error: bad file descriptor" being output when you use pygments: cd /c/Ruby192/lib/ruby/gems/1.9.1/gems/albino-1.3.3/lib; patch < 0001-albino-windows-refactor.patch

Set up the GitHub Pages Repository

Install msysGit.
Create a new repo as per the GitHub Pages documentation.
- An easy way to get started is to copy or fork an existing repo.
Write and save a post in the _posts folder.

Test

Run "Start Command Prompt with Ruby" and change to your pages repo's working copy.
Enter jekyll --auto --server to build your site and launch a web server that rebuilds your site every time you save in your editor.
Browse to http://localhost:4000 to see your site.

Publish

Create a GitHub repository named after your user name (e.g., bgrainger.github.com).
Push your local repository to GitHub.

Profiling JScript, Part 1

2004-11-01T00:00:00-08:00

Finishing my rewrite of the Logos Bible Software Home Page wasn't sufficiently appealing, so I started looking at JScript.dll in more detail. I decided to get a feel for the architecture of jscript.dll by examining the call stack for a simple property access on one of our COM objects. Placing a breakpoint in CLbxScriptUtil::get_Date (the function invoked by Application.ScriptUtil.Date in JScript) gave the following call stack:

ScriptUt.dll!CLbxScriptUtil::get_Date()  Line 158   C++
oleaut32.dll!_DispCallFunc@32()  + 0xc3
oleaut32.dll!CTypeInfo2::Invoke()  + 0x20c
oleaut32.dll!CTypeInfo2::Invoke()  + 0xc4
oleaut32.dll!CTypeInfo2::Invoke()  + 0xc4
ScriptUt.dll!ATL::IDispatchImpl::Invoke()  Line 4412    C++
jscript.dll!IDispatchInvoke()  + 0x6f
jscript.dll!InvokeDispatch()  + 0x72
jscript.dll!VAR::InvokeByName()  + 0x10c19
jscript.dll!CScriptRuntime::Run()  + 0xcd5d
jscript.dll!ScrFncObj::Call()  + 0x6a
jscript.dll!JsEval()  + 0xe2
...

jscript.dll!IDispatchInvoke() looked like an interesting function. Getting its decorated name out of jscript.dll and running it through UnDecorateSymbolName gave the following method signature: (Note: parameter types and names cleaned up by referring to the IDispatch::Invoke() documentation)

long __stdcall IDispatchInvoke(
  class CSession * pSession,
  struct IDispatch * pDispatch,
  DISPID dispIdMember,
  REFIID riid,
  LCID lcid,
  WORD wFlags,
  DISPPARAMS * pDispParams
  VARIANT * pVarResult,
  EXCEPINFO * pExcepInfo,
  unsigned int  * puArgErr
)

Although I can't prove it, I'm going to assume that this is a helper function used by JScript internally whenever IDispatch::Invoke needs to be called. A simple profiling exercise would be to log how many times this function is called. You can see that when it's calling our code, it immediately invokes ATL::IDispatchImpl (in the common header file atlcom.h). We could log all calls to Invoke by changing atlcom.h and recompiling almost every source file in our application. Or we could try to intercept the function at runtime and log accesses that way. Not only is the latter method a lot quicker (no recompilation necessary), but it greatly increases the chances of GPFs (since this would involve programatically changing the bytes of a running application). It seemed like the obvious choice.

I decided to write three functions: one to hook the IDispatchInvoke function, one to unhook it, and the hook function itself (which would just increment a counter). For ease of testing, I'd invoke these functions from a JScript-accessible COM method. I chose to make ScriptUtil.Application install the hook, ScriptUtil.Parent remove it, and ScriptUtil.Date log the current counter value.

While a “production” version of this code would need to handle different base addresses for JScript, it's more expedient to just hard-code the addresses of the code I want to change. The nop bytes that need to be changed to a long jump are at 0x75C6672E; the mov edi, edi that needs to be changed to a short jump is at 0x75C66733. We also need the address of our hook function (which is easy to get at runtime by using a function pointer). The fourth address needed is that of the five nop bytes in our hook function that we'll overwrite with a jump back to the third byte of JScript's IDispatchInvoke function.

The blog post about mov edi, edi that I mentioned in the last installment had a cryptic reference to something called “Detours”. A little googling showed that Detours is a Microsoft Research project that enables interception of arbitrary functions at runtime. It seems quite sophisticated, but assumes that the function being “detoured” doesn't have the mov edi, edi instruction at the beginning. As a result, Detours needs to copy the first bytes of the function to a temporary location, since it overwrites code that's essential for the function's correct operation. Secondly, while you can call the original function (through a “trampoline”), you need to supply all the original parameters (just like a normal function call). While this makes sense conceptually, it could lead to subtle bugs if the calling convention or parameter types weren't declared identically. I'm not currently using Detours, but did take a few ideas from their description of its implementation.

My implementation is fairly simple. I have a __declspec(naked) hook function that increments a global counter and has five nop bytes that are reserved for replacement with a long jump. A C++ helper class, CEnablePageWrite, calls FlushInstructionCache and VirtualProtect (in its constructor) to enable writing to the page containing the code to be modified; the destructor restores the original protection to the page. Consultation of the Intel Architecture Software Developer's Manual, Volume 2: Instruction Set Reference Manual gave the necessary bytes for the jmp opcodes. The hook installation function creates the short and long jumps in jscript.dll, and creates the long jump inside the hook function. The hook uninstallation function restores the jscript.dll bytes to what they were originally. CLbxScriptUtil::get_Date just logs the current value of the counter to the logfile.

All this code is running well on my machine; you're welcome to come over and take a look at details of the implementation or at its output.

Profiling JScript, Part 0

2004-10-30T00:00:00-07:00

As is my wont, I recently started looking at JScript.dll disassembly again and wondering about the possibility of implementing a JScript profiler. It seems that with XP SP2 I now have symbols for JScript.dll on my system, which makes it a lot easier to understand the code. (Symbols provide function names and parameter lists for those functions.) While reading the assembly listings, I noticed that most functions look like this:

mov     edi,edi
push    ebp
mov     ebp,esp
   ...
ret
nop
nop
nop
nop
nop

The second and third lines are standard x86 function prologue (setting up the stack), but I couldn't understand the apparently-useless mov edi, edi (or edi = edi in C++) instruction at the begining of every function. There's also a five-byte gap between functions (that seems to be always five bytes, and not a variable number, e.g., to force functions to align on 16-byte boundaries). A quick Google search turned up Ishai's blog post "Why does the compiler generate a MOV EDI, EDI instruction at the beginning of functions?". In short, this extraneous two-byte instruction (and the five bytes of nop (no operation, i.e., do nothing) at the end) are inserted to enable hot-patching of the application. The five nop bytes can be replaced with a "long jump" instruction (the jmp opcode, and a 32-bit address to jump to), while the two-byte mov edi, edi instruction can be replaced with a "short jump" instruction (the jmp opcode, and an 8-bit offset) that jumps to the long jump. But why use mov edi, edi instead of two nop instructions? If you are patching the application while it's running, it's theoretically possible that one thread could have executed the first nop but not the second. If your patch thread then changes both bytes (to the short jump), the thread that's about to read the second nop instruction will instead read the offset your patch just inserted, and attempt to interpret that offset as a valid x86 opcode. Bad things would almost certainly happen. mov edi, edi takes two bytes, and so it's impossible for the instruction pointer to be pointing at the second byte. This should enable us to hook certain functions in JScript.dll very easily. Since JScript.dll is loaded into our process's address space, we can change the bytes making up its code easily. We first get the address of our profiling function and overwrite the five nop bytes with a jump to that address. We then change the mov edi, edi instruction to jump to our long jump. Now, whenever that function is called, execution will quickly pass to our profiling function. We just have to make sure we return to the right instruction with the stack exactly as we found it. How hard could that be?

First VS2005 bug report

2004-07-28T00:00:00-07:00

Microsoft has recently launched a Product Feedback site, which includes a bug reporting section. This turns out to be a great place to report bugs they introduced in VS2002/2003 and still haven't fixed.

An example of this would be a subtle, insidious bug in compiler-generated code that was introduced with ATL 7.0 in Visual Studio 2002. This was the problem that caused LDLS to crash if Dragon Naturally Speaking was running. It's easy enough to work around, but it would be nice to have the auto-generated code be bug free. Here's my bug report; let's see what happens with it…

Update 2011-09-09: The bug is still tracked on Connect all these years later. And Microsoft fixed it!

Cracking a DLL, Part 2

2004-03-29T00:00:00-08:00

Introduction

In Part 1, I explained the motivation for cracking X.dll, and gave a brief look at what a bottom-up investigation of the DLL revealed. For a variety of reasons, that approach ultimately ended in failure. I tried again, this time with a top-down approach. (I also tried harder.)

Top-down Approach

Finding Differences

I decided to run the code twice, once with the current date, and again with the system time set to a month in the future. Any time I reached an "if" statement in the code, I would note whether the code took the if branch or the else branch. (Technically, I would note whether the program jumped to a different instruction or not each time it reached a conditional jump instruction. If statements are coded using the x86 conditional jump instructions: ja (jump if above), jae (jump if above or equal), je (jump if equal), jne (jump if not equal), etc. The "conditional" part of the jump tests the processor flags, which are set by a previous instruction (e.g., cmp (to compare two numbers), or test (to test a bit in a number)).)

The first time through, I would step through the CreateData function in X.dll, and not step into any functions it called. I would note the return value of each function, and if they returned something different, I'd run the program again (with both dates), stepping into that function to see where the execution differed that time.

Set Up

Firstly, I disassembled the entire DLL by running dumpbin /disasm:bytes X.dll > X.asm. This provided the assembly language instructions along with the equivalent machine code bytes. That would let me tie the disassembly I was seeing in the debugger back to the actual bytes of the DLL if I ever needed to make any modifications.

Debugging CreateData

I fired up the program and stepped into the CreateData function. This immediately dumped me into disassembly view, since no source code was available. I selected the instructions (from the current location to the first ret (return from function) statement) in the debugger window and copied it into TextPad. As I recognised standard assembly language conventions (set up function call frame, allocate local variables, call functions with parameters, etc.), I annotated the source code with what I thought it was doing. I also annotated the source with the code path that was taken.

Very quickly I found a difference in the behaviour of the program with the different dates. With a valid run date, the function at offset 0x1C6B4 (hereinafter called Init) would return 0x00D40BA8. In the future, this function would return 0. This value was stored in memory (in a global variable). The next function that was called would take a while to execute with a valid date (probably loading data from disk), but would throw an exception when called with a future date. The Init function obviously deserved more scrutiny.

Debugging Init

As with CreateData, I copied the source code for Init to TextPad and started annotating it as I stepped through. This time I found a point where the program would not follow a jae instruction when running with a valid date, but would jump when running with a date in the future:

call    20008E88     ; Function calls of mystery
call    2000289C     ;
call    2000530C     ;
cmp     edx, 0       ; First comparison -- edx=0 before and after bomb-date
jne     2001C7C0
cmp     eax, 1087h   ; eax=0x1085 before bomb date, eax=0x1089 after bomb date
jae     2001C7E6     ; jumps after bomb date, doesn't jump before
jmp     2001C7C2     ; program always jumps here (before bomb date)

The condition being tested was whether eax (an Intel x86 register, i.e., a variable) was greater than or equal to 4231. There were three function calls in a row immediately before this test; I decided to examine each in sequence.

When I stepped into the first function, I realised that I had seen this function before: it was the function that called GetLocalTime and validated the date, returning a floating point number. The second function that Init called was simple to figure out; it just converted that floating point number to an integer. (I had become confused about what was happening where with my bottom-up approach the day before; I now saw that these were two separate functions.) The third function was complicated and had lots of if/else statements, but the end result was that it divided one 64-bit integer (the number of days) by another 64-bit integer (the hardcoded value 9) and returned a 64-bit integer.

The cmp tests following those function calls were now simple to figure out. The first checked that the result of this division was less than 2³² (specifically, that the upper 32 bits of the result were 0). The second checked whether the lower 32 bits of the result were less than 4231. In short, X was checking that fewer than 4231 9-day periods had elapsed since 1 Jan 1900. (It appears that the programmer was trying to be slightly sneaky here. By dividing by 9, he changed the hardcoded value that he was checking against (4231) to something one might not be expecting. Indeed, at first look, I had no idea what 4231 referred to.) If the test succeeded, Init loaded the magic number 0x00D40BA8 out of memory and returned it, otherwise it returned 0.

Changing the DLL

The hardest part was deciding how to change the DLL. Do I extend the number of 9-day periods that are allowed? Should I skip over the whole date check, and just return the magic number? Perhaps just change the test so that it always jumps to the correct instruction, whether or not eax < 4231? Adding or modifying any jump instructions would require me to pull out a hex calculator and perform some subtraction, because the instruction to which you jump is specified as an offset from the current instruction (to save bytes). This sounded like hard work, so instead I decided to just comment out the whole jae instruction. I did this by overwriting the appropriate bytes in the file with 0x90, the opcode for nop (no operation; do nothing). The original source code looked like this (my annotations added after semicolons):

push    0            ; use the number 9 as the hardcoded parameter to DivInt64
push    9
call    20008E88     ; call GetCurrentTimeAsDouble
call    2000289C     ; call ConvertDoubleToInt64
call    2000530C     ; call DivInt64 (divide one 64-bit integer by another)
cmp     edx, 0       ; make sure upper 32 bits of quotient are 0
jne     2001C7C0
cmp     eax, 1087h   ; is quotient < 4231?
jae     2001C7E6     ; greater than or equal, jump to BadDate
jmp     2001C7C2     ; must be less than, jump to GoodDate

I edited the DLL in a binary editor, replacing the opcodes for the cmp and jae instructions (seven bytes total) with 0x90. This effectively comments out the comparison aginst 4231 and the jump if the value is greater, letting the program fall through to the unconditional jmp at address 2001C7BE:

push    0            ; use the number 9 as the hardcoded parameter to DivInt64
push    9
call    20008E88     ; call GetCurrentTimeAsDouble
call    2000289C     ; call ConvertDoubleToInt64
call    2000530C     ; call DivInt64 (divide one 64-bit integer by another)
cmp     edx, 0       ; make sure upper 32 bits of quotient are 0
jne     2001C7C0
nop                  ; do nothing
nop
nop
nop
nop
nop
nop
jmp     2001C7C2     ; always jump to GoodDate

It's not the most elegant fix in the world; I certainly changed much more than I needed to. A good one byte fix would have been to change the comparison against 0x1087 to 0x7f001087 (a date centuries in the future). But why be satisfied with changing one whole byte? We could change the number to 0x40001087, changing just one bit. I'll try even harder next time.

Cracking a DLL, Part 1

2004-03-26T00:00:00-08:00

Preface

At the last company meeting, it was revealed that the resident company "cr4x0r" had removed a bomb date from a DLL used in an upcoming product.

The company meeting was a bit light on technical detail, so (thanks to Jacob's prompting) I'm going to blog how it was done for any interested parties.

Background and Motivation

The specialised search engine is implemented by a library named X. Somehow, someone found out that X.dll would stop working after 1 April 2004.

Since this DLL is the core of the product, it was essential to modify it so it would continue to work. Not having the source code for the DLL, this was a non-trivial operation.

Possible Approaches

All that we have to work with is the machine code output from the Delphi compiler. Somewhere in the DLL must be a check of the current time against a bomb date. Maybe the bomb date is hard-coded in the DLL, maybe it is in the datafiles. Maybe the bomb date is the actual date, maybe it's the number of 100-nanosecond intervals since 1 Jan 1601. Searching for the bomb date itself would not be profitable; however, there must be some code with the following structure:

if (CurrentDate < BombDate) then
    proceed
else
    crash

There were two ways to go about finding this code (or the assembly language representation of it) in the DLL:

Bottom-up — find the code that gets the current time and examine the surrounding code in hopes of finding the check.
Top-down — debug the program one line at a time, with the system clock set before and after the bomb date, looking for differences in program flow (i.e., finding the if/else statement).

I elected to pursue a bottom-up approach first; it seemed that this would require less assembly language code to be examined.

Bottom-up Approach

Finding the Code That Gets the Current Time

At some point, X.dll has to ask the operating system what the current time is. Windows provides the functions GetSystemTime, GetSystemTimeAsFileTime and GetLocalTime for this purpose. If we were lucky, we would find a direct call to one of these functions in X.dll. If we were unlucky, X would descramble the desired function name at run time (so that you couldn't find the name in the binary) and dynamically link to kernel32.dll in order to call the function.

I fired up Dependency Walker and, sure enough, X.dll contained a reference to GetLocalTime (in kernel32.dll).

(This reference is stored so the OS loader can fix up addresses when the DLL is loaded. Because DLLs can be loaded at different addresses in memory (and because kernel32.dll is different in different versions of Windows), the actual location of the GetLocalTime function can't be hardcoded in X.dll. Instead, a relocation table is used. The relocation table basically looks like this (well, in pseudo-code):

...
void GetLocalTimeStub(LPSYSTEMTIME lpSystemTime) { goto GetLocalTime; }
void GetSystemTimeStub(LPSYSTEMTIME lpSystemTime) { goto GetSystemTime; }
...

The DLL's compiled code calls the GetLocalTimeStub stub function instead of the real kernel32.dll function. At load time, the OS walks the relocation table and fills in the actual address of the functions in other DLLs. Since those DLLs have already been loaded into memory, the real addresses of the functions are known.) I suddenly realised that I didn't know how to read relocation tables. I wanted to put a breakpoint on GetLocalTimeStub so that I could debug the program at the point where it called this function, but I didn't know where this function was located in X.dll. However, Dependency Walker told me that GetLocalTime is located at an offset of 0x15A68 in kernel32.dll. Combine that with Process Explorer telling me that LDLS.exe loaded kernel32.dll at 0x77E60000 and we know that the code for GetLocalTime is located at 0x77E75A68.

In the debugger, it's as simple as pulling up the Disassembly window, entering that address in the address box, and setting a breakpoint on that byte of code. (With debugging symbols installed, the debugger will even helpfully put "_GetLocalTime@4:" above that line of assembly language.)

Analysing the Code That Gets the Current Time

We now invoked the CreateData function in X.dll. The debugger immediately stopped on a call to GetLocalTime. The call stack window revealed that we were four functions deep inside X.dll. The function that was calling GetLocalTime was the first thing to investigate. It allocated 32 bytes on the stack (the size of a SYSTEMTIME struct), passed the address of that struct to GetLocalTime, then called another function with the st.wYear, st.wMonth, and st.wDay values. This in turn called a third function which appeared to check the validity of the date. It ensured that the year was between 1 and 9999, that the month was between 1 and 12, and that the day of the month was between 1 and the maximum number of days in that month (having already called a fourth function to determine if the current year was a leap year). Since we just got the date from the OS, it seemed unlikely that it would be invalid, but it can't hurt to check, I guess. It then calculated (based on the year/month/day values) the number of days since 1 Jan 1900 (just making a guess at this epoch start date, actually, but it's probably somewhere around that time) and converted that to a floating point number. A similar validity-checking procedure was carried out with the hour, minute, and second values, and then they were converted to a fraction between 0 and 1 and added to that floating point number. The program then immediately converted that floating point number to an integer, stripping off the fractional part. We continued debugging the code, making notes on what we thought the program was currently doing, until we realised that I was hopelessly lost. We hadn't written down the execution path of the program in great detail, and I didn't know where I was in the program. The date check might already have been done. Or the date might just have been stored in memory, and the check was going to be performed elsewhere. At that moment, D.F. called and invited us to see a movie. It seemed like we had reached a good stopping place.

Continue reading Part 2.

ATL proxy/stub DllCanUnloadNow bug

2004-03-26T00:00:00-08:00

All our of COM server DLLs have proxy/stub code, which is used when data needs to be marshalled across process boundaries. For example, LibSys.exe loads the actual code for the LbxMetadataCache object, but LDLS.exe calls this code. Proxy code is loaded into the LDLS.exe process, which forwards the call to the stub code in LibSys.exe, which then calls the actual LbxMetadataCache code. By and large, this proxy/stub code is created automatically for you. With ATL, you have the choice of building the proxy/stub as a separate DLL, or merging the proxy/stub code with your main COM DLL. We chose the latter route, since it halves the number of DLLs we need to ship. Whichever route you choose, the ATL COM server project wizard generates the appropriate code for you. Part of the generated code is the DllCanUnloadNow function. This method returns S_OK (aka true) if it's safe for Windows to unload the DLL, or S_FALSE (aka false) if it's not. Your implementation should keep track of whether any clients are using the objects in your DLL, and only return S_OK if you're not being used any more. If the proxy/stub code is merged into your DLL, your DllCanUnloadNow method has two things to keep track of:

Are any clients using the COM objects defined in this DLL?
Are any clients using proxy code to communicate with COM objects loaded into another process?

You can only unload if the answer to both of these questions is "no".

ATL 7.0 changes the auto-generated code for this function significantly. Here is the code from ATL 6.0:

STDAPI DllCanUnloadNow()
{
#ifdef _MERGE_PROXYSTUB
    // proxy/stub code
    if (PrxDllCanUnloadNow() != S_OK)
        return S_FALSE;
#endif

    // check module lock count
    return (_Module.GetLockCount() == 0) ? S_OK : S_FALSE;
}

Here is the code generated by ATL 7.0:

STDAPI DllCanUnloadNow()
{
#ifdef _MERGE_PROXYSTUB
    // proxy/stub code
    HRESULT hr = PrxDllCanUnloadNow();
    if (FAILED(hr))
        return hr;
#endif

    // check module lock count
    return _AtlModule.DllCanUnloadNow();
}

The problem is that S_FALSE is still a success value, and FAILED(hr) will be false. So even if the proxy says "Don't unload me; clients are still using me as a proxy", the DLL will be unloaded if no clients are using the DLL directly.

So, how did we find this out? If Dragon NaturallySpeaking is running in the background, the system becomes a lot more aggressive about checking for DLLs that are no longer being used. It will very frequently query metcache.dll to find out whether it can be unloaded. The proxy says, "I'm still in use, don't unload me". DllCanUnloadNow ignores that result and asks if any clients are using the COM object directly. This is false, so the DLL tells Windows to unload it. Windows then unloads the DLL. Fractions of a second later, LDLS calls the LbxMetadataCache proxy code, which no longer exists in memory! This, as you might expect, is no good. An exception occurs, and error tips start popping up on the user's screen.

I fixed the problem by changing:

    HRESULT hr = PrxDllCanUnloadNow();
    if (FAILED(hr))
        return hr;

to:

    HRESULT hr = PrxDllCanUnloadNow();
    if (hr != S_OK)
        return hr;

This causes the DLL to stay in memory if any clients are using it as a proxy. Only when no one is using it as a proxy and no one is using the COM objects directly will the DLL be unloaded.

Code Comments

2003-09-24T00:00:00-07:00

One of our interns left some, uh, interesting comments in the code. Here are some I found just going through one source file. (Comments are word-wrapped here. In the original source, each comment was on just one line (up to 500 characters long).)

// the "i" on the end of the [regular expression] won't work if it's in a string format (which it must be to get that i on the end...AAH, CATCH-22, MY ANCIENT NEMESIS)

// This is necessary because this main window actually modifies the start date passed in from the properties dialog (global variables at work, oh ya), so when we pass it back in if we decide to open the properties dialog again once we've looked at our calendar, the start date is bad and causes the properties dialog to crash! I fixed this by simply keeping track of what the properties dialog returns separately from the gStartDate variable.

// Get all the g00dz from the source window

// we are about to perfom some serious loadification

// MAN, I HOPE THEY DON'T EVER DECIDE TO HAVE A DIFFERENT NUMBER OF DAYS IN THE WEEK THAN SEVEN!@#!@3~4112!@

// Collect the latest and greatest saving xml from the peerless and altogether unmatched CreateSaveXML function [ editor's note: I deleted that function. ]

// append the goodness to the main document's XML container for subsequent savification

static in C++

2003-08-28T00:00:00-07:00

What does the C/C++ keyword static mean? When used inside a function, e.g.:

int f(int a)
{
    static int b = 1;
    return a + ++b;
}

it means that a single instance of this variable is allocated for the lifetime of the program. In terms of storage, it's very similar to a global, but its scope is restricted to that function.

C++ inherits this sense of static for class definitions. If a variable is declared as static in a class, there is one instance of that variable (again, allocated very similarly to a global) that is shared between all instances of that class. Declaring the variable as protected or private controls its scope.

There is another meaning of static. In C, it was the opposite of extern; it meant that a variable had internal linkage semantics and wasn't visible from any other translation unit. (That is, you could only access it from within the file in which it was defined.) This use of static is deprecated in C++; use an unnamed namespace instead:

namespace
{
    int s_variable = 1;
}

Since this use is deprecated, static should only be used inside a class or function definition. This rule would prohibit, for example, declaring static constants (or worse yet, variables) in a header file. The problem with this is that every source file that includes that header file gets its own copy of the constant or variable. There are no errors about duplicate symbols, because the linker respects the old C meaning of internal linkage. In the case of constants, barring linker optimisation, the constant would appear multiple times in the binary, instead of just once. In the case of variables, although it might initially appear that the functions in different source files are sharing the variable, each source file has its own copy and changes to the variable won't be seen between different source files.

Changes to Asynchronous Pluggable Protocol Handlers

2003-08-27T00:00:00-07:00

After dealing with yet another customer (through tech support) who was having problems directly attributable to a full/corrupted Temporary Internet Files folder, I decided to change the implementation of our lbxfile: and lbxres: protocols. (We're alpha testing, after all :-).)

Current Approach

Even though we can open a stream directly on an embedded file, our protocols work by saving the data to a temp file (so that the embedded file isn't locked while IE is accessing it). Until now, this temp file has been stored in the Temporary Internet Files folder (TIF).

Problems with Current Approach

There appear to be numerous problems with using the UrlCache functions to store this data in TIF:

The size of this folder seems to grow without limit, even though there is a limit set in the IE > Tools > Internet Options > General > Temporary Internet Files/Settings dialog. D.D. just reported deleting 10GB of files. Just accessing a few reports can write a megabyte of data to the cache (Common.js is 184KB).
The files are never cleaned up properly. This is a minor point, but many megabytes of source code are easily accessible in TIF.
When the cache gets full, new files are deleted without warning. A symptom of this is when View > Source in IE opens an empty Notepad window. The file is successfully written to the cache, but is deleted immediately before the filename of the cache file is returned to the client. This manifests itself as users' command bars failing to load (because the images for the buttons are inaccessible).
Filling the cache breaks other components/programs. IE's View > Source is one example, but Libronix Update also uses the cache to download files. Way back in the original Alpha days, we had a machine that couldn't launch Libronix Update because the .lbxupd file itself was deleted before Libronix Update could open it.
Other weird errors. NewVerseTemplate.xslt mysteriously failing to load was almost certainly a TIF problem; changing .Item to .OpenStream (in BibleUtility.js) fixed it.

New Approach

It still seems advantageous to write the data to a temporary file (to avoid locking problems). We're now writing it to a location we control: a subfolder of the user's temp folder. lbxfile: writes files to %TEMP%\lbxfile and lbxres: writes files to %TEMP%\lbxres.

Problems with New Approach

There is still the thorny issue of file lifetime management. The protocols create an object (that implements IStream) that reads from the temp file. It would be nice if we could delete the temp file as soon as this object is released. However, as part of the protocol operations, we call ReportProgress(BINDSTATUS_CACHEFILENAMEAVAILABLE), which lets the caller know the name of our temp file. If we omit this call, IE will still load dialogs and reports properly, but UrlDownloadToCacheFile or UrlDownloadToFile, which are used by CommandBars and TrayIcons, will fail. However, by reporting the filename, we're implicitly granting clients the right to access that file later on. Our current solution is to have a worker thread that runs in the background. Every five minutes it wakes up, scans the temp folder and deletes files that were last accessed over ten minutes ago. The folders get cleaned when the thread is started (when FileProt.dll or ResProt.dll is loaded) but are left intact when the app shuts down (since there isn't enough time to set a DllBeingUnloaded event then wait for the thread to wake up and delete everything when the DLL is being unloaded). (If we wanted maximum purity, the app itself could delete the known folders after waiting for threads to shut down.) The result of this is that at most fifteen minutes worth of data (probably several megabytes) could be left in the user's temp folder in between runs of the program.

So?

This will be in the release after 2.1 Alpha 3, which should probably be an alpha (instead of a beta) release just because of this change. The code's still fairly untested under heavy use (I zipped the shell and some addins on my system, and it worked well enough), so it will be good to have some reports that it's working correctly. There probably won't be any amazing success stories with it (maybe one or two users who were having weird problems will be fixed?), but we shouldn't see any more problems that I currently attribute to TIF (unless there's a bug in the new implementation).