Entropy Overload

On the Difficulty of setting the Windows 7 Desktop Wallpaper

2009-11-10T16:59:00.007Z

You need to be a software engineer to precisely and correctly set the Windows background in Windows 7.

I have a multi-monitor setup, and I use a very particular method for laying out windows on my secondary screen: I have them cascaded, from the top-left corner of the screen towards the bottom right, such that a little of the bottom-left corner of every window is visible on-screen at all times, like this:

The corners form a mechanism for switching between windows based on spatial layout. Certain windows, such as terminals, browsers, email apps, etc. have "natural" dimensions. Browsers generally like to be page-shaped; terminal windows are quite squat; documentation browsers are somewhere in between, with a treeview on the left and a content pane on the right. These natural shapes guide their spatial positioning. So, all my terminal windows end up cascaded one after the other on at the top left of the screen, followed by Firefox's downloads window, then my bug tracker client, then Thunderbird, then MSDN documentation, with Firefox at the bottom-right, filling the full height of the screen.

Switching between applications based on their spatial layout turns out to usually be more efficient than almost any other scheme, particularly when one's hands aren't on the keyboard. Even alt-tab doesn't distinguish very clearly between different instances of e.g. a terminal app, when in fact each one may be logged in via ssh to different machines and thus it matters very much which one I select.

This scheme works well enough, but the bottom-left corners can get a little ragged. To make aligning the corners easier, I have a special wallpaper which consists of a line carefully placed so that it shows up on the bottom-left corner of my second monitor. When the wallpaper mode is set to "tile", a wallpaper which is big enough to cover the entire desktop area will show up at the expected offset, and everything works out OK.

This wallpaper, a 16-colour bitmap, is so small when compressed that I can embed it here uuencoded directly:

begin 644 wallpaper2.bmp.bz2
M0EIH.3%!629365%3W2<`1PST[.0U0`!`0!`"`0!``!``0`!$``,(L`#9`0JC
M4#1HR::%&C(&C3(T"E4TAM30`&=?M"6=)(E^T$2WA+9$L*JDF-E$NG?'AI55
M0/>6^L(CB*T(550`Q-.DJ".(B"O,*JH`8F(WU0$<@^P((\>[801S_$;<:YG'
:LR**SC<GG'.8JJ@!M,13_%W)%.%"045/=)P`
`
end

If you save that to a file with a '.uue' extension, many common archivers, such as WinZip or WinRar, will be able to open it up. Here's what the meat of it looks like, anyway:

However, it turns out, if you try to set this background using the "Personalization" Control Panel applet in Windows 7, it does some helpful "conversions" on it: it appears to round-trip the damn bitmap through JPG, resulting in artefacts. I've enhanced a zoom of the line after Windows has butchered it:

The line looks unclean, and in particular, it ends up with wave of intensity throughout its length, which is quite distracting and not the desired effect.

Things can get worse though; if you try to use Windows Photo Viewer to set the background using the original bitmap, it will actually resize the thing down to fit the dimensions of your primary monitor (!!!), all but zooming the line out of existence. This drove me up the wall for about 20 minutes, as I tried to figure out where my line had gone when I first transitioned from XP to Windows 7.

The easiest way I currently know to correctly set the Windows 7 desktop wallpaper is to write a program that uses the SystemParametersInfo function directly, to ensure the bits get through unmolested.

I have a lot more Windows 7 annoyances where this came from, but this one cost me quite a bit of time, so it sticks out. My verdict on Windows 7 is pretty mixed. Many applications, particularly games, are incompatible, and overall I dislike the shell compared to XP, especially Windows Explorer and the Vista-esque Start Menu. About the best things I can say in favour of the move is having less concern about future compatibility, and having a 64-bit mode that mostly works, despite all the horrific hacks going on beneath the covers to make it happen. Oh, and I'm running with UAC disabled. I can only take so many access denied errors as I poke around the system before I crack - and command-line apps generally just give you the error, rather than popping up an elevation prompt.

Bad week

2009-10-24T23:27:00.002+01:00

It's been a bad week. My main hard drive crashed last Monday or so, while today my scooter is gone, presumed stolen. On the bright side, the final component of my new PC arrived and I was able to get Windows 7 up and running.

The new machine has an i7 920, so things like %NUMBER_OF_PROCESSORS% returns 8 - it's a quad core with 2x hyperthreading. Having so many logical cores has some downsides: maxed-out single-threaded processes only take up 12.5% CPU, so it can look like the machine is not particularly busy even though a process is working as hard as it can. Other specs include 12GB RAM, AMD 5870 graphics, 80GB Intel SSD and 1TB Samsung F3.

Maximizing the value of the small boot drive has been interesting. Windows 7 all by itself takes up a good 18G of space, what with the side-by-side (SxS) feature keeping lots of versions of the same APIs, and a big hiberfil.sys in the root - I moved the page file to the big drive. Because of the acute space constraints, I've been strongly motivated to keep almost all the junk that apps usually dump into the file system off the boot drive. By and large, I achieve that by creating directories on the 1TB drive and pointing to them with junctions on the boot drive. That way, apps think they are installing onto the C drive but they don't steal too much space. My prior drive - the one that died - was a 10K WD Raptor, so I've had some experience with this, and I wrote a script to automate the process:

#!/bin/bash

. _die

function usage
{
    echo "usage: $(basename $0) <directory>..."
    echo "Replace directories with junctions pointing to /bulk, after moving contents."
    exit 1
}

test -n "$1" || usage

for f in "$@"; do
    
    test -d "$f" || die "'$f' isn't a directory"
    
    dest="$(abs-path "$f")"
    dest=/bulk/mathom/"${dest//\//_}"
    
    test -d "$dest" && die "'$dest' already exists"
    
    # Try to copy stuff over; if an error, delete target and give up.
    cp -r "$f" "$dest" || {
        rm -rf "$dest"
        die "couldn't copy '$f' to '$dest'"
    }
    
    # Rename source, in case something can't be deleted.
    aside="$(move-aside "$f")" || die "failed in move-aside"
    
    # Create junction.
    junction "$dest" "$f" || die "failed to create junction"
    
    # Remove the aside.
    rm -rf "$aside" || {
        echo "warning: couldn't delete '$aside'"
    }
done

This script uses a number of little wrapper utilities I wrote - at this point I really ought to be thinking of putting them in an online repository. Here's a summary so you can follow along:

abs-path: Canonicalizes and makes an absolute path by using cygpath twice, once to Windows format and once again back to Unix format, and prepending $PWD if necessary.
_die: Defines a die function which prints out appropriate error and exits.
move-aside: Renames a file or directory so that a failed attempt at creating a junction can be rolled back. Also helps with locked files - generally these can be copied and renamed, but not deleted. The new name is printed out on standard output.
junction: This wraps SysInternals junction to create a junction in the TARGET LINK_NAME argument order familiar from ln, and permits using Cygwin paths.

With this script, and the big drive mounted at /bulk with a directory called mathom in its root, any given directory tree can be moved across readily enough. However, things in Windows 7 x64 are slightly more complicated.

Some of the directories I moved over to /bulk/mathom included Program Files, Program Files (x86), and ProgramData. Windows 7 itself has some junctions set up (the rats nest here is a proper horrorshow, I should document it at some point just for my own use, if nothing else). For example, "ProgramData\Application Data" points at ProgramData itself, and a simple cmd /c dir will fall into the infinite recursion and print out errors when the file path gets too long. So, in moving these directories across, I had to examine them for existing junctions and replace these as necessary. The Cygwin tools of Unix heritage are more robust against cycles in the file system hierarchy - for example, find keeps track of inodes to avoid infinite recursion.

All this checking of existing junctions wasn't sufficient to get everything working nicely, though. The permissions generally also need tweaking, particularly for folders that get modified by installers. These generally perform operations using either the NT SERVICE\TrustedInstaller or NT AUTHORITY\SYSTEM accounts - and sometimes both, in a dance of setup.exe and msiexec.exe. So, by process of checking, trial and error, using SysInternals Process Monitor to track installation failures, I put together a new script to ensure Program Files, ProgramData and folders of their ilk have the right set of permissions:

#!/bin/bash

function usage
{
    echo "usage: $(basename $0) <file/directory>..."
    echo "Resets files and directories to Windows 7 general install permissions."
    echo "Essentialy this is: SYSTEM/TrustedInstaller/Admin: full; AuthUsers: Modify; Users: Read"
    exit 1
}

test -e "$1" || usage

for arg; do
    
    # First, take ownership so we can definitely set all ACLs.
    # /F <file>; /R - recurse; /D <Y/N> - take ownership if no directory listing
    takeown /F "$(cygpath -w "$arg")" /R /D Y > /dev/null
    
    # Reset ACLs recursively to inherit from parent
    icacls "$(cygpath -w "$arg")" /reset /t /c /q > /dev/null
    
    # Set appropriate ACL on root, to progate recursively
    icacls "$(cygpath -w "$arg")" \
        /grant "Administrators:(OI)(CI)F" \
        /grant "SYSTEM:(OI)(CI)F" \
        /grant 'NT SERVICE\TrustedInstaller:(OI)(CI)F' \
        /grant 'Authenticated Users:(OI)(CI)M' \
        /grant 'Users:(OI)(CI)RX' \
        /inheritance:r /c /q > /dev/null
    
done

The logic is applied in three phases:

Take ownership, with takeown
Reset permissions recursively so that they inherit from parent
Set ACL on the root, so children will inherit from it, but make sure the root itself doesn't inherit permissions.

I tried for a while to get it all into one icacls invocation, but I couldn't get the two effects working together: enabling inheritance in the children, but disabling it in the root. This is something you can do in one "Apply" step using the Security tab in Explorer, though it's quite tedious picking out all the users and setting the appropriate checkboxes.

As an aside, the Security tab is quite confused by junctions; when permissions have been inherited from a directory on a different drive, because the current file / directory is in a subtree pointed to by a junction, the Security tab will search in vain through the parents looking for what provided the given inherited permission. Sometimes it gets lucky, and finds a random parent that coincidentally has the same permission; otherwise, it bails out with "Parent Object" as the indicated "Inherited From" column. But my proper list of Windows 7 issues will have to wait until I really feel the need to rant, which will no doubt happen sooner or later.

I should add that all the drastic file system surgery I write about here was performed with UAC effectively out of the picture, and all Cygwin tools running with the administrative privilege bits set. It's been my experience that command-line apps don't pop up the required UAC prompt; instead, they fail with an access denied message, which is rather vexing when you're trying to automate things.

Public interest rule of thumb

2009-07-15T13:15:00.002+01:00

TechCrunch is having a crisis of conscience over what they'll do with internal Twitter documents they've received.

My rule of thumb is pretty simple. The documents are stolen. Unless the documents reveal wrongdoing greater than stealing, such that the public interest served in publishing is greater than the private harm, then it's unethical to publish. Idle musings about "they are going to be published somewhere on the Internet" is specious; the morality of one's actions don't depend on whether someone else is "doing it", much less everyone.

If TechCrunch goes ahead and publishes these documents, it'll only further cement my already pretty low opinion of Arrington. (I'm not a habitual reader of TechCrunch, or otherwise I'd waste away over all the food I'd be vomiting up.)

StackOverflow and religious language in programming

2009-07-14T19:45:00.002+01:00

I was reading Why I Don’t Listen to the StackOverflow Podcast any More, and in reading the comments, I found it interesting to see the amount of religious language.

#21 (sbohlen) I’d have to agree that they do indeed appear successful, but success / failure isn’t a boolean [...] The world is full of people who succeed due to all kinds of factors including dumb-luck, [...]

This reminds me intensely of "the race is not always to the swift" etc. Eccl 9:11.

I changed adopters to believers to accentuate the effect:

"#16: [...] I have also experienced that the non-[believers] aren’t (gnerally) non-[believers] out of choice but are instead non-[believers] largely out of ignorance and inexperience — they just don’t know what they don’t know [...]"

#16: "the [types] who aren’t interested in self-improvement; there’s not a damned thing I can suggest to help these people — I cannot MAKE someone want to better themselves, that’s gotta come from within"

#16: "I [...] think that Jeff and Joel are indeed preaching [...] they are (IMO) abdicating the unofficial professional responsibility that comes with having a pulpit from which to preach."

#14: "I am a strong believer"

#13: "@sbohlen. I fully agree with the sentiment that “up and running” is not a valid metric"

I'm not going to go on with this, though I could - the about page for the blog is particularly ripe for further examples of religious language - but I do want to make a further point.

Could it be that latent religious feelings are responsible for a good portion of the minor angst on display in these comments? Could it be that people are unconsciously annoyed, not with StackOverflow's success, but rather its success even in the face of quite mild contempt at the unit-testing / SOLID religion?

My own position: I'm a "believer" in evolution and the market. Neither evolution nor the market have any moral content (they can't say what "ought" to be), and both only measure success by effects rather than processes. In a phrase, "up and running" is not just a valid metric - it's the only metric, until it stops running. If SO continues to thrive, and doesn't die under the weight of its lack of unit testing (which seems to be the logical end result of the beliefs of the believers in the comments), where then is the truth in the proclamations of doom from the prophets of the true religion?

I don't think unit-testing is the only way to avoid long-term bit-rot. I think it's a good way, but not the only way; moreover, I don't think unit testing is especially useful in the very early stages of a project, where the design changes often and rewriting unit tests becomes a disincentive to larger refactoring and redesign. I think every one of the concepts embodied in S.O.L.I.D. is debatable in certain scenarios (except perhaps LSP).

In my experience, premature abstraction has been responsible for a similar number of ills to premature optimization: abstractions chosen at the wrong boundaries, for the wrong reasons, because designers thought they could foretell the future and anticipate where changes would come. Thus I am wary of merchants of abstractions and patterns, principles and practices. I'll use them when appropriate, but never with religious zeal.

WPF / VS2010 Font rendering: blurryville

2009-05-22T13:03:00.008+01:00

I downloaded and installed VS2010 Beta 1 to see how it's looking. It's not pleasant though - it makes me feel like I'm looking at the screen through a layer of coke-bottle plastic:

Compared to the old rendering, where my preferred fixed-width code / terminal font, Dina, is correctly chosen, as opposed to falling back to what looks like Courier New:

The new rendering hurts my eyes (they keep trying to focus more, but it doesn't help!) and makes VS2010 Beta unusable, as far as I'm concerned.

The "recommended solution" for text clarity issues in WPF is "use the largest font size possible". That's a bit of a cop-out for the working programmer, where displaying as much code on-screen as is comfortably legible is usually the best option.

FWIW, I registered a feedback item on this.

Sharing a printer, the hard way: writing your own spooler

2009-04-28T13:47:00.006+01:00

I have an ultra-cheap HP Color Laserjet 1600 printer, the kind that costs less than a full set of replacement toner cartridges, and the ones that are shipped in the box are only 20% full. From the driver publisher info it appears the I/O protocol logic of the printer was provided by a company called Zenographics, which appears to now be either a part of or owned by Marvell. It works well enough when you have it directly connected to a machine via the USB cable, but sharing it is a nightmare.

In particular, the drivers are very unstable when targeting shared printers. Especially problematic is printing from Vista to XP, or XP to XP, or XP to Vista. With some combinations, the print spooler crashes; with others, it simply hangs, and the spooler service needs to be restarted to get the document printing at all. The folks on the HP forums seem to imply (a) that this is by design, and (b) sharing the printer effectively is impossible. I find this hard to swallow; since the printer works when directly connected, there is no valid reason why printing across the network should not also work, since the physical printer doesn't come into the equation. Bits are bits; if the driver can't consistently send some bits across the network, either there's some atrocious QA going on inside Zenographics, or worse, malice.

I had despaired of ever getting it working nicely as a small workgroup printer, but I saw a faint glimmer of hope when setting up my home NAS, based on Solaris. CUPS is available for Solaris, and there's a Foomatic printer driver available for this specific model as well.

It turns out that I couldn't get CUPS configured with this printer and driver setup. CUPS uses a web page interface for configuration, but it was desperately slow, and every time I tried to print a test page, it informed me that the printer had just gone offline. I could see all the background processes it had started up to print, and the printer device at /dev/printers/0 was indeed opened for writing, but it never got anywhere.

But that didn't matter. The foo2hp driver provided the only real tools I needed: a PostScript to ZjStream (the printer protocol) filter, which simply reads the raw PostScript and outputs the raw print data. Windows ships with a generic PostScript driver; I use the PScript5-based "HP Color Laserjet PS". A simple shell pipeline from source to printer device would do.

To share the printer, I needed some kind of queue for buffering. I decided on a simple filesystem-based approach, which seems to me to be in the Unix tradition. I created a share on my Solaris box specifically for files to be printed. Windows clients can use the aforementioned generic printer driver with a FILE: target, and place the output in this shared directory. The spooler script I wrote polls this directory periodically, and attempts to print whatever it finds.

The printing logic is pretty simple, at its core:

foo2hp2600-wrapper -p 9 input.ps > /dev/printers/0

The actual code is a mite more sophisticated, of course; it does logging, moves documents to be printed into a directory categorized by date and time, etc. And in future it can be extended to other document types than PS, e.g. printing PDFs or photos simply by copying them to a directory.

Another advantage of this approach is that I can shell out to ps2pdf to create an archival version of the printed document. The PDF is usually far smaller than the PS and is viewable from Windows without installing a PostScript viewer. This archival version is useful if I ever need to print it again. Consider online flight checkins, for example - some websites (such as EasyJet) prevent printing the boarding pass more than three times, which is a little inconvenient if you're having printer trouble.

On the downside, printer options that can't be expressed directly in the PostScript with the generic driver can't be configured. For example, the way I have things configured I'm limited to printing monochrome on A4 by default. In practice this isn't really a problem, since the vast majority of print jobs fit this template; I could create other spooler directories for different defaults.

I expect that with enough research, I'll be able to get CUPS configured correctly, or to get Samba to look like a PostScript printer that dumps the raw network data into my spooler directory. However, I'm pretty happy with my hack for now...

ZFS/Solaris as a NAS

2009-03-22T03:24:00.004Z

I've finally got a semblance of a Solaris system up and running with a nice fat ZFS storage pool. It was neither trivial nor pleasant, however.

Here's the executive summary of what I learned:

Don't use OpenSolaris 2008.11; use at least NexentaCore 2 instead; others may work.
Don't use an Intel or Realtek network adapter; if you must, use Realtek with an older version of the gani driver.
Don't use Solaris's built-in CIFS implementation, smb/server, aka hereafter "cifs" (to contrast with "samba").
Don't let ZFS to default to 128KB max block size unless all your files are big (greater than about 2MB) or tiny (less than about 64KB).

OpenSolaris 2008.11, the version I tried, is not production ready unless you have extremely specific hardware and quite narrow requirements. I finally ended up using NexentaCore Unstable v2.0 Beta 2.

The prime requirements for a working NAS are:

Lots of attachment points for disks
Good physical network connectivity
Good network protocol implementation
Good filesystem implementation

There is no one of these that I didn't have issues with.

The first one, attachment points, was a nice-to-know gotcha that it took me a while to figure out. My motherboard (an old Abit AW9D-MAX) has 7 internal SATA attachment points, as well as 1 PATA channel for two extra PATA devices. This being insufficient, I procured an extra PCIe two-port SATA adapter. I was up and running fine with 5 disks attached (1 boot drive and 4 drives in a raidz ZFS pool - effectively RAID 5 without the need for NVRAM), and in preparation for creating my second raidz vdev (virtual device in ZFS lingo), I started attaching more drives, one by one so that I could label them (and thereby know which one failed when it fails - which it will, eventually). When I had 7 drives attached, the OS occasionally failed to boot, but would reliably hang upon execution of the 'format' command (the handiest way of figuring out the drive device name under Solaris).

The gotcha, however, is that the BIOS defaults to IDE emulation mode, which, for this particular BIOS, supports a maximum of 6 devices. It packs the first 2 hard drives into pretend-PATA (complete with "master" and "slave") even if they aren't actually PATA, and then labels the next 4 drives as SATA 1 through 4. I had to change the BIOS's mode to AHCI to get it to support more drives. Luckily, my boot drive is an old PATA one, so it didn't need compatibility to stay in the same place (master on the first PATA channel).

Physical network connectivity was a far, far harder task, and one that still isn't quite complete. My motherboard has two physical 1Gb Ethernet adapters built in. Unfortunately, OpenSolaris defaults to a driver it calls rge for the chipset (RTL8111/8168B PCIe), but this driver has big issues, that I suspect are related somehow to duplex operations. Big transfers using big buffers using cifs (the Windows-compatible protocol Solaris ships with) will work with reasonable performance (30MB/sec or so), but streaming video (small reads that expect low latency) performs abysmally. Even worse, at non-deterministically random times (as far as I could ascertain), the rge driver would fall into a mode where transfers were extremely slow, as in less than 1MB/sec slow. I took to keeping a ssh session open to the Solaris machine and simply echoing '.' back and forth (echo '.' loop piped to ssh cat -), and mysteriously enough, so long as there was small back and forth traffic, transfer performance rocketed back up again.

This being unsatisfactory, I tried to use the gani driver. I never could get gani-2.6.3 to work with my chipset, however. I didn't know if it was a problem with the driver, a conflict with my chipset, or my poor configuration skills with Solaris. All I know is that I tried every technique I could find, up to and including patching driver_aliases and running sys-unconfig to start things off from a fresh basis.

So, I bought an Intel 1000/PRO Gigabit ethernet adapter. Folks online, in response to reports of rge not working correctly, seemed to say that the Intel adapter "just worked", but I should have dug deeper... The driver Solaris uses for this is called e1000g. The version of this driver shipping in OpenSolaris 2008.11 doesn't work. It drops packets. Simply pinging the Solaris machine from the outside shows packet loss exceeding 5%. With a ssh session open to Solaris, simply holding down '.' on the keyboard and watching the character getting echoed in the terminal is sufficient to demonstrate the hiccups: the pauses are visible and disturbing. It actually affects typing in ssh, almost as if one was at the end of a dodgy Internet connection, perhaps some half-baked cafe wifi link in Marrakesh. A recursive diff of about 100MB of data across about 1000 files using cifs took a whole 45 minutes. A trivial test using dd to copy a 640KB blob of data using a 64KB buffer size took about 15 seconds and dd reported transfer rate of 45KB/sec.

It was this network connectivity that prompted me to start trying out other distributions. NexentaCore 2 was first in line. It defaults to the gani driver for the Realtek adapter, but an earlier version, 2.6.2. This works, but only at 100Mbps; try as I might, I can't convince it to go higher. (Windows running on the same hardware, same patch cable and same switch, never, ever dropped down to 100Mbps - something I can't say about the other Windows machines I have here.) Alos with significant importance, NexentaCore doesn't ship with a GUI (so no need to disable it), and does ship with apt-get and a reasonable selection of packages, including my preferred Linux/Unix editor, joe (very similar to old Borland DOS IDE text editors). I detest vi - I've never used a program that gleefully punished more heavily every transgression of the user.

Anyhow, that's how I got one reasonably-working 100Mbps connection out of three 1Gbps adapters, four driver versions and two operating system installs.

Next, the network protocol. Solaris defaults to using something that is colloquially called cifs, although the usual identifiers used when e.g. enabling and disabling with svcadm are smb/server and smb/client. I can't say much about smb/client - I'm sure it works well enough, but I don't have much use for it - but smb/server barely works at all. It's implemented as a kernel module, which unfortunately means it has odd and painful limitations compared to how user-level programs view the file system. In particular, only one ZFS filesystem can be navigated from a cifs share, significantly harming the usefulness of creating lots of ZFS filesystems. A ZFS file system mounted inside an existing filesystem, where the mountpoint is visible from a cifs share, will show up as an empty directory to clients, whereas it appears correctly mounted locally on the Solaris box.

Another limitation is symbolic links. By apparent design, cifs prohibits what Samba calls "wide links": symbolic links that resolve to locations that are outside the share's hard-linked subtree. Such symbolic links look much like dead links from Windows, i.e. the text little files they are implemented as. Samba defaults to "wide links" on for performance reasons if nothing else.

A final limitation is hard links created from Windows using CreateHardLink. Cygwin 'ln' ultimately uses this API. For whatever reason (and I didn't dig too deeply, like investigating Wireshark traces), Cygwin determines that 'ln' isn't supported on cifs shares and falls back to copying instead. Cygwin 'ln' works correctly on Windows shares, however, and it also works correctly over Samba.

The verdict: cifs isn't worth it. Needs more time to bake. Use Samba instead.

Finally, ZFS itself. ZFS is the big draw, and the reason I chose to use Solaris. Userspace implementations are available in Linux via FUSE, and also in FreeBSD, but I had bad performance experiences of NTFS-3G/FUSE in Linux, and FreeBSD's implementation sounded dangerously non-production ready. ZFS largely works as advertised. The primary limitation is the inflexibility in removing drives from pools. Raidz vdevs can't have drives removed at all. A word of warning, though: ZFS has what I can only consider a bad bug for files in the 128KB..2MB range. If you have a file of about 129KB, depending on how it was created, it ought to be using up about 100% more space than it should be. In some largeish directories of files I was seeing wastage of about 35% (as measured by 'du' versus 'du --apparent-size'), whereas NTFS on Windows had wastage in the region of 5%. Paring back the ZFS default block size (dynamically settable on ZFS - yay! - but only affects subsequent file operations) to 8KB or 16KB improves things immensely, but still not quite as space-efficient as NTFS.

In conclusion, ZFS/Solaris as a NAS can work well with very carefully chosen hardware and select software configuration. If you get it working better than I describe herein, be very careful what you touch, and whatever you do, don't upgrade your zpools until you're sure that whatever step forward you take is better than what you had previously.

OpenSolaris, ZFS, Dvorak and VI

2009-03-06T06:14:00.002Z

Been experimenting with OpenSolaris to try out ZFS. OpenSolaris is not a pleasant experience even after using an average Linux.

Its termcap and terminfo databases don't include complete information for xterm as implemented by the very gnome-terminal it defaults to, much less gnome-terminal itself; nor does it include any entries for rxvt derivatives (so I have to export TERM=xterm upon ssh in from Cygwin). Thus, there is much fun with Del, Home, End, PgUp, PgDn etc. brokenness: Ctrl+A/E instead of Home/End, Ctrl+D instead of Del, f/b in less rather than PgUp/PgDn, etc. The terminfo DB is spartan even by comparison with Cygwin, and termcap rather, eh, antique:

solaris$ find /usr/share/lib/terminfo/ | wc
    1716    1716   58553
cygwin$ find /usr/share/terminfo/ | wc
   2544    2544   79257

solaris$ grep 1982 /usr/share/lib/termcap
# From research!ikeya!rob Tue Aug 31 23:41 EDT 1982
# From jwb Wed Mar 31 13:25:09 1982 remote from ihuxp
# Extensive changes to c108 by arpavax:eric Feb 1982

Also, there is apparently no text-mode Dvorak layout shipped with OpenSolaris. I managed to patch something rudimentary together reverse engineering the contents of /usr/share/lib/keytables and applying with loadkeys. Within X, xmodmap can load an appropriate translation, but getting that working before gdm comes up for the login and password screen is tedious. Getting a vnc server session running gdm is easier.

Finally, I have to resort to using vi to edit basic files. Vi is a line editor masquerading as a text editor. Now I feel like I'm living in the 80s. The early 80s.

SecuROM and SysInternals tools

2009-02-04T18:34:00.004Z

If you're like me, you like visibility into the workings of your OS, which processes are running, what modules are loaded and from where and whether they are signed or not, what files are open, what files are being touched and what registry entries are being frobbed.

I use the SysInternals tools for getting increased visibility into these things. In particular, I use Process Explorer (procexp) alongside Task Manager (the two aren't complete overlaps for functionality), with File Monitor (filemon) and Registry Monitor (regmon) on XP and Process Monitor (procmon) on Vista. Filemon and regmon are more focused to their specific tasks and have easier to configure filters than procmon, so I usually prefer them, but they are not compatible with Vista.

Any time I see an unexpected filesystem locking error, such as a failure in Windows Explorer to delete a folder, I can search for the culprit in procexp's handle view. Often the culprit is Windows Explorer itself - perhaps caused by a shell extension like Winzip or Winrar - and I can force closed the handle right there from the procexp interface itself.

I have a set of "verification columns" configured procexp that makes visible the Company, Description and Verified Signer columns for processes and modules (DLLs). With Options | Verify Image Signatures turned on, this makes for easy checking for suspicious DLLs and kernel drivers (modules loaded into the System process) loaded in the system.

Procmon supports a whole lot more analysis than the easier to use tools like filemon and regmon, but the most important distinct feature is probably the stack trace on every file and registry operation. Double-clicking an event in the procmon view pops up a dialog that has a Stack tab, wherein the stack trace is displayed. When Windows symbols are correctly configured, procmon will call out to windbg.dll and show symbolic information such as function name and offset for each code address in the stack trace, up to and including downloading the symbols PDB if necessary.

Since I don't think I've done it already on this blog, I'll outline the sequence of steps required to get symbols correctly configured.

Download and install WinDbg. WinDbg is a useful tool in its own right for low-level Windows debugging, and particularly for .NET debugging when using the SOS command extension DLL.
Choose a location - i.e. an empty directory - for the symbol cache on your system. Once chosen, create an environment variable called _NT_SYMBOL_PATH with the value "SRV*[symbol-cache-dir]*http://msdl.microsoft.com/download/symbols" (without quotes), but replace [symbol-cache-dir] with the path to your selected directory.
Now, the SysInternals tools can be configured with these settings. Procmon and Procexp both have Configure Symbols dialogs in their Options menus. Set the DbgHelp.dll path to the DbgHelp.dll from the installation location of WinDbg, and set the "Symbol paths" textbox to the value of the _NT_SYMBOL_PATH environment variable created earlier.

However, the reason I wrote this post is not merely to laud the SysInternals tools and the work of Mark Russinovich. The fact is that SecuROM, a DRM solution used in many games these days, has panic attacks when it sees evidence of SysInternals tools being used. I don't really understand what SecuROM's justification for its conniptions could be, since monitoring the process with the tools could only help in an initial cracking attempt, yet game cracks appear to come out no slower than they ever did. All it seems to do is annoy honest users - and in this case, developers. Not only does SecuROM want you to exit the applications in question, but it also wants you to reboot the entire system, ignoring the fact that you might have concurrent long-lived tasks that don't merit interruption by a mere game

Well, I couldn't let that stand. I created a tool to enable me to simply exit the SysInternals utilities in question, rather than having to reboot the machine. The key to SecuROM's detection of the tools is in the communication mechanism between the SysInternals tools and their kernel driver counterparts, in particular the named devices in the Windows Object Manager namespace, visible using the WinObj tool. It turns out that merely by changing the ACL (access control list) on the device object to a single deny-all ACE (access control entry) is sufficient to fool SecuROM from thinking the device is no longer loaded - though SecuROM also looks for top-level windows with particular window classes, so the utilities do have to be exited as well.

So, I wrote a simple Delphi console application, called cacls_driver, to modify the ACL of a driver device. A key set of libraries that made this easy was the JEDI Win32 API library, in particular the units JwaWinType, JwaWinBase, JwaWinNT, JwaNative and JwaSddl. I've appended the tool's source code to this entry.

Using the tool is simple enough. The most important thing to know is the name of the driver one is changing the permissions of. For the versions of filemon and regmon I have installed, the correct driver names are filemon701 and regmon701 respectively. I figured this out by running the aforementioned WinObj and looking in the 'GLOBAL??' folder for devices with names starting with filemon and regmon. For the original Crysis, SecuROM even complained about Process Explorer, so I had to include procexp110 (IIRC) as well. SecuROM support, on the other hand, recommended that I upgrade Process Explorer, which of course used an incrementally different device name (procexp111) and thereby avoided the SecuROM sanction.

Here's an example of the tool in use hiding the regmon and filemon driver links ($ is my command-line prompt, and # are comments):

# This simply shows the current ACL
$ cacls_driver -s filemon701 regmon701
filemon701 :: D:(A;;CCRC;;;WD)(A;;CCSDRCWDWO;;;SY)(A;;CCSDRCWDWO;;;BA)(A;;CCRC;;;RC)
regmon701 :: D:(A;;CCRC;;;WD)(A;;CCSDRCWDWO;;;SY)(A;;CCSDRCWDWO;;;BA)(A;;CCRC;;;RC)

# This disables the ACL. There's no output from this command.
$ cacls_driver -d filemon701 regmon701

# Verifying that the ACL has been set.
$ cacls_driver -s filemon701 regmon701
filemon701 :: D:
regmon701 :: D:

These ACLs having been applied - and ensuring the tools in question aren't still running - I'm free to run my games without the odious requirement for a reboot.

Source code follows...

program cacls_driver;

{$APPTYPE CONSOLE}

uses
  SysUtils, JwaWinType, JwaWinBase, JwaWinNT, JwaNative, JwaSddl, Generics.Collections;

type
  TOperation = (opNone, opApply, opShow);

  TOptions = class
  private
    FOperation: TOperation;
    FDrivers: TList<string>;
    FDacl: string;
  public
    constructor Create;
    destructor Destroy; override;
    property Operation: TOperation read FOperation write FOperation;
    property Drivers: TList<string> read FDrivers;
    property Dacl: string read FDacl write FDacl;
  end;

function ParseOptions: TOptions;
const
  // http://msdn.microsoft.com/en-us/library/cc230374.aspx
  // These were the defaults on my system (WinXP SP3)
  // SID tokens (last two letters in ACEs):
  // SY = local system; BA = builtin administrators; WD = everyone; 
  // RC = restricted code
  // Permissions:
  // CC = create child; RC = read control; SD = delete; WD = write DAC;
  // WO = write owner
  DefaultDacl = 'D:(A;;CCRC;;;WD)(A;;CCSDRCWDWO;;;SY)(A;;CCSDRCWDWO;;;BA)(A;;CCRC;;;RC)';
  NullDacl = 'D:';
var
  i: Integer;
  opt: string;

  procedure Usage;
  begin
    Writeln(Format('usage: %s <command> <driver...>', [ParamStr(0)]));
    Writeln('Modify discretionary ACL for the given global namespace driver symlinks.');
    Writeln('  -d       Disable access (deny-all ACL)');
    Writeln('  -e       Enable access (apply default ACL: ', DefaultDacl, ')');
    Writeln('  -s       Show ACL for the given drivers');
    Writeln('  -a <acl> Apply an explicit ACL');
  end;
  
  procedure SetOpOnly(Op: TOperation);
  begin
    if Result.Operation <> opNone then
      raise Exception.Create('only one of -s, -d or -e allowed');
    Result.Operation := Op;
  end;

  procedure SetOp(Op: TOperation; const ADacl: string);
  begin
    SetOpOnly(Op);
    Result.Dacl := ADacl;
  end;
  
  procedure SetOpArg(Op: TOperation);
  begin
    Inc(i);
    if i > ParamCount then
      raise Exception.Create('missing argument to option ' + ParamStr(i - 1));
    SetOp(Op, ParamStr(i));
  end;
  
begin
  Result := TOptions.Create;
  try
    i := 1;
    if ParamCount = 0 then
    begin
      Usage;
      Exit;
    end;
    
    while i <= ParamCount do
    begin
      opt := ParamStr(i);
      case opt[1] of
        '/', '-':
        begin
          if Length(opt) > 2 then
            raise Exception.Create('invalid option');
          case opt[2] of
            's': SetOpOnly(opShow);
            'd': SetOp(opApply, NullDacl);
            'e': SetOp(opApply, DefaultDacl);
            'a': SetOpArg(opApply);
            '?', 'h': Usage;
          else
            raise Exception.Create('invalid option');
          end;
        end;
      else
        Result.Drivers.Add(opt);
      end;
      Inc(i);
    end;
  except
    Result.Free;
    raise;
  end;
end;

{ TOptions }

constructor TOptions.Create;
begin
  FDrivers := TList<string>.Create;
end;

destructor TOptions.Destroy;
begin
  FDrivers.Free;
  inherited;
end;

procedure EnablePrivilege(Process: THandle; const Name: string);
var
  privs: TTokenPrivileges;
begin
  FillChar(privs, SizeOf(privs), 0);
  privs.PrivilegeCount := 1;
  privs.Privileges[0].Attributes := SE_PRIVILEGE_ENABLED;
  Win32Check(LookupPrivilegeValue(nil, PChar(Name),
    privs.Privileges[0].Luid));
  Win32Check(
    AdjustTokenPrivileges(Process, False, @privs, SizeOf(privs), nil, nil));
end;

procedure ElevatePrivileges;
var
  process: THandle;
begin
  Win32Check(OpenProcessToken(GetCurrentProcess(), TOKEN_ALL_ACCESS, process));
  try
    EnablePrivilege(process, SE_SECURITY_NAME);
    EnablePrivilege(process, SE_DEBUG_NAME);
  finally
    CloseHandle(process);
  end;
end;

function NtErrorMessage(Status: TNTStatus): string;
var
  buf: PChar;
  h: THandle;
begin
  h := LoadLibrary('ntdll.dll');
  try
    FormatMessage(
      FORMAT_MESSAGE_ALLOCATE_BUFFER or FORMAT_MESSAGE_FROM_SYSTEM or
        FORMAT_MESSAGE_FROM_HMODULE,
      Pointer(h),
      Status,
      0,
      PChar(@buf),
      0,
      nil);
    try
      Result := TrimRight(buf);
    finally
      LocalFree(Cardinal(buf));
    end;
  finally
    CloseHandle(h);
  end;
end;

procedure NtCheck(Status: TNTStatus);
begin
  if Status <> 0 then
    raise EOSError.Create(NtErrorMessage(Status));
end;

procedure WithDriver(const Name: string; Mask: TAccessMask; const Proc: TProc<THandle>);
var
  path: string;
  driverName: TUnicodeString;
  driverAttr: TObjectAttributes;
  handle: THandle;
begin
  path := '\GLOBAL??\' + Name;
  RtlInitUnicodeString(@driverName, PChar(path));
  try
    FillChar(driverAttr, SizeOf(driverAttr), 0);
    driverAttr.Length := SizeOf(driverAttr);
    driverAttr.ObjectName := @driverName;
    
    NtCheck(NtOpenSymbolicLinkObject(@handle, Mask, @driverAttr));
  finally
    RtlFreeUnicodeString(@driverName);
  end;
  try
    Proc(handle);
  finally
    NtClose(handle);
  end;
end;

function GetKernelObjectDacl(Handle: THandle): string;
var
  len: Cardinal;
  desc: PSecurityDescriptor;
  buf: PChar;
begin
  GetKernelObjectSecurity(handle, DACL_SECURITY_INFORMATION, nil, 0, len);
  desc := AllocMem(len);
  try
    Win32Check(
      GetKernelObjectSecurity(handle, DACL_SECURITY_INFORMATION, desc, len, len));

    Win32Check(ConvertSecurityDescriptorToStringSecurityDescriptor(desc,
      SDDL_REVISION_1, DACL_SECURITY_INFORMATION, buf, nil));
    try
      Result := buf;
    finally
      LocalFree(Cardinal(buf));
    end;
  finally
    FreeMem(desc);
  end;
end;

procedure SetKernelObjectDacl(Handle: THandle; const Dacl: string);
var
  secDesc: PSecurityDescriptor;
begin
  Win32Check(ConvertStringSecurityDescriptorToSecurityDescriptor(PChar(Dacl), 
    SDDL_REVISION_1, secDesc, nil));
  try
    Win32Check(SetKernelObjectSecurity(Handle, DACL_SECURITY_INFORMATION, secDesc));
  finally
    LocalFree(Cardinal(secDesc));
  end;
end;

procedure ApplyToDriverHandle(Drivers: TList<string>; const Proc: TProc<string,THandle>);
var
  driver: string;
begin
  ElevatePrivileges;
  
  for driver in Drivers do
    WithDriver(driver, MAXIMUM_ALLOWED,
      procedure(Handle: THandle)
      begin
        try
          Proc(driver, Handle);
        except
          on e: Exception do
            Writeln(Format('failed on "%s": %s', [driver, e.Message]));
        end;
      end);
end;

procedure DoShow(Drivers: TList<string>);
begin
  ApplyToDriverHandle(Drivers,
    procedure(Driver: string; Handle: THandle)
    begin
      Writeln(Format('%s :: %s', [driver, GetKernelObjectDacl(Handle)]));
    end);
end;

procedure DoApply(Drivers: TList<string>; const Dacl: string);
begin
  ApplyToDriverHandle(Drivers,
    procedure(Driver: string; Handle: THandle)
    begin
      SetKernelObjectDacl(Handle, Dacl);
    end);
end;

begin
  try
    with ParseOptions do
      try
        case Operation of
          opShow: DoShow(Drivers);
          opApply: DoApply(Drivers, Dacl);
        end;
      finally
        Free;
      end;
  except
    on E: EOSError do
      Writeln('error: ', e.Message);
    on E: Exception do
      Writeln(e.Message);
  end;
end.

Irony

2009-01-30T15:22:00.003Z

I just received two magazines through my letterbox. This is the first one, addressed to a previous occupant of my home, is from the UK Ministry of Defense:

The second is my weekly subscription:

Awesome sentence fragment

2009-01-22T12:15:00.002Z

Best phrase I've read in some time: an army of over-promoted, ideologically vetted homunculi

Jeff is Wrong and don't listen to him

2009-01-15T17:18:00.005Z

I rarely (I hope) post specific corrections to misleading information out there on the web, not least because there's so much of it and it's usually a futile effort, but Jeff Atwood's latest post really rubbed me up the wrong way.

In his post, Jeff writes some unidiomatic C and bewails its ugliness and pain etc.:

b1 = (double *)malloc(m*sizeof(double));

By way of comparison, he then writes some C# (I expect that's what it's supposed to be) that doesn't actually use the GC, since it's not allocated on the heap:

Double b1;

(Of course, if it's not C#, and is in fact supposed to be e.g. Java, it's still not an example since it hasn't been initialized.)

Now Jeff's core message (at least at the start of the post) that GC is a Good Thing, I'm all in favour of. I strongly believe that manual memory allocation has good reasons to be used only in about 5% (or less) of programming tasks, usually restricted to things like operating systems and embedded devices. I consider well-defined, trivially provably correct zoned allocation to be a good 80% of the way to GC, so I would include good uses of that in the 95% case.

Jeff's final point, however, about "disposal anxiety" is where he casually reveals that he doesn't pay much attention to a very important issue: disposal of resources. He mocks a particular piece of code that has at least the core concept right:

sqlConnection.Close();
sqlConnection.Dispose();
sqlConnection = null;

Two-thirds of this code is redundant - the last line is always so unless sqlConnection is read later in the same routine, while either of the first two would do for resource disposal. This wouldn't be so bad but for Jeff saying:

Personally, I view explicit disposal as more of an optimization than anything else, but it can be a pretty important optimization on a heavily loaded webserver, or a performance intensive desktop application plowing through gigabytes of data.

Disposal of resources in a long-running application using GC is not a performance issue. It's a correctness issue.

Garbage collection knows about memory. With most collectors, GC is only invoked when the GC is asked for more memory than is immediately available, taking specific tuning parameters into account. In other words, the GC is only sensitive to memory pressure. It doesn't know about resources that it sees only as pointer-sized handles, it doesn't know how much they "cost", and indeed those resources might be on a different machine or even spread across many different machines.

More critically, garbage collection gets its performance surplus over and above manual garbage collection by not collecting until as late as reasonably possible, where "reasonable" is usually a function of how much free memory is available without swapping to disk. The long-run-average optimal collector for a program that has a machine all to itself won't collect at all until every last remaining byte of RAM has been allocated, which may of course take some time. (Added to clarify: this is a theoretical optimum, and not how most GCs act in practice. They collect much sooner e.g. the youngest generation may fit in L2 cache and so be very fast to collect.)

Precise tracing garbage collectors work somewhat paradoxically not by collecting garbage, but by collecting live objects. The "garbage" is everything left over after all the live objects have been collected. The more garbage as a fraction of live objects there is, the cheaper, proportionally, it has been to collect. (Added to clarify: this means that the collection of garbage is amortized; any amount of garbage costs the same to collect, providing the set of live objects is held constant.) This is how GCs outperform manual memory allocation on average and with sufficient free memory. Ideally, the GC never runs at all, and program termination cleans up.

With this insight under your belt, it should be clear that expecting the GC to clean up resources is to be ignoring one of the key benefits of GC. Not only that, but you shouldn't be expecting the GC to finalize your objects at all. If your resources must be disposed of - and almost all resources should, e.g. TCP sockets, file handles, etc. - then you need to take care of that yourself, and deterministically. Leaving e.g. file handles to be disposed of by the GC is opening up the program (and possibly the user) to odd non-deterministic failures when they find files on disk are still locked, even though the program should have closed them.

Implementing user-defined copy-on-write data structures in Delphi

2009-01-13T18:02:00.003Z

There was some discussion about user-defined class operators in the Delphi non-technical forum. Some folks seem to be asking for operators for heap-allocated classes, but in the absence of either GC or reference counting, these are problematic.

I suggested that records combined with interfaces can get much of the benefit of both worlds: user-defined operators on the record, life-time management via interface reference counting, and arbitrary data sizing through the interface being a heap reference.

To demonstrate the idea through example, I have written up a quick copy-on-write array type that has a user-defined '+' operator that concatenates two lists. Let me build it up from the bottom. First, I've defined a generic dynamic array type to avoid the pitfalls of nominal typing that Delphi has inherited from Pascal (a common newbie mistake, particularly with the syntactic ambiguity with open arrays in parameter lists):

type
  TArray<T> = array of T;

Next up is the interface. Using an interface for communicating between the record and the heap value means that I get reference counting, and thus lifetime management, for free.

  ICowArrayData<T> = interface
    function GetLength: Integer;
    function MutableClone: ICowArrayData<T>;
    function GetItem(Index: Integer): T;
    procedure SetItem(Index: Integer; const Value: T);
    function ToArray: TArray<T>;
  end;

  TCowArrayData<T> = class(TInterfacedObject, ICowArrayData<T>)
  private
    FData: TArray<T>;
  public
    constructor Create(const Data: TArray<T>);
    function GetLength: Integer;
    function MutableClone: ICowArrayData<T>;
    function GetItem(Index: Integer): T;
    procedure SetItem(Index: Integer; const Value: T);
    function ToArray: TArray<T>;
  end;

The only "interesting" method really is MutableClone, which I'll be calling to verify that I have a unique instance before I perform any mutating operations. This is necessary to get copy-on-write behaviour. The implementation class doesn't add anything beyond a constructor, as all communication will be through the interface.

Here's the actual record, and the only type that the user of this bundle should concern themselves with:

  TCowArray<T> = record
  private
    FData: ICowArrayData<T>;
    function GetItems(Index: Integer): T;
    procedure SetItems(Index: Integer; const Value: T);
    function GetLength: Integer;
  public
    constructor Create(const Data: TArray<T>); overload;
    constructor Create(const Data: array of T); overload;
    property Items[Index: Integer]: T read GetItems write SetItems; default;
    property Length: Integer read GetLength;
    function ToArray: TArray<T>;

    class operator Add(const Left, Right: TCowArray<T>): TCowArray<T>;
  end;

It adds a nice default Items property so that values can be indexed like arrays, exposes a Length property much like .NET, and adds in the Add operator to prove the point.

The implementation of most the methods are pretty routine, but a few are important. First up, because interfaces are managed types, they get zero-initialized by default. Thus, we can't expect the record's FData field to be initialized. We can take a leaf out of Delphi's built-in array (and string) types, and let zero-length arrays be represented by a nil FData. That affects the implementation of the record methods, e.g.:

function TCowArray<T>.GetLength: Integer;
begin
  if FData = nil then
    Exit(0); // nil => zero-length
  Result := FData.GetLength;
end;

Another important method is the implementation of the copy-on-write itself. The only mutating method on this type is SetItems, and here it is:

procedure TCowArray<T>.SetItems(Index: Integer; const Value: T);
begin
  FData := FData.MutableClone;
  FData.SetItem(Index, Value);
end;

This isn't particularly efficient, but it does make sure that we have a unique reference when required. The implementation of MutableClone is simple enough too:

function TCowArrayData<T>.MutableClone: ICowArrayData<T>;
begin
  if RefCount = 1 then
    Exit(Self);
  Result := TCowArrayData<T>.Create(ToArray);
end;

Using the inherited RefCount from TInterfacedObject, we can determine safely if we are only referred to by our caller. In a race scenario, where RefCount might e.g. spuriously be 2, there won't be any correctness problem if we duplicate the array anyway.

The implementation of the '+' operator is pretty trivial too:

class operator TCowArray<T>.Add(const Left, Right: TCowArray<T>): TCowArray<T>;
var
  resultArr: TArray<T>;
  i: Integer;
begin
  SetLength(resultArr, Left.Length + Right.Length);
  for i := 0 to Left.Length - 1 do
    resultArr[i] := Left[i];
  for i := 0 to Right.Length - 1 do
    resultArr[Left.Length + i] := Right[i];
  Result := TCowArray<T>.Create(resultArr);
end;

Finally, let's look at the program body itself, testing these types:

procedure WriteArray(const Msg: string; Arr: TCowArray<Integer>);
var
  i: Integer;
begin
  Write(Msg, ':');
  for i := 0 to Arr.Length - 1 do
    Write(' ', Arr[i]);
  Writeln;
end;

var
  x, y: TCowArray<Integer>;
begin
  try
    x := TCowArray<Integer>.Create([1, 2, 3]);
    y := x;

    Writeln('Starting out, both x and y refer to same instance data');
    WriteArray('x', x);
    WriteArray('y', y);

    Writeln('Modifying x; note that y doesn''t change:');
    x[1] := 42;
    WriteArray('x', x);
    WriteArray('y', y);

    // Add operator as concatenation
    Writeln('Concatenation:');
    y := x + y;
    WriteArray('x', x);
    WriteArray('y', y);
    
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
end.

The output does indeed show that the array is copied on writes, and that concatenation works as expected:

Starting out, both x and y refer to same instance data
x: 1 2 3
y: 1 2 3
Modifying x; note that y doesn't change:
x: 1 42 3
y: 1 2 3
Concatenation:
x: 1 42 3
y: 1 42 3 1 2 3

As an addendum, let me add the full source for reference. Note that due to some bugs in the current compiler's generics implementation for dynamic arrays, the code won't work if it's part of a unit - it needs to be in a single whole for now, unfortunately.

{$apptype console}
uses SysUtils;

type
  TArray<T> = array of T;
  
  ICowArrayData<T> = interface
    function GetLength: Integer;
    function MutableClone: ICowArrayData<T>;
    function GetItem(Index: Integer): T;
    procedure SetItem(Index: Integer; const Value: T);
    function ToArray: TArray<T>;
  end;
  
  TCowArrayData<T> = class(TInterfacedObject, ICowArrayData<T>)
  private
    FData: TArray<T>;
  public
    constructor Create(const Data: TArray<T>);
    function GetLength: Integer;
    function MutableClone: ICowArrayData<T>;
    function GetItem(Index: Integer): T;
    procedure SetItem(Index: Integer; const Value: T);
    function ToArray: TArray<T>;
  end;
  
  TCowArray<T> = record
  private
    FData: ICowArrayData<T>;
    function GetItems(Index: Integer): T;
    procedure SetItems(Index: Integer; const Value: T);
    function GetLength: Integer;
  public
    constructor Create(const Data: TArray<T>); overload;
    constructor Create(const Data: array of T); overload;
    property Items[Index: Integer]: T read GetItems write SetItems; default;
    property Length: Integer read GetLength;
    function ToArray: TArray<T>;

    class operator Add(const Left, Right: TCowArray<T>): TCowArray<T>;
  end;

{ TCowArray<T> }

class operator TCowArray<T>.Add(const Left, Right: TCowArray<T>): TCowArray<T>;
var
  resultArr: TArray<T>;
  i: Integer;
begin
  SetLength(resultArr, Left.Length + Right.Length);
  for i := 0 to Left.Length - 1 do
    resultArr[i] := Left[i];
  for i := 0 to Right.Length - 1 do
    resultArr[Left.Length + i] := Right[i];
  Result := TCowArray<T>.Create(resultArr);
end;

constructor TCowArray<T>.Create(const Data: TArray<T>);
begin
  if Data = nil then
    FData := nil
  else
    FData := TCowArrayData<T>.Create(Data);
end;

constructor TCowArray<T>.Create(const Data: array of T);
var
  arr: TArray<T>;
  i: Integer;
begin
  if System.Length(Data) = 0 then
    FData := nil
  else
  begin
    SetLength(arr, System.Length(Data));
    for i := 0 to System.Length(Data) - 1 do
      arr[i] := Data[i];
    FData := TCowArrayData<T>.Create(arr);
  end;
end;

function TCowArray<T>.GetItems(Index: Integer): T;
begin
  Result := FData.GetItem(Index);
end;

function TCowArray<T>.GetLength: Integer;
begin
  if FData = nil then
    Exit(0);
  Result := FData.GetLength;
end;

procedure TCowArray<T>.SetItems(Index: Integer; const Value: T);
begin
  FData := FData.MutableClone;
  FData.SetItem(Index, Value);
end;

function TCowArray<T>.ToArray: TArray<T>;
begin
  if FData = nil then
    Exit(nil);
  Result := FData.ToArray;
end;

{ TCowArrayData<T> }

constructor TCowArrayData<T>.Create(const Data: TArray<T>);
begin
  FData := Data;
end;

function TCowArrayData<T>.GetItem(Index: Integer): T;
begin
  Result := FData[Index];
end;

function TCowArrayData<T>.GetLength: Integer;
begin
  Result := Length(FData);
end;

function TCowArrayData<T>.MutableClone: ICowArrayData<T>;
begin
  if RefCount = 1 then
    Exit(Self);
  Result := TCowArrayData<T>.Create(ToArray);
end;

procedure TCowArrayData<T>.SetItem(Index: Integer; const Value: T);
begin
  FData[Index] := Value;
end;

function TCowArrayData<T>.ToArray: TArray<T>;
var
  i: Integer;
begin
  SetLength(Result, Length(FData));
  for i := 0 to Length(FData) - 1 do
    Result[i] := FData[i];
end;

procedure WriteArray(const Msg: string; Arr: TCowArray<Integer>);
var
  i: Integer;
begin
  Write(Msg, ':');
  for i := 0 to Arr.Length - 1 do
    Write(' ', Arr[i]);
  Writeln;
end;

var
  x, y: TCowArray<Integer>;
begin
  try
    x := TCowArray<Integer>.Create([1, 2, 3]);
    y := x;

    Writeln('Starting out, both x and y refer to same instance data');
    WriteArray('x', x);
    WriteArray('y', y);

    Writeln('Modifying x; note that y doesn''t change:');
    x[1] := 42;
    WriteArray('x', x);
    WriteArray('y', y);

    // Add operator as concatenation
    Writeln('Concatenation:');
    y := x + y;
    WriteArray('x', x);
    WriteArray('y', y);
    
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
end.

Semantics, Formats and Markets

2009-01-12T16:05:00.003Z

I was reading Dare's response to Bill's post on format proliferation and RDF. Bill hopes that more folks will understand RDF and seems to see it somewhat as a silver bullet for the problem, while Dare is more pessimistic.

The way I see it, there are distinctly two issues here. The first is the obvious one, solving the n*m problem, where n = format count and m = format producer and consumer count. The idea behind any common format or format mapping / shaping / semantic extraction is to reduce it to n*1 + 1*m, i.e. n+m.

That's all well and good, but it has problems: lowest common denominator, semantic loss in conversion, inhibition of innovation (gated on lowest common denominator), barriers to entry (new producers want only to be concerned with their specifics, not a huge standard, while new consumers don't want to have to implement the world before being useful).

More germane to the RDF situtaion, a requirement to understand a meta-model as well as any given model itself is so large a barrier to non-specialists just trying to get their work done that it's highly unlikely to ever receive serious attention. More on that in a bit.

Official standards-body approaches are the other issue. Formats in an area of innovation act like a bubbling market; take-up of formats by consumers and producers determine the winner, until eventually network effects reduce the total number of formats to just a handful. The problem with trying to manage this market process via a committee is much like the problem of trying implement socialism: the total information embodied in various choices of one format over and above another, which a free market reveals naturally, is not available to committees. Committees tend to be dominated by large market players which have various strategic and political objectives that may be quite distinct, and indeed sometimes covertly opposed, to the average market desires for any given format.

The working programmer, at the end of the day, is either putting square pegs into square holes (in which case, no problem), or trying to put square pegs into round holes, and then having to create an adapter to convert square pegs into round pegs. An arbitrary selection of square or round as a "peg standard" doesn't necessarily help him for his specific needs; similarly, pointing at some generalized framework for describing the semantic meaning of square and round pegs respectively is far too abstract for him to get his job done efficiently - i.e. without investment whose cost exceeds the value of getting the original job done.

iPod Touch, iTunes, and unwanted processes

2009-01-03T00:42:00.003Z

I recently got a second-generation iPod Touch. I don't make enough phone calls to make a phone contract worthwhile, much less an iPhone contract with O2 in the UK; and O2's PAYG (pay as you go) appears to charge GBP 7.50 per roaming MB, while I pay EUR 1 for first 50 MB with my Vodafone Ireland PAYG, whether I'm in the UK, anywhere else in Europe, or in the US. So, iPod Touch it is.

It's a nice device, both style-wise and as a hand-held web-browsing experience. The browser is good enough to create a paradox; limitations that only surface because of the increased expectations start to get a little annoying. For example, up to eight separate pages can be browsed simultaneously, but as soon as one of the pages gets big enough, information about the other open pages is forgotten beyond the URL / request parameters. This means that e.g. you don't want to leave a purchase page open while browsing in a parallel window, or you'll break navigation flow / possibly pay twice.

The browser also crashes a lot. I've had the device for about two days, and suffered many (8+) unprompted "back to main menu" transitions, with nary a hint from the device that the browser had crashed. It does make me wonder; how much of the reputation Apple has for good firmware is due to pretending that errors don't happen? The subsequent OS-level cleanup / resource management doesn't seem too solid either, since searches on the topic suggest that a clean reboot is what's necessary to restore this BSD-based Unix kernel to good running. This doesn't inspire confidence.

As a device for playing music, it's too large and heavy for my taste; I'm still using my second-generation Nano, even though I also have a third-generation Nano - the wheel is too small on it.

The draconian limitations that are de rigeur with Apple firmware chafe quite a lot. Even my humble K800 phone can create folders in the file system, browse it, start applications from arbitrary locations, open videos, music and pictures from arbitrary locations, etc., all using the same tree that you see when browsing the CF card from a PC. The iPod Touch doesn't have any of this: it's Apple's way or screw you, to be blunt. iTunes synchronization is a useless to me; I'm a file system guy - give me scripting, cron and hard & symbolic links and I'll create the structure I prefer myself. I suspect I won't be happy until the device is jailbroken.

Anyhow, the other reason I wanted to write this post, other than to praise and complain about the device, is the little setup I created to cope with iTunes 7, which I was reluctantly forced to upgrade to. The iPod Touch doesn't work without iTunes 7, and it also doesn't work without a bunch of other background services running, most importantly, the 'Apple Mobile Device' service.

Rather than have half a dozen Apple-related processes running, even though I'm not running iTunes and don't have a device connected to my machine, I wrote a little script to start up the necessary services upon iTunes startup, and kill off the unwanted processes after iTunes shutdown. They rely on Cygwin and some little utilities I wrote myself.

First up is 'hide.exe'. This simple executable, written in Delphi, runs a given process with a list of command-line arguments, but in a hidden window, by passing SW_HIDE as the nShow parameter. This basically lets a console hang around running my script while iTunes is running, waiting for it to exit, so that the script can clean things up later. The hide.exe executable itself is a GUI subsystem app, though it doesn't have a message pump or anything.

The second is a very simple killall script for Cygwin:

#!/bin/bash

if test -z "$1"; then
    echo "usage: $(basename $0) ..."
    echo "Kills all specified processes."
    exit
fi

while [ -n "$1" ]; do
    ps -W | grep -i "$1" | cut -b -10 | print0 | xargs -0 kill -f
    shift
done

Obviously, when using this script you don't want to be too ambiguous about your process search string. The 'print0' in the pipe is another little utility I wrote to bridge the gap between line-oriented programs, word-oriented programs and programs that can accept null-terminated strings. It simply reads each line one at a time, and prints out the same line with a null terminator instead of a newline. Without it, any programs with spaces in their names would be parsed by xargs as multiple separate arguments, since xargs, by default, breaks arguments on any whitespace, not just newlines.

With that aside, my iTunes wrapper script (I call it start-itunes) is fairly simple:

#!/bin/bash

itunes="${itunes:-/c/other/itunes/itunes.exe}"

net start 'Apple Mobile Device' > /dev/null
"$itunes" || messagebox "Failed to start \"$itunes\""
net stop 'Apple Mobile Device' > /dev/null || messagebox 'Failed to stop "Apple Mobile Device"'
net stop 'iPod Service' > /dev/null
killall SyncServer.exe distnoted.exe

It simply starts the required service, lets iTunes run to completion, and then stops the redundant services and blows away the crap left behind by iTunes. I don't use iTunes to "sync" anything, so I'm assuming that blowing them away doesn't hurt. I haven't had any problems, anyhow.

This script uses 'messagebox', yet another little utility I wrote, to display errors using the Win32 MessageBox function. This is necessary, otherwise the errors wouldn't be visible - the script is run from a hidden window.

The final step is the shortcut itself. Cygwin has a utility, mkshortcut, to create shortcuts, though I don't like its command-line syntax and wrote a wrapper script to make it look more like ln and friends. However, a Cygwin mkshortcut command-line for creating an appropriate shortcut for my script above might look a bit like this (watch the backslash, added for nicer PRE formatting):

mkshortcut -a "$(cygpath -w $(which bash.exe)) $(which start-itunes)" \
    -n start-itunes.lnk -i /c/other/itunes/iTunes.exe $(which hide.exe)

Since start-itunes and hide.exe are useful in themselves, they're on my path, so 'which' is able to find them.

Somewhat more efficient smart pointers

2008-11-02T23:44:00.003Z

There was a little to and fro in the comments on yesterday's post on more fluent smart pointers.

It wasn't my intention to create the ultimate in performance for the smart pointer, so I didn't pay much attention to it; I focused mainly on getting an effect from composing a number of simple reusable primitives and ideas.

However, I'd like to point out that since method references are just interfaces, a more efficient implementation can simply implement the interface directly. A yet more efficient implementation might choose to construct a vtable directly, and use a simple 64-bit value on the heap (32-bits for the reference count, 32-bits for the instance pointer), but I'll leave that as an exercise for the reader.

Anyhow, here it is: construction is now sufficient to assign to a location of type TFunc<T>, rather than needing an extra Wrap method:

unit ObjHandle2;

interface

uses SysUtils;

type
  TObjectHandle<T: class> = class(TInterfacedObject, TFunc<T>)
  private
    FValue: T;
  public
    constructor Create(AValue: T);
    destructor Destroy; override;
    function Invoke: T;
  end;
  
implementation

constructor TObjectHandle<T>.Create(AValue: T);
begin
  FValue := AValue;
end;

destructor TObjectHandle<T>.Destroy;
begin
  FValue.Free;
end;

function TObjectHandle<T>.Invoke: T;
begin
  Result := FValue;
end;

end.

Reference-counted pointers, revisited

2008-11-01T19:05:00.002Z

Some time ago, I blogged about writing smart pointers (i.e. reference-counted auto-destruction) in Delphi. While having dinner with some of the speakers at the EKON 12 conference I attended last week, a more fluent interface for using smart pointers in Delphi occurred to me.

I'm using the same TSmartPointer<T> class that I started out with in the previous article, though I've renamed it TObjectHandle<T>. The main tricks I'm pointing out here are (1) to use method references to avoid having to use the Value property all the time, and (2) to use aliases at the point of class definition to make construction slightly more palatable.

So, here's my new TObjectHandle<T> class; the main change, apart from the name, is a new method called Wrap:

type
  TObjectHandle<T: class> = record
  private
    FValue: T;
    FLifetimeWatcher: IInterface;
  public
    constructor Create(const AValue: T);
    property Value: T read FValue;
    class operator Implicit(const AValue: T): TObjectHandle<T>;
    class function Wrap(const AValue: T): TFunc<T>; static;
  end;

The implementation of the new method is pretty simple too:

class function TObjectHandle<T>.Wrap(const AValue: T): TFunc<T>;
var
  h: TObjectHandle<T>;
begin
  h := AValue;
  Result := function: T
  begin
    Result := h.Value;
  end;
end;

The capture of the h local variable will mean that the handle will be kept alive as long as the method reference constructed from the anonymous method is kept alive.

Here it is in use, as two versions, so that the usage difference can be seen. This is also where the additional lubrication of declaring aliases comes in. I start out with a little TCanary class which can keep track of destruction, and has a Name property to demo the fluency of the technique:

type
  TCanary = class
  private
    FName: string;
  public
    destructor Destroy; override;
    property Name: string read FName write FName;
  end;
  
  OHCanary = TObjectHandle<TCanary>;
  HCanary = TFunc<TCanary>;

The destructor prints out the name of the canary when it is destroyed. The two aliases represent an Object Handle for TCanary and a Handle for TCanary respectively. The fluent technique relies on both; the second is used for smart pointer locations and the first for smart pointer construction. There is a tradeoff involved in the technique, between construction fluency and usage fluency:

procedure Test1;
var
  canary: OHCanary;
begin
  // easy construction (implicit operator)
  canary := TCanary.Create;
  // but cumbersome access - always need Value accessor
  canary.Value.Name := 'Test1 canary';
end;

The new style has slightly worse construction, but better actual use:

procedure Test2;
var
  canary: HCanary;
begin
  // cumbersome constructor
  canary := OHCanary.Wrap(TCanary.Create);
  // but much nicer access
  canary.Name := 'Test2 canary';
end;

Without having to access everything by prefixing every access with .Value, a lot of fluency is gained, IMHO.

To summarize, here's the entire ObjHandle.pas unit:

unit ObjHandle;

interface

uses SysUtils;

type
  TObjectHandle<T: class> = record
  private
    FValue: T;
    FLifetimeWatcher: IInterface;
  public
    constructor Create(const AValue: T);
    property Value: T read FValue;
    class operator Implicit(const AValue: T): TObjectHandle<T>;
    class function Wrap(const AValue: T): TFunc<T>; static;
  end;
  
  TObjectHandleArray<T: class> = array of TObjectHandle<T>;

procedure MakeDestroyer(Obj: TObject; out Result: IInterface);

implementation

{ TLifetimeWatcher }

type
  TLifetimeWatcher = class(TInterfacedObject)
  private
    FProc: TProc;
  public
    constructor Create(const AProc: TProc);
    destructor Destroy; override;
  end;

constructor TLifetimeWatcher.Create(const AProc: TProc);
begin
  FProc := AProc;
end;

destructor TLifetimeWatcher.Destroy;
begin
  if Assigned(FProc) then
    FProc;
  inherited;
end;

procedure MakeLifetimeWatcher(out Result: IInterface; const AProc: TProc);
begin
  Result := TLifetimeWatcher.Create(AProc);
end;
  
procedure MakeDestroyer(Obj: TObject; out Result: IInterface);
begin
  Result := TLifetimeWatcher.Create(procedure
    begin
      Obj.Free;
    end);
end;

{ TObjectHandle<T> }

constructor TObjectHandle<T>.Create(const AValue: T);
begin
  FValue := AValue;
  MakeDestroyer(FValue, FLifetimeWatcher);
end;

class operator TObjectHandle<T>.Implicit(const AValue: T): TObjectHandle<T>;
begin
  Result := TObjectHandle<T>.Create(AValue);
end;

class function TObjectHandle<T>.Wrap(const AValue: T): TFunc<T>;
var
  h: TObjectHandle<T>;
begin
  h := AValue;
  Result := function: T
  begin
    Result := h.Value;
  end;
end;

end.

And here's the entire demo program:

{$apptype console}

uses SysUtils, ObjHandle;

type
  TCanary = class
  private
    FName: string;
  public
    destructor Destroy; override;
    property Name: string read FName write FName;
  end;
  
  OHCanary = TObjectHandle<TCanary>;
  HCanary = TFunc<TCanary>;

destructor TCanary.Destroy;
begin
  Writeln(FName, ' died.');
end;

procedure Test1;
var
  canary: OHCanary;
begin
  // easy construction (implicit operator)
  canary := TCanary.Create;
  // but cumbersome access - always need Value accessor
  canary.Value.Name := 'Test1 canary';
end;

procedure Test2;
var
  canary: HCanary;
begin
  // cumbersome constructor
  canary := OHCanary.Wrap(TCanary.Create);
  // but much nicer access
  canary.Name := 'Test2 canary';
end;

begin
  Test1;
  Test2;
end.

Types: Values versus Locations

2008-10-20T11:28:00.003+01:00

Minor thought I had this morning: I was doing some prep work for the conference I'm speaking at next week, and I noticed I was being perhaps overly pedantic about the terminology of types in a way that only matters for imperative languages.

I habitually make explicit the distinction between values and locations of a particular type. I might say, the storing of values of multiple types in a location of a single type is an instance of polymorphism. In describing a class, I might say, this is "an iterator over a stream of values of type T", instead of just saying "an iterator over a stream of T" (no, not a teapot!).

The distinction is important in languages that have mutable state. Locations can have their address taken, they are subject to polymorphism, and the location itself has an identity independent of its value. Values, on the other hand, cannot have their address taken - at most, the value is or contains an address. Values always have a fixed type, but locations may contain values of different types if the type of the location is polymorphic. Locations may be lvalues or rvalues, but values are always rvalues (unless you dereference, index or field-access them). Especially important is the fact that implementations of closures in imperative languages like C#, Delphi, Ruby, etc. have almost always opted to capture locations, not values.

However, consider a pure language, like Haskell. If the language doesn't have mutable state, there's no such thing as a location (at the conceptual level). If you have an iterator in such a language (perhaps modeled using a tail-call continuation design), it's redundant to say "iterator over a stream of values of T" - it's always OK to say "iterator over a stream of T" instead. And the closures might capture values or locations, as performance demands, since the semantics don't change.

Painfully Amateur Philosophy addendum: locations are a pretty physical concept - you know the data is in there in memory somewhere - but values are more of a platonic concept, existing in some pure universe, and we can only refer to them by metaphor and convention by using specific bit patterns interpreted in precise ways. I specifically use the word metaphor in the conceptual metaphor sense: in no way are the electron levels inside the machine representing the ASCII characters of "cat" anything like the furry mammal. Rather, the bit pattern is like a pointer only the human mind can dereference, once it has been transformed into what humans have agreed to be a semantically equivalent representation on screen (which is just a different part of memory) or on paper (which is just bits streamed out over the wire).

Perhaps most people find this obvious and boring, but what interests me is the way the representational power of the bits is entirely unmagical, yet it permits meaning to be stored inside the machine. I say meaning, by which I mean that we humans, the only judges of what is meaningful, find to be meaningful, but no more: I do not think meaning is something inherent in objects, just in judges. What if brains had no more magic in their neurons than the circuits in the machine? (It seems entirely plausible to me, and to many programmers I imagine. And this would constrain those judges to be mere boolean functions over matter patterns.)

In such a scenario, qualia would be beyond our power to deconstruct using physical means, since the brain/machine could be evaluated "on paper", and such an evaluated brain would report the same qualia as you or I, and we would have no way to argue otherwise in a one on one dialogue (this is how I think of the Turing test - as a philosophical concept, not an actual benchmark, which I think is silly and pointless). Under this assumption, there isn't any argument that could prove that the machine isn't conscious which doesn't itself rely on arbitrarily chosen boolean functions which explicitly return false for non-mysterious matter patterns (e.g. capable of "understanding"). Closely related is the problem of other minds, about which I'm on Turing's side - if we can't tell the difference, then there isn't any.

Perhaps consciousness and the experience of qualia is just what matter feels like when it's part of a causal chain? (The technical term, I understand from my Googling, is "type physicalism", though epiphenominalism is related.)

Philosophy over. I need to get back to work :)

WASD's little known alternative, QWAS

2008-10-18T23:26:00.006+01:00

I was reading a review of a new Microsoft gaming keyboard today and I noticed that it has extra highlighting for the traditional WASD keyboard layout most commonly used by first-person shooters (FPSs).

This interests me because I don't use WASD; I use a variant that I have almost never seen anyone else describe, namely, QWAS, with W for forward, Q/S for strafe left/right, and A for reverse.

It was a long time in gestation. I started out playing Doom with just the keyboard, like most other people at the time, but then I read - I forget where and can't find the reference - about using the mouse for looking and the keyboard, specifically the Z and X keys, for strafing, with Space, Ctrl and mouse buttons used in some combination for Use, Fire, Forward, Run, etc. I didn't find this combination especially easy to use, though, so it wasn't long before I stumbled onto the QWAS hold I use today. I do, however, find it puzzling that so few other people use it.

The reason I don't use WASD is because I think the layout is awkward for the wrist when the keyboard is centrally positioned, ready for touch-typing:

Moreover, since, like most men, my ring finger is much longer than my index finger, WASD means my ring finger on the A key is in an uncomfortable cramp-prone position:

When this is compared with the QWAS layout, the advantage in comfort and ergonomics - for my hand shape and keyboard positioning, at least - is clear:

The Search for Differentiation and Build vs Buy/Adopt/Coopt

2008-10-18T19:22:00.003+01:00

There's been some chatter in the development blogosphere recently about that old chestnut, the build vs buy decision; but in these days of open source development, there are more alternatives to build: adoption and cooption (private forking / customized modules etc.).

Jeff Atwood wrote an article about not being ashamed to write your own software (i.e. build) when the functionality in question is core to your business. Much of the reaction to Jeff's fairly strongly stated article has been pulling in the opposite direction. I'd like to argue back in Jeff's direction, though.

The basic trouble I have is that the vast majority of open source is crap. Once you tread outside the critical path of high-user-base server-side software, like (to take three examples completely at random) Linux, Hibernate and MySQL, you very quickly get stuck in the mire; even then, much of the "good" open-source software is only good in parts.

For example, if you're looking for a mature, well-designed and stable interface for writing drivers for Linux - exactly what is needed to make supplying binary drivers viable for hardware vendors - tough luck, buddy. Economically speaking, incentives in open source are almost always not aligned with consumers' interests, but rather with the companies which hire the developers: and those guys are usually consultant / service shops, i.e. aligned with corporate customers, not consumers. Firefox is an exception because of Google advertising revenue, which is correlated with consumer adoption, thus their incentives are aligned.

Rails has a solid concept and has a reasonably good API, but its execution is, IMHO, not great. I have little patience for server-side technologies that aren't thread-safe, as request/response cycles are trivially parallel; though, from an historical viewpoint I can understand it, as relatively few scripting languages implement threading. All the same, though, the kind of coding that would make a server-side library thread-unsafe, I wouldn't tolerate even if the platform was single-threaded: it implies poorly-factored mutable global state. I wouldn't touch PHP with a barge pole - I have zero technical respect for the language.

To compete, you need differentiation: you need to do something, for some subset of the market, in a way that is head and shoulders better than every other choice. Open source (or, indeed, buying) can only give you what everyone else has; you need to add in the magic yourself that makes you stand out. Private forks of open-source software are one way of doing that, but is vulnerable to all the usual problems that forking has (staying current, merging bugfixes, divergent evolution, etc.). Creating your differentiating software yourself means your codebase will be smaller and more perfectly adapted to your problem domain, and thus easier to maintain and keep up to date than a private fork of an ongoing public open-source project.

The only real question is which software in your organization is, in the language of strategic management, going to help give you a sustainable competitive advantage. Open source software and third-party libraries can never give you that (unless you own the founding team & brand), because they are equally available to your competitors. Choosing the balance of build vs buy means trading off risk and cost versus lack of differentiation. You have to measure and compare the resources available to you and the risk of non-completion of differentiated software, taking into account opportunity cost, with the lack of differentiation that would come from the buy / ad/coopt decision.

Where the balance lies thus depends on both sides: how much [human+monetary] capital you have, versus the competitive landscape on the other. Whether a third-party library exists or not, in and of itself, is not a sufficient reason to always use it. Making users happy isn't sufficient; you must make them happier than the competition, and that's not going to happen if you don't do something the competition isn't already doing. The only thing that holds you back is risk & capital versus the reward in differentiation.

Anonymous methods as an alternative to 'out' parameters

2008-09-25T19:05:00.003+01:00

Out parameters are a useful language feature for multiple results, particularly when the language in question doesn't have tuples as a first-class feature. Even though Java omitted pass-by-reference (meaning both var and out in Delphi parlance), C# did not follow its lead and includes both parameter-passing semantics. And so it follows that the .NET method System.Int32.TryParse(String,Int32&) is possible in C#, but not in Java.

The API works well enough when you want to put the extra return value(s) into local variables or visible fields. What about times when you want to put the return value into a property, though? Since you can't (generally) take a reference to a property, you have to manually create a local variable, pass the local variable by reference, and then, in a separate statement, assign the variable to the property.

Anonymous methods can provide an "out" here, if you'll pardon the pun. By passing a setter instead of a mutable reference, you can escape from this constraint. Here's are two overloaded ReadLine functions, one using the out mechanism, the other using the setter pattern:

function ReadLine(const Prompt: string; out Dest: string): Boolean; overload;
var
  line: string;
begin
  Write(Prompt, '> ');
  Flush(Output);
  Readln(line);
  Result := line <> '';
  if Result then
    Dest := line;
end;

function ReadLine(const Prompt: string; const Setter: TProc<string>): Boolean; overload;
var
  line: string;
begin
  Write(Prompt, '> ');
  Flush(Output);
  Readln(line);
  Result := line <> '';
  if Result then
    Setter(line);
end;

As can be seen, the code that uses setters versus out parameters is little different. (It would be more different in a language that requires definite assignment, like C#, as otherwise we would have to set Dest to some value in the case of no entry.)

The real difference is in the usage point of this idiom. (And don't forget, this is an idiom; it might look odd the first time you see it, but when you understand anonymous methods, or even just the idiom, it's a trivial pattern and easy to understand.) Here's the definition of a class with a couple of properties, and a couple of functions. The first uses the out idiom, and the second uses the setter idiom:

type
  TFrob = class
  private
    FFoo: string;
    FBar: string;
  public
    property Foo: string read FFoo write FFoo;
    property Bar: string read FBar write FBar;
  end;

// Using 'out' idiom - requires separate assignment statement
function WorkWithFrob: Boolean;
var
  frob: TFrob;
  temp: string;
begin
  frob := TFrob.Create;
  try
    if not ReadLine('Give me Foo', temp) then
      Exit(False);
    frob.Foo := temp;
    if not ReadLine('Give me Bar', temp) then
      Exit(False);
    frob.Bar := temp;
    Writeln(Format('Your foo is %s and bar is %s', [frob.Foo, frob.Bar]));
    Result := True;
  finally
    frob.Free;
  end;
end;

// Using 'setter' idiom - can be done inline
function WorkWithFrob2: Boolean;
var
  frob: TFrob;
begin
  frob := TFrob.Create;
  try
    if not ReadLine('Give me Foo', procedure(x: string) begin frob.Foo := x; end)
        or not ReadLine('Give me Bar', procedure(x: string) begin frob.Bar := x; end) then
      Exit(False);
    Writeln(Format('Your foo is %s and bar is %s', [frob.Foo, frob.Bar]));
    Result := True;
  finally
    frob.Free;
  end;
end;

This idiom isn't useful very often, but when the right situation comes up, it's really quite nice to have, especially when you need persistent references to properties - e.g. when you need to store the reference in some structure somewhere, and update it with different values over time.

Viewed more generally, anonymous methods / method references can act as kinds of pipes that join up producers and consumers of data, with getters (TFunc<TResult>) for consumer-pull and setters (TProc<TArg>) for producer-push.

Google Developer Day 2008, London

2008-09-17T17:40:00.003+01:00

I was at Google Developer Day here in London yesterday. Most of it was pretty light on technical details, but there was some good info on V8, Google's new JavaScript engine which is used in Google Chrome.

The specific sessions I went to were Intro to Android, Intro to Android SDK, Google Data API mashups, and V8 - "the Chrome Engine" (sic). The heavy tilt towards Android was chiefly driven by curiosity after Mike Jennings took out a pre-production OHA phone and demoed it in the keynote. It looks reasonably neat, borrowing some nifty effects from Apple's iPhone; no pinch-zooming though.

Intro to Android was a mix of business / very lightweight technical details about the phone. Apparently, Google has twisted some arms in back-rooms to get buy-in from operators & OEMs, so anything with the OHA / Android branding etc. should have some minimum level of openness etc. Should be a welcome relief from the vice-grips of Apple. There wasn't much description of the Dalvik VM running behind the scenes, other than that you take your .class files from javac and run them through a processor to get running on the device, and apparently the guy Mike talks to about the VM is "very bright". That was mentioned a number of times, so it must be important.

The Android SDK intro was given by a guy (Carl-Gusaf Harroch) from a local startup, not an actual Google guy. He very roughly described content providers, and also briefly outlined some entries in the application manifests, which are the mechanism by which the Android OS figures out what events your application is interested in (phone call arrived, moved certain distance according to GPS, that kind of thing). There was a laboured comparison of content providers with REST, in that there are methods that correspond to CRUD operations, but apparently there are other concerns such as observability etc. which make them not as simple as REST (and thus an invalid comparison, in my view). Apparently content stored on the phone and exposed to other applications is heavily skewed towards assuming that the content is living in an SQLite DB.

The GData mashup session wasn't interesting to anyone who has interacted with Google REST / AtomPub APIs even trivially. Once upon a time I wrote a blogger post app, so I didn't learn much.

Oh, and if you are writing a client to GData etc., I recommend that you don't start by trying to grok any of the Google API libraries unless you need deep integration. I didn't like the look of them last time (Java-itis, factories etc. everywhere), and I'm fairly sure they haven't improved.

Finally, I went to the V8 talk by Kevin Millikin. This was the best and most technical by a long, long way; to be honest, if it hadn't been for this talk, the day would have been a waste, on net. He described some V8 implementation details.

Roughly speaking, the main approach V8 takes is to try and shoe-horn classes into the JavaScript dynamic type system so that other, more classical dynamic object-oriented optimizations can be applied in the future. The way it works is with hidden classes. Every freshly new'd up object obj gets the same class, call it class_0. The first time you add a property, call it 'x', to obj, a new class class_1 is created which has a single property called 'x'. Additionally, a class transition edge is added to class_0 pointing at class_1, and this edge is labeled 'x'. This means that all objects which are freshly new'd up and have a single property called 'x' added to them will all share the same class. So, in theory, every object in the system which has the same properties in the same order will share the same class. The class transitions described above form a tree (I asked), and can't close over to make a dag, because if that happened, enumeration order would change. (I believe this is an implementation detail of ECMA, so people shouldn't be relying on it; however, it's something the V8 folks discovered that they could not change from extant practice.) This in turn means that if you do have conditional branches in your JavaScript constructors, you should assign to all properties in the same order, if that is feasible.

Anyhow, the use of classes as indicated above means that object use sites can now be optimized based on the runtime class of the object. Here's a specific example: whenever you access a property in JS running on V8, the access site will be a little stub function using one of 4 templates: uninitialized, pre-monomorphic, fast-monomorphic, and megamorphic. The first time the access site is invoked, the runtime class is inspected and noted, and the stub moves to the pre-monomorphic state. The second time it's accessed, the runtime class is checked against the previous class and if it's the same, the stub moves to the fast-monomorphic state, and is written so that it is very simple: compare object type, if it's as expected then dereference to object storage, then load property at the specific offset (stored in the property in the class but inlined as machine code in the access site). If the class wasn't as expected, then the stub is changed to the megamorphic state. Finally, the megamorphic state is the slow path that falls back to hash-based lookup, just like most other JS implementations.

Since arrays in JS are just tightly-packed hash buckets with particular key patterns, the same approach could be taken but it wouldn't necessarily be fast. Apparently the V8 folks discovered that a lot of artificial JS benchmarks were based around array manipulations, so they put a little work in this, but they're not finished in this area (as far as I could make out). In any case, array access strategy is governed by a heuristic; for small packed arrays accessed with integers, a direct lookup can be made. For larger, sparse arrays, the property access mechanism is used.

Variable capture in closures (a topic dear to my heart at this point) also works along similar lines as property creation, from what Kevin described. There is a caveat: if you are using the "with" statement in JavaScript, or there is eval( between the variable definition and the access site in your closure, then the access can't be optimized; these constructs fall back to the hash-based lookup, because they effectively need to use dynamic scope. So, don't create closures in that way unless you're prepared for the speed bump.

The majority of the JS standard library, things like array.join etc., are implemented in JavaScript itself in V8. They're using a special %AddProperty function to make them non-enumerable. V8 also uses a freeze-dried heap mechanism to reduce the cost of initializing the standard library. It can essentially save and restore the heap, with appropriate relocations etc. as required, so that it doesn't have to reparse everything on startup.

The garbage collector for V8 looks less interesting as a source of performance. It's almost certainly an improvement on what other JS implementations are using for GC, but I think it's some way from the last word on the topic. It has only 2 generations, so intermediate live objects that get promoted when collecting new-space will eventually force a costly old-space collection. Kevin didn't say that they were using write barriers to reduce need to scan old-space looking for pointers to new-space, but grepping the V8 sources turns up some write barrier hits, so maybe they are. V8's GC is definitely better than reference counting as implemented in IE of older days, of course.

Since the main approach taken thus far was just to give values a class, and use that to optimize property access, there is still a lot of scope for optimization in V8. I wouldn't be surprised to see significant (2-5x) performance improvements in the not too distant future in V8, as more techniques are integrated. They're currently going straight from the JS AST to machine code, no inlining of aforementioned property access sites (AFAICT - there was a 'ret' at the end of the demo access site). The main thing is (a) they have objects pinned down to types now, and (b) hopefully as JS developers learn how to make code run fast under this paradigm, objects will look even more type-ful and thus increase scope for other optimizations.

Exogenous Exceptions (oh and another Vista rant :)

2008-09-11T04:01:00.003+01:00

Eric Lippert has just posted an entry about "vexing exceptions", talking about the various buckets he classifies exceptions into and different approaches for handling them:

Vexing exceptions are the result of unfortunate design decisions. Vexing exceptions are thrown in a completely non-exceptional circumstance, and therefore must be caught and handled all the time.

[...]

The classic example of a vexing exception is Int32.Parse, which throws if you give it a string that cannot be parsed as an integer. But the 99% use case for this method is transforming strings input by the user, which could be any old thing, and therefore it is in no way exceptional for the parse to fail.

[...]

And finally, exogenous exceptions appear to be somewhat like vexing exceptions except that they are not the result of unfortunate design choices. Rather, they are the result of untidy external realities impinging upon your beautiful, crisp program logic. [Example of file-not-found follows, and points out that File.Exists check would only be a race.]

However, I strongly disagree on his suggestion that you try to catch exogenous exceptions, and somewhat disagree on the "vexing" exceptions (here, it's probably just the specific example he chose that I don't really agree with).

In my opinion, exceptions should, in general, be propagated all the way to a user-visible dialog box / notification message, if the exception represents an error by the user. So, for example, if a user enters a floating-point value or a string value in a box that should semantically be an integer, it is fine for an exception in Int32.Parse to bubble its way back into the user's view - so long as the message in the exception is meaningful to the user, and is written in the correct language / jargon etc. If not, then certainly the exception should be wrapped in another exception and re-propagated, but the exception itself shouldn't be just caught.

Of course, if there is valid logic for a failure case that doesn't simply mean telling the user about a problem in their input / what they're trying to do, then that's a situation where TryParse etc. makes sense.

On the exogenous question, catching these is extremely problematic, because when the user eventually sees the message, they'll need to figure out what went wrong. So, if you have a file access problem, the very first exception raised - the exogenous one when the CreateFile call at the heart of things failed - should have a message associated with it indicating that such-and-such file couldn't be accessed. This is the same message that should eventually be propagated to the user, either in an event log, a dialog box, or other notification mechanism, particularly if the high-level attempted action was user-initiated.

To do otherwise leaves the user stranded with a generic, non-specific (and thus meaningless) "I couldn't do something" message. I've seen far too much MS software that gives non-specific and non-actionable error messages in failure situations that this kind of advice really annoys me unless it is very carefully qualified and described.

If anything, exceptions should be wrapped rather than caught, with higher-level semantic information wrapping the underlying reason. So, if you're trying to e.g. modify a record in a data entry application, the chain of failure might be "couldn't modify record because" -> "couldn't contact database for locking because" -> "couldn't connect to database server because" -> "remote server name FooBar could not be found". This kind of message has information about every level of the stack, and should a user have to e.g. contact IT, the full message (the technical details can be hidden in a pop-out dialog etc.) is 100% actionable, and even regular users, let alone power users, may find it actionable.

Software does not have AI-level capabilities, and is very far from it. Describing what went wrong is 100x more useful than presenting something vaguely actionable. Unless the error case is very common, and thus you are very certain what the fix is, you should not try to present "actionable" advice over describing what went wrong, simply because to give good actionable advice in general, you need to embed an expert system; without populating the expert system with data, it needs to include IT-support-level AI, which like I said, isn't happening any time soon.

Here's a specific example that really burned me just the other day. Vista Ultimate has a full PC backup capability. I found out that my main HD is failing (SMART alert). I wanted to restore that backup onto a different disk of the same make and size (actually, right down to consecutive serial numbers). However, my machine is rather complex - there are 7 SATA disk devices in the machine. The Vista OS DVD failed to restore the backup to my perfectly-matched disk, but I have no idea why. All I do know is that the error message was "vaguely actionable":

Error details: There are too few disks on this computer or one or more of the disks is too small. Add or change disks so they match the disks in the backup and try the restore again. (0x80042401)

This message is completely and utterly useless, because it does not describe what went wrong, only "how" to fix it - but because the software isn't AI-level and doesn't include an expert system, it itself can't produce a specific set of instructions.

In this machine, I had 4x1TB disks, 1x400GB disk, and 1x200GB disk; the backup was on the 400GB disk and the target of the restore was the 200GB disk. 2 of the 1TB disks were blank and formatted. Thus, there was no lack of disks or space. Similarly, the target of the restore was at Disk 0, achieved through careful selection of the SATA connection on the motherboard. Still didn't work though, and I can't fix it because the error message is following the wrong philosophy for our current knowledge of AI.

FWIW, here's another user's experience with this problem. Notice the procedure to actually show the user the actual error, rather than the useless message:

Boot from the Vista DVD
Go to Repair Computer -> Command Prompt
Go into Regedit
Under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\
Create the key: Asr
Under Asr create the key LogFileSetting
Under LogFileSetting create the dword EnableLogging with the value 1
Under LogFileSetting create the string LogPathName (string) with a value such as D:\Asr.log
- you should specify a physical drive (e.g. I used the drive you are going to restore from) not the ramdrive (X:) so that the log is saved after reboot.
Exit Regedit
From the Repair menu launch Complete PC Restore
Attempt the Complete PC Restore
When you get the error, check the logging path to be sure the Asr.log file exists. I did that by going back to the Command Prompt and getting a directly listing before rebooting.

This is disgraceful, and frankly, unforgivable. Don't do this with your exogenous exceptions.

Vista LanmanServer falling apart on you?

2008-09-09T11:05:00.004+01:00

I've been having a lot of trouble over the past day or so with my Vista machine. Since I can't tolerate it long as a main desktop, it's primarily acting as a file and print server.

Over the past day, however, the Vista network sharing service, LanmanServer, has been falling over on me. All attempts to connect to Vista shares fail with error 1130, aka ERROR_NOT_ENOUGH_SERVER_MEMORY, and the event log on Vista is filled with 2017 (primarily) and 2021 Event IDs:

2017: The server was unable to allocate from the system nonpaged pool because the server reached the configured limit for nonpaged pool allocations.

2021: The server was unable to allocate a work item %d times in the last 60 seconds.

Fixing it required stopping Computer Browser (net stop browser) and Server (net stop server) and restarting them (net start etc.) on the Vista machine.

Other people have been having this problem too; here's a relevant thread where you can read up about these frustrations:

[...] Vista Ultimate file/print server that is accessed by various Vista and XP client workstations. Recently - and increasingly - the XP machines have been spontaneously losing their connections to the Vista file server. This started out as a minor annoyance about a month ago, but has escalated to the point where the XP clients cannot stay connected for more than 5-30 minutes[...]

Anyhow, I've found a solution that appears to work, linked to at the end of the previous thread (which makes me suspect it fixed the problem for those folks too).

Smart pointers in Delphi

2008-09-04T22:47:00.003+01:00

Strongly-typed smart pointers are now possible in Delphi, leveraging the work on generics. Here's a simple smart pointer type:

  TSmartPointer<T: class> = record
  strict private
    FValue: T;
    FLifetime: IInterface;
  public
    constructor Create(const AValue: T); overload;
    class operator Implicit(const AValue: T): TSmartPointer<T>;
    property Value: T read FValue;
  end;

Here it is in action, where TLifetimeWatcher is a little class that executes some code when it dies:

procedure UseIt;
var
  x: TSmartPointer<TLifetimeWatcher>;
begin
  x := TLifetimeWatcher.Create(procedure
  begin
    Writeln('I died.');
  end);
end;

Here's the full project code that defines TSmartPointer<>, TLifetimeWatcher, and the above test routine:

{$APPTYPE CONSOLE}

uses
  SysUtils;

type
  TLifetimeWatcher = class(TInterfacedObject)
  private
    FWhenDone: TProc;
  public
    constructor Create(const AWhenDone: TProc);
    destructor Destroy; override;
  end;

{ TLifetimeWatcher }

constructor TLifetimeWatcher.Create(const AWhenDone: TProc);
begin
  FWhenDone := AWhenDone;
end;

destructor TLifetimeWatcher.Destroy;
begin
  if Assigned(FWhenDone) then
    FWhenDone;
  inherited;
end;

type
  TSmartPointer<T: class> = record
  strict private
    FValue: T;
    FLifetime: IInterface;
  public
    constructor Create(const AValue: T); overload;
    class operator Implicit(const AValue: T): TSmartPointer<T>;
    property Value: T read FValue;
  end;

{ TSmartPointer<T> }

constructor TSmartPointer<T>.Create(const AValue: T);
begin
  FValue := AValue;
  FLifetime := TLifetimeWatcher.Create(procedure
  begin
    AValue.Free;
  end);
end;

class operator TSmartPointer<T>.Implicit(const AValue: T): TSmartPointer<T>;
begin
  Result := TSmartPointer<T>.Create(AValue);
end;

procedure UseIt;
var
  x: TSmartPointer<TLifetimeWatcher>;
begin
  x := TLifetimeWatcher.Create(procedure
  begin
    Writeln('I died.');
  end);
end;

begin
  try
    UseIt;
    Readln;
  except
    on E:Exception do
      Writeln(E.Classname, ': ', E.Message);
  end;
end.

Update: Fixed capture - was capturing a location in the structure, which of course will be freely copied etc.