Entropy Overload

Life with Cygwin: Paths, Notepad and Clipboard

2008-05-21T11:00:00.004+01:00

I have in the past written about my Cygwin setup, but several weeks ago, while I was in Scotts Valley and having dinner with Adam Markowitz and Steve Trefethen, Steve mentioned that I should write a bit more about my setup.

While the defaults for a new Cygwin install today are better than they have ever been, there are still a lot of things to be desired. Using, as I do, a bash shell as my main command line, yet still being a Windows programmer running on Windows, means that I need to integrate with Windows command-line programs. Herein lies a problem: Cygwin uses Unix-like paths with '/' and no drive letter or colon (which is a path separator on Unix systems), while Windows inherits the usual CP/M/DOS traditions. Incidentally, I mount those drives to root letters to make converting between Windows and Cygwin paths easy:

$ mkdir /c
$ mount 'c:\' /c

Mounting removable drives like floppy disks and DVD-drives in the same way is more problematic, as the 'ls --color=auto' command (which wants to colour in files and directories corresponding to thier types) will try to read the contents of these directories, which of course will be mounted to removable drives. This would normally cause delays when doing a listing of the root directory, as the removable drives in the system spin up etc. Consequently, for removable drives I use a different technique. For example, my DVD-drive is 'O:' (because it's round, and because I frequently add and remove drives and I don't like drive letters changing because it breaks things), so this is how I integrate the DVD-drive into the Cygwin file system:

$ ln -s '/cygdrive/o' /o

This creates a symbolic link which will just be a broken link when there is no DVD in the drive. I do similar things for my iPods, pen drives, floppy drive (I still keep one around, just in case :), etc.

Anyway, back to Cygwin/Windows path interaction. Cygwin does provide a command to convert between paths, called 'cygpath'. It can be used fairly easily in an ad-hoc way on the command-line:

$ notepad /etc/bash.bashrc
# (this will fail, as notepad can't cope)

$ notepad $(cygpath -w /etc/bash.bashrc)
# (this will work here but fails when the Windows path has spaces)

$ notepad "$(cygpath -w /etc/bash.bashrc)"
# (this is more resilient)

winexec

Using cygpath manually is a bit of a pain, so I wrote a little bash script I call winexec to capture the pattern:

#!/bin/bash

function usage
{
    echo "usage: $(basename $0) [options]  [...]"
    echo "Executes an executable with arguments, converting non-options into Win32 paths."
    echo "Options:"
    echo "  -f    Only convert paths to files or directories which actually exist."
    echo "  -s    Use cygstart to execute detached from console."
    echo "  -k    Skip converting paths until '**' found in arguments (and remove the '**')."
    echo "  --    Terminate $(basename $0) options processing."
    exit 1
}

# Process options to winexec itself.

while [ "$1" ]; do
    case "$1" in
        -f)
            ONLY_FILES=1
            ;;
        
        -s)
            USE_CYGSTART=1
            ;;
        
        -k)
            SKIP_TO_STAR=1
            ;;
        
        --)
            shift
            break
            ;;
        
        -*)
            # Give an error on unknown switches for future compat.
            usage
            ;;
        
        *)
            break
            ;;
    esac
    shift
done

EXECUTABLE="$1"
shift

test -z "$EXECUTABLE" && usage

# Options conversion and caching.

declare -a OPTS
function add_opt
{
    OPTS[${#OPTS[@]}]="$1"
}

function add_file_opt
{
    if [ -n "$SKIP_TO_STAR" ]; then
        if [ "$1" = "**" ]; then
            SKIP_TO_STAR=
            # Eat '**' but don't add.
        else
            # Haven't seen star yet, so add unconverted.
            add_opt "$1"
        fi
    else
        if [ -n "$ONLY_FILES" ]; then
            if [ -f "$1" -o -d "$1" ]; then
                add_opt "$(cygpath -w "$1")"
            else
                add_opt "$1"
            fi
        else
            add_opt "$(cygpath -w "$1")"
        fi
    fi
}

# Process arguments to executable.

while [ "$1" ]; do
    case "$1" in
        -*)
            add_opt "$1"
            ;;
        
        *)
            add_file_opt "$1"
            ;;
    esac
    shift
done

# Actually start the executable.

if [ "$USE_CYGSTART" ]; then
    cygstart -- "$EXECUTABLE" "${OPTS[@]}"
else
    "$EXECUTABLE" "${OPTS[@]}"
fi

For an example of how I use that, I have another script called 'dir', for when I feel like I need classic 'dir' options:

#!/bin/bash

winexec -f -k cmd /c dir '**' "$@"

All these scripts, BTW, go in my ~/bin directory and are chmod'd 0755 to make them executable:

$ mkdir ~/bin
$ chmod -R 0755 ~/bin/*

My system's PATH (i.e. the Windows PATH, from System Properties | Advanced | Environment Variables) includes my home directory's bin directory before the Cygwin bin directories, but it also includes those. There can be some knots here though, which I won't get into today. The scripts also need to use Unix line-endings, though Cygwin was less strict about this in the past. It's easily enough done, though: the dos2unix command will normalize to Unix any text files given as arguments.

n

Notepad is a classic programmer's tool - as in "all I need is Notepad and the compiler" (or maybe just Notepad ;), etc. Since Notepad doesn't react so well to multiple file arguments, it isn't completely suitable to the winexec trick. I have a customized script for Notepad:

#!/bin/bash

if [ -z "$1" ]; then
    echo "usage: $(basename $0) ..."
    echo "Starts notepad on the file(s)."
    echo "If  is -, then standard input is redirected to a temp file and opened."
    exit 1
fi

for file in "$@"; do
    if [ "$file" = "-" ]; then
        file=$(mktemp)
        cat '-' > $file
        (
            notepad "$(cygpath -w "$file")"
            rm $file
        ) &
    else
        cygstart -- notepad "$(cygpath -w "$file")"
    fi
done

Having created this little utility, I can open multiple files in notepad just using the bash wildcards:

$ n /c/windows/*.txt
# (there aren't too many of these guys)

Similarly, I can capture a program's output into Notepad for reference in a separate window and possible printing:

$ dir /c | n -
# (opens a notepad window containing the directory listing for C:\)

Copy and Paste

Finally (for now), good Windows integration requires good clipboard integration. The native-Windows rxvt terminal which ships with Cygwin already support automatic copy on selection and paste with middle-cilck or Shift+Ins, familiar to Unix console and X users. However, I often want to copy the output of a command to the clipboard, or get a copied piece of text into a file, or transform the contents of the clipboard (perhaps to do a search and replace on it), etc. Thus, I wrote two little utilities in Delphi, copy-clipboard.dpr and paste-clipboard.dpr:

Copy

{$APPTYPE CONSOLE}

uses
  SysUtils, Classes, Clipbrd;

var
  list: TStringList;
  line: string;
begin
  try
    list := TStringList.Create;
    while not Eof(Input) do
    begin
      Readln(line);
      list.Add(line);
    end;
    Clipboard.AsText := list.Text;
  except
    on e: Exception do
      Writeln(ErrOutput, e.Message);
  end;
end.

(Freeing objects that have no external effect when freed before you're about to exit the program is the height of pointlessness, in case you were wondering.)

Paste

{$APPTYPE CONSOLE}

uses
  SysUtils, Clipbrd;

begin
  Write(Clipboard.AsText);
end.

These two utilities, having been compiled, renamed to c.exe and p.exe, and moved to my ~/bin directory, come in very handy. For example, should I myself have wanted to copy one of the above scripts, I normally just select and copy script text, and:

$ p > ~/bin/winexec
$ chmod 0755 ~/bin/winexec

Similarly, I sometimes want to search and replace on text on the clipboard:

$ p
Similarly, I sometimes want to search and replace on text on the clipboard:
# (showing what's on the clipboard)
$ p | sed 's| |_|g' | c
# (replace all spaces with underscores)
$ p
Similarly,_I_sometimes_want_to_search_and_replace_on_text_on_the_clipboard:

A not usually unwelcome side-effect of my clipboard commands is that they normalize line endings and add a newline sequence at the end of the text, if there isn't one already.

I hope I've given a few folk some ideas about optimizing their environment, particularly if they're command-line junkies like me.

In Defense of Steve Vinoski and Erlang

2008-05-18T20:16:00.002+01:00

There's been a minor scuffle going on between Ted Neward and Steve Vinoski over the wisdom of Erlang's approach to concurrency: whether it should be baked into the language or not on one hand, and whether it should be running on the JVM or CLR on the other.

I've already articulated my position on VMs, and I think it makes a lot of sense, particularly for prototyping, to build a VM specifically for a language implementation, particularly if the language has some primitives that are not normally available in commodity VMs. And to be frank, if one's language doesn't have some interesting new primitives or combination of primitives, it is unlikely to be moving the state of the art forward.

Erlang uses the actor concurrency model combined with lightweight aka green threads, though the execution engine may spawn as many threads as needed in order to get genuine concurrency from an underlying parallel architecture, such as multiprocessor or multicore. Erlang works under a shared-nothing model, however, so the "green threads" are more like "green processes".

I think Ted misses some appreciation of the power of the Erlang model, and in particular, its choice of primitives. Ted points to an implementation of the Actor model using Lift (written in Scala), and in particular some ballpark performance numbers:

We also had an occasion to have 2,000 simultaneous (as in at the same time, pounding on their keyboards) users of Buy a Feature and we were able to, thanks to Jetty Continuations, service all 2,000 users with 2,000 open connections to our server and an average of 700 requests per second on a dual core opteron with a load average of around 0.24... try that with your Rails app.

One of the obvious problems with this comment is that it doesn't sound very impressive when actually compared with Yaws, implemented in Erlang:

Our figure shows the performance of a server when subject to parallel load. This kind of load is often generated in a so-called "Distributed denial of service attack".

Apache dies at about 4,000 parallel sessions. Yaws is still functioning at over 80,000 parallel connections.

What this disparity tells me is that the JVM and CLR are likely lacking some primitives that help Erlang achieve this kind of result. For straight-up processing code, Erlang isn't terribly fast, by all accounts. The reason it wins is likely because it is avoiding the context-switching overhead through the use of lightweight processes. This in turn suggests to me that the equivalent of green threads, or some kind of automatic CPS transformation or native support for continuations is needed for CLR and JVM to be credible target platforms for Erlang. Lift is currently using Jetty Continuations, and when you read up about its implementation:

Behind the scenes, Jetty has to be a bit sneaky to work around Java and the Servlet specification as there is no mechanism in Java to suspend a thread and then resume it later. The first time the request handler calls continuation.suspend(timeoutMS) a RetryRequest runtime exception is thrown. This exception propagates out of all the request handling code and is caught by Jetty and handled specially. Instead of producing an error response, Jetty places the request on a timeout queue and returns the thread to the thread pool.

When the timeout expires, or if another thread calls continuation.resume(event) then the request is retried. This time, when continuation.suspend(timeoutMS) is called, either the event is returned or null is returned to indicate a timeout. The request handler then produces a response as it normally would.

Thus this mechanism uses the stateless nature of HTTP request handling to simulate a suspend and resume. The runtime exception allows the thread to legally exit the request handler and any upstream filters/servlets plus any associated security context. The retry of the request, re-enters the filter/servlet chain and any security context and continues normal handling at the point of continuation.

... you can see that it's clearly a hack to work around the limitations of the JVM - i.e. the fact that it doesn't have user-schedulable green threads (on top of native threads, not as a replacement, a bit like fibers on Windows), or an automatic CPS with some AOP-style weaving, or native continuation support.

In conclusion, the primitives in the VM matter a great deal. With the right primitives, wholly different styles of application become possible, because of the radically different performance profiles.

The point of VMs

2008-05-18T19:35:00.002+01:00

There was a post on the Delphi newsgroups that stuck in my head for some reason, and I felt I had to write a reply. The reply ended up being a lot longer than I originally intended, because I felt I had to justify my stance. I'm reposting it here in edited form.

"Virtual machine" has acquired pejorative overtones due to historical and social reasons that are probably too emotive to go into. Suffice it to say that I think it's another case of "good ideas don't win, proponents of bad ideas die out instead".

The way I see it, a virtual machine (in the context of programming language implementations) is a software implementation of an abstract machine with a closed-by-default set of semantics.

Let's take that definition apart:

software implementation: Here, I don't mean that the machine cannot be implemented in hardware. Rather, I mean that if it's going to be "virtual", it is usually implemented in software, which gives rise to certain characteristics, which in turn imbue "virtual machine" with extra shades of meaning. It turns out that software implementation is better than implementing in hardware, largely because of flexibility.
abstract machine: Every programming language has an abstract machine implicit or explicit in its definition, or otherwise its promised semantics are meaningless - you need a machine at some point to actually do things, and have effects. So, the abstract machine bit isn't controversial; it's its qualities that matter. Note that I differentiate between two different abstract machine concepts: a language's abstract machine, which it uses to model effectful operations, and a platform as an abstract machine. A CPU (+ memory + etc.) specification is an abstract machine, and a platform; the physical device, however, is a real machine, running on the laws of physics.
closed-by-default semantics: Here, I mean that at the abstraction level of the abstract machine in question, undefined behaviour is outlawed. In defining our machine, we humbly accept our human frailties, and do our best to prevent "unknown unknowns" becoming a problem by reducing the scope of the problem domain. We limit the power of the machine, in other words.
Since we do, eventually, want to be able to talk to hardware, legacy software and the rest of the real world, there do need to be carefully controlled holes and conduits built-in. But they're opt-in, not opt-out.

Let's look at some of the ramifications of this conception of VMs.

Software implementation delivers a tremendous amount of flexibility. Some examples: runtime metaprogramming (e.g. runtime code generation, eval); dynamic live optimization (e.g. Hotspot JVM [1]); auto-tuning garbage collection; run-time type-aware linking (solving the template instantiation code-bloat problem); rich error diagnostics (e.g. break into REPL in dynamic languages).
Abstract machine: Developments in programming language fashions have made object orientation come to the fore (perhaps even too much to the fore). However, our physical machines map much closer to procedural code and a separation between code and data than the trends in language and architecture design.
In other words, the platforms that historically popular type-unsafe [2] languages (like C++ and Delphi) have targeted aren't a close match for those languages' abstract machines. When they want to interoperate, either with other modules or with modules written in different languages, they face barriers, because their common denominator is the abstraction of the physical CPU. Hence C-level APIs being de facto industry standards, along with limited attempts to raise the abstraction level with COM (largely defined at the binary level in terms of C, explicitly referring to vtable concepts that are otherwise just hidden implementation details of other languages).
So, moving the abstraction level of the target machine closer to the average language abstract machine makes compiler implementation easier, reduces interoperation barriers, and provides more semantic content for the (typically) software implementation to work its flexibility magic.
Closed-by-default eliminates whole categories of bugs. Type-safety can be guaranteed by the platform. Never again [3] have a random memory overwrite that shows up as a crash 5 minutes or 5 hours later. It also improves security [4] by having a well-defined whitelist of operations, rather than trying to wall things in with blacklists and conventions ("this structure is opaque, only pass to these methods" etc.).

[1] Some notable optimizations that become feasible when the program is running live include virtual method inlining, lock hoisting and removal, redundant null-check removal (think about argument-checking at different levels of abstraction), etc. Steve Yegge's latest blog post, while rambling, covers many optimizations that apply equally to static languages running in a virtual machine and to dynamic languages (but of course he's interested in promoting them as the apply to dynamic languages):
http://steve-yegge.blogspot.com/2008/05/dynamic-languages-strike-back.html

[2] Any language that has dynamic memory allocation that it expects to be reclaimable (i.e. no infinite memory) and doesn't have a GC isn't type-safe. A single dangling pointer to deallocated memory kills your type safety: if a value of a different type gets allocated at the same location, you have a type violation.

[3] Unfortunately, RAM may occasionally flip bits due to cosmic rays etc. So, we want to use ECC RAM and checksum critical structures when it matters. Edge case nit.

[4] IMO, the capability-based security model is the best of those available, ideally including eliminating ambient authority.
http://en.wikipedia.org/wiki/Capability-based_security
Guess what: you need a type-safe virtual machine to make some strong guarantees about capabilities, otherwise someone could come along and steal all your capabilities by scanning your memory.
See Capability Myths Demolished for more info:
http://srl.cs.jhu.edu/pubs/SRL2003-02.pdf

In an odd coincidence...

2008-05-16T15:13:00.001+01:00

Jeff Atwood has a new post up about forking open source projects, and in particular, pointing out how difficult it is. This very closely corresponds with the point I made yesterday...

The new "Open" isn't as open as it seems to be

2008-05-16T00:09:00.003+01:00

I was reading Scott Koon's (lazycoder) post about RIAs (rich internet applications) being a platform play, what with Adobe AIR, Microsoft Silverlight and JavaFX. Scott noted that these platforms are all "open enough" that independent implementations can be made.

This idea reminded me of a podcast on open source business models with Dirk Riehle that I listened to a number of weeks back. The core ideas are also on the Dirk's website.

Dirk makes clear a number of features of the open software market. In particular, he distinguishes between community open source and commercial open source, where a single vendor controls the direction of the project and employs the key contributors. Even though e.g. Silverlight isn't exactly open source, Microsoft has been open enough to let Novell's Moonlight be implemented. The strategies of Sun and Microsoft for their new platforms appear to be based around the ideas of commercial open source. Their profit items are hardware and services in the case of Sun, and OS and database licenses in the case of Microsoft. What Adobe's long-term strategy for monetizing its platform play isn't yet clear to me, but when you've got folks locked in, you can start selling your captive audience one way or another.

Anyhow, my point is that this new openness isn't as open as it seems to be. Community open source needs strong personalities to deliver direction ~~, and even then, it seems to work best when reimplementing a previously proprietary, closed-source technology~~. Commercial open-source has several key advantages in controlling the platform, because focus and longer-term strategy means it can usually force any potential competitors into trying to keep up with it, rather than forking and going their own way. Open platforms tend to rally around key personalities, which essentially become brands, and the commercial style means that the body corporate owns that brand. Microsoft can come out with all the IronPython and IronRuby they like, but they are unlikely to get much acceptance if they try to introduce features that Guido or Matz don't agree with. More corporately, it's unlikely that many competitors to Sun would ever become "go-to guys" for issues with the Java platform (though if it would be anybody, I'd bet on Azul in the long term).

Upshot is, going open isn't as open (or risky) as it might seem, provided that you've got something else to sell alongside.

Update: Fuzzyman in the comments has convinced me that I was wrong to suggest that the most used open source isn't that innovative, hence the overstrike. However, that wasn't really essential to the point of the post...

1001 Books you must read before you die

2008-05-13T01:38:00.004+01:00

A long list of books, with many titles where I've said "I must read that some day". I read a lot more when I was a kid, before I got into this whole computer racket, and started focusing on more non-fiction. I've only read 50 or so from the list. At this rate, I'll never get more than 15% or so.

I wonder how much you can tell about a person from their reading list? Here are the ones I read, I may have missed one or two though:

Slow Man – J.M. Coetzee
Choke – Chuck Palahniuk
Super-Cannes – J.G. Ballard
Memoirs of a Geisha – Arthur Golden
The Information – Martin Amis
Time’s Arrow – Martin Amis
London Fields – Martin Amis
The Long Dark Teatime of the Soul – Douglas Adams
Dirk Gently’s Holistic Detective Agency – Douglas Adams
The Old Devils – Kingsley Amis
Money: A Suicide Note – Martin Amis
The Hitchhiker’s Guide to the Galaxy – Douglas Adams
High Rise – J.G. Ballard
One Hundred Years of Solitude - Gabriel García Márquez
The Third Policeman – Flann O’Brien
One Day in the Life of Ivan Denisovich – Aleksandr Isayevich Solzhenitsyn
A Clockwork Orange – Anthony Burgess
Stranger in a Strange Land – Robert Heinlein
Catch-22 – Joseph Heller
The Tin Drum – Günter Grass
Breakfast at Tiffany’s – Truman Capote
The Lord of the Rings – J.R.R. Tolkien
Lord of the Flies – William Golding
Foundation – Isaac Asimov
The Catcher in the Rye – J.D. Salinger
I, Robot – Isaac Asimov
Nineteen Eighty-Four – George Orwell
Animal Farm – George Orwell
The Glass Bead Game – Herman Hesse
Of Mice and Men – John Steinbeck
The Hobbit – J.R.R. Tolkien
Brave New World – Aldous Huxley
The Castle – Franz Kafka
The Trial – Franz Kafka
A Passage to India – E.M. Forster
Heart of Darkness – Joseph Conrad
The Hound of the Baskervilles – Sir Arthur Conan Doyle
The War of the Worlds – H.G. Wells
The Invisible Man – H.G. Wells
Dracula – Bram Stoker
The Island of Dr. Moreau – H.G. Wells
The Time Machine – H.G. Wells
The Adventures of Sherlock Holmes – Sir Arthur Conan Doyle
The Adventures of Huckleberry Finn – Mark Twain
The Brothers Karamazov – Fyodor Dostoevsky
Around the World in Eighty Days – Jules Verne
Through the Looking Glass, and What Alice Found There – Lewis Carroll
War and Peace – Leo Tolstoy
Crime and Punishment – Fyodor Dostoevsky
Alice’s Adventures in Wonderland – Lewis Carroll
A Christmas Carol – Charles Dickens
Castle Rackrent – Maria Edgeworth
Aesop’s Fables – Aesopus

HT to Jason Kottke.

C++ evaluation order gotcha

2008-04-08T00:16:00.005+01:00

My girlfriend was looking through an introductory C++ programming book the other day, but while she was comparing actual results with expected results, she saw a discrepancy and I was called in to figure out what was going on.

The program is pretty simple:

#include <iostream>

int f(int x, int y)
{
    std::cout << "f()\n";
    return x + y;
}

int main()
{
    std::cout << "Calling f()\n";
    std::cout << "Result is: " << f(3, 4);
    std::cout << "\n";
    return 0;
}

The book indicated that it should print out the following, which indeed it does (using bcc32 6.10, g++ 3.4.4 and cl 9):

Calling f()
f()
Result is: 7

The question my astute girlfriend has is: why is 'f()' printed out before 'Result is: ', when 'Result is: ' appears earlier in the expression? In other words, why doesn't it print out:

Calling f()
Result is: f()
7

... as you might expect, following the logic from top to bottom and left to right?

The answer comes down to how C++'s designers have, in their infinite wisdom, decided to implement I/O. The left shift operator, '<<', is overloaded and so the critical expression gets parsed as a set of function calls, roughly like this:

operator<<(operator<<(std::cout, "Result is :"), f(3, 4));

This is somewhat explained on the MSDN page allegedly describing Order of Evaluation (C++).

So, the most nested things need to get evaluated first, that's pretty unavoidable. Because of the associativity of the '<<' operator, things on the left end of the line get printed first, followed by each successive item.

Here's where the subtlety is, though: that most noble order, C++ definers, have decreed that the order of evaluation of arguments is beneath their dignity; that is to say, order of argument evaluation is undefined.

However, there are some general things we can say about C++ as implemented on x86. It usually defaults to the C calling convention, which pushes arguments on the stack from right to left, in order to support varargs. Since the arguments need to be pushed from right to left, the compiler will generally take a hint and evaluate them from right to left too. That's where our strange out of order result comes from.

Delphi isn't much less guilty on this specific front, however: it doesn't have a standard in the first place, and it also generally evaluates arguments from right to left. That's often more efficient when starved for registers: usually one will want to use EAX, EDX and ECX - the registers available for Delphi in the default calling convention - for purposes of evaluating the excess arguments (if more than three), which will end up pushed on the stack. If the first argument, which will ultimately end up in EAX, got evaluated first, and so on, many valuable registers would be tied up only to calculate values that end up pushed on the stack, no longer needing a register. The alternatives, spilling to the stack or evaluating to a temporary, are wasteful.

However, for the specific instance of I/O - Writeln - Delphi breaks up multiple outputs into separate calls. So, our code, in Delphi:

 Writeln('Result is: ', f(3, 4));

get translated, behind the scenes, into something vaguely like:

Write0LString(Output, 'Result is: ');
WriteLong(Output, f(3, 4));
WriteLn(Output);

... with much less confusion for the beginner.

Procedurally-typed expressions redux

2008-03-13T04:01:00.004Z

A few days ago I blogged about some issues that the Delphi parser has with certain constructs. A little discussion ensued in the comments, so I think I want to clarify what I meant a little.

Specifically, there were two problems I pointed out:

Calling procedurally-typed values in non-trivial expressions
Creating procedurally-typed values from no-arg procedures (and functions and methods etc.) that appear in slightly non-trivial expressions.

The first problem can be solved by interpreting the appearance of an argument list (introduced by '(') in an expression as an attempt to call what is effectively the left hand side of the '(' operator. The second problem can be solved by more deeply analysing the expression tree during argument processing so that it isn't confused by something as simple as an extra set of parentheses. The first problem is more pressing as it's affecting certain features I'm trying to get into the product.

Nothing will be imposed. There will be no breaking changes in syntax or semantics, if it can possibly be helped. All that should change is that previously invalid code becomes valid.

While there is an ambiguity when looking at some hypothetical function to a function to a function, where, when calling f()(), it might appear unclear from the rules which function is being called, this syntax is currently not valid at all. Currently, the first set of parentheses is parsed as part of the first call to f, but the attempt to call the return value (of type function to a function) will fail. Extending this so that the second call succeeds shouldn't be problematic, because there's a simple rule: the '()' on a no-arg function is optional, but it will be parsed if it's found. It can't be "delayed", such that the raw 'f' is interpreted as a function call and then the '()' applied to the return value. The parser eats the tokens when it sees them and it has a rule for matching them.

The further testing I did for this post exposed another problem, potentially more severe than the others - it's a type hole:

{$apptype console}
{$T+} // you'd like typed-@ to help you here, but it doesn't...

type
 TF = function: Integer;
 PF = ^TF;

function MyF: Integer;
begin
 Result := 42;
end;

// P is separate procedure so 'f' stands out as a stack variable
procedure P;
var
 f: TF;
 x: PF;
 // use absolute to get around oddities of procedural types
 f1: Integer absolute f;
 x1: Integer absolute x;
begin
 Writeln('MyF is at ', Integer(@MyF));
 Writeln('f is at ', Integer(@@f));
 f := MyF;
 Writeln('f is pointing to ', f1);
 x := @f;
 Writeln('x should be pointing to f, but is pointing to ', x1);
 // The following line doesn't do what you'd expect: it crashes.
 // Writeln(x^);
end;

begin
 P;
end.

Unhappily enough, this prints out the following on my machine, with no warnings or errors during compilation:

MyF is at 4211536
f is at 1245092
f is pointing to 4211536
x should be pointing to f, but is pointing to 4211536

The line commented out does in fact crash.

Update: Craig in the comments asks why it's wrong. There are at least two things that could be wrong, and one of the unfortunate problems here is that there's no standard for Delphi beyond what the compiler currently does, so we can't say for sure. Either the '@f' when assigned to x should take the address of f rather than just inhibit calling the function pointer, or it should result in a compiler error - particularly as typed-@ operators are turned on here. The commented out line, x^, ought to call the function pointer being pointed to by x, but it doesn't, since it's simply pointing to the function 'MyF', rather than the function pointer 'f'.

Not a Delphi post: PC Games

2008-03-11T14:57:00.004Z

A couple of things I saw yesterday got me thinking about games, something I've been meaning to write down my thoughts about for some time. From long-time experience, I've gathered that my tastes in games are somewhat (but only somewhat) off the beaten track, if only because of the lack of market supply for what I like to play.

First off, "PC game" or "video game" etc. is too broad a category to talk about meaningfully, so it needs to be broken up. However, I'm not so happy with the usual division of games into genres like real-time strategy (RTS), turn-based strategy (I'll call this TBS), first-person shooter (FPS), third-person shooter (I'll call this TPS), etc., though sometimes those divisions accidentally coincide with the divisions I have in mind.

I look at games using several axes: the kind of skills being exercised, the depth of immersion and storytelling, and the nature of the in-game stress / release cycle and pacing.

By the kinds of skills, I mean things like strategic thinking (e.g. Civilization - non-repeatable long-term plans to achieve goals), tactical thinking (e.g. most FPSes - repeatable short-term templates of action that solve categories of problems), mastery (e.g. simulation games), hand-eye coordination (twitch games like almost every arcade game, and also including many FPSes, RTSes and TPSes)

By the depth of storytelling, I'm talking about how essential the story is to one's experience of playing the game. Is the story just an add-on to rationalize the gameplay? Is it just very effective atmosphere, made effective by how the level design directly reflects the story? Or is the story essential to successfully playing the game, or perhaps even ultimately user-written, as they make choices in the world? And by immersion, how easily can I forget than I'm at a computer, and instead think of myself as being a character in the game?

When I talk about an in-game stress / release cycle, I'm thinking in particular of how stress builds up in the player, to what degree it builds up, how it is built up, what options the player has to reduce this stress, and the pacing refers to how long the stressful periods are versus the (relatively) relaxed periods. For example, many FPSes have "action bubbles", where lots of bad guys surround some kind of goal than the player usually has to physically get to. If the FPS is heavily corridor-oriented (e.g. pretty much all the Doom games), where there's really only one way to go, then the player will probably have to kill almost every bad guy met, and can only stop for a breather if any pursuing bad guys have been killed off. On the other hand, there may be relatively quiet corridors linking areas that trigger action after a certain point is reached, something done very well by Half-Life (the original).

My favourite games have been Thief series, Far Cry, Deus Ex, Outcast and Civilization 2, roughly in that order. I've enjoyed playing games like Doom, Half-Life, Crysis, Bioshock, Oblivion, System Shock 2, but none of them had the long-lasting appeal to me that my favourites have. I tried to replay Half-Life several years ago, but it didn't take long in-game before I discovered I'd lost my love for it. On the other hand, I've replayed Thief 2 (in particular) pretty much at least once a year since it came out - an older game (older technology-wise as it uses the same engine as Thief 1, which was weak compared to the upgraded Quake engine behind Half-Life, both being 1998 vintage), but one with far more potency for me.

So, coming back to those axes, what do I like in a game? Story and immersion is very, very important, probably the most important thing. Almost all the games above are FPSes, because that's the format which lends itself best to immersion. Civilization is so sparse on the story side - the only real identity you have is as a nation of peoples, rather than a character - that one's imagination fills in the gap. This usually leads to even more identification with the game than a heavy-handed plot, because it's entirely user-driven. Far Cry has the weakest story of the lot, but its immersion is extremely effective - you really do feel like you're on a tropical island, sneaking through the bushes. Far Cry also makes up for the weak story on the other axes. The other games, in particular the Thief series, are strongly story-driven, but they don't lack in the other axes. Deus Ex has a degree of user participation in the story creation, as certain choices you make affect the plot fairly substantially - for example, you can choose to save your brother or leave him to die (which he urges you to do). The music in Outcast is amazing and highly atmospheric. The feeling of being in an Arabic-style souk is pretty strong in the city of Okriana in Talanzaar, and I have visited e.g. the souks of Marrakesh, with that atmosphere coming back to me.

I like to use strategic skills over tactical or twitch skills in games. So, even though Doom can be fun for a 30-minute blast (on a Windows 3D port, with jump and mouse-look in the Y dimension enabled), it's not what gets me hooked. The Thief series' player character, Garrett, is sneaky enough to be able to assess the ground and the disposition of enemies without being seen. The player's job is then to plan and navigate a path through the ground without being seen or heard, within the resource constraints of the gadgets available. A good friend of mine, far and away my better in twitch games like Quake multiplayer, found progressing in e.g. Thief 2's Bank level very difficult - he got stuck and ultimately bored in many situations that simply increased my appetite for the problems. Civilization is all about strategy - there isn't even a constraint on turn time. Far Cry, Outcast and Deus Ex have sufficient freedom of movement that you can usually plan your encounters from afar and engage on your own terms. All things considered, I prefer engaging from a position of strength (usually means being higher up) and unseen where possible. Up-close and twitchy is too risky: I don't like dying in a game, because that breaks my immersion. In real life, there's no way back from death: why should I play my games like it was any different? Evaluating the enemy, navigating unseen to that position of strength and planning the encounter is usually more fun for me than the encounter itself; however, there is the gratification of a plan well executed and performing as anticipated, something very akin to the pleasure of programming. It also turns out that freedom of movement is very important to permitting a strategic approach to the game: strictly linear games like Half-Life 2 almost actively prohibit strategic thinking.

About the stress & release cycle: one is usually stressed in a game because one is threatened with in-game death. There are different graphs one could draw of the typical stress patterns of different games and gaming styles. A game like Doom 3 or Half-Life 2 often rely on "surprises" - quiet areas / dark alcoves which are obvious traps, and the only "surprise" is the exact trigger location or time. This is probably a feature I hate most about some games: knowing that something bad is going to happen, but not being able to control it. Half-Life wasn't too bad, because one had tactical weapons like grenades and strategic weapons like trip-mines wherein action bubbles could be analysed and solved pre-trap-triggering, but Half-Life 2 (and especially Doom 3) used such cheap tension tactics that I stopped playing the games in disgust. These games often have a ::^^::^^::^^:: kind of tension graph: medium tension interspersed with high tension. On the other hand, a game like Thief 2 looks more like ..:..:..:^^.^^.^^:...: lots of low-tension observation, occasional medium-tension moments when encountering danger areas while scouting, followed by a long stretch of alternating high-stress navigation and resting safely in shadows. The key is that the player has control over the resting and the initiation of the next high-stress section. I far prefer games where I have control over the stress level pacing and have time to consider my options before proceeding.

If you've read this far, well done for putting up with me! I just wanted to get those thoughts out of my head, but if it's been interesting to someone else, then that's gravy.

Odd corner of Delphi: procedural expressions are not expressions

2008-03-07T21:03:00.004Z

Here's an odd corner of Delphi: expressions of a callable type (e.g. a function pointer or method pointer) are not fully-fledged expressions; or rather, the expression grammar for function application doesn't acknowledge function pointer types as proper first-class parts of the type system. For example:

type
  TFoo = procedure(x: Integer);
  TBar = function: TFoo;

var
  bar: TBar;
begin
  bar(42); // doesn't compile; too many args for TBar
  (bar)(42); // doesn't compile; bar's value doesn't flow through
  bar()(42); // still no good; bar()'s value doesn't flow through
  (bar())(42); // again, no good, for same reasons as previous two
end.

The reasons are probably directly related to two things: the historical lack of general function pointers in Pascal and the rule that no-argument procedures can omit the parentheses.

The following are valid expressions that call the function pointer, but the resulting value can't be invoked in turn:

  bar; // ok: return value dropped
  bar(); // ok: return value dropped

The above results in some serious drawbacks if one wants to program Delphi in a functional style. Currying can't work, for example: there's no way you could turn foo(1, 2, 3) into foo(1)(2)(3), since the latter syntax isn't valid.

I personally consider it a bug, but this behaviour, at the heart of expression parsing, is tricky to fix while guaranteeing not to break anything that already works. For example, there are other odd semantics around these Delphi function pointers:

type
  TFrob = function: Integer;

function MyFrob: Integer; begin Result := 42; end;

procedure Baz1(x: TFrob); begin end;
procedure Baz2(x: Integer); begin end;

procedure BazO(x: TFrob); overload; begin end;
procedure BazO(x: Integer); overload; begin end;

begin
  Baz1(MyFrob); // passing MyFrob
  Baz2(MyFrob); // calling MyFrob and passing result
  BazO(MyFrob); // guess which?
end.

The above call to Baz1 is currently handled, believe it or not, by parsing as a function call and then later, in the middle of function application, throwing away the call bit and turning it back into taking the address of the function. You can see this by hiding the function call just a tiny bit, so the function application logic can't see it so easily:

  Baz1((MyFrob)); // no longer passing MyFrob

The lesson to me in the above case is pretty clear. If you want to have function pointers in your language, you shouldn't make function application look like a function value - that way madness lies. This is why the '@' operator was invented:

  Baz1((@MyFrob)); // works again; passing MyFrob

Unfortunately, its use in Delphi for these cases is optional.

The two issues discussed in the above post are distinct. However, I may need to fix the first one in order to get certain features into the product...

Lang.NET Symposium videos are up - in WMV

2008-02-20T22:15:00.004Z

The Lang.NET symposium videos are up, but the user interface over there is pushing Silverlight. The videos are actually in WMV format. The URLs are hidden in StartPlayer.js. If you want to avoid having to install Silverlight, these urls should do (mind the spaces):

http://langnetsymposium.com/talks/Videos/1-00 - Keynote - Jason Zander.wmv
http://langnetsymposium.com/talks/Videos/1-01 - CSharp3 - Anders Hejlsberg.wmv
http://langnetsymposium.com/talks/Videos/1-05 - Lively Kernel - Dan Ingalls - Sun.wmv
http://langnetsymposium.com/talks/Videos/1-06 - JScript - Pratap Lakshman.wmv
http://langnetsymposium.com/talks/Videos/1-07 - Irony and ERP Language Challenges - Roman Ivantsov.wmv
http://langnetsymposium.com/talks/Videos/2-00 - Democratizing the Cloud with Volta - Erik Meijer.wmv
http://langnetsymposium.com/talks/Videos/2-01 - Newspeak - Gilad Braha - Cadence.wmv
http://langnetsymposium.com/talks/Videos/2-02 - Resolver One - Giles Thomas - Resolver.wmv
http://langnetsymposium.com/talks/Videos/2-03 - Retargeting DLR - Seo Sanghyeon.wmv
http://langnetsymposium.com/talks/Videos/2-04 - Visual Basic - Paul Vick.wmv
http://langnetsymposium.com/talks/Videos/2-06 - PHP - Wez Furlong.wmv
http://langnetsymposium.com/talks/Videos/2-07 - Phalanger - Tomas Petricek.wmv
http://langnetsymposium.com/talks/Videos/2-08 - Pex - Peli de Halleux.wmv
http://langnetsymposium.com/talks/Videos/2-09 - Numerical Computing with the CLR - Jeffrey Sax - Extreme Optimization.wmv
http://langnetsymposium.com/talks/Videos/2-10 - remotion Mixins - Stefan Wenig and Fabian Schmied - rubicon.wmv
http://langnetsymposium.com/talks/Videos/2-11 - CodeIt - Serge Baranovsky - submain.wmv
http://langnetsymposium.com/talks/Videos/3-00 - IronRuby - John Lam.wmv
http://langnetsymposium.com/talks/Videos/3-01 - Ruby.NET - Wayne Kelly.wmv
http://langnetsymposium.com/talks/Videos/3-02 - FSharp - Luke Hoban.wmv
http://langnetsymposium.com/talks/Videos/3-03 - Parsing Expression Grammars in FSharp - Harry Pierson.wmv
http://langnetsymposium.com/talks/Videos/3-04 - NStatic - Wesner Moise - SoftPerson.wmv
http://langnetsymposium.com/talks/Videos/3-05 - Moonlight and Mono - Miguel de Icaza.wmv
http://langnetsymposium.com/talks/Videos/3-06 - Visual Studio Shell - Aaron Marten.wmv
http://langnetsymposium.com/talks/Videos/3-07 - Modeling and Languages - Don Box.wmv
http://langnetsymposium.com/talks/Videos/3-07 - Modeling and Languages - Don Box_1.wmv
http://langnetsymposium.com/talks/Videos/3-08 - Cobra - Chuck Esterbrook.wmv
http://langnetsymposium.com/talks/Videos/3-09 - Intentional - Magnus Christerson.wmv

You didn't read it here :)

Edit: As nbrooks commented, you'll need to install the WVC1 codec to view the videos (if you haven't actually installed Silverlight). You can get it here: http://support.microsoft.com/kb/942423

Continuations: Wistfully yearning

2008-02-02T05:47:00.000Z

During the past week, there was a conference at Microsoft's campus in Redmond, Lang.NET. I didn't attend, but I did look over again at the videos from the previous speakers (one of which was by Danny Thorpe).

However, it wasn't Danny's talk that I found most interesting. Instead, it was Shriram Krishnamurthi's presentation on continuation-based web application programming. Ian Griffiths wrote a disparaging post on continuations describing how they can work in a web application, in case you're not aware of their nature. I strongly disagree with Ian, though, to the point that I'm not actually sure if he's willfully misrepresenting those who would advocate continuations.

In my previous job, we developed a web application framework, running on top of ASP.NET, for data-entry intensive applications in the finance industry. The product is now called Topoix. I contributed a bunch of things to the underlying architecture and implementation, but one of them is particularly relevant to continuations: the way Topoix implements modal dialog boxes that ask the user a question. Essentially, displaying a dialog caused execution of business code on the server to stop and resume correctly, based on the reply, when the dialog was dismissed. The stopping and resuming didn't tie up a thread on the server, and nor did it require stickiness to a particular server in the farm, but obviously it required some state - persisted to disk and perhaps cached in memory, depending on how you wanted to configure things.

Now, I want to address some of the (IMHO) mistaken criticisms in Ian's article.

This requires a certain amount of magic under the covers. Supporting continuations in special cases such as function calls or iterators is much easier than providing completely generalised support. Continuations do not fit all that well with the stack-oriented execution model offered by the JVM or CLR.

I agree to the point that code that hasn't been transformed into continuation-passing style doesn't fit well with a stack-oriented execution model. But, however, bear in mind that translation from stack-orientation to continuation-passing style is a safe and semantics-preserving operation. Things really only get hairy if one uses callbacks from non-managed code, and one expects to be able to pass around a continuation with the non-managed code on the stack.

Ian continues:

Continuations can look attractive on the web, because they offer a tool that lets you capture the shape of a sequential user journey in the structure of your code.

Continuations can do more than capture sequential user journeys; if you take care to capture values, rather than references, they can capture an arbitrary history tree of independent, concurrent user journeys. This is because the stack in CPS is essentially converted into a linked list of activation records in the heap, where each activation record points to its parent. It's trivial to see that continuing at an earlier point (equivalent to pressing the back button in your browser) simply creates a new activation record which points to some tail of the linked list, and doesn't disrupt the concurrent "main trunk" of interaction. Providing that you don't mutate variables from previous activation records (unless that's desirable - consider e.g. site preferences, or login status), you don't have a problem with non-sequential navigation.

However, I think this is a bad idea. Although the relationship between the code and the user navigation path is apparently simple, it hides subtle but significant details. This makes the same mistake as we did in the 1990s with distributed object models. We’d like to believe we’re hiding a lot of implementation details behind a simple abstraction. In practice we’re hiding important features behind an inappropriate abstraction. Our industry has more or less learned the lesson that procedure calls are an unworkably naive abstraction for request/response messaging over a network. I suspect that it’s equally naive to attempt to manage the complexities of user/web server interactions with simple sequential code.

Ian here is saying that bad abstractions are a bad idea. I fully concur; when building abstractions, it's important to abstract the correct things, and not encourage writing the wrong sorts of programs. There are a few things to say about this, though. The first is that the really egregious abstractions are those which enable writing programs that one would not actually *want* to write, if one fully understood the abstractions in hand. I'm thinking in particular here of cross-machine COM or CORBA: we should prefer REST or document-style SOAP for reasons that are beyond the scope of this post, but I'll just say that they relate to the nature of object orientation as used in the small, with shared memory, on local machines, and how that falls apart when distributed.

However, I think Ian will have to come up with something better than condemning continuations as only being capable of abstracting "simple sequential code", if for no other reason than it's a falsehood.

So, Ian tries to come up with other reasons to slay the continuation beast. However, it's worth bearing in mind, when considering his objections, what the alternative is, and then asking one's self if the alternative isn't actually a continuation written out the long way.

Let's see.

Abandoned Sessions. Sometimes the user just walks away.

Well, a continuation that one persists to a cookie, hidden form fields, encoded in the URL, or to server-side storage, as necessary / appropriate, puts this one to rest. And besides, how does one solve this issue in the average trivial shopping-cart site? One must save state in exactly the same locations, but you have to encode the resumption of logic yourself, rather than having it done for you. The abstraction doesn't look weak here.

Now, there are things Ian says that aren't related to continuations per se, but rather to coding a web application as if it had state.

Continuation-based web servers aren't a way of pretending that the server is stateful when it needs to be stateless. Rather, they're a way of easing the pain of getting back to where you left off. That's why, when Ian says:

The problem with this is that a lot of the techniques we have learned for resource management stop working. Resource cleanup code may never execute because the function is abandoned mid-flow.

... it's not very relevant. One can't be holding any non-persistable resources when persisting a continuation, and with language and library support, this can be flagged at runtime or ideally at compile time (I can't help it, I'm a static guy, even though it's not fashionable in the web world). And since the continuation-persisting code needs to, you know, actually persist the objects on the linked-list stack, it can trivially flag the obvious problems when it discovers something it can't persist.

Thread Affinity. With an ordinary sequentially executing function, I can safely assume one thread will run the function from start to finish. But if I’m using continuations to provide the illusion that I’ve got sequential execution spanning multiple user interactions, then I might get a thread switch every time I generate a web page.

I see this as kind of a bogus problem. How often, when writing an ASP.NET application, does one work with objects or resources that have thread affinity, and expect them to be on the same thread when you next enter the server? I should think one doesn't expect this at all - one doesn't normally have the ability, much less the "problem", of handing an object from one request/response cycle to the next.

Web Farms. This is essentially the same problem as the thread affinity issue, but at the machine level: in a web farm, your sequential function might end up executing on a variety of machines over its lifetime. However, you’d probably avoid this problem in practice using sticky sessions. (And unless your continuations are serializable across machine boundaries you’ll have to do this.)

I don't see any point to using continuations if they can't be saved and resumed at will, across process boundaries, machine boundaries, server restarts, etc. If you limit the power of the continuations you consider, it's pretty easy to condemn them as not being powerful enough.

The Back Button and Branching

This one’s the killer.

Your web site may present linear user journeys, but that doesn’t mean your users necessarily follow them. I often don’t.

I habitually do two things that will confound any web site that expects the user to do things in a particular order. First, sometimes I use the back button. Second, sometimes I bifurcate my navigation - I’ll open a link in another tab. Both of these will confuse any web site that thinks it knows what my ‘current page’ is. The notion of a current page is not enshrined in either HTTP or HTML, and I enjoy the flexibility this offers me when browsing sites. Indeed, it’s one of the reasons I really like tabbed web browsers.

Here, I don't know what Ian is talking about. I mean, I understand exactly what he's saying, but it's a complete non-sequitur to jump from this behaviour to saying that it's incompatible with continuations. Continuation state forms a linked list on the heap. One can share the tail of a linked list among an arbitrary number of heads with zero problems.

This means I have to write my function in such a way that it can cope not only being rewound, but also to being split so that multiple threads execute the function simultaneously, each taking different paths. But of course because I’m using continuations, each of these threads gets to use the same set of local variables. The fact that I enabled users to inject gotos into my code at will is now looking like a walk in the park - now they can add arbitrary concurrency!

Here, when Ian says "use the same set of local variables", he's talking about mutating the tail of the list, which might be shared by other heads. And he's right in one sense: if you modify the tail, you better mean it, because you're modifying shared state, not local state.

However, I have a couple of more aces up my sleeve. If desired, the transformation that turns sequential code into CPS can also turn mutating assignments into single static assignments. Single static assignment is a form used in some compilers in the back end to ensure that a variable can only ever have a single definition. If one could annotate one's variables, indicating if they should be shared across multiple continuations (stored in the tail) or strictly local to every continuation that redefines it (essentially, hidden in a scoping sense by the redefining activation record), this problem goes away.

The deeper objection I have to Ian's criticism is that you have to deal with this problem anyway, and program transformation just makes your life easier. Consider what you'd have to do to handle Back Button and Branching without using the approach sketched above. You have all the same problems. You still need to make sure that wherever you stuff away your state, that it isn't inappropriately used by the branches. SSA transformation can do this automatically and declaratively.

Consequently, this approach requires you to write your code in such a way that it can tolerate sudden halts, thread switches, rewinding, and forking of execution.

Sudden halts, thread switching, rewinding, forking: what is it in the tear-down and re-setup of the ASP.NET page cycle that stops you from seeing these things, if you choose to describe them that way? Tearing down the page is surely a violent halt. Setting up again, possibly on another machine, is surely a wrenching thread switch away from your carefully constructed state. To support rewinding / forking (aka Back button / new tab / window), you still need need to juggle your state so that it's correctly stored, keyed and associated with its page which was originally displayed to the user. But continuations can do *all* of these things with easier to read code, and selective SSA transformation turns a fiddly choice (should this data be in the database, in memory, in a cookie, querystring, hidden field, client-side state sent back through AJAX) into a declarative form which your Web library should be abstracting for you.

Does your web development environment supporting continuations? If not, why not?

I will give Ian something, though. It's certainly not true (and shouldn't be true) that one simply writes a sequential application using continuations without thought for the network and then expect it to work fine across the web. That can't happen. However, I do believe that with the correct libraries, language, annotations and transformations, a far better language for web apps can be created. Whether it'll be along the lines of Volta, or one of the frameworks described herein, I don't know. However, I strongly suspect new language constructs, or at least flexible DSL support, is necessary. I can even think of implementing continuations using monads, so perhaps it'll be Haskell!

Async via LINQ

2008-01-16T12:15:00.000Z

Andrew Davey has posted an interesting article on composing async operations that doesn"t require a spaghetti-code style state machine. It doesn"t look particularly hardened to exceptions on End* calls, but the technique is interesting, using the monad-like nature of LINQ to compose continuations.

I blogged about something very similar over a year ago, and it looks like it's getting easier to do now.

Monads in C#

2008-01-11T02:54:00.001Z

Probably the best outline of monads I've seen yet. Easy to understand for the C# programmer.

Volta, Continuation-passing style

2007-12-22T13:29:00.000Z

Over a year ago, I pointed to a particular pattern in C#, using delegates with asynchronous methods, that used continuation-passing style (CPS) to make writing scalable IO-heavy code easier:

A lot of work? Yes, but better than people who write business code for a living having to work in a nest of hand-written continuations just to get scalability.

If the folks advising asynchronous code are serious about recommending this approach to scalability, then the enabling tools need to be available. It's conceivable that the CLR could do this automatically, with some kind of [AutoAsync] attribute, along with a couple of utility methods to access the generated Begin/End pair via reflection or whatever. That would keep the C# language clean while leaving the CLR open to a lazy implementation approach (using ThreadPool threads), but also with the power to create the transformations discussed above.

Now it looks like some of the folks over at MS are having similar ideas, in their Volta preview. It's not quite the same, you have to write your async method in a particular way, but with the automatic transformation of the callsite into the appropriate format, it's a good step forward:

Asynchronous Calls
Hiding network latency requires asynchronous calls. In the first technology preview, Volta allows programmers to add asynchronous versions of methods on tier boundaries.

To make a method asynchronous, define the CPS-equivalent method signature and annotate it with the Async attribute. Volta will generate the body and modify the call-sites accordingly.
    [Async]
    public static void F(string s, Action k);
If the programmer invokes an asynchronous method F, then Volta will launch the invocation on another thread and invoke the continuation upon completion of the call to F.

I'm not entirely delighted with some of the ideology behind Volta - I don't think hiding the network's existence in and of itself is necessarily a good thing - but making the right kind of code easier to write is definitely a step in the right direction.

Cygwin

2007-12-10T18:03:00.000Z

It's been a long time since I last posted. I've been working on the Delphi compiler all the while, but with uninteresting bug fixes (usually too boring to talk about), learning more about the code base (not general enough to talk about) and working on new features (not public enough to talk about), it's been rare enough that I found something interesting enough to talk about. So, I've decided to take a different tack.

I've had the "opportunity" to set up a number of PCs and laptops recently for my personal use, general and development work. Quite a bit of customization is necessary before I'm comfortable with a system. Apart from the usual handful of utilities (Firefox with extensions, Winzip, Winamp, Forte Agent), there's one other thing that I can't really live without - Cygwin. The bash shell running in an rxvt terminal window is my main interface with Windows.

There's one big reason for this. The shell is very programmable, so any time I need to do work of a repetitive nature, I can write a script. Whenever I find myself doing similar things, I can refactor my work and write a new utility script containing the commonality. My life would be a lot harder without Cygwin. Sometimes I wonder why it isn't more popular than it is; but then, the default Cygwin setup isn't very appealing. I remember when I first installed it, and got to use the usual Unix utilities I was familiar with from Linux, only this time in Windows, but I was still using the crippled cmd.exe shell from NT, running in the crappy console that Windows provides by default. I'm going to write a series of posts on how I beautify my Cygwin environment.

Installation

The first step is to install Cygwin. I do this in a four-step process:

Download setup.exe from cygwin.com.
Run setup.exe, and tell it to download (but not install) everything to a local directory. I choose c:\cygwin\opt\var\cygwin-cache. I then rename the ugly directory name it generates (based on the mirror you chose) to something briefer.
I then run setup.exe again (setup.exe normally resides in c:\cygwin\opt\var\cygwin-cache on my system), and perform an install, selecting those packages that I know I want and need. Those include:
- All the programming languages, from clisp and SWI-Prolog through byacc and cocom to gcc and nasm.
- The text editor joe and the rxvt terminal.
- ImageMagick, Ghostscript, TeX - stuff I seldom use, but like to have available.
Finally, I add the main binaries directory (c:\cygwin\bin by default) to my system path.

Actually, I'm lying. I don't normally use setup.exe to download the cygwin-cache directory, since I already have one. I actually use wget to download the cygwin packages directly from a mirror on a monthly basis, but that presents a bootstrapping problem: one wouldn't normally have wget without already having installed cygwin.

In any case, once I've gotten Cygwin installed, it's time for me to set up my home directory (the ~ directory in Unix parlance; defaults to c:\cygwin\home\<username>). This is created the first time you run a Cygwin login shell, such as from the shortcut the setup.exe program creates (or by running bash --login). All the stuff up until now is pretty regular Cygwin orthodoxy: people usually stumble along this far all on their own. I know I did the first time, but I found bash running in NT's command window pretty unpleasant to use. Here's where I make Cygwin usable, at least to me.

Beautifying

The main objective is to get bash running inside rxvt. This can be done pretty trivially:

Create a shortcut to c:\cygwin\bin\rxvt.exe (assuming one chose the default install directory)
Modify the shortcut and change the target so that it has a few extra arguments: "c:\cygwin\bin\rxvt.exe -e bash --login".

That tells rxvt to use bash rather than sh as the shell, and tells bash to run ~/.bash_profile (amongst others) as its startup script, rather than ~/.bashrc.

However, the default shell is still pretty unappealing: an ugly white window with black text. That's why I use some customizations. Rxvt reads in .Xdefaults from your home directory when it starts up, even though it's a Windows program, not an X one. .Xdefaults uses a pretty simple line-oriented file format. '!' starts a comment, which extends to the end of the line. Here are some lines I add to make things more suited to my preferences:

Rxvt*geometry:120x43

Rxvt*background:rgb:00/00/20
Rxvt*foreground:rgb:C0/C0/C0

Rxvt*color1:red
Rxvt*color2:rgb:72/A0/32
Rxvt*color4:skyblue

Rxvt*color3:yellow
Rxvt*color5:magenta
Rxvt*color6:rgb:20/C0/A0
Rxvt*color7:rgb:C0/C0/C0

Rxvt*color9:red
Rxvt*color10:rgb:72/A0/32
Rxvt*color12:skyblue

Rxvt*color11:yellow
Rxvt*color13:magenta
Rxvt*color14:rgb:20/C0/A0
Rxvt*color15:rgb:C0/C0/C0

Rxvt*colorBD:rgb:C0/C0/C0

This remaps the 16 console colours a little, inverting the colour scheme. I get the names for these variables (called resources in X parlance) from the rxvt manual page, available from "man rxvt". Here are some other relevant lines in my .Xdefaults:

Rxvt*jumpScroll:True

! I use Dina-9, from the font Dina, which I installed separately
!Rxvt*font:-*-lucidatypewriter-medium-*-*-*-12-*-*-*-*-*-*-*
Rxvt*font:Dina-9

! I have an ego
Rxvt*title:Barry's Bash

! Scrollbar on the left looks odd.
Rxvt*scrollBar_right:True

! In case I forget something; this is somewhat better than cmd.exe's usual limit
Rxvt*saveLines:30000

Rxvt*termName:rxvt

! This is the number of pixels between text rows; aesthetic issue, depends on font and size
Rxvt*lineSpace:1

After having done this, I have something I can live with.

Broken by Default

2007-06-11T16:39:00.001+01:00

I recently had reason to use Firefox's bookmarks' feature, "Open All in Tabs". I was unpleasantly surprised to discover that it closed my open tabs, and navigated away from my existing pages (including breaking my VPN connection to Borland's network). It's one of those features that's broken by default, by design, because - well, I don't know why. Apparently there's been some secret discussion somewhere 3+ years ago, that decided that the current behaviour is best. I have no doubt that the reasons are bogus :)

Anyway, I found out about about:config setting browser.tabs.loadFolderAndReplace, which fixes this behaviour and sensibly creates new tabs, rather than overwriting your current tabs and bringing your network down around your ears.

Here's another, more verbose link, describing how to fix the broken behaviour. Yes, I'm editorializing, but I'm just pissed because I had to fix everything back up again, after this "feature" exploded in my face.

Acropolis: The Second Coming of MFC

2007-06-04T22:33:00.000+01:00

I notice that MS have released some early bits of a new rich client application framework. I also note that the XAML namespace prefix in the demo video ("Introducing Acropolis") is Afx, the same as the old MFC header file name.

The initial wizard looks eerily familiar, very MFC - but everything has been updated, and something that's particularly interesting to me is how the Application XAML - an object serialization format, not necessarily limited to UIs - is presented in a design view that looks quite similar to a web page. In a past life I worked on a system that used a lot of metadata, and one of the many design metaphors we looked at was document-like layout for non-document (or at least, non-traditional document) data.

I'm waiting for the day when people incorporate 3D zooming transitions into these document layouts, taking advantage of the rich client power and modern hardware / WPF, to pack more data in the spatial screen landscape. Defining a 2D/2.5D (I count zoomed depth as 0.5 of a dimension) mapping for an application's structure has for a long time seemed to me like a good way of getting a higher-level, more abstract visualization / editing experience for today's larger application source bases.

Having worked on some fairly complex systems from front to back, top to bottom, and known how everything was put together - having written most of it - it was always frustrating to me not having first-order unified view (as opposed to derived, or via stale documentation) that explained how the application fitted together.

My comment is at quite a tangent to the specifics of the Acropolis release, but this little artefact hinted of a direction that I hope development UIs take in the future. For what it's worth, I don't believe in UML - I'm talking about something which is qualitatively different, in that the visual mapping function is domain-specific and tailored to the different functional aspects of an application, is 2-way with its component structure, and has hyperlinks connecting related parts. Perhaps the mapping function could be rotated around different axes, to get different perspectives on the app. Making the source code for a system look more like blueprints. Musings for the future...

XML, the right format for object graph serialization

2007-05-22T06:08:00.001+01:00

I was reading this blog entry, and my annoyance meter was piqued:

One of the advantages of having a canonical representation of an object tree, is that it serves as a great interchange enabler. That canonical representation doesn't need to be Xml, but it seems like a good choice.

Sigh. The assumption of an "object tree" - object graphs usually aren't trees, they typically have lots of cross-references. Objects in memory form edge-labeled graphs, while XML is a node-labeled tree. Converting objects to XML requires some naming scheme for those objects which have relationships more complex than parent-child.

If you design your objects along functional lines and make them immutable after the constructor has finished executing, this scheme can work, but I don't believe there are many useful structures with no cross-references whatsoever. It works best if you have a rooted namespace context like TComponent and its Owner, or something similar where objects can be identified by a URL - ideally one and only one URL.

I believe that in some decades from now, we'll look back at this objectification of the world, this current habit of seeing the power of objects in e.g. GUI libraries and then applying the same hammer to all manner of delicate problems, as one of the bigger mistakes in the history of programming. Basically, both hierarchical data (e.g. XML) and relational data (collections of fact tuples) are bad fits for object orientation, and in the code side of the house (as opposed to data) lightweight processes (think Erlang) haven't had their due. And popular programming languages are starved of higher-order abstraction facilities.

I wonder how this is going to change. C# is gaining its extensions (and cruft - having both lambdas and anonymous delegates in the language is a bit of an eyesore), but it's still pretty poor in many other ways. It's very deeply wedded to imperative, linear thinking, with very few transformations before execution. Static typing is fine, but metaprogramming and dynamic programming's rise show why it is also constraining. Statically typed languages that lack a higher order, a way of programming over the type domain as well as the value domain, end up falling back to reflection and other weakly typed loopholes to overcome traditional (C, Pascal, Java) static typing's inflexibility.