Words in Boxes

Migrating repositories from Bitbucket to Github

2012-02-05T13:23:00.000-06:00

A few years ago, I decided to use Mercurial & Bitbucket instead of Git & Github for my public repositories, because Mercurial seemed simpler and worked better on Windows. But times have changed. I use OS X now, and Github has become both overwhelmingly better and more popular. So it’s time to pick up and move.

This is a guide on how to do just that. It assumes that your repos have a small number of committers, and that you’re comfortable with the command line.

1. Clean up your Mercurial repository

Chances are your Bitbucket and Github usernames are not the same. Or if you’re like me, you didn’t keep your ~/.hgrc settings the same across machines, so your commit usernames aren’t consistent even if they’re all actually yours. The migration is a chance to tidy up.

To do this, we can use Mercurial’s convert extension, which can create a new, ‘filtered’ repository from an existing repository. In our case, we’ll alter the usernames to match what we want to appear on Github.

The convert extension is enabled by adding these lines to our ~/.hgrc:

[extensions]
hgext.convert=

Next, we’ll create a text file that maps old usernames to our new usernames. This is easy with some bash magic. Go to your repository directory and on the command line type:

hg log | grep user: | sort | uniq | sed ’s/user: *//‘ > users.txt

users.txt contains a sorted and filtered list of all the usernames attached to commits, like this:

username
username@localhost
username <username@gmail.com>

Edit this file so it maps old usernames to new usernames. If you have any existing Github repositories, I recommend running git logto get the exact username / email pair Github expects.

username=Username Fullname <username@gmail.com>
username@localhost=Username Fullname <username@gmail.com>
username <username@gmail.com>=Username Fullname <username@gmail.com>

We’re ready to convert (replace SOURCE_HG_REPO and CLEAN_HG_REPO with real directory names, of course):

cd ..
hg convert --authors SOURCE_HG_REPO/users.txt SOURCE_HG_REPO CLEAN_HG_REPO

And bam, you have a new, tidied mercurial repository in CLEAN_HG_REPO.

2. Convert the repository to Git

Converting a repository from Mercurial to Git is simple. We’ll use a script called fast-export. The following instructions are adapted from Dan Benjamin’s.

The easiest way to get fast-export is to clone it from its Git repository:

cd ~/tmp
git clone git://repo.or.cz/fast-export.git

Now we’ll create a new, empty Git repository, and use fast-export to populate it.

git init DEST_GIT_REPO
cd DEST_GIT_REPO
~/tmp/fast-export/hg-fast-export.sh -r CLEAN_HG_REPO
git checkout HEAD

And bam, you have a new Git repository, complete with your entire commit history. Don’t forget: you’ll need to convert your .hgignore into .gitignore.

3. Push the new repository to Github

Now it’s time take your shiny new Git repository and slap it up on Github. Go to your Github profile page, and click “New Repository.” Fill out the form, and follow the instructions to import an existing Git repo, which goes something like this:

cd DEST_GIT_REPO
git remote add origin git@github.com:username/REPO_NAME.git
git push -u origin master

4. Don’t forget to migrate other data

I haven’t found a way to automatically migrate them, but don’t forget about:

Issues
Wiki. I tended to use a one-page Wiki as a landing page. The Github convention is to use a readme.md markdown document. You’ll have to convert the Wiki markup to Markdown.

5. Deleting your Bitbucket repository

On the “Admin” tab of your Bitbucket repository page, you can choose to delete the repository. Make sure to enter the url of your new Github repository in the “Redirect to” field!

Wordcycler 2.0 Released

2011-12-21T08:39:00.002-06:00

It’s been a long time coming, but I’ve released a new version of Wordcycler, the program for two-way Instapaper sync for Windows and your e-book reader. There is only one major change — now, instead of scraping the site, Wordcycler uses the offical Instapaper full API.

This means two things for current users:

Wordcycler will be more reliable and future-proof. Since Wordcycler was nothing but a glorified site scraper, it has always been one minor Instapaper site upgrade away from being completely broken. We’ve been lucky so far, but it’s a bomb waiting to go off. The official API is guarranteed to keep working.
To use Wordcycler, you must be an Instapaper subscriber. Only subscriber accounts have access the full API. It’s only $1 a month and helps keep Instapaper running, so I recommend you subscribe if you haven’t already.

I realize this is a trade-off, so if this is a deal-breaker you should not upgrade. But I will not be updating the 1.x versions of Wordcycler, so if Marco Arment changes the HTML and breaks Wordcycler, you will be on your own. Using the full API is the right thing to do, if only just to be a good Instapaper citizen.

The Future of Wordcycler

The fact is, I don’t use Wordcycler myself much anymore. I own a Mac now, and I do most of my Instapaper reading on my iPhone. Wordcycler is not a money-making project, so my time and attention has been spent on my day job, my personal life, and other projects – the usual reasons. Wordcycler 2.0 has been sitting on my drive half-finished for about six months, but I wanted to finish it up to leave things in a good spot.

So that’s a long way of saying that this could be the last release of Wordcycler. It’s been a fun ride, and it’s been very gratifying to see so many people get use out of it. But it’s time to stop kidding myself that I have the time to maintain it, especially now that I’m no longer an active user myself. I’m exploring the possibility of open-sourcing Wordcycler, but no promises yet.

What I’ve Been Reading (History)

2011-11-07T20:03:00.000-06:00

David Crane, Scott of the Antarctic. My recent fascination with Scott’s two polar expeditions and the “heroic age” of Antarctic exploration probably merits its own post. Apsley Cherry-Garrard’s first-hand account, the aptly-named The Worst Journey in the World is also a great read.
Alfred Lansing, Endurance: Shackleton’s Incredible Voyage.
Charles Mann, 1491: New Revelations of the Americas Before Columbus. This is maybe the best history book I’ve ever read. Interesting facts on almost every page.
Charles Mann, 1493: Uncovering the New World Columbus Created. Also endlessly fascinating, if not quite as strong as its predecessor.
Max Hastings, Winston’s War: Churchill, 1940-1945. I was surprised by how desperate the British plight in WWII was, how impotent they actually were through most of it, and how much it bankrupted the empire.

That will probalby be the last of the non-fiction for a while; I’m just starting 1Q84, which clocks in at over 900 pages.

On Switching to a Standing Desk

2011-07-10T09:07:00.004-05:00

About a month ago I decided to try working at a standing desk. Four trashcans, two shelves, and several books later, I created this:

Despite its humble appearance, it functioned beautifully. It's since been replaced by an honest, actual adjustable-height desk, and I'm still standing and liking it a lot.

So why did I do this? This whole idea of standing while working has gotten a lot of talk recently, most prominently in the New York Times article Stand Up While You Read This. This blog post by Jesse Noller and Episode 21 of Build and Analyze finally convinced me to give it a shot.

Here what I was hoping to get out of it:

Better posture. Like many people, I tend to slouch in my chair. I find it easier to keep better posture while standing.
More active. The thing about standing at a desk is that I'm not just standing -- I'm fidgeting, stepping back and forth, and generally moving around. This makes me...
More alert. I tend to get sleepy after lunch. Having to keep myself upright without the aid of a chair counteracts that. It keeps me on task.
Respect(?) When people first come across you using your standing desk, they give you an expression that could either be respect or what's-up-with-that-dude wariness (I can never tell).

My initial goal was to stand nonstop all day. During my two weeks with the nonadjustable desk, that's all I could do. It was a good way to get over the initial habit-building hump, but let's just say that by the end of the day, my feet were ... tired.

Since I've gotten the adjustable desk, I've settled into a 2/3 standing, 1/3 sitting routine. Much of what I read recommended just powering through the pain until it gets better, but I find that simply breaking up the standing with an couple of hours of sitting in the middle of the day helps enormously.

So, if you want to get started, here's what I recommend:

Rig something on top of your existing desk. It's cheap and easy (and fun!). I used parts scavenged from around the office. Some have used soft drink cans. Adjustable desks are expensive so you want to make sure that you'll actually use it before shelling out.

As for actually making it a habit, there are two philosophical camps out there. The first camp says that humans are evolved to walk around barefoot and that any pain you experience is just your lazy body getting readjusted to its true evolutionary mode of being. The other camp recommends getting good insoles and a stress mat, and generally working your way into the whole standing routine.

I probably fall more in the second camp, but I haven't bought anything special. I simply wear my normal shoes and sit down when it starts to hurt. Seems to work for me.

Clockster: A Simple Online Countdown Timer

2011-07-05T06:15:00.002-05:00

I've created a small, HTML5 online countdown timer called Clockster. There's not much to it - you enter a time, the browser starts counting down in big numbers, and then beeps and flashes when time expires. It has a few interesting features:

It's completely HTML based. Most of the timers out there are Flash based, because until recently that was really the only way to play audio in the browser (for the beeping). But now all the major browsers support HTML5 audio, rendering it unnecessary.
If you resize the window, the numbers resize as well. So you can either see it from across the room, or make a small clock in the corner of your screen.
There are controls (pause, restart) on the countdown screen.
The time is specified in the url. For example, http://clockster.net/25minutes/.
The front page or any specific timer can be pinned to the Windows taskbar using Chrome or Internet Explorer.

On the server, Clockster is written in ASP.NET MVC, and hosted on AppHarbor. While the server-side aspect of Clockster is very light weight, I've been nothing but happy with AppHarbor. Deploying is as simple as pushing your Git or Mercurial repository to AppHarbor, which handles building and deploying the site automatically, usually within a few seconds. (It's like Heroku, if you're familiar with it.) For a small app like Clockster, it's completely free.

On the client, Clockster uses jQuery and date.js. The audio is handled by HTML5 audio, which why you need a modern browser to use it. (That's the fun of doing a completely non-commercial personal site).

Clockster was also a great opportunity to check out Nuget, the .NET package manager. I'm really impressed. It's obviously heavily inspired by Ruby gems, and it makes installing all sorts of project dependencies (like jQuery and Elmah) really easy.

Clockster was a fun project and I hope you check it out.

NReadability

2010-12-08T12:47:00.001-06:00

I just released Wordcycler 1.4. The only change is that Wordcycler will now fetch every page of multi-page articles, which has been a long-time user request.

If you're a frequent Instapaper user, you might ask why it has taken so long when Instapaper itself has been doing this for a while. The answer is that Wordcycler, since version 1.2, doesn't pull the article text from Instapaper.com; instead it pulls each articles from its original site and cleans it locally. I do that to avoid slamming Instapaper with hundreds of requests at once (you'd be surprised how many people maintain 400-item reading lists).

The technology behind Wordcycler's page cleaning is NReadability, an open-source C# port of the Javascript Readability bookmarklet. It strips away all the non-article content of a web page, and cleans up the formatting - just like Instapaper's text view.

NReadability is maintained by Marek Stój, and powers his Instafetch app for Android and Windows Phone 7. I was extremely lucky to find NReadability - he posted it a mere days before Instapaper stepped up its rate limiting, which broke any Wordcycler version prior to 1.2. I plugged it in to my code was able to release a repaired version 1.2 almost immediately.

The one thing NReadability didn't do was fetch multipage articles, so porting that from readability.js was my side project for a few weeks in November. About a week ago, Marek accepted my patch and released version 1.3.1.0. I’m happy I was able to contribute back to this great project.

Making Emacs Server Play Nice With Windows

2010-09-21T20:29:00.001-05:00

Some people like to use Emacs for everything, even Google Maps. That's way too much for me, but I do like to have Emacs available with a minimum of fuss. Here are two tricks that I use to make that happen: (1) creating a right-click "Edit in Emacs" menu item in Windows Explorer, and (2) open the current file in Visual Studio in Emacs. I originally attempted this a while ago, hit a wall, and gave up. I just got it working, so I wanted to write up what I did.

If Emacs were a normal Windows program, this would be easy, but it predates Windows by over a decade so I guess it gets to play by it's own rules. In order to open a file in an already running instance of Emacs, you have to use the Emacs client executable, and not the regular Emacs executable. The client connects to a server running in Emacs, which you have to explicitly start.

(The following instructions are all a distillation of the instructions available from the Emacs Wiki.)

First, add this to your .emacs:

(require 'server) 
(server-start)

This will start the server when Emacs starts. To make things even more fun, this sometimes fails on Windows 7. See here for a solution.

Windows Explorer

We're going to add a right-click menu to Windows Explorer labeled "Open with Emacs," which does exactly what it says it does.

The next step is to edit the registry. Here's how:

1. Open up the Registry Editor (Windows-R, type "regedit").
2. Create a new key [HKEY_CLASSES_ROOT\*\Shell\Edit in Emacs\Command].
3. Set the value of the command to "C:\Program Files\emacs-23.2\bin\emacsclientw.exe" -na "C:\Program Files\emacs-23.2\bin\runemacs.exe" "%1" (adjusting the paths as necessary).

And that's it. This first attempts to open the file with a running Emacs server, but if there isn't one (i.e. Emacs isn't running), then it opens a completely new instance (that's what the "-na" option is for). If you normally launch Emacs with a batch file (which is common), the –na argument can point to that instead.

Visual Studio

Sometimes when editing a file in Visual Studio, I want to punt the file to Emacs for a bit and work on it there. Fortunately, it's really easy to call the Emacs client from what's called an External Tool in Visual Studio. You can even assign it a keyboard shortcut, such as C-k,C-e, and bam, the file is opened in Emacs.

The theory is the same as before, but the details are a bit different. Go to Tools > External Tools, and Add a new tool with the following configuration:

Title: Edit In Emacs
Command: C:\Program Files\emacs-23.2\bin\emacsclientw.exe
Arguments: –na "C:\Program Files emacs-23.2\bin\runemacs.exe" "$(ItemPath)"

Now, when you go to Tools > Edit in Emacs, the current file should appear in Emacs.

To assign this to a keyboard shortcut, go to Tools > Options, then Environment > Keyboard. In the "Show commands containing" text box, type in "externalcommand." You'll see a bunch of options that look like this:

As you can tell, this is more complicated than it needs to be. Each "Tools.ExternalCommandX" entry corresponds to an external tool. The number matches the position on the external tools list. In other words, if I made "Edit in Emacs" the first external tool, then I would use "Tools.ExternalCommand1." Choose that, put your cursor in "Press shortcut keys," type your keyboard shortcut (C-k, C-e), and press "Assign." That's it.

Announcing Wordcycler

2010-06-29T06:00:00.000-05:00

Earlier this year, Carlo Zottmann released a little program called Ephemera, which gave Mac users the ability to synchronize articles between Instapaper.com and their USB e-book reader. I heard nothing but great things about Ephemera, and I really wanted to use it myself. But I'm a PC, not a Mac, so I was out of luck.

But that’s nothing a bit of code couldn’t fix. So a few months of weekend hacking later, I'm happy to announce Wordcycler, which brings two-way Instapaper syncing goodness to Windows. So what is Wordcycler?

Wordcycler lives in your tray. When you connect your reader via USB, Wordcycler fetches your unread Instapaper items, saves them on the device as single articles, and (if you want) ejects it. All automatically and in seconds.

When you finish reading an article, delete it from your reader, and it will be archived on Instapaper.com the next time you sync. And any articles that you read and archived on Instapaper.com in the meantime will be deleted from the reader as well.

Wordcycler can also download the pre-made all-in-one bundles available from Instapaper. Either way you choose to consume, Wordcycler is the fast, easy, Whispernet-charge-free way of experiencing Instapaper on your reading device.

If this kind of thing interests you, please check it out. I'd love to hear your feedback, bug reports, etc. I plan to keep adding features in the coming weeks (support for Instapaper's folders being the big no-brainer.)

XSpec 0.2 Released

2010-03-23T06:39:00.001-05:00

Last week, a new version of XSpec, an open-source unit-testing framework for XSLT (and now XQuery) was released. XSpec allows you to write test suites (called scenarios) to test XSLT functions and templates and XQuery functions.

It’s good stuff. If you do any work with these technologies, I recommend checking out the Getting Started guide and the User Guide for an introduction.

I’m particularly excited about this release because I’m involved in the project – well, in a bit part anyway, mainly providing testing and documentation. The overwhelming bulk of the credit for the new release is due to Florent Georges, who has really kicked the project back into gear and has rewritten the core functionality to support XQuery in addition to XSLT. After a year of dormancy, both development and the mailing list are active, which means that XSpec will continue to improve.

George Orwell on Writing and Thinking

2010-02-28T10:48:00.001-06:00

From Politics and the English Language:

A scrupulous writer, in every sentence that he writes, will ask himself at least four questions, thus: 1. What am I trying to say? 2. What words will express it? 3. What image or idiom will make it clearer? 4. Is this image fresh enough to have an effect? And he will probably ask himself two more: 1. Could I put it more shortly? 2. Have I said anything that is avoidably ugly? But you are not obliged to go to all this trouble. You can shirk it by simply throwing your mind open and letting the ready-made phrases come crowding in. They will construct your sentences for you – even think your thoughts for you, to a certain extent – and at need they will perform the important service of partially concealing your meaning even from yourself.

…

What is above all needed is to let the meaning choose the word, and not the other way around. In prose, the worst thing one can do with words is surrender to them. When you think of a concrete object, you think wordlessly, and then, if you want to describe the thing you have been visualizing you probably hunt about until you find the exact words that seem to fit it. When you think of something abstract you are more inclined to use words from the start, and unless you make a conscious effort to prevent it, the existing dialect will come rushing in and do the job for you, at the expense of blurring or even changing your meaning. Probably it is better to put off using words as long as possible and get one's meaning as clear as one can through pictures and sensations. Afterward one can choose -- not simply accept -- the phrases that will best cover the meaning, and then switch round and decide what impressions one's words are likely to make on another person. This last effort of the mind cuts out all stale or mixed images, all prefabricated phrases, needless repetitions, and humbug and vagueness generally. But one can often be in doubt about the effect of a word or a phrase, and one needs rules that one can rely on when instinct fails. I think the following rules will cover most cases:

(i) Never use a metaphor, simile, or other figure of speech which you are used to seeing in print.

(ii) Never us a long word where a short one will do.

(iii) If it is possible to cut a word out, always cut it out.

(iv) Never use the passive where you can use the active.

(v) Never use a foreign phrase, a scientific word, or a jargon word if you can think of an everyday English equivalent.

(vi) Break any of these rules sooner than say anything outright barbarous.

Worth rereading every once and a while.

Switcheroo version 0.4

2010-02-07T13:14:00.001-06:00

I’ve released a new version of Switcheroo, the humble incremental-search task switcher for Windows. Changes in this version include:

The list that prevents certain windows from appearing in the main window is now editable from the options dialog. (For example, “Program Manager.”)
The main window now resizes to the full width and height of the list.
Fixed a few minor bugs and tested on 32-bit Windows 7.

Essential Windows Tools

2010-01-26T06:00:00.000-06:00

When working, I like everything on my computer “just so.” I want to do things quickly and with minimal mental overhead. Here’s a list of (developer-centric) tools that make that possible:

VirtuaWin. This is hands-down the best virtual desktop manager for windows. With a bit of tweaking, you can switch between desktops instantaneously by bumping your mouse pointer at the edge of the screen while holding down the control key.
xplorer2. I use the free “lite” version. Great features:

Tabbed views (just like your browser). Keeps the taskbar clean.
Filter a directory listing quickly using ctrl-h.
When in a folder, pressing F10 and enter starts a command prompt in that folder. (With a bit more tweaking, get a cygwin bash prompt.)

Cygwin + minTTY. If you spend time on the command prompt, it’s worth it to learn your way around bash. And MinTTY’s native-windows interface – with fully resizable windows and transparency – is worth the price of admission alone. I don’t use the built-in command prompt any more, and neither should you. Great features:

Color coded directory listings and grep results.
Press control-r and search backwards through your command history.

Switcheroo. Yes, I wrote it, but what of it? Switching between (and closing) running applications using incremental search keeps me from reaching for the mouse and breaking my concentration.
Slickrun. A floating, auto-completing prompt that lets you quickly open programs, folders, and websites. With some batch-script ingenuity, the possibilities are endless. For example, I can type “vup” or “vdown” to quickly adjust the volume. Or “newmail” to write a new e-mail in Outlook. For most commands, I only have to type the first few letters. Set up tip: change the colors, make the font larger, and set it to autohide and chase your mouse cursor.
Emacs. The One True Editor. Learning it is a commitment, but worth it. If you take the plunge, grab Ctrl2Cap and remap your caps lock key as another control key. With a bit of setup, plays very well with Cygwin.
Winsplit Revolution. Not as essential with Windows 7’s built-in window positioning, but the fusion mode is still great.
Password Safe. Securely encrypt a list of all your passwords. Now you only have to remember one, which lets you online banking passwords more complex and harder to brute-force. And you’ll stop using the same one everywhere.

Knowledge and productivity are like compound interest

2010-01-22T08:05:00.001-06:00

What Bode was saying was this: ``Knowledge and productivity are like compound interest.'' Given two people of approximately the same ability and one person who works ten percent more than the other, the latter will more than twice outproduce the former. The more you know, the more you learn; the more you learn, the more you can do; the more you can do, the more the opportunity - it is very much like compound interest. I don't want to give you a rate, but it is a very high rate. Given two people with exactly the same ability, the one person who manages day in and day out to get in one more hour of thinking will be tremendously more productive over a lifetime.

– Richard Hammin, in You and Your Research

Recent Projects

2009-11-01T20:26:00.001-06:00

It's been a while since I've written here. In the interim, I've completed a few different small projects. Here they are in no particular order:

Switcheroo. The humble incremental-search task switcher for Windows.

I wrote this partly to learn a bit of WPF, and partly to scratch my own itch. I got tired of alt-tabbing through a (long) list of open windows, and really wanted something like Emacs's IDO mode buffer switching. This little program allows me to quickly switch to any window by typing in just a few characters of its title.
eXist XProc Extension Library. A set of XProc extension steps for interacting with an eXist XML database from a client. Using these steps, you can conduct common eXist management tasks from XProc - loading resources, extracting resources, querying data, etc. They fill much the same role as the eXist Ant tasks.

I wrote it as an experiment in creating a non-trivial reusable library in pure XProc. I think it came out fairly well.
Ruby Blue Visual Studio theme. I really liked the various ruby blue themes available for other editors and wanted one for Visual Studio.

XProc Roundup

2009-05-26T20:43:00.001-05:00

Interest in XProc has really picked up recently, which is exciting to see. Here's a roundup of some of the recent activity:

XProc: Step by Step. Vojtech Toman (who's working on EMC's Calumet implementation) wrote a great introduction to XProc. Calumet itself is scheduled to be released under a developer's license on June 15. I don't use any of EMC's products, but it's encouraging that such a big player in the market is embracing XProc.
XProc Tutorial. Roger Costello updated his excellent tutorial. It's a bit of a shame that it's in PowerPoint, but it's well organized and comes with several sample scripts.
Why XProc Rocks. Joel Amoussou talks about XProc and news publishing.
XML Pipelines / XProc for bioinformatics. Pierre Lindenbaum describes an interesting use of XProc.

The XProc spec is tantalizingly close to making the transition to proposed recommendation, and all the recent (and positive) attention makes me optimistic that XProc will be a standard that's actually used in the real world.

Debugging XProc Pipelines

2009-03-16T22:21:00.001-05:00

As I've worked more with XProc, I've written a couple of utility steps to help debug pipelines. Although they started as quick hacks, they've continued helping out long enough to be worth sharing. You can download the library here.

The more interesting of those is a step (wxp:assert) which asserts that the result of a given XPath expression evaluated against one document must be equal that that of a second XPath evaluated against a second document.

Practically speaking, I can assert that the output from a step must match certain expectations. If this assertion is met, the wxp:assert step functions just like a p:identity step, passing XML through its primary input to its primary output without modification. If the assertion fails, the step sends an error message to the console, and optionally throws an XProc error and terminates pipeline execution.

For example, here's an excerpt from an upconversion pipeline:

<p:xslt name="group-sections">
  <p:input port="stylesheet">
    <p:document href="group-sections.xslt"/>
  </p:input>
</p:xslt>

<wxp:debug-output step-name="6-sections-grouped" debug="true"/>

<wxp:assert label="No sections were deleted during grouping" 
            xpath-source="count(.//subsection)"
            xpath-alternate="count(.//*[self::sect-1 | self::sect-2 | 
                             self::sect-3 | self::sect-4])" 
            fail-on-error="false">
  <p:input port="alternate">
    <p:pipe port="result" 
            step="parse-initial-subsections"/>
  </p:input>
</wxp:assert>

(This also shows the wxp:debug-output step, which has probably been rendered obsolete by the latest version of Calabash. It simply functions as a p:identity step that also writes the XML it receives to disk for later review.)

This instance of wxp:assert asserts that the number of subsection elements in the primary source input port matches the number of sect-1 and sect-2 elements on the alternate input port. The source document is produced by the preceding step, and the alternate document is piped in from a step earlier in the pipeline. This is a quick way to ensure I don't do anything too lame-brained in the "group sections" step.

Here's another instance from the same upconversion pipeline. This tests the output from a XSLT step that uses regular expressions to parse out section numbers from title elements.

<wxp:assert label="No empty enums after parsing sections"
    xpath-source="count(.//enum[not(.//text()[string-length(normalize-space(.)) gt 0])])" 
    xpath-alternate="0"
    fail-on-error="true"/>

In this case, the alternate XPath expression is a constant (0), so I don't need to provide a document on the alternate port.

Using wxp:assert steps does slow down a pipline, so I usually remove them once I finish debugging the pipeline. The major exception to this rule is that I leave them active in upconversion pipelines where manual intervention is acceptable and accuracy is more important than speed.

There are a few interesting in the step's source worth checking out. Here's a summary of how wxp:assert works:

It combines the two source documents into a single document using p:pack.
That is passed to an XSL stylesheet, which uses the saxon:evaluate extension function to evalutate the two XPaths against their respective nodesets.
Depending on the value of the fail-on-error option, throw an XProc error and terminate the pipeline.
Use a final p:identity step to pipe the initial input from the step's source port to its result port.

Or course, using a Saxon extension function does tie the pipeline to Calabash (or at least, a Saxon-based processor). I'm interested in hearing ideas about alternate approaches.

Code Review: Commenting XSLT Regular Expressions

2009-02-11T18:25:00.003-06:00

You learn a lot from reading other people's code. For example, the other day I ran into a clever trick in Jeni Tennison's XSpec code for commenting regular expressions in XSLT:

<xsl:variable name="attribute-regex" as="xs:string">
  <xsl:value-of>
    \s+
    (\S+)        <!-- 1: the name of the attribute -->
    \s*
    =
    \s*
    (       <!-- 2: the value of the attribute (with quotes) -->
      "([^"]*)"  <!-- 3: the value without quotes -->
      |
      '([^']*)'  <!-- 4: also the value without quotes -->
    )
  </xsl:value-of>
</xsl:variable>

The trick is the <xsl:value-of /> instruction, which casts its contents as a string. An especially nice thing about this method is that you can refer to other variables within the declaration:

   (\S+)    <!-- 12: the name of the element being opened -->
   (        <!-- 13: the attributes of the element -->
     (      <!-- 14: wrapper for the attribute regex -->
       <xsl:value-of select="$attribute-regex" />  <!-- 15-18 attribute stuff -->
     )*
   )

Of course, to ignore all the extra white space in a regex constructed this way, you'll need to set the "x" flag in any <xsl:analyze-string />, replace(), or matches() that refers to it.

“Broken gets fixed. Shoddy lasts forever”

2009-02-04T19:35:00.001-06:00

Jack Moffett:

One of the developers I work with said this after I complained about a lingering issue in one of our products. It rings true. When deadlines are tight, and there is more work to get done than there are developers or hours in the schedule, it’s not the squeaky wheel, but the jammed one that gets the grease. The lesson, then, is to make sure it gets done right the first time. You never know when you’ll have the opportunity to revisit it.

(via Daring Fireball)

Stark and Adjustable Meanings

2009-01-20T21:07:00.001-06:00

An interview of Alan Kay:

But at PARC our idea was, since you never step in the same river twice, the number-one thing you want to make the user interface be is a learning environment—something that’s explorable in various ways, something that is going to change over the lifetime of the user using this environment. New things are going to come on, and what does it mean for those new things to happen?
... Even if the user is an absolute expert, able to remember almost everything, I’m always interested in the difference between what you might call stark meaning and adjustable meaning.
I did quite a bit of study on that over the years to understand the influence of having something that you can read. It’s known that our basic language mechanism for both reading and hearing has a fast and a slow process. The fast process has basically a surface phrasal-size nature, and then there’s a slower one. This is why jokes require pauses; the joke is actually a jump from one context to another, and the slower guy, who is dealing with the real meanings, has to catch up to it.
There have been many, many studies of this. This argues that the surface form of a language, whatever it is, has to be adjustable in some form.

Money Is Gas in the Car

2009-01-11T10:09:00.001-06:00

Tim O'Reilly talks about the importance of working on things that are important, not just things that simply make money:

First off, though, I want to make clear that "work on stuff that matters" does not mean focusing on non-profit work, "causes, or any other form of "do-goodism." Non-profit projects often do matter a great deal, and people with tech skills can make important contributions, but it's essential to get beyond that narrow box. I'm a strong believer in the social value of business done right. We need to build an economy in which the important things are paid for in self-sustaining ways rather than as charities to be funded out of the goodness of our hearts.
...

Some of you may end up working at highflying companies. Some of you may succeed, and some of you may fail. I want to remind you that financial success is not the only goal or the only measure of success. It's easy to get caught up in the heady buzz of making money. You should regard money as fuel for what you really want to do, not as a goal in and of itself. Money is like gas in the car — you need to pay attention or you'll end up on the side of the road — but a well-lived life is not a tour of gas stations!

What's Your Semantic Exit Strategy?

2009-01-08T18:08:00.001-06:00

Quoting Cafe Con Leche quoting A List Apart:

We’ll start by posing the question: “why are we inventing these new elements?” A reasonable answer would be: “because HTML lacks semantic richness, and by adding these elements, we increase the semantic richness of HTML—that can’t be bad, can it?”

By adding these elements, we are addressing the need for greater semantic capability in HTML, but only within a narrow scope. No matter how many elements we bolt on, we will always think of more semantic goodness to add to HTML. And so, having added as many new elements as we like, we still won’t have solved the problem. We don’t need to add specific terms to the vocabulary of HTML, we need to add a mechanism that allows semantic richness to be added to a document as required. In technical terms, we need to make HTML extensible. HTML 5 proposes no mechanism for extensibility.

HTML 5, therefore, implements a feature that breaks a sizable percentage of current browsers, and doesn’t really allow us to add richer semantics to the language at all.

This is an important warning for anyone creating or maintaining a custom XML document schema. Creating a tight, semantically correct element for every type of information in your existing documents is easy (and dangerously seductive); creating a mechanism by which the schema's users themselves can gracefully expand the semantic tagging as documents grow is much harder, and ultimately much more important. Be as general as you can get away with, and as specific as you dare.

Lessons from a Windows Reinstall

2009-01-06T06:45:00.000-06:00

The Windows install on my fiancee's laptop, a 3-year-old Gateway, had become unstable and slow (as Windows does if left alone for three years) so it was time to wipe the hard drive and start again. I methodically prepared, backing up all the data to two different locations — Dropbox and an external hard drive. Also, at my father's urging, I created a drive image using the free trial of Acronis True Image, which I stored on an external hard drive. I didn't think I'd need it, because the current install was so horrible that anything was preferable to it.

So, I rebooted, I popped in the Gateway system restore disk, and told it to format the drive and reinstall the operating system.

When it was done, the computer booted into Windows. Happy surprise, the restore CD didn't reinstall all of the stupid cruft software that all consumer laptops come loaded with. However, a not-so-happy-surprise — the CD also didn't reinstall any of the Gateway-specific drivers the laptop needed to function, so it couldn't use its wireless card or suspend.

No problem, I thought, I'll get them from the system restore partition on the primary hard drive. Hmm, the restore process repartitioned the drive and killed it. (Really?) Okay, no problem, I'll go to Gateway's website and download the drivers. Hmm, those don't work. (Seriously?) I'll go to the chipset manufacturer's website. Hmm, not there. (Oh, no.)

This problem was entirely my fault. I should have made absolutely sure I had the drivers I needed before wiping the disk, so I'll spare you the whole rant of WHY DOESN'T GATEWAY'S CUSTOM SYSTEM RESTORE DISK CONTAIN THE DRIVERS TO ACTUALLY, YOU KNOW, RESTORE THE SYSTEM? Because, that would be petty.

After some Internet searching, I discovered a free utility called Double Driver, which allows you to backup and restore all the drivers installed on your system. Which was great, except I'd already killed the Windows installation with the drivers I needed.

To make a long story short, here's what I did:

Created a new drive image of my partially finished reinstall with Acronis True Image.
Restored the original, pre-wipe drive image.
Booted into Windows and ran Double Driver to create an archive of all the installed drivers on my USB drive.
Restored the in-progress drive image.
Used Double Driver to restore all the previously installed drivers.

Success! (Well, Double driver actually missed one file, which I had to retrieve from the backup image. But close enough.)

Also, although Acronis True Image saved the day, I can't recommend it as a product, since its background service slowed to a crawl the two computers I installed it on. Which is too bad.

So, recapping today's lessons, if you're reinstalling Windows, you should first:

Backup all your data. Twice.
Create a drive image of your current install. It will save your bacon.
Create an archive of all your drivers using DoubleDriver. This too will save your bacon.

XProc, Part III — Turtles All The Way Down (Steps, Reuse, and Encapsulation)

2008-12-22T12:11:00.002-06:00

This is my third post on XProc. (To see the first two, go here and here.) In this post, I'll talk about the different categories of steps, how to create your own steps and reuse them across pipelines, and how this all relates to the fundamental metaphor of XProc.

One task I do frequently is removing Arbortext Editor change-tracking markup from documents. Here's a simple pipeline that does just that:

<p:pipeline type="u:accept-changes">
  <p:xslt>
    <p:input port="stylesheet">
      <p:document href="accept_changes.xslt"/>
    </p:input>
  </p:xslt>
</p:pipeline>

The fundamental unit of work in XProc is the "step," which you can think of in the same way you do subroutines in other programming languages. In general, a step takes XML as input, performs an operation on it, and outputs XML. There are three types of steps in XProc:

Atomic steps. These are the most basic building blocks of XProc pipelines, and each carries out a single fundamental XML operation. There are a number of built-in atomic steps, such as p:xslt (as shown above). These built-in atomic steps will be the foundation for almost everything you do in XProc, so learn them.
Compound steps. These are assembled from other XProc steps. Sometimes they are just a series of implicitly connected atomic steps, and sometimes they use built-in logical control structures, such as p:for-each, to control their execution.

It turns out that the above accept-changes pipeline — and in fact any pipeline you create — is also a compound step. (The fact that p:pipeline is a shortcut for p:declare-step is a dead giveaway.) This is an important idea, and we'll discuss it more below. But for now, remember this: a pipeline and a step are the same thing.
Multi-container steps. There are only two multi-container steps: p:choose and p:try. These contain two or more alternate pipelines. You cannot define your own custom multi-container step.

No doubt, in your day-to-day work, you'll never feel the need to stop and consider whether a certain step fits one category or the other. But the difference does have consequences, so it's worth spending a bit of time on it.

Let's say that you want to use the above accept-changes step as a step within a larger pipeline — for example, a pipeline that removes all of the notes from 2nd-level sections in a document. Here's how you would do that:

<p:pipeline name="delete-notes" 
    xmlns:p="http://www.w3.org/ns/xproc" 
    xmlns:atict="http://www.arbortext.com/namespace/atict"
    xmlns:u="http://www.wordsinboxes.com/xproc">

  <p:serialization port="result" encoding="utf-8" method="xml"  
    omit-xml-declaration="false"/>

  <!-- Declare accept-changes step -->

  <p:pipeline type="u:accept-changes">    
    <p:xslt>
      <p:input port="stylesheet">
        <p:document href="accept_changes.xslt"/>
      </p:input>
    </p:xslt>
  </p:pipeline>
  
  <!-- Pipeline flow starts here -->  

  <u:accept-changes name="remove-tracking-markup" />
  
  <p:delete name="delete-level-2-notes" 
    match="/chapter/section/section/note" />
     
</p:pipeline>

What's happening is that we are declaring a new step type called "u:accept-changes," and then invoking an instance of it as the first step in the pipeline. A few notes:

The u:accept-changes step declaration itself is not executed directly; only it's instance on line 21 is. More formally, a pipeline element that is the child of another pipeline element (unlike what we've seen before) is used to define a step that can be invoked later. As an analogy to other programming languages, you are defining a object type and then creating an instance of it. As a result, there is no need in this pipeline for explicit piping; the default implicit connections suffice.
A step's type attribute is different than its name attribute. The value you place in type becomes the name of the element you use to invoke it (lines 11 and 21). That instance can itself be given a name, which can then be used when controlling the flow of XML using pipes. So, a type refers to the object, and a name refers to the instance.
A step's type must be in a non-XProc namespace, which you must remember to declare.
The p:serialization element defines how a specified pipeline output port serializes XML when outputting from the pipeline. It is the equivalent of the xsl:output instruction in XSLT, and takes many of the same options.

Of course, if you're going to just reference your new step type once within the same pipeline that you've declared it in, there's not really much of a point.

The real power of declaring your own step type comes from the ability to place it in an external library and create instances of it within any number of pipelines. Here's how you do that, using p:import. First, the library document:

<p:library 
    xmlns:p="http://www.w3.org/ns/xproc" 
    xmlns:atict="http://www.arbortext.com/namespace/atict"
    xmlns:u="http://www.wordsinboxes.com/xproc">

  <p:pipeline type="u:accept-changes">    
    <p:xslt>
      <p:input port="stylesheet">
        <p:document href="accept_changes.xslt"/>
      </p:input>
    </p:xslt>
  </p:pipeline>
       
</p:library>

Note the root element p:library, which is used as a container for a collection of custom steps. If I wanted, I could have made the p:pipeline element the root element, and imported that single step, but using p:library leaves more room for future growth.

Here's the final pipeline, which imports the above library:

<p:pipeline name="delete-notes" 
    xmlns:p="http://www.w3.org/ns/xproc" 
    xmlns:atict="http://www.arbortext.com/namespace/atict"
    xmlns:u="http://www.wordsinboxes.com/xproc">

  <p:serialization port="result" encoding="utf-8" method="xml"  
    omit-xml-declaration="false"/>

  <!-- Import accept-changes step -->

  <p:import href="lib.xpl" />
  
  <!-- Pipeline flow starts here -->  

  <u:accept-changes name="remove-tracking-markup" />
 
  <p:delete name="delete-level-2-notes" 
    match="/chapter/section/section/note" />
     
</p:pipeline>

There. A nice and simple pipeline. I (and anyone else) can reference this u:accept-changes step in any number of pipelines, and never worry about how it actually works. (And, if you stored the transform inside the u:accept-changes definition using p:inline, it would be even more portable.) Later, if I decide that I'm better served by using a Python script to accept changes instead of a transform, I can simply update the step declaration (using p:exec) once in the library. My hope is that XProc will become a common language for sharing XML manipulation tasks.

It's also worth noting that if I gave this note-deleting pipeline a type attribute, then it too could be invoked as step in another pipeline. And that pipeline could be invoked as a step, and so on. Turtles all the way down.

Here's where the difference between an atomic step and a compound step comes into play. Although we defined our accept-changes step as a compound step (because it contains a subpipeline of one step), when we invoke an instance of it, that instance is an atomic step.

If that is confusing (and it sure confused me at first), it helps if you think about the step from two separate perspectives — from within, looking at the implementation details, and without, looking at what actually enters and leaves the step. We've already covered the inside. From the outside, when you call an instance, that instance is a black box which XML enters and XML leaves — exactly like a built-in atomic step. The fact that it's actually implemented by assembling XProc steps is irrelevant to the calling pipeline.

There's one more important XProc fact I've talked around but never explicitly stated. The only information that can flow between steps through pipes is XML. At first, I thought this a stifling restriction, but as I work more with XProc, I think that this prohibition is the key to its power.

Because, when you combine all these ideas, you end up with a very high level of enforced encapsulation that provides a very simple yet powerful abstraction for working with XML. It allows you to worry about what needs to be done instead of how. When any pipeline can be used as an atomic step, you can work fractally, making parts out of the same raw material as the whole, creating higher and higher levels of abstraction.

So, to summarize:

A pipeline is a step.
Although the declaration of a step may be a compound step, its invocation is not.
XML is the only information that can flow between steps through pipes.

That's it for now. If you're looking for more XProc information, Dave Pawson has been working on his introduction to XProc, which is shaping up to be a great resource.

I may start posting my XProc-related material in a more suitable space, away from my blog. I'm considering using Docbook Website system. Any thoughts?

Recently In Markup

2008-12-16T07:30:00.000-06:00

Cafe con Leche
"The W3C Core Working group has broken faith with the XML community by publishing an XML 1.0, fifth edition spec that is incompatible with all previous versions."
Joe Gregorio | BitWorking | CS Broke
"What part of the computer in front of you is fractal?"
Batch files
More than you ever wanted to know about how to write Windows batch files.
Main Page - Exslt2
The beginning of the discussion for EXSLT 2.0.
Marklogic review
Dave Pawson reviews MarkLogic ... and gives up.
Overlap, Containment and Dominance | Jeni's Musings
Let's talk about Python 3.0
"You build a nice big room-sized cage, and in one end of it you put five monkeys. In the other end you put the banana."

This Week in Google Reader

2008-12-13T08:00:00.000-06:00

Laugh-Out-Loud Cats #1006
(via HOBOTOPIA)
Teachers, Quarterbacks, and Markets
(via Uncertain Principles)
Photo
(via garfield minus garfield)
Wireframe Lamborghini
(via We Made This)