Xenomachina

Multi-diff with Vim and Git

2012-02-10T14:42:00.001-08:00

I just pushed some stuff to github that you may find useful if you're either a git user, a vim user, or (best of all) both.

git-multidiff

For git users, there's git-multidiff, which works kind of like git difftool, except that it invokes your tool of choice once on the entire set of files, instead of once for each pair. This is handy if you have a diff tool that'll let you view multiple diffs simultaneously.

Full installation instructions are in a comment at the top of the file, but it basically consists of putting git-multidiff and _git-multidiff-helper in your path and adding an entry to your .gitconfig. Note that it requires Python (I've tested it with 2.7.2).

tab-multi-diff.vim

Speaking of “diff tools that'll let you view multiple diffs simultaneously”, that's what tab-multi-diff.vim is for. It lets you do a “vimdiff” on multiple pairs of files, with each pair in a separate tab.

To install it, just save tab-multi-diff.vim in your vim plugins directory (typically ~/.vim/plugin/).

To use it, you can invoke vim (or gvim) with a command like:

gvim -c 'silent call TabMultiDiff()' old-foo foo old-bar bar

Thats obviously kind of long, so you probably want to wrap it in a shell script. My script for doing this is vd (which also depends on v). Note that that it imposes some of my personal preferences, so you may only want to use it as a starting point.

Using Them Together

To use git-multidiff and tab-multi-diff.vim together I have the following in my .gitconfig:

[multidiff]
  tool = vd -f

Note that the tool option for multidiff is a command line prefix, not a “tool name” as it is for git difftool. That’s why it’s possible to include a flag. The -f flag shown here is to prevent backgrounding. (It's always seemed weird to me that git difftool has this extra layer of indirection.)

A Sufficiently Advanced Violin

2011-12-04T15:12:00.000-08:00

The reactions to the recent story about CT scans being used to recreate a Stradivarius violin are interesting. For example, in the comments on Engadget there's a lot of denial that it could sound as good as the original, as well as people saying it won't sound as good in 300 years. I have to wonder if the latter even matters. If we can cheaply create a clone of a 307-year-old Stradivarius, you can just make a new one when it stops sounding good. And who knows if a 600-year-old Stradivarius will actually sound good?

Musicians Centre has an interesting take: “Why do we have to keep going back and trying to replicate the past when it comes to instruments?”

I agree, but I don't think the scanned Stradivarius has to be just about replicating the past. If we are able to scan instruments that sound good and produce replicas, that means we can experiment with modifications to the design, and iterate to produce better instruments. Without having any way to measure what makes a Stradivarius “good” means iteration is hard, and you end up with people talking about trees that don’t exist anymore or a mysterious fungus that can’t be replicated.

That said, improvements will most likely have to overcome a subjectivity problem. On a large scale there are objective ways of determining that one violin is better than another, but at a finer scale things might not be so clear cut. Assuming you could make a violin that sounds slightly better than even the best Stradivarius by some objective measure, would it just end up sounding weird to people who are used to the real thing?

PSA: Netflix for Android spontaneous deactivation fix

2011-05-29T22:59:00.000-07:00

Today Netflix on my Android phone (a Nexus One) started giving me this error:

It looks like Netflix has been deactivated on this device. It could be an issue with your account or perhaps your device was deactivated on netflix.com. (2004)

Netflix only lets you have 6 devices activated per account, so at first I thought I might be bumping into the limit, but it turned out that that wasn't my problem.

The thing that eventually worked was to clear all data for the Netflix app. To do this:

Go to the home screen.
Press the menu button.
Select "Manage apps" (or "Settings", then "Applications", then "Manage applications" on older versions of Android).
Select the "Downloaded" tab.
Select the Netflix app.
Click on "Clear data".

The next time you open the Netflix app you'll need to sign in again, but then it should be working correctly.

I talked to Netflix customer support about this issue and apparently they had a ton of devices spontaneously deactivate in the last day or so. It sounded like they either don't really understand the cause, or just didn't want to share the details. Based on the fix it seems like some sort of authentication token either got corrupted or had the server-side rug pulled out from under it. Clearing the app data seems to force it to get a fresh token.

Android's 2D Canvas Rendering Pipeline

2011-05-09T14:30:00.000-07:00

This is a conceptual overview of how Android's 2D Canvas rendering pipeline works. Since Android's Canvas API is mostly a pretty thin veneer on top of Skia it should also serve as a reasonable overview of Skia's operation, though I've only looked at Skia code that's reachable from Android's SDK, and when the Skia and Android terminology differ (which is rare, modulo “Sk” prefixes and capitalization) I've used the Android terminology.

How and Why I Wrote This

I wrote this overview because I've been doing some Android development recently, and I was getting frustrated by the fact that the documentation for android.graphics, particularly when it comes to all of the things that can be set in a Paint object, is extremely sparse. I Googled, and I asked a question on Stack Overflow but I couldn't find anything that explained this stuff to my satisfaction.

This overview is based on reading what little documentation exists (often “between the lines”), doing lots of experiments to see how fringe cases work, poring over the code, and doing even more experiments to verify that I was reading the code correctly. I started writing it as notes for myself, but I figured others might benefit as well so I decided to post it here.

Caveats

I say this is a “conceptual” overview because it does not always explain the actual implementation. The implementation is riddled with special cases that attempt to avoid doing work that isn't necessary. (I remember hearing some quote along the lines of “the fastest way to do something is to not do it at all”.) Understanding the implementation details of all of these special cases is unnecessary to understanding the actual end-result, so I've focused on the most general path through the pipeline. I actually avoided looking at the details of a lot of the special-case code, so if this code contains behavioral inconsistencies I won't have seen them.

Also, there are cases, particularly in the Shading and Transfer sections, where the algorithm described here is far less efficient but easier to visualize (and, I hope, understand) than the actual implementation. For example, I describe Shading as a separate phase that produces an image containing the source colors and Transfer as a phase producing an image with intermediate colors. In reality these two “phases” are interleaved such that only a small set (often just one) of the pixels from each of these virtual images actually “exists” at any instant in time. There is also short-circuiting in this code such that the source and intermediate colors aren't computed at all for pixels where the mask is fully transparent (0x00).

This does mean that this overview can't give one an entirely accurate understanding of the performance (speed and/or memory) of various operations in the pipeline. For that it would be better to performing experiments and profile.

Also keep in mind that because this is documenting what is arguably “undocumented behavior” it's hard to say how much of what is described here is stuff that's guaranteed versus implementation detail, or even outright bugs. I've used some judgement in determining where to put the boundaries between phases (all of that optimization blurs the lines) based on what I think is a “reasonable API” and I've also tried to point out when I think a particular behavior I've discovered looks more like a bug than a feature to rely on.

There are still a number of cases where I'd like to do some more experimentation to verify that my reading of the code is correct and I've tried to indicate those below.

Entering the Pipeline

The pipeline is invoked each time a Canvas.drawSomething method that takes a Paint object is called.

Most of these drawing operations start at the first phase, Path Generation. There are two exceptions, however:

drawPaint skips Path Generation entirely and Rasterization consists of producing a solid opaque mask.
drawBitmap has different behavior depending on the supplied Bitmap's configuration.

In the case of an ALPHA_8 Bitmap, Path Generation and Rasterization are both skipped and the supplied Bitmap is used as the mask.

For other Bitmap configurations the Shader is temporarily replaced with a BitmapShader in CLAMP mode. This means that setting a Shader to be used with a drawBitmap call with a non-ALPHA_8 Bitmap is pointless. The pipeline is then executed as though drawRect had been called with a rectangle equal to the bounding box of the Bitmap.

According to Romain Guy, this behavior is intentional.

Overall Structure

The overall structure of the pipeline. This diagram is available in Gzipped SVG or PDF formats for use as a quick reference card.

At the top of the diagram are the two main inputs to the pipeline: the parameters to the draw method that was called (really multiple inputs) and the “destination” image — the Bitmap connected to the Canvas.

There are four main phases in the pipeline. The details of these will be covered below. While there are exceptions, all of the phases (mostly) follow this pattern: There are two or more sub-phases, the first of which computes an intermediate result, while the later ones “massage” this intermediate result. These later sub-phases often default to the identity function. ie: they usually leave the intermediate result alone unless explicitly told to do otherwise by setting properties on the Paint.

Path Generation

The output of the first phase is a Path.

This phase has three sub-phases:

An initial Path is constructed based on the draw* method that was called. In the case of drawPath, this is simply the Path supplied by the client. In the case of drawOval or drawRect, the output is a Path containing the corresponding primitive.
If the Paint has a PathEffect, it is used to produce a new path based on the inital Path. The PathEffect is essentially a function that takes a Path as its input and returns a Path.
If no PathEffect is set then the initial Path is passed on to the next phase unmodified. That is, the default PathEffect is the identity function.

PathEffect implementations include CornerPathEffect, which rounds the corners of the Path, and DashPathEffect which converts the Path into a series of “dashes”.

One interesting quirk: if the Paint object's style is FILL_AND_STROKE the PathEffect is “lied to” and told that it's FILL. This matters because PathEffect implementations may alter their behavior depending on settings in the Paint. For example, DashPathEffect won't do anything if it is told the style is FILL.
The final sub-phase is “stroking”. If the Paint.Style is Path this does nothing to the Path. If the style is STROKE then a new “stroked” Path is generated. This stroked Path is a Path that encloses the boundary of the input Path, respecting the various stroke properties of the Paint (strokeCap, strokeJoin, strokeMiter, strokeWidth). The idea is that later phases of the pipeline will always fill the Path they are given, and so the stroking process converts Paths into their filled equivalents. If the style is FILL_AND_STROKE the result Path is the stroked Path concatenated to the original Path.

The method Paint.getFillPath() can be used to run the later sub-phases of this phase on a Path object. As far as I can tell this is the only significant part of the pipeline that can be run in isolation.

Rasterization

Rasterization is the process of determining the set of pixels that will be drawn to. This is accomplished by generating a “mask”, which is a alpha-channel image. Opaque (0xFF) pixels on this mask indicate areas we want to draw to at “full strength”, transparent (0x00) areas are areas we don't want to draw to at all, and partially transparent areas will be drawn to at “partial strength”. This is explained more at the end of the final phase. (When visualizing this process I find that it helps to think of opaque as white and transparent as black.)

Rasterization has two completely different behaviors depending on whether a Rasterizer has been set on the Paint.

If no Rasterizer has been set then the default rasterization process is used:

The Path is scan-converted based on parameters from the Paint (eg: the style property) and the Path (eg: the fillType property) to produce an initial mask.
Pixels “inside” the Path will become opaque, those “ outside” will be left transparent, and those on the boundary may become partially transparent (for anti-aliasing). The mask will end up containing an opaque silhouette of the object.

The Path object's fillType determines the rule used to determine which pixels are inside versus outside. See Wikipedia's article on the non-zero winding rule for a good explanation if these different rules.
If there is a MaskFilter set, then the initial mask is transformed by the MaskFilter. The MaskFilter is essentially a function that takes a mask (an ALPHA_8 Bitmap) as input and returns a mask as output. For example, a BlurMaskFilter will blur the mask image.
If no MaskFilter is set then the initial mask is passed on to the next phase unmodified. That is, the default MaskFilter is the identity function.

If a Rasterizer is set on the Paint then, instead of the above two steps, the Rasterizer creates the mask from the Path. The MaskFilter is not invoked after the Rasterizer. (This seems like a bug, but I've verified this behavior experimentally. Romain Guy agreed that this is probably a bug.)

The only Rasterizer implementation in Android is LayerRasterizer. LayerRasterizer makes it possible to create multiple “layers”, each with its own Paint and offset (translation). This means that when n LayerRasterizer layers are present there are n + 1 Paint objects in use: the “top-level” Paint (passed to the draw* method) and an additional n Paint objects, one for each Layer.

LayerRasterizer takes the Path and for each layer runs the Path through the pipeline of that layer's Paint starting at the PathEffects step and rendering to the mask. This has some interesting consequences:

Each layer can have its own PathEffect. These are applied to the Path that was generated by the top-level PathEffect (if one was set). So if the PathEffect of the top-level's Paint is set to a CornerPathEffect and a layer's PathEffect set to DashPathEffect that layer will render a dashed shape with rounded corners.
Each layer can have its own Rasterizer. Recursive rasterization is recursive.
Each layer can have its own MaskFilter. This MaskFilter applies to a separate mask in the sub-pipeline. Remember, the entire pipeline is being run again. For example, if there are two layers and one has a BlurMaskFilter the output of the other layer will not be blurred regardless of the order of the layers.
The destination Bitmap of this sub-pipeline is an alpha bitmap, so only the alpha-channel component of the Shading and Transfer phases have any relevance.

Also note that LayerRasterizer does not make use of the MaskFilter in the top-level Paint. Since the top-level MaskFilter is not invoked after invoking the Rasterizer, there is no point in setting a MaskFilter on a Paint if the Rasterizer has been set to a LayerRasterizer. (Perhaps other Rasterizer implementations could make use of the top-level MaskFilter, but LayerRasterizer is the only implementation included with Android.)

Shading

Shading is the process of determining the “source colors” for each pixel. A color (can) consist of alpha, red, green, and blue components (ARGB for short) each of which ranges from 0 to 1. (In reality these are typically represented as bytes from 0x00 to 0xFF.)

At a high level, the output of the Shader can be thought of as a virtual image containing the source colors: the “source” image. The actual implementation doesn't use a Bitmap, but rather uses a function that maps from (x,y) to an ARGB color (the “source color”) for the given pixel, and this function is only called for coordinates where the corresponding pixal may be altered by the source color. This is really just an optimization, however.

Like the previous phases, Shading also has two sub-phases:

An initial “source” image is generated by the Shader. If no Shader has been set it's as if a Shader that produced a single solid color (the Paint's Color) was used.

The Shader does not get the mask, the Path, or the destination image as inputs.
If a ColorFilter has been set then the colors in the source color image are transformed by this ColorFilter.

The only input to the ColorFilter during the pipeline are ARGB colors. The ColorFilter does not get the mask, the Path, the destination image, or the coordinates of the pixel whose color it is transforming, as inputs.

Transfer

Transfer is the process of actually transferring color to the destination Bitmap. The transfer phase has the following inputs:

The mask generated by Rasterization.
The “source color” for each pixel as determined by Shading.
The destination bitmap, which tells us the “destination color” for each pixel.
The transfer mode (XferMode).

Once again, there are two sub-phases:

An intermediate image is generated from the source image and destination image. For each each (x,y) coordinate the corresponding source and destination colors are passed to a function determined by the XferMode. This function takes the source color and destination color and returns the color for the intermediate image's pixel at (x,y).

Note that the mask is not used in this sub-phase. In particular, the source-alpha comes from the Shader, and the destination alpha comes from the destination image.

If an XferMode hasn't been set on the Paint then the behavior is as though it was set to PorterDuffXferMode(SRC_OVER).

The second sub-phase takes the intermediate image, the destination image, and the mask as inputs and modifies the destination image. It does not use the XferMode.

The intermediate image is blended with the destination image through the mask. Blending means that each pixel in the destination image will become a weighted average (or equivalently, linear interpolation) of that pixel's original color and the corresponding pixel in the intermediate image. The opacity of the corresponding mask pixel is the weight of the intermediate color, and its transparency is the weight of the original destination color.
In other words, a pixel that is transparent (0x00) in the mask will be left unaltered in the destination, a pixel that is opaque (0xFF) in the mask will completely overwritten by the corresponding pixel in the intermediate image, and pixels that are partially transparent will result in a destination pixel color that is proportionately between its original color and the color of the corresponding intermediate image pixel.

This is the final phase. The pipeline is now complete.

More on Porter Duff Transfer Modes

The most commonly used transfer modes are instances of PorterDuffXferMode. The behavior of a PorterDuffXferMode is determined by its PorterDuff.Mode. The documentation for each PorterDuff.Mode (except OVERLAY) shows the function that is applied to the source and destination colors to obtain the intermediate color. For example, SRC_OVER is documented as:

[Sa + (1 - Sa)*Da, Rc = Sc + (1 - Sa)*Dc]

This means:

Ra = Sa + (1 - Sa) * Da
Rr = Sr + (1 - Sa) * Dr
Rg = Sg + (1 - Sa) * Dg
Rb = Sb + (1 - Sa) * Db

Where Rx, Sx, and Dx are the intermediate (result), source and destination values of the x color component.

Some additional notes on the PorterDuff.Mode documentation:

The documentation uses “Sc” and “Dc” rather than describing each red, green, and blue component separately. This is because Porter Duff transfer modes always treat the non-alpha channels the same way and each of these channels is unaffected by all other channels except for alpha.
SRC_OVER and DST_OVER are the only two modes that have the left hand side of this equation, “Rc”, in their documentation. I'm guessing this inconsistency is a copy-and-paste error.
The alpha channel is always unaffected by non-alpha channels. That is, Ra is always a function of only Sa and Da.
The documentation for ADD refers to a “Saturate” function. This is just clamping to the range [0,1]. (I don't know why they use such an odd name for clamping, especially “saturation” usually refers to an entirely unrelated concept when talking about colors.)
The definition of many of these modes, including OVERLAY, can be found in the SVG Compositing Specification. The Skia code actually links to (an older version of) this document. It has some good diagrams, too.

References

The android.graphics documentation.
This answer to “Android Edit Bitmap Channels” on Stack Overflow. Seeing this answer motivated me to learn more about how the pipeline actually works.
The Android codebase. Since the documentation was so sparse and there didn't seem to be much information I looked to the source. My initial look stopped short when I realized everything was just a wrapper around “native” code.
Skia documentation, particularly SkPaint. Skia is the vast bulk of “native” (C++) code involved.
“Stack Overflow: How do the pieces of Android's (2D) Canvas drawing pipeline fit together?”, a question I asked on Stack Overflow. One member of the Android team actually responded, but didn't really provide the details I was looking for.
The Skia codebase. The code for SkCanvas::drawPath is a good place to start.
SVG Compositing Specification: W3C Working Draft 30 April 2009. This document is mentioned in the Skia code.
SVG Compositing Specification: W3C Working Draft 15 March 2011. This document supercedes the one mentioned in the Skia code. I believe the relevant bits still apply, but there's more detailed explanation and some good diagrams.

What's good for the Twitter is good for the Apple

2011-03-19T11:05:00.000-07:00

A lot of people have been talking about Twitter's recent stance on third-party apps. I think Mike Loukides of O'Reilly really hits the nail on the head:

...you can't tell people where (or how) to innovate, and where not to. Innovation just doesn't work that way. The best way to prevent "think big" innovation from happening is to cut off the small ideas.

Even John Gruber, unabashed Apple fanboy, agrees:

It’s not that I think Twitter is wrong in any moral sense to do whatever they want with their own API — it’s that I think they’d be foolish to do anything that dampens the diverse ecosystem of client software that has evolved around Twitter. They’re acting against their own self-interest, but apparently don’t realize it.

Whether it's "moral" or not is open to debate. There does, however, seem to be general consensus that the changes in Twitter's policies are bad for developers, bad for users and in the long term bad for Twitter.

The general form of the argument, which I wholeheartedly agree with, goes like this:

Artificially restricting developers hurts innovation. (See Loukides's quote, above.)
Hurting innovation hurts users.
Hurting users hurts the platform creator.

These can be long term things, which makes them hard to measure. You can't just change your policy and see the effects overnight. For example, it might have taken years before a particular sort of ground-breaking third-party product would appear on a restriction-free platform, so in the short term having restrictions that forbid its existence might not appear to have significant detrimental effects. Likewise, most users won't miss the utility of a product they don't know exists, or even can exist. It generally takes a competing, less restricted, platform to come along before people really start to realize what they're missing. This is further slowed down by network effects.

What's interesting is that this exact same chain of reasoning also applies to Apple and their App Store policies. Just as Twitter API clients should not "compete" with the official Twitter clients, apps for iOS are not allowed to compete with Apple products (or even other established iOS apps, to a degree). The iOS policies are actually far more restrictive on innovation than Twitter's policies, as the iOS policies largely forbid using Apple's APIs in any way that Steve Jobs didn't already imagine. "Think Different", indeed. (As an aside, I think Gruber is at least partially aware of the similarity, or he wouldn't have so carefully prefaced his statement with "It’s not that I think Twitter is wrong in any moral sense".)

The parallels run even deeper. Even people who have come out in Twitter's defense on this issue often point out that Twitter's platform was in many ways built by the Twitter community (hash-tags and at-replies were being used by users before Twitter even had special support for them) and the large variety of Twitter clients also contributed to Twitter's success. For Twitter to suddenly institute draconian policies seems like a betrayal to some.

If Twitter betrayed their users by being open at first and then closing up once they achieved popularity then Apple is just as guilty. Apple's trick was to stretch things out over a much longer time frame. Historically, Apple hardware was touted as being quite open. The Apple IIe was easily hackable both in a software and hardware sense. Apple's products weren't marketed as the "computer for the rest of us" just because they were easier or prettier than the competition, but also because they purportedly made it easier to create all sorts of things, including visual art, music and even computer programs. (I say "purportedly" because the Amiga and Atari ST were arguably just as good if not better when it came to certain sorts of creative work.) Remember Hypercard? A third-party equivalent to HyperCard wouldn't even be allowed given the current iOS App Store policies.

One last thing to note is Twitter's stated reason for the policy change:

If there are too many ways to use Twitter that are inconsistent with one another, we risk diffusing the user experience.

Hmmm, sounds like they're worried about "fragmentation".

PayPal stupidity

2010-12-18T09:53:00.000-08:00

It seems that every year, while doing my Christmas shopping for relatives in Canada, I discover another major e-commerce site that doesn't understand that billing addresses and shipping addresses aren't necessarily in the same country.

This year I was surprised to discover that PayPal, who you would think would have a clue, doesn't let you set a shipping address outside of your account's country. I was attempting to order an item from a Canadian website to be shipped to a Canadian address but because my PayPal account is a US account it will only let me create US shipping addresses.

This issue isn't unknown to PayPal, either, as evidenced by the "adding a shipping address in canada" and the "How do I use a foreign address?" threads on the PayPal's Community Help forums. This appears to be the official response:

It is not possible to add an foreign address to your PayPal account within PayPal. You can open a new account with your Canadian address and Canadian financial information.

Given that this appeared to be my only available option I decided to try and set up a Canadian PayPal account. This required that I come up with a new e-mail address for the account, since PayPal uses a single namespace for all accounts (arguably the right thing to do, but it doesn't interact well with the boneheaded policy of requiring a separate account for each country). Luckily I have an unlimited supply of e-mail addresses. The sign-up process then wants you to enter banking or credit card information. Of course, they are restricted to the country that you have selected, in my case Canada. I do not have a Canadian bank account or credit card (anymore). I was about to give up, but then I realized that I could just click on “my account” and bypass this step entirely. To complete my purchase I then:

Attempted to purchase with the merchant. This was just to find out the exact amount I was going to be charged.
Transferred funds from my US PayPal account to my Canadian PayPal account by “sending money” to myself. Having a second browser open was useful for this step. Thankfully, I was able to choose which currency to use in my US PayPal account so I didn't need to do any currency conversions by hand.
Waited several minutes for the funds to show up in my Canadian PayPal account.
Actually purchased the item from the merchant.

A few minutes (!) after purchasing the item PayPal actually called me on the phone. They wanted to make sure that I "still had control of my account", referring to the new account I had just created. I told him I did, and then I mentioned the annoyance of having to create a second account just so I could ship to another country. They confirmed that what I did was basically the only option, and said the reason for this is to make sure that each account complies with the laws of the country that it is associated with. This seems bogus. I could maybe understand only allowing banking information from a single country per account, but there's no good reason to put the same restriction on shipping information. PayPal does nothing with the shipping information except pass it along to the seller.

iOS Developer Agreement: Too Little Too Late

2010-09-09T11:37:00.000-07:00

It looks like Apple might be regaining some of their sanity given the recent update to the iOS developer agreement.

Compilers

Section 3.3.1 has been updated to only restrict the use of private APIs. This is a perfectly reasonable restriction. The clause which stated that “applications must be originally written in Objective-C” (in my mind, the most offensive part of the iOS developer agreement) has been removed. I'm very glad to see it's gone.

Interpreters

They also updated section 3.3.2, the “no interpreters” section. The language has changed but the meaning apparently hasn't:

An Application may not download or install executable code. Interpreted code may only be used in an Application if all scripts, code and interpreters are packaged in the Application and not downloaded. The only exception to the foregoing is scripts and code downloaded and run by Apple’s built-in WebKit framework.

The old version of this rule was confusing and unclear, and the new version, despite being less verbose, still leaves a lot open to interpretation. For starters, what does “install” mean in this context? If the user of the app manually constructs the executable code, is that allowed or not?

The definition of “executable code” isn't entirely clear either. My inclination is to assume that this means a Turing complete language, but one could argue that there are even non-Turing complete languages that count as “executable code”. For example, I wonder if an iOS port of the classic 8-bit educational game Rocky's Boots would run afoul of this rule. In the game you would construct machines out of various bits including Boolean logic gates and then use these machines to solve various puzzles. “Running” the machines in the game requires the interpreting of executable code.

Either way, the restrictions imposed by this section probably don't affect as many developers as the old 3.3.1 restrictions did. However, in some ways this rule is actually worse. The old 3.3.1 only restricted how one could build apps but it didn't really limit the types of apps that one could build. The no interpreters rule, however, actually makes it impossible to implement several classes of useful software on the iOS platform, including:

Web browsers that interpret JavaScript on the client.
Emulators of legacy platforms, like 8-bit computers or old game consoles, that allow the user to run their existing software (e.g.: game ROMs, etc.).
Educational development tools like Scratch.
Mathematics software like Mathematica or Maple.
Electronic circuit simulators.
PostScript or TeX viewers (both are Turing complete languages).

Apple implies that the reason for this rule is “security”:

In particular, we are relaxing all restrictions on the development tools used to create iOS apps, as long as the resulting apps do not download any code. This should give developers the flexibility they want, while preserving the security we need. [emphasis mine]

It's a pretty sad to see Apple is falling back on “security” as an excuse for limiting what customers can do with the products that they purchased. This is the same thing Sony did a few months ago when they removed “install other OS” (an advertised feature) from the PlayStation 3. In Sony's case the security issue had to do with their DRM. In other words, it wasn't a customer's security they were concerned for, but their profit's. One has to wonder if Apple has similar motives. An interpreter acts as a sandbox, so un-trusted code execution there is generally not as big a deal as arbitrary native code execution, as might result from a buffer overflow or similar bug in native code. Last I checked, Apple wasn't prohibiting string manipulation in native apps.

Analytics

Like 3.3.1, section 3.3.9, the privacy and analytics section, has also changed for the better. The language that specifically forbade Google's AdMob is gone, meaning developers can decide which advertising platform to use.

Why?

Apple says in their announcement:

We have listened to our developers and taken much of their feedback to heart. Based on their input, today we are making some important changes to our iOS Developer Program license in sections 3.3.1, 3.3.2 and 3.3.9 to relax some restrictions we put in place earlier this year.

Apple clearly didn't anticipate the backlash that would be caused by 3.3.1 when the “originally in Objective-C” clause was added. Not only were developers angered by that rule, but since its addition, people have been looking much more closely at what's in the developer agreement. Apple doesn't want this scrutiny as it brings to light already existing ridiculous rules, like 3.3.2, and makes people more likely to question Apple's motives when new rules are introduced, like 3.3.9. It also made many developers (and tech savvy users) who liked Apple (myself included) re-evaluate whether this was really a company they wanted to purchase products from or develop for.

I think there's also a possibility that the recent changes to 3.3.9 were made in order to avoid legal issues.

Neither of these are really great reasons for Apple to change their behavior. I think Steve Jobs preferred the older set of rules, but it became clear that developers, and potentially even the law, wouldn't stand for them.

To iOS or not to iOS

The current developer agreement is a lot closer in meaning to the pre-iPad developer agreement. Back when the iPad came out I had considered getting one so that I could experiment with developing for iOS. I gave up on that plan when the “originally in Objective-C” rule was added. So now that the rules are pretty much back where they were, am I going to get an iOS device?

Probably not. Apple has lost my trust, and in order to win it back they'll have to do more than just change things back to the way they were. For starters, I'd like to see them make a rule for themselves that the developer agreement will apply not just to third-party developers, but also to Apple's own iOS apps. For existing Apple apps that violate the rules they can then choose to revise the agreement for everyone, fix the app, remove the app. Apple already has an advantage over third-party developers, so for them to impose rules whose only apparent purpose is to strengthen Apple's advantage over third-party developers is reprehensible. I'm looking at you Safari.

Better yet would be to make it possible for people to distribute native iOS apps without going through the App Store. I'd care a lot less about the App Store policies are if there were other ways to get apps on iDevices. I'm fine with this being a setting users need to enable (as it is on Android devices), but requiring that the user "jailbreak" their device to get such basic functionality is not acceptable.

Steve Jobs and iPhone Developer Agreement Section 3.3.1

2010-04-13T11:41:00.000-07:00

On reading Steve Jobs’s response on Section 3.3.1, there are two things that struck me:

First, like John Gruber, he doesn't seem to understand the implications of the new text in section 3.3.1. In particular, it says:

Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine...

This does not prohibit the use of cross-platform toolkits, which is what he seems to think it does. There are many cross platform toolkits that are written in C and C++. What it does prevent (as I mentioned in my post yesterday) is using other languages, something Jobs's argument completely ignores. That he wouldn't understand what the the text of this section actually means is pretty disturbing.

The other thing that struck me is his reason for wanting to keep cross-platform toolkits off of the iPhone OS. He's afraid that cross platform toolkits will make it easier to write apps that exist on both iPhone OS and competing platforms, and that if the same apps exist on other platforms people won't have a reason to purchase iPhone OS products. This suggests a lack of faith in the technology of the iPhone OS platform.

With the changes Apple has made to section 3.3.1 they are setting a very bad precedent. People have complained about the behavior of other platform owners in the past, including IBM, Microsoft, and the various game console manufacturers, for imposing all sorts of nasty restrictions on developers, but none of them have ever gone as far as to forbid the use of either toolkits or source-to-source translation. That Apple would do this suggests to me both a lack of respect for developers (and hence, a failure to understand that without third party developers a platform is basically worthless), and also a lack of faith in their own platform's merits.

iQuality™

2010-04-12T10:45:00.000-07:00

iPhone/iPad app developers:

Pay $99 for the privilege to improve Apple's platform by developing new apps for it.
Have only one option for distributing their software, which involves paying Apple 30% of their revenue.
Risk having their apps disapproved after the app has already been developed. Reasons for disapproval include:
- Having functionality that overlaps with existing or future Apple products.
- Including an interpreter of any sort as part of the app.
Note that initial approval doesn't mean you won't get pulled from the App Store later on because Apple decided to change the rules.
And now, due to Apple's recent changes to section 3.3.1 of the iPhone developer agreement, developers must use an Apple prescribed programming language: Objective-C, C or C++. This new rule means that you can't even use a compiler that emits one of the allowed languages.

Apple apologists often claim that these hurdles improve the user experience by acting as a sort of quality control. Sometimes a comparison to the "Nintendo Seal of Quality" is made.

The Nintendo Seal of Quality was developed by Nintendo in the 80's as a response to the "video game crash" that happened earlier that decade. One of the reasons for the crash was a glut of crappy games that made people wary of buying games in general, and eventually lead to the (albeit, prolonged) death of Atari and much of the video game industry at the time.

A recent example of this sort of argument was John Gruber's Daring Fireball post on the change to section 3.3.1 of the iPhone developer agreement. In it he says:

On the one side, this rule should be good for quality. Cross-platform software toolkits have never — ever — produced top-notch native apps for Apple platforms. Not for the classic Mac OS, not for Mac OS X, and not for iPhone OS. Such apps generally have been downright crummy.
...

The iPhone OS Kindle app is excellent, a worthy rival in terms of experience to Apple’s own iBooks. The Mac Kindle app is a turd that doesn’t look, feel, or behave like a real Mac app. The iPhone OS Kindle app is a native iPhone app, written in Cocoa Touch. The Mac Kindle app was produced using the cross-platform Qt toolkit.

There are a few serious flaws with his reasoning here.

First of all, there's the question of whether cross-platform toolkits have ever produced top-notch native apps for Apple platforms. (The "for Apple platforms" qualifier here seems superfluous to me, but I'll let it go.) He claims that they never have, but I doubt that he has actually performed a exhaustive survey of software for Apple platforms to determine this.

Virtually all software with a graphical user interface that exists on more than one platform makes use of some sort of cross-platform toolkit. While it's easy to spot a badly ported application, applications that have been ported properly are indistinguishable from ones that were written "natively". It wouldn't surprise me at all if a number of the "top-notch apps" Gruber uses actually do use a cross-platform toolkit without his knowledge.

In fact, Firefox is built using XPCOM, a cross-platform toolkit (the "XP" actually stands of "cross-platform"). One could even argue that WebKit is a cross-platform toolkit as well (though it's specific to HTML rendering, rather than being a general GUI toolkit). So if Gruber is right then none of Safari, Google Chrome, or Firefox are "top-notch native apps".

But for the sake of argument, let's say Gruber is right and that cross-platform toolkits really have never produced top-notch native apps for any Apple platform. It doesn't matter, because the "cross-platform toolkits" argument is a complete red herring. Take a close look at the text of section 3.3.1:

Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).

There's no talk about toolkits here — it's talking about languages. There is nothing about this rule that prevents you from using a cross-platform toolkit provided that it's in Objective-C, C, or C++. Qt is a C++ toolkit, so if it weren't for the fact that it relies on a pre-processor (which is unusual for cross-platform GUI toolkits) then there's nothing about section 3.3.1 that says you couldn't port a QT app, like the Kindle application, to the iPhone OS.

Some of the things section 3.3.1 actually will prevent are:

Writing apps in other languages that teams may be more proficient in (Lisp, OCaml, Erlang, Scala, Python, Ruby, Java C#) even if C, C++ or Objective-C is used as an intermediate step.
Hand-coding inner loops in assembly code (a common practice in games).
Using domain-specific languages for specific tasks. For example:
- Using Lex, YACC, ANTLR or any other compile time parser generator. (for example: to produce a parser for parsing data fetched from the web)
- Using a compile-time template engine. (for example: to generate HTML for emails)
- Using gperf for perfect hashing.
Using code transformation tools. For example:
- Internationalization tools that perform build-time string extraction and reinsertion.
- Macro processors like m4.
Using existing libraries written in other languages. For example, many numerical libraries are written in Fortran.

As a software engineer, I'd be very surprised if any of the above had a correlation with lower quality applications. On the contrary, it tends to be the more proficient software developers that use a wider variety of tools, including ones like those listed above. (Luckily, actually identifying apps that violate this policy would be very hard, if not impossible, without the source code. If it weren't for this fact, we'd likely see a number of existing apps get pulled off of the store.)

Other App Store policies have similar problems. For example, it seems extremely unlikely that containing an interpreter is positively correlated with low quality. Instead, the no interpreters rule locks out a certain class of app. You cannot have a PostScript or TeX viewer without an interpreter, for example. Emulators for old machines are also affected. (Technically an emulator is an interpreter, though Apple seems to allow emulators as long as they don't let the user supply new code to be emulated/interpreted.) Even Safari runs afoul of this rule given that it includes a JavaScript interpreter, but of course Apple doesn't need to follow their own rules.

But never mind what I think. Even if the intent behind these policies was to keep low quality apps out of the App Store (which is hard to believe) the fact that the App Store is full of crappy apps is evidence enough that they've failed in this regard. These policies don't work to keep low quality apps out. The various high profile examples of apps that have been either disallowed, left in review limbo or lobotomized also speak strongly against the notion that Apple's policies have anything to do with improving things for their users.

So to sum up:

Many of Apple's policies don't appear to have been designed to improve quality.
If they were designed to improve quality, the large number of garbage apps on the App Store make it clear that they aren't really working.

AT&T Wireless: getting off to a bad start

2009-09-12T15:12:00.000-07:00

I have an unlocked Android (G1) phone. Miranda would like an iPhone, so we've decided to switch to AT&T. She can get her iPhone and I'll put a SIM card into my Android phone. (I'm okay with 3G not working on the Android.)

My initial thought was to get a free phone from AT&T, But then it occurred to me that this would use up my phone subsidy for two years. I tried adding a second line to a family plan on AT&T's website and then removing the phone, but it wouldn't let me complete the order. I decided to chat with their customer support. Here's the conversation I just had with them:

▶Thank you for your patience! An AT&T sales representative will be with you shortly.
▶You are now chatting with Tiffany, an AT&T sales representative.
Tiffany: Hello, Laurence.
Tiffany: Welcome to AT&T online Sales support. How may I assist you with placing your order today?
Laurence: I want to get a family plan with 2 lines, but only want one phone, as I already have an unlocked GSM phone that I would like to use.
Laurence: Is there a way to do that?
Tiffany: I will be glad to assist you.
Tiffany: You may order a free phone so that you can get the SIM Card from that phone.
Laurence: Im thinking about if the phone I'm using breaks or something, I'd probably rather get a better phone than just using the free phone. By getting the free phone I'm essentially locking myself out of a subsidized phone for 2 years though, right?
Laurence: (It also seems like a waste to get a phone I'll never actually use.)
Tiffany: If you would like, you may purchase a SIM Card in the store for $25, but once you sign the contract, it will be 2 years before a upgrade discount is given.
Laurence: I have to wait 2 years to get a subsidized phone even if I don't get any phone to begin with?
Tiffany: Yes, since the terms are for two years.
Laurence: So there isn't any way to get the contract, but hold off on getting a phone until later unless I want to either wait 2 years, or pay the full, unsubsidized, price?
Tiffany: You may set the line up online as a individual plan, then when the phone arrives, you may merge the lines in the store as a family plan and inquire on later getting a phone at a discounted price if you are not purchasing a phone at the time.
Laurence: I'm a bit confused.
Laurence: Let's pretend I was just getting an individual plan.
Laurence: Is there a way I could get an individual plan with just a SIM card (no phone), and then a few months later get a phone at the same price as if I'd gotten it when I got the plan?
Tiffany: The SIM Card will provide you with service, the service is for two years, it is not necessarily the phone, it is the contract that you are agreeing to which determines your discounts on the phones.
Tiffany: You term will still be 2 years, making it that amount of time to upgrade to a new phone for a discount.
Laurence: I understand that. I'm just surprised that I have to use the discount at the beginning of the contract or I lose it.
Tiffany: I do understand.
Tiffany: I do recommend that you visit your local AT&T Store to inquire on the procedure of using it at a later time.
Laurence: Ok, I'll try that. Thanks.
Tiffany: It was my pleasure, may I be of further assistance today?
Laurence: No, that's all I wanted to ask about.

The part that's especially ridiculous to me is that AT&T is essentially forcing me to waste their money. If they would let me save my subsidized phone discount for later, chances are I wouldn't even use it. Instead they're telling me I have to use it now, which means I definitely will and on a phone that I'm not even going to use. I don't even want to think about what that $25 charge for a SIM card that Tiffany mentioned is about.

GXP Now Open Source

2008-07-23T09:30:00.000-07:00

Google XML Pages (GXP) is now open source! See the announcement on the Google Open Source Blog for more info.

Speaking at OSCON 2008

2008-07-18T12:11:00.000-07:00

I'll be speaking at O'Reilly's Open Source Convention (OSCON) next week about Google XML Pages. GXP is an XML-based templating system that I originally built almost 7 years ago, and have been working on (mostly 20%-time) since then. It's hard to believe, but it's finally being open-sourced.

UI Design in Sunshine

2008-02-28T08:36:00.000-08:00

I just saw Sunshine, and while going through the extras I was pleasantly surprised by this short video about the design of the user interfaces on board the ship in the movie. The whole video is pretty interesting, but the part that made me literally laugh out loud starts just after 1:47.

Flickr Photo Page Bookmarklet

2008-01-21T21:16:00.000-08:00

Occasionally I come across images that are hosted by Flickr, but which don't have a link back to the photo page on Flickr. This seems to be especially common on message boards and image bookmarking sites like FFFFound. Being able to get to the Flickr photo page is nice for a number of reasons: you can see other photos by the same author, see the metadata, notes, comments, others sizes, etc.

A Flickr photo URL looks something like this:

  http://farm2.static.flickr.com/1407/1085850486_23589455e2.jpg

Flickr has some nice documentation for their URLs. The URL above is of the form http://farmfarm-id.static.flickr.com/server-id/id_secret.jpg, so the photo ID is 1085850486 and the "secret" is 23589455e2. It's possible to find the photo page by using Flicker's API call "flickr.photos.getInfo", which returns XML containing the photo ID. That's tedious, and won't work in a bookmarklet.

Some more searching turned up the http://flickr.com/photo.gne?id=id URL. I can't find any official documentation for this, only some posts in the Flick help forums. It appears to work though, and it wasn't too hard to turn it into a bookmarklet.

To install the bookmarklet, just drag the following link to your bookmarks or toolbar.

[Flickr photo page]

To use the bookmarklet, first make sure that the photo is the only thing open in the browser tab/window. In other words, the URL in the address bar should be for the .jpg hosted on flickr.com. If the photo is embedded in a page you'll generally need to select "View Image" (or equivalent) from the context menu. (FFFFound is an exception: it caches the images, so you'll instead need to click on the image to be taken to the original) One you're on the image, just click the bookmarklet to be sent to the Flickr photo page.

Rant: Bad Internationalization

2007-07-11T23:09:00.000-07:00

A lot of websites don't seem to "get" the fact that the Internet is international. I'm not talking about things like being translated into multiple languages or outputting content in UTF-8 (in this post). What really annoys me are sites that seem to actually go out of their way to make life difficult for international users and/or customers.

Credit cards have the nice property that they work internationally. If someone accepts Visa or Mastercard they'll normally accept those cards no matter what country the cards originate in or what currency the account is in. Currency conversion is handled automatically by the credit card issuing bank. A number of websites seem to be incapable of dealing with this properly, however.

Air Canada

An example I recently encountered is Air Canada's website. I was going to purchase some plane tickets for some family in Canada. After going through the ordeal of finding flights that met the various constraints I got to the payment page. I then entered my credit card number and most of my billing address only to find that "United States" wasn't in the country selector. "Is it listed as America? USA?" I thought to myself. Nope. It just wasn't there.

After a few minutes of trying to figure out what was up I noticed a paragraph in the sidebar that helpfully pointed out that the United States, as well as a handful of other countries, had been intentionally left out of the available options based on the country selection I had (apparently) made when I first visited the site. The only rationale I can think of for this is that it's a way to charge customers in different countries different prices. Blech.

Dell

Another incident happened back around Christmas. I was going to order a present for my father, who lives in Canada, from Dell. I first tried using dell.com, but it wouldn't let me ship to Canada. I then tried dell.ca, but the billing page had the country field hard-coded to Canada. My US-based credit card wasn't going to work.

After failing with both websites, I decided to try and get in touch with a human. "I'll just call the number on dell.ca", I thought. The number is 1-800-WWW-DELL (phone over HTTP?), and it's the same number on dell.com. Calling the number from the US connected me to Dell US, not Dell Canada, and they wouldn't ship to Canada. I asked if they could connect me to Dell Canada, and the sales person suggested that I just have the present delivered to myself, in California, and then I could ship it to Canada myself.

This idea is all kinds of terrible because it would mean more work on my part, be more expensive, take longer for the gift to arrive, and it'll also mean that the recipient would have have to pay GST -- not a very nice Christmas present. (GST is a tax applied to items as they cross into Canada.) By ordering from Dell Canada I could pay the GST, rather than the recipient having to pay it, the shipping would be direct (and presumably from a warehouse in Canada), and would be included in the price.

Eventually I was able to convince the Dell US salesperson to give me a real phone number for Dell Canada sales, and I was able to order the gift. Still, this would've been far less painless if dell.ca just let me choose the billing country for my credit card. Better yet, dell.com should allow shipping to Canada by forwarding my order to Dell Canada for me.

With the Dell example I think the reason for this screwup was less malice, more oversight/ignorance/stupidity. However, in some ways it's even more sad that even Dell, a company known for being relatively 'net savvy, can't seem to get this right.

Voodoo Electronics

2007-06-03T12:05:00.000-07:00

I'm interested in robotics, but building robots requires three things: mechanics, electronics and software. (I'm intentionally ignoring BEAM here -- I'm not interested in building software-free robots.) I know software, and I feel like I have an okay grasp of mechanical systems. I don't know enough about electronics, though.

My understanding of electronics is that there are basically three types of electronics:

AC electronics. Not especially relevant to building robots.
DC digital electronics. This part I feel like I pretty much understand. It's based on boolean logic, and so a lot of my software knowledge is applicable. The only extra bits are pretty simple:
- you're dealing with high versus low rather than true/false, so you need to come up with a mapping convention for each circuit and stick with it.
- many components (like ICs) also need power. Just get a power source that's the right voltage and enough current, and this should be okay.
- there are a few different "families" of digital components. Stick with one for any one circuit.
DC analog electronics. This has two main parts. The first part is the "simple bits": things like Ohm's law which every single electronics tutorial feels the need to tell you about. The second part is what I've come to call "voodoo electronics" -- the parts that every electronics tutorial I've ever seen simply hand-waves away.

A recent example of "voodoo electronics" that I encountered was in the most recent issue of Make. This issue had a number of electronics tutorials. Some of these were very helpful, like Your Electronics Workbench. The article on the 555 timer chip, however, The Biggest Little Chip, suffered from the voodoo electronic problem. The article contained the following schematic:

It then went on to explain the purpose of each of the various components one by one. The explanations were a bit terse, but reasonable. Eventually, the entire circuit had been explained except for R3 and C2. Here's how the article explained their purpose:

R3 protects the LED from excessive current, while C2 protects the 555 timer from random noise.

"Excessive current"? "Random electronic noise"? How am I supposed to design my own circuits without understanding how to both predict when these problems will exist and how to devise a defense against them? So far I have yet to see a single electronics tutorial that explains these things.

I ran into the same thing when I found some schematics for a stepper motor controller on the web. All of the schematics made sense to me, except a mysterious zener diode which was there (according to the documentation) "to absorb reverse EMF". No more explanation was given.

Are there any electronics tutorials out there that actually explain how to predict when part of a circuit will be susceptible to "random electronic noise", "excessive current", or "reverse EMF" and how to build protection against them, or is my only hope to take a full-fledged course in electrical engineering?

Update

After posting this I realized that I could just post a comment on Make's website. The author of the 555 article posted a very helpful response. Now I just need to figure out what that zener diode in that stepper motor controller was for.

Copyright Law and the King

2007-05-01T11:10:00.000-07:00

From the U.S. Copyright Office's What Does Copyright Protect? page:

How do I protect my sighting of Elvis?
Copyright law does not protect sightings. However, copyright law will protect your photo (or other depiction) of your sighting of Elvis. Just send it to us with a Form VA application and the filing fee. No one can lawfully use your photo of your sighting, although someone else may file his own photo of his sighting. Copyright law protects the original photograph, not the subject of the photograph.

Easier Links with Google AJAX Search API

2007-03-10T16:21:00.000-08:00

Ever get annoyed by the complexity of adding links in blog or message board posts? Sure, it isn't rocket science, but between getting the angle brackets, quotes and start and end tags balanced correctly and getting the URL right, adding links is a pain.

This weekend I did some playing around with the Google AJAX Search API and made a nifty bookmarklet for creating links in textareas. I've only tested it with Firefox (various combinations of 1.5 and 2.0 on Linux, Windows and Mac), and it only works with plain old multi-line textareas (not 1-line text fields or rich/"wysiwyg" text controls). Aside from those caveats, it seems to work pretty well.

Installation is easy: just drag this link to your bookmark toolbar: [linkify] Note that if you're reading this in a feed reader you'll probably need to visit my site for this step, as feed readers usually strip out JavaScript. (Readlings: no smirking.)

Here's how to use it:

Go to some page that has a textarea. The comments area on most blogs should work. The Blogger posting form in "Edit Html" mode also works.
Type some text in the textarea.
Select the part you'd like to turn into a link.
Activate the bookmarklet. This will bring up a search pane on the right side of your browser window.

The search query defaults to the text you'd highlighted. You can edit the query if necessary.
Click on "create link" next to the search result. The popup will disappear, and the text you selected earlier (in the textarea) will be replaced with a link. The link's text will be the original selected text, and its href will be the URL of the search result you picked.

Thanks to Chris Wetherell for telling me about inter-frame communication with URL fragments, which turned out to be handy.

Update

I've been told that the bookmarklet works in Opera, but not in Safari. I'll look into the Safari issue.

Intellectual Tug-O-War

2007-01-10T13:59:00.001-08:00

Why is it that almost all of the online debate about intellectual property is between extremists? On the one hand we have Richard Stallman, Cory Doctorow and piracy apologists. On the other hand we have the RIAA, MPAA and Microsoft/Apple astroturfers. I tend to disagree with both of these groups because both of them go too far -- just in opposite directions.

The most recent example of this is the Analysis of Microsoft's Suicide Note page. It's on fsf.org, so of course it's no surprise that most of the comments are from free-software groupies. The only exceptions are a few posts from "sreiser". I actually agree with some of his points, but I strongly disagree with:7

by affording basic intellectual property protection within the operating system, they [Microsoft] are doing great service towards cultivating a more responsible public

The responses are, of course, equally extreme, but diametrically opposite. For example:

...private piracy, on the other hand, does not cause a real harm to the manufacturer, due to its nature. By private piracy I refer to those people who download a certain product from Internet or otherwise make unauthorized copies of it only for self use. This is, of course, still illegal (although there are legal holes in many countries than enable it), but even so it requires a closer look: most of 'private pirates' (most probably more that 99%) copy the product just because they cannot afford buying it. So, if you disable such user from doing that, this won't make him/her to buy that product, since cannot afford it. Instead that user won't use the product at all.

I don't agree with either of these quoted bits. On the one hand, if I'm buying an operating system, that operating system should serve me, not the content providers. If there was somehow some sort of DRM that didn't cause any harm, then I'd be okay with it, but not causing any harm means:

It shouldn't make my machine slower.
It shouldn't make my machine run hotter.
It shouldn't increase the cost of the operating system or computer.
It shouldn't increase the liklihood of bugs.
It shouldn't make any legitimate tasks that I might want to perform more difficult. This includes fair-use tasks like backing up my media, moving it between my machines, burning it to a CD so I can listen to it in my car, etc.

Since no such DRM exists (or can exist, given the fuzziness of what is fair use) it's better to do without.

On the other hand, the "private piracy doen't hurt anyone" argument simply doesn't hold water. There generally isn't a black and white distinction between what one can afford and what one cannot. Can the typical teenager afford to buy thousands of CDs? Probably not. Can they afford to buy tens or even hundreds of CDs? For most American teenagers the answer is probably yes. So is it okay for someone to buy as many CDs as they can afford, and then to pirate the rest of their music?

Even if you did consider such an absurd situation to be morally justified, I argue that it would rarely happen. Once someone accepts piracy as a legitimate option they'll pirate whenever the percieved benefit minus cost outweighs that of purchasing. Given that many people's wants expand to exceed their means, and that physical goods are notoriously hard to pirate, people who have taken this path will not purchase media even if they can afford it. Worse, this sort of behavior can spread to those well outside of the "can't afford it" group because it lowers the social stigma of pirating, while at the same time decreasing the percieved value of the purchased product.

One of the craziest things I've experienced when trying to talk with these extremeists is that they generally see me as being on the opposite side of the spectrum from them. The anti-copyright crowd, upon hearing that I think content creators should be able to use copyright in order to get compensation for their work will invariably accuse me of being against open source/"free" software. I actually like open source (I'm typing this on my Ubuntu box) and I've even contributed to some projects. I think of "open source" as a feature though, not a moral imperative. While I prefer my software to be open source, I don't consider the developers of non-open-source software to be infidels.

Likewise, members of the the pro-DRM camp, upon hearing that I dislike DRM, will generally accuse me of being a software pirate. I just don't want fair use to be harmed. The "don't even talk about circumvention" bits in the DMCA are also frighteningly similar to thoughtcrime.

Am I really the only one here in the middle? In the end, I'm inclined to side with the "information wants to be free" crowd, but mostly because they seem to be on the losing side of the tug-of-war.

Book: On The Edge

2006-12-08T21:31:00.000-08:00

I recently finished reading On the Edge: The Spectacular Rise and Fall of Commodore. I had a string of Commodore computers from the time I was 8 up until I was in university, so it was great to find out more about the history behind these machines.

One thing that comes to mind when I think about the Commodore machines I had is how there was so little hardware abstraction. In modern PCs you rarely talk directly to the hardware. Instead there are high level languages and APIs piled on top of APIs. I'm not saying that having little hardware abstraction was unique to the Commodore machines. It was probably actually very common in other machines of that era as well, since the machines were so underpowered (by today's standards) that you couldn't really afford the extra overhead imposed by layers of abstraction. However, as far as I can tell, the Commodore machines tended to have even less hardware abstraction than their contemporaries.

I think a large part of this had to do with Commodore's habit of using custom chips in almost every machine since the VIC-20, which was actually named after the VIC chip the machine was based around. In the Commodore 64 there were the VIC-II and SID chips. On the Amiga, there was of course the famous Amiga custom chipset.

This hardware-centric philosophy permeated the system software as well. I remember programming on the Commodore 64 in BASIC 2.0, which provided absolutely no high-level support for sound or graphics whatsoever. This was on the machine that arguably had some of the best sound and graphics capabilities for its time. On the C64 everything had to be done in PEEKs and POKEs. Even on the Amiga, which had a "real" operating system, many hardware details would still peek (or is that poke?) through, and applications frequently took advantage of hardware features that were unique to the Amiga.

From reading On the Edge I think I have a better understanding of why the Commodore machines were like this. A big factor in this seems to be that the Commodore 64 and the Amiga development teams included chip designers. The C64 pretty much started with the VIC-II and SID chips, and the rest of the computer was built around them. This sort of makes sense, since Commodore was one of the only computer makers of that era to have its own chip fab. (According to the book, this was because Jack Tramiel, Commodore's founder, was obsessed with "vertical integration".) Likewise, the Amiga's design centered around its own custom chips largely because it was originally going to be a game machine. It even sounds like one of the reasons Amiga was willing to be bought by Commodore had to do with Commodore owning a chip fab.

The book talks about both the engineering and the marketing behind the Commodore machines. As a software engineer I'm obviously really interested in the engineering stories, but some of the marketing stories are also fascinating, especially given Commodore's notoriously bad marketing in its later years.

Here's a bit about Commodore's marketing tactics in 1983:

Commodore instigated a strange offer that literally removed the competition from homes. ... "Send in any sort of computing device and you can get $100 off your Commodore 64." ...
Commodore employees speculated that the offer actually increased sales of Timex-Sinclair computers. Customers sometimes purchased $50 Timex-Sinclair computers just to take advantage of the $100 rebate offer. The excess Sinclair computers became a running joke within the halls of Commodore. "We had all these Sinclairs," says Bil Herd. "I started using them as doorstops."

What's especially interesting is how these stories can sometimes intersect in surprising ways. During the development of the Commodore 128 there was difficulty getting the C64's CP/M and Magic Voice cartridges to work on the C128 prototype:

To fix the problem, Herd required the C128 to start at memory address zero, but the 8502 started elsewhere. "One night, everybody left and it was broken," says Herd. "During the night, I said, 'I have no way to fix this, unless we startup by not starting at that address.' I said, 'Hey, Von. The Z80 chip starts from zero, doesn't it?' He said, 'Yup.' I said, 'Cool. I need somebody wire wrapping tonight.'"
The hour was too late to purchase a Z80 chip, so Herd looked elsewhere. "Everybody had doorstops that were actually Sinclairs," he recalls. "I went and tore open my doorstop because we didn't own a Z80 chip in the place."

The Z80 in the C128 literally originated in a doorstop!

If, like me, you grew up with Commodore computers, or if you're just interested in the history of personal computers in general, then I highly recommend reading On the Edge.

Fubars for sale

2006-11-14T18:38:00.000-08:00

I looks like fubars really exist. This review cracked me up:

... if all you want to do is knock down some walls or kill some enemies, this is the way to go.

Scala Programming Language

2006-11-07T23:56:00.000-08:00

Today I went to a talk on the Scala Programming Language. According to its creators, Scala "smoothly integrates features of object-oriented and functional languages". It's also got static type checking, and from the little bit I've seen so far the type system seems fairly powerful. These are all things I've wanted in a language for quite some time.

The languages that I use at work, mostly Java and Python, are normally classified as "object-oriented". Python does have a bit of a functional flavor to it as well, but it seems that most people think of functional and object-oriented styles of programming as almost completely disjoint. I've been finding that I've been gradually moving to an ever more "functional style" of object-oriented programming, even when programming in Java, so I was happy to see a language designed to integrate the two styles.

Nerds versus geeks

2006-08-30T10:03:00.000-07:00

Ask Yahoo recently had a post titled "What's the difference between a nerd, a geek, and a dork?. Their definitions of nerd and geek are pretty consistent with the ones I've become used to since living in the SF Bay Area, but they aren't consistent with the way these terms were used when I was growing up.

In my high school a geek was someone who was unpopular while a nerd was someone who was unpopular but smart. I remember when I first started reading Slashdot, shortly after I moved to the Bay Area, I was really surprised at their use of the word "geek". "Geeks don't know how to use computers", I thought, "they're too busy learning to speak Klingon or memorizing the names of Star Wars characters". Sure, there are certain things that both geeks and nerds tend to be interested in, like science fiction, comic books and role playing games, but nerds are the ones who know how to do "useful" things. I became even more surprised when I learned that many slashdotters seemed to use an inverted set of definitions for geek and nerd.

Wikipedia has a bit of an explanation:

Pundits and observers dispute the relationship of the terms "nerd" and "geek" to one another. Some view the geek as a less technically skilled nerd. Others view the exact opposite.

They also reference an excellent Cat and Girl comic which, incidentally, defines nerd and geek in a way that's consistent with the definitions I grew up with.

It sounds like the definitions of these two terms are regional. The Wikipedia page suggests that this may be an east-coast versus west-coast thing: on the east coast people think nerds are smart, while on the west coast people think geeks are smart. That's certainly consistent with my experience, as I grew up in Ontario. It would be interesting if someone made a map-poll like the Pop vs. Soda Page. LazyWeb, don't fail me now!

Top 10 Java Classes I Love to Hate

2006-08-17T18:30:00.000-07:00

I haven't posted here in months. What better way is there to end a blogging dry spell than a good rant?

Here are ten Java classes in the standard API that annoy me whenever I have to deal with them, in no particular order:

java.io.File

An abstract file representation... or is it? It exposes system specific things like the File.separator and File.pathSeparator, yet it doesn't understand what "." and ".." do unless you canonicalize the File object. It's also tied to the system's filesystem. This means you generally need to build an abstraction layer on top of File. Finally, File.lastModified() returns a long. Why not a Date? (lastModified() used to be measured in arbitrary units from some arbitrary time. Not very useful if you want to be able to communicate the last modified time to the user or even another program. This was later fixed to be measured in milliseconds since the epoch.)

java.io.Serializable

First, it's a marker interface. Marker interfaces are generally a bad smell. They're a good sign that someone was being lazy, or not thinking very carefully about the problem they were trying to solve. In the case of Serialization it's especially bad because there are methods that probably should be in the interface: writeObject(), readObject() and getSerialVersionUID. Instead, reflection is used to find methods that have "magic names". A definite no-no in my book.

Of course, even that wouldn't fix the bigger issue which is that Java's serialization (like Python's pickling) is a very fragile mechanism for persisting objects which relies on the internal state of objects rather than on their public interfaces.

java.lang.Cloneable

Like Serializable, Cloneable is a marker interface. It's even easier to see what's wrong with Clonable, though. Where's the clone() method? The docs mention its absence. That doesn't stop it from being a bug.

java.text.MessageFormat

Overcomplicated. It defines a whole programming language for messages. This is great in theory, but it's not something you could actually give to translators. Hint: most translators are not computer programmers. Even if you manage to find a translator that can deal with the craziness of MessageFormat chances are they don't know every language you want to translate into.

java.text.SimpleDateFormat

Almost every time I see this class used I see the same bug in the code that uses it. SimpleDateFormat.format() is not reentrant and is not thread safe. Beyond that, DateFormat has one of the most bizarre APIs ever. It has a setCalendar method, but what does setting the "calendar" do? Why, it lets you get it back with getCalendar! It also lets you stomp on some of format's internal state if you call it concurrently, but presumably it isn't meant for that.

java.util.Calendar

What isn't wrong with this class? Well, I guess it isn't a marker interface, at least.

An instance of the Calendar class represents what? The answer should be "a calendar", but in fact an instance of this class represents a date. A mutable date whose mutators follow the rules of a particular calendar. Truly bizarre.

java.util.Date

Date isn't so bad. It has two main sins: First, Date objects are mutable. Second, somehow it made someone feel the urge to write Calendar.

java.util.Locale

Here's a dictionary definition for "locale":

lo·cale (lō-kăl')
n.
A place, especially with reference to a particular event: the locale of a crime.
The scene or setting, as of a novel.

Some of the constants in Locale fit this definition, and clearly represent "places":

static public final Locale CHINA = new Locale("zh","CN","");
static public final Locale FRANCE = new Locale("fr","FR","");
static public final Locale GERMANY = new Locale("de","DE","");

Others, not so much:

static public final Locale CHINESE = new Locale("zh","","");
static public final Locale FRENCH = new Locale("fr","","");
static public final Locale GERMAN = new Locale("de","","");

This is a pretty clear example of a common problem I've noticed with internationalization: a lot of people seem to confuse location with language. Is it because of the Locale class, or is the Locale class merely another victim of some sort of mass-hysteria? I don't know. In any case, the class probably should've been called language since it's obviously based on RFC1766: Tags for the Identification of Languages.

java.util.Stack

Stack is mostly annoying because it used up a good name. I occasionally want a stack, but I never want a stack that extends Vector. At least Vector and Hashtable had the decency to not use the only good names for what they do, but what else do you call a Stack? A LIFO? Java 1.6 will actually add Deque which can be used as a stack, but the interface is a lot "fatter" than than I'd like for the cases where I really just want a stack.

java.util.WeakHashMap

Good idea, poor execution. First, the name is ambiguous. Weak what? Weak keys? Weak values? It turns out that it's weak keys, but it would be nice if the name said as much. It would also be nice if there was a weak value version so that people were more aware that they had to make a decision, as the two types of "weakness" are not interchangeable. You typically want weak keys when you're trying to "annotate" existing objects, but don't want the annotations to outlive the objects. You typically want weak values when you're trying to implement a weak cache keyed by something that's "recreatable" (typically "value objects", like Strings or numbers).

Since it is a weak key hashmap it should actually be a weak key identity hashmap. It isn't, though. WeakHashMap uses Object.equals() despite the fact that weak references operate on identity.

Amazing Spaghetti

2006-05-31T09:15:00.000-07:00

On Gadgetopia I read that "Someone scanned in all the pages from the classic book BASIC Computer Games". I remember back in elementary school and high school I used to get magazines and books from the library with program listings.

I rarely ever actually typed in the listings, though -- I was too lazy for that. Instead, I'd often try to figure out how the code worked. My favorites were the listings that also had an explanation of how the program worked. Then I'd typically ignore the listing entirely and just try to write a program based on the description. I remember writing a fractal landscape generator in BASIC 7.0 for the Commodore 128 based on an article in Compute! which had a listing for an IBM PC fractal landscape generator.

Anyway, back to BASIC Computer Games: Looking at the scanned cover, I felt like I remembered this book. I remember borrowing it from the public library back in high school and being surprised by the seemingly ancient version of BASIC that it was using. I don't know if it was actually an "ancient version of BASIC", that's just what it seemed like to me at the time. The book was published in 1978, well before the Commodore 64, my first computer, even existed.

There was one program in the book that really intriguied me: a random maze generator called "Amazing". By the time I got my hands on the book my programming language of choice was Turbo Pascal, so I decided to try and port "Amazing". I never actually succeeded because Turbo Pascal is structured, while "Amazing" made copious use of spaghetti like GOTOs.

The first thing I did when I found out about the scan of the book today was to check if this really was the maze generator I remembered. I remembered that the program had lots of "GOTO 1000" statements, and that line 1000 was itself a GOTO statement (making the program worthy of TheDailyWTF, IMHO). Sure enough, if you check the listing of Amazing you can see that spaghetti I was up against.

The funny part is that the program listing is only about 150 lines long, which is actually pretty short. I think I'll take another crack at translating the program, though this time I'll probably use Python rather than Turbo Pascal.