Ted Benson

Adam Marcus on Remote Labor

2015-02-10T00:00:00+00:00

A few years ago, Adam Marcus gave a great lecture on all things related to remote labor over the web. Friends on twitter just brought up the talk in reference to an article on the sharing economy by Robert Reich, and I realized that the audio recording of it had fallen off the web.

So here it is:

Audio from the talk: Your browser does not support the audio tag! (download link)

PhD Defense: Creating Rich Designs and Read-Write-Compute Pages with a Relational Layer for Web Content

2014-08-05T00:00:00+00:00

Proud to finally have this flyer going out to the MIT email lists. If you are in Cambridge, please do attend!

Thesis Defense: Creating Rich Designs and Read-Write-Compute Pages with a Relational Layer for Web Content

Speaker: Ted Benson
Speaker Affiliation: MIT CSAIL
Host: David Karger

Date: Thursday, August 07, 2014
Time: 3:00 PM to 4:00 PM
Refreshments: 2:45 PM
Location: 32-G449 (Patil/Kiva) (32 Vassar Street)

Committee: David Karger (MIT CSAIL), Rob Miller (MIT CSAIL), Tim Berners-Lee (MIT CSAIL)

When we browse the web, we experience rich designs and data interactivity. But creating such web sites requires great engineering effort. As a result, novices are largely limited to content management systems like WordPress and experts rely on complex software toolchains like Ruby on Rails. My work shows that adding a declarative, relational layer to the web stack reduces the complexity of authoring and reusing web content by providing a way to reason about how structures on the web fit together and what should happen when they change. I apply this framework to the two goals of design reuse and content management. For design reuse, it enables new and more usable authoring and deployment techniques. For content management, it enables the authoring of read-write-compute applications without any procedural programming at all. User studies show that HTML novices can learn to apply this framework in only a few minutes, increasing their creative capacity beyond read-only rich text. I will show demonstrations of software created using this approach and discuss implications for web authoring going forward.

Relevant URL: For more information please contact: Ted Benson, eob@csail.mit.edu

Magical Source Code Highlighting

2014-08-02T00:00:00+00:00

This is perhaps the coolest command line trick I have learned in recent memory.

Any time I need to syntax highlight a snippet of HTML to insert on a presentation slide, I simply copy it to the clipboard, run highlight-html on the command line, and voila! The clipboard now contains properly syntax highlighted code ready for pasting.

Here’s how to set your computer up so you can do it too.

First, install the highlight package for brew

brew install highlight

Then create executable shell scripts located in $PATH of the following form:

pbpaste | highlight --syntax=html -O rtf | pbcopy

This one is named highlight-html. I’ve got others for other languages (CSS, Javascript, Python, etc).

That’s it.

I suspect this is the beginning of a beautiful relationship between pbpaste and pbcopy and my shell scripting.

Data quickie: Wordpress themes are getting more complicated over time

2013-10-22T00:00:00+00:00

Here’s a quickie for you. The other day I scraped all the themes from WordPress.org (615 in total, or at least that’s what my script pulled down) and then plotted how complex the theme was based on when it was created.

I used two ways to measure complexity: how many files were in the theme package and how many bytes were in the non-image files inside the theme package. These two metrics are somewhat related (we would expect the number of bytes to increase as the number of files increases) but also tell slightly different stories.

Number of files in a WordPress theme over time

Number of bytes in non-image theme files over time

Not counting bytes in image files is important because we should expect the quality (and thus file size) of images to increase over time.

Conclusion

Writing a theme is getting harder over time, and I suspect that this could be said of web sites in general. We need to work to make sure the tools for web authoring improve at the same rate as the level of sophistication expected by web authors and surfers.

Here are the data files: YearVersusNumberFiles.csv and YearVersusBytesNoImages.csv. And here is the R Script I used to create the figures: plot-data-versus.r

Flexible data formats are used more flexibly

2013-07-07T00:00:00+00:00

Proponents of self-describing storage cite this ability to uniquely tailor the information stored to each item as one of the style’s big benefits. Adding “one-off” properties to items, they claim, is a natural need of information storage.

But is it?

Well, here’s a short analysis I did the other morning that provides some evidence to help answer that question. I took about 1,800 datasets that I scraped from visualizations on the web and looked that property coverage for each property in the dataset. Let’s define the property coverage for property P as the number of items that have property P divided by the total number of items in the dataset. Property coverage of 1 means every item has the property. Property coverage of 0.5 means half of the items do.

Some of the datasets for these visualizations were authored in JSON, a self describing format. Others were authored using Google Spreadsheets, a schema-based format. If customizing items with one-off properties is truly one of the empirical benefits (not just claimed benefits) of self-describing storage, then we should expect to see more low-coverage properties in the JSON datasets.

And here is the graph with the results:

It’s nicely split into three general segments: authors who use JSON tend to create more properties with 0-50% coverage. Between 50% and 70% it’s a wash. And spreadsheet authors tend to use more properties with between 70% and 100% coverage.

While you can’t make any strong claims from this data, it does suggest that one-off (low coverage) properties are actually used in datasets in the wild. And if this chart generalizes, then users of self-describing formats make use of low coverage properties more often than users of schema-based formats, supporting the idea that this is one of the benefits to the approach in practice, as well as theory.

Eval in Template Languages Considered Harmful

2013-02-02T00:00:00+00:00

I’m of the opinion that supporting the eval statement in template languages is a Bad Idea, precisely because it’s such a useful outlet to have. How many of us have been delighted to write a short script in PHP, only to later bemoan the spaghetti code that ultimately results over the lifetime of the project.

A site called Egghead.io recently released a series of videos about Angular.JS, a really cool Javascript framework. They haven’t even gotten one minute into the first video (55 seconds, actually!) before they issue this disclaimer about Angular’s templates:

Many expressions can be evaluated in here. Now there’s a lot of things you shouldn’t do in here [inside the template’s eval affordance]. You should keep this logic to a minimum. Because this is your view and presentation. But there’s really quite a lot of things you can do in there.

Careful planning of architectural constraints is a bit like user interface design. Like user interfaces, development frameworks offer affordances (the commands they support), require learning (tiptoeing around the framework’s particular constraints), and benefit from homogenization.

But when you provide support for eval, it creates an easy escape hatch that prevents homogenization. If something doesn’t fit into your framework well, rather than fix the problem (or decide it isn’t important), one can just jump through the eval escape hatch, skirting the framework’s architectural choices and inlining arbitrary Javascript code.

This opinion might sound extreme, but I think it is called for when you consider that good framework designers strive to provide guarantees about the behavior and performance of code using their framework. This requires reasoning about code, and reasoning about code requires a declarative view of computation which eval destroys. It’s the difference between a toolbelt, like backbone.js and what you might call a Proper Architecture, like MapReduce.

Celebrity as Parable

2013-01-22T00:00:00+00:00

(Reaction to this blog post, which got some time on Hacker News a while back.)

Occasionally, our habit of fixating on celebrity leaves tech hopefuls feeling lost. It can be hard to watch other men and women attain riches while thinking, “I, too, have similar skills; what did I do wrong?”

You might have similar skills, but rocks rolling down from the top of a mountain end up in very different places depending on the bumps they hit on the way down. Just because you’re similarly shaped doesn’t mean you’ll end up with the same outcome.

I’m not claiming that there is nothing special about Steve Jobs or Bill Gates, just that these men are not singularly gifted. Each of our lives unfolds as collaboration between ourselves and those around us. If we’re to believe Gladwell’s message in Outliers, the mountain has as much, if not more, impact on trajectory as the shape of the rock.

When celebrating the Houstons and Zuckerbergs of the world, we should congratulate the achievements of the person, yes. But for our own purposes of self-growth, it is probably more healthy to think of them as parables, not people. Regardless of how you feel about the truth of Gladwell’s thesis, these CEOs represent 0.01% of the outcome space for starting a company. As such they’re all so far away from the mean that it is emotionally unhealthy and probably counterproductive to use them as personal role models anyway.

Instead, let them be archetypes. You can use Drew as the character through which to tell the story “scratch your own itch” or “symmetric sharing incentives.” You can use Bill to tell any number of stories, like: “license and become platform.” And so on.

Distill the essence of what they did well and let that–not their success–be your guide. It’s a bit similar to the idea of Chengyu (成語) in Chinese: short idioms that serve as poetic allusions to some situation. Sometimes these idioms call out a historical figure by name, like saying “Custer’s Last Stand” to describe a situation in [American] English.

Just for fun, here are three. I’m not a native speaker, so forgive me (and email me) if I’m mischaracterizing:

Yu Passed by his House Three Times without Entering (大禹三過家門而不入) Refers to Yu the Great, who designed and built a system of irrigation canals that tamed the floods which had been plaguing the Chinese people. He was so dedicated to this task that he didn’t return home to his wife for thirteen years until it was complete, despite passing by the very doors of his house three times during that period. Today the saying refers to someone wholly dedicated their country or work.
The old man lost his horse (塞翁失馬) refers to a story about an old man who once lived lived near a place called Xiongnu. When things happened to him–good or bad–he demonstrated wisdom by refraining from snap judgement, knowing that things that appear good or bad in the short term can often turn out to be the opposite in the long term. (Side note: what a great parable to include in a CS class on Greedy Algorithms!)
Dongshi Imitated a Frown (东施效颦) is like saying “cargo cult”, except with the additional meaning that the cargo cult imitation creates a bad (instead of merely neutral) outcome. Dong Shi was a woman who feigned chest pains, jealous of her beautiful friend who got lots of attention when she fell sick. But instead of getting more attention, Dongshi’s feigned illness scared people away.

Anyway, the point is, don’t fixate on celebrity. Everyone has the capacity to touch lives and do great work, and some people end up getting famous. By all means try for fame and glory, if that’s what you want, but it’s probably best if you use celebrities as parable, not role model, along the way.

Virtualization Makes Things Weird

2012-11-14T00:00:00+00:00

Ah, for the simple days, when a taxi was a taxi, and my video games weren’t helping terrorists behind my back.

Exhibit A: Johnathan Zittrain on Ethics Laundering via Mechanical Turk. 30 minutes but worth the watch.
Exhibit B: We’re not a Taxi Service; We just connect riders to cars

We’ve begun to virtualize many parts of the economy. And things are getting weird as a result.

Take the case of these new peer-to-peer taxi services popping up. It’s a cool idea, for sure. But we also agree that regulations are usually a Good Thing, providing they’re used to protect consumers instead of simply raising the barrier to entry of local business (see: Why It’s Illegal to Braid Hair Without a License).

But how do you regulate a virtualized taxi livery? A company that provdes taxi-like services without actually owning cars or employing drivers? They just play switchboard for anyone who wants to sign up.

There’s a game in Japan called Pachinko. It’s also weird. I can’t figure it out, despite having spent a good hour trying in Kyoto this summer. Pachinko involves shooting BB-gun sized metal balls into a Galton Box of sorts, in which you receive points for certain landing positions. As best as I can tell, the configuration of the game creates a calculable expected return for every possible angle from which you shoot these balls. This means the game boils down to (A) guessing what these expected payouts are and (B) holding your wrist at precisely the angle that achieves them.

Anyway, the point about Pachinko is, it’s gambling. It’s a billion dollar business. Except for a small problem: you can’t gamble in Japan. To avoid the prohibition, they’ve taken Chuck-E-Cheese style gaming to the next step.

You give Pachinko money
Pachinko gives you points
Points can be redeemed for toys, and (here’s the kicker)
Coincidentally, next door is a shop that purchases toys for money

It comes full circle.

Anti-gambling law avoided. Gamblers, proceed.

Virtualization is weird.

One might argue that the financial crash of 2008 was a result of the same song-and-dance. I can’t figure out how to concisely make the proper allusions here, but The Big Short is an excellent read if you’re interested.

I have some friends who are lawyers. They tell me the interesting thing about the law is that every case is a story, and the job of the court is to interpret that story in the context of existing law and precedent. Initially this frustrated me. Doesn’t this guarantee inconsistent outcomes? Can’t we just have a giant lookup table?

Section 3.4.5: An Eye.
Section 3.4.6: A Tooth.

But, seeing how fast things change, and how strange things get when companies virtualize our daily activities past the point where the law, as written, can address them…well, it makes me glad the courtroom involves stories, instead of a simple dictionary look-up.

Cross Origin Resource Sharing Gotcha: Don't forget about OPTIONS requests

2012-11-07T00:00:00+00:00

Anyone in the business of hacking the web past its original boundaries knows the importance of the browser’s cross-origin security model. But you also know how often we try to sneak around it.

For static content, JSONP – the typical sneak route – seems a bit hacky. Dynamic JSONP callback names require me to serve up static resources with a dynamic wrapper. And static callback names just don’t seem very principled: what if I have multiple requests in the pipeline? Yuck.

The workaround is to use Cross-Origin Resource Sharing (CORS), which is a relatively new family of HTTP headers that communicate to the browser that a server authorizes the use of its content by external origins. If you can swallow incompatibility from the usual suspects, it’s a nice option because it allows you to load static cross-origin content as if it were same-origin content.

All it takes is a few special HTTP headers. You can add these headers to a whole directory of static content by adding the following lines to your .htaccess file:

Header add Access-Control-Allow-Headers "x-requested-with"
Header add Access-Control-Allow-Methods "GET, PUT, POST, DELETE, HEAD, OPTIONS"
Header add Access-Control-Allow-Origin "*"

And here’s the catch that prompted me to write this post: most how-to snippets around the web seem to focus on the Allow-Origin header, which is the meat and potatoes of the whole thing. But it is important that you add the Allow-Methods one, too, because some browsers will do what is called a preflight request, which is an HTTP OPTIONS call to the endpoint to see, essentially, what’s cooking on the other end of the line.

The preflight is the browser’s first impression before it goes and fetches the real thing you want.

If you don’t explicitly signal that CORS is also OK for these OPTIONS requests, the server might make a bad first impression. The browser will think that CORS is not enabled, and then infer (incorrectly) that any subsequent GET request will fail. Wanting not to waste your time and bandwidth, the browser will then never even attempt the GET fetch you asked for. But – and here’s the rub – your browser console will report this failure to you as a failure of the GET request, throwing you off the scent of the real bug (a failed OPTIONS request).

So that’s it. CORS is your friend. And make sure not to forget about the OPTIONS preflight.

A Useful Little Pattern for Web Scraping

2012-07-17T00:00:00+00:00

I’ve been doing a lot of web and Google Spreadsheet scraping recently, and one situation I’ve run into a lot is that the schema of the data source doesn’t quite fit into the schema I’m trying to dump the data into. The data source might expose someone’s full name, for instance, whereas I want to store the first and last name separately. I’ve developed a useful little coding pattern to help with that situation that I thought I’d share here.

Let’s say that the scrape of any data source produces a result result : Map[A,B], where A and B are almost always strings in real life. For example:

val result = Map(
  "name" -> "Ted Benson",
)

Let’s define a “fixer” as a function that takes your result and outputs the missing key-value pairs that you would have liked to have seen.

type Fixer[A,B] = (Map[A,B]) => Map[A,B]

And then we define a function fix which simply applies a fixer to the result and folds its output back into the original map (possibly overwriting some keys).

def fix[A,B](m : Map[A,B], f : Fixer[A,B]) : Map[A,B] = m ++ f(m)

Now we’ve got an incredibly useful little function that can help us tidy up any schema-misaligned data that we’re pulling in. To split the full name into first and last components, we might do the following (pardon the lack of error checking):

val FixName = (m : Map[String,String]) => {
  val fl = m("name").split(" ")
  Map("first" -> fl(0), "last" -> fl(1))
}

and then, given our result object from above, we can fix it by simply saying

val fixed = fix(result, FixName)

which will result in the map

val fixed = Map(
  "name" -> "Ted Benson",
  "first" -> "Ted",
  "last" -> "Benson"
)

Why go through all this trouble to wrap a simple modification to a data object? Because if you’re scraping many different sites, you’re going to need a pipeline to automate the work for you. And by folding a “fix-it” step into this pipeline, and formalizing it like this, you can write your scraper bot in a domain-independent manner and then simply provide it with a chain of Fixer functions for each URL pattern you request of it.

Democracy has a Built in Pressure Release Valve

2012-03-27T00:00:00+00:00

I remember the first time I saw a picture of a steam engine governor. The device is ingenuous: its very design relies on the laws of nature to maintain a relatively constant speed. A rotating shaft, driven by steam, is attached to weight-bearing arms that hang down by its side. As the rotational speed increases, the centrifugal force causes the arms to swing outward, thus dampening the rotation of the shaft. The faster it tries to go, the more force prevents it from going fast. Anyway, the point is that, rather than an external digital controller, as we might build today, this old design is self-regulating.

It seems to me that democracy has this self-regulating property. At any given point in a nation, there plenty of people angry at the government. It builds up pressure, sows discord in the nation. A monarch or dictator would fear this pressure. But in a democracy this pressure is relieved by the act of voting, and all the political rituals (debates, etc) that go along with it.

What’s interesting about this is that it works regardless of whether or not “democracy” actually means that people control the fate of the nation. Modern American government is controlled by political parties, congressional voting blocs, and lobbying organizations – all of these are persistent, non-democratic entities. The real muscles which flex power in our country do not answer to voters.

And yet, we feel as if we control the government, because every four years we get our chance to “vote the bastard out” if we’re unhappy. Even if that act only amounts to changing the kabuki mask worn by the thousands-deep organizations that actually constitute power. The mask is important – it sets the tone and tenor of delivery, and has veto power – but ultimately it is just the presentational interface to a vastly more complex body behind it.

This is an interesting property of democracy when compared against systems which maintain a narrative of “we’re in control because we know what’s best for you” such as communist regimes. There is no built-in pressure release valve in those narratives. When you get upset at the government, how can you vent your steam?

Perhaps this is why communist regimes work so hard to censor information. With no built-in pressure release valve, they have to either construct an artificial one (as North Korea does, using the USA as its Emmanuel Goldstein) or sit in fear that their citizens will boot them out French Revolution style.

Democracy idealized inspires us. Democracy ritualized mollifies us.

Who Occupies Wall Street?

2012-03-14T00:00:00+00:00

I participated in the Cambridge get-together of the nationwide Occupy Data Hackathon 2 this weekend. Lots of interesting people, projects, and analyses going on.

Here’s the visualization I whipped up, which lets you browse the results from the Occupy Research General Demographics and Participation Survey, and the post on the [Occupy Research](http://www.occupyresearch.net] blog that linked to it. The data was cleaned up by Charlie DeTar of the Media Lab, and he also has a great – and much higher-fidelity – visualization of it available here. The code for both is available on GitHub.

NICAR, from a Programmer's Eyes

2012-03-06T00:00:00+00:00

Last week I attended NICAR 2012, a conference for computer assisted investigative reporting. I was there to help David teach reporters how to use tools such as Datapress and Exhibit and to learn about the needs and state-of-the-art of computers in reporting. It is always a privilege to get to visit and observe someone’s world as an outsider; and even more so to see how they are using tools from your world to do their work. So here are some parting thoughts I took away from the conference: NICAR from the eyes of a computer scientist.

Visualization is just the “last mile” of data in reporting. And an optional one at that. We spend a lot of time talking about data publication and visualization tools in Haystack, but this is just one small slice of the needs of computer-assisted reporting. A story might take months of investigative work- gathering data, cleaning data, interviewing people, assembling scraps of paper — and a presentation of that data is only prepared in the final run-up to publication. That presentation isn’t always a wiz-bang interactive graphic, either. Many times a data-intensive story might be presented entirely as narrative, if the medium fits better.

Scraping tools seriously needed. Web scraping wins the award for highest ratio of need over capability. Scraping the web is absolutely essential to do good investigative work when it comes to municipalities, many of which publish web pages with daily administrative information (such as arrests) while removing the previous days’. Without a scraper, reporters would have to spend a large portion of each day copying and pasting this information down into a spreadsheet by hand.

Big Opportunities for Automation. Ben Welsh of the LA Times gave a great talk about how he automates his reporting through a combination of web and email scrapers, databases, and automated copy generators. The goal being to auto-generate 100% of reactive stories (deaths, arrests, etc) and then go back and rewrite the most important ones by hand. Reminiscent of AtomsMasher for the newsroom.

And machine learning, too. Tools that enable reporters to perform topic modeling and hierarchical clustering are making a big splash. They can go a long way toward helping a reporter understand a big data dump so they know what documents to focus on. I think the coming years are going to big for the dissemination of machine learning components into a lot of consumer software. Tools that enable reporters to say “give me more documents like these ones” will be a bit hit.

The CMS is Broken. One refrain I heard over and over is how much the reporters have to fight with their CMS. For reasons both technical and administrative. A common solution seems to be for custom news apps to be hosted out of a subdomain on third party sites (AWS, Heroku, etc) with window dressing to provide the illusion of being a part of the main news site. However as a totally separate entity, these news apps don’t get integrated into the standard RSS feed, advertising system, top stories feature, and other critical elements of the CMS-managed web of data, casing traffic and revenue challenges for their authors.

What happens when your paper is the business of causing protests? I went one session where the creators of curbwise.com, two employees from The Omaha World-Herald, discussed the hopeful (revenue-wise) but new territory of transforming the news into apps. Here’s the gist: print ad revenues are falling, but online ad revenues are a pittance in comparison. To make up for lost revenue, newspapers can exploit their intimate knowledge of a locale by creating community-specific information sites and charging for them somehow. But what happens when that site is, like Curbwise, a way to protest your home’s valuation? Now the newspaper has a financial interest in causing people to protest their home valuations. Is that territory we should be comfortable with newspapers occupying? Do they occupy it already (to the extent that extreme news is news that sells well, so there’s always an incentive to fan a fire)? Food for thought.

Appification of news, in general. Continuing on the previous thought, there was enormous interest in the idea of transforming news into for-fee “apps” that deliver a targeted news experience. Such as paying a small fee to get your kid’s high school football scores in a format that looks like ESPN.com.

The need for computer science as a liberal arts requirement. If ever I haveeen a good argument for computer science as a liberal arts requirement, going to NICAR was it. It was amazing and energizing to see the extent to which computers are enabling better reporting and storytelling. In some cases, surprising to see how programming has become an essential tool for some areas of reporting. In today’s world, knowing how to program better equips you to make sense of the information around you and communicate your findings to others.

The need for computer scientists to grok liberal arts. On the other hand, we as computer scientists need to be delivering tools — serious data crunching tools, visualization tools, curation tools, scraping tools — that are built for use by people who spend their days thinking about things other than computers. Because I want my local reporters to spend their days fact checking the good stories, not brushing up on Python.

Roasted Cocoa Ice Cream

2011-11-12T00:00:00+00:00

1. Roast the Cocoa

Roast 60g of cocoa nibs @ 350 for 5 minutes

Watch out – these things burn fast! I ruined the first batch when I wasn’t attentive enough to a funny, almost burning plastic smell coming from the oven. The nibs come out almost as bitter as when they went in, but much chewy-er.

2. Make Some Cocoa Tea

Bring to a boil with 400g wholefat milk, turn off the heat, cover, and let steep for about 10 minutes.

3. Junk the Nibs

Strain ‘em outta there.

4. Custard it Up

In a bowl, mix two whole eggs with 80g sugar. Add it to the milk, stirring with a whisk. Put the temperature on low heat, and stir constantly, scraping the bottom, until you reach custard consistency. You know you’re there when a spoon dipped in comes out with a thick coat, but swiping a finger across the spoon leaves it clean. When you decide it’s time, immediately transfer the pot into a bath of ice water to stop the eggs from cooking.

Watch out: if you overdo this step everything will get grainy and lumpy. But fear not: I suck at this step but the result remains delicious in spite of me. You really can’t fail when the main incredients are cream and sugar.

5. Chill Out

Let the custardy milk cool down. I put it in the refridgerator or outside (in the winter). Or in the freezer if I’m already late to a party.

6. More Cowbell

Once it’s cool, mix in 200g heavy cream.

7. Churn baby, churn.

The machine knows what to do.

Enjoy with a few nibs on top.

A new generation of nuclear power opponents?

2011-03-12T00:00:00+00:00

Nuclear power is as close to magic as we’ve come when it comes to economically viable energy production, but it’s also controversial. A lot of the opposition is from the older generation who remember back to events like Chernobyl and Three Mile Island. Will the ongoing nuclear scare in Japan create a new generation of people opposed to nuclear power?

We’re witnessing the cost of a nuclear meltdown right now. Hopefully this is the worst things will get, but it’s already scary enough for me.

People don’t place bets if losing the bet carries too high a cost. And any engineering system is a bet. The output of a nuclear power plant isn’t actually Electricity + Waste as we typically think of it, the output is coarsely:

Success with probability p
Failure with probability (1-p)

Presumably, success has some positive value to society, Failure has some negative value, and p is very close to 1.

How do we, as a society, evaluate the output of the plant? As the expected return, Successp + Failure(1-p)? This is how the auto industry works, for example: they design with a tolerable expected deaths per vehicle-mile (like 0.0000000something).

The problem is expected return doesn’t work so well if failure is catastrophic. The reason is that whatever mathematical model we use estimate a value for p is inherently flawed, as all models are. Models don’t reflect the world, they are just rough approximations of it.

So the real probabilities are

Success with probability p-epsilon
Failure with probability (1-p+epsilon)

Where epsilon is some probability mass that represents the un-modeled failure situations – Donald Rumsfeld’s “unknown unknowns”. The problem with this term is that we don’t know how big it is. If we have catastrophic failure with probability (1-p+epsilon) and we don’t know how big epsilon is, that’s pretty scary.

The failure in Japan right now is happening somewhere in the probability mass covered by this epsilon term:

When Fukushima was built, TEPCO rated and tested reactors 1 & 2 to tremors equivalent of a 7.9 earthquake – the highest they thought was possible for the region. No need to test for an 8.9er – not factored into their risk model.

An 8.9 quake is the strongest in recorded history for Japan – 10 times worse than the 7.9 Fukushima was rated for (logarithmic scale). And it wasn’t just the quake that caused the reactor problems. Engineers spend a lot of time planning for shit to hit the fan. And when you look at catastrophic failures – the (1-p+epsilon) scenarios – it’s often a perfect storm of cascading failures that cause all of the checks and balances to fail.

In the case of this reactor, it was:

Earthquake hits, shuts down nuclear reactor, power goes out
Tsunami floods, takes out diesel backup generators that keep cooling system running
Backup batteries run out because the above national infrastructure is too torn up to replace them
Plant infrastructure (pipes, outer containers) crumble and burn, further damaging cooling system

That’s pretty complicated. And low probability.

The worst failure condition for a nuclear reactor doesn’t just kill people, it curses the earth for kilometers and decades. And that’s the scary thing: that there are unknown unknowns not factored into the risk assessments of engineering systems with potentially catastrophic results.

I think a whole new generation of people will grow up thinking that any power solution that places a nonzero – and worse, inherently unknowable – probability mass on nuclear fallout is unacceptable.

There’s an easy solution to this, of course: don’t build systems whose cost of failure is catastrophic. Will that be the policy mandate for years to come, or will we give in to the economic pressures created by scarce energy?

Join us for a Crowdsourcing Community Service Hackathon this weekend

2011-01-08T00:00:00+00:00

Calling all hackers, designers, and journalists in Boston!

A friend and I are organizing a “Crowdsourcing for Community Service” hackathon at MIT Friday-Sunday January 14-16. It should be a fun time and we would love for you to come out and join us.

All the details, and a signup form, can be found on the website.

We’re trying to get a group of creative people together for a weekend to try to solve a community service problem using mobile and crowdsourcing technology.

How often do you think: “If only we could each pitch in a small bit, the world would be a much better place.” Well, let’s get together, brainstorm a bit (Friday), create a tool or platform that helps us each pitch in that little bit (Sat & Sun), and then use it to better the Cambridge and Boston communities!

We’re interested in people of all skills that could contribute to a project like this, whether you’re a hardcore web hacker, a graphic designer, a writer, or a community organizer.

Hope to see you there,

Ted

Scala can be Scary

2010-12-07T00:00:00+00:00

With syntax like this, who needs enemies?

object ListCase {
	def matcher(l:List[Int]) {
		l match {
			case List(1,3,5,7) => println("Primes")
			case List(_,_,_,3,_) => println("3 on 3")
			case 1::rest => println("List starting with 1")
			case List(_*) => println("Other list")
		}
	}
}

Only at the Media Lab

2010-11-20T00:00:00+00:00

I was doing work on the 5th floor of the Media Lab the other day and pop music was echoing throughout the entire building. After about an hour, I finally went to investigate what it was:

The Semantic Web needs a MySQL

2010-11-16T00:00:00+00:00

One thing was clear in the comments of many industry-facing participants of ISWC 2010: a big impediment to adoption of semantic web technologies is the lack of an off-the-shelf triplestore that “just works.”

There are many other problems, of course: RDF an awkward format when it comes to real world programming because the graph model doesn’t align to the object-dictionary model of OO programming; JavaScript favors JSON instead of RDF; URIs and namespaces can be a burden to craft the first time around. But these problems can be lessened, or eradicated, with good development frameworks.

Underlying these surface problems is a deployment one: even if a company wanted to, there’s no clear hassle-free solution to getting a triplestore up and running with the same ease, access, and reliability that relational solutions such as MySQL and Postgres provide. And as long as this is the case, otherwise semantic-web savvy individuals are going to continue to live in the relational world. When people are spread thin, and want to focus on user experience instead of database administration, they’ll pick the database product that allows them to focus on other things.

So what gives? Do we wait for a Mike Stonebraker of the triplestore world to come around? Or do we try to bolt our technologies onto non-relational databases with gaining momentum such as MongoDB or CouchDB?

The Toothpaste Problem and Choosing the Right Data to Publish

2010-11-09T00:00:00+00:00

People who visit a toothpaste isle with only 4 products walk away much happier than those who visit the typical supermarket isle crammed with 40 variants of Colgate. Why? Because they don’t get overwhelmed by a tsunami of possibilities that leaves them wondering if they made the wrong choice.

When it comes to a large organization publishing data, perhaps a similar problem arises. Given all the information in the world that we could publish in structured form, how are we to know which important bits to address first?

Hans-Jörg Happel proposed an interesting way to solve this problem in the Social Semantic Web track at ISWC 2010 today. If we can quantify the need for a particular morsel of information, we can prioritize our efforts to structure and publish data. The question, then, becomes how to quantify information need.

Happel’s idea is to do this by examining missing values from query results. When someone performs a query, they’re stating that they need a particular data set. When one of the items in the query result is empty (such as missing 2010 GDP value for Mexico), that’s a known piece of information that someone needed and didn’t get. If we count up the number of times each of these NULL values occurs, we can begin to keep a priority queue of desired, but missing, data.

So if Mexico’s 2010 GDP is missing from WikiPedia, is that a problem? Well, count up the number queries that returned a NULL for this item and judge quantitatively. If the number is comparatively high, maybe we should prioritize the addition of Mexican economic stats.

He’s created a plugin for Semantic MediaWiki, called Semantic Need, which does exactly this. The list of prioritized information is called the “Extended Knowledge Base” — those things that we want to know, but don’t. As a programmer, I find this project very clever. Developers usually think of NULL values in query results as mere annoyances. But this work turns that around and makes them useful.

One of the themes of the Haystack Group is that focusing on user needs can direct research toward results that are immediately useful. On the semantic web, picking an explicit user goal (helping users communicate effectively using data) can be more effective than picking an abstract goal (building a web of linked data). Our project DataPress attempts to follow this philosophy by helping users add interesting visualizations to their blogs, and as a side effect, showing those users the value of structuring their data. Semantic Need follows this philosophy in another way: it attempts to quantify an existing, realized need for pieces of data so that we know which data is actually useful for structuring right now.

While the presentaiton didn’t address it, the idea behind this talk could be incredibly useful for government data. What if governments provided not links to data sets (as data.gov does) but rather some ontology and a query interface. Then it sits back and sees what users query for. Using an approach like this, the “what data should we publish” problem solves itself: the queries people ask will tell you what data to prioritize for publishing.

Here’s a link to the paper: Semantic Need: Guiding metadata annotations by questions people #ASK

If James Bond were a Linguist

2010-10-09T00:00:00+00:00

MIT is hosting the 2010 Empirical Methods in Natural Langauge Processing conference this year, and I noticed a clustering of papers in the program that would make for a fun session, possibly titled “Linguistic Security”. The session would cover both offense (what can we tell about you from the language you use) and defense (how can you hide messages in your word choice).

Since the EMNLP program is already fixed, the “2010 Ted’s Blog Workshop on Linguistic Security (TBWLS)”, will have to suffice :).

Here’s the program. Each item is taken from the real program:

Keynote: Why do we call it decoding?
Kevin Knight
Improving Gender Classification of Blog Authors
Arjun Mukherjee and Bing Liu
Modeling Perspective using Adaptor Grammars
Eric Hardisty, Jordan Boyd-Graber and Philip Resnik
Practical Linguistic Steganography using Contextual Synonym Substitution and Vertex Colour Coding
Ching-Yun Chang and Stephen Clark
A Latent Variable Model for Geographic Lexical Variation
Jacob Eisenstein, Brendan O’Connor, Noah A. Smith and Eric P. Xing

Bit by Flexibility: Implicit Conversions to Java with Scala 2.8

2010-10-06T00:00:00+00:00

Scala 2.8 includes a library that helps implicitly convert Scala objects to Java objects so you can keep your data in Scala-land while still using Java API calls. Just import this package in your code:jj

import scala.collection.JavaConversions._

The problem is sometimes the conversion library fails at compile time because there are just too many possible conversions it can make. It can’t decide between all the possibilities. Talk about being a victim of your own success!

Here’s an example: I have a scala.Iterable of items, and I want to implicitly convert it to a java.lang.Iterable

val trainingData:Iterable[ILabeledSeqDatum] = processTrainingFile(trainingFile) 
_crf.train(trainingData)

But the implicit conversion dies here with the following message:

[error]found   : scala.Iterable[edu.umass.nlp.ml.sequence.ILabeledSeqDatum]
[error]required: java.lang.Iterable[edu.umass.nlp.ml.sequence.ILabeledSeqDatum]
[error]Note that implicit conversions are not applicable because they are ambiguous:
[error]both method asCollection in object JavaConversions of type [A](i: Iterable[A])java.util.Collection[A]
[error]and method asIterable in object JavaConversions of type [A](i: Iterable[A])java.lang.Iterable[A] [error]are possible conversion functions from Iterable[edu.umass.nlp.ml.sequence.ILabeledSeqDatum] to java.lang.Iterable[edu.umass.nlp.ml.sequence.ILabeledSeqDatum]
[error]   _crf.train(trainingData)

So here’s the fix: you can wrap your data to indicate the particular conversion you would like to occur. A list of wrappers is here. In my case, I want a java.lang.Iterable, so I’ll wrap it as so:

val trainingData:Iterable[ILabeledSeqDatum] = processTrainingFile(trainingFile) 
val iw = new IterableWrapper[ILabeledSeqDatum](trainingData) 
_crf.train(iw)

This removes the ambiguity, allowing the compiler to proceed without baffling itself by its own cleverness.

The Right to Advertise Adult Services

2010-09-08T00:00:00+00:00

I don’t have a deep understanding of the law, but I think the recent closing of the “Adult Services” section on Craigslist is a fascinating moment to reflect how complicated and confusing regulating sin is.

Here’s the situation, as described by Matt Zimmerman of the Electronic Frontier Foundation, in case you don’t read geek news:

On Saturday, after years of pressure from law enforcement officials, Internet classified ad web site Craigslist bowed to demands to remove its “Adult Services” section which critics charged encouraged prostitution and other sex-related crimes.

At first glance, it seems like this might be pretty cut and dry:

X is illegal
Y has an advertising section for X
Y is therefore an accomplice to acts of X
So Y should be punished

But Craigslist isn’t advertising sex – its users are – and this turns out to be an important difference. Important because this situation represents a sort of edge-case between two separate goals we have as a society:

We want to make it illegal to do, or be an accomplice to, certain acts
We also want to protect telecom carriers from being liable for the messages they carry

The 1996 Communications Decency Act puts this second goal into law. It protects people who serve as a carrier of information from having liability for the third-party information they carry. An analogy in the physical world would be that the Department of Transportation isn’t responsible for how people choose to use their roads. If you drink and drive, it’s your responsibility, and the DOT isn’t an accomplice just because it “carried” your car. In the same way, providers of “online roads” (like Comcast, Google, and Craigslist) are not liable for the particular 0s and 1s that individuals choose to put into their systems. This is a critical protection for the internet to be a viable business platform. If it weren’t there, your ISP would be an accomplice if you planned a bank robbery over instant messenger.

But when Craigslist hosts a classified forum titled “Adult Services,” it is pretty clear what the intent of that forum is. They’re not just asking for any third-party messages, they’re asking for a particular type of third party messages, in this case one that tends to be illegal in most places.

Does the debate actually come down to semantics? Is it a crime to host a site titled “Post illegal prostitution ads here”, while a site with the innuendo-laden, yet ultimately nonspecific title “Adult Services” would be protected under the Communications Decency Act? What about forums where drug users hang out? Or a forum where people discuss ways to speed without getting a ticket? It gets grayer and grayer pretty quick, which is why this is an important issue to stop for a moment and think about.

My gut reaction on this issue is, “Well, yeah. If the act is illegal then of course they shouldn’t be allowed to host advertisements for it.”

But then I am reminded that the junctures at which it is most important to stand our ground on freedom of speech tend to be exactly those situations where it might not be comfortable to do so. Because if we start putting footnotes on which types of communication carriers are allowed to carry, then we’ve removed their protections entirely. If Google has to filter one message, then they have to filter them all. And that removes the very foundation of freedom of speech online.

Islamophobia

2010-09-07T00:00:00+00:00

Sometimes America makes me so proud. Sometimes it just makes me sad.

It Takes 6 Clicks to Protect your Geo-privacy on Facebook

2010-08-20T00:00:00+00:00

The battle of the geo-APIs has begun! Loopt and Foursquare out to the early lead. Facebook hot off the starting line. Twitter revving its engines at the gate. Geo-location APIs are going to bring about a huge, cool shift in the types of mobile information services we can use, but they also present many serious privacy and safety concerns that need to be carefully managed. The “if you don’t have something to hide…” argument against privacy simply doesn’t cut it in the real world: life and relationships are complex. Sometimes privacy lets you just be alone when you want to be alone, other times it can save your life.

Don’t take it from me, take it from Sam Altman, founder and CEO of Loopt, who describes in an interview to Charlie Rose just how delicate the privacy implications of these technologies are. He even explains why it is important for these services to let you lie about your location.

So now that the geo race has begun, it’s important to pay attention to how each player respects your privacy.

Facebook released its “Places” feature recently, which allows users to inform others where they are. What you might not know is, this feature also allows your friends to tell the world where you are. In typical Facebook fashion, it isn’t immediately clear how to disable this feature. In fact, it takes a total of six clicks to protect your location privacy on Facebook.

Here are screenshots from my Facebook account so you can see how to change your settings.

August 20, 2010 It Takes 6 Clicks to Protect your Geo-Privacy on Facebook Edit Delete Tags Autopost The battle of the geo-APIs has begun! Loopt and Foursquare out to the early lead. Facebook hot off the starting line. Twitter revving its engines at the gate. Geo-location APIs are going to bring about a huge, cool shift in the types of mobile information services we can use, but they also present many serious privacy and safety concerns that need to be carefully managed. The “if you don’t have something to hide…” argument against privacy simply doesn’t cut it in the real world: life and relationships are complex. Sometimes privacy lets you just be alone when you want to be alone, other times it can save your life. Don’t take it from me, take it from Sam Altman, founder and CEO of Loopt, who describes in an interview to Charlie Rose just how delicate the privacy implications of these technologies are. He even explains why it is important for these services to let you lie about your location. So now that the geo race has begun, it’s important to pay attention to how each player respects your privacy.

Click on “Account”
Click on “Privacy Settings”
Click on “Custom”
Click on “Customize Settings”
Scroll down to the “Things others share” section
Click on the “Select One” box next to “Friends can check me into Places”
Select either Enabled or Disabled

Whether people want to let their friends share their location is up to them. There’s no problem in publishing information that you truly don’t mind being published. But it is critical that we create software, and a culture around that software, that make it easy for users to choose how their information is used and divulged, and to that end, six clicks seems a bit many.

Solving Problems versus Selling Products

2010-08-10T00:00:00+00:00

Does your company solve a problem or sell a product? When times are good, the two missions may appear the same, but when times are hard, differences will become obvious.

The company whose mission it is to solve a problem embraces change, pivots with it, and works toward a future in which their customers have fewer problems. Will their business look the same in ten years? Don’t ask them: they operate like a good defense, responding to the necessities of the surrounding environment surrounding, possibly changing the plan totally.

The company whose mission is to sell a product fights change, because change perturbs the existing market base, and that creates risk. Their goal is to temporarily solve the effects of a problem, not to remove the problem. If you remove the problem entirely, why would anyone need your product? Will their business look the same in ten years? They hope so. That way the product production can be put on autopilot while they grow business by pursuing other products in other areas.

This distinction is important because it is a choice that must be made early on in a company’s growth, but it is one that probably won’t be noticed until much later in your company’s timeline. It’s an abstract decision and a deep one, a decision that will affect your entire company culture.

Intuit, apparently, has chosen to sell a product, as the LA Times reports. California is working on some great options to help its citizens file taxes for free, and Intuit, maker of the TurboTax program, has been spending barrels of money to prevent Californians from getting access to free and easy tax filing services. Intuit has demonstrated that their mission is not to solve the problem of complicated taxes – it is to sell you a product. What will your mission be?

Should the President have an Internet Kill Switch?

2010-06-13T00:00:00+00:00

A new bill in the Senate proposes to give the president the authority to shut down the internet in the event of an emergency. A kill switch, essentially. It will be interesting to see the debate on this, particularly from the standpoint of how the government views its role with regard to the network infrastructure of the country.

It is easy to easy to see why the pentagon would want a power like this. The internet is a vector by which enemies could digitally travel to and attack the country, but unlike airports, seaports, and border crossings, there is no way to enforce the border. It doesn’t even have to be international; domestic crime groups could attack our nuclear power plants, wall street, or any number of institutions and, right now, there is no way to pull up a paddy wagon and cordon off the area, so to speak. The people pushing this bill clearly see an “internet kill switch” as one way to provide the digital equivalent of shutting down a port, grounding airplanes, or enforcing a curfew.

The problem as I see it is that a kill switch is a rather brute way to respond the risk, and it comes with collateral costs both to our liberty but also to public safety.

From the liberty standpoint, at what point should the government be allowed to instruct citizens to stop communicating? That’s essentially what the internet boils down to – communication – so it isn’t an unfair question, even if the communication is done via a federally funded infrastructure. Some countries have already begun initial discussions about whether internet access should be considered a fundamental human right, and, to the extent that it is, perhaps the rights of the people outweigh the government’s need to feel secure.

I think a much stronger argument can be made against the bill from the standpoint of safety. The internet so intertwines our businesses today that it is impossible to predict what the effects of a national shutdown would be. Would surgical teams suddenly lose the ability to control their robotic instruments mid-surgery? Would hydropower plants suddenly lose their links to the computers that control them? Would police departments be unable to coordinate investigations? While not everything depends on the internet, I think an analogy to shutting down the electrical grid is in the right direction.

We need a way to protect our electronic infrastructure, and a new government department that manages it would probably be a good thing to have around (also a new division of the military). But instead of a kill switch, perhaps a more targeted approach could be taken. Identify risk areas of our current infrastructure and work to build better perimeters around those, or possibly localized kill switches: around just the wall street trading network, for example, or around just the a nuclear power plant’s network.

With a more targeted approach, the president could always still call up the handfull of big telecommunications companies and say “pull the plug,” but it would be an off-the-books, controversial move instead of a gun ready and waiting to be drawn from the holster. And with a move that big, maybe that’s the way it should be.

Browser Plugins Considered Dangerous

2010-06-12T00:00:00+00:00

It seems like the market for browser plugins is thriving. Firefox, Chrome, IE, and now even Safari all support them, and some are popular enough that even non techies are likely to have them.

In general, plugins are a cool and useful capability to have around, but I think they are also holding the web back.

Think about flash. A lot of ink has been spilled lately about the flash versus html5 issue. But had browser plugins never been a possibility, we would have had to develop a rich, standards compliant canvas tag long ago, because the only vector for innovation would be better html features.

I fear the same thing will happen to browser-side databases. With the likelihood diminishing that HTML5 will be able to specify a database standard that pleases the required parties, we might be left to pursue the “make up” option of the (now defunct) Gears-style database – all the goodness of a client-side database, but wrapped in a browser plugin. And just functional enough to keep us from feeling the urgency to standardize the technology.

I say throw away these system-level browser plugins and lets return to the days of no-holds-barred unauthorized HTML extension. Let Chrome and Firefox support incompatible supersets of HTML, because only then will experimentation be done in a way that is both disrupting – the losers will have no choice but to implement the standards of the victors – but also open. Because when you compete by extending HTML (instead of writing a plugin), you do so in a way that anyone is able to reproduce if they decide it is worth it to them. And that is far better than the closed world of browser plugins.

History is filled with alternating cycles of stability and disruption. After a good run by the Church of Web Standards Orthodoxy, I think we’re ready for some disruption.

AT&T' Clever Trick

2010-06-04T00:00:00+00:00

I think I know what AT&T is up to with the recent data plan change: they’re trying to switch everyone to a pay-per-MB plan just before videoconferencing hits the mobile market big time.

I, perhaps like you, only use about 100 MB a month because I only use my iPhone for information. Email, a website or two per day, and maps. I use my iPhone a lot, but I don’t think of myself as a multimedia user. So when the new rate plan came out, my first thought was, “Excellent! I can save $15 per month with no change in my behavior!”

And that’s the trick: Within the next year, the behavior of iPhone users is about to change, and ATT wants to get us to willingly opt-in to a pay-per-MB plan before we realize that. Because once videoconferencing hits the iPhone, even email-only iPhone users will be gobbling up huge data charges if they choose to see their phone conversations in addition to hearing them.

And then we’ll all wish we had stayed with the all-you-can-use plan.

Facebook's Product is You

2010-05-22T00:00:00+00:00

Much of the debate about Facebook’s new policies is missing another large issue. So far, the focus has been on privacy: essentially how to draw the messy line between public and private in the socially networked age. Just as important as where that line falls, however, is who owns the rights to use and redistribute the information artifacts left behind by your digital presence.

Here’s an example from the offline world: I wrote a book for a publisher who likes to put author photos on the front cover, and so I was required to grant them the rights to my image. Two years later, I was walking through a book store and saw my picture on a combo-pack of books that I didn’t even know existed. That was weird, but it was what I was explicitly told to expect when I signed my contact. I’m now a co-owner of my own image, but I got paid to share and I have a contract filed away. I can live with that.

Here’s the question, though: what about that photo of you holding two beers and wearing a cowboy hat? Or the fact that you once listed “AIDS” as your relationship status, even though you quickly changed it. Facebook’s database remembers. What about your religion (“Christian - Other”) or your interests (“acoustic rock, avocados, unicycling”) and the fact that three years ago your religion was something else (“Wiccan”). What about the fact that you keep looking at Christy’s beach pictures over and over – Facebook’s database now shows 42 times in a month – or that you are a member of the “Yes We Can!” group?

Forget about the privacy question for a moment, and put on your green accountant’s visor. Let’s talk about you, the product.

Every piece of information and every mouse click you make online is product. It is the product the modern web is making its money off of. Oo Nwoye hit the nail on the head when he put it this way: “On Facebook, the users are the products while the advertisers are the customers.” The question is, where’s your contract? How much agency do you have over the distribution of this product, and what do you get out of the bargain?

Thinking about it this way is important, because if you divide people into two groups, based on their privacy stance, I think you end up with two very different answers to the “Facebook problem”:

If you are not OK with your information being sold, would you be willing to pay a monthly fee to have Facebook access but be guaranteed your information wasn’t provided to third-parties? Facebook has to generate revenue to pay for the services they’re providing you, and if they can’t sell your personal information, you’re going to have to cough up a subscription fee. (As an aside, danah boyd has a great article about social networking as a public utility, addressing the “I don’t approve, but can’t cancel my account” problem).
If you are OK with your information being sold, why are you letting someone else make all the money off of you? Why not sell it yourself? For those people who really don’t care if corporations know that you are the founding member of the “Soccer is for Sissies” group, then why shouldn’t you be the one to profit from that? Either sign a profit-sharing contract with Facebook or act as your own free agent. There is a huge market opportunity here: create a browser plugin that offers your personal data to every web page you visit. You have an asking price (say, 5 cents per page) and they have an offer. If their offer meets your asking price, you send them your personal data. Triple your asking price and you sell them your entire social history as well.

Here’s my point: the privacy debate is real and important, but in order to tackle the societal challenges that the social networking era is throwing at us, we also need to start having a conversation that addresses the marketplace for our personal information. We are the products of this marketplace, every one of us, but I get the impression that we would rather pretend otherwise because it sounds bad to say it that way. Unless we acknowledge this bizarre new world, I fear we won’t build a legal infrastructure around it that gives us the individual agency we deserve.

Ten Rules for Radicals

2010-04-30T00:00:00+00:00

Carl Malamud gave an inspiring opening keynote for the final day of WWW 2010. Malamud has spent his career fighting to free government data, video, and documents so that the public can have access to them.

Here are his Ten Rules for Radicals. (Note: These aren’t all direct quotes since they were given as part of an hour-long speech.)

Everything you do is an experiment.
`When the authorities finally fire that starting gun, run as fast as you can, so that when they get queasy and have second thoughts, it is too late to stop. As a small player, the elephant can step on you, but you can outrun the elephant.
Eyeballs rule. Build a user base, and you have much more leverage than if you are just blowing smoke.
When you achieve your objective, don’t be afraid to turn around and be nice. You can bang on the table and be a pain in the ass, but when you finally get what you want from someone, its time to cooperate.
Keep asking. Keep rephrasing the question until they can say yes.
When you get the microphone, make sure you make your point clearly and succinctly.
Get standing. One can criticize government all one wants, and they will often ignore you. But if there is something clearly wrong and you can document it and publish it, they have no choice but to listen. Get standing, and you can insist.
Try to get the bureaucrats to threaten you.
Look for places where the government is clearly overreaching – something that is totally nuts. Hit them there. (Refusing to publish state statutes is an example.)
Don’t be afraid to fail.

And a hilarious comment on finding out that the FBI had been called in to investigate a “security incident” that his team of information-freeing volunteers had caused:

“Now I’ll grant that 20,000,000 document downloads was probably more than what the government was expecting, but I’ll remind you: surprising a bureaucrat isn’t illegal.”

Time Travel on the World Wide Web

2010-04-27T00:00:00+00:00

What if you could flip a switch in your browser and browse the web like it’s 1999? If the Memento project succeeds, you’ll soon be able to.

A joint project between the Los Alamos National Laboratory and Old Dominion University, Memento seeks to add time travel as a standard capability to the world wide web. “Time travel” refers to the ability to perform a computer operation as if were a different time than “now”: query a database as if it were yesterday, for example. On the web, this would mean a standardized way to access historical web resources.

“We have a web that doesn’t have a time dimension, and it is really important that we add one,” says Herbert Van de Sompel, one of the researchers who presented Memento at the Linked Open Data workshop at WWW 2010.

Their proposal would have time become one of the negotiable attributes of a web request, joining other negotiable parameters such as content type and language. In this system, the cnn.com home page could publish a bit of metadata pointing to a “Time Gate” responsible for handling time travel requests for that page. A web user would be able to request cnn.com along with a time parameter specifying “December 31, 1999″. The Time Gate would help negotiate a connection to an archival server that would provide the cnn.com page as of that point in time.

If this proposal was adopted, researchers would have a standardized way to download a web page’s contents as they existed at several points in the past, a task that is difficult (sometimes impossible) and time intensive today. Casual web users would also be able to travel back in time to see how a page looked on important days in history. The team is currently developing browser plugins that would take a user’s entire web session back in time, literally allowing you to browse the web as it was on the eve of the millennium, for example.

The proposal still has some challenges to work through. First is the “Time Gate” design that it uses to negotiate time. While this design allows third-party archival services and plays well with existing web caches, it poses scalability questions from the standpoint of the HTTP negotiation process. What if some future negotiable HTTP parameter also utilized third-party server to handle negotiation? In such a case, it would be unclear which third-party server needs to be consulted, in what order, for which set of negotiable parameters.

Another question relates to the abstract nature of resources on the web, as Jeni Tennison and John Sheradon pointed out. When a user asks for the population of London in 2000, are they asking about the population of London as published in 2000 (which may be data from 1998) or are they asking about the actual population of London in 2000 (which may not be published until 2001)? From the standpoint of the web, the former is time travel on the document (the data) and the latter is time travel on the abstract resource (the City of London). If we are to increasingly link our data sets, this scenario will arise often, and some guidelines for interpreting the precise meaning of a time travel request will have to be addressed.

Now, or later, or later simulating now, check out the project home page: http://www.mementoweb.org/

The Scale of Things, from Yahoo!'s Perspective

2010-04-26T00:00:00+00:00

Sometimes it is fun to get a picture of the scale that different organizations deal with in their daily tasks. I just saw a talk from Yahoo! about their use of Hadoop which was rich with such numbers. Here are a few of the cooler ones:

Compute Power

Yahoo has roughly 30,000 nodes running in tens of clusters.
Each node has roughly four 1TB disks, 8 cores, and 16 GB RAM

This means that they’ve got 240,000 cores running.. That’s a lot of silicon. Their largest cluster is 4000 machines, which they believe to be an upper limit with the current version of Hadoop. This cluster handles in excess of 100,000 hadoop jobs per day.

The Web is Big

The data that their Hadoop clusters have access to sums to over 3 Petabytes, compressed and un-replicated.

The web itself has over 2 trillion links.

Hadoop makes things Fast [1]

Using Hadoop has improved Yahoo!’s ability to process web-scale data. Pre & Post-Hadoop statics are pretty stunning:

For clustering and ranking: 100s of features -> 1000s of features, weeks of processing -> hours of processing.

Dictionary building across web logs: 4 weeks to run -> 30 minutes to run, 2-3 weeks to develop, 2-3 days to develop

Link analysis: capable of handling 100s of billions of URLs -> capable of trillions of URLs, took months -> takes days

[1] Footnote: Of course, MapReduce systems only work well for certain types of computation. They don’t make everything fast.

Inflection Point of the Social Web

2010-04-22T00:00:00+00:00

Sweaty, exhausted, the lacrosse team huddles together at half time. They are losing 0-4 and the mood of the home-team crowd in the stands has already quieted to reflect the impending defeat.

The whistle blows to start the second half, and they win the face-off. The play they’ve devised is crazy, but it works: four seconds into the second half, they score their first goal. The fans go wild. The team’s heats beat with a renewed determination. They go back to win the game, 5-4.

An inflection point is the point on any graph where the curvature changes. Its the mathematical equivalent of the pivotal goal that turns the momentum of a sports game around. Melodramatic sports metaphors aside, I think the social web might have just hit an inflection point. And oddly, Facebook – the walled garden of the social web – is the one who might have just given it to us.

I just had a chance to scan the F8 announcements from the other day, but two important bits stuck out at me as both surprising and exciting:

First, the Open Graph protocol. Facebook has created a standard for embedding social links within the HTML of a web page. What is important isn’t the protocol itself – many people have made similar attempts. What is important is that Facebook, the 1000lb gorilla in the room, is putting their weight behind one, and it appears to be one that plays nicely with the vision of linked data that so many in the research community share.

Second, Facebook appears to be removing their policy that prevents data export. Perviously, third parties were only allowed to retain your data for twenty-four hours. This created the ultimate dependence relationship on Facebook: make any social application you want, but you will forever be dependent on the Facebook API to make it work. Now, if I’m hearing Zuckerberg’s keynote correctly, Facebook will allow third-party services to import your Facebook data into their own databases. There may be a catch – I haven’t read the TOS yet – but if there isn’t, this is a big deal because it bulldozes an enormous hole through Facebook’s walled garden.

I think Facebook may have assessed their adoption numbers and trajectory and decided that user lock-in is no longer important because of the sheer momentum they have behind their social platform. So they can now relax their lock-in policies and aim their sites on web-wide service integration.

What are your reactions to Facebook’s announcement?

Newspapers are Dead. Long Live News!

2010-04-09T00:00:00+00:00

I saw an interesting talk at CSAIL today by Nick Bilton, the lead technology writer for the New York Times Bits Blog.

He talked about a lot of things – data visualizations, the multi-tasking lifestyle, Internet-enabled journalism. He left an impression that the New York Times is way ahead of the curve in terms of news in the Internet Age. While their flagship offering may still be traditional ink on pulp, they have a fully-staffed R&D lab dedicated to prototyping new ways of finding, composing, and distributing news with technology. For an organization that uses old English text for its logo, that’s impressive.

But what struck me as most interesting, given his position, was his matter-of-fact attitude toward the changes in news: Newspapers will become extinct. Maybe in five, ten, or twenty years – whenever flexible OLED screens become cheap enough to use instead of paper. And in the face of the free-for-all publication platform that is the web, the traditional model of a select few organizations providing news to the masses will likely die out too. Yet there will always be news. And there will always be a market for news organizations and journalists, just as there is a market for experts in any field. They just won’t interact with us the same way they do now.

Close your Laptop and pull out a notepad

2010-03-25T00:00:00+00:00

The Irony of Wikileaks

2010-03-24T00:00:00+00:00

It’s been an interesting year for WikiLeaks. Having broken a number of important stories, they’re famous enough now that people outside the internet community have a chance at knowing who they are. But they’ve also had problems. They’ve shut down the site for a significant fundraising goal, and now, according to their twitter feed, they’re being pursued by the US Government for preparing to leak a murder video in which Uncle Sam is allegedly caught with the smoking gun.

First I want to acknowledge that a site like WikiLeaks is treading on grey territory by its very definition. There is a fine line between classified information and whistleblower-worthy information. While the ideological goal of WikiLeaks — to provide the world’s first truly free press — is noble, there is also a reason why organizations, like individuals, need privacy. Publishing the details of banking negotiations between Iceland and Britain, for example, doesn’t seem very noble to me. No more noble than if I published messy divorce proceedings I found in a neighbor’s trash can. But there have also been many instances where WikiLeaks has fearlessly exposed wrongdoing that would have otherwise been covered up.

Jumping over the philosophical argument though, the recent events concerning WikiLeaks got me thinking: how could we design a better WikiLeaks that is resistant to the problems it currently has? Namely, a WikiLeaks that 1) doesn’t need fundraising, and 2) can’t be shot by angry spies. What if WikiLeaks was a sort of Wiki-BitTorrent, served by thousands of people around the world running a background process on their computers. No fundraising needed here, it’s all P2P. And if the Wiki is maintained WikiPedia style, then we no longer need to pay for a centralized staff to curate it. Plus, there’s no centralized staff to kill or jail. It can be a decentralized, collaborative process.

And then I realized the irony. What if this experiment in citizen’s power is successful precisely because it is a dictatorship?

If WikiLeaks was P2P, then it could be attacked — poisoned at the infrastructural level by the governments and corporations that do not want to see it succeed. Even worse, a P2P WikiLeaks could be left standing, but its content could be poisoned with fake and misleading files. Sort of how the RIAA uploads music files to file-sharing networks that actually just contain 3 minutes of silence. With the signal-to-noise ration low enough, the whole operation would be discredited, and the site would essentially be useless. Even if such information-poisoning wasn’t known by the public, how could anyone trust an anonymously curated site to provide reliable truth on such secretive subjects?

It seems a single point of control — the editor — might be necessary for WikiLeaks to succeed, because we need a way to know whether to trust the explosive documents it publishes. If only we could place our absolute trust in an editor, then we can be assured of the veracity of the leaked documents.

And that makes WikiLeaks just as old fashioned as it is revolutionary.

Data Driven Computer Science

2010-03-22T00:00:00+00:00

Does a growing segment of computer science have more in common with particle physics than algorithm design?

Consider five small factoids:

When David Karger posted about List.it’s growing user adoption (more than 14,000!) a few weeks ago, the primary benefit he cited were the mounds of usage data the Haystack group now has available to analyze
Any major systems conference today is bound to have several star papers from the big industry players — Google, Yahoo, Microsoft, Facebook, etc — containing research made possible by company-proprietary data sets
Academics in natural language processing struggle to compete with companies like Google when it comes to algorithms that need lots of data. Google simply has so much data, and so many computers, that they can do heavy computational lifting others can not.
One of the prizes in Yahoo!’s Key Scientific Challenges contest is access to a portfolio of their private data sets for research
The Eyebrowse project is motivated, in part, by the fact that as researchers, we have no idea how people actually use the web. This is the sole privilege of companies like Google, Microsoft, and Yahoo! that own and run advertisement networks that track your movements across the web. Among other things, Eyebrowse is a research project to help researchers gain access to this information.

It is clear that there is a growing subset of computer science that is not about computers, but rather about the information we suddenly have available as a result of computers. What’s interesting is that this new type of study is found scattered throughout the subfields of computer science, yet it is distinctly different in nature than traditional computer science research. This begs the question of whether we need to adapt our existing approaches to research and education to reflect this new type of work.

Does the research start at the generation of the data set, for example, or the analysis of it? This is an incredibly important question because a good data set will make or break a research paper. Should PhD students be spending their first two years building and marketing a platform — essentially running a startup company — and then the following four years analyzing the usage data it generated? Should they take a lesson from Business School students and embed themselves in corporations, providing them access to proprietary data sets for study? Should they limit themselves to studying only public data sets?

Who pays to build data sets? Good data sets are expensive to obtain. Particle physicists spend billions of dollars constructing particle accelerators just so they can record a few milliseconds of good data. But governments willingly provide the money and resources to help them gather this data because there isn’t a market for gluon data. There is, however, a market for your social networking behavior and web advertising clicks, so we shouldn’t hold our breaths waiting for the NSFs of the world to fund a multi-billion dollar social network just to gather behavioral data.

Should we require researchers to publish data sets alongside their papers? My sense from biology students is that some biology labs today defensively guard their data to make sure they beat others to publication. How do we avoid data hoarding while still respecting the fact that generating a good data set takes a lot of insight and work?

If trends continue, datasets will become an increasingly important fuel for computer science research. Hopefully we can learn from the other scientific disciplines about how to cope with being data-driven and adopt community standards that encourage an environment of collaboration and sharing.

Uccello's Chalice

2009-10-01T00:00:00+00:00

Amazing. This is Paolo Uccello’s wireframe rendering of a chalice in the year 1450 – right around the time when perspective was being invented.

To think that the birth of the Renaissance would look so much like the birth of computer graphics..

How Safari and Firefox handle HTML 5 Manifest files

2009-09-26T00:00:00+00:00

I was doing some experiments with Adam in the lab on Friday, and we discovered some interesting variations in the way that Firefox and Safari implement the HTML 5 Cache Manifest specification. I think this is a particularly important feature to have implemented consistently across platforms because it is the make-or-break feature of HTML5 that will permit web applications to function offline.

First, what is the manifest?

For people who haven’t heard about this feature before, the manifest is essentially a special file that lists portions of a web site that should be cached locally for offline access. This is the feature of HTML 5 that will standardize the type of “airplane mode” access that GMail users have with Google’s custom Gears plugin. The manifest is served as a regular old file, with MIME type text/cache-manifest, and is linked from the html tag itself, as follows:

<html manifest="site.manifest">

 ..

</html>

Once a web site is marked as being cached, then the browser will use the local cached copy of all the files specified in the manifest instead of attempting to load them from the internet. Say you’re on an airplane and type in the URL for http://my_cached_site.com. The browser will recognize it as a cached one, load it from its local storage instead, and then use a new JavaScript API to inform the web site that it is running in offline mode. So now for the important part, how do these two browsers (Firefox and Safari) handle this file?

Firefox

Upon loading an HTML5 document with a manifest attached, Firefox firsts asks permission to cache the site offline before requesting the manifest file from the server. Here is how the toolbar looks on my browser:

And here is the server log (I’m using a Rails project to test this) to show that the manifest was not yet loaded:

If you choose to allow offline caching, the web browser then requests the cache file, as can be seen from this screen shot.

Now here’s the cool thing, I set the headers on the manifest file such that the manifest file itself should also be cached on the client side:

 headers["Expires"] = "Fri, 30 Oct 2010 14:19:41 GMT"
 headers["Cache-Control"] = "max-age=3600, must-revalidate"

And the result of this is that the subsequent load, no files at all are loaded from Firefox – it operates entirely offline. Notice the completely empty server log as I reload the site 2..n times.

Safari

Now let’s look at how Safari does it. Upon loading the web page, Safari also does not load the manifest file, as can be seen from this screen shot:

However, it also does not ask any questions about offline access. The next time I load the web page, something strange happens. Safari checks the manifest file twice and then doesn’t load the actual HTML page (because it doesn’t have to). The double-loading of the manifest file appears to be on the second page load, not split 1/1 between the page departure and subsequent reload. A little strange, if you ask me.

Furthermore, when I reload the page, despite the HTTP headers specifying that the manifest should be cached, Safari reloads the manifest file. Though, at least it only loads it once for every subsequent time:

Conclusion

I’m no spec-master, but it seems like Firefox’s implementation of this feature is what I would want to happen as a web architect, while Safari’s behavior seems a bit strange.

Firefox:

Only loads the web page once
Asks the user for permission to enter offline mode
Only downloads the manifest file once if given permission
Then obeys HTTP Cache Control headers to suppress reloading the manifest file on future loads

If Safari were to also behave like this, there are a few fixes that need to be implemented. Namely:

Ask the user if offline access should be allowed
Load the manifest when the user loads the page the first time (and approves offline mode), not the second time, when the user might be on an airplane
Stop loading the manifest file multiple times in a single page load
Start obeying the HTTP cache headers so that zero web connections are necessary if the cache says so

Safari’s Manifest handling quirks aside, both browser teams should be applauded for so aggressively implementing the HTML5 spec. It is a real treat as someone researching web platforms to get to test the in-progress spec on real browsers instead of just talking about what might eventually happen down the road.

This article is cross-posted on the Haystack Blog.

Will the Namespace Traffic Jam Kill RDFa in HTML5?

2009-09-21T00:00:00+00:00

One of the most exciting aspects of the (in-progress) HTML5 specification is the number of data-centric features it contains. It’s almost as if the committee is saying a big, “OK, OK! We heard you!” to all the data-heads out there and is providing not one, not two, not three, but four different ways to access and manage structured data from within the client browser:

Data Attributes, are key-value pairs that may be added to any DOM node
Microdata provides a way to interweave objects and object-properties amidst the DOM
RDFa provides a way to interweave RDF amidst the DOM
Client-side Database Support provides a full relational data access from JavaScript (the spec says this will be SQL compliant, but in reality it will likely just be the SQLite subset of SQL).

These are all great developments, and will no doubt bring about a lot of creativity about how data can be used on the client-side, but what interests me the most is why the HTML5 working group felt the need to include Microdata alongside RDFa. The capabilities of HTML5 Microdata and RDFa are nearly identical, albeit with slightly different terminology. Both provide a way to embed data within HTML attributes and tag contents. Both allow for both named entities and blank nodes. And both allow for a variety of more complex constructions, such as lists and HREF property values.

One of the only real differences, as I can tell from glancing over the specs, is that RDFa requires URIs whereas Microdata simply uses ordinary strings to reference entities and properties. And that is what worries me: one of the biggest benefits of RDF is its use of URIs, yet URIs seem to be exactly what is preventing the adoption of RDF. One problem is probably that URIs look funny as data model elements, even to a programmer.

“A person has name” is much more natural sounding than “A http://csail.mit.edu/Contact#Person has a http://csail.mit.edu/Contact#name”. We think of our code in natural language terms, and URIs obfuscate our realp world metaphors. Far more serious a problem is the namespace traffic jam that currently exists. If I want to publish an RDF document that describes this blog, for example, best practice would have me draw class types and property types from no less than six ontologies!

The RDF ontology to describe object properties
The RDFS ontology to describe object classes and labels
The Dublin Core (DC) ontology to describe the titles, authors, and the like
The Friend of a Friend (FOAF) ontology to describe my contact information
The XSD ontology to describe literal dates, strings, and numbers
And yet another, custom, ontology to describe everything else particular to the blog

That is already 6 ontologies, and we haven’t even raised the possibility of using OWL Time, Snap, Span, and GeoOWL for things like time and space description! Even for a semantic web developer, the complexity of managing all of these ontologies, and the namespaces that go with them, becomes pretty burdensome pretty quickly.

And that is why I worry about the future of RDFa in HTML5. It appears that the Microdata specification in HTML5 is essentially the RDF graph data model with the URIs neutered out. Given essentially the same data model, no doubt most developers will pick the easier of two formats to implement. In order to get more people on the RDF bandwagon, we need to make the RDF path just as easy to follow as the Microdata one.

How can this be done? If you ask me, the best way is to get rid of this namespace traffic jam and cultivate a set of community-oriented ontologies. Rather than trying to create base ontologies that address abstract universal concepts, why not try to have each community standardize a single ontology for their particular domain. Have WordPress and Blogger sponsor the Blog Ontology. Have Amazon.com and eBay sponsor the Marketplace Ontology. Have Facebook and MySpace sponsor the Social Ontology.

Then, instead of reusing bits from other ontologies, such as dc:creator or foaf:name, have each of these community-focused ontologies be self-sufficient, covering all the concepts necessary for their domain. We can always apply mapping rules to distinguish between social:name and store:book-author-name later.

With only a single ontology per domain area to worry about, the namespace traffic jam will disappear and it will be easier for people to get on board with RDF and RDFa. All in all, it seems the good news coming out of the HTML5 spec is that we can expect rich data annotation to soon be arriving to HTML content everywhere. But what we need to work on as a community is a way to make URIs, and the Ontologies that give them meaning, easier for programmers to use so that the web won’t just be full of data with Microdata, but full of linked data with RDFa.

A More Subtle Wordpress Security Hole

2009-09-08T00:00:00+00:00

This has come up twice in one day over here at the Haystack group, so I thought I’d post it here. Using Emacs and Vi to configure Wordpress can expose your database login and password to the world.

Here is why: Wordpress stores this configuration information in a file called wp-config.php that lives in the root of your install. This files contents aren’t visible to web visitors because its .php extension causes to be interpreted by PHP instead of returned to the web user. Emacs, by default, will save a backup file of anything you edit in the same directory. This file will end with a tilde character (~).

This means that if you use a non-customized version of Emacs to edit your wp-config.php file, you’ll get a second file called wp-config.php~. The .php~ extension is not registered to any particular MIME type, and so it will be returned, in full, to the remote user as text, revealing your database server, login, and password with it.

So please, if you use Emacs and have Wordpress, check your WP install directory for any Emacs backup files and delete them! There’s enough Wordpress hackery going around already this week :)

More Six Degrees: An Experimental Study of Search in Global Social Networks

2009-09-08T00:00:00+00:00

The followup paper to the prior one I summarized here, also published in Science, re-does the Milgram letter experiments over email. Before going further, here is the citation:

 Dodds, P. , Muhamad, R., Watts, D.
 An Experimental Study of Search in Global Social Networks.
 Science Vol 301. 8 August 2003.

They take the letter-writing experiments global, ultimately involving over 60,000 people in 13 countries, all trying to route messages to one of 18 targets with only local information. They found the following bits of interesting information:

Friends are routed to far more than other relationship-types
Work ties, School ties, and family ties are the three highest sources of the acquaintance, in decreasing order
People route to “fairly-close” friends more often than very close friends.

They verify that email, as with letters, reaches its target in 5-7 hops using local-knowledge social routing. And they find that routing in this experiment was fairly egalitarian – that is, no one person was critical to the success of the messages reaching targets. This is an interesting experiment to see how the prior letter-writing one would apply to the digital world. In addition, it collected some useful statistics about the types of people we choose when trying to route information to a target.

Controller Specific TTNavigation URLs with Three20

2009-09-04T00:00:00+00:00

One of the more unique parts of the programming style that comes with the Three20 framework its its URL navigation style via the TTNavigator class. This class is intimately tied in with table cell-based navigation. But what about when you have a URL that is context sensitive. That is, it doesn’t route to a general function, but rather a specific instance. Here is a scenario like the one I have in my code: I have a “profile” view that allows people to publish various social links about themselves, and one of these is email. When you click the email table cell, I would like it to trigger a message to that controller which opens up the email client: [self sendEmail];

The problem is that TTTableItem objects want URLs, not delegates and selectors. How do I wire up a URL to this particular controller instance instead of just a general one? The answer is to stop thinking of TTNavigator routes as something that has to be setup at application start time.

I don’t know if you think like this, but I realized I had subconsciously been thinking about routes the Ruby on Rails way – parameters that are only mutable during initialization. But TTNavigator routes can be changed any time. So here is how I implemented the email link. Each time the PersonController initializes, I have the following code:

 TTNavigator* navigator = [TTNavigator navigator];
 TTURLMap* map = navigator.URLMap;
 
 [map removeURL:@"app://person/email"]
 [map from:@"app://person/email"
      toViewController:self
      selector:@selector(sendEmail)];

Notice how I first remove any existing routes, and then I re-wire that route to the particular controller that has just been initialized. Then when I create my table cell, I can just use a URL as always:

 [cells addObject:
        [TTTableImageItem itemWithText:@"Send Email"
                          imageURL:@"bundle://email.png"
                          URL:@"app://person/email"]];

And that is it! Let me know how you have solved the same problem. As Three20 doesn’t have much in the way of documentation, I would love to hear how other Objective-C hackers are handling situations like this.

A Formal Model for Travers & Milgram's Letter Experiments

2009-09-03T00:00:00+00:00

Back to school next week, so in my never ending quest to learn how to do Real Research, I’m going to try to begin reading papers regularly. To keep momentum (and to be a better blogger), I’ll write summaries here.

First up is Identity and Search in Social Networks by Watts, D.J.; Dodds, P.S.; Newman, M.E.J. from Science Vol 296, 2002. They create a formal model to describe social networks which predicts similar path-lengths as Travers and Milgram’s famous letter-writing experiments in the 1960s.

For those have haven’t heard, these experiments are where the term “six degrees of separation” come from. The two scientists randomly chose residents in Nebraska and Massachusetts to send a letter to a target person in Boston, and, if they didn’t know the person directly they were to mail it to the person they thought might know them with the same instructions.

They found that the average length of the chain from random sender to target was about 6 (and, as an aside, that the probability of termination at any point in the path was 0.25). Watts, Dodds, and Newman creates a tunable social network model that is able to predict the results of Travers and Milgram, but also resembles many real-world computer science problems.

The essence (read: oversimplification) of their model is that individuals are described by (1) connections to friends, and (2) a feature vector of traits, and (3) a tendency to interact with and make social decisions based on hierarchy, where hierarchy is expressed within the feature vector. Setting up the model as such, it stands to reason that an individual with only local information – that is, knowledge of their own feature vector and the feature vectors of their friends – would route a message to a target T to the friend whose feature vector best matches that of the target. Approaching the world hierarchically, such a friend is most likely to be in an overlapping social group to the target. They then test their model with a variety of different parameters, with a few interesting findings:

If H, the feature vector, is of too high a dimensionality, the path becomes a random walk. This matches real-world intuition: if we were unable to bucket our social network into groups, we would have trouble deciding which friend to turn to for a particular need.
Searchable networks (those for which senders can locate their targets with relatively short paths) tend to have a positive homophily parameter. Homophily, apparently, is the tendency of individuals to associate with like individuals, as opposed to dissimilar people. This also makes intuitive sense: if you can reliably predict what types of associations a friend has by that friend’s feature-vector, then you can make good guesses as to whether your friend is a good routing choice for a particular target.

And, perhaps most interestingly, when tuned to simulate the conditions of the Travers and Milgram experiments, the model predicts that the average path length between a randomly chosen source and target is 6.7 – right in line with the “six degrees of separation” we all recite. Takeaways: An interesting model both for its assumptions of how social networks can be formally characterized and how localized routing decisions work. Its routing algorithm produces paths of equal average length to observed human behavior.

HTML's Three Competing Data Formats

2009-08-28T00:00:00+00:00

I just read an interesting blog post about the HTML5 DataSet specification which really got me thinking about the state of data on the web. In short, it seems the battle for bringing well-structured data to HTML has been won. Not that there was ever really an opposition, but it is clear now that the future of the web really does hold potential for structured re-use of the data within.

Custom Data Attributes

The custom data attribute section of the HTML5 spec provides a standard way to embed key-valued metadata inside any HTML element. Essentially, any attribute beginning with data- will be treated as a special data item that you can access from the DOM API. For the jQuery crowd, it appears to natively provide the same functionality that jQuery.data does. While this is a very rudimentary form of data storage, it will go a long way toward making the DOM useful to developers. From the standpoint of Model-View-Controller separation, the DOM is currently a view-only part of the puzzle. This enables it to serve as the model as well, possibly resulting in reduced complexity in your code. Look at the example the W3C provides, below:

 <div class="spaceship" data-id="92432"
      data-weapons="laser 2" data-shields="50%"
      data-x="30" data-y="10" data-z="90">
  <button class="fire"
          onclick="spaceships[this.parentNode.dataset.id].fire()">
   Fire
  </button>
 </div>

Here we see a DIV node which, in addition to representing a spaceship visually on the page, now also can contain the data attributes about that spaceship necessary for the web application to keep its state. Can it be scraped? Sure, but this probably isn’t where you want to stash data intended for external re-use. Data attributes don’t support URIs, for one thing, nor do they support anything more complex than simple key-value attributes attached do DOM nodes. For these features you’ll need RDFa or Microformats.

Microdata

Going one step further, HTML5 provides a definition for something it terms Microdata. Microdata, like data attributes, are a way to define key-value paris within your HTML text, but they have a few important differences:

Items can be named:
Properties can be added outside of the node that represents the item:

Name: <span itemprop="name"&gtAmanda</span>
Property subjects can be specified explicitly to avoid ambiguous parsing:

Name: <span itemfor="amanda" itemprop="name"&gtAmanda</span>
Items can have other items as values, not just literal types:

Band: <span itemfor="amanda" itemprop="band" item id="someBand"&gtThe Scripting Javas</span>

Nasty Bug in Rack 1.0

2009-08-12T00:00:00+00:00

I spent most of the afternoon tracking down a nasty bug that ended up being in the Ruby Rack middlewear and thought I’d post a fix here in case there are any Rails developers out there who read my blog. In short, a call to .params on a Rack::Request object will fail with an uncaught exception whenever @env["rack.request.form_input"].eql and @env["rack.input"] are both nil. This causes request.POST to return nil (line 137 of request.rb), which then causes line 160 of request.rb to throw an exception:

 160 self.put? ? self.GET : self.GET.update(self.POST)

I wouldn’t be surprised if this bug is already fixed in the latest git repository, but I wanted to post the fix here in case you are on a host where you are frozen at Rack 1.0 and can’t update to the lastest. Basically, instead of using request.params to access the union of GET and POST params, use the following method:

 def extract_params(req)
   if req.put?
     return req.GET
   else
     (req.GET || {}).update(req.POST || {})
   end
   rescue EOFError => e
     req.GET
 end

That implements the code a bit more safely and will return an empty hash instead of nil. Don’t hate me for using Ruby’s notorious (possibly_nil_var || backup_var) syntax. Now to finally finish the real work I was trying to get done….I guess the good news is I learned how to use ruby-debug. I was using Rack::Request to extract the useful data out of the env variable that is presented with the rack request. In certain circumstances, the

Palm Pre secretly tracks you by GPS, sends info to Palm

2009-08-12T00:00:00+00:00

It was bound to happen one day with these new GPS and Internet-enabled phones. First we find out AT&T has been illegally spying on US citizens, now we find out that companies are beginning to do it for their own personal gain. We dearly need someone on the hill to craft legislation that prevents companies from this type of cyber-stalking, as it can only lead to the types of societies that people don’t want for their kids.

Using UISearchDisplayController with Three20

2009-06-29T00:00:00+00:00

Joe Hewitt’s Three20 project is a great set of libraries for iPhone development, but one of the inevitable consequences of using it is that Apple eventually releases official versions of some of the same “missing widgets” found in Three20. One such example is the search functionality in tables.

I have been using Three20 for an exciting project I’m working on over the summer, but also wanted to leverage the new search functionality of the iPhone OS 3.0 instead of the Three20 search feature. It took about a day of poking around the code-base of both, but ultimately only a few changes to the code, which I’ll describe here.

Step 1: Implement the search delegates in your TTTableViewController

In your TTTableViewController interface, tack on the UISearchDisplayDelegate and UISearchBarDelegate interfaces to say you’ll be implementing those. They will let the search bar and controller send your controller events to let it know it is taking over.

Step 2: Add the search bar to your table header

In your loadView method, create a new UISearchBar object and add it to your table’s header, like so:

// [SNIP]
UISearchBar *searchBar = [[UISearchBar alloc] initWithFrame:CGRectMake(0, 0, self.tableView.frame.size.width, 0)];
searchBar.delegate = self;
searchBar.showsCancelButton = YES;
[searchBar sizeToFit];
self.tableView.tableHeaderView = searchBar;
[searchBar release];

Step 3: Add a SearchDisplayController to your controller

In the same loadView method, create a new UISearchDisplayControllerand add it to self:

// [SNIP]
UISearchDisplayController *searchDisplayController = [[UISearchDisplayController alloc] initWithSearchBar:searchBar contentsController:self];
[self setSearchDisplayController:searchDisplayController];        
[searchDisplayController setDelegate:self];
[searchDisplayController setSearchContentsController:self];
[searchDisplayController setSearchResultsDataSource:self.dataSource];        
[searchDisplayController release];

Notice above that you are pointing the SearchDisplayController at several different items: the UISearchBar from before, self, and also your TTTableViewDataSource.

Step 4. Implement the required delegates

Next we’ll implement the required delegate methods that will be called when the user interacts with the search bar. Notice below that we’re using these methods to essentially pass information through to the data source.

- (void)filterContentForSearchText:(NSString*)searchText scope:(NSString*)scope {
  [self.dataSource filterContentForSearchText:searchText scope:scope];
} 

- (BOOL)searchDisplayController:(UISearchDisplayController *)controller shouldReloadTableForSearchString:(NSString *)searchString {
  [self filterContentForSearchText:searchString scope:
    [[self.searchDisplayController.searchBar scopeButtonTitles] objectAtIndex:[self.searchDisplayController.searchBar selectedScopeButtonIndex]]];
    // Return YES to cause the search result table view to be reloaded.
    return YES;
}

- (BOOL)searchDisplayController:(UISearchDisplayController *)controller shouldReloadTableForSearchScope:(NSInteger)searchOption {
  [self filterContentForSearchText:[self.searchDisplayController.searchBar text] scope:
  [[self.searchDisplayController.searchBar scopeButtonTitles] objectAtIndex:searchOption]];
  // Return YES to cause the search result table view to be reloaded.
  return YES;
}

- (void)searchDisplayControllerDidBeginSearch:(UISearchDisplayController *)controller {
  DKSearchableDataSource *ds = self.dataSource;
  [controller setSearchResultsDelegate:self.tableView.delegate];
  ds.searchActive = YES;
}

- (void)searchDisplayControllerDidEndSearch:(UISearchDisplayController *)controller {
  DKSearchableDataSource *ds = self.dataSource;
  ds.searchActive = NO;
}

Important! Notice the setSearchResultsDelegate call above. This occurs here instead of the loadView method above because the table delegate hasn’t yet been created when we’re still inside of loadView – if you were to set it up there then selecting search result cells wouldn’t trigger a callback.

Step 5. Modify your data source to be aware of search

Finally, we need to modify our data source to be aware of this new “search mode.” I did this with a pretty simple fix: I created a boolean property named searchActive that is set by the owning controller (see the above code). Then, in all the important methods, I put in a simple if..else statement that returned one value for search mode results and another value for normal usage. Here’s an example:

- (NSInteger)tableView:(UITableView *)tableView numberOfRowsInSection:(NSInteger)section {
  if (self.searchActive &amp;&amp; (self.searchText != nil)) {
    return self.filteredItems.count;
  }  else {
    return self.items.count;
  }
}

In the above conditional, I check for searchText to see if the user has begun searching yet. If they haven’t I want to display all results.

Finally, I need to implement the actual search method, which is simply passed through from the controller class.

- (void)filterContentForSearchText:(NSString*)searchText scope:(NSString*)scope {
  [self.filteredItems removeAllObjects]; // First clear the filtered array.
  self.searchText = searchText;
        
  /*
   * Search the main list for products whose type matches the scope (if
   * selected) and whose name matches searchText; add items that match to the
   * filtered array.
   */
   for (id *item in _items) {
     NSComparisonResult result = [[item searchText] compare:searchText options:(NSCaseInsensitiveSearch|NSDiacriticInsensitiveSearch) range:NSMakeRange(0, [searchText length])];
     if (result == NSOrderedSame) {
       [self.filteredItems addObject:item];
     }
   }
}

Conclusion

I left a lot of explanation out here, but I hope this is enough to get you started on using Apple’s “official” search functionality with the Three20 library. Leave a comment if you have any questions or need a bit more explanation of the code and I’ll be glad to amend the post.

What is Machine Learning

2009-06-25T00:00:00+00:00

I’m interested in Machine Learning, but still quite a newbie in the field, so I thought I would start posting articles about it to force myself to read more and begin synthesizing the information in my mind. Since this is the first, I’ll start with the basics: what is Machine Learning, anyway?

Machine Learning has its origins in the Artificial Intelligence community but it is actually a different branch of Computer Science than what one often thinks of as AI. When people say they are doing “Artificial Intelligence,” it often refers to the quest to make computers more humanlike – able to think, reason, emote, etc. Machine Learning, on the other hand, uses a slightly different set of tools to solve basic engineering problems that humans alone can not solve. Machine Learning is specifically focused on problems that are easier for a computer to learn how to solve than for a human to figure out how to solve.

That characterization might sound a bit strange, so here is an example: You are given 10,000 patient records involving heart disease patients over the past five years and asked to find out if there are any drug combinations that appear to be having adverse affects. With that quantity (and complexity) of data, it is near impossible for a human to approach the task of answering the question. Using Machine Learning, the human can instead teach the computer, how to learn how to find the answer, and then let it loose. It is sort of like meta-math: you teach the computer how to solve the problem of solving a problem.

Here is a breakdown of the categories of problems that you can solve via this method:

Supervised Learning involves taking an existing set of data for which you have example inputs and “answers”, and then learn how to guess a new answer from a new input that hasn’t been seen before. This can be divided into two further cases based on whether your answer-space is continuous or discrete:
Regression is performing supervised learning on a continuous data set. You might be trying to predict weight based on a person’s age, for example. In this case, you would train your algorithm on an example set of weights and ages, and then you would predict future weights using just an age by itself.
Classification is performing supervised learning on a discrete data set. You might be trying to predict a person’s favorite ice cream flavor based upon a number of other attributes about them. Your training set consists of multiple people with those predictive attributes, along with their choice of ice cream flavor.
Unsupervised Leaning is the act of solving a problem for which there is no existing set of solutions to provide the computer as examples. In unsupervised learning, the question you are asking the computer is, “is there any interesting structure in this data?” The typical example of this type of learning is clustering, the act of grouping items into different buckets, without knowing in advance what those buckets are. You might be trying to explain the habitat of Canada Geese, for example, and have a large data set that consists of their location at randomly selected time instants. Using unsupervised learning, the computer might cluster these locations into two general categories: Canada and Florida. While this example could easily be performed by the human eye, remember that computers can deal with multiple, sometimes infinite-dimensional data that no human could ever process.
Reinforcement Learning can be thought of as learning that takes place gradually, with feedback after each step. Whereas supervised and unsupervised learning uses a data set presented in its entirety at the beginning of the problem, with reinforcement learning the computer is presented instead with a goal (described mathematically, of course). The computer then takes actions (also described mathematically) which generate either positive or negative feedback. This feedback is incorporated into the computer’s strategy which evolves over time as it learns to meet its goal. An example of feedback learning is the autopilot on an airplane. The goal is to keep the airplane flying to its destination in a steady manner. The actions the computer can take consist of altering air speed, pitch, yaw, and rudder controls, and the feedback the computer receives is a combination of how the plane is actually performing versus how the autopilot settings are asking it to perform.

So that is my quick and dirty run-down of Machine Learning. I’m no expert, so if there are problems with this post please leave a comment and I’ll correct them. Either way, I hope this is the start of a series of useful articles explaining some of the basic principles and techniques of the field.

Marvin Minsky on the AI Winter

2009-06-11T00:00:00+00:00

I went to the morning half of the Living Heritage AI Workshop held at CSAIL today and it turned out to be a cool get-together of many of the founders of artificial intelligence from the MIT AI Lab heydays. Marvin Minsky spoke for a while, and a small, but interesting, part of his talk was a rundown of why he thought AI research essentially came to a halt in the 80s: because focus changed from solving small, focused problems to trying to come up with a single approach to solve all problems. Here is the rundown he gave of these one-size-fits-all approaches, and their shortcomings:

Neural Networks - tend to get stuck on local peaks
Rule-based Systems - don't yet use enough reflective layers
Baby Machines - all, so far, have failed to keep growing
Statistical Methods - fail to explain what causes exceptions
Genetic Programs - fail because they lack explicit goals
Situated Action - needs higher-level representations
Formal Logic - can't exploit reasoning by analogy
Fuzzy Logic - cannot support reflective thinking
Simulated Evolution - Fail to learn the causes of failures
Algorithmic Probability - More general, but needs approximations.

He followed with the comment: “Physicists prosper by showing where old theories fail. AI-researchers seldom publish their programs’ faults.”

Another cool tidbit is that he said he never remembered technically admitting anyone to the AI Lab back in the day. They had a lot of funding at the time, and people would just show up from universities overseas, and sometimes they would stay for a week and sometimes they would stick around for good.

Alone in Tokyo

2008-05-15T00:00:00+00:00

</embed>

This made me really miss Japan.

Don't Become a Buddhist

2008-05-12T00:00:00+00:00

After that flight, the Dalai Lama invited our group to his home in Dharmasala, India. Following a cordial greeting, he said something that seemed most unusual, given his position as the leader of a spiritual movement. “Don’t become a Buddhist. The world doesn’t need more Buddhists. Do practice compassion. The world needs more compassion.

John Perkins

As You Dream It

2008-05-11T00:00:00+00:00

“The world is as you dream it,” he told me. “Your people dreamed of huge factories, tall buildings, as many cars as there are raindrops in this river. Now you begin to see that your dream is a nightmare.”

John Perkins

Book to Print!

2008-05-09T00:00:00+00:00

Ah, the fresh smell of paper pulp! An unexpected box in the mail containing twenty copies of The Art of Rails can mean only one thing — the book has gone to print! For the sake of getting things done, only two observations:

I really need to finish up this W. Web story, but I’ve been dragging my feet ever since I ended up starting a war!
All of the Wrox authors really need to get together and make a pact to send in menacing pictures for their book covers. That way the book store can place the Wrox books across the isles from each other and it will look like different topics in computer science are staring each other down.

Declaring Something Doesn't Make it True

2008-02-09T00:00:00+00:00

Although New York Sen. Hillary Clinton won the largest states…her presidential campaign spent the rest of the week repositioning her as a kind of underdog…This week, Clinton and her advisers declared that Obama is the “establishment” candidate…It’s hard to exaggerate the distance between Clinton’s previous image and her new underdog image.

Peter Overby, NPR News, 8 Feb 2008

What really puts a bad taste in my mouth is the way in which the Clinton campaign doesn’t see reality as the summation of past actions, but rather as a marketing campaign that can be whatever they pay money to brand. “Oh, now the voters want something new now? That’s OK! Yesterday we were the voice of experience, but today we’ll be the voice of the outsider!” What was it that George Orwell write about? Memory holes?

Introduction to The Art of Rails

2008-02-08T00:00:00+00:00

There is a certain state of mind, a certain transient condition that arises, where everything seems to resonate and effort becomes effortless. Athletes call it being in the zone, some others call it flow. Flow has nothing to do with triumph or accomplishment; it isn’t the product of your labors. Flow is the merging of a watchmaker and his watch or an artist and her paints.

The dot com bust was a confusing time for web development, but rising from the burst dreams of instant wealth, something strange and exciting happened. The web development community as a whole reached a kind of flow. In a world filled with duct-tape solutions and proprietary formats, suddenly web developers were clamoring for standards compliance, for elegance and simplicity. And it wasn’t just to fend off browser compatibility issues, but because the code looked beautiful.

Through the fits and starts, the competing ideas, and the explosion of development frameworks that followed, an identity began to emerge. This identity is as much a philosophical statement about what the web could be as it is a technical one about how to accomplish those goals. This identity is still emerging, and there are still many problems to be solved, but one thing is now certain: web application development has come of age as a rich discipline of programming that stands up on its own.

Ruby on Rails is just one part of this much larger story taking place, but in many ways it is the symbol of this coming of age. Rails challenged the web development community to rethink what it meant to build a web application. It provided an entire application ecosystem when most developers were embedding their code inside HTML files. It made unit testing not only easy, but cool, at a time when debugging web applications was, at best, a black art. It introduced a new generation of web developers to the ideas of meta-programming and domain-specific languages. And most of all, it found the voice of the change that was taking place: that the web provides a natural and elegant architecture on which to write applications if only we can create the right metaphors to harness it.

Ouch

2008-02-07T00:00:00+00:00

One part of the Clinton mystique maintains: Deep down journalists think she’s a political Rasputin who will not be dispatched. Prince Yusupov served him cupcakes laced with cyanide, emptied a revolver, clubbed him, tied him up and threw him in a frozen river. When he floated to the surface they found he’d tried to claw his way from under the ice. That is how reporters see Hillary.

Peggy Noonan, 8 Feb 2008, Wall Street Journal

Book Edits are Killing Me

2008-02-03T00:00:00+00:00

[eom]

Programming Book Sales and Contracts

2008-01-21T00:00:00+00:00

John Resig of the jQuery world has a really interesting post on his blog about programming book sales and profits from the standpoint of a technical author. The comment stream is worth a read too, with inputs from both writers and publishers.

The Internet is a Strange Place

2008-01-19T00:00:00+00:00

Neil Stephenson needs to step in and write a novel about internet gangs. I just saw this headline:

Scientlogy hacked by /i/, documents leaked

With the comment thread at the end of the article

nvrmind: wtf is /i/?

wafflez: in this case, the invasion board on 711chan.

altheahouse: The chan’s first wave. There is a war going on between Scientology and Anon. The smart money is on the later. Anonymous does not forgive. Anonymous does not forget. Anonymous does not receive. Anonymous delivers.

Running rSpec from TextMate

2008-01-13T00:00:00+00:00

Just a quick tip for installing the rSpec bundle for TextMate: it is important that the bundle that you install is synced with the version of rSpec that you are actually running on your machine / in your project. Otherwise you will get a bunch of Ruby errors.

Close TextMate, open up a shell, and go to the bundles directory for your installation, which is probably something like this:

/Users/USER/Library/Application Support/TextMate/Pristine Copy/Bundles/

And then run the following code to pull down the latest (you may want to use a particular tag) from their repo:

svn export http://rspec.rubyforge.org/svn/trunk/RSpec.tmbundle/

Then you can enjoy nice testing by pressing Command-R from within one of your spec files: (IMAGE REMOVED - My blog was hacked and I lost data)

PeepCode Love

2008-01-01T00:00:00+00:00

I know I am just the latest of many to praise PeepCode, but Geoffrey Grosenbach has really produced something fantastic. In addition to being a great resource for tutorials on specific topics in the Rails world, it is a very effective (and entertaining) way to learn.

He’s from Seattle, so I can’t help wonder if the PeepCode logo: (IMAGE REMOVED – My blog was hacked and I lost a lot of data) Is in any way an homage to Seattle’s famous Lusty Lady (a landmark near Pike’s place, oddly juxtaposed next to upscale apartment buildings and boutique shops): (IMAGE REMOVED – My blog was hacked and I lost a lot of data) In any event, I just bought my first 5-pack of “shows.”

Code in Text

2007-12-16T00:00:00+00:00

I took about a month hiatus from writing (much to the frustrated grumbles of the fine folks at Wiley), but I’ve gotten back into the groove and am preparing to turn in Chapter 8, “Playing with Blocks.” One thing that is markedly different about this chapter than the others: it’s chock-full of code. Up until this point, the book has been about design patterns, tricks, and techniques. Code is certainly used all over the place, but it is only used to demonstrate the examples. This chapter is also about design, but its about design centering around one of Ruby’s unique language features among the OO family of languages – Procs and blocks. While it has been fun writing text-heavy material thus far, it is really nice to read over this chapter and see lines of code weaving in and out of the text all over every page. It’s good to get down and dirty in the code every once in a while :) (I’ve certainly never examined Ruby’s C source code until this chapter..)

Chapter 8

2007-12-07T00:00:00+00:00

The plot thickens! An innocuous chapter on designing with block-based methods brings terrible news of a coming conflict. Will W. Web survive? (Thanks to Rob B. for the idea)

W. Web sat, listening intently as his companions bantered back and forth as if according to a script. They had been sitting at the table for some forty minutes, most of them spent discussing the day’s protest and how it had gone. This group, it turned out, was exactly who Web was sent to find – prominent representatives from the clans allied with his, sent to show support and presence at the URL protest downtown.

Wall-mounted oil lamps flickered oranges and yellows off the shadows in the basement tavern as discussions of the protest moved to of motivation, as such conversations tend to do. The occasional shouts and nasal laughter of a group of lingering URLs, their scrawny fry-like bodies in baggy homemade t-shirts, could be heard amidst the clinking of glasses from the bar across the hall.

“Flexibility,” Schema said, slapping his hand definitively down on the table and looking at Web, “So what we need is some flexibility. You have no idea what it’s like having to do the same thing all day.”

“But what else is there to do?” Rusty responded, “You create data, you update it. If you’re lucky, someone might come and read it too. And then one day, you send it to the big bit bucket in the sky.”

“That’s exactly the myopic response that gets us into this type of problem. Look at us! You, six feet tall. Me, with a mustache! People grow, things change. We don’t always know what we want to do with a thing at the time we build it.”

“But there’s a medium to be kept. The best we hope for is some grounding stability amidst the process of change. Certain structures to scope and limit our operations.”

“Look,” Matz jumped in, “I think we can all agree for the needs of both flexibility and structure, but I’m not so sure we need to characterize the solution as a tension between the two. Could we not attempt to live each through the eyes of the other?”

“Agreed,” Jen offered, “what is most important is that we take a holistic approach that incorporates all of our strengths. We can’t afford to factionize ourselves in the face of…” she trailed off, looking uncomfortable.

Faces grew somber and nods accompanied downward stares around the table. A URL laughed loudly in the background.

“In the face of what?” Web replied.

His party looked nervously among themselves, communicating with their eyes in a silent, reluctant argument to decide who would be the one to explain. Rusty leaned in towards the table, the rest of the group following.

“The compilers,” he said in a near whisper, almost mouthing the word. “They have spies, so we can’t discuss it in the open. This rally was a cover for the resistance movement. The secret negotiations have fallen through and the time has run out.”

Web wanted to disregard the statement as silly, but the solemn expressions on the faces looking at him made shivers run down his spine and out his limbs. The air felt suddenly electrified in a way that could not be denied.

“What do you mean, run out? What resistance movement?” Web said, his voice now low and intense.

“There’s a war coming, Web! And it’s too late to stop! They’re gathering by the day. We’ve been tracking them for a year now. But the final preparations have only recently begun.”

“What was the protest for then?”

“To safely gather the leaders of the resistance movement under the guise of an internal struggle,” Jen replied, “until last week, we had only met through messages, passed through unreliable transport.”

“And me? Why was I sent here?”

“Because this war is about YOU, Web” Rusty said, his voice a strained shout hidden in the whisper, “you need to start paying attention to the events around you! Stumbling around on the street like you do!”

“You were sent here so that you could be protected,” Jen said in a much softer tone. Web sat back in the bench, stunned. “Their signal to attack is the Flying Toasters screensaver.”

It was too much to process, but he could see the connections starting to form. The Flying Toasters screensaver? The screensaver! He had been processing orders for that screensaver for months! The biggest resurgence of a software product in years! And the compilers. Nothing on the web was compiled… The electric feeling of the air embedded itself into Web’s skin as Web looked at the faces of his new colleagues, realizing now, for the first time, why he was here.

EJS Moving to Google Code

2007-12-05T00:00:00+00:00

The folks over at JavaScript MVC have been using EJS in their framework, and have made a lot of great chances to it, so I’m going to move it over to Google Code this week so that others can more easily track development of it and contribute changes.

Code can be Funny

2007-12-03T00:00:00+00:00

Here’s a tiny excerpt from the current chapter I’m writing: “Playing with Blocks”. So far I’ve only posted the narratives that accompany each chapter. This piece of code made me chuckle, though, so I thought I’d share it. There’s a lot of great discussion in the chapter about Methods, Procs, and blocks, and how to apply the differences to your coding style. So if this piques your interest, then look for the book next Spring:

While the adverb metaphor doesn’t encapsulate nearly all of the uses of blocks, it is a particularly good one to demonstrate how becoming familiar with block-based programming will change the way you think about code. Languages with blocks let you do things sneakily:

def sneakily
   # There actually isn't such thing
   # as System.logger...so don't go
   # planning any bank hoists with this
   # example!
   System.logger.turn_off
   yield
   System.logger.turn_on
end

# Don't tell!
sneakily do
  1000.times {
     votes << Vote.new("Ted")
  }
end

Or, if you don’t mind the consequences, even incorrectly

def incorrectly
   # This one actually works, but can
   # really mess up the interpreter depending
   # on what you put inside the block!

   # Temporarily randomize the result of addition
   class Fixnum
      def genrand(other) ; rand(other) ; end
      self.class_eval {
         alias :oldplus :+ ; alias :+ :genrand
      }
   end
   yield
   class Fixnum
      self.class_eval { alias :+ :oldplus }
   end
end

incorrectly do
   votes.count
end

Let’s call the general idea code wrapping: creating code that is meant to wrap around other code. Code wrapping can be applied many ways. Block-based iteration is just code wrapping. A block-based iterator implements the iterating loop with an empty body waiting to be filled by the block provided by the caller.

Many elements of aspect-oriented programming are just code-wrapping, and later you will see how aspects such as logging and performance can be implemented in this way. Finally code-wrapping provides an excellent way to construct hierarchical documents such as HTML and XML by using block-based programming make Ruby look like a domain-specific language.

That’s just the draft, of course. In the final copy, I’ll make sure to vote 2000 times.

What's new is what's old

2007-11-29T00:00:00+00:00

I’ve been busier lately than just about any time I can remember and haven’t had much time to write here, let alone get much else done. I’m usually against making comparisons between computers and human brain, but when you’re overloaded with stuff to do, it sure feels a lot like a computer that’s thrashing – context switching so often that all your resources are dedicated to overhead and none to the actual tasks that you need to get done.

I just wanted to stoke the fire of a previous post predicting that the iPhone SDK would actually be WebKit. (as in, the new SDK is the old SDK, we just shot the rendering engine with steroids). Here are some of the feature highlights from WebKit 3.0:

Enhanced Rich Text Editing
Faster JavaScript and DOM (~ 2x)
Faster Page Loading
SVG support
XPath support
Improved JavaScript XML technology (XSLT, DOMParser, XMLSerializer, and enhanced XMLHttpRequest support)
Styleable form controls
Additional advanced CSS support: 2.1, 3.0, and experimental.
Reduced memory use (~14%)
Web Developer Tools included

A new chapter, and thus a new installment of W. Web, to come this weekend, I promise, and eventually I’ll continue posting the Fuji hike stories.

iPhone Native Apps Prediction

2007-10-24T00:00:00+00:00

Post-hoc update: hah! boy was I wrong!

Jobs finally let the inevitable plan leak that Apple was going to support native apps from 3rd party developers on the iPhone. (This is the whole reason I bought one, so good news for me). He mentioned the understandable difficulty of releasing such a framework because of security reasons, which I think makes complete sense: the prospect of mobile viruses is potentially disastrous. Here’s the prediction then: “Native” apps are really going to be HTML-based Widgets, armed with new WebKit features & a rich JavaScript API into the phone’s features. Several things support this theory:

The only way to create a truly secure native-app framework based on binary files would be Apple inspecting and signing each binary to give their approval. This creates a bottleneck that would stifle the type of experimentation and hacking I think Apple knows it needs. Therefore a non-binary format used as the native app language is preferable. Java byte code is already out, as evidenced by Apple’s failure to include Java on the iPhone. MS CLI is out for obvious reasons. That pretty much leaves Flash and Web-based solutions.
We know the iPhone already supports widgets, even though we aren’t allowed to create our own. A good way to guess what bones we’re going to get thrown is to look at which one’s Apple is already chewing on.
Apple is embracing the web in a very big way. The flourishing of the web has done a lot of good for Apple – when your company has been plagued for years with detractors saying “well so-and-so-program won’t run on a Mac,” the platform independence of web applications is a welcome arrival. It makes sense for them to continue embracing web technologies on other devices, such as the iPhone.
WebKit just added support for local storage. One of the first problems with third-party apps is how to let them store information without being able to corrupt the iPhone. Some sort of managed storage solution has to be added to provide persistence without true filesystem access, and this is exactly what the WebKit team just added.

Democracy: New words needed; Apply within

2007-10-21T00:00:00+00:00

Today I had a disturbing thought: what if we no longer have a single word sufficient to describe our ideals of “for the people, by the people” government. Societies develop new words as certain concepts become relevant enough to require concise differentiation from others. That is why we have one-hundred different ways to describe our Waffle House hash browns, while (if my fourth grade teacher is to be believed) the Inuit have equally many words to describe snowfall.

Maybe I’m just too young to realize, but it seems that the word “democracy” used to be a pretty powerful one: it encompassed not just the voting process but the very ideals by which a society held themselves accountable. Today, the word too often seems a hollowed shell, used to describe the mechanism of electoral voting and nothing more.

The problem is, having a ballot box doesn’t guarantee representation, self determinism, or personal liberty, unless a lot of other factors come into play along with it. It is the casual failure to examine the death of these other factors that scares me. To get an idea about what I mean, here’s a relevant clip from a documentary about Fox News. For the politically minded, please remember that the finger I’m pointing is at everyone in the abstract – this Fox News example is simply one I found in a short period of time. I’m not talking about Right or Left here, I’m talking about self-determinism.

</embed>

The problem is, the sheer mechanics of voting isn’t enough. The votes have to mean something. That means that the public has to be fairly informed. The voter registration process has to be easy and fair. The populace has to be free from fear of rebuke. The elections free from fixing. And the nation not a slave to debt.

For each one of those qualities, any number of nations could be found that are “democratic” in mechanism, but trapped under oppression of some means in reality. The American public, with consent manufactured via Hollywood style news that tells us what to think instead of showing us the world; the nations of South America and South-East Asia, trapped in World Bank debts crafted so deftly around them; Eastern Europe, where you know who you’re supposed to vote for, but who’s counting anyway?

A parting, that I’m pretty sure is fabricated, but nonetheless poingant:

I have the greatest admiration for your propaganda. Propaganda in the West is carried out by experts who have had the best training in the world – in the field of advertising – and have mastered the techniques with exceptional proficiency…Yours are subtle and persuasive; ours are crude and obvious…I think that the fundamental difference between our worlds, with respect to propaganda, is quite simple. You tend to believe yours…and we tend to disbelieve ours.

Chapter 7

2007-10-15T00:00:00+00:00

Woah, it’s been a while. I’m skipping Chapter 6, which is going to be on RESTful design (and possibly ActiveResource if I can pull it off without being too reference book-y). So I’m waiting until the official Rails 2.0 release and moving on ahead without it.

I guess W. Web will have to travel back in time to make up the gap in the story. One thing I’m going to have to determine soon: just where is this Web character going to end up? It’s apparent from Chapter 7: a conversation is about to take place, a plot to conspire. More to come…

For now, Web has teleported past untold events of Chapter 6 right into the basement of a smoky pub, waiting for Rusty.

Web sat waiting in a booth at the back of the smoky pub, staring at the aged carving in the wood above his table.

LOKE ERST YE THENK, AND BE YE THE WOLRDE

And the phrase beneath it, in the same ornate style but newer, he thought, given the brighter color of the wood bezel.

FRIDAY NIGHT FISH AND CHIPS SPECIAL

The pub was a strange maze of tables and shadows, wood and brick columns holding up the floors above. It was underneath a building holding commercial space, an old wine cellar that used to belong to a distributer on the seventh floor. Before that it was a factory floor for a parachute company, and even before that it was rumored to have been an underground alchemy supply shop.

Now it was a pub. A damp, earthen smell was permeated under the sounds and odors of fried fish and pipe smoke.

“Hi! My name is AJ and I’ll be your server today! Is there anything I can get you to drink?”

Web thought the waitress looked out of place in this old tavern. She beamed at him waiting his reply with her spiked up neon blue hair, a shining pink plastic halter top and costume-like makeup. She was balancing a tray of empty pint glasses on one hand and a skateboard beneath her feet.

“Actually liquids tend to give me a short. Still some old ISDN lines in here,” Web said, patting his stomach, “Could I just have some chips? You know what, throw in the fish, too.”

“Sure thing. No liquids, fish, chips, coming right up,” she said as she rolled away on her skateboard.

From afar web could see the dim outline of the matre d’ pointing a group of people toward his table. He recognized Rusty from before, along with two others he didn’t know. He stood up to shake their hands as they approached the table.

“Web, it’s good to see you again! Let me introduce you to my friend Gr–”

“Napkin!” AJ cut through the group of people on her skateboard and left Web with a napkin in his outstretched hand.

”..Schema, and Matz,” Rusty continued unfazed. Web gave each a hearty greeting and the four sat down in the booth. Before the newcomers even had time to pick up their menus, AJ was back, tossing a plate down on the table.

“Plate!,” AJ’s voice trailed past as she skated without slowing, the empty plate left rattling in a circular motion on the table.

Matz looked at the spinning plate and inspected the cover of the menu closely,

“How strange. I’ve heard of these new places. What did you order?”

“I got the f–”

“Fish, no liquids!” AJ was flying by on her skateboard in the opposite direction this time. She managed a perfect shot from four feet away, the fish landed square on the far side of the plate and didn’t even slide a centimeter.

“Fish. And chips,” Web replied, starting to worry about how his knife and fork would be delivered to him.

Chapter 5

2007-09-18T00:00:00+00:00

W. Web finally makes it to the Big City on his first mission from the compound. In other news, I’m beat.

W. Web stepped off the bus and forced a nervous swallow as he entered the crowd of noise and signs. He had arrived as close to the heart of Application Square as the bus could get, which was about two blocks away. As far as he could see, up to the big plaza in the distance, were URIs and other regulars of their crowd, holding signs and milling about. It was not unlike a football game, Web thought, remembering videos of tailgating parties that had been uploaded to him before.

He waded through the URIs trying to get closer to the square. The URIs were tall, French Fry-like creatures with short, stubby legs and tyrannosaurus-like arms. For some reason, they all talked like they were from the Bronx, even the Unicode ones. As groups of them chanted their slogans, their nasal voices occasionally cracked at the exciting parts.

No wonder these guys get pushed around, Web thought.

The real geeks were the ones that hadn’t even bothered to register domain names for themselves. IP addresses blazoned up their chests, they were a bit pastier than the others and were wearing thick glasses.

“Hey, watch it! Geez, buddy, can’t we even get some respect at our own protest?”

Web had been searching for the stage so hard that he had inadvertently walked straight into one of them.

“Oh! I’m sorry…uh, 208.97.177.118,” Web said, tilting his head sideways to read the IP address written up its chest, “I was just looking for the main stage. Do know if there is one?”

“What am I, DNS? That resource is 301 Moved Permanently! Aheiahahiaha!” the URI snorted in reply.

Web stood there unsure of what to say.

“Hey, loosen up, buddy,” the URI punched him in the shoulder, “it’s a beautiful day. The stage is over that way,” he gestured with his fry-like head.

“Thanks, I appreciate it. Say, what exactly are you protesting?”

“You mean nobody gave you a pamphlet?,” the creature exclaimed, “those dynamic URIs, you can never count on them to be there when you need ‘em. We’re protesting the ‘stablishment, buddy boy, cause we get no respect and its time we deserve some.”

“What do you mean,” Web said, “everyone uses you guys. You’re great – you should see how large my browser’s bookmark folder is! In fact just the other day I saw one of you on a billboa–”

“Use us and throw away, use us and throw away. That’s the way it always happens. Throw-away labels, that’s what we are. Nobody stops to think that we’ve got depth to us. Do you think we like being three-hundred characters long? ‘Oh, doesn’t matter,’ you say ‘nobody ever stopped to appreciate the beauty of just a mere URI!’ Just a mere URI?!”

It didn’t give Web a chance to respond.

“And social services - we deserve them too. Do you know how much funding unit tests got last year? Over 1.2 billion. Billion! Ya know how much we got? Didn’t think so – well I’ll tell ya - a big zero. Zilch. Nothin. How’s it come about that we’re the face of the internet and nobody’s ever stopped to think that we need attention too?”

Web could see that this was the beginning of a long rant this URI had given many times before and knew he had to get to the stage before things wound down.

“I couldn’t agree more. We need more funding devoted to URIs,” he said hurriedly, “Look, I’m sorry to run, but I’ve really got to get to that stage. Thanks for your help!”

And before the URI could say anything more Web had pushed back into the crowd, weaving his way forward toward the edge of the square.

This Binary Universe

2007-09-05T00:00:00+00:00

Most amazing album I’ve heard in a long time.

</embed>

In high school my friend Susannah introduced me to the world of electronic music with BT’s early work (Ima, ESCM) and he has remained one of my favorite artists ever since. After a many-year lull in his true electronica while he pursued more pop-style music and remixes, this new album reminds me what made his early music so different from anything else in the electronic section of the music store.

iWork and SVN don't play well together

2007-08-28T00:00:00+00:00

My publisher requires that I write in MS Word, but it’s so clunky on a Mac (and iWork so smooth) that I write each chapter in a mixture of text files and iWork documents and convert it to MS Word at the last minute. But there’s one big problem with the way iWork saves files.

I have a subversion repository for all my writing projects. Every time I step away from the computer after having made a change to one I check everything in so next time I have an idea I can pick up where I left off no matter where I am. I started noticing a funny pattern with iWork documents that has since caused me to put svn:ignore status on all of them: iWork keeps overwriting my .svn directory.

The problem is that iWork apps save their files as directories. In Finder, they look like ordinary files, but try opening up the console to see their true nature for yourself:

$ ls -l
total 688
-rw-r--r-- 1 ted ted 349696 Aug 27 06:52 189481_ch04.doc
drwxr-xr-x 8 ted ted 272 Aug 27 06:36 189481_ch04.pages

When you do an svn add on the pages files, it does what SVN normally does when it encounters a directory: it traverses recursively through it, adding .svn directories along the way, and adds every file within it.

For the initial checkin, everything works fine, but once you open up iWork and edit your file the problems start.

$ svn update
svn: Working copy '.' locked
svn: run 'svn cleanup' to remove locks
(type 'svn help cleanup' for details)

Uh-oh..

$ svn status
L .
M 189481_ch04.doc 
~ 189481_ch04.pages

Apparently iWork doesn’t like other programs adding content to their files.. er.. directories.. er.. file-directories. The .svn folder is overwritten, completely messing the whole version controll tree (if you’re the kind of person that likes to commit and update in batch). It seems that Apple is moving toward the “managed directory” style of information management. Instead of one large file to store a particular document or presentation, you get a whole directly that masquerades as a file (I think NeXT did this a lot..).

That is a cool choice, I suppose – as a developer I would certainly rather manage a directory of files than one ginormous monolithic on. But at some point Apple is going to have to address the fact that many command line utilities will either get confused at this style of file storage or, like SVN and CVS, will try to add their own data into the mix. Pages is a wonderful word processor, and as of iWork 08 I don’t really see a need to ever fire up Office X unless I have to (Office for Windows is great…the OS X version is just a piece..). But this is a huge problem for anyone who uses version control regularly, and I hope either a fix or a work around is devised soon.

Chapter 4

2007-08-27T00:00:00+00:00

Enough of the heavy stuff, let’s get on with the story. Over the hump of Chapters 1-4. Late but picking up stride.

The bus continued down the road and eventually onto a bumpy highway. Between the hum of the wheels in the back of the bus and the grain of the road shaking his seat, Web began fighting to keep his eyes open, and before he knew it..

“Excuse me,” said a frail, old voice, its owner tapping him softly on the shoulder.

Web opened his eyes, momentarily confused about where he was. He looked with the distant gaze of interrupted sleep at the elderly woman who was sitting next to him.

“I’m sorry to wake you. Your head was getting heavy on my shoulder. And you were drooling on my shawl.”

Web blinked and felt the conscious world returning to him. He hadn’t even seen the woman get on.

“I’m terribly sorry,” he said suddenly as the ability to speak clicked in, “it’s been a long week. I’ll try my best to use the window.” And then, to try to win some points with the old woman, “I’m the World Wide Web, you know.”

“That’s nice,” she smiled, “I thought you looked like a Jeffrey.”

Web gazed out at the road passing by and thought of his week at the retreat. Everything had been so regimented, yet it all seemed to work together smoothly. The more he learned how to fall into their pattern of going about things, the less he found himself distracted by the day-to-day worries that once kept him busy.

It was still a ways before Web’s stop – a protest down in the Big City. The members at his new camp said they were were pleased at how well he understood their beliefs, and they wanted him to be their representation at the event. URLs were gathering to protest for equal rights, it seemed, right under the marquee at Application Square. He job was to attend the protest and attempt to identify and arrange a meeting with the leaders of each respective group at the protest. Web still wasn’t sure what to do after that, but his new friends had assured him that the rest would take care of itself.

Chapter 3

2007-08-10T00:00:00+00:00

Our W. Web leaves the compound on his first assignment. His bus ride turns out to be a time for contemplation:

W. Web boarded a bus heading back into the city from the Rails compound. It had been a long orientation week, and he wasn’t sure if he was ready to shave his head like the rest of them, but he liked a lot of what he had heard.

He was on his fist trip for the group – a rally being held down in the Big City. It would be interesting for sure. Web squeezed through the crowd to find a seat near the middle, right next to a tall man with a buzz cut Web thought looked like a “Rick.”

Rick was riding the bus home from work. He had the reverse commute, living in the city but working out in the country. It was an unusually crowded day for a Tuesday – people of all destinations packed together, muffled music seeping from earphones, blank stares emanating from tired faces. Rick inched sideways to create more room as strange man who appeared to be wearing a paper mache globe around his body struggled to fit into one of the center-facing seats next to him.

As the bus pulled away from its next stop on the city’s edge, Rick smelled a strong stench get on. A man, dirty and tattered, pushed his way to the middle of the bus.

“Move it!” he shouted gruffly as he pushed a small boy and his mother out of the way, The smell of liquor oozed out his pores.

Rick, having just received his black-belt in Karate, was not about watch a drunken bum push a small child. And to be honest he was a little excited at the chance to put his hard-earned skills to use. He stood from his seat, chest out like a Marine, and moved swiftly down the bus toward the low-life. His every move signaled confrontation.

“Hey!” the voice shot toward the drunk.

But the voice was not Rick’s, it was that of a frail old man observing the situation from the rear of the bus. The man smiled warmly, leaning at an angle to see through the crowd.

“Young man, I have a seat for you right here!” he shouted up to the front of the bus, patting an empty seat next to him.

The man tromped his way unsteadily past Rick, who had been stopped in stride by the old man’s voice, and to the back of the bus. He sat down next to the old man.

“Aren’t you glad spring is finally here?” the elder asked the drunk with excitement to his voice, “I’m on the way home to garden with my wife. You should see how happy she is when she gardens, and the Hydrangea are just about to bloom!”

The drunk wrinkled his face, blinking, and his lips began to quiver. Within seconds a tear had broken free and began to roll slowly down his cheek.

“I had a wife, and she loved to garden too,” he managed to stammer, “I lost my job last year, and then lost my life to the bottle.”

The old man patted his shoulder, a small gesture to acknowledge his grief. And as the two talked and the story unfolded, Rick watched from the middle of the bus, his face softening in pity.

At the next stop, W. Web watched curiously as Rick stepped off the bus. Five minutes and two miles ago, the man’s hair had been raised and his stance ready for a confrontation. But the man getting off the bus was different, with all the look someone contemplating the hard blows life can deliver.

(Story adapted from Steve Hagen)

I will not write about coding. I will not write about coding.

2007-08-06T00:00:00+00:00

Chapter 3 is Late. With a capital L. It’s 4:39 AM on a Monday morning – the time of the morning when studies show fatigue degrades driving performance to that of a drunk driver. So I’m going to bed.

Dear book readers of the future, I pledge to not write this book while drunk (well, functionally drunk, I suppose, as the cup in front of me contains tea).

Sincerely, Ted

There is a another pledge that I’m trying hard to stick to: to talk about design of code rather than the mechanics of coding. Picking a book format for technical books is tough because there are a lot of Right Ways to do it:

The how-to book: Let’s try cooking a pie. First, take the tin out of the freezer and put it on the counter. Next, …
The reference tome: The oven has three settings. Bake, Broil, and Clean, and a temperature dial. Pie crust requires 35 minutes at 350 to reach a golden brown.
The cookbook: Cooking a Cherry Pie. Ingredients: cheeries, crust, pie tin. Directions: Step 1 - …
The patterns book (like an abstract cookbook): Pattern #35: Baking disc-shaped objects filled with fruit in a metal casing…
The Head-First books: (oh god, please make the fish-eye lens stop!)
The cyberpunk novel: Zer0-t0x ran to the alley where he saw one of the last functioning pay phones in the city. Reaching for his backpack, he pulled out a blue box and some … ok this is really more of a novel than a technical book.

It seems when it gets late I start enumerating things in <li> tags. But those are the styles as I see them.

I’m trying to write in a style different than the ones above, that blends technical how-to with the types of discussion you’d want to read on a train. Design-focused like a patterns book but without restricting myself to patterns. I’m taking what I believe are the most important topics in web application design and Ruby on Rails and devoting a chapter to each, tackling it from the perspective of someone interested in developing their ability to design good applications rather than just punch out code.

The tough part is that I’m trying to stay away from both code listings (as in “refer to Listing 2-23 for a 50-line XML file embedded in this book”) and code instruction. The former is easy to do, despite making my page-count deadlines more challenging. The latter is really tough because it means I must assume that the reader understands everything going on in the code examples from a syntax and Rails API point of view.

The assuming parts aren’t hard – all of the examples are short, simple and to the point. Plus people are smart. Most technical books probably err on the side of explaining too much of the syntax anyway. The hard part is keeping myself from delving into long syntactical or API-related discussions – something I’d rather let other books do so that The Art of Rails can focus on design issues.

The Art of Rails is a book for Rails developers about designing web applications with the Ruby on Rails framework and style, not a Ruby on Rails API reference. It is funny to me to find that one of the hardest parts of writing is keeping oneself from explaining too much. You’d think filling page after page would be hard, but when you really sit down and start do to it, you realize that there is actually a lot to say. More than can fit coherently in one chapter, or even one book.

So you have to be very careful to pick and choose the right things, and ruthless at deleting portions you’ve written and like, but ultimately detract from the main focus of the book you’re writing. Anyway. I think the Art of Rails is off to a great start. (despite being a bit behind schedule)

And I’m going to bed.

Chapter 2

2007-07-22T00:00:00+00:00

Web’s saga continues. It is the must-read book of the summer! (cough it will be out next spring)

W. Web approached the picnic tables set up for breakfast cautiously, clutching the orientation binder in his hands. He waved as he saw Jeff, the man he had met preaching on the street outside the hospital. Here in the camp outside of the city all of the cult members he remembered from the recruiting sessions were much more relaxed, their nervous edge gone. Web grabbed a paper plate from the serving line and began carousing down the platters deciding what to eat.

“Excuse me, I’m new here, do you know if there is a serving spoon for the eggs?” he said to the woman in front of him.

She turned, amused, and smiled with a quizzical brow, “serving spoon? No, just imply to the platter that you’d like some.”

“Imply to the.. I’m sorry, what?”

The women held her empty plate out to the platter and looked expectantly at it. W. Web couldn’t tell if he had blinked while she pulled a fast one or if he just hadn’t noticed eggs on her plate all along, but when she pulled the plate back it was full.

“We always felt serving spoons were redundant anyway,” she said with a smile, “after all, what else were you going to do with it?”

Web inched his plate toward the inanimate platter looking a bit confused. He jerked his hands back as an omlete suddenly weighed down the plate.

“There you go! A natural. If you want scrambled, try not to look so confused. And smile for sunny side up.”

Web offered a timid smile back and muttered his thanks, shuffling away to clear his head in a less strange environment. He slid past the filling picnic tables and headed back up toward his cabin, the woman’s voice wafting behind him.

“If you’re headed back to your bunk, make sure you touch the door with your left hand first, otherwise the door won’t unlock!,” and then after a pause, “don’t worry! It will all start making sense!”

Chapter 1

2007-07-20T00:00:00+00:00

As chapters are turned in, I plan on posting a little bit about each one. Since I can’t publish the content of the chapter (ps, you can download the new Harry Potter at this link <a href=http://##$.'@03##CARRIER LOST), I’m instead releasing a short story-esque blurb that makes sense in light of the chapter. So here you are, Chapter 1:

W. Web knew immediately something was wrong.

He had suffered stomach pangs before, but never like this. Stumbling out of the taxi cab and into the hospital, he mopped the sweat from his brow and pushed his way across the sidewalk traffic. Inside everything was a dizzy blur flowing past him. Nurses, patients, a police officer. Several computer technicians hitting a computer monitor and mumbling something about the Intranet going down.

“I know, I know!,” Web thought as he struggled past them for the emergency patient entrance. Luckily for W. Web this particular hospital works on a triage system, and when you explain to the nurse at the front desk that you are the Internet you get bumped to the front of the line. There’s a lot riding on your health.

As the Web lay in his hospital bed, passing other familiar technologies as the nurse pushed him down the hall, he realized that he had made the right decision to stop ignoring the pangs. It was going to be OK.

April Fools: Constitution just a Suggestion

2007-07-20T00:00:00+00:00

I promise to keep politics out of any writing I do on here, but this goes beyond politics to plain and simple factual conflict with our Constitution. The Internets and sites like digg and Reddit have already been going ablaze with this, so perhaps you’ve already seen it, but here it is again: This recent Executive Order by President Bush on 17 July 2007 effectively repeals the 5th Amendment to the constitution.

Here is the 5th Amendment:

No person shall be held to answer for a capital, or otherwise infamous crime, unless on a presentment or indictment of a Grand Jury, except in cases arising in the land or naval forces, or in the Militia, when in actual service in time of War or public danger; nor shall any person be subject for the same offence to be twice put in jeopardy of life or limb; nor shall be compelled in any criminal case to be a witness against himself, nor be deprived of life, liberty, or property, without due process of law; nor shall private property be taken for public use, without just compensation.

And here is a summary of what this new Executive Order states:

Section 1. The government may freeze and seize all assets of any person that has committed or might commit an act which undermines our Iraq efforts.
Section 2. It is prohibited to act counter to this order if you are one of the people targeted by it.
Section 3. This applies to anyone on US-controlled territory (including foreign locations), US Citizen or not.
Section 4. To deal with any such person targeted by this order is prohibited.
Section 5. The government does not have to tell you, or anyone that you have been targeted by this order.
Section 6. Powers to enforce this order may be delegated to any arm of the US Government.

That is my summary of the order. If you want to view the full text of it, follow the link above.

When I have children one day and they are growing up in school, I hope that, like other great democratic nations of the world, we are still able to teach them about the inherent rights and protections from abuse of power our democratic system affords them. But if Congress continues to allow the Executive Branch to toss the Constitution out the window bit by bit – due process, protection against unreasonable seizure, habeas corpus – then I fear they will grow up with the twisted belief that these fundamental rights were somehow Achilles heels in the fight for stability and security, not realizing that the enforcement of these rights are the very definition of stability and security.

Update: Pardon the early-morning politics outburst. They happen.

Broken R Key

2007-07-17T00:00:00+00:00

Writing with a broken R key is a bit difficult.

They say that when you work with a particular thing all day, you become a connoisseur whether you like it or not. Welders develop a fine taste for rebar, academics for chalk. Well I am now a world-renound expert on which words have the letter R in them. Let me tell you why (no Rs in that sentence). Every time I have to produce an R, I must to punch down on the key with great speed and force, turning my left index finger temporarily into a comet smashing into the earth, improbably landing precisely on my R key every time.

The rhythm of my typing ends up sounding like a syncopated jazz song.

click-click-THUNK-click-click-click-click-(pause)-click-THUNK THUNK-(pause) THUNK (pause) THUNK THUNK!

I’ll mail a $4 bill to anyone who can guess what word that was.

I did it to myself. I rearranged the keys on my keyboard too many times (that is another story entirely, but the moral is, don’t fool yourself into thinking a Dvorak layout will make you learn Dvorak).

And now my “” key is boken as a esult.

So when the book comes out and there are no Rs in it, or when you run across words like “though” when you are pretty sue it should have been thRough, you’ll know what’s going on.

In other news, I got an iPhone and it is totally sweet.

The Thing about Chapters One through Three

2007-07-14T00:00:00+00:00

I’m two weeks away from my first book deadline and trying to pace myself to prevent long nights in the days running up to when I fire up my FTP client and upload a bunch of ZIP files. But in these last two weeks, I’m especially reminded that writing is tough. Especially when it is chapters 1 through 3.

The flexibility factor. This time around things are working a bit differently than the previous book. Instead of month-to-month deadlines (roughly a chapter a month), I have quarterly deadlines, which means that every couple months I have to turn in 3-4 chapters. While this is pretty cool because it allows parallel writing of the chapters (which is a lot easier, as you can’t force your mind into always thinking about things contiguously), it makes time management really tough. I can’t tell you how many times I’ve spent time allocated for writing the book writing other stuff, whether for personal reasons or professional. The argument I always use to convince myself: “It’s like practice!” Any writing is good writing, but two weeks away I wish I had a few more pages down and a few less to go. This is closely related to:

The errand factor. I’m working 5 days a week, but only 3 are at my day job. The other two I’m working on the Art of Rails. But oh, how easy it is to run errands on those two days, and errands can add up to a lot of time. I’ve got a lot of respect for people who work from home and pull it off.

The research factor. Research and can be a big time sink, an infinite one if you let it. Writing is necessarily a combination of fact and advice (advice being a euphemism for opinion), and no matter how strongly you know a certain topic or have opinions about it, once you put it down and print it, it is hard to take back. So you better be darn sure you meant what you said. During the initial stages of writing, “working on my book” can mean anything from writing toy web apps to try out a technique to listening to podcasts about development to laying on my back staring at a tree wondering why I thought it would be a good idea to sign a contract obligating me to write 300 pages.

The Chapter 1 factor. I need to get over it. I’m just a random dude writing about a topic I enjoy and care about, and no matter what I put down in Chapter 1 it will just be another random Chapter 1. But I tell you, it is hard to follow that advice if you’re the one writing the words. It’s like it is the first date and you spent an inordinate amount of time getting ready because you know its the first time the girl is going to get a real impression of you. I’ve written a whole book’s worth of Chapter 1s. And that took up a lot of time. I’m still not finished, but I think I’ve learned a valuable lesson: say what you have to say and move on. Because that is what, as a reader, I’d want to be reading.

And a final note to my editor: John, if you’re reading this, don’t worry! :)

Cap City

2007-06-27T00:00:00+00:00

Washington is such a funny town. Everyone works for the government, and if you don’t work for the government, then you work for someone who works for the government. We’re the only city in the US with gangsters in Escalades blasting C-SPAN radio out the window.

A whole language of euphemisms and body language has sprouted to support the fact that the government works through a combination of compartmentalization and embracement of the idea that “needing to know” makes you cool. At least, it makes you cool for the fist few years out of college. You don’t work for the NSA, you “do work up near Fort Meade”. And nobody works for the CIA, you are an “analyst for State”.

The charade might work if it weren’t for the glimmer in the eyes that says, “and next year they totally promised me that I’ll be James Bond.” I must admit to ignorance of what it is really like to be in-the-know, in the need-to-know sense of the word, but, like everyone in the Capital City, I know enough people who do live such lives to be able to laugh with them at the reality behind the recent-college-grad romance with security clearances.

Place: Work. Or at least, a large, human sized metal safe cut off from the outside world that you refer to as “Work”

Music: White noise coming out of pores in the wall

Mood: Wishing there was a window
DATE: 07 JULY 07 STANDARD MESSAGE TRAFFIC THE PRESIDENT WILL BE WEARING A
BLUE TIE AND ARGYLE SOCKS TODAY. HE HAD GRAPE NUTS FOR BREAKFAST. MRS.
BUSH IS NOT WEARING UNDERWEAR AGAIN. PLEASE KEEP THE TEMPERATURE IN
BRIEFING ROOMS HIGH ENOUGH TO PREVENT EMBARRASSING BLOUSE INCIDENT BUT
LOW ENOUGH TO AVOID POSTERIOR SWEAT.

This is hilarious! I must tell my friends! Drat. That would be a federal crime, for which I would be thrown in to prison. I don’t have time anyway, I have to write a summary of important information for the last hour and send it to my superior.

DATE: 07 JULY 07 HOULY INTELLIGENCE SUMMARY MEXICO CITY IS THE CAPITAL OF
MEXICO. IT IS RAINING OUTSIDE.

Phew. Sent just before the deadline.

Information is so compartmentalized that you can pretty much allude to anything and come off as a badass.

A recent dinner conversation that may or may not have taken place (sorry, you don’t need to know):

Friend 1: Apple picked only one carrier in Canada for the iPhone too. I wonder how long until they will be able to open it up to the rest of the market.

Friend 2: Well, heh, I can assure you, it won’t be until at least two years.

Friend 1: Why do you say that.

Friend 2: Let’s just say we have too many nuclear-powered robots navigating the magma under Canada for us to let them have iPhones everywhere.

Friend 1: What?

Friend 2: Please don’t repeat that, I’ve already stepped outside of what I can tell you. How are your lima beans?

All at the table agree Friend 2 must have an awesome job. Of course, if you get too crazy, it won’t work. Claiming that DARPA funded a Gay Bomb to make an opposing army simultaneously homosexual and so horny as to crush its threat via mass orgy, or that the government has a secret facility that trains men to telepathically explode goat hearts – now that would be silly. The government funds real projects, people.

My Office is on Fire

2007-06-11T00:00:00+00:00

“SNARGL-ZZZZZT-WOLA-fragl-wozz-UP-YARISPLASH”-[ smack! ]

I keep my alarm set to the AM dial and in between stations because at high volume it is apparently the the only sound on the planet that induces fear and panic in every species that was carried on Noah’s arc (Scientific American, 24 May 2343 BC). Plus anything other than Exorcist-style AM static just gets incorporated into my dream. Which is why until I got my nice new radio-alarm clock I would dream I was a garbage man every morning (beep beep beep..), only to wake up and realize that not only was I not a garbage man, but I was late for work.

Mondays are AM Radio alarm days because Mondays are 8am meeting days. Under normal circumstances, scheduling a meeting for 8am on a Monday is sick and cruel. The detainees at Guantanamo don’t even have to attend Monday meetings until at least 9:30. But my manager is an ex-Marine, and a great guy, so I figure it is a good excuse to get up early enough to see what this morning thing people keep talking about is like.

This morning, however, my office building was on fire. It is 7:50 and the brow-sweat from walking up seven mile long Rosslyn escalator has just started to flow. I turn a corner and notice a strange number of fire trucks outside my building for 7:50 AM. Usually our building doesn’t burn down until at least 9:30. 7:50 is far too early for this to be a drill, and two woman standing across the street told me that this was the real deal.

And then I had one of those experiences in which you catch yourself in the middle of completely idiotic behavior rationalized by the attempt to make reality match your expectations – I started planning how I could break into the building.

See, I had a mission! Normally I show up at 8:10 and sheepishly slink in to the lab with a meeting already in progress. But it was only 7:50! Gloriously, I could stroll into the lab and take any chair I wanted, my self-centered feeling of victory to culminate rather anticlimactically with the words “Ted, could you dial into the conference call for us?”

Today is Monday! 8am meeting day! I must prevail! I didn’t see any smoke – a good sign. Firemen were blocking the main entrance but nowhere near the garage entrance to the building. Cars were flowing into it as usual. I could walk in through the Garage and figure out where the fabled car-owner elevators are and use those.

Oh, right, fire. I’ll have to take the stairs.

Wait.. Fire??! In the middle of an office-worker filled intersection, a dream sequence:

Fireman: [in full gear running down the stairs, carrying a rescued office-cat] Run! Get out! What are you doing! Me: It's OK, I'm going to be the first person into an 8am meeting! Fireman: But everything is on fire!
Me: .... [ awkward pause ]
: *A stunt man from a movie comes running down the stairs on fire. Followed by three more cats and a giraffe.*

Yeah.. screw the meeting, I’m going to pick up some breakfast.

– Update –

Because several people asked: Yes, there were fire engines parked all over outside this morning and firemen preventing people from entering. But no, it can’t have been a big one. I didn’t even ever see smoke or flames, and only a few floors were evacuated.

Traffic

2007-06-06T00:00:00+00:00

I write like a nomad in fast forward. I can never sit for more than an hour and a half without having to change locations to clear my mind, and that has introduced me to Northern Virginia’s famed attraction: traffic.

My commute to work is normally on the metro. It takes about 35 minutes, 10 of which are on foot and the rest is either on the train or waiting for it. I suppose that is a relatively normal commute for city living, but I’m always moving so it doesn’t seem so bad. Here, at Dunkin Donuts (mmmm, donuts), on my first day of writing full-time, I had time to reflect on the experience of driving during the workday. It took me about as long to get here as it does to work, except it is only two miles away.

Stop. Drive forward 10 feet. Screech to a stop. Bolt forward ten feet again. Stop suddenly. Prepare to drive forward only to have some guy in a Ford Expedition slam his metal fortress at high speed into the impossibly small space that has opened up between me and the car ahead of me. Inch forward into the intersection only to stop again. Wait. 3 inches forward. Wait. The light turns yellow. Start looking around for options to bail out but see none – its too late to squeeze in a right turn.

The light tuns red.

I’m a calm person. I enjoy meditation and yoga. I even lived in a Buddhist monastery in Japan for a week just to see what it was like.

But boy do I understand Samir from Office Space when I’m in traffic like this. “Fck! sht! ass! ass! yak! rutabega! pencil sharpener!” I want to scream at the top of my lungs, punching the steering wheel in a blind rage until my horn gives out.

Maybe I’ll ram my car into an old lady’s sedan, just so she knows that Northern Virginia traffic is all her fault. But instead I sigh and smile apologetically, mouthing the word “Sorry” to the driver on my right, who is honking his horn because I’m stuck in the middle of the intersection. And I turn up C-SPAN radio to hear what the Judiciary Committee has to say about the attorney firings.

3/5 the time is 5/3 the fun

2007-06-04T00:00:00+00:00

So far the work week is like any other, but this is the start of working half-time for the summer that I’ve planned to get the bulk of the Art of Rails done. I haven’t been working on it lately due to a lot of stuff going on. The lack of progress is only brow-wrinkling right now, but given my upcoming deadlines the 2 extra days a week opening up are pretty crucial.

So I’ve got my iced tea chilling in the fridge, have my Bermuda shorts, and am ready to sit on a lawn chair in the paved paradise of Northern Virginia writing to Jimmy Buffet music for a good part of the summer. I have to admit I’m pretty excited.

Abusing Javascript for Multiline Strings

2007-05-05T00:00:00+00:00

You realize how funny the technology world is when you come upon a technology from 2005 and are immediately distrusting and suspicious that it has already been replaced with something new. “Two years ago!? How antiquated! I’m sure by now we’ve tackled the problem with an extra embedded API added on for ordering triple half-calf mocha skim ole 2% orange juice lattes wearing a bathing suit! … from Emacs!”

You understand my suspicions, I’m sure.

So I begin with my distrust, but ultimately feel compelled to express my excitement about the ability to abuse the the E4X extension of JavaScript to handle multi-line strings. For those who don’t know (I sure didn’t), E4X stands for ECMAScript for XML (E4X) and was introduced in 2004 and subsequently updated in 2005. It appears that Mozilla-based browsers support it, but I’m not so sure about IE (grumble).

The gist is this: JavaScript doesn’t support multi-line strings, which is a bummer. Ruby has it. So does Python. Even C# does, with the @” quote. You’d think peer pressure alone would have forced JavaScript to adopt it, not to mention the common need to assemble long chunks of HTML from within JavaScript, but it still ain’t so. E4X provides a sneaky way to accomplish it, however, by embedding the string within HTML tags. E4X is essentially a way to use DOM fragments as native JavaScript objects, so you can have a variable defined like this:

    var address = "" + (<r><![CDATA[
    127 Prime Ln.
    55005
    ]]></r>);

The developer gets all sorts of nifty ways to interact with this data, but the toString function is the sneak-route for abuse. Let’s say we want to construct a multi-line string for some poetry:

    var htmlFragment = "" + (<r><![CDATA[
    l(a
    
    le
    af
    fa
    
    ll
    
    s)
    one
    l
    
    iness
    
    - e.e. cummings
    ;
    ]]></r>);

Or perhaps we want to construct a fragment of HTML from with JavaScript but don’t want to use a builder:

    var htmlFragment = "" + (<r><![CDATA[
    <div id="somethingorother">
      <ol>
        <li>Item 1</li>
        <li>Item 2</li>
      </ol>
    </div>
    ]]></r>);

Why in the world would you put your HTML inside a CDATA block, you say? For a top secret project, I tell you! The variable htmlFragment now contains this multiline string, since the the “”+ at the beginning caused the subsequent E4X DOM fragment to be cast as a string for concatenation. Test it out for yourself by setting the innerHTML of your page body to the variable above.

    $('the_body').innerHTML = htmlFragment;

Pretty clever…

Abstraction is Everything

2007-05-05T00:00:00+00:00

A computer can’t really do much in theory: read a number from disk, write it back to disk, add one to it, see if it equals zero, and then maybe jump to a new location on the disk. If I remember back to the CS Theory classes in college right, then I’m pretty sure that is about it.

Which is why, when someone finally figures out how to coax a computer into synthesizing a believable human voice, all the programmers in the world have been planning to throw a secret party in which we’ll laugh the laughs of movie villians: “Did you see the NYT headline? ‘Intelligent Computer Unveiled’! We’ve tricked them! All we’ve done is add 1 and compare to 0 over and over!”

But to claim the humble premise of a computer makes it a stupid device is a bit misleading, I suppose. It is a bit like atoms. There isn’t much to a proton, neutron, or electron, but the ravioli I had for lunch is a pretty impressive accomplishment. Something simple repeated enough times at a fast enough speed doesn’t just give the illusion of complexity, it is the only possible source of it. So 1s and 0s are the Real Deal.

It all boils down to Abstraction. Programmers deal with the inherent simplicity of a computer by creating metaphors for themselves that can be reused as building blocks to increasingly complicated systems. So out of bits come bytes, from bytes come characters, from characters come strings, and so on. Pretty soon you have buttons on a screen and MP3 music playing in the background. It works wonderfully and is the fundamental practice that enables us to enjoy the technology we do today. When developers look at code and ooh and aah in admiration, it’s either because of one of two things: low-level feats of magic or elegant abstraction.

But abstraction comes with a catch. Users at each layer of abstraction are largely ignorant about the finer-grained layers beneath them. This is absolutely necessary to deal with the complexity of our world – we couldn’t function at higher levels of thoughts if we had to be concerned with every detail underpinning existence. Have you ever tried to make a Twinkie by hand? Me neither. But not understanding the building blocks of an abstraction can be a big risk if the abstraction turns out to be flawed. Most people take it for granted that a Twinkie is a particular abstraction of food, in the delicious category. But what if, due to some design flaw at the molecular level, a Twinkie was actually petroleum-based product? You’d never know! But incorporating this abstraction into your diet would persist this error through your system.

So it is useful to allow ourselves to be ignorant of the finer-grained details beneath an abstraction, but it can also be hazardous to not understand these details when necessary.
This is why learning to be a good programmer is an art of mindset rather than an art of syntax. Learning a programming language is different than learning to program. Learning a programming language introduces you to a specific set of operators and abstractions comprising one perspective of controlling a computer. But really learning to program is the art of abstraction. Realizing when new metaphors are necessary, seeing through the ones that currently exist, and constructing them to be relevant in a generalizable fashion.

Waltham, MA

2007-04-30T00:00:00+00:00

Waltham is a wasteland, at least the part I’ve seen so far. Remember those old-west towns from your high school history book? The ones that cropped up on the side of a trail for no purpose other than to house the passing settlers by and the occasional agent computing conference? Apparently I-95 is the new Rt. 66 and Waltham, MA is such a town.

2:00 PM. I call the Westin from the bus to ask which stop to get off at. “Oh, we’re not the Westin downtown. You’ll have to hail a cab,” they inform me. Whoops – but not a problem.

I hop off the bus at the next stop and nearly straight into the cab behind it. It turns out to be maybe the best driveby taxicab tour of a city I’ve ever had, complete with a description of MIT so concise it deserves to be in their admissions brochure: [in my attempt at transcribing the Boston accent]

“Tat ovah thear is MIT, wheah all te crazy smart fucks invent fucked up shit. Usually I get frequent fliah businessmen who wanna go theah. Sit in the back ahn theah Blackberrys. Those guys, wahkaholics. They sit at Cape Cod in the summah wit te’ fuckin Blackberrys sendin shit on fukin email don’t know how to fuckin relax. Ya kid’s in the watah! Quit ya workin!”

40 Minutes and $65 later. We pull off the outer beltway of Boston into Waltham. “Waltham is one ‘a tose business oasis pahhks where all the companies go from MIT to build shit like patriot missiles, but tey’re real quiet about it cause ta people’d get upset. So what you gahtta do is darive 10 minutes down tis highway, see, cause tat’s wheah tah Wellesley girls ah. Radcliffe girls hate Wellesley girls.”

The hotel looks like a business, covered head to toe in green-tinted glass. After checking in I walk up and down the street to see if there was anything better to do. The place is a ghost town. Nothing here but chain hotels, gas stations, and empty business offices. The sidewalk on the street begins and ends randomly, as if they had excess asphalt in some parts of the road and figured it was better than no sidewalk at all. Hotel, hotel, gas station, hotel, empty office building, rental car place. Car rentals must be a big industry in Waltham.

The Charles river is nice, if inaccessible by foot, just past a half-finished construction project and a giant highway bypass. The bypass is right outside my window, cars wooshing by on the outermost beltway of Boston, but no buses. No buses or public transport comes to this side of town, the girl at the front desk told me with the same tone of voice a doctor would use to inform a family that the operation didn’t go well and, we’re sorry to say, amputation was necessary. Nearest restaurant? 10 minutes by car. So needless to say I’m about to have the first room-service meal of my life. The hotel is beautiful; I’m sure the conference will be interesting. But if anyone ever suggests a vacation to Waltham, MA to you, run in the opposite direction as fast as you can…

By the way, I’m here to present a paper on Fido, a framework for information query and dissemination in distributed multi-agent societies. It’s pretty cool, if you are into that stuff, so check it out.

Melodic Impact

2007-04-24T00:00:00+00:00

This Saturday Grace and I are playing piano in a concert that she and her friend Jenny organized to raise money for Special Love, a Washington, D.C. area organization that helps children with cancer. We’re playing a 4-hand/1-piano duet of Under the Sea, and I’m also playing the Fantasie Impromptu by Frederic Chopin.

Here are the details:

“Melodic Impact” A Spring Concert for Kids with Cancer Saturday, April 28th, 6:00-8:00 pm (reception follows) Mt. Olivet U.M. Church - 1500 N. Glebe Rd, Arlington, VA

After a 5 year stretch in which my piano gathered dust in the corner of my living room, it has been wonderful practicing for the concert and knowing what it feels like to be immersed in music again, although I remain pretty nervous about the performance. Particularly nervous about playing through the performance induced hand sweating that will inevitably occur. For those lucky ones who aren’t afflicted by “hyperhidrosis” (and I gather this is the majority of the population), it like a virtuous cycle without the virtuous part: nerves cause sweat, sweat causes nerves, repeat.

Grace and Jenny have done an amazing job putting everything together and have really shown that people who work hard and believe in a good cause create their own luck. Planning a benefit concert was just an idea floated between the three of us last fall, and now through their hard work we are one week away from the concert and have hundreds of dollars in in-kind donations for an auction, a roster of performers that includes world touring musicians and Kennedy-center veterans, and a corporate sponsor to handle the advertising expenses. It is inspiring to be around people who don’t just keep good ideas in their heads and actually act upon them. So if you happen to be in the area and see this, come out and have a good time.