Visual Website Optimizer’s interview (here) with direct response copywriter Jeremy Reeves is a dollars-and-cents example of how experimentation (in this case A/B testing) can yield better results than intuition.

I’d have thought short-form digital content and pithy calls-to-action would outperform longer variations, prior to reading Jeremy’s interview.

• That’s been my own experience with usability testing, as well as my own browsing/reading behavior online.
• And it also seems to be the overall trend in digital: Twitter and Youtube being two standout examples.

Yet real life is more complex.  One long-form sales letter (all 50 pages of it here!) Jeremy developed for a client outperforms by 50% its short predecessor (a video, a few bullets, summary of the offer, etc) .

Personally, I’m skeptical.  This sales letter evokes too many childhood memories of ads for Veg-O-Matics and Ginsu knives on late-night TV.

But I admire and respect the data-driven savvy of direct marketers like Lester Wunderman who built the market for performance-based advertising and paved the way for Google.

Jakob Nielsen wrote a great piece this week (here) analyzing the miserable ratings reviewers have given the Wall Street Journal’s iPhone app.  Across 68,418 reviews, the WSJ’s app gets an overall 2 star rating.

Nielsen attributes this to a confusing interface design.  ”It’s clear that people are deeply offended by being asked to pay again for mobile access to the newspaper when they’re already paying for a wsj.com subscription.”  Yet, “mobile app access is free to paying website subscribers: they simply have to log in with their existing userid and password.”

He explains that after downloading the WSJ app, the subscribe option is the strongest call to action on this startup screen.

This made me wonder: how is the New York Times’s mobile app doing in Apple’s App Store?

Turns out it’s a tied game at 2 all. (Rated at two stars each, that is.)

The NYT’s app does a much better job forthrightly outlining access options, much in line with Nielsen’s recommendations for the WSJ.  It’s the poor performance of the Times’s app that produces a lot of 1- and 2-star ratings: freezes, slow loading, crashes, etc.

You be the judge.  Here are subscription options — spread across three screen grabs – from an article on the NYT’s app.

The folks at Aardvark.com have described in blog posts and presentations (like this one) how they combined user centered design and nimble engineering to create a hockey stick of engineering progress.

I think their product development process is mighty impressive and according to co-founder Max Ventilla their investors thought the same, giving “money as much for our process as for our team and concept.”

While going on gut instinct would have been “twice as fast,” Aardvark’s user driven process “dramatically reduced the chance that we would make wildly wrong bets and have to double back, abandoning large periods of work.”

Rather than going on gut instinct, they collected new ideas with a net cast widely, and mitigated risk by staging solutions in small chunks developed by small teams.

The product development process is detailed in posts here, here, and here on the Aardvark blog.

For me, 3  aspects of their process  stand out:

Qualitative/direct user research methods rather than quantitative/indirect sources like website analytics or A/B testing.  A few examples:

• Every week a new group of 6-12 users would visit the office for prototype testing (paper, or click-through) or to be interviewed and observed using the product.
• After testing a new feature on the live site with a small group of 10-20 users, each was sent an email containing 3-4 questions about their experience with it.
• Phone interviews, a community forum, feedback sent via email, and “overheard” feedback on Twitter and Facebook.

Horizontal organization.  Each Friday the entire Aardvark team discussed the “Weekly Learnings” collected through research above.

• Prioritized features or “learnings” were then solved by small groups of 3-4 people each — an engineer, a researcher, a designer, etc.
• How learnings were prioritized isn’t clear.  Google, the founders’ alma mater, funds only the 4s and 5s on a five point ranking system.  That’s described here.

A deeply embedded evolutionary approach. Evolution progresses through small changes/adaptations and so did the “chunked” implementation of features at Aardvark.  Further, each “chunk” passing through the process was run through two think/make/check cycles  – the first three steps and the last three.

By the numbers:

• Founded in July 2007.
• Five concepts abandoned during prototyping.  The sixth idea, Aardvark, run as a Wizard of Oz experiment for 9 months during 2008 until publicly released as a minimum viable product.
• Most compelling stat from October 2009 data: 87.7% of questions submitted were answered, and nearly 60% of them were answered within 10 minutes.
• In February 2010 Google paid $50M to acquire the company.

Janice Fraser describes how TaskRabbit earned an extra $26K in revenue by launching a redesigned homepage 30 days before the design was “pixel perfect” and completely integrated throughout the site.  A/B testing the redesign on the homepage only, they found a 9.75% “task posted” conversion, where the existing design converted at 5%.

Watch live video from Startup Lessons Learned on Justin.tv

Judging by the concentration of backlinks on TaskRabbit’s homepage, this was a smart way to rollout the new design.

For a site with traffic distributed across more landing pages or a “longer funnel” an accelerated rollout would have been trickier.