al3x

The Tapir Book

2009-07-07T00:00:00-07:00

The Tapir Book

When my coauthor and I began working on Programming Scala last year, the most frequent comment I got was, “you’re never going to want to write another book once you’re done.” Well, as of this morning, I’m pretty much done, and I can say that I’d very much like to write another book. I don’t think, though, that I would commit to another one while working a full-time job, and I certainly wouldn’t recommend trying to write a book while working at a rapidly growing startup.

When I announced that I was going to be working on the book, a critic insinuated that it would detract from my work at Twitter. In practice, the opposite was true. I had many an evening or weekend afternoon of writing interrupted by fires at work that needed fighting. For that reason, I wouldn’t consider taking on another book project until I can give it my full attention. The value of sabbaticals as practiced in the academic world was made crystal clear to me throughout the writing process.

There’s a reason why Dean’s name comes first on the book’s cover: he is unambiguously the principal contributor to Programming Scala. We worked from his outline, and though much was changed and reorganized collaboratively, the backbone of the book is absolutely Dean’s. I couldn’t have asked for a better coauthor. If anything, I wish that I could have contributed more equally to the text. What I did contribute, though, I’m proud of. We handed off our content to the production team at O’Reilly earlier today, and I can hardly wait until October to have the final product in my hands (and on my Kindle).

I took on the book in part to develop a mastery of Scala, and I’ve looked forward to learning something new every time I sit down to write, week after week. Though I understand more of the language than I did when I started, I still don’t feel that I’m on the level of folks like David Pollak, Jorge Ortiz, Daniel Spiewak, and the rest of the Scala gurus who dove into the language well before Dean or I. Still, it’s been an incredible learning experience, and I’m extremely grateful to everyone who made it possible, not least of all our editor, Mike Loukides.

This coming weekend will be my first in many months that is completely open, free of examples that need to be written, sections that need to be reorganized, reader feedback to incorporate, and unfamiliar concepts that need to be researched before I can write about them with confidence. With the free time I now have again, I can get back to writing open source code, mixing music, exploring the Bay Area, posting to this blog, and all the other things I’ve set aside while working on the book.

It’s liberating to be done, but bittersweet.

Since I Left You

I have managed to do a bit of this and that while wrapping up the book. We’ve set up a blog for the Twitter platform, and I’ve contributed a few posts to that. I’ve managed several talks, and committed to a couple more later this year.

Most cathartically, I’ve laid open my virtual notebook of ideas for all the web to see. I’m going to be at Twitter for a couple more years at least, if all goes well, and it seems a shame to let ideas sit in a text file and rot in the interim. Some of the ideas I fully intend on seeing through myself, in time. Others I hope will be picked up, improved upon, or made irrelevant by people smarter and more talented than myself. Either way, it’s been an experiment in radical openness (a topic I’m preoccupied with), and one that’s paid off from day one.

It should be less quiet around here, again, now.

Reading The Web on Kindle 2

2009-05-17T00:00:00-07:00

Reading The Web on Kindle 2

When I ordered a Kindle 2 shortly after the device was announced, I promised that I’d review it. Thanks to the book, I haven’t had time for much personal writing, but I’m sneaking this in today between edits.

The Kindle works so well that it’s difficult to review it in any critical sense. It delivers the experience you’d imagine it does: you buy books and periodicals, they show up instantly on the device, you read them. Text looks great, it’s easy to use, and it does indeed have the widely reported effect of making you read more. I had to suffer a couple of lemons before I got one that worked, but Amazon was courteous and quick about replacing them.

Really, the Kindle is a no-brainer purchase. Go buy one; you’ll be happy with it. That is, if you only want to read books and periodicals on it.

Dun-dun-dunnnn.

The Kindle is great for traditional media, but I wanted to see how it fared with hypertext. Shortly after I got my Kindle, I stopped using Google Reader. I dumped my feeds into Kindlefeeder, pruned out the purely visual and aural subscriptions, and set up a delivery schedule such that I had a fresh digest waiting for me when I got home from work. When I found things during the day that I wanted to read in depth, I’d pop them into Instapaper and they too would be batched into a digest and delivered once a day.

For a while, this worked well. I was no longer tempted to check the feeds or read interesting articles during the work day, as I had no convenient way of doing so. I got into a routine of reading my daily Kindlefeeder digest just before bed and going through the Instapaper items on the weekends. Looking at text on the Kindle is such a vastly improved experience over reading on a computer or iPhone that I forgave the occasional formatting or delivery error.

If the hypertext you read doesn’t make use of links, pictures, movies, block quotes, or essentially anything other than paragraphs and markup for emphasis, you’re set. In my experience, though, the Kindle is a pretty lousy platform for reading hypertext. The browser is filed under “Experimental” for a reason: the web looks weird in it, and the browser doesn’t behave quite like the rest of the Kindle. Bouncing between a Kindlefeeder digest and the browser is slow and clumsy. Technical articles I’d saved via Instapaper lost critical formatting and diagrams. Frustrating, all around.

This is nobody’s fault, per se, certainly not Kindlefeeder’s or Instapaper’s. Amazon is pretty clear about what you’re supposed to use the Kindle for, and that’s books and magazines and maybe the very occasional text-only blog. The Kindle is awesome for these things. It’s just not a multimedia device.

“Well hurr”, you say, but I guess I hadn’t realized how many of the feeds I read are composed of a rich mix of text, audio, video, and still images. I’ve been doing the blog thing for long enough that I remember when feeds with images were pretty unorthodox. Tumblelogging has really opened the world of syndicated online publishing up to more than just text. That’s a good thing for the web, but a bad thing for constrained devices like the Kindle.

So, after a couple months of reading web content pretty much exclusively on the Kindle, I’ve ultimately decided to go back to Google Reader and a browser on my laptop. Doing so requires more self control, and text doesn’t look as good, but it opens back up a world of multimedia content that, much to my surprise, I had come to miss.

In a way, I prefer the idea of a barrier between my Kindle and the web. I want my reading time to be about deep engagement with substantive content. Some blogs qualify for that time, but not many. I’d rather flip through my feed reader over my morning cup of coffee and keep the Kindle for novels, The New Yorker, and perhaps an academic article or two.

Mending The Bitter Absence of Reasoned Technical Discussion

2009-04-04T00:00:00-07:00

Mending The Bitter Absence of Reasoned Technical Discussion

There’s a counterpart to my post on technology journalism that I’ve been hesitant to write. Just as most professional journalism on high technology fails us today, so too does the online discussion amongst technologists as a community.

Social media (blogs, community news sites like Reddit and Hacker News, Twitter and such) have swept in to fill a vacuum between peer-reviewed academic journals and water cooler conversation amongst software engineers. Anyone with a blog can publish development war stories, benchmarks, or an interview with another developer. It’s a world of engineer’s notebooks laid wide open, and in theory, we should be more informed as a profession than we ever have been.

In practice, the conversations that are most widely heard in the tech community are full of inaccuracies, manufactured drama, ignorance, and unbridled opinion. In discussing these Internet-spanning debates with non-technical friends, comparisons to Hollywood tabloids come first to mind. It’s a time sink for an industry that should be a shining example of how to use the newest of media for constructive debate.

Faith and Numbers

My first harsh exposure to this sort of discussion was in 2007, several months into my contract on Twitter. I was asked to do a short email interview about our system and its development. I asked the interviewer if he’d prefer to interview the more senior developer on our team, but he insisted that he wanted an “in-the-trenches” perspective. I answered his questions to the best of my ability, and went back to work.

Imagine my surprise a few days later when my answers to a series of straightforward questions on a blog were the talk of the web application development community. One comment that seemed particularly uncontroversial to me was at the core of this nerd firestorm: Ruby is slow.

For the next several days, I could think of only one thing: why is something you can measure controversial? Are we not engineers? Is this not Computer Science? Why are we discussing the performance of software as if it were a matter of faith, or opinion, or preference? If you disagree, simply run an experiment. Did we somehow lose our Enlightenment values on the way to Web 2.0?

Of course, what I had said about Ruby was of no surprise to those who had been working with the language in earnest. In a private IRC channel, I was told by the Rails elite that I’d said nothing they didn’t all know. It was simply a tradeoff: work with a language you enjoy, accept the slower performance, buy more servers. Unless you have some deep insecurity about your choice of programming language, there’s simply no reason to be offended by such an assertion.

Time melted away my frustration, Twitter continued to grow, and I eventually got a chance to speak in person with some of the most vicious responders to my little interview. Amazing how suddenly reasonable and transformed a person can be when they’re forced to look you in the eye.

Two Years On

Earlier this week, I gave a presentation at the Web 2.0 Expo in San Francisco. As I made clear at the beginning of my talk, I was presenting in part because I’m co-authoring a book about the Scala programming language, one that’s being published by the same company that puts on the Web 2.0 Expo, O’Reilly Media. Cross-promotion, I believe it’s called. Scandalous.

For me, writing the Scala book and giving these talks are a labor of love. I’ve given up my free time to deliver the book, just as my co-author has taken time out of his personal life and consulting schedule. We’re getting paid for our work, but it’s no secret that writing a technical book is not exactly the fastest route to staggering wealth. I don’t get paid to speak; at best, my travel expenses are covered. I speak and write because it’s a way I can contribute to the Scala community, and I want very much to share the brilliant work they’ve done with anyone who will read or listen.

My presentation argued that Scala is a great choice of language for the next wave of Web 2.0 businesses, which must operate in a far more restrictive economic climate than their predecessors. My post on recession engineering sums this idea up. I said that people chose languages like Ruby and Python for early Web 2.0 businesses because those languages are a pleasure to develop rapidly in, and that Scala has that same quality while delivering better performance and leveraging the assets of the Java Virtual Machine, which can translate to dollars saved on hardware and new development.

I went out of my way to explain where Ruby’s strengths are as we see them at Twitter, without spending so much time on the issue that it detracted from showing off Scala’s features and making the business case for the language. I briefly covered why we didn’t make use of other languages. I did not discuss why nobody should ever use these other languages, but simply why we (Twitter) currently do not. It would be absurd and disrespectful to tell a room of other engineers why their choice of language for their projects – projects they know everything about and I know nothing about – is wrong. I try to speak only about what I know first-hand.

That a language is “fun” or “agile” or “beautiful” is unmeasurable and fuzzy. That a language is fast in comparison to another language, or can make use of code written in other languages without penalty, is not. Those incontrovertible facts, coupled with Scala’s specified and tested features, were the crux of my argument. Nobody who was actually present at my presentation found these facts controversial enough to stand up and dispute them. Indeed, the reaction amongst attendees seemed to be one of newfound curiosity in Scala, which is all I was trying to accomplish.

Some hours after I’d left the conference, the dual-headed distortion machine of the tech press and social media went to work. Before long, the story about my presentation had gone from “Scala is a nifty language and you should think about it for your business” to “Twitter engineer spits on the grave of Ruby, exalts Scala as shining new deity”. Time warp to 2007, numb and dismayed.

How To Have A Reasoned Technical Discussion

We’ve come to accept that trying to have a reasonable discussion on the Internet is like insert any number of increasingly offensive metaphors here. Usenet, IRC, forums, blogs, and now media like Twitter have all been black-marked as houses unfit for reason to dwell within. And so we roll our eyes, sigh, and quietly accept the idiocy, the opportunism, and the utter disrespect for our peers and ourselves that is technical discussion on the Internet.

This need not be the case. It is possible to have a reasoned technical discussion on the Internet. People do it every day, particularly in smaller online communities where social norms are easier to enforce. We can do it.

The next time you’re thinking about engaging in a technical discussion, run through these questions before you hit the “post” button:

Are you responding to facts? With facts?
Have you read any primary source materials on the issue you’re discussing?
Do you have any first-hand experience with the technologies or ideas involved?
Do you have any first-hand experience with those technologies operating at the scale being discussed?
Have you contacted the individuals involved in the discussion for further information before making assumptions about their findings?
Are you falsely comparing technologies or ideas as if there was a zero-sum competition between them?
Are you addressing your peers with respect, courtesy, and humility?
Are you sure that what you’re posting is the best way to promote your self, product, project, or idea? Does it demonstrate you at your best?

Et cetera, et cetera. Or, essentially, a brief reintroduction to logical thought and civil society.

And a final tip:

Some technical discussions veer towards the purely aesthetic. Thankfully, the humanities have provided us tools for reasoning about that which hard science may not be able to measure. Spend some time with art and theater criticism and you’ll find intellectual instruments aplenty for the comparative evaluation of seemingly intangible qualities such as beauty, theme, and emotion.

Conclusion

I don’t get much out of writing this sort of proscriptive, parochial content, but it’s my honest hope that someone will read the above, stop, think, and make a better contribution to the discourse of the technical community.

This mode of discussion is lower than us. Yet, lest we make an effort to rise above it, it’s all we deserve. Don’t waste your life screaming into the void. Make things, measure them, have reasonable and respectful conversations about them, improve them, and teach others how to do the same. Set emotion aside, and think how much we could accomplish if we had the humility and grace to learn from our peers.

Towards Better Technology Journalism

2009-03-03T00:00:00-08:00

Towards Better Technology Journalism

Rarely does technology journalism produce informed, correct, relevant, and readable content. This is a sorry and damaging state of affairs.

I’ve been drafting this post in my head for ages, and bringing the topic up to friends and colleagues ad nauseam. One approach I could take is to rantingly provide example after example of miserable technology journalism. For anyone immersed the culture of high tech – that is, those of you who care about this issue and have read at least this far – those examples are practically unnecessary.

Most technology professionals I know roll their eyes at our industry’s press. “What are you going to do? Can’t live with ‘em, can’t get publicity for new products without ‘em” seems to be the mindset. To ask for truly superb coverage of anything more than the latest gadget is asking too much in today’s tech media landscape. As an engineer, a consumer, and an avid reader, I’m unsatisfied with this.

Before I dive in, I’d like to clarify that this is entirely my personal opinion, and in no way reflects that of my employers. To maintain focus in my job, I ignore any and all press about Twitter that I’m not forced to read while strapped to a chair with my eyes pried open, Clockwork Orange-style. My personal projects have been covered with reasonable accuracy. This is not axe-grinding. Moreover, my agenda is not about ensuring that technology business and research gets a pass from the press; if anything, our industry should be regarded more critically.

With that said: instead of harping endlessly and unproductively on the culprits, I’ll briefly feature two recent examples of inept tech reporting. Then, I’ll expand on the problem and offer some potential solutions.

Example One: “TechCrunch Are Full Of Shit”

That phrase has been a rallying cry in the web community of late, urged on by a post on Last.fm’s blog. Long story short, the popular web-oriented technology news site TechCrunch reported on a rumor, something the site does seemingly as standard operating procedure. Generally, companies and individuals don’t bother to retaliate when slandered by TechCrunch, as to do so would lend an iota of legitimacy to to the site, while reducing the victim to their level of pettiness.

Last.fm bucked this informal policy and took a stand. They were quickly validated for doing so. The damage to the company’s reputation, though, is done. In an industry where ending your consumer relationship with a company is one click on a “delete my account” button away, misleading and false press can utterly undermine a business.

TechCrunch hardly count as “journalists” or “the press” by any reasonable definition. They’re a tabloid masquerading as a legitimate news outlet, a sort of Drudge Report for nerds; they lack even the sense of humor of actual tech industry tabloids like Valleywag. While TechCrunch may have started out as a blog, free of the restrictions and expectations of traditional journalism, their content is now syndicated by the Washington Post. TechCrunch’s is one of the most widely-heard voices in technology reporting. This should be considered an embarrassment to our industry.

Still, to really illustrate what’s wrong in technology journalism, an example from a credible publication is necessary.

Example Two: Dirtying a Clean Slate

In mid-February, the New York Times published a piece by John Markoff with the provocative title Do We Need a New Internet? The article discusses a fascinating research project called Clean Slate, set to “reinvent the Internet” with an eye towards security. Anyone with some network engineering experience understands that while the Internet is a clear success on a human scale, there’s room for improvement at the level of bits and bytes, particularly when it comes to security.

Markoff does this essential research project (and credulous readers alike) an enormous disservice by veering away from actual reporting at the end of his article. The last several paragraphs are nothing more than speculation on the author’s part, and not even speculation that’s of particular relevance to the aims of the Clean Slate project. Beyond being generally in the ballpark of security stuff, there’s nothing pertinent there. Though its placement in the “Week In Review” section may account for the editorialization, it’s serious subject matter being treated carelessly.

While not outright slanderous, the New York Times have not, to my mind, fulfilled their journalistic responsibilities when discussing Clean Slate. Airy speculation is for blogs (cough) and barrooms. No wonder that security experts like Ben Laurie were aghast at the irrelevance and incorrectness of Markoff’s conclusions.

The poor quality of technology journalism is not simply an infection plaguing unaccountably popular blogs. It’s real and present in the most trusted names in news.

The Problem

The scary truth of information technology is that it’s just too huge a domain to be an expert in, even if you’re a full-time engineer. I’d wager there’s just a handful of people on this planet who can claim expertise in everything from silicon up to human-computer interaction. Even if most engineers were halfway-decent writers, most engineers aren’t equipped to write about technology in the large.

The majority of technology journalists are even less equipped. Many have no engineering background. They’ve never built anything like the things they write about. Or, if they were once engineers, they haven’t written a line of code or soldered a circuit in years. In a fast-moving industry, professional engineers get left behind the state of the art all the time. How can journalists without any engineering expertise possibly hope to keep up? Simply tapping expert sources isn’t enough. A reporter can’t simply string together quotes from PhDs and CTOs and end up with something cogent, accurate, and informative to a non-technical reader.

We shouldn’t be content to trust the public record of high technology to individuals ill-equipped to report on it accurately. In an age where new media have enabled the people who make technology to produce a dynamic record of its creation and use, the role of the technology journalist is to tell a story that reaches outside our industry and community. It is, then, partly our responsibility as an industry and as a community to ensure the quality of that shared story.

The worth of accurate technology journalism produced by qualified professionals is unquestionably high to the technology industry and the public it serves. How do we fix this?

Solutions

This is not an easy problem to solve. Journalism of all sorts is under attack; the lack of quality reporting and the corresponding lack of trust and engagement amongst readers is not unique to the technology industry. High tech has the advantage, though, of a track record of interdisciplinary problem-solving. Solutions are out there.

Here’s several, to get the ball rolling:

Teach technology reporting in J-school (better)—Journalism schools offer specialized tracks for business reporting, financial reporting, even sports reporting. There are a handful of programs in technology journalism, but these tend to focus on the hard sciences. High tech industry professionals should help instructors develop accurate and informative curricula, and make themselves available to speak directly to journalism students in the classroom. Want someone to talk to your journalism class about how web applications work? Email me. I hope other engineers will contribute their expertise to the classroom.
Report on your 20% time—An increasing number of tech companies encourage side-projects and open source contributions. Why not use that time for journalism as well? Let’s get working engineers in the field, reporting on the subject matter they know better than anyone. This would require a strong editorial hand, but the risk seems worth the reward of highly informed technology reporting.
Incentivize technology reporting as a career—It’s hard to make a buck as a technology journalist, particularly one who reports on something more substantial than gadgets and empty enterprise software press releases. No wonder that TechCrunch has gone the route of sensationalism; it drives ad clicks and sparks debate, making a potentially dreary beat profitable and exciting. Tech journalism isn’t sexy, but it could be made so. That change starts with breaking the cycle of low-quality tech reporting, giving prospective technology journalists a set of role models they can aspire to.

Your solutions are needed. Without them, the only hope for technology journalism is that communication channels like blogs and Twitter outpace the inaccuracies of bad reporting with the distribution of fact. To a degree, this is already happening: friends only heard about the Last.fm scandal because the corrected story was making the rounds on Twitter, “routing around the damage”. This is small comfort, though, and a shallow goal for our industry and community.

You can assume the inevitability of mediocre technology journalism, or you can contribute solutions and make a change. The fidelity of the public history of high technology is in your hands.

Why I Don't Allow Comments, and More on Everything Buckets

2009-02-24T00:00:00-08:00

Why I Don’t Allow Comments, and More on Everything Buckets

I don’t allow comments on this site. I have my reasons.

There are certain types of sites for which comments work well. Metafilter is probably the best example of a long-lived web community that still boasts valuable, cogent comments. Investor Fred Wilson’s A VC blog also consistently has friendly, insightful discussions about finance, music, and more. I’m willing to admit that comments can be done right.

For most sites, though, comments are worse than useless. The anonymity of the Internet inspires hit-and-run attacks, unintelligible ramblings, and truckloads of spam. I believe that comments are evil by default, and the sites above that seem to have healthy communities are blessed flukes.

That’s all secondary, though. The main reason I don’t allow comments is that I want to inspire debate. I think people do their best writing when they’re forced to defend their ideas on their own turf. It’s one thing to leave a comment on someone else’s blog, but quite another to put your argument in front of your own readers. It forces a level of consideration that, without fail, results in a higher quality exchange of ideas.

I couldn’t have been happier with the reactions to my Case Against Everything Buckets. For one, it spawned way more discussion than I was ever expecting. But what’s more, it was a great discussion. I love being told I’m wrong, and I love it even more when I’m being told I’m wrong by smart people who write well. And, for the most part, that’s exactly what happened.

The Reactions

My favorite (mostly) positive reaction was from Adam Rice, in no small part for archly and correctly dubbing my piece “ranty and prescriptivist”. While a future in which meaning is extracted from unorganized data may be coming, it’s not here yet, and Adam elaborates on that beautifully.

The developer of ShoveBox, one of the applications I called out, wrote a thoughtful piece that defends his product as a kind of experimental inbox. He makes some great points about the awkwardness of the desktop metaphor, and in doing so brings up a glaring flaw in my piece: I sort of assumed people realized that I use the Terminal for a lot of this stuff, and spent too much time suggesting that the Finder is the way to go for most file creation/manipulation/organization tasks.

The developer of Tinderbox, a visual information organizer, made a compelling argument that Everything Buckets are best used for data that may have future use. My personal filing system overlooks this, for the most part. As I’ve written before, I mostly eschew that which I can’t immediately put into a contextually appropriate tool. Some data – quotations, for example – don’t fit this model. They need to age, to mature, to eventually find a context and a use. Flat files work reliably for storing this data, but they don’t add anything to it. Point taken.

Longtime blog buddy Buzz Anderson wrote a defense of VoodooPad against my argument. I enjoyed reading it, but I’m afraid it was unnecessary. I specifically didn’t call out VoodooPad because I don’t think it’s an Everything Bucket. It’s a great application for writing hypertext, and hypertext is structured data (at least more structured than plain text). VoodooPad may encourage you to “put your brain in it”, but it also suggests sane boundaries about what belongs in it and what doesn’t. So while I appreciate what Buzz has to say, he’s preaching to the choir on this one. Michael McCracken picked up on this too.

At the same time, though, I won’t apologize for my “condescending, engineering-centric view of the world” when it comes to recommending the use of structured data. Basically, I’m trying to be the neighborhood greasemonkey, flatly stating that if you learned how your goddamn car actually works, you’d get better mileage and take it to the shop less often. Sure, I’d love to live in a future where users can input their data as finger-painting and get meaningful results back, but as Adam Rice recognizes above, we’re just not there yet. Tut-tut.

Funny, too, that Buzz talks about VoodooPad – which requires each new page to be named by the user – as a “frictionless” tool when John Gruber takes pains to discuss Untitled Document Syndrome, in which having to figure out what to call a file is an affront to humanity. I actually liked Gruber’s response a great deal, and took a sobering look at the applications in my dock after reading it. Except for the tools I use for programming, none of them require me to explicitly create and title documents. The “library paradigm” domination Gruber wants to see might already be here today.

All of that speaks to why I don’t allow comments. Maybe those rebuttals would have been written if I did, but I can’t be sure. What I do know is that the authors took the time to call me out on their own turf, and I think it made for some great debate.

Addendum: A Bucket I Can Live With

I went furniture shopping this past weekend. It was a reconnaissance mission, scouting ahead for potential purchases when I move house in the coming months. En route, I realized with some chagrin that I needed an Everything Bucket. I needed a place in which I could put both text and photos, numbers and labels. I needed a flexible organization scheme. I needed a way to gather a variety of information while mobile and then sort through that information on my Mac when I got home.

While heading over the bridge to Emeryville, I downloaded Evernote. I used it to snap pictures of items and price tags, and to note down in text what couldn’t be captured photographically. For the most part, it worked great. When I got home, I downloaded the desktop Evernote client and got everything in (gasp) sync. I can’t see putting everything in Evernote, but it’s certainly handy to have a mixed-media data capture system when you need one.

I’d like to think that my original point still stands: you should pick the best tool for the job, the one that’s going to do the most for your data. Some jobs require tools that can be misused. It’d be possible to dump all sorts of things into Evernote that it doesn’t handle particularly well, like to-do lists. But for a certain type of job, used with discipline, a potential Everything Bucket like Evernote really works. I’ll be keeping it around.

The Problem With Email Clients

2009-02-08T00:00:00-08:00

The Problem With Email Clients

A little over a week ago, Gmail made it possible to “go offline” and take the contents of your email archive wherever you like. Slate’s technology columnist, Farhad Manjoo, wrote an effusive piece declaring Gmail the victor in a battle between desktop email clients and webmail that’s been raging since the mid-1990s:

“Gmail has bested the Outlooks of the world [...] If you’re still tied to a desktop app—whether Outlook, the Mac’s Mail program, or anything else that sees your local hard drive, rather than a Web server, as its brain—then you’re doing it wrong.”

Self-described “pixel-pusher” and member of the greater Twitter Mac cabal, Neven Mrgan, retorted with this gem:

“Well, I’m convinced. I guess I’ll just switch to an email client that doesn’t allow me to drag a goddamn file into the message to attach it.”

While chuckle-worthly, this isn’t strictly true: it’s possible to drag a file onto any file-type input form element in Safari, and the site-specific browser Mailplane makes more general drag-n-drop possible with Gmail on a Mac. Neven’s commment does, however, illustrate the core problem with email clients, desktop or web-based: they’re all utter failures at something.

The Problem With Desktop Email Clients

Manjoo is more than right in claiming that Gmail has bested desktop email clients on most fronts. He talks about Gmail’s speed, its efficiency, that the user doesn’t have to worry about storage, about how its search actually works. Weirdly, though, he doesn’t elaborate on one of Gmail’s best design decisions: conversations.

Gmail presents email threads as one long conversation, starting with the oldest message and ending with the most recent. When coming back to a conversation, older messages are collapsed until you need them. No matter who joins the thread, the conversation will remain one unified, chronological, vertical stack of messages. You can archive entire conversations, and they pop back into your Inbox when continued by new messages.

Anyone who’s given Gmail a fair shake will quickly find conversations indispensable. Going back to any other email client is agonizing and disorienting, like being knocked around and dumped out of the back of a pickup on the outskirts of a strange town. In desktop email clients, new messages arrive completely bereft of context. The only way to orient yourself is to either remember what the conversation was about or read through the mess of quoted text that may or may not be present at the bottom of the message, depending on what kind of email client or prefences the sender has. You could try searching to re-orient yourself, but good luck with that in Outlook or Mail.app.

With conversations, Google has offered the only advancement in the information architecture of email clients in decades. Apple, on the other hand, has given us basically bupkiss, rendering Neven’s defense a bit silly. While they clearly don’t expect many people to use it judging from the attachment toolbar button on the New Message window, it’s lovely of Apple to support drag-n-drop in their Mail.app. But without fundamental improvements like conversations, that’s sort of like bragging about the automatic windows on your car sitting on cinderblocks in the front law: nobody’s impressed anymore, and it’s not going anywhere.

I’ve been trying the beta releases of a desktop email client called Postbox. Other than being a slightly more palatable version of Mozilla Thunderbird, there’s precious little to recommend it. It learns essentially none of the lessons of Gmail, which is probably why one of the top requests from beta testers is a real conversation view. The developers are focusing on nigh-on useless features like social software integration, similar to Apple wasting its last major release of Mail.app on templates, notes that don’t go anywhere, and half-assed RSS reading. It’s one of the only new non-amateur desktop email clients to come out in ages, and it’s no savior.

The problem with desktop email clients is that they’ve gone nowhere, and appear to be going nowhere fast. While Google has delivered solutions that make email less excruciating, Apple, Mozilla, Microsoft, and now Postbox are twiddling their thumbs and staring at the ceiling. The primary recommendation for a desktop email client is that you end up with a copy of your messages on your hard drive, and even that argument is moot. So why do people keep using desktop email clients?

The Problem With Webmail (Really, With Gmail)

First of all, most webmail isn’t Gmail. Most webmail in 2009 is designed to look like a desktop email client. Web interfaces that attempt to mimic desktop software have been perpetual and stunning failures, so it’s baffling as to why they’re still designed this way. I’m not going to waste time tearing down the Yahoo! Mails and Hotmails of the world, or whatever terrible webmail interface your university, company, or ISP forces you to use. They’re abominable, and that’s why Slate has columnists covering Gmail and nobody gives a rip about Yahoo! Mail even though they’re the market leader.

Gmail does everything it possibly can to tastefully straddle the line between web and desktop application. For all its functionality, it still feels more like a web page than a web app. There’s no attempt at drag-n-drop, fake windows, big candy icons, or any of the usual signatures of desktop software. Other than their custom buttons, Gmail tries not to break too many of the web’s rules and conventions in its attempt to provide a fast and efficient interface to one’s email.

Unfortunately, this restraint isn’t enough to keep me from trying desktop clients like Postbox, or even secretly firing up Mail.app once a month to see if I can stand living without conversations for more than two minutes (I can’t).

This is probably heresy coming from a web application developer, but I think web applications are mostly ghastly. I hate using them. When I’m faced with a computing problem, I want to solve it with a polished, stable, native application for my operating system that looks and feels like it belongs on my computer. I don’t believe in Rich Internet Applications—they’re a boogeyman that I keep hoping will disappear.

The problem with Gmail is that it could be a “real” application. While its conversations and search-that-actually-works are (sadly) innovative, they’re not impossible to implement as part of a platform-native email client. I enjoy using Gmail, but I’d enjoy it even more if it obeyed the rules of my operating system, not the rules of the web. The web has a lot to offer certain types of problems, but I’m not convinced that email is one of them.

The Problem, In Summary

The problem with desktop email clients is that they’re not webmail. The problem with webmail is that it’s not a desktop email client.

My hunch is that the very thing geeks seem to like about desktop email clients-that you end up with a copy of our messages on your hard drive-is the very thing that drives most users to webmail. Desktop email clients have to deal with synchronization, and there’s no good way to hide from the user the impedance mismatch of syncing a local and a remote data store. Less technical users get frustrated by the schizophrenic behavior of desktop email clients: messages they thought they moved to a folder never left the Inbox; messages they thought they sent are sitting in their Drafts folder hours later; messages they thought were deleted are still there on the server.

Webmail alleviates this problem by allowing users to interact more directly with the data store that contains their email archive. It’s an illusion, but it’s a more complete illusion than desktop email clients can offer. When using Gmail, you feel like things are actually and really happening when you send an email or archive a conversation. Until, of course, your internet connection falters and you discover that you’ve lost your most recent actions in some local event queue in your browser.

If you’ve managed to make one or another of the current lousy options work for you, congratulations: way to get that car off the cinderblocks. The rest of us loathe our email clients, and would basically kill for something better.

The solution, I think, is to make desktop mail clients more like webmail, and absolutely not the other way around. The more webmail tries to behave “native”, the worse it works. The more desktop mail clients strive to provide an intelligent information architecture and reliable synchronization, the more users win. But before life with email really improves, the major desktop email client vendors need to catch up to Gmail.

The Case Against Everything Buckets

2009-01-31T00:00:00-08:00

The Case Against Everything Buckets

The Mac software ecosystem faces a plague. A plague of Everything Buckets. Indulge me.

If you search for “productivity” or “organization” software for the Mac, you’ll find variations on a particular type of application. These applications claim to be “your outboard brain” or “your digital filing cabinet” or similar. They go by many names: Yojimbo, Together, ShoveBox, Evernote, DEVONthink. There may be differences in their implementation and appearance, but these applications are all of the same sinister ilk. They are Everything Buckets.

An Everything Bucket, since you’re probably wondering, is what I call applications that encourage the user to throw anything and everything into them. They’re virtual scrapbooks, applying a lightweight organization system to (often) unrelated data of varying types. These applications typically employ a proprietary database, or at best, build atop the SQLite database technology that Apple ships with Mac OS X. They usually default to storing information in Rich Text Format (RTF) or Portable Document Format (PDF). They are Not A Good Idea.

Why Everything Buckets Are Not A Good Idea

Computers work best with structured data. Everything Buckets discourage the use of structured data by providing a convenient place to commingle “structureless” data like RTF and PDF documents. Rather than forcing the user to figure out the rhyme and reason of their data (for example, by putting receipts in a financial management application and addresses in an address book), Everything Buckets cry: “throw it all in here! Search it! Maybe I’ll corrupt my proprietary database, but maybe I won’t and you’ll have the joy of sifting through a mire of RTF documents. Doesn’t that sound great?”

This proposition should not sound great. If you think you’re going to save time in the long run by throwing your data into a big bucket now, then sifting through it later, you are mistaken. There are better ways.

The Filesystem

If you want to store data of differing types within a lightweight organization system, I encourage you to check out the filesystem.

On Mac, an application called Finder provides a pleasant interface to listing, organizing, and previewing information in a filesystem. It’s free, and it’s running on your Mac right now. Oh sure, it’s got some quirks, but for the most part you’ll find it reliable. You can create directories, and even create directories inside directories (a feature that Yomjimbo currently lacks).

Everything Buckets are selling you a filesystem, and removing the step of creating and saving a new file within that filesystem. That’s their primary value. Whatever organization scheme they may claim to offer, you can replicate on the filesystem. I promise. Even tags (symlinks, aliases – look ‘em up).

If you still feel like paying $39 for a proprietary second-rate filesystem, well, bless your heart. You’re the stuff a software developer’s dreams are made of.

The Search Illusion

Most Everything Buckets also let you search. Let’s talk about that.

We live in the age of search. “Why can’t I just Google for it?” is a common complaint from anyone who’s ever misplaced a file and doesn’t know better. In response to this complaint, desktop search technologies like Spotlight have emerged. How do they work? They sift through your structureless data periodically and build a structure from it. This structure is called an index. Without an index, searching through all but smallest quantities of unstructured data takes ages. Once again: computers work best with structured data.

So while you, the user, are presented with the illusion of “Googling” for that phone number you threw into your Everything Bucket, your computer is constantly hauling ass to make up for the fact that you couldn’t be bothered to put the phone number in your virtual address book. “What do I care how much work my computer does?” you say, until the next time the Spotlight indexer kicks in and your Mac grinds to a halt.

This posits a stronger correlation between Everything Buckets and things like Spotlight than I really mean to put forward. The point is that if you don’t do some organization of your data up front, you probably won’t like the ways in which it’s done for you later.

With an Everything Bucket, you also miss out on opportunities to do interesting things with data. Once data is normalized and structured, finding correlations is faster and easier. Remember the first time you really got a spreadsheet to do something cool for you? You can’t do that with a big steaming pile of RTF files.

Applications That Actually Do Things

One of my Rules For Computing Happiness is: “do not use software that does many things poorly.” Everything Buckets violate this rule up, down, and sideways. They’re poor filesystems, poor text editors, poor databases, poor to-do lists, poor calendars, poor address books, poor bookmark managers, and poor password managers. At their worst, they’re even poor web browsers, poor encryption systems, and poor synchronization schemes.

The corollary to that rule is: “use software that does one thing well.” When you need to store some data, there are so many wonderful applications to pick from. From recipes to receipts, photographs to music, journal entries to to-do list items, there’s a great application out there for what you need to do. Chances are good that the right application structures your data so that you can get more out of it. Use an application that actually does something more than holding data. You’ll be happier.

If you don’t want to be forever accumulating applications, store stuff as plain text on the filesystem. It’s dead reliable, you can shuffle it about however you want, and you’ll be able to search through it quickly, even without an index.

Conclusion

Get your brain out of an Everything Bucket and in to an application (or two, or three) that adds some value to your data. Of the culprits named above, at least DEVONthink can claim some intelligence in finding correlations between documents. But chances are good that if you think about the task you’re really trying to accomplish, there’s an application out there for it that will make your life easier. Do some research on a site like I Use This or MacUpdate and find an app you trust.

Your computer wants your data to be structured. Throw it a bone.

The Thing About Security

2009-01-12T00:00:00-08:00

The Thing About Security

Last week, Twitter, the thing I work on, had some security issues. Earlier in the week a phishing attack started going around. Then, someone used a dictionary attack to grab the account of one of our support staff, who has administrative privileges on the site. We cleaned up both of these issues just as fast as we could. The company communicated quickly and openly about what happened, and if anything, the press seemed to encourage more people to check Twitter out. It was a stressful week, but it forced some positive changes.

It stings a bit to have experts like Bruce Schneier call us out – and rightly so – for not shoring up holes like this sooner. Defending against a dictionary attack is basic stuff. Our fix for it was an afternoon’s work. But dictionary attacks just aren’t something that’s defended against by most web authentication stacks out of the box, which means the vulnerability lingers until a security review or a successful attack. In part, I think this is due to increasing specialization in computer security education, but that’s a topic for a longer post.

The thing about security is that it requires stakeholders. I have a security background, but Twitter’s security isn’t my job. In fact, my job is pretty much the opposite: I open up as much of Twitter’s functionality as I can without (hopefully) making the system insecure. So while I’ve usually been a “first responder” to security incidents because of my background, it requires a major mental context switch from the work I normally do.

Several months after I joined Twitter in early 2007, I suggested to the team that we do a full internal security audit. Stop all work, context switch to Bad Guy Mode, find issues, fix them. I wish I could say that we’ve done that audit in its entirety, but the demands of a growing product supported by a tiny team overshadowed its priority. Now we’re in an unwelcome position that many technical organizations get into: so far into a big code-base that’s never seen any substantial periodic audits that the only way to really find all the issues is to bring in some outside help – something I sincerely hope we end up doing, but is not my call.

Outside security review is about spending enough money that your organization becomes its own security stakeholder. Invest in a high-priced security consultancy’s time for a week and you’d damn well better do something with their findings, right? It’s still a reactive model for a product’s security life-cycle, rather than a proactive one, but it’s something. Ultimately, outside security audits are the price a company pays for not building security mindfulness and education into day-to-day development.

My advice: be a stakeholder in your organization or product’s security from day one. If you don’t have time to be that stakeholder, find one, either inside or outside of your company. Make sure that dollars are on the line, or it won’t get done. There are hundreds of security issues in any complex system that fall under the heading of “basic stuff”, and they won’t get fixed until that happens.

Though maintaining users’ privacy and trust is incentive enough, the cruel reality is that good security is fueled by more than just good intentions and best efforts.

My Interview with Waferbaby

2009-01-02T00:00:00-08:00

My Interview with Waferbaby

My friend Daniel, perhaps better known as Waferbaby, has started a new section of his ever-evolving web presence that features interviews with people about their hardware and software setups. It’s called, appropriately, The Setup, and I’m flattered to be the first person Daniel interviewed for it.

I’ve reproduced my answers here for my own record-keeping. In case you’ve ever wanted to know basically anything about how I compute, I tried to answer Daniel’s questions in extreme detail. The speculative final question was the most fun to answer.

So who are you, and what do you do?

I’m Alex Payne. I go by al3x around the Interwho. I work at Twitter in San Francisco as their API Lead. Our API is how we expose the things you can do with Twitter in a way that programmers can use in their own applications and websites. Part of my workday is spent writing code to make the API go, and part is spent helping developers out with their questions and suggestions.

In my free time, I’m working on a book about the Scala programming language. I geek out on programming languages, economics, culture, and theory. I DJ with a friend once a month.

What hardware are you using to get your work done?

I have a 15” MacBook Pro, one of the final revisions before the current black-bordered unibody design. It’s got a 2.6 GHz Core 2 Duo processor and 4 GB of memory. I hook it up to a 30” Cinema Display when I’m at work, and I prop it up on Rain Design iLap when I’m at home. I type Dvorak, but I leave my keycaps in their factory default QWERTY layout.

Instrumental to having just one machine is a good networked storage and backup strategy. I have a ReadyNAS NV at home with about 750 GB of expandable RAID storage. Most of the disk goes to store media that I’d flip out if I lost, and the rest goes to Time Machine backups. I’ve also started backing up essential files with Backblaze.

I always have my original (pre-3G) 8 GB iPhone with me. I listen to Shure SE110MPA headphones on the go and Audioengine 5 speakers over an AirPort Express at home. My home network is tied together by a Linksys WRT54GL wireless router running the Tomato firmware.

And what software?

I spend most of my day flipping between TextMate, Terminal, Safari, and a Fluid app for Gmail. I write Scala and Ruby code in TextMate, run tests and such in Terminal, and answer hell of email with Gmail. I keep Safari mostly ad-free with Safari AdBlock and speed up searching with Inquisitor.

I use iChat for IM and Twitterrific to interact with Twitter. iCal syncs up to Google Calendar for scheduling, and Things keeps track of my tasks on both my MacBook and my iPhone. I’ve recently started journaling, and I use MacJournal for that. I keep a big library of PDFs (academic papers, presentations, books, &c.) in Yep. I’ve taken to storing sensitive files in encrypted disk images managed by Knox.

I also run some of the usual “power user” Mac enhancements: TextExpander to make emails go faster, LaunchBar to get to what I need quickly and to keep my clipboard history accessible, Growl and HardwareGrowler to get visual notifications from software, and Caffeine to keep my Mac awake while watching web videos and such. I keep tabs on my machine with iStat pro.

We use Dropbox at Twitter to share documents, and we make pretty extensive use of Google Apps. We use Trac for ticketing internally, but I keep API-related tickets public on Google Code. We do code reviews with Review Board and browse our Git repositories with cgit.

I’ve usually got iTunes or Last.fm playing, though lately I’ve been listening to Bloom play itself when I need to focus. Speaking of focus, I use Isolator to get distractions out of the way. When I need distractions again, I see what’s on my Instapaper or Google Reader.

What would be your perfect, ultimate setup?

Bill Joy said at this year’s TED conference, “I’d argue today that we have incredibly powerful computers, but we don’t have very good software for them”. I’m of the same mind.

Hardware-wise, my current setup is a dream. I love having just one machine to look after, and I love that it’s both portable and powerful; my only wish would be for a workday’s worth of battery life. I can get decent network access nearly anywhere via HSDPA if Wi-Fi isn’t available. The Cinema Display lets me sprawl out. My ReadyNAS gives me a sense of security about my data. My iPhone lets me leave the house with just my keys, my wallet, a pair of headphones and this one little magical device that can connect me to anyone, direct me anywhere, and entertain me for days. It’s a good time for hardware.

Software is another story. I’m perpetually dissatisfied with my text editor, the tool I spend the most time with. OS X is the best Mac experience there’s ever been, but it’s hardly the best computing experience I can imagine. The iPhone is an incremental improvement, but I can’t accomplish the majority of my daily computing tasks on it comfortably.

I want to interact with my computer in a fundamentally different way. The desktop metaphor is dead; a generation of children have grown up never working with the physical objects that the virtual desktop represents. What I really want is a modernized Plan 9, a software platform that’s designed from the ground up for our networked, distributed world. If you try to placate me with the assertion that the Web is this new operating system I will become violent.

I want better software: more usable, more accessible, more open, more secure, more integrated, more seamless. I want a better software development experience. I want better programming languages with better development toolkits. Fundamentally, I want better abstractions for the same computation I can do today with all that lovely hardware.

Why I Don't Work In Information Security

2008-12-31T00:00:00-08:00

Why I Don’t Work In Information Security

Yesterday, the web was abuzz with news about a paper released at the 2008 Chaos Communication Congress. This paper details an attack against the SSL certificates that web browsers and other software use to verify and encrypt communication between client and server. Some of my younger security industry friends were feverishly announcing that the attack “breaks the Internet”. Old security hands like Bruce Schneier and John Viega said, respectively, “[t]his isn’t a big deal” and “there is little to worry about”. Which is it?

Over lunch, I commiserated with a coworker at Twitter about this odd event, as it’s a prime of example of why he and I no longer work in information security.

Much of the tech world is obsessed with engaging in macho pissing contests, but no part more so than computer security. In the case of yesterday’s announcement, the researchers in question were more concerned with their ability to present their findings at a popular hacker conference than with guaranteeing the safety of the Internet. Why else would they put the organizations they disclosed their findings to under NDA and not consult the authorities on the most widely-deployed SSL implementations? Building reputations and managing PR is the order of the day.

This culture of one-upsmanship doesn’t mean that computer security is a stagnant discipline. It does, however, mean that the people who choose a path of humility about their work don’t get the rewards – financial and otherwise – that they deserve. This is a shame, and it’s to the detriment of digital security as a whole. My coworker suggested that an academic, peer-reviewed approach to security research would ultimately be more beneficial to the Internet community as a whole. I don’t have the authority to comment, but I do feel that most anything else would be an improvement on the traveling hacker conference circus we have today.

Secondarily, I don’t work in information security because of the technologies that have become prevalent in the industry. Not that I have any particular talent in them, but both exploit development and reverse engineering are all about Windows, and have been for several years. Working with Windows day in and day out is a mind-numbing prospect, even if the theoretical payoff is control of 90% of the computers on Earth. Application-level attacks are mostly about the web, and honestly, I’m deathly bored of the client-side web.

These reasons are a big part of why I’ve ended up working in the “under-the-hood” part of web application development for the past two years, even though I spent most of my adolescence dreaming of getting paid to hack. Interestingly, I’m not alone. I’ve met a number of ex-security people who’ve ended up building high-performance systems for the web. It’s still a pissing contest, of course, but a much less public one, and with much less of an agenda.

Someday, I expect I’ll come back to information security. Despite the politics, it’s compelling, high-impact work. I hope, though, that the field’s culture has changed by that time. I’m not holding my breath.

Addendum, Jan 17, 2009: Two of the people involved with the SSL hack replied to me about this post via email. Alex Sotirov said:

“We already have an academic, peer-reviewed approach to security and it is not working. Academic papers about the problems with MD5 collisions have been published since 2004. The exact same attack that we did was described in an academic paper in 2007, but even this was not enough to get the CAs to discontinue using MD5.

When we got up on stage and presented our real rogue CA certificate, the CAs were finally forced to do what they should have done years ago. Verisign stopped using SHA-1 just 5 hours after the talk.

I am not disputing that this is a pissing contest and that my personal motivation to do this work is far from altruistic, however the result is that the Internet is a safer place today than it was on Dec 29.”

Jake Appelbaum had this to say:

“[W]e were actually working towards making the internet safer; we weren’t harming anyone and that harm reduction is a large part of how we planned our release. We informed the relevant parties and we didn’t release anything that could harm anyone. The information to do the chosen prefix attack had been public since 2007.”

Both are smart guys, and they know the security world far better than I do. It sounds like they had the best of intentions, along with their professional interests, in mind.