Mike West

Instapaper is Amazing

2009-03-22T20:37:07+00:00

I'm subscribed to a slowly-expanding list of something like 230 RSS feeds (I take the firehose approach to news-gathering, apparently), and for all practical purposes, my internet experience centers around NetNewsWire. Most mornings, I sit down, skim through the headlines of some subset of my feeds, and open tabs for all the the articles think I might find interesting. Tabs are NetNewsWire’s killer feature, so far as I'm concerned. I can pop open a few tabs and NNW happily keeps track of them across restarts so I don’t miss out on whatever it was that I found interesting if I don’t read the article immediately.

Unfortunately, this is sometimes as far as I get. On Friday, I had 97 tabs open; some had been sitting there since January, which is a bit absurd. These were articles that I'd almost certainly find interesting, but that I hadn’t made time to actually read. Reading ought to be the entire point of the tab-opening process, but here I was, opening tab after tab after tab, regardless of the queue of content I'd already identified. Obviously something has gone seriously wrong with my workflow.

Carlo has been singing the praises of Instapaper for a while now, this mountain of built up content seemed like a perfect test case. After digging around a bit, I came up with a script to “integrate” my Instapaper queue with NNW (NNWInstaPost), and I started pushing articles to Instapaper, and closing tabs. This worked halfway decently (I'd like a ‘to Instapaper’ toolbar button in NNW. Or a keyboard shortcut. Or anything other than clicking through the Scripts menu for each article individually). After a day or two, I've come to the conclusion that Carlo was right: Instapaper (in particular, Instapaper.app on my iPhone) is gobsmackingly brilliant.

I think I've read more articles with Instapaper in the last two days than I have in the last two weeks with NNW alone. Walking around with a queued-up list of content in my pocket is really quite wonderful. I can read on the bus. I can read while waiting for the bus. I can read while walking from the bus to the subway. I can read on the subway. Etc, etc.

I love it. I think you will too, go try it out.

Opera Web Standards Curriculum: The JavaScript Bits

2009-02-08T12:19:27+00:00

Last year, I jumped on the opportunity to sit down and write some articles for Opera’s Web Standards Curriculum. I bit off a bit more than I could chew, and Chris Mills exhibited the patience of a saint as I finished the first quickly, the second slowly, the third very slowly, and then completely failed to deal with the rest. Regardless, those were released along with the rest of the JavaScript bits to complete the curriculum.

I'm not particularly proud of my end of the process of getting these articles released, but I'm happy to see them released, and happier still to see the company they keep. Chris did a brilliant job organizing a brilliant group of authors; Opera (and Chris in particular) have organized a great body of work, which I'm proud to have had a hand in.

My articles are:

I hope you enjoy them.

Centralized Bug Tracking

2009-01-10T19:32:15+00:00

I liked many things about working at Yahoo. I'm coming to realize that what I (in hindsight) like most is probably the piece of software I thought about the least positively, namely Yahoo’s mostly centralized and completely open bug tracking system: Bugzilla. We abused it more than a bit, attempting to layer task and project management on top of a system that wasn’t really designed to support it, but all told, Bugzilla made my work life better.

As a generic employee, the centralization of bug tracking meant that I was able to quickly and easily file bugs against any Yahoo property. I didn’t have to know who was responsible for a project in order to raise bugs against it. I didn’t need the group responsible for a project to know me. When I saw an issue on a Yahoo site, I filed a bug against the project, and knew someone with the capability to fix the issue would be notified about it. Bugzilla minimized the friction caused by unclear answers to the question “I found a bug, now what?”. Instead of sending out a few emails, looking for someone to stick with a problem, it gave everyone in the company a clear “next step”, and (in the best cases) fostered a corporate culture of reporting bugs rather than avoiding them.

As a developer, Bugzilla meant that I didn’t have to keep the list of bugs on my projects. The bug database was maintained for me, triaged and prioritized by my managers, and brutally honest. Every bug that was reported against News sat in my queue, staring at me pleadingly until I fixed it. I made appropriate comments on each bug when necessary, which simple integration with CVS made trivial, with the cumulative effect that I didn’t worry about forgetting to fix something, or losing track of a bug’s status. Everything was maintained for me, removing a burden from my shoulders.

This isn’t to say Bugzilla was perfect. It was a bit of a mess, honestly, often difficult to use, full of confusing forms and confused categorizations, and plagued by an understaffed team of developers who played with the UI far too often. For these reasons and more, it probably annoyed me more than any piece of software at Yahoo, but it’s existence was hugely advantageous. In hindsight, I'm coming to consider this a critical component of any development team; a central bug tracking system provides visibility and accountability in a way difficult (impossible) to replicate with personal to-do lists and email.

Don’t read this as an endorsement of Bugzilla in particular, but as an endorsement of the concept of bug tracking. Working without a centralized bug database makes your work life more difficult for no good reason. It’s something I highly suggest that you avoid.

If you'd like to get started quickly with an externally hosted bug tracking system, I've heard good things about Lighthouse and Sifter. I'm still looking for a locally hosted system that I like, but I've been recommended Mantis, FogBugz, and, of course, Bugzilla. Honestly, even a hand-maintained text file in Tasks format that you print out and pin to the wall for people to write on is better than nothing. For the sake of your own sanity, use something.

Some Thoughts Regarding Caja

2008-12-16T20:39:27+00:00

Yesterday, Yahoo! made some announcements regarding The Future™ of many of their high profile properties. Specifically, they’re (slowly) opening up, enabling third-party developers to build applications that can be seen on and interact with your My Yahoo! page, or your mailbox. I think this is a great step, and one I wish they'd made before they laid me off.

Ah well.

One of the core technologies that’s behind this set of features is called Caja. Caja is a code sanitizer: it takes an HTML fragment, and JavaScript that operates on it, and “cajoles” it into a chunk that can be embedded into a page without the risk of maliciousness. I'd like to ramble about that, briefly, at a very high level. I'm still trying to wrap my head around it’s details

JavaScript is, simply, dangerous.

If you've paid attention to any of Doug Crockford’s presentations, you’ll know that the browser security model is simply broken-as-designed. The internet, therefore, is a place where one can barely trust first-party code, much less code written by your neighbor. You have to keep a constant eye out for new cross-site scripting vectors, and be very careful about how you filter third-party input before making it available as “user generated content.”

Seen in this light, Yahoo! has a massive problem to confront with it’s new “open” initiatives. On the one hand, they must protect the security of their sites. On the other, they want to pull in content from their users, and not just text, but code. Working applications, written outside of Yahoo!, running directly on a Yahoo! site. The project specification itself is basically a nightmare scenario for the security team. They need to find a way to include third-party JavaScript safely and sanely onto Yahoo! pages. This mechanism needs to be pretty automatic, as they can’t dedicate an engineering team to manually review (potentially) thousands of applications.

There are two broad paths to take to this end:

Code sanitization, which reads unknown code, processes it, and outputs a sandboxed version (if possible). Caja is the best known example of this tact.
Static analysis, which reads unknown code, parses it, and gives a thumbs-up if it only does known-safe things. AdSafe is a work-in-progress along these lines.

Yahoo!’s running with the former, so let’s dive in.

Code Sanitization

Untrusted JavaScript must not be allowed access to the document or window objects. This means that it must not gain direct access to any DOM node, as every DOM node enables you to crawl back up to document. event objects are right out as well, as they contain dangerous references as well. Really, when you get right down to it, you have to throw out practically everything of practical use.

I mentioned that JavaScript was broken, right?

Caja is an attempt to distill a safe subset of JavaScript out of this mess through server-side sanitization. A third-party uploads an application, consisting of JavaScript, CSS, and HTML fragments, and Caja transforms it into something that can be guaranteed not to damage the page into which it’s embedded. This is a good thing.

Caja is a capability-based system, meaning in practice that it begins by defining an extremely restrictive sandbox in which code must run, and enabling well thought-out bits of functionality by selectively injecting access as needed.

Think of it this way: when you hand your car to a valet, you would be better off if you gave them the valet key, which only enables them to drive the car. You shouldn’t give them your key, which would also let them rummage through your glove box, etc. In the same way, you shouldn’t give a program access to everything in the DOM if it’s only supposed to change a background colour in a particular location. You'd be better off if you could restrict it’s scope of access. Caja attempts to do this.

The token you give the valet, the key, can be looked at as a set of capabilities. The valet key enables “drive the car”, your key enables much more. Similarly, handing an object to a program is handing it capabilities, enabling it to act in whatever ways are exposed by that object. Since JavaScript’s default objects are so overpowered, Caja exposes a new set of objects that completely wrap things like the DOM or the event model, and rewrites input programs to use these objects instead, severely limiting the damage they can do. These wrapper objects contain a series of runtime checks to ensure that a malicious program hasn’t somehow broken out of their constraints, and the entire rewriting process fails if the program is written in such a way as to make it impossible to sandbox.

Big Drawback

Though I understand Caja’s practical necessity, I really don’t like the way it works. In short: it breaks progressive enhancement completely, and introduces a hard dependency on JavaScript for functionality.

As a quick demo, let’s look at the following code:

<script src="searchbox.js"></script>
<link rel=stylesheet href="searchbox.css" /> 
<form> 
  <input type="text" size="60" name="q"> 
  <input type="button" value="Search" onclick="doSearch(this)"> 
</form>

After “cajoling”, it will look something more like:

...
IMPORTS___.htmlEmitter___.p('form') 
    .a('onsubmit', 'return false') 
    .ih('  <input type="text" size="60" name="q">\n' 
      + '  <input type="button" value="Search"' 
      + ' onclick="return plugin_dispatchEvent___(…)">\n') 
    .e('form'); 
...

The HTML is transformed into a series of JavaScript method calls that generate HTML. This makes sense, as it enables Caja to retain complete control over what’s written to the page, but it has the side effect of making the form completely inaccessible to anyone who isn’t running JavaScript.

I'd much prefer more thought to be put into AdSafe, which sets up the same sort of wrapped-object sandbox, as well as a series of rules which third-party developers must follow. The system them simply verifies that they have done so, rather than rewriting their code to ensure that they have. If the rules are solid, the effect will be the same as can be achieved with Caja, but much more elegant, and with more respect for the fundamentals of the web.

Crockford has (finally) put up some example code on the AdSafe site. I'd suggest that you go take a look at it. It looks like a very interesting way to program indeed.

My job's value

2008-11-30T15:50:24+00:00

Recently, I wrote a short article on the effect a team’s sense of ownership in it’s projects can have on the finished product. The surprising twist in my professional life last week has led me back onto the same train of thought, but I'm coming to it from a slightly different angle. I discussed “ownership” in a narrow sense, relating to a team’s involvement with and responsibility for decisions made about a project’s vision and direction. This isn’t the only way in which the word has impact; one also has “ownership” of a project in the broader sense of pure possession.

A relatively famous study offered a representative sample of students a straight choice between a bar of chocolate and a coffee mug. Opinions were split: neither bars nor mugs were substantially preferred. If, however, a group was given one, and then asked if they'd like to trade it for the other, surprisingly few would. Even though the items were roughly equivalent in the abstract, possession (however brief) seemed to imbue them with irrationally high value. My coffee mug is much more important to me than some random bar of chocolate with which I have no relationship, even if I've only had my mug for a few minutes. Even if it’s a bit chipped and cracked, it’s mine, and that counts for something.

The economist Richard Thaler coined the term “endowment effect” to describe this phenomenon, and I'm reminded very strongly of it as I start looking around for a new job. Now that my projects at Yahoo! have been taken from me, I'm starting to wonder whether I perhaps I've been valuing them more than I rationally ought. There are a lot of good jobs out there, and I'm relatively sure of finding one quite quickly. Perhaps my job wasn’t the best for me after all. Perhaps that bar of chocolate across the room is significantly tastier than I expect. Perhaps.

This, of course, is nothing more than a thinly veiled attempt to rationalize to myself the thought that it’s really not all that bad to have my position yanked out from under me. It’s not quite working.

Yet.

Has Mike been laid off? Yes. Yes he has.

2008-11-21T17:13:13+00:00

Yahoo! has decided to close down the engineering team in it’s German office, of which I am (er… was) a part. I'm suddenly incredibly motivated to look for new work.

If you happen to know of an exciting web company somewhere in or around Munich that’s in need of a solid webdev, please drop me an email (mike@mikewest.org), ping me on twitter, take a look at my resume (or linkedin profile, if that’s your style), or give me a call ( +49 176 4854 6453 ). I'd really love to hear any and all leads you might have. :)

I ♥ GitHub

2008-11-16T16:53:09+00:00

Over the last two or three weeks, a substantial subset of my friends and colleagues have started using GitHub to host some of their personal projects. I'm really enjoying this influx, and it’s inspiring in a way I didn’t really expect. GitHub has done nothing less than to make my friend’s coding activity visible to me, and mine visible to them. This doesn’t sound like much, but it’s simply transformative; If this is how “normal” people feel about Facebook, then I can start to understand how it’s captured so much mindshare.

Coding in the Open

Visibility is inspiration and accountability. Watching talented developers Get Things Done™ around me gives me impetus to start putting something together of my own. Neil was creating a new repository every day at one point, and Norm’s potential set of projects is great to see. The whole group of Londoners are setting an example I'd like to live up to, and at the same time generating gentle social pressure for me to build something exciting of my own. Watching practically everyone I know fork Norm’s homedir immediately after he put it online is simply brilliant. I want that to happen to my projects. You couldn’t ask for better, more constructive peer pressure.

To that end, I'm putting as much of my current code as possible out into the open. I'm not generating a whole lot of code that I'd expect to be of use to anyone but myself at the moment, but even so, I work harder when I know that people I respect will be seeing what I'm producing. I'm coding for myself, but an audience changes the way I think about what I'm doing; The simple fact that my friends are following my progress on GitHub is reason enough to try to exceed their expectations.

Moreover, I've been able to convince myself that even robot’s opinions matter. Just knowing that CalendarAboutNothing is watching my every commit gives me reason to make sure that I make a little bit of time every day to sit down and write something resembling quality code. It’s a simple thing, and has been much more effective than I thought it would be over the last ~2 weeks. I highly recommend trying it out; soon you’ll be just as hooked on big red X’s as I've become.

An Admonition Regarding Details

2008-11-11T22:33:41+00:00

If Apple’s taught me anything about design, it’s that details are everything. The overall product might be brilliant, but it’s the tiny bits of perfection that really bring things together and imbue an experience with a sense of wonder and care. When I noticed that Rob Goodlatte (who has gone dark, apparently?) replaces the ampersands on his Lucida Grande dominated page with lovely, lovely Baskerville, I was thrilled. The first time I saw the little bit of bounce-back at the end of an iPhone’s scrolled list, I was hooked. These almost insignificant changes have an effect on the overall experience far out of proportion to their apparent importance.

It’s important, however, not to miss the forest for the trees. Attention to details will often make or break a project, but first laying down a solid foundation of functionality in broad strokes is critical. If you haven’t yet built a bit of your application, worrying about making it pixel-perfect cross-browser and subtly animated to amaze your users is nonsensical and counterproductive.

Put (virtual) pen to (virtual) paper, and start working. Details will fall into place naturally, either in the nooks and crannies of unconnected code you cleverly hack together to solve a problem, or in the long periods of iteration and polishing that you’ll start to go through near the middle of a project when things mostly work.

Test-Driven Development generally advocates that you should begin by ignoring (irrelevant) details and “Do the simplest thing that could possibly work.” Neil Crosby similarly says “Make it work, Make it pretty, Make it right.” My Dad (enthralled with the message while completely missing the commercial point of the Nike campaign) always told us “Just do it.” These are starting to resonate with me, and I like the idea of the development process as a continual process of iteration, building something delightful

This, of course, is a long-winded way of justifying the gaudy hack I've just put into Fallow to handle simple conditionals in templates. It’s ugly, but functional, and I know I can make it cleaner tomorrow. But right now, it works; That’s better than yesterday, and I can live with that.

The Inspiration of Ownership

2008-11-10T19:49:27+00:00

On the bus home from work, I was listening to this week’s On the Media. In particular, I was struck by a great interview with the man who designed the field-organizer and volunteer training programs for Barack Obama’s campaign: Marshall Ganz. If you’re at all interested in the political angle, I'd suggest you listen. If you’re at all interested in how I'm planning to apply this seemingly unrelated topic to the technical field of web development (et al), keep reading.

Ganz’s greatest accomplishment in this race was to empower a lower tier of volunteers to an extent that simply hadn’t been accomplished in previous elections. By exposing a breadth of information and actionable intelligence that went beyond what campaigns in the past had been able or willing to share, his team was able to create a wide-reaching group of volunteer leadership that came to believe that Obama’s campaign was their campaign too. This self-identification enabled Obama’s volunteer staff to develop individual personal narratives that were far more compelling than anything the campaign could have come up with on it’s own. Information lead to a feeling of ownership, which lead to acceptance of greater responsibility, and exhibition of greater creativity.

The thing is, this isn’t at all limited to the political sphere: software projects work exactly the same way. In particular, and regardless of the relative “objective” value, I’ll work harder on my features or fixes than I will on yours.

Fostering Ownership

Especially in a large company working on large projects with many layers of indirection, the importance of a particular bug or task to a project’s leadership is almost guaranteed to diverge in some (hopefully small) way from the dreams and desires of the team actually implementing the changes. Personal buy-in is simply hard to come by when tasks are handed down from on high, but that personal involvement is absolutely critical to a project’s success. What to do?

I have a few suggestions from my perspective, primarily as an implementor looking upwards:

Influence

Reduce the distance between decision and implementation. The more layers a task has to go through to reach it’s eventual executor, the less likely that it has any potential to inspire. Flattening the decision structure such that the implementing team feels like it has a hand in the project’s direction is one way to increase personal involvement, and to make the team feel that the project is somehow, for each of them, individually “mine”.

Moreover, bringing the decisions about priorities, and the rationale for the decisions, out from shadowy backrooms and into the harsh light of (probable) criticism is the single best way to make sure that the project is going in the right direction. If the product vision can’t stand up solidly in front of the team that’s meant to implement it, it’s not going to go anywhere in the market, even if it’s perfect in every way. The development team has to believe in what they’re building; they have to be in on decisions that they feel they are making.

Information

Share information, even if you don’t think it’s relevant. You'd probably be surprised to learn exactly what your development team actually cares about. Good (and bad) feedback from users or new data from Comscore isn’t something that you should talk about once a quarter in planning meetings, but could instead be a source of inspiration and innovation for the entire team, every day.

Campaign volunteers are closer to the voters than the campaign leadership, and look at the raw voter data differently. Unforeseeable (or simply unseen) trends might be easily explainable with the application of their local expertise. Likewise, engineers and web developers work right at the heart of a site or application. Flood them with up-to-the-minute information you consider relevant and irrelevant. They will draw connections you couldn’t anticipate, and fill gaps you didn’t see, and that creates visible success that inspires.

Innovation

Allow room for surprise and innovation. The products and applications that the development team builds are pure thought-stuff, forged and hammered through will alone. This has a few impacts on the project’s process, the most important of which is that the solution to a problem can almost never be specified at the same time the problem itself is confronted and documented, because the statement of the problem is never complete enough to illuminate a complete solution. Experimentation and innovation, and the feeling that both are allowed and encouraged are critical.

Obama probably didn’t tell each of his volunteers individually how they should campaign in their communities. He and his staff trusted in their ability to assess the general problem of increasing voter turnout, and turned them loose. In the same way, the development team shouldn’t be given detailed technical descriptions of solutions to implement, but instead detailed technical and nontechnical descriptions of problems to solve.

The code written must be based on their expertise, their choices, and their understanding of that problem. This personal involvement in the problem solving inspires a sense of ownership that can’t be created by mandate. If the final solution that’s written doesn’t address the issue at hand in a way that’s somehow surprising or innovative, then the development team is bored, uninspired, and uninvolved.

In a nutshell…

If you make me believe that a product, or a feature, or a fix is mine by giving me influence, information, and room to innovate, I’ll move mountains to accomplish more than you expect. If, however, my paycheck alone is supposed to inspire me to greatness on your project, then I’ll slowly stumble over every molehill on the path to mediocrity.

Flickr's API is driving me nuts

2008-11-09T17:00:13+00:00

I'm trying to do something with the Flickr API that I consider to be relatively trivial. I have the impression that the API is fighting me every step of the way. Why, oh why, can’t the wonderful people who designed Del.icio.us’s new API hop over to Flickr and slap together something that makes sense from the perspective of the end user?

Basics

I'd like to have a list of my flickr sets as a block on my homepage, displaying the most recent sets in the order they were published, along with some simple metainformation and a thumbnail. I'd also like to display photosets in my archive pages, interspersed throughout the rest of the content at the proper point in the timeline.

Most of this is easy to get with a single call to flickr.photosets.getList. The XML that’s returned looks something like:

<?xml version="1.0" encoding="utf-8" ?>
<rsp stat="ok">
<photosets>
    <photoset id="[ID GOES HERE]" primary="2914968443" secret="[SECRET GOES HERE]" server="3007" farm="4" photos="2" videos="0">
        <title>2008-10 - Driving</title>
        <description />
    </photoset>
    ...
</photosets>
</rsp>

Simple enough to parse, and in combination with Flickr’s standardized url structure, this gets me relatively close to the data that I'm looking for. The small bit that’s missing is an actual publication date, which is significant for my plans. Without it, I can’t correctly insert the photosets into my archive pages, and so far as I can tell, the data simply isn’t exposed via the API. So let’s dig around a bit.

Since I know what the primary photo is for the photoset, I can grab it via flickr.photos.getInfo, which gets me the photo’s dates:

<?xml version="1.0" encoding="utf-8" ?>
<rsp stat="ok">
<photo id="2914968443" secret="[SECRET GOES HERE]" server="3007" farm="4" dateuploaded="1223227911" isfavorite="0" license="3" rotation="0" originalsecret="c87ab26ef7" originalformat="jpg" media="photo">
    ...
    <dates posted="1223227911" taken="2008-10-04 18:12:42" takengranularity="0" lastupdate="1223283362" />
    ...
</photo>
</rsp>

Leaving aside the strange change of timestamp format, and the nonsensical lack of timezone information, I can mash those into the photoset data to get an approximation of it’s creation date. That works pretty well, actually. It means, however, that I have to make an additional API request for each photoset. That’s more than a little annoying.

But now I have all the information I need, so putting the code together is straightforward. Straightforward, that is, until I start thinking about the necessity to automatically update this data periodically.

Updates

The Del.icio.us API is really good about handling this scenario: posts/all returns all the your bookmarks, and allows you to filter the list by tag or date range. If I know, for instance, that I last polled for changes at 10:00 this morning, I can ask Del.icio.us to send me only the bookmarks that came in after that point in time. This makes the update-handling code on my end quite simple: I ask for all the updates since the last bookmark I've stored locally, and when I get a response, I treat the whole thing as new.

Flickr doesn’t, so far as I can tell, support the same mechanisms. This has the effect of pushing validation down to my layer: I grab a list of all my photosets and then walk through the list, checking locally to see if I've already got the information stored. This is stupendously inefficient, and actually becomes more inefficient as time goes on and I add more photosets.

The closest Flickr comes to the Del.icio.us level of efficiency is flickr.photos.getRecent, which returns a list of up to 500 recent photos. It doesn’t, however, provide the same benefit as the Del.icio.us feed, as it doesn’t allow you to specify what “Recent” means. It simply pulls a set number of photos off the top and throws them back over the wall.

I would love to see something like the Del.icio.us functionality added to the Flickr API. It would make my particular use case quite a bit simpler, and if applied across the board, it would make the entire Flickr API better suited to the polling-based tasks it’s being put to.