kstewart

API Design is Interaction Design

2009-06-10T00:00:00-07:00

For the past few weeks, I have been working with my team on refining the REST API for our Web service. While there will be some new additions in order to meet requirements for applications that will build upon the API, many of the changes are refinements. Early adopters and our own experience has provided a treasure trove of information that has guided the development of the latest version.

From the very beginning, I chose a REST Oriented Architecture (ROA) for the service. To me, this just seemed obvious. I’ve followed the development of XML-RPC and SOAP for years. For awhile, XML-RPC seemed pretty cool and simple to me especially when compared to the mess that is SOAP (which evolved into WS-Death Star). However, in practice I always ended up sending some simple, Plain Ol’ XML over HTTP. For the majority of cases, HTTP and URIs covered the functionality I needed sufficiently.

As REST became a more prevalent meme, I got a copy of RESTful Web Services and read it cover-to-cover. I recommend this book to ANYONE building Web applications. It put what was common practice for me into context and really illustrated what REST and ROA are really about and what tradeoffs need to be made in every application.

One part of the book that really stood out for me were the sections on resource design. Instead of the highly procedural XML-RPC or my own custom-coded XML schemas in my POX/HHTP implementations, REST made me think more about the actual objects, or resources, that I wanted to interact with over the network. Doing this forced me to think harder about naming and behaviors (whether or not they mapped cleanly to GET, PUT, HEAD, DELETE and, of course, POST). For error conditions, there was almost always an HTTP status code that would work for us. And, HTTP headers were incredibly useful in dealing with metadata.

So, on my first pass with our API I attempted to adhere to RESTful principles. After identifying my resources, I spent even more time thinking about representations, the content types and schemas for the data my service would consume and produce. Many of my early assumptions changed quickly as I worked through this process (and again as we started working on the latest version) For instance, I once considered XML a safe, de facto format to support in an API. Then, along came JSON. When, I looked at some of my earlier representations I saw almost an order or magnitude difference in the size of an XML response compared to its JSON counterpart for exactly the same data! Now, in many cases, XML is a requirement. However, if you have the ability to define exactly which representations you will support, think long and hard about when to use XML. Given the proliferation of libraries in almost every conceivable language, JSON is almost as de facto a choice as XML and has even more practical benefits although there are also downsides, like potential security vulnerabilities. As in everything else, one should pick the tools that help solve the problem at hand.

The first versions of our API worked fine. The true test was seeing the number of people who just went to our documentation page and started coding against it. Generally, we received fewer than half a dozen questions via email before they were able to productively use the service. That’s a huge win! We also ate our own dog food by prototyping clients for the service using:

curl
Ruby (using Merb as our framework)
Flash (ActionScript 3 and the Flex framework)
Adobe AIR

All the clients worked well, with the minor exception of Flash. Unfortunately, Flash has some limitations with respect to things like:

HTTP response codes other than 200
PUT and DELETE
Custom HTTP headers
Using the User-Agent to identify the Flash Player

Surprisingly, Adobe AIR works just fine with our REST API. Here, we had a nicely defined API but needed to make it work with Flash. The thought of either creating a separate API or moving to a least common delimiter approach for everyone was not appealing. However, there was actually a simple solution. If we added a prefix to all our API endpoints, we would know that the client was the Flash Player. In our Web service code, we wrote a proxy that would extract input from the request and then call the REST API directly. When the API code generated a response, the proxy would repackage it appropriately for the player:

Always generate a 200 response
In the response body, use an XML document to house:
- The actual HTTP response code
- Standard and custom headers
- Response body from the REST API

The beauty of this approach is that it is fairly generic. As we add new endpoints, the proxy does not need to be changed. The Web frameworks router just tries to match everything after the prefix against our existing RESTful routes. If we have a match, then the proxy calls that API. Otherwise, we return an error. Simple. So, how is this interaction design? Well, developers are users, right? For the most part, we try every new API by using curl or these days, rest-client. If the API seems too cumbersome using either of these tools, then we’ve probably done something really wrong. And, we did. There were a few APIs that were a little unwieldy or confusing due to the naming, so we focused on them a lot for the latest version. Deprecation can be a good thing.

As developers, we use tons of APIs from various sources. Some are good, some not so good. Alex Payne, the API Lead at Twitter gave a great presentation the interaction design of APIs. I almost didn’t write this blog post because Alex’s presentation was so spot on. But, I think it is a topic that may not get the attention it deserves so I’ll throw my hat in the ring. In our API, we leverage an internal user authentication service as opposed to rolling our own. This service had an “XML API” that was essentially a Plain Ol’ XML over HTTP endpoint that called into their Java code. Not really RESTful, but generally workable. The problem was that their XML API did not have feature parity with their Java and ActionScript APIs. We were told that this would be addressed with the next version of the API and there would be a new “REST API”. Great!

Well, maybe not so great. The user authentication service was growing to meet the needs of multiple clients and, like us, did not want to rewrite their code to address their non-primary clients. Their approach was to use a proxy approach as well that generated interfaces from their Java class and method definitions. See where this is going yet?

The XML generated from the class and method definitions was approximately an order of magnitude larger than the previous XML. Sure, we got feature parity but good luck trying to figure out how to USE the API. I had to have at least 5 browser tabs open and look through JavaDocs to figure out how to create an account! I know, I know… once we get it done it’s done…or is it? What happens when they change method names or parameter lists? Alex defines “The Humane API” as:

Explorable
Predictable
Consistent

This user authentication API is not humane currently. I suspect that this is rooted in its implementation. It is a Java-based Web service and there are tons of tools and APIs that can turn Java objects into Web services. Unfortunately, the services generated are best consumed by other Java applications and services. My team uses ScrumWorks Pro for our agile planning and task tracking. It too has one of these Java-generated Web service APIs which I just find inscrutable. I’ve seen similar things in the .NET world as well. On the Web, Web service clients can be as simple as curl or as complex as a desktop Windows or Mac application like iTunes. Every programming language/tool has its own idioms, but they all have one thing in common: they likely have one or more libraries for communicating via HTTP.

And that is the interaction design advice I’d like to impart. Recently, there have been a proliferation of projects that help you build applications and services with nothing more than HTTP, Javascript/JSON and general RESTful principles. CloudKit bills itself as an Open Web JSON Appliance. Once you’ve downloaded and installed the Ruby gem, you can expose a resource and immediately interact with it using curl. Awesome! And then there is CouchDB, which is getting a ton of attention and for good reason. Jacob Kaplan-Moss, one of the creators of Django, said this of CouchDB in a blog post called Of the Web:

Let me tell you something: Django may be built for the Web, but CouchDB is built of the Web. I’ve never seen software that so completely embraces the philosophies behind HTTP. CouchDB makes Django look old-school in the same way that Django makes ASP look outdated. Let me try to explain what I’m talking about. You want an API without having to write a line of code? It’s called curl, and it ships with your MacBook. And just look at how simple the APIs are in your favorite language.

If you are building a Web application or service, strive to make it Of the Web.

Web Development vs. Software Development

2009-04-13T00:00:00-07:00

From Eric S. Raymond, The Art of Unix Programming, Chapter 11: The Web Browser as a Universal Front End

For a large class of applications, it makes increasing sense not to write a custom GUI front end at all, but rather to press Web browsers into service in that role. This approach has many advantages. The most obvious is that you don’t have to write procedural GUI code – instead, you can describe the GUI you want in languages (HTML and JavaScript) that are specialized for it. This avoids a lot of expensive and complex single-purpose coding and often more than halves the total project effort. Another is that it makes your application instantly Internet-ready; the front end may be on the same host as the back end, or may be a thousand miles away. Yet another is that all the minor presentation details of the application (such as fonts and color) are no longer your back end’s problem, and indeed can be customized by users to their own tastes through mechanisms like browser preferences and cascading style sheets. Finally, the uniform elements of the Web interface substantially ease the user’s learning task.

For most of my career, there has been endless, tedious debate over the nature of software development. Is it math, science, engineering or craft? Well, having gone to an engineering school to study computer science and in the process take a TON of math, science and engineering courses I can tell you that the majority of what people call software development is a craft and we should stop obsessing over that fact.

I have been a self-professed computer science bigot for a long time. I tended to make a distinction between people who had degrees in computer science (or at least studied the foundation stuff) and self taught developers. You know, a caste system. The high priesthood vs. the unwashed masses. And, in the dotcom days, that bias was reinforced when I saw the “designs” created by the Perl and PHP hackers. I have never, and probably will never, write a line of Perl in my life. Of course, some think that Ruby is the new Perl so I may have just lied! :-)

However, after a few years writing Active Server Pages applications with VBScript and then moving back to a more comfortable language in Java with servlets, JavaServer Pages and JDBC my perspective slowly changed. Sure, I wrote model-view-controller (MVC) code in classic ASP just as I did in Java. But, I learned many things by working closely with Web designers. I gained greater appreciation for aesthetics, user experience and interaction design as I built more Web applications. While my stomach churned over some of the hackish code I saw, I realized their true beauty… they worked and provided value for their users.

Most computer science programs unwittingly reinforce this foolish caste system mentality from the 60s and 70s because faculty see themselves as divorced from the trivialities of pragmatism, usefulness and real-world scenarios because they are teaching theory and not practice. Some institutes of higher learning have at least acknowledged the need for a separate track in software engineering that focuses more on what practicing developers need to know: version control, debugging, cross-functional collaboration, etc. However, the very term software engineering still conveys the wrong sentiment that what we do is equivalent to electrical, aerospace or chemical engineering. It, emphatically, is not.

Going off on a tangent for a moment, I used to feel the same way about game developers. I used to read Game Developer magazine, especially their Postmortem column, for years and marveled at how little these people seemed to know about basic development principles like object-oriented design. Yet, like Web developers they made products that people paid good money for and enjoyed using. Who cares then if few of them knew how to write a class that properly demonstrated encapsulation, inheritance and polymorphism?

Perhaps, the differences could better be described as web design vs. web development? Unfortunately, this too is a false dichotomy. Web design without development often leads to brochureware. Or, some Web designers fool themselves into thinking that what they are doing is not creating code. I’ve seen some self-professed designers churn out some awesome Javascript and PHP or Python yet they don’t believe that they are developing software (even as they check their code into their Subversion repository…)

I may be in the minority here, but I actually believe that Apple did the WRONG thing with the iPhone SDK. I think they should have stuck to their guns and NOT come out with a way to build native applications for the iPhone. (Brief pause as multiple people pick their jaws up from the floor.) That’s right, I said it. I would have been much more impressed if they stuck to the idea that Web applications ARE applications and can be used across multiple device contexts. Apple could have exposed more of the iPhone’s features as Javascript objects that developers could utilize in their “Web” applications. By caving to the pressure to provide a “real” SDK Apple has continued to perpetuate the caste system. Granted, the iPhone is such a compelling device that many “Web developers” grumbled but grudgingly started to learn Objective C and Cocoa in order to build applications for it…

Wait. Web developers learned a language like Objective C that you compile down to a binary in order to deploy on a device? Isn’t that the domain of software developers?!? The false dichotomy breaks down! There is no us vs. them or hacking vs. engineering. There are only the people who like to tirelessly debate false dichotomies:

vim vs. Emacs
Tabs vs. spaces
Python vs. Ruby
Microsoft vs. everyone else

and those of us who care about our craft enough to focus on building beautiful things that people love to use.

Seven Things

2009-01-25T00:00:00-08:00

Damn you, Brian! for tagging me with this meme. But, since it is currently snowing (again?!?!) here in the Seattle area, I figured I needed a diversion. So here goes…

One

I was born left-handed. My parents are from the West Indies and apparently, being left-handed was considered BAD. So, at a very young age whenever my Mom saw me doing something with my left hand, she would say “Use your right hand!”. My wife told me she thought I was left-handed when she met me because I tended to do things with that hand. Ironically, both my son and our new President are left-handed. I’ve also learned that lefties tend to be more creative and possibly smarter. See what you’ve done to me, Mom?

Two

I met my wife on a blind date. I was at a party where I met up with some mutual friends. After hours of drinking and talking, one of the girls remarked, “I can’t believe you don’t have a girlfriend. You have to meet our friend, Donna.” Oh great. Since she wasn’t there and all the girls that were were…hot, then this must be the ugly one. :-) A few weeks pass by and a friend of mine calls me up one night, saying “Hey, remember that girl that those girls wanted you to meet?” Me: “Yeah…?” Friend: “Here she is.” (click)

One tentative conversation led to a first date. It could have been disastrous since I was late driving out to the boondocks of Long Island after spending the day helping a friend move. She was annoyed and I was tired. However, a good Italian meal and a bad movie made things better. A couple of more dates led to a vacation in Jamaica and the rest, as they say, is history.

Three

I used to be very good at math. I was an honors student all through grade school and most of high school. My math skills declined (and my social skills improved) by the time I hit college and took my first calculus course. I suspect the cause of the change may have been either:

My mostly Japanese-speaking professor
My complete disinterest in anything other than my Computer Science courses
Girls

Now, my math skills seem to be stuck at intermediate algebra.

Four

I love comic books and comic book movies. While I no longer collect them, I do have a pretty big collection in my garage. Batman and Iron Man are my all-time favorites. Superman is a punk (he turned down Wonder Woman’s advances in one issue!! What?!?!) So, when the live action Iron Man movie came out in 2008 as well as Dark Knight, I could have died happy and complete.

Five

I am technology agnostic. I’ve tried lots of different computing platforms. The first computer I ever used was an Apple II and the first one I owned was an Apple IIc. I’ve had and developed for PCs running DOS, Windows (3.1 – XP, including NT and 2000). I have worked on numerous workstations running SunOS, AIX, HP/UX and Irix. I tried owning a Linux workstation running Red Hat, but it frustrated me no end as a desktop. I might try again soon, as Ubuntu looks promising. Now, I am primarily a Mac user as it hits all the right notes for me.

Note to all Microsoft-bashers: Stop it! It’s not necessary. And the people that work there are not all bad; in fact, they’re mostly good people trying to do the right thing in a large corporation. If you don’t like Microsoft’s products or behavior, just vote with your dollars! That’s what really sends the message.

Six

My perfect vacation involves a week on an island (generally in the Caribbean) at some sort of all-inclusive resort. My days should consist of deciding whether to swim in the pool or the ocean first, enjoying great meals and getting to catch up on a ton of reading. One day can be allocated to some sort of touristy activity. But, just ONE day.

Seven

It is better to work for me than with me. At least, that’s what my co-workers say. I fiercely protect my team so that they can focus on doing great things. And, I have trouble managing up but I am trying to get better at that. However, at the end of the day I am an engineer and logic rules my world. My measure of success is building (or helping people build) great products.

Tag! You’re it!

Is Agile or Lean Enough?

2009-01-03T00:00:00-08:00

At the end of the year, I tend to reflect on the progress my team has made and what directions we should take in the new year. Actually, I think about this all the time. However, since the company shuts down I have extra time to be introspective. This year given the snowstorms in Seattle, I’ve had even more time.

Generally, I have been a proponent of agile techniques. They’re basically codifications of common sense best practices. For the past few years, I’ve been particularly attracted to Scrum for managing my development projects. While the basics are pretty easy, Scrum (like most agile methodologies) actually requires a lot of discipline. So, my team and I spent a lot of time increasing our discipline and paying attention to the issues that Scrum surfaced in our development process.

While that approach definitely delivered some benefits, I constantly felt like there were huge gaps in what agile in general and Scrum in particular were bringing to the table. A recent blog post by James Shore, The Decline and Fall of Agile, captured some of my concerns:

It says that the team should be cross-functional and recommends co-locating the team in a shared workspace. It says the team should deliver a valuable, shippable product at the end of every Sprint, and that the team should self-organize, discover impediments, and remove them.
Oh, and it also has a few mechanical things about a monthly Sprint and daily Scrum. Trivial stuff compared to the rest. But guess which part people adopt? That’s right—Sprints and Scrums. Rapid cycles, but none of the good stuff that makes rapid cycles sustainable.

Taking part in many agile discussions at work as well as practicing it on my project, I recognized this as one of the bigger problems. As much as my team and I may want to work in an agile fashion, there were other factors conspiring against us. Co-location was the biggest one and getting a fully cross-functional team in place is not always possible. Shore also states:

It’s human nature to only do the stuff that’s familiar and fun, and that’s what has happened with Agile. People look at agile methods as a chinese menu of practices, choose the few that look cool, and ditch the rest. Unfortunately, the parts they leave out are the parts that make Agile work. Scrum makes it worse by ignoring important (but hard) agile engineering practices, and the Scrum Alliance makes it worse still with their armies of trainers some good, some not issuing dubious “ScrumMaster” certificates to people who demonstrated competence in connecting butt to chair for two days.

I’ll be honest, our company offered Scrum training with Ken Schwaber and I took it (Hey, it was FREE!) And yes, I am now a Certified Scrum Master. And what enlightenment did I receive from this training. Not a whole heckuva lot. In fact, at the conclusion of this training I painfully recalled an enjoyable post from Steve Yegge, Good Agile, Bad Agile:

Up until maybe a year ago, I had a pretty one-dimensional view of so-called “Agile” programming, namely that it’s an idiotic fad-diet of a marketing scam making the rounds as yet another technological virus implanting itself in naive programmers who’ve never read “No Silver Bullet”, the kinds of programmers who buy extended warranties and self-help books and believe their bosses genuinely care about them as people, the kinds of programmers who attend conferences to make friends and who don’t know how to avoid eye contact with leaflet-waving fanatics in airports and who believe writing shit on index cards will suddenly make software development easier.
You know. Chumps. That’s the word I’m looking for. My bad-cholesterol view was that Agile Methodologies are for chumps.

And if you have taken this particular Scrum course and recall the “secret handshake” at the end, the chump title probably seems like a better name for the certification you received. Sure, there were a few useful bits in terms of clarifying a few ambiguous parts of the standard Scrum literature. However, the course offered no practical help with day-to-day development issues. Specifically, the disconnect from one or more development practices that pair up well with Scrum as Shore notes.

So, reflecting on the past year I see two fundamental issues with our adoption of agile practices. The first is the lack of top-down adoption of agile in our company. While there are many teams adopting agile (specifically Scrum), there is still the cross-functional gap. That leads to a lot of wasted time spent context switching between Scrum and the “traditional” development process. I’ve witnessed a lot of teams say that they’re doing Scrum but they’re just re-labeling portions of the legacy process with terms they learned from Scrum. More fragile than agile. It seems as if there is not an organizational drive towards continuous improvement that is driven from the top, then it is easy to regress to past behavioral patterns. In Scaling Lean and Agile Development, Craig Larman summarizes this nicely:

There are better ways to build large systems than with many developers in many places. Rather, build a small group of great developers and other talents that can work together in teams, pay them well, and keep them together in one place with product management or whoever acts as the voice of the customer.
But of course you are still going to do large, multisite, or offshore development. This is because your existing system is already structured that way, or because in the case of large groups there is the mindset that “big systems need lots of people.”

THAT’S what our current situation feels like. And that’s not something that can be addressed from the bottom up alone.

And my second issue? Exactly what Shore mentioned:

Scrum makes it worse by ignoring important (but hard) agile engineering practices

Scrum does not prescribe any development practice to use alongside it. That’s OK, but one would think that by now the Scrum practitioners would have strong recommendations based on real-world experience. The common wisdom is Extreme Programming (XP), but I have several issues with XP. It seems to be even more of a take-it-all-or-leave-it-all religion than Scrum is. And XP doesn’t work for all teams. And most agile processes over-focus on the development team, but gloss over the other cross-functional team members.

Now that the Scrum backlash is in full swing, people are starting to embrace Lean. What they fail to realize is that Scrum is just the gateway drug to Lean. Scrum is not a pure pull system, which is what you would have once you Got Lean. After reading Corey Ladas’ blog post and book some of the ideas resonated with me. But, this was only because I had years before been a fan of the Theory of Constraints and Eliyahu Goldratt’s books (sidebar: his books are great due to his ability to communicate dry, academic topics via engaging novels). And again, the pragmatic issues around cross-functional product development processes are still largely ignored.

So, in 2009, how will I evolve my team’s processes? We’ll probably continue with Scrum or even “evolve” towards Lean. But our focus will be on sharpening our engineering practices. We’ve been pretty good about constant communication and unit testing, but we can probably tighten up our feature designs and code reviews. It’s all about balance.

Welcome!

2008-12-29T00:00:00-08:00

Welcome to [working code]