Stardragon

Building Web 3.0

2011-07-14T13:56:26-04:00

This article was originally published on the New Relic blog.

When many technology experts speculate on the "next big thing" for the World Wide Web, several commonly mentioned concepts include the Semantic Web, personalization, and intelligent search.

Although these are useful concepts, none are particularly new or revolutionary -- especially when compared with Tim O'Reilly's definition of Web 2.0. Personalization has been around since the late 1990s, when Yahoo! launched the personalized web portal, and Amazon started offering personalized recommendations based on your shopping history. XML has been widely used for over 10 years. Semantic data standards such as Resource Description Framework (RDF) and Web Ontology Language (OWL) matured around 2004. And personalized search tools have been available for many years.

What's lacking from the Web 3.0 conversation is any discussion of interaction and integration between applications and web sites. It's one thing to be able to export, import, and mash-up data. But synchronizing changes to data entities is still either a manual process, or one that requires developing customized integrations with proprietary web services. And when a company employs multiple services to manage various parts of the business process, as mine does, time spent managing that data synchronization adds up quickly over time.

Let's focus on a particular example: the help desk. For many technology companies using cloud services to manage customer support, their environment looks something like this:

While many service providers offer an API to allow data to be exported or imported, and even modified in some instances, there has been a lack of common semantics to allow updates to a support request to propagate automatically between these systems. For instance, if a customer support issue is related to a system bug, updating each system of the progress can be a tedious and time consuming process to keep the customer informed of progress, and when the fix is deployed to production.

Although the internal process of managing customer support requests often can be highly customized to suit the requirements of a particular organization, most support tools are built around a common paradigm:

A customer sends a support request, or an incident is automatically generated by an incident monitoring system. We'll refer to these requests as tickets.
Tickets have a progress status.
Tickets are updated with annotations and attachments. These annotations may be public (visible by the customer), or private (visible only to support staff).
Customers should be regularly notified of the progress of and updates to a support request.

For many customer support organizations, as much as two-thirds of the overhead involved in support requests involves the manual synchronization of updates from one system to another. If only help desk service providers agreed to, and implemented, a common API for sharing and synchronizing information, these systems could be easily integrated, reducing the overhead of managing support requests greatly. Furthermore, customers would receive more frequent and timely notifications of progress.

Enter the Networked Help Desk: a new open standard for customer support management. This API will allow service providers to seamlessly integrate development tracking, issue tracking, CRM, application monitoring, and call center service systems and create a more efficient and effective customer support infrastructure. I'm very excited that my company is a founding member of the Networked Help Desk consortium, and look forward to sharing the progress of this effort in the future.

When buying a more expensive tool saves you money

2011-01-02T14:15:21-05:00

A few days ago, I saw this tweet from Giles:

In a tongue-in-cheek manner, I responded, "Ubuntu is the poor man's OSX." On my mind at the time was the fundamental complaint lobbied by Apple detractors: Apple hardware is expensive than other brands, and Apple all but forces you to run OSX on their hardware; hence it is more expensive to use.

What I didn't expect was this rebuttal blog post from Giles, which makes a reference to Dr. Thomas Stanley's book, The Millionaire Next Door. One of the core rules of the wealthy, as explained in this book, is the rule of always living below your means. I agree with that sentiment. Although I have not read Dr. Stanley's books, I have read several titles on building wealth by Robert Kiyosaki, and he also stresses this principle when coaching people on how to build wealth.

However, a key factor is missing from Giles' argument: productivity and the value of time. While money is a finite resource, it is one that, given one's acumen and effort, can be be built and accumulated indefinitely. Time is not. Rich or poor, we all have 24 hours each day.

When I first moved to New York City after graduating from college, I was earning a relatively paltry salary at my entry level job as a programmer analyst. I made a lot of frugal choices in order to save money: I rented a studio apartment in Brooklyn instead of Manhattan, furnished it with Ikea, sold my car, shopped for clothes in bargain basement stores, and cooked my own meals whenever practical. Another choice I made was to do my own laundry and shirts, which I thought was a prudent cost saving measure.

Doing my own laundry proved to be a poor choice. I had to schlep my clothes to the laundrymat, wait for the machines to be availble, watch them so no one would steal my clothes, carry them back, fold them, and, gag, iron my dress shirts. This took about 3.5 hours of my time each week, at a cost savings of about $15. I was, in effect, earning less than minimum wage doing this chore myself. By sending out my shirts and laundry instead, I gained 3.5 hours for work, rest, play -- whatever I wanted to do. My realization was that although money and time may be finite, it's possible to exchange one for the other. By making wise trades I could actually have more of both available.

So what does this have to do with Ubuntu and OSX? In a nutshell: OSX is the more productive operating system than Ubuntu on the desktop or laptop. Installation is simple and few choices are required. Upgrades are automatic and you're prompted when. Many OSX applications require little more than downloading the disk image, opening it, and dragging an icon over to your Applications folder (or running an automated installer program). Need to set up automated backups? Just plug in an external disk, or connect to a server, and turn on Time Machine. Even better: with just a few clicks of the mouse, you can revert a file, folder, or your entire system back to its state at a specific point in time. Purchased a new camera, printer, or wireless hub? No problem, OSX will automatically find and install the drivers for you. Try doing these tasks in Ubuntu, and compare the time spent vs. OSX.

When I was working on a Ruby / Rails project at a previous company, and given no choice but to use a Windows based PC, my team chose to install VMware and Ubuntu in a dual boot environment. Ubuntu was, after all, better organized and less difficult to install than most other Linux distributions. However, each time I had installed or upgraded Ubuntu, I had to spend about 4-5 hours tweaking my installation. While Ubuntu has a fairly decent package manager, the myriad library of available packages if mind boggling. You have to choose among many desktop environments, text editors, compilers, office productivity tools, etc. Often you have to repeat the process many times because of conflicts between competing packages, missing depndencies, and out of date server entries.

With OSX, installing most applications requires little more effort than downloading a disk image, opening it, and either running an automated installation script or dragging an icon to the Applications folder. You don't need to worry about security overrides; most applications will prompt you when you need to grant the installer systems access. And you always have the option of configuring, compiling, and installing the source code from the shell if you need.

That said, OSX isn't all wine and roses. I've had my tales of frustration: reinstalling Ruby and Rails on top of a broken configuration on Tiger, a rogue Java installation that corrupted Leopard altogether, requiring a clean reinstall, and sliding down the bleeding edge when I upgraded to Snow Leopard and ran though a gauntlet of obscure tasks to get a local MySQL instance working properly in Ruby. But the time spent on these issues was small compared to all the tweaking and retweeking I spent with my Ubuntu installations.

With all that said, it's time to challenge Giles' claim that, "Ubuntu is the rich man's OSX." He claims Ubuntu is cheaper because it's free and you can run it on less expensive hardware. But is it really less expensive in the long run when you consider the reduced productivity involved? When comparing my experiences between both OS'es, I spent an estimated 10-15 hours of extra time managing my workstation with each installation / upgrade of Ubuntu over OSX.

If you're a software engineering consultant making, say, $75 per hour, that amounts to as much as $1125, and that's on par with the added cost of buying a Mac Pro over a PC laptop. But the cost savings is even bigger if you're an employee and your company chooses a Mac over a cheap laptop and Ubuntu. When you consider the total value of a software engineer to an organization -- salary, payroll taxes, benefits, overhead, and opportunity cost -- that can amount to as much as $500 - 1000 per hour. Considering those factors, choosing Ubuntu over OSX for a development workstation becomes far more expensive in the long run.

Note that my most recent experiences with Ubuntu date back to Gutsy Gibbon around 2008. Certainly Ubuntu has made advances since then, and as one recent benchmark suggests, Ubuntu's actual performance is on par with OSX, with neither showing a clear advantage for all operations. The latest comparisons I found on the web tend to be biased, and therefore I wasn't able to locate a task speed test of any sort. That's a bummer, because I would like to include some emperical data here to back up my argument. If anyone knows of such an analysis, let me know.

Finally, I'm not here to claim that OSX is superior to Ubuntu. Both have advantages in certain areas. For production servers, the obvious choice is Ubuntu (or most flavors of Linux) for cost, scalability, and ease of setting up and replicating multiple instances on the cloud or in your server farm. Also, If you prefer to have more control over your graphical desktop environment (or eschew GUIs altogether), have specific hardware support needs, or want to develop in exotic flavors of C or C++, then Ubuntu is probably the better choice. But if, like myself, you want an easy to install and configure OS with seamless hardware integration and support, no-fuss backups, synchroniztion for you iPhone and iPad, and want to spend less time tinkering and more time getting shit done, OSX may actually be the more effective and economical choice. Not to mention avoiding sticking out like a sore thumb when I bring the laptop to Ruby and Rails conferences.

The Year of Programming Dangerously: Why View Testing Should Never Be Overlooked

2010-05-25T22:01:55-04:00

An interesting thread on LinkedIn's Ruby on Rails Group and the Ruby on Rails blog reminded me of the importance of testing the output from your controllers.

If you've been paying attention, you may have noticed the unusual number of Rails releases (2.3.6, 2.3.7. 2.3.8) deployed in rapid fashion recently. It turns out the reason behind this was a bug caused by back porting of some XSS protection features from the upcoming Rails 3. And it turns out that those bugs went undetected because of a lack of comprehensive view tests in a Rails module.

There are several ways you can approach testing the output from your application controllers, such as:

In your functional tests, using assert_select on the output (or alternatively, searching for patterns or substrings).
Writing RSpec tests that mock your output.
Using Webrat to examine output.
In Selenium or Watir, by searching and examining elements in the output.

What I find puzzling is that few developers, including myself, bother writing view tests at all. Yet the application output can often be the source of many common bugs that are not easily detected visually, such as form parameter name mismatches, incorrect field state settings, JavaScript typos from AJAX actions, and element visibility (or invisibility) issues.

So why is this? There are two possible factors that come to mind:

Lack of view testing examples in popular screencasts and blog posts that cover testing. I won't name names, but several tutorial screencasts that cover testing frameworks simply ignore or eschew view testing and don't offer examples. Because software developers learn by imitation, it lends to a culture that suggests that view testing is an esoteric facet that doesn't require much attention.
Cumbersome API's. The assert_select method, and XPath for that matter, are painful to use. I have a hard time testing output without having the assert_select cheat sheet in hand.

What Rails needs is a more abstract set of built-in APIs for examining HTML output in an abstract and object oriented manner, particularly for important interface points such as form field parameters and state, and style directives on elements, especially with respect to visibility.

And while testing your UI interactions using Watir or Selenium can sometimes cover the same area, it's important to note that running these tests tend to be slow. Functional or view tests, on the other hand, can be run repeatedly and quickly without waiting for the browser set-up. Same goes with Webrat.

If anyone has suggestions on gems or plugins to make view testing easier, I'd love to hear them.

I'm an old fashioned SQL database programmer

2010-04-14T19:28:28-04:00

There's been a lot of buzz growing in the blogosphere about NoSQL databases. As of today I count six sessions on the schedule for RailsConf 2010 that are directly or indirectly related to non-relational database storage engines. None of the planned sessions cover SQL exclusively (although two cover both realms to some extent).

Certainly there are some appealing features about NoSQL. Let's face it: SQL is an awkward language. Many relational concepts are difficult for an inexperienced programmer to grasp. What's the difference between WHERE and HAVING in an query? Do you know the syntax for constructing an outer join? Most RDBMS engines to this day treat data types in a rather primitive manner. And the biggest failure of SQL is its inability to handle hierarchical queries in an efficient manner. Try to fetch a subtree of objects with an arbitrary depth, for instance. And let's not get started with the differences in dialects between the various engines. I had to deal with that once when I ported a database application from SQL Server to Oracle, and had to write an RDBMS-agnostic query to obtain paged results from a table without using a cursor.

When it comes to writing web applications, there are certain problems that are best solved in a non-relational environment. Document storage, structured data storage, Cache lookup, queue management, and full text searches are several examples that can be best served by systems such as Memcached, Redis, Tokyo Cabinet, Lucene, or the new Cassandra. I've used them in quite a few Rails projects to manage these sorts of tasks. And when it comes to dealing with networks of data, such as connections for a social web site, you'll want to break out a dedicated engine to crunch that data continuously, as most of the big sites do.

But when it comes down to the nitty gritty of storing key business data for an application, and generating reports on that data, I always come back to good old SQL. Need to run an aggregation with grouping? Want to find documents with the most common tags to the document you're currently displaying? Need to enforce referential integrity in the event that a user record or a document is deleted from the database? It's hard to do that efficiently with non-relational databases.

Back in the 90's, I experimented with several object database and network-relational database solutions such as GemStone and ObjectStore. I was seduced with the idea of a native object storage or network object engine. I even convinced one of my employers to pluck down $25,000 for an ObjectStore license -- a very dear sum for a startup 12 years ago. But sadly, none of those systems ever rose to the popularity of Oracle or even DB2. And the core reason is this: you can't get the query performance from object, network, or key-value storage systems that you can get from an RDBMS. And maintaining referential integrity in a network database takes an incredible amount of overhead.

SQL, for all its faults, happens to be the one language that has remained constant throughout my career. And I'm very excited about the prospect of building my queries using the hot new ActiveRecord Query interface in Rails3. But remember that it's all still SQL under the hood. And understanding how SQL works will help you create more efficient queries.

The Importance of Face Time

2010-04-06T15:39:30-04:00

As I write this, I'm on my way to my company's Portland office. Even though we're living in the age of corporate frugality, my manager doesn't simply permit me to travel regularly between the San Francisco corporate headquarters and the Portland office, where most of the research and development occurs. In fact, it is encouraged.

There are several good reasons for this. The first, and most obvious reason, is that regularly interacting face-to-face with my remote associates builds a team culture and puts a personal face on the other end of the email, phone call, or chat session. But even more importantly, it fosters more spontaneous dialogue on technology and programming.

All the modern conveniences of free multimedia communication over the internet have failed to eliminate the inertia of communication. Having to pick up the phone, open a new email message, or even type in the chat window requires me to pause my train of thought. I have no idea what's happening on the other end of the line. What is the general mood of the office? Is my colleague busy and focused in a task? Will I even get an immediate response, or do I have to send a message and wait? And does my seemingly insignificant thought even worth the interruption?

Being in the same room with my coworkers eliminates many of those considerations and reduces the risk of interruption. If my teammate is busy, it's usually obvious. If they wish not to be interrupted, they'll have the headphones on, and I can gauge how important my communication is. But even more than that, the ability to have casual and spontaneous conversation sometimes reveals significant insights that make an impact on our collective task and sometimes even the overall mission.

The most successful companies have managers understand the fine balance between fiscal prudence and maximizing team productivity. That occasional trip to the field office may seem expensive on paper, but it can yield intangible benefits far more valuable.

Ruby, Rails and The Semantic Web

2010-03-29T10:35:30-04:00

This year, for the first time ever, I submitted a proposal to a conference, RailsConf 2010. In fact, I submitted three. This post is based on the motivation behind one of the proposals.

The Semantic Web

Let's look at three of the most successful web sites today: Google, Facebook, and Twitter. What makes them particularly popular is not just their excellent web interfaces, but the ease in which you can integrate their sites with third party applications and interfaces. Building a killer web site isn't good enough anymore: you need to also offer easy to integrate API's and web services to build a seamless flow of information between your site and others.

The World Wide Web has evolved magnificently over the past 15 years. You could characterize its development in three eras. [The years are approximate.]

Web 1.0 (1994 - 1999)

The web was in its infancy. Data sharing was difficult, expensive, and error-prone. [Recall how many broken images and 404 pages did you hit in an average day in these days.] Sharing data with another web site required either lots of cooperation, buying syndicated content, or hiring a programmer to write a Perl script to scrape data, and a messy data transformation to import the results. into the site. Portal sites such as Yahoo, Excite, and Go were king.

Web 2.0 (2000 - 2006)

The rise of XML, JavaScript, RSS, and SOAP made data sharing considerably easier and more uniformly. Web services makes it possible to create mash-up sites that combined data from multiple sources. Google ascended to greatness during this era with their mail, calendar, AdWords, and maps tools, just to name a few.

However, exploiting external services and tools into your site was still a time consuming process. The lack of standard conventions for defining services and data schemas required lots of parsing and transformation. SOAP required building custom clients to cooperate with transactions. While Java and .NET offered the requisite tools for developing these interfaces, the lack of a comprehensive framework made this development time consuming.

Web 3.0 (2007 - present)

Tim Berners-Lee envisioned a Semantic Web where content, links, and transactions can be analyzed and shared. This is now being realized with sophisticated, integrated applications such as Facebook and Twitter applications, federations of services. You can use one site to search for a cool event, buy and pay for tickets, add it to your electronic calendar, and share it with your friends, all from the same interface.

Key Technologies: REST and ActiveResource

Much of my recent work has involved integrating my company's application with partner sites and tools. REST and ActiveResource in particular have made this task inexpensive and straightforward. Now we have a standard means of not just encapsulating our data, but in representing and manipulating it through the concept of resources.

Taking advantage ActiveResource requires REST, and this is of course a cooperative process. By designing your web site to be RESTful not only allows another Rails developer to build an ActiveResource client to access your site, but it also allows developers in other development environments -- Java EE, .NET, PHP, and Python -- to build clients with minimal effort.

Why I'm not lining up to buy an iPad...yet

2010-03-30T09:10:34-04:00

I vividly recall that moment back in January when Steve Jobs single-handedly brought several web sites to their knees, and halted all activity in the workplace, with his announcement of Apple's latest gizmo, the iPad. I was just as enthusiastic about the upcoming device as anyone.

However, I'm not about to buy one, at least not right now.

Apple is making a bold move to return to a form factor with a long troubled history, especially for Apple. Remember the Newton? John Sculley's pet project of the last millennium was supposed to revolutionize hand held computing. It didn't work out for several key reasons:

It pushed way beyond the technology barrier at the time. The handwriting recognition software was buggy. The CPU's speed was too slow. The OS crashed frequently.
Synchronization, especially with a PC, was balky at best, frustrating at worst.
Network connectivity was limited and hung frequently.
Software incompatibilities made it difficult to use as a replacement for a regular computer or PDA.
The form factor had limited practicality, except in specific circumstances. Doctors loved it, but few others did.
It was expensive.
It lacked multitasking ability.

Now that Apple has finally gotten on the Tablet Computing 2.0 bandwagon, we expect things to be different. And certainly technology has come a long way since the days of the monochrome LCD. But can we expect the iPad to really succeed where the Newton and many other pad computers didn't?

One triumph is synchronization. With the MobileMe service and iTunes, synchronization between my iPhone and my laptop is a breeze. I expect the iPad to work just as well.

Apple was smart to abandon the stylus and handwriting recognition in favor of the touch screen. And I admit that I do pretty well typing on my iPhone, thanks to the smart word recognition software built into the OS. But it's still far from perfect, and it often corrects mistakes I didn't make. So I'd probably want to have a wireless Bluetooth keyboard handy for whenever I wanted to do serious typing.

Network connectivity, as is the case with the iPhone, should be a snap. However, without a WiFi network, I'd still be suffering with AT&T's abysmal reliability and performance, especially in network-saturated cities such as San Francisco.

Let's look at the last four reasons that the Newton failed. iPad doesn't make much headway with these. Where's the Flash player, for instance? Can you write software programs on the device itself? What about using an alternative web browser such as Firefox or Opera? Can you edit Word documents on it?

The form factor is also problematic. It's still too big for the pocket, so I'm not going to take it anywhere I wouldn't take my laptop. At least it weighs a lot less.

Battery life is a new problem. With the Newton, the rechargeable (and replaceable!) battery packs would last a week or more. With the iPad, expect to recharge the unit daily, or even more often than that. So I'm not about to replace my Kindle with an iPad for reading books anytime soon.

While the iPad is less expensive than the Newton MP2000, it's still quite pricey. The cheapest version, at $499, has too little memory to be practical, and no 3G wireless connectivity. I expect most folks will shell out at least $600 for the fatter versions, or wait until the end of April to pick up the 3G version.

And the last item on the list — lack of multitasking ability — is the show stopper for me. My number one frustration on my iPhone is my inability to switch between applications without the risk of losing work. You're at the mercy of the app designer whenever you leave the app in the middle of a task. Some apps behave the way you expect, but others do not.

So let's root for the initial wave of iPad sales to spell success for Apple. Let's hope that they solve this multitasking problem once and for all. And most of all, let's hope that Apple lowers the price this fall.

The slow adoption pace of Ruby 1.9

2010-03-29T09:44:23-04:00

The latest State of the Stack Report, published on RailsLab, shows one curious trend: while adoption of Ruby 1.9 continues at a slow rate, many sites have been quick to update Rails itself. Only 1% of sites reporting use Ruby 1.9.1, despite its being out in the market for over a year. Contrast that with Rails 2.3.5, released just last November, with a nearly 14% adoption rate, and Rails 2.3.4, released in September, having a commanding 32.6% adoption rate.

Ruby 1.9 has some very compelling features. Here are a few of my favorites (complete list here):

Nicer Enumerable syntax reducing the need for blocks.
Block variables behaving as true locals.
Full Unicode support.
Order preservation of hashes.
Non-blocking I/O.
Native threads.
YARV: compiling ruby codes into opcodes

So how come there's no stampede? One obvious reason is gem dependencies. Many developers are reporting problems with some popular gems, although the number of ports and success stories is increasing.

Another reason is support, and this may have as much impact, is hosting provider support. Fortunately, there is some progress here as well. heroku now has 1.9.1 available in their newest stack. Engine Yard is planning to support Ruby 1.9.2 later this year, which will include some string encoding conversion fixes.

It's about time

2010-03-28T19:21:24-04:00

I've been planning to start a blog for some time now, but always put it off for one excuse or another. Being too busy with work, too pre-occupied with home life, whatever. Well, it's about time that I create a home for all the contributions I've been making on various sites. Over the next few days, You'll be seeing many articles reprising topics that I've covered recently, just to get the site going and presenting some interesting content for you to read. Stay tuned.