Aaron Dunnington

Array-Based Queue on Aerospike

2015-03-21T11:00:00+00:00

Aerospike is a high-performance, horizontally scalable key-value store. Running 1M transactions per second without breaking a sweat, this NoSQL database is delivering 10-15x price-performance gains. Notably, through flash optimizations, RAM-like performance can be achieved using local SSDs. Further, Aerospike operates as an AP data store.

Similar to Redis, Aerospike provides built-in support for complex data types such as lists and maps. If you a migrating a Redis based application for nice performance rewards, you can conveniently use a complete Redis veneer. Note, when storing built-in Aerospike data structures in records, you are bound by the typical record size limitation of 1MB. The Large Data Type (LDT) Lua extensions, further, provide support for ordered lists, sets, stacks, and maps. Using sub-records, LDTs may grow freely, only limited by the underlying storage.

Through User Defined Function (UDF) modules, data structures can be managed directly within Aerospike on the server. The Redis veneer above provides a great example of this record management from which this UDF module is based on. To enable runtime characteristics similar to that of a memory resident linked list in Redis, you can implement efficient queuing using two array list buffers. Let's begin with the enqueue User Defined Function (UDF) in Lua. NOTE: For a more scalable approach with similar runtime properties, please see the aerospike-LDT-techniques repo.

function enqueue(rec, bin, value)
    local q = rec[bin]
    if q == nil then
        q = map { size = 0, pos = 0 }
    end

    if q.rear == nil then
        q.rear = list()
    end
    list.append(q.rear, value)
    q.size = q.size + 1

    rec[bin] = q
    l_update(rec)
    return q.size
end

The UDF above participates in the main flow of the transaction. This method sets up a map to manage the queue data structure if it does not already exist. Subsequently, it appends the incoming value to a rear array list buffer before updating the record with the call to the local helper function l_update. Next, let's take a look at the work of our dequeuing function which manages our second buffer.

if q.front == nil then
    q.front = q.rear
    q.pos = 1
    map.remove(q, "rear")
end

local item = nil
local last = list.size(q.front)

if last == 1 then
    item = q.front[1]
    map.remove(q, "front")
else
    if q.pos < last then
        item = q.front[q.pos]
        q.front[q.pos] = q.front[last]
        q.pos = q.pos + 1
    else
        item = q.front[last]
    end
    list.remove(q.front, last)
end
q.size = q.size - 1
return item

The dequeuing logic above manages the front buffer of the queue. If no front buffer exists, it will move the rear buffer to the front. As we are using array lists for our buffers, it would be a linear operation to dequeue items from the beginning of the list since all subsequent items need to be shifted left. By maintaining a position in our queue, we can dequeue more efficiently. As items are dequeued, we iteratively reverse the list by copying the item currently in the last position into the old index of the item being dequeued. Once we hit the end of the list, we dequeue from the last position until the front buffer is empty.

The complete queue.lua UDF module is available here. To install, use the udf-put command below.

ascli udf-put queue.lua

To work with the queue through the aql console, it can be called with the execute command.

aql> execute queue.enqueue('Q', 99) on demo.queues where pk='q'
+---------+
| enqueue |
+---------+
| 1       |
+---------+

aql> execute queue.dequeue('Q') on demo.queues where pk='q'
+---------+
| dequeue |
+---------+
| 99      |
+---------+

Using two buffers, we can achieve runtime characteristics similar to that of a linked list. As this queue example stores the data structure within a single Aerospike record, it is bound by the aforementioned record storage limits.

References

Thanks to Peter Milne for creating the excellent Redis veneer article which this UDF module is based on. Additionally, please see Peter's aerospike-LDT-techniques repo for FIFO utilizing LDTs. As opposed to the in-record storage limitations discussed above, the LDT approach scales queueing by utilizing a LLIST as a map with counters positioned at the head and tail.

HTTP Request Context in Golang

2015-01-07T11:00:00+00:00

The Go HTTP stack offers a very efficient web server implementation. As the Request object passed through to Handlers does not support memory resident storage itself, context specific to the lifetime of each request needs to be managed outside net/http.

Go web frameworks such as Goji, Beego, and Revel have built-in support for request context. Further, the Gorilla context package provides support for global request variables, though it relies on mutex to guard access to request state. If you are running directly on net/http, you can also inject a context object at the beginning of the request lifecycle which flows through the handler chain. By passing context through, rather than relying a global context object, state specific to each request can be managed without introducing locks.

type Handler func(http.ResponseWriter, *http.Request, *Context)

func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    ctx := &Context{
        Config: Config,
        DB:     DB,
        Items:  map[string]interface{}{},
    }
    h(w, r, ctx)
}

To build a chain of Handlers which accept context, the creation of the custom Handler type above can be wrapped. The outer Handler returned will contain a reference to the Handler chain passed in. Upon execution by the networking stack, the outer Handler's ServeHTTP method will be invoked, creating the custom Context object before forwarding to the entire handler chain.

func Csrf(h Handler) Handler {
    return Handler(
        func(w http.ResponseWriter, r *http.Request, ctx *Context) {

            // Your super secure CSRF logic here...

            ctx.Items["csrf"] = token
            h(w, r, ctx)
        })
}

As Handler implements the ServeHTTP method, the first handler instance in the chain can be registered with a given pattern on your router.

func home(w http.ResponseWriter, r *http.Request, ctx *Context) {
    Render("home.tmpl", ctx.Items["csrf"])
}
router.Handle("/", Csrf(Handler(home)))

At runtime, the Csrf handler executes first, calling ServeHTTP to setup the request environment. Next, the Csrf handler itself is called which sets a token within the Context. Finally, the home Handler is invoked wherein request context state is made available.

Autoscale Resque Workers on Heroku

2011-11-27T11:00:00+00:00

A common technique for autoscaling Resque workers on Heroku uses the after_enqueue and after_perform hooks that run within each Resque worker to adjust the current worker scale up and down, respectively. The after_perform hook watches the pending job count. When no jobs remain, this hook will scale all workers down.

When scaling workers up, the Heroku API is additive; thus, additional workers can be scaled while others are actively processing jobs. As the Heroku API does not currently allow individually specified workers to scale down, however, it is necessary to wait until all workers are inactive before performing a scale down operation.

While the after_perform scale down approach mitigates against shutting an active worker down while pending jobs exist, a race condition can occur where the pending job count hits zero, and a scale down operation is invoked on the Heroku API prior to a subsequent job being enqueued from the app and reserved on a worker which will be shutdown in the current scale operation.

Recently, I have been working on the resque-heroku-scaler gem which autoscales Resque workers through a separate monitor process. The scaler monitor process polls for pending jobs against the specified Resque Redis backend. When scaling down, this scaler process locks each worker from attempting to reserve any future pending jobs before a scale down operation is performed.

To get started, include the scaler tasks in lib/tasks/scaler.rake

require 'resque/tasks'
require 'resque/plugins/heroku_scaler/tasks'

task "resque:setup" => :environment

In your Procfile, configure the scaler process using:

scaler: bundle exec rake resque:heroku_scaler

To run the scaler process, use the following command.

heroku scale scaler=1

Require the worker extensions within the app running your workers. For example, in lib/tasks/resque.rake. These extensions to the worker run loop poll for a shared flag within the Redis backend set by the scaler process which indicates the worker should enter an inactive scaling state. After a worker has entered this scaling state, it will cease attempting to check for incoming pending jobs. Once all workers have indicated they are inactive, the scaler process issues the scale down operation using the Heroku API.

require 'resque/tasks'

task "resque:setup" => :environment do
    require 'resque-heroku-scaler'
    ENV['QUEUE'] = '*'
end

In your development environment, the scaler process can run local worker processes using the rush library. To configure, use the following in your scaler rake file.

require 'resque/tasks'
require 'resque/plugins/heroku_scaler/tasks'

task "resque:setup" => :environment do
    if Rails.env.development?
        require 'resque-heroku-scaler'
        ENV["RUSH_PATH"] ||= File.expand_path('/path/to/app', __FILE__)
        Resque::Plugins::HerokuScaler.configure do |c|
            c.scale_manager = :local
        end
    end
end

Autoscale Delayed Job Workers on Heroku Cedar

2011-07-23T11:00:00+00:00

Heroku provides an excellent platform to scale your Ruby or Node.js application through the cloud. Built on top of Amazon EC2, Heroku alleviates much of the server administration work typically associated with scaling applications out as user demand increases. Heroku is PaaS; Salesforce and Matz agree.

On the new Cedar stack, scaling your app’s web and background processes across N machines can be achieved using the scale command. While explicit scale commands work well for web processes as traffic increases, background workers tend to have a lot of free time if not managed automatically, running $36 a month per worker.

Several libraries currently autoscale background workers. For Resque based workers, Daniel Huckstep provides a cool approach to scale based on load. For Delayed Job, Heroku’s Pedro Belo has provided a DJ fork which supports autoscaling on Bamboo and Aspen. Additional DJ autoscale libraries include HireFire and Workless.

While I would prefer to use Github's Resque for performance and manageability, to avoid an additional, potentially paid dependency on Redis at this stage, I am currently using Pedro's Delayed Job autoscaling fork with a few changes to support Cedar.

This implementation will autoscale a single worker up and down as background requests are generated by the application. Job retries are not currently supported. To use, create an initializer similar to below. In your local development environment, the rush library provides process management.

Delayed::Worker.auto_scale = true
Delayed::Worker.max_attempts = 1

if Rails.env.production?
    Delayed::Worker.auto_scale_manager = :cedar
else
    Delayed::Worker.auto_scale_manager = :local
end

To schedule a custom background job or delay send from a mailer, respectively:

Delayed::Job.enqueue YourAwesomeJob.new
UserMailer.delay.welcome_email(context)

In your Procfile, add:

worker: bundle exec rake jobs:work

Efficiently autoscaling your background workers eliminates spending for idle dynos. For a background load of 20 hours per month, you’d pay $1 autoscaling compared to $36 for an idle background worker running the entire month.

Recurring Billing with Braintree's Transparent Redirect

2011-07-18T11:00:00+00:00

Over the past several years, PCI DSS has prompted quite a bit of change on the web in credit card processing. There is much to be read on the topic; though, in short, since 2004 a consortium of credit card companies have established the governing standard for how merchants must deal with credit cards online. Achieving compliance, however, can be quite costly.

Enter Braintree, a service based payment solution providing a merchant account, your processing gateway, and recurring billing on a single platform. By externalizing the credit card surface area within your application, Braintree’s unique Transparent Redirect technology allows you to preserve the sales experience within the context of your site while alleviating the hurdles associated with achieving PCI compliance.

In concert with the Braintree's Vault, Transparent Redirect enables developers to productively integrate recurring subscription billing. To remotely create customers within the vault using transparent redirect, call the create customer data method to embed the redirect data within the form as a hidden field.

class PaymentController < ApplicationController
  def signup
    @tr_data = Braintree::TransparentRedirect.create_customer_data(
      :redirect_url => "http://example.com/payment/confirm"
    )
  end
end

This data field will include the callback location to your confirmation action to be invoked after sensitive credit card data has been externalized to Braintree in the initial remote form post.

<%= form_tag Braintree::TransparentRedirect.url, :method => :post do %>
  <%= hidden_field_tag "tr_data", @tr_data %>
  <%= text_field_tag "customer[credit_card][number]" %>
  <%= text_field_tag "customer[credit_card][expiration_date]" %>
<% end %>

In your confirmation callback action, the customer record can be confirmed using the query string parameter below. The result will contain the identifier tokens which represent the remote customer and credit card, respectively.

result = Braintree::TransparentRedirect.confirm(request.query_string)
customer_id = result.customer.id
payment_token = result.customer.credit_cards[0].token

Lastly, to establish a recurring subscription associated with the remote credit card, use the payment token along with the selected plan.

result = Braintree::Subscription.create(
  :payment_method_token => payment_token,
  :plan_id => "super_awesome_plan"
)

Braintree's innovative approach to externalizing credit card information through transparent redirection takes the pain out of PCI compliance while preserving the user experience within the context of your site. Braintree is paving the way in today’s demanding world of web based credit card processing. Use them.

Asynchronous MongoDB in Tornado with asyncmongo

2010-12-05T11:00:00+00:00

EDIT: This blog now runs on Golang

This blog runs on the Tornado web framework. Tornado is one of a handful of C10K web servers which scale concurrent requests through a single threaded, event-driven asynchronous architecture. By using a single thread to manage multiple requests, a single C10K server can scale to thousands of concurrent requests.

One thread for the entire web server, doesn't that block other requests when hitting the database you might ask? Absolutely it does if you're using blocking style I/O. Tornado differs from conventional web servers which use multiple threads to handle concurrent requests. When using traditional threaded web servers, each blocking call to the database or a remote service will be isolated in a separate thread to unblock execution of other requests. While multi-threaded servers provide a very clean programming model to handle concurrency, the overhead of managing multiple threads can pose challenges when C10K scale is required. Over the past decade, this has led many large web offerings to take a bet on event-driven servers. Some notable sites using C10K servers include YouTube, Wikipedia, Wordpress, FriendFeed, and bit.ly.

So, what's the catch? While the single-threaded nature of event-driven servers is great for scaling concurrency, additional care is required in the mid-tier to preserve the non-blocking programming style. For remote web service calls, Tornado includes a non-blocking HTTP client.

While this is great for HTTP based services, most database access in Tornado has typically been blocking. FriendFeed uses a blocking MySQL client under the assumption that if database queries were causing concurrency issues in the mid-tier, the database servers would not be able to scale to the corresponding load. FriendFeed also uses nginx as a load balancer in front of Tornado which helps to distribute any momentary blocking calls to the database amongst several frontend machines.

Bummer, all that C10K perf only to block on every call to the database? Not a problem anymore for MongoDB users. Bit.ly recently contributed their work on asyncmongo to the community which provides non-blocking access to mongo over the PyMongo API.

import asyncmongo
import tornado.web

class Handler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def get(self):
        db = asyncmongo.Client(pool_id='mypool', host='localhost', 
            port=27107, dbname='mydb')

        db.users.find_one({'username': self.current_user}, 
            callback=self._on_response)

    def _on_response(self, response, error):
        if error:
            raise tornado.web.HTTPError(500)
        self.render('template', first_name=response['first_name'])

Bit.ly's asyncmongo fills a huge gap in data access for Tornado. This client driver preserves the non-blocking programming style required by event-driven architecture in the mid-tier to a major NoSQL player focused on scaling big data. Looking forward to seeing other client libraries follow suite. Pycassa users have been calling for it.

Data-Driven iPhone Apps with Property Lists and PHP

2009-12-16T11:00:00+00:00

When driving iPhone apps with data from the cloud, the device's underlying connection speed is often a potential performance bottleneck. As devices consume increasingly larger sets of data over the air, an app's ability to provide a smooth experience can be constrained. Thus, when selecting payload formats or compression techniques, service aware mobile apps can really benefit from a healthy respect for just how much an EDGE or 3G connection can hose your transfer rates.

One way to add a performance optimization to the payload format consumed by data driven iPhone apps is through Property Lists. Property Lists provide a great mechanism for iPhone apps to transparently marshal data in and out of an app using either XML or binary formats.

As great as XML is for exposing data to a wide range of clients, it is verbose by design, and often bloats wire payloads in favor of interoperability. If you can afford to introduce a little coupling of your mobile app to your service, the binary serialization format of Property Lists may potentially boost perf 5 to 20 times over text based Property Lists. Though, even the XML based Property List schema arguably would add some coupling to the Mac platform. On the other hand, if you'd rather give up the seamless interaction the Property List APIs have with native data types on the iPhone, it would be interesting to see how a minimalistic POX based format performs against binary Property Lists.

Nevertheless, there are a number of options for exposing Property Lists on the server depending on your language. I've seen Property List implementations for Python, Perl, and PHP. If you're running OSX on the server, of course both Cocoa and Core Foundation offer Property List APIs; or, you could run plutil from the shell. Lastly, there's always parsing the raw bytes if that's more your bag. Though, unless you really want to understand Apple's binary serialization format for Property Lists by parsing it yourself, the Cocoa, Core Foundation, plutil, PHP, and Perl implementations are the options I've seen thus far that offer binary support.

The CFPropertyList project provides a Property List implementation for PHP. Here's a simple Zend controller that exposes a list of entries using CFPropertyList.

<?php

require_once 'CFPropertyList/CFPropertyList.php';
require_once 'models/Entry.php';

class BlogController extends Zend_Controller_Action
{
    public function init()
    {
        $this->_helper->viewRenderer->setNoRender();
        $this->_helper->layout->disableLayout();
    }

    public function indexAction()
    {
        $table = new Entry();

        $select = $table->select();
        $rows = $table->fetchAll($select);

        $plist = $this->createPropertyList($rows);

        $this->getResponse()->setHttpResponseCode(200);
        $this->getResponse()->setHeader('Content-Type', 
            'application/octet-stream', true);
        $this->getResponse()->setBody($plist->toBinary());
    }
}

Here, we grab some entries from the database, then call createPropertyList (which we'll write next) with the results, and lastly serialize the response in the binary format by calling the toBinary method on the returned plist object. Now, let's generate the property list from our data set below.

private function createPropertyList($rows)
{
    $entries = new CFArray();
    foreach ($rows as $row)
    {
        $entry = new CFDictionary();
        $entry->add('id', new CFNumber($row->entry_id));
        $entry->add('title', new CFString($row->title));
        $entry->add('author', new CFString($row->author));
        $entry->add('summary', new CFString($row->summary));
        $entries->add($entry);
    }

    $d = new CFDictionary();
    $d->add('feed', $entries);

    $plist = new CFPropertyList();
    $plist->add($d);

    return $plist;
}

This method iterates over the result set while creating corresponding Property List structures. The CFPropertyList PHP library provides type wrappers for the corresponding property list data type, for example, CFDictionary, CFArray, CFNumber, and CFString in the above snippet. This function produces a Property List with a top level CFDictionary that has one item named "feed". The value of the "feed" item is a CFArray which contains a CFDictionary object for each entry.

Then, on the client, here's a quick look at how to consume the property list based service. Here, we make a remote call to the service, and turn the resulting NSData into a Property List to print out the title of each entry. The remote call in this code blocks with the invocation of sendSynchronousRequest:returningResponse:error: for the sake of brevity; though, in practice, processing the request asynchronously with NSURLConnection may be more appropriate.

NSURL *url = [NSURL URLWithString:@"http://localhost/blog"];
NSURLRequest *request = [NSURLRequest requestWithURL:url];

NSData *data = [NSURLConnection 
                    sendSynchronousRequest:request
                    returningResponse:&response
                    error:&error];

id pList = [NSPropertyListSerialization
                propertyListFromData:data
                mutabilityOption:NSPropertyListImmutable
                format:&format
                errorDescription:&errorStr];

NSDictionary *d = (NSDictionary *)pList;
NSArray *entries = (NSArray *)[d objectForKey:@"feed"];

for (NSDictionary *entry in entries) {
    NSLog(@"Title: %@", [entry objectForKey:@"title"]);
}

CFPropertyList's baked in support for binary is cool. It provides a great way for PHP services to shrink the wire payloads compared to text based formats in order to serve data hungry iPhone apps over constrained connection speeds. Given the growth in machine readable formats across service offerings over the past couple years, coupled with the huge demand for data centric iPhone apps, it'd be interesting to see how common the various hydration approaches are in the market today.

AJAX: Great Stuff, Why Now?

2005-09-04T11:00:00+00:00

Those into web applications have no doubt heard the buzz of late on Asynchronous JavaScript and XML (AJAX), an approach to web development that focuses on harnessing the existing client side processing capabilities of modern browsers. In short, a larger portion of web application code, typically JavaScript, can be downloaded to and executed on the client machine. Further, the client code leverages the XMLHttpRequest object built in to most browser engines to communicate with the server in the background using XML and HTTP.

The most compelling reason for using the AJAX technique is that it enables a more fluid user experience. In other words, the user is not constantly interrupted to jump to a new page. Thus, with the understanding that client processing is far less expensive than network communication, the AJAX approach allows web applications to behave more like desktop applications.

As more and more code moves from server to client, however, intellectual property vulnerabilities may become an issue. Of course, this is very specific to the organization responsible for the application. If the organization is driving an open source effort or primarily focused on providing a service that is coupled to hosting infrastructure, exposing the client application may not be as much of an issue. On the other hand, if the corporate advantage resides in the client application, depending on customer audience, intellectual property exposures should be evaluated. Preventative measures can be taken through JavaScript obfuscation and potentially the future development of JavaScript security standards. Further, for very large applications, the amount of code downloaded to the client machine may pose interesting performance challenges. Could we use AJAX precepts to download more AJAX precepts?

Overall, I am very excited to see the improvement of the user experience in web applications. One parting question: what has changed recently to cause the flurry of AJAX activity? Was it the acronym itself? These technologies have been around since 1998. Why haven't application infrastructures and tools been maturing over the years? Possibly more importantly, is there a technology out there now that will raise similar questions seven years from now?

Trailblazing Application Frameworks

2005-06-16T11:00:00+00:00

Developer productivity, or said another way, budget, is usually of paramount concern in the majority of business application projects. There are several application frameworks addressing this ubiquitous need for an increase in productivity in innovative ways. These frameworks, moreover, have been gaining considerable momentum and popularity. Ruby on Rails, Spring Framework (as well as Castle), and Zope are a few such projects that I am very excited about.

To start us out, Ruby on Rails, an open-source web application framework, unifies web development with a single language and particular assumptions about application architecture as well as coordination with an underlying model en leu of a proprietary architecture and verbose configuration files. As a result, Ruby on Rails drastically reduces coding time to create applications that process and move data around. Okay, at a this level, sounds like a magic bullet. Curt Hibbs has written a great introduction to Ruby on Rails that depicts just exactly how Ruby on Rails jives. Basically, by using certain naming conventions from relational model to object model (although this can be circumvented at the expense of a little more time), Ruby on Rails is able to automate persistence, object relationships, basic request processing, presentation, and validation. Lastly, Ruby is an interpreted, cross-platform language and the Ruby on Rails framework provides database drivers for the popular RDBMSs. Thus, Ruby on Rails provides an extremely productive programming environment that alleviates the developer from the minutia while staying decoupled from any particular platform.

Next, the Spring Framework and Castle project provide light-weight, container-based frameworks that rely on Inversion of Control (IoC) for Java and .NET, respectively. These two frameworks are very similar in their application of dependency injection and Aspect Oriented Programming (AOP). These concepts facilitate the testability and API neutrality of our code as well as allow us to flip switches for security, transactions, and the like. Furthermore, they are alike in that they both offer basic Model-View-Controller (MVC) get-ups that enable us to plug in different view components like Velocity and NVelocity. Spring and Castle both even allow the usage of different persistence engines, for example, Hibernate and NHibernate. Where these two outfits differ, however, is in their assumptions about domain model. This is where I believe that Castle shines. Whereas Spring relies on lengthy XML configuration files to direct Hibernate, Castle's NHibernate provider is able to reflect upon the object model and automatically generate persistence configuration. This is a huge time saver. While tinkering with Castle, I sicked the ActiveRecord Generator on a database of approximately 200 tables and in a matter of minutes, I had a comprehensive, persistent object model. The code-reduction was quite impressive. All in all, these frameworks are very comparable, Castle leaning more towards a Ruby on Rails paradigm of assumptions. Of course, Spring and Castle are rooted in static languages; consequently, these frameworks are not quite as flexible and dynamic as Ruby on Rails. Yet, they have gotten quite close with reflection, AOP, and transparent proxy classes.

Last, we have Zope, a full-fledged application server written in Python. Unlike the previous frameworks, Zope relies heavily on an internal object database, or ODBMS. Several years ago, object databases were predicted to make a big splash; however, many would say that they never fulfilled this expectation. Well, Zope begs to differ. Python, like Ruby, is an interpreted, object-oriented language. This allows applications written towards the Zope framework to benefit from all the productivity gains of these pragmatic, dynamic approaches wherein the Linux Apache MySQL PHP/Perl/Python (LAMP) camp is very popular as well. Another effort that contributes to Zope's promise is ERP5, an open-source enterprise framework built on top of Zope. The "5" in ERP5 stems from the ERP5 Universal Business Model which, at an abstract level describes business entities as Paths, Nodes, Movements, Items, and Resources. What has me so excited about the combination of these projects is ERP5's design and usage of Zope's foundation. Dr. Jean-Paul Smets, CEO of Nexedi, wrote a great introduction on ERP5 and the interplay with Zope where he describes how they mitigated typical performance problems in the object-database world by indexing objects in a traditional relational database. The Zope application framework combines the pragmatic approach of Python with the elegance of an object database, and when used in concert with a relational database as an indexing facility, is extremely powerful. Performance of relational databases with no impedance mismatch. Now, that is pretty cool.

To sum it up, Ruby on Rails, Spring, Castle, and Zope are all exploring new avenues for application development. Each with their unique approach to increasing developer productivity, I believe these frameworks are cultivating the foundation for the next generation of web applications. I would like to thank all that have contributed to these projects; I look forward to seeing how each project affects the way that we develop web applications.

Web Services: Interoperable or Easy to Develop?

2005-06-14T11:00:00+00:00

The World Wide Web Consortium (W3C) defines Web Services as a "standard means of interoperating between different software applications." Well, that is pretty cool. If only we, the industry, could do that.

Some really bright guys at HP, Steve Loughran and Edmund Smith, recently authored an article entitled "Rethinking the Java SOAP Stack." Basically, they suggest, with quite sound evidence, that the current Web service stacks have failed to realize what Web services set out to do in the first place, that is, interoperate. Many current Web service stacks (emphasis on SOAP) take this great idea of Service Oriented Architecture and bastardize it by applying classic middleware approaches.

Steve and Edmund do a great job at depicting how we are currently experiencing an impedance mismatch from message (documents) to objects much like we see in the relational database to object mapping world. In short, these really great, flexible technologies like XML and XML Schema are shoved onto OOP. Since XSD and WSDL are at the very least mildly unpleasant to write by hand, the Web service stacks try to help us by making assumptions about document to object mapping. As a result, we as programmers are spared the details of schema and WSDL; however, the framework just serialized a definition of some platform specific data type:-) Now, that won't be very pleasant in another world.

So, Steve and Edmund accurately point out that what we are left with is a brittle, constrictive approach (much like RMI, Remoting, etc) to some great open, interoperable standards. How did that happen?

Enter Alpine. The HP crew suggest a lightweight SOAP stack that is only a SOAP stack. By delegating the mapping to the user application, the framework imposes no blanket assumptions about mapping. The caveat, we have more work to do with XML. But, at least, in theory, Alpine will get closer to that interoperable thing that Web services are supposed to do.

Okay, so now some questions:

Why are objects so hard to map to? Impedance. Are our modern-day programming languages in need of some rethinking (in particular, message documents and relational models sort of throw a wrench into OOP when a mapping paradigm is used)? Are SOA concepts creeping into OOP? How could we express SOA better in a high-level language like Java or C#? I mean annotations (and attributes) are great; but, are they really the answer to making services easier to deal with? Personally, I think Steve and Edmund are right; we should suck it up and do contract-first development and work with message objects until there is a better form of expression for services. But, what and when will that form of expression be?

Thanks Steve and Edmund; I look forward to hearing more about Alpine!