It feels as though, over the past couple of years, the pace at which I have been learning has slowed. This troubles me. One of the reasons I love the software industry is that there is always something new to learn. I love learning about new ways to solve problems with technology. And, learning is what you must continue to do if you are to stay relevant as a software engineer.

Now, this doesn’t mean that I haven’t learned anything over the past couple of years. Quite the contrary actually. I’ve really been enjoying the boom our industry is seeing in non relational data storage systems. We’ve been utilizing many of these newer technologies at Signal with great success. I’ve been an active member of the ChicagoDB users group, attended a few conferences on this topic, and even gave a few talks…a first for me. I’ve also been learning first hand how to scale a system as we’ve taken the Signal application from one capable of sending thousands of messages per “blast” to one that regularly sends millions of messages per “blast”.

However, something is missing. I used to read more. I used to blog more. I used make a sincere effort to learn a new programming language every year. I used to constantly download and tinker with new tools. These things have fallen to the wayside. And, I really miss them.

Instead, over the past couple of years, I have been spending the majority of my time creating. When creating a new project, there is usually something new to learn. That’s what drives me to create the project in the first place. But as time goes on, the cost of the time I spend working on the project starts to outweigh the benefit. I reach a point where I am no longer learning anything while working on the project. If I’m not benefiting from working on the project in any way (now that the learning part has mostly dried up), there is little motivation for me to keep working on it.

Part of me feels obligated to keep these projects going, and to get them to a point that I would consider “done”. I’ve never been a quitter, and stopping work on these projects sounds like quitting to me. However, it makes little sense to continue working on something that gives me nothing in return. I will still fix bugs that are reported, but any new development on most of my active projects is likely going to cease. Time is precious, and I need to make the most of it.

So as of right now, I’m making an adjustment in priorities. I’m using my free time to invest in myself. To read more. To blog more. To get back to learning a new programming language a year (Clojure is on tap for 2012…I’ve been wanting to learn it for 3 years now). To resume downloading and tinkering with tools that spark my curiosity. I want to start practicing how to write code, like I learned at the code retreat I attended over the summer.

I’ll still be creating, as I’m someone who learns by doing, but I have a feeling from now on my projects will be much, much smaller.

It is impossible to “find” time to do something. You have to make time.

One Monday morning about a month ago, I was browsing through open issues in our bug tracker looking for something to work on. It was my week on “technical support”. At Signal, the engineer on technical support spends a week working on non customer facing issues with our infrastructure, process, etc, that are too large to simply knock out as you encounter them.

An issue entitled “Add monitoring to detect jobs that do not start” caught my attention. We already had monitoring in place to alert us when a job fails. However, we had nothing to alert us if that job never starts in the first place. We have many scheduled jobs that run in the background, some pretty important, and have been bitten on more than one occasion by reoccurring jobs that had not run for quite some time.

I bounced some ideas off the team, and we all agreed that that it made sense to create a new application to monitor our scheduled tasks. The app would know what is supposed to run, and when. We would change all of our jobs to “ping” the app when they start and finish. If the app detects that something did not run when it was supposed to, or did not finish when it was supposed to, it could alert the team.

And just like that, Proby was born.

“What’s with the name?”, you ask? I likened the application to a probation officer, keeping an eye on wayward tasks.

Since going live, Proby has caught several issues with our scheduled jobs that would have gone unnoticed…many more than I would have anticipated. It has also evolved into a fairly complete application. It is easy to setup new tasks. It holds onto the execution history of each task, allowing you to see a trend of the run time for each task, and which runs resulted in an alarm. It supports alarms via email or SMS, using Signal’s SMS messaging API. It has several settings that allow you to tweak exactly when an alarm is sent, cutting down on the number of false alarms. And, thanks to designer extraordinaire Drew Myler, it looks amazing!

Shortly after creating Proby, we realized that there is no way we are the only ones who have had scheduled tasks go days without running, unnoticed. So today, we are making Proby available to the general public. It is currently in closed beta, and we are letting people in slowly to flush out any bugs or other issues. Use of Proby is also FREE while it is in beta.

Interested? Head on over to http://probyapp.com/signup and signup to participate in the closed beta! And, don’t forget to follow @probyapp on Twitter for updates!

What the hell is a code retreat?

For those who have never attended a code retreat, the format is simple. The group is given a problem. In our case the problem was Conway’s Game of Life. The code retreat is then broken up into several 45 minute sessions. Each session you pair with somebody new, and you take a stab at solving some portion of the problem. All code is written using TDD. At the end of each session you delete all of the code you created. The problem, and the time limit on the sessions, are specifically chosen so that there is little to no chance that you will come up with a complete solution in the allowed time. Solving the problem is not the point. The point is to practice.

We as software developers don’t give ourselves enough time to practice. At work, or even with open source or side projects, we’re always trying to get something done. Rarely, if at all, do we practice solely for the sake of practice. This is what we focused on at the code retreat. At the end of each 45 minute session, we deleted our code, and discussed what we learned. We were encouraged to try new ideas, to really push ourselves outside of our comfort zone. Knowing that the code would be deleted in a matter of minutes, there was no need to worry about screwing up the design, or creating something that would ultimately become unmaintainable. We had a safety net, and were encouraged to use it.

Corey said that we never get the chance to write “perfect code”. Perfect code doesn’t exist in the real world. In the real world products need to ship. It is simply not practical to spend the amount of time on a piece of code that is required to make it “perfect”. There are always trade offs of one form or another. However, code retreats give you the opportunity to practice writing perfect code. There is nothing to ship, nothing to “get done”. The code lives for 45 minutes, and then it disappears, giving you a unique opportunity to practice writing tiny pieces of perfect code.

Structured chaos

Sounds pretty hectic, right? But, the sessions were far from unstructured. Throughout the retreat, we were encouraged to keep in mind the four rules of simple design:

• Runs all the tests
• No duplication
• Proper naming (expresses ideas and reveals intent)
• Small (small methods, small classes)

The morning sessions were primarily focused on getting familiar with the problem. As the day progressed, we were encouraged by Tyler and Corey to try out certain ideas, or to think about tackling old problems in new ways. Can you solve the issue using polymorphism instead of if statements? Can you do it by capping all of your methods to 3 lines? What happens when you focus on proper naming of your tests and your methods? These “challenges” were designed to push us out of our comfort zone. It worked.

Some thoughts

I quickly realized that 45 minutes was not enough time to solve the problem. However, it took me almost the entire day to finally get comfortable with the fact that even solving a significant portion of the problem was out of reach. About half way through the day, I caught myself trying to solve a chunk of the problem that was way too large for the amount of time we were given. This resulted in me rushing a solution in an attempt to get my way too large of scope test to pass. At the end of the session, I was left confused with a pile of crappy code (unnecessary abstractions, no encapsulation, etc), at least half of which I wasn’t sure why we wrote.

I love that feeling you get when you create something, or solve a problem. This is why I am a programmer. Not being able to finish the problem, or even a significant portion of it, was incredibly frustrating. It took me the entire day to finally let go of that.

It was also great to see the different ways people were approaching the problem. At the start of the code retreat most pairs were pursuing a similar approach. But by the end of the day, the variety of approaches was simply amazing. With each session, everybody learned something new. Since we were required to find a new partner for each session, the combined experience of each new team seemed to produce a new approach, slightly different from the approaches each of the partners had tried in previous sessions. It was almost like looking at an old tree. Each session resulted in the forking of each branch on the tree. By then end of the day, there were a whole lot of branches.

Practice, practice, practice…

At the end of the code retreat, we all briefly shared with the group what we learned that day. Everybody had something new to say. The code retreat had a way of showing all of us what we needed to work on and what we needed to explore.

When writing this blog post, it suddenly hit me.

This is exactly what practice is designed to do!

When I practice something else (karate basics or kata for example), it has the same effect. Serious practice makes it very clear what needs to be developed. It makes no difference what it is that you’re practicing. The key to getting better at something is to practice, and to continue developing the “problem areas” identified by your practice. But it can’t be just any practice. As Corey stated during the code retreat:

Practice doesn’t make perfect. Perfect practice makes perfect.

Perhaps the single biggest thing I took away from the code retreat is that it showed me how to practice writing code.

What I learned

First off, I learned that I need to slow down…way down. All too often I think I know the solution to a problem before my fingers even touch the keyboard. And, perhaps worse, is that I am so focused on implementing that solution, that I will often ignore warning signs (code smells) that should be alerting me that my solution may perhaps not be the best solution. This code retreat made me realize that I need to do a much better job of listening to my tests, and listening to my code. I need to do a better job of keeping my nose open at all times for code smells, and to stop when I smell something. Something that doesn’t feel right in the tests is a sure sign that something is wrong with the design. Stop, take a step back, think for a minute about what is causing the smell, and fix it.

There is nothing wrong with having a solution in mind before you start coding. In fact, I think that most times you want to have a solution in mind before your fingers touch the keyboard. However, you should keep an open mind as you start to build your solution. Don’t assume your initial solution is the best solution. In fact, it’s probably safer to assume that it is not, and allow your tests to guide you to a better one.

Go to a code retreat!

If you’re serious about getting better as a programmer, and have never been to a code retreat, your missing out on an incredible learning experience. Make it a point to go to one. Since code retreats are essentially group coding practice, you can attend them over and over and be sure to walk away with something new every time. I’m already looking forward to the next one.

I have so much to learn.

To address this issue, we decided to create daily summaries of the subscription list data, and report off of the summary data instead of the raw data. The vast majority of our list reporting is already broken out by day, so this seemed logical. And, we decided to use MongoDB to store the summary data.

Why MongoDB?

We are already running MySQL and CouchDB in production. Why not use one of those? Why introduce a 3rd database product to the architecture?

The summary data is live data, not archived data (like the data we are storing in CouchDB). As subscriptions are created/destroyed for the current day, we need to increment/decrement the appropriate metrics. There are also some use cases where we need to alter summary documents for days in the past. This means we need support for atomic operations in the underlying database, to prevent race conditions from skewing the stats. CouchDB does not have the ability to atomically update a value in a document with a single call, so that took it out of the running. Some of our more active lists see several new subscriptions a second. Dealing with document update conflicts would have been a nightmare.

We decided to go with MongoDB over MySQL because of the structure of the summary data. We had several metrics that we wanted to keep tabs on (opt ins per day, opt outs per day, etc). Some of these metrics had a nested nature to them. For example, in addition to keeping track of the number of opt-outs per day, we also want to keep track of the reasons those subscriptions were canceled. Also, there are several ways that a user can opt-in to a subscription list. For example, each list can have several different keywords for entry via SMS. We needed a way of breaking all of these metrics out by method of entry.

Also, the main data structure containing the stats is repeated twice within the document. Once for the current day’s stats, and once for the totals for that subscription list up until the date in the document. This “to_date” data structure keeps us from having to evaluate ALL of a list’s documents in order to determine how many subscribers there are on a given date. With the “to_date” data structure, the specific day’s document is all we need.

We decided that this data structure was better represented as a single JSON document instead of a series of tables in MySQL. It seemed much cleaner to have a single document per list, per day that contained all of the information for that list’s daily activity than to have it scattered about in a series of relational tables. Choosing MongoDB meant giving up on SQL’s aggregate functions (AVG(), MAX(), MIN(), SUM(), etc), but the benefits provided by the simplicity of the data structure seemed to make up for the loss of these functions.

An example summary document

{
  "campaign_id": 1,
  "subscription_campaign_id": 2,
  "account_id": 1,
  "date": ISODate("2010-10-13T00:00:00Z"),
  "stats": {
    "sms_101": {
      "confirmed_opt_ins": 100,
      "unconfirmed_opt_ins": 15,
      "unconfirmed_opt_outs": 10,
      "unconfirmed_opt_out_reasons": { "UI": 5, "CNF": 5 },
      "confirmed_opt_outs": 20,
      "confirmed_opt_out_reasons": { "UI": 10, "CNF": 10}
    }
    "sms_102": {
      "confirmed_opt_ins": 200,
      "unconfirmed_opt_ins": 30,
      "unconfirmed_opt_outs": 20,
      "unconfirmed_opt_out_reasons": { "UI": 5, "CNF": 15 },
      "confirmed_opt_outs": 35,
      "confirmed_opt_out_reasons": { "UI": 15, "CNF": 20 },
    }
    "email": {
      "confirmed_opt_ins": 300,
      "unconfirmed_opt_ins": 35,
      "unconfirmed_opt_outs": 25,
      "unconfirmed_opt_out_reasons": { "UI": 20, "BULK": 5 },
      "confirmed_opt_outs": 30,
      "confirmed_opt_out_reasons": { "UI": 20, "BULK": 10 }
    }
  }
  "to_date": {
    "sms_101": {
      "confirmed_opt_ins": 1100,
      "unconfirmed_opt_ins": 1000,
      "unconfirmed_opt_outs": 200,
      "unconfirmed_opt_out_reasons": { "UI": 100, "CNF": 100 },
      "confirmed_opt_outs": 400,
      "confirmed_opt_out_reasons": { "UI": 300, "CNF": 100 }
    }
    "sms_102": {
      "confirmed_opt_ins": 2200,
      "unconfirmed_opt_ins": 200,
      "unconfirmed_opt_outs": 150,
      "unconfirmed_opt_out_reasons": { "UI": 100, "CNF": 50 },
      "confirmed_opt_outs": 400,
      "confirmed_opt_out_reasons": { "UI": 250, "CNF": 150 }
    }
    "email": {
      "confirmed_opt_ins": 2050,
      "unconfirmed_opt_ins": 125,
      "unconfirmed_opt_outs": 75,
      "unconfirmed_opt_out_reasons": { "UI": 75 },
      "confirmed_opt_outs": 750,
      "confirmed_opt_out_reasons": { "UI": 600, "BULK": 150 }
    }
  }
}

A possible relational database schema

MongoDB’s Atomic Operations

As I mentioned above, MongoDB’s atomic operations were key to us choosing MongoDB for this task.

MongoDB does not support locking or transactions like you would find in a traditional relational database. However, it does support a number of operations that are guaranteed to be atomic on a single document. If your data is designed so that all related data (or at least data that would need to be updated at the same time) is contained within a single document, then the atomic operations supported by MongoDB should be more than enough to handle the majority of use cases.

For this project, we made heavy use of the $inc operation. $inc will atomically increment or decrement a value anywhere in the document by the specified value. If no property exists in the document with that name, $inc will create it, and set its initial value to the value you wanted to increment it by. For this project, we simply initialized the data structure holding the metrics to an empty JSON hash when the summary document is created. The first time $inc is used to increment or decrement some metric, it will insert the metric into the hash, along with the proper initial value. Subsequent calls to update the document using the $inc operator will then update that value accordingly.

Using the $inc operation also meant we didn’t have to read the document to get the current value of the field in order to increment/decrement its value. We were simply able to increment/decrement the value by making one call to the database, keeping things nice and simple.

Atomically incrementing a document’s confirmed subscriptions count

collection = db.collection('subscription_statistics')
today = Date.today.to_time
collection.update({'campaign_id' => 1, 'date' => today},
  {"$inc" => {"stats.sms_101.confirmed_opt_ins" => 1}},
  {:safe => true})

MapReduce

MongoDB also supports MapReduce, providing you the ability to evaluate your data in ways that simply can’t be done using their standard queries. For this project, we needed to support summing the values of specific keys across several documents (to calculate the total opt-ins over a date range, for example). Initially, this sounded like a good fit for MapReduce. However, MapReduce will run the specified map function against each document in the database. The more documents your database has, the longer it will take MapReduce to run. I hoped that since we had built an index for the fields that MapReduce was using to determine if a document should be selected, that MongoDB would utilize that index to help find the eligible documents. Unfortunately that was not the case.

In our case, since we are only dealing with 365 documents for a year’s worth of statistics, it was considerably faster for us to find the documents using MongoDB’s standard queries and sum the data in ruby code, than to use MapReduce to do the same. If we were evaluating ALL of the documents in the database, then MapReduce would have been a much better option. I understand that 10gen is hard at work on making MapReduce faster for MongoDB 2.0. Having a strong MapReduce framework can be a powerful tool for a statistics database. Hopefully we’ll be able to utilize it in the future.

The Results

The results were staggering. On our largest list, the overview chart we display showing the current number of subscriptions per day over the last 30 days went from taking 37726ms to load to just 502ms. And the summary report for the list, which contains a wealth of statistics for the list including opt-ins / opt-outs per day, subscriber count by opt in keyword, and a series of summary statistics, went from taking 64836ms to load to just 322ms.

Dramatically reducing the size of the data being evaluated had an equally dramatic effect on the amount of time it took to evaluate that data. And, MongoDBs atomic operations and dynamic queries made this project a blast to work on.

def some_method(required_1, required_2, options={})
  # Do something awesome!
end

some_method("foo", "bar")
some_method("foo", "bar", :option_1 => false, :option_2 => true)
some_method("foo",
            "bar",
            :option_1 => false,
            :option_2 => true,
            :option_3 => "something",
            :option_4 => "something else")

This may not look like much. However, this feature alone is capable of producing some very readable code, and is used extensively in APIs throughout the Ruby ecosystem. Consider for a moment what these APIs would look like if Ruby did not have this capability, which isn’t hard to imagine for those of us with a background in a language like Java. You would either be forced to require that each parameter be specified:

# What is this code doing?  What do the nil values,
# or even the true and false values map to?
some_method("foo", "bar", false, nil, true, nil)

or accept a hash or a request object that contains all of the necessary parameters:

# This is much more readable, but requires that the
# options hash be created on its own line.
options = {:option_1 => true, :option_2 => false}
some_method("foo", "bar", options)

Providing optional parameters via hash key/value paris at the end of a method call produces code that is incredibly readable. You have the names of the attributes right next to their corresponding values! There is no ambiguity whatsoever as to which values match up with which parameters.

It is also very flexible. The order of the attributes in the hash does not matter, like it does for required attributes. And, it is very easy to add new options, or delete old ones.

This approach also makes it easy to specify default values for options that were not specified when calling the method:

def some_method(required_1, required_2, options={})
  defaults = {
    :option_1 => "option 1 default",
    :option_2 => "option 2 default",
    :option_3 => "option 3 default",
    :option_4 => "option 4 default"
  }
  options = defaults.merge(options)

  # Do something awesome!
end

There are however a few minor drawbacks to this approach. The first is documentation. Methods that take a hash of options as a parameter do not convey any information about the valid options in the method definition alone. And, it is possible that the method in question simply forwards the options to another method, sending you on a wild goose chase to determine the set of valid options the code supports.

# Looking for a list of valid option keys...no help here.
def some_method(required_1, required_2, options={})
  do_something_awesome_with_the_options(options)
end

This is why it is so important do document your public API if you are using this approach. Take a look at the ActiveRecord::Associations::ClassMethods documentation. This page documents, in a very clear and easy to read mannor, all of the supported options for each method.

It is also worth pointing out that while this approach is great for optional parameters, it is ill suited for required parameters. Required parameters should be specified outside of the options hash, making it clear that values for the required parameters must be provided. While it’s true that stuffing all of your parameters inside a hash means you’ll never have to look at another wrong number of arguments error again, it will make your code difficult to understand, and easy to misuse.

This was our first ‘bot’ in the room, and it felt a little weird. My teammate Ray created a persona for the Jenkins bot by associating it with the one and only Leroy Jenkins, changing the name of the bot from “Jenkins” to “Leeerrroooy J”. We also changed the sender name of the emails that Jenkins delivers to match.

But something was missing. Every time Leeerrroooy J. popped into our chat room with something to say, we’d all scream “Leeerrroooy Jenkins!”. I thought it would be awesome if we could have Campfire do this for us! But sadly, I couldn’t find any scripts to play audio in Campfire in response to a certain message.

So, I created one. Now, every time Leeerrroooy J. lets us know the status of a build, this will play. Makes me laugh every time.

Realizing this script could be useful in other contexts, I made it a tad more configurable, and put it on github. Right now, you can setup the script to play the sound at a given URL any time a specific word is said in the room, or when a certain user posts a message.

Hope you enjoy it as much as I do.

• Mean reads / second
• Mean writes / second
• Mean requests / second for DELETE, GET, HEAD, POST, and PUT requests
• Mean view requests / second
• Mean bulk HTTP requests / second
• Counts for various HTTP response codes

In addition, there is a plugin for individual CouchDB databases and individual couchdb-lucene indexes. The database plugin will report:

• Database size
• Number of documents
• Number of deleted documents
• Number of update operations

The couchdb-lucene plugin will report:

• Size of the index
• Number of documents indexed
• Number of deleted documents

The kind folks over at Scout have just released two new, official plugins based on the ones I created. The CouchDB Overall plugin combines some of the more important CouchDB metrics into a single plugin, and the CouchDB Database plugin reports the same set of the stats as the database plugin listed above.

The original plugins can be found at https://github.com/signal/scout-plugins/tree/master/couchdb. More information can be found here. I hope you find them useful.

Thanks to Doug Barth for some help on the plugins, and Derek over at Scout for putting together the official plugins.

However, polyglot persistence comes with its own set of drawbacks. One of those drawbacks is the loss of foreign keys, which are very important in maintaining data integrity. Another drawback is that Object/Relational Mapping (ORM) libraries typically focus on a specific database, or type of database. So, writing code that manages relationships between objects backed by different databases hasn’t been nearly as easy as writing code to manage relationships between objects in the same database.

Tenacity’s goal is to address some of these issues. Tenacity is a ruby gem that provides an ORM independent way of managing relationships between models backed by different databases.

Tenacity works by extending popular Ruby ORM libraries to respond to a set of methods that the tenacity core uses to build and manage relationships between objects. By extending the ORM libraries to implement this interface, tenacity is able work with the objects in a generic way, without having to know what database is backing the given objects. This approach also allows you to continue using your favorite ORM libraries. To use tenacity, you simply need to include Tenacity inside your model.

Tenacity is heavily based on ActiveRecord’s associations, and aims to behave in much the same way, supporting many of the same options.

This initial release of tenacity supports belongs_to, has_one, and has_many associations, and the ActiveRecord, CouchRest, and MongoMapper ORMs. However, there is still plenty of work to be done. Feedback, bug reports, and code contributions are always welcome.

Tenacity is free and open source, and can be found on GitHub at https://github.com/jwood/tenacity.

Example

Download

gem install tenacity

The Rise of the App

However, the web does have limitations. One such limitation has long been the ability to interact with the hardware you are using to browse the web. Over the years technologies have been developed to help with this limitation (Java, Flash, etc), but I see those technologies as extensions the web, and not part of the web itself. Those technologies have carried their own set of drawbacks as well, such as requiring the installation of client software, performance issues, and security vulnerabilities.

Nowhere is this limitation more obvious then in the mobile arena. Today’s smartphones are amazing. You have a camera, a GPS, a MP3 player, a web browser, oh…and a phone too, all wrapped up in one device. It’s like walking around with a tiny computer in your pocket. But currently, the web is only able to utilize a tiny fraction of this functionality. Its document centric design, which has been stretched, twisted, and pulled in directions that its creators never could have imagined, appears to have finally reached its limit.

In addition to hardware access limitations, may web applications were not designed to be viewed on a device with such a small screen and no mouse. It can be very difficult to view a 900 pixel wide web page on a screen that is only a couple of inches wide. And while you can zoom in over certain areas of the page, not being able to view and interact with the page in its entirety introduces some big usability issues. Also, many web applications rely on mouse over events to interact with the user. Drop down menus are a perfect example of this. There is no mouse on today’s touch screen smartphones, making these sites very difficult to navigate when viewed on a mobile device.

These limitations have given rise to “the app”. The app, a native application that runs on the smartphone itself, it not a new idea. Smartphones have been around for quite some time, and these phones have always had native applications. But only recently did Apple change the game by providing independent application developers with a way to easily sell and distribute their applications. As a result, there are now hundreds of thousands of applications out there for various platforms, many of which do incredible things with the hardware on the mobile device. Things that simply could not be done via a web application.

The rise of the app came on so quick and so strong that many began to ask if the app killed the web. This is certainly a reasonable question to ask. A native application has access to all that the device has to offer; the camera, the GPS, and yes, even the web. Apps are also designed to run on a mobile device. So, the text is easy to read, the controls are easy to use, and the user experience is generally much better than a web application.

Problems with Native Applications

While native applications have full access the the hardware and address many usability issues that web applications have on mobile devices, they introduce a new set of problems.

Some platforms charge program fees that must be paid to develop applications for the particular platform, even if you plan on creating a free app, or an app you never plan on releasing. While generally inexpensive, it is a new cost that web application developers have never had to deal with.

Deployment is more complicated for native applications than it is for web applications. Even though Apple’s App Store, Google’s Android Market, and other application marketplaces have made the distribution and installation of native applications much easier, it still does not compare to the ease of deployment of a web application. On the web, you have complete control over what you deploy, when you deploy it, and to whom you deploy it to. This is not the case with native applications, which can be subjected to drawn out, illogical review processes before being made available to the public. If you have a critical bug that needs fixing, or you’re trying to beat your closest competitor to market with some killer new feature, this process can be agonizing. And, even after publishing the latest version of your application to the marketplace, there is no way to force your users to upgrade. This could leave you supporting older, buggy versions of your application, or maintaining old services still in use by those old versions of the application.

If you charge for your application, distribution of that application is not free, or even cheap. Apple’s App Store, Google’s Android Market, and Palm’s App Store all take a 30% cut of the sale price of your application. Some may argue that this is a small price to pay for distribution to such a large audience. However, the audience is larger on the web, and distribution is piratically free.

The development of native applications is largely platform dependent. Programming languages and APIs vary significantly from platform to platform, making it practically impossible to develop a single application capable of running on multiple platforms. Cross platform development tools, such as PhoneGap and Rhomobile, address this issue by allowing you to develop native applications using HTML, Javascript, and CSS. However, these tools don’t address the other issues mentioned in this section. In addition, you also run the risk of some App stores outright banning applications not built using their tools (looking at you Apple). Though Apple has recently softened their stance on this issue, I would not rule out future restrictions.

A Mobile Web App in Native App’s Clothing

The sad part is, the majority of today’s apps don’t need to be native applications. They do nothing to utilize the features exclusive to native applications. Most of these apps simply fetch data from a web service and display it nicely to the user. Some take advantage of the GPS, or store data locally on the device. However, these apps could just as easily be mobile optimized web applications.

“Look and feel” is often cited as a reason to develop a native application instead of a mobile optimized web application. Though this argument certainly has its merits, there are plenty of options out there to help make a web application look and feel great on a mobile device. A bit of mobile optimized CSS (fluid layouts, properly sizing images, etc) can go a long way in terms of making a web application look awesome on a mobile device. And, there are projects out there, such as jQTouch, jQuery Mobile, and Sencha Touch that can help your web application look and feel more like a native application if that’s what you are looking for.

Web applications also have access to a few features that, just a short time ago, were only available to native applications. The device’s location is accessible via Javascript’s Geolocation API, eliminating the need to build a native application solely for access to the device’s GPS. Web applications also have access to persistent local storage on the mobile device via the HTML5 Web Storage API. While the Web Storage API falls short of a being a true database (it’s more like a hash, allowing you to store key/value pairs), it can go a long way in addressing the storage needs of most applications.

A good comparison between mobile apps and native apps can be found here.

Making the Web a Friendlier Place for Mobile Web Apps

The mobile web is growing by leaps and bounds, but much work is needed in order to make the web a friendlier place for mobile web applications. The W3C realizes this, and have started a Device APIs and Policy Working Group to create a specification that will allow web applications to access device hardware and services (contacts, calendar, camera, microphone, SMS, MMS, email, etc) in a standardized way. The specification is far from complete, and it will likely be quite some time before these APIs have been implemented by the various web browsers. But, the mere fact this is being worked on is a great sign for the future of mobile web applications.

There are non-technical hurdles preventing the web from being a more mobile friendly place as well.

First, browsing the web on a mobile device isn’t a very pleasant experience right now. Having “real” web browsers on mobile devices, such as WebKit, have gone a long way in fixing this. However, many sites are not optimized for mobile devices, making them difficult to use. I think this prevents the majority of people from browsing the web on their mobile device like they would on their desktop or laptop computer.

Second, we’ve been told for years that “There’s an app for that”. Personally, when I’m looking for a mobile application take care of some task, the first place I hit is the Android Market. Having one place, where apps are broken up into categories, and searchable, makes finding the right app for my needs incredibly easy. There is no such place for web applications. This used to be the case for desktop applications as well, but not anymore. The Ubuntu App Center, and now the Apple Mac Store, aim to serve the same purpose as the App Store or the Android Market, but for desktop computers instead of mobile devices. Perhaps it won’t be long before we see a categorized, searchable index for mobile web applications as well.

Conclusion

Opting to build a mobile optimized web application instead of a native application is not a new idea. However, the idea has been slow to catch on, due to some of the reasons I’ve outlined above.

Obviously, even after the Device APIs being worked on by the W3C have been implemented, there will always be a need for native applications. Games that need hardware accelerated graphics, or apps with complex user interaction will most likely stay native applications for the foreseeable future. But I hope that the other applications, especially the applications that are simply a front to a web service, are re-born as web applications at some point in the near future. For these apps, it’s time to come home.

Kudos to Heroku for continuing to find ways to make their platform easier to use.

The script is below. Hope you find it useful.

After graduating from college I immediately started work at a very large, multi-national communications technology company. At the time, I liked the idea of working for a large company. I thought that it would provide room to grow, job security, and the ability to switch roles easily without leaving the company. Over the past 10 years, I have moved from a company with about 100,000 employees, to a company with about 2,000 employees, and then to a company with 4 employees. Every time I moved to a company with substantially less employees, I noticed that my job satisfaction increased.

I have never been more happy with a job that I am at Signal. And, there are several reasons for this.

The ability to wear all hats

When I started at Signal there were only 2 other engineers (3 if you include the technical co-founder who was looking to focus less on technology and more on growing the business). With only three people on the engineering team, you need to be able to work on any part of the system, at any time. Application design and development, user interface design and development, database administration, system administration, IT, etc. In an environment like this there are many opportunities to learn something new. Database administration and user interface design were not strong areas of mine, but working at a startup gave me the opportunity to develop those skills. Much more so than at company where teams are segmented based on the functional areas of the application. And, even after a year and a half, I’m still learning new stuff like crazy.

This isn’t to say that some engineers aren’t stronger than others in certain areas, and you can’t play to those strengths. You’d be foolish not too. But at this size, it’s simply not practical to have one engineer do all the UI work, one do all of the database work, and one do all of the application development.

The chance to make a really big impact

I was hired as Signal’s fifth overall employee. When there are only five people in the entire company, everybody has the chance to make a really big impact. The harder you work, the bigger contribution you get to make. And contributions can take many forms, like a newly implemented feature, providing awesome support to a customer, a new idea for a product, or a new way of resolving a problem that has been facing the company. There is no product group or customer service team at a very small startup, but the needs traditionally serviced by those roles still exist. In addition, you are able to see the impact of your contribution immediately, be it positive or negative.

You also have the opportunity to play a bigger role in how your product or service develops. If you see an area that needs improvement, a bug that needs fixing, or a gaping hole in the product that could be addressed by a new feature, it’s up to you to address it. There is no committee or change board that needs to approve the change, and no need to line up resources from the database and UI teams to help with the work. You are expected to drive the change. This, for me, is a major source of inspiration and motivation.

The ability to choose your tools

One of the things a software engineer likes to complain about the most are the tools they are forced to use at their company. And with good reason. Using tools that you are comfortable with can make you more productive, and quite frankly, happier. Little is more frustrating than being forced to solve a problem one way when you know you can solve it quicker, and better, another way. This is generally not a problem at startups. While you rarely have the ability to choose all of your tools (programming language for example), you will generally have the ability to choose your work machine, operating system, editor/IDE, web browser, mail client, etc. And, since you are your own IT department, there is no question that you will have root access to your work machine. These may sound like minor details to most people, but any geek will assure you how important this actually is.

The ability to choose the right technology for the job

Many companies standardize on technologies. There are many valid reasons for this. However, sometimes the standardized option makes little sense for the task at hand, and you are forced trying to fit a square peg through a round hole. This generally doesn’t happen at startups. One of the best things about my job is that the engineering team gets to decide which technologies to use after evaluating the technical details of the problem it is trying to address. One might think that a stance like this would lead to an explosion of tools and technologies in our stack, but this is hardly the case. Being responsible and professional developers, we understand that we will have to live with and support the decisions we make. So, we carefully evaluate any significant new technology before bringing it into the stack.

A voice in choosing your teammates

When working on a small team, everybody needs to pull their weight in order for the company to be successful. Bringing new people onto the team is a big deal. A single bad team member can be a cancer to an otherwise strong and healthy team. Therefore, people interviewing for the company talk with everybody. This may be a bit overwhelming for candidates, but I think most appreciate how closely we protect our team dynamic. Now, this isn’t to say that everybody “interviews” each candidate. Our VP of marketing isn’t going to ask a candidate for an engineering position to solve a coding problem on the whiteboard. Instead, a few people will determine if the candidate is qualified for the position, and the rest will determine if the candidate would be a good fit, personality wise, on the team. As an engineer, I think this is great. When hiring a new engineer, we all get to ensure that the candidate is smart, loves technology, knows how to solve problems, and will get along with the rest of the team. And when hiring for a non engineering position, we get to ensure that the person we’re bringing onto the team isn’t a jerk.

A great work culture

Startup culture is great. I’m not talking about $1,000 desk chairs or having a company masseuse. Hell, two other engineers and I were coding around what the CEO called a “nerd campfire” the other day (can’t say that I disagree with that terminology) because we currently have no desks. I’m talking about the camaraderie; eating lunch together, getting to meet each others spouses and children, chilling out and having a few drinks with each other. I think this is very hard to find outside of small company. It seems to me that most startups have a casual, relaxed, and fun work environment. Flexible hours. No dress code to speak of. Signal even has a kegerator in the office which is always fully loaded with great beer. They do pretty much anything they can to make their employees feel comfortable so they be productive.

Flexible work options

Flexible work hours are very popular now-a-days. Even at the larger companies where I have worked, it wasn’t uncommon for some people to come in at 11am and stay until 7pm or 8pm. Having the ability to fit work around your personal schedule is a big plus, and I feel it makes for a much happier employee.

Even more important to me is the ability to work from home. I live about 30 miles from the office, which takes about 90 minutes to travel during rush hour by car or train. I’ve tried not too complain too much about my commute, because nobody forced me to live where I do. It was my choice, and I’d make the same choice again. But, when I was given the opportunity to work from home, I jumped on it. Working from home saves me 3 hours a day. That is huge for me, as it always feels like there are not enough hours in the day. Working from home 4 days a week has given me the ability to see my wife and kids more often, start running, and to spend a little more time on some side projects to improve and expand my skills as a developer. Seeing my kids more often is by far the biggest benefit of working from home. Now that my oldest is in school, I barely see him when I go into the office. Our schedules only overlap for about 30 minutes in the morning, and 30 minutes at night. This is not the case when I work from home.

No bullshit politics

When there are only 13 people in the entire company, there are no politics. There is no jockeying for position. There is no posturing to give the impression that you are an all star employee when you’re really just a leech. There is no room to hide. If you’re not contributing, everybody knows it. If you are trying to exploit others for personal gain, everybody sees it.

Your opinion matters

It’s hard to make a decision that impacts the company in complete isolation at a startup. The founders at Signal always consult the team when it comes to making a big decision. Make no mistake, they have the final say. But they value the opinions of the team and constantly request feedback. This gives the employees a sense of ownership, which is great for morale and can be very motivational.

Conclusion

I think many of these points are the product of a few major differences between a big company and a startup. First, productivity is paramount, and anything that can be done to increase productivity is done. If employees aren’t producing at their highest level, it greatly increases the risk of the startup folding (delaying time to market or being surpassed by a competitor). Second, startups run on the bare minimum. There is no extra weight to carry around. There are no employees that need to justify their existence by adding unnecessary overhead. Everybody has more than enough “real work” to do in order to move the company forward.

Working for a startup does have some drawbacks as well, but the positives far outweigh the negatives. The comparison is not even close as far as I’m concerned. As I mentioned at the beginning of the post, each company I have worked at has been substantially smaller than the previous one, and my job satisfaction has increased with each move. While some large companies can have some really interesting problems to solve (especially when it comes to problems of scale), I’m not sure I could ever go back to working for a large company.

Yo Ho, it’s a startup’s life for me.

See everybody next month!

Information about the meeting can be found here. I hope to see you there!

First and foremost, this was an incredibly well organized and wonderfully run conference. Ray Hightower and the WindyCityDB team did an amazing job. The space at IIT was perfect. The food and drink was ample and tasty. And the talks were interesting and thought provoking. To top it off, all proceeds from the conference, approximately $1,000, was donated to the Greater Chicago Food Depository.

One theme that took me by surprise was the broad support for relational databases. Almost all of the speakers stated that the relational database still has a place in this new world, along side all of the NoSQL technologies that we had gathered to discuss. This is something I completely agree with, but being a NoSQL conference, I had imagined that there would be quite a bit of RDBMS bashing going on. Nothing could have been further from the truth. Everybody seemed to agree that the RDBMS is a very functional and reliable piece of software that is still very much needed. Everybody also seemed to agree that there are some things the RDBMS doesn’t do too well based on how it is designed, and that the different NoSQL technologies aim to fill this gap. Nobody was advocating their particular NoSQL database as a replacement to the RDBMS. Instead, all were pitched as alternatives, created to solve different problems.

Another theme that sort of caught me off guard was the wide acceptance of Polyglot Persistence as “the future” of application persistence methods. I say “sort of” because I was there to speak about Polyglot Persistence, which of course means that I think it can be a great tool if used properly. However, I thought it was a bit more controversial of a topic. As it turns out, there are many already taking a polyglot approach to solving their persistence needs. This was great to hear, as made me feel a little less insane for wanting to use multiple databases within a single application. Well, I could still be insane, but at least now I know I’m in good company :) Several speakers mentioned Polyglot Persistence in their talks, and I spoke with several individuals at the conference, and after the conference, who are actively using a polyglot approach to address their persistence requirements.

I also found John Nunemaker’s prediction that hosted database solutions are going to be the future of application data persistence to be very interesting. If you think about it, this is exactly what happened with web hosting. Most companies, especially small companies, now use a 3rd party to host their production environments. In fact, I’d bet that the thought of hosting your own environment doesn’t even cross the mind of the vast majority of small companies now-a-days. I can easily see how hosted database solutions, such as MongoHQ or Cloudant, could follow a similar path. You would no longer need to worry about hardware issues, scaling issues, backups, maintenance, etc. It would allow you to simply focus on your data.

Even though I didn’t get a chance to play with it, the WindyCityDB lab was a nice touch, especially for those who had never tinkered with some of the NoSQL databases on display. The lab consisted of several iMacs, each loaded with a particular NoSQL database. All of the databases were preloaded with the Stack Overflow data set. Sharing a data set was a great idea, as it gave you a good way to compare and contrast how the different databases handled the same set of data.

I really enjoyed myself at this conference, and am already looking forward to next year’s. At the rate the NoSQL movement is moving, I can only imagine what the landscape will look like a year from now.

The case for Polyglot Persistence was made throughout the day by several of the speakers. Most people seemed to acknowledge the fact that it is very likely one database may not have all of the tools you need to get your job done. Instead of coercing a single database to try and do things it was never designed to do, it is becoming more common for applications to use multiple databases, utilizing each for their respective strengths. But, of course, using multiple databases in a single application comes with its own set of issues, and you should make sure there is a real need for Polyglot Persistence before making that leap.

Thank you very much to the WindyCityDB organizers for putting on such a great event. I had a great time, learned a ton, met some interesting people, and participated in some great conversations. What more can you ask for in a tech conference?

So I started taking notes, documenting everything I could regarding what we learned about CouchDB, the design decisions we made for our application (and their respective trade-offs), and the migration of our application code to use CouchDB. I organized those notes, and posted them on this blog as a case study, hoping it would help others looking into CouchDB.

I received a lot of positive feedback from the posts, making me feel like I had in fact filled that need, at least to some degree. Even better, earlier this year I was contacted by Couchio about combining the series of posts into a white paper that would be posted on their site. This would put the case study in front of a larger audience, potentially helping even more people. I was thrilled.

Today, that white paper was released as the “Epic Interactive Mediums Whitepaper” (love the Epic :)). You can get it here. Many thanks to the kind folks over at Couchio for putting this together. I hope people will find it helpful.

I saw the re-write as a good opportunity to get more familiar with Rails 3 and MongoDB. After getting Rails 3 setup to use MongoDB via MongoMapper (Ben Scofield’s setup script gets you most of the way there), I quickly realized that the tests would not run.

jwood-mbp:addressbook jwood$ rake test
(in /Users/jwood/dev/personal/addressbook)
Errors running test:units, test:functionals, test:integration!

jwood-mbp:addressbook jwood$ rake test:units
(in /Users/jwood/dev/personal/addressbook)
rake aborted!
Don't know how to build task 'db:test:prepare'

Some Googling revealed this post by Yehuda Katz, in which he says:

Additionally, users can completely remove the ActiveRecord gem, and be rid of the generators, rake tasks and other peripheral elements. An example: if DataMapper implements a db:test:prepare rake task, a Rails developer can replace ActiveRecord with DataMapper, and the test:units rake task will use DataMapper’s new tasks.

We’re not using DataMapper here, but the same thing applies to MongoMapper. So, all that is needed is to implement a db:test:prepare task, which I did in lib/tasks/mongo.rake

namespace :db do
  namespace :test do
    task :prepare do
      # Stub out for MongoDB
    end
  end
end

As of right now, the task does nothing. I may end up adding some functionality to it as I work on the project. We’ll see.

Running the tests again, I can see that was all that was needed to get going.

jwood-mbp:addressbook jwood$ rake test:units
(in /Users/jwood/dev/personal/addressbook)
Loaded suite /Users/jwood/.rvm/gems/ruby-1.9.1-p378@rails3/gems/
rake-0.8.7/lib/rake/rake_test_loader
Started
...
Finished in 0.062854 seconds.

3 tests, 3 assertions, 0 failures, 0 errors, 0 skips

Standup Timer is free and open source. The source code can be found at http://github.com/jwood/standup-timer. Standup Timer can be found in the Android Market.

First, Standup Timer was my first mobile app. Aside from a very small amount of time I’ve spent playing around with the Blackberry development kit (really too small to even mention), I have had no prior experience developing software for a mobile device. My background is largely web development and distributed applications development. Second, I have been doing Java development for quite some time, so I am very comfortable working in Java. Third, if you’re not familiar with Standup Timer, it is a very simple application. It only interacts with a few components of the Android platform, and only consists of a few screens. No GPS, no web, no multi-touch, etc. So, my tour of the Android SDK was far from complete.

Dealing with multiple devices

Supporting multiple screen sizes

The topic of supporting multiple screen sizes has been a popular one, especially when comparing Android development to iPhone development. Supporting multiple screen sizes in your application does add a little complexity to the development process, but for my app (and I’d imagine most apps), it is very manageable.

This issue did not take Google by surprise. They knew that if Android was to be successful, it would need support devices with a wide range of physical characteristics, including screen size. So, from the beginning, they made it possible for applications to support multiple screen sizes with little effort. The Android platform contains many features to help with this, including the ability to pre-scale images for phones with different resolutions, allowing you to specify your screen component sizes in density independent pixels (dips), and allowing you to easily center and stretch your components to fill the screen. These features, along with a set of best practices for supporting multiple screen sizes, are documented very well at http://developer.android.com/guide/practices/screens_support.html.

For applications that do not use the standard Android view components, or do custom graphics, it may be a different story. I have read blog posts from some Android game developers saying it is a larger issue, and some who say it is not. So, I’m not sure. However, for the vast majority of applications, the standard view components work just fine, making supporting multiple screen sizes a very manageable issue.

One thing that did bite me regarding multiple screen sizes is the fact that views will not scroll by default if they happen to flow off the screen. You need to anticipate this, and wrap any views that may flow off the screen with a ScrollView to enable scrolling behavior.

Supporting multiple devices

To me, this issue is much larger than the multiple screen size issue. Android is an open source project, which gives anybody the ability to modify the code however they wish. Google prevents carriers and cell phone manufacturers from abusing this by holding back a few key applications from the open source release, including the Android Market application. Without the Android Market application, phones would not have access to the 20,000 apps currently available for the Android platform.

However, phone manufacturers do tweak the platform to work for their specific hardware. Android 2.1 on one device is not necessarily the same as Android 2.1 on another device. For an example of this, compare 2.1 on the Motorola DROID with 2.1 on the Nexus One. These seemingly minor modifications can be very troublesome for application developers. The web has been flooded with reports of developers growing frustrated with the complexity matrix of Android versions and the exploding number of phones running them, all potentially containing their own tweaks to the platform. Browsing through the change logs of applications I have installed on my Motorola DROID, I can see that several apps have made changes to fix issues on specific devices.

This issue is forcing developers to not only test on every device they plan on supporting, but also to write device specific code to work around any known issues for a device. As Android continues to grow, this level of support will be unsustainable. Developing for a common platform should mean that your application will run fine on any device running that platform. But, because of these device specific tweaks, this is quickly turning out not to be the case for Android. I worry that if Google doesn’t find some way to control this, developers will continue to abandon the platform.

I have even run into what appear to be device related issues with Standup Timer, which is miniscule compared to the size and complexity some of the other apps available in the Android Market. Standup Timer uses an Android API that prevents the user’s screen from blacking out while a timer is in progress, letting you always see how much time is remaining for a meeting. Just the other day somebody left a comment in the Market indicating that this was not working on their device. So far, all of the other reviews have been positive, leading me to believe that this could be an issue with that user’s particular device. The commentator didn’t provide any information about the device they were using, or how they could be contacted. So, I’m not sure if I’ll ever be able to track down this issue. Even if I knew the device experiencing the issue, I’m not sure I would be able to help. Unless you have access the device in question, reproducing these issues is practically impossible. The Android simulator tool is great, but you cannot create a simulator for a specific Android device, running that device’s flavor of Android. You can only create simulators running the “generic” version of an Android release.

Development

Application life cycle

The Android platform primarily communicates with an application through a series of life cycle events. The platform will let the application know when an activity (a screen) has been created, paused (lost focus), resumed (regained focus), etc. The life cycle events are easy to understand, and are specific enough so that I never need to examine the state of my application or the platform within one of these callbacks. The fact that a specific life cycle method was called tells you what state your application is in.

My largest complaint about how the platform manages an application is how it destroys and re-creates the activity (the object that backs the screen) when the phone rotates from portrait mode to landscape mode, or visa versa. It seems to me that this would best be handled by making the appropriate life cycle callbacks to the same activity instance, giving it a chance to redraw the screen. Instead, you are forced to save all of your activity’s state, and then reload it when the new activity instance takes over. Forgetting to save the state of a particular variable means that value will be lost when the phone is rotated. This was the source of several bugs when writing Standup Timer.

Android APIs

I have no major issues with the Android APIs. They seem complete (for what I needed), and most importantly, behaved as expected. The database API seems a bit archaic, especially in today’s age of sophisticated O/R mapping tools. But, at least I didn’t have to catch SQLException after each operation, like the JDBC API.

However, the lack of decent documentation for some of the APIs is a problem. There were a few instances where I was forced into using trial and error to determine the purpose of a method parameter. I feel this level of unclarity is unacceptable for a public API.

UI Creation

The Android SDK offers developers two methods for creating UI screens: programatically using Java code or declaratively using XML. I did not attempt the programatic approach, as it brought back too many nightmares of building UIs for Java applets. The XML approach to creating screen layouts, which is the recommended approach, is very straight forward. Java developers, especially those working in the “enterprise”, have grown to hate XML over the past few years, due to its verbosity and proliferation. But, I think Android’s use of it is very tasteful. At no point did I consider myself to be in XML-hell. Looking back on the final XML that declares my user interface, it doesn’t seem overly verbose or complicated.

I did however struggle at times to figure out how to get the UI to look exactly the way I wanted it. Again, this seems to be the result of poor documentation of the XML elements and their corresponding attributes. Some more complex examples in the sample code included in the SDK would have also helped.

If you keep your screens simple, and don’t display too much information on any given screen, then you shouldn’t have much trouble dealing with how that screen looks in portrait or landscape mode. Screens with a list of items, for example, will usually look just fine when viewed in either portrait or landscape mode. When in landscape mode, the user will just see a little more blank space in the list. For more complicated screens, the Android platform allows you to specify an alternate UI layout for landscape mode. This allows you to completely restructure the elements on a screen to deal with the the wider, shorter screen size without affecting how the screen looks in portrait mode. Given how I struggled to get the views to look exactly the way I wanted, I tried my best to avoid the need to specify an alternate layout file for landscape views. Only two of the screens in Standup Timer actually needed a landscape specific layout file.

adb

The Android Debug Bridge (adb) is a nifty little tool that allows you to interact with an Android device or simulator from the command line. You can install/uninstall applications, interact with files on the device, or run a shell on the device. It also provides a series of commands that allow you to more easily create scripts to run on the device. Not only is this tool useful by itself, but it also enables the creation of sophisticated tools for Android development. Learning how to use adb is a wise investment of your time.

The Eclipse plugin

The Eclipse plugin for Android development is great. It provides tight integration between Eclipse and the Android SDK. Code completion is available for the Android APIs as well as the different types of XML files used by the Android platform. It has great support for Android resource files. You can easily start your application on a simulator with the click of a button, and running your tests is a breeze. I would not attempt to write an Android application without this plugin, or a similar plugin for a different IDE.

However, I did run into a few issues regarding how Eclipse interacts with the Android simulators. On more than a few occasions, when starting the application or running the tests, the Eclipse plugin started a simulator running a version of the Android platform that did not match my project settings (although, I have not completely ruled out user error here :) ). Also, the Eclipse plugin seems to have a few bugs around interacting with multiple simulators. adb allows you to specify the target device or simulator when issuing a command, which is useful when you have more than one simulator running at a time (which is almost always the case given the current state of the Android platform). However, the Eclipse plugin doesn’t always utilize this capability. Sometimes it will detect multiple simulators running, and ask you which one you’d like to target. Other times it will not. In addition to the simulator interaction issues, there are a few areas that could benefit from a bit of polish. The UI builders, for example, are difficult to work with. It became obvious very quickly that I would be better hand coding the XML for the UI than attempting to use the builders.

Testing

Simulators are awesome

The Android simulators are awesome. Once you have the proper versions of the SDK installed, it is trivial to create simulators with different screen sizes, different amounts of available storage, running different versions of the Android platform. And, as previously mentioned, adb is great for interacting with the simulators. While not perfect (you should always test your app on a real device before publishing it), they are pretty darn close.

The only thing missing, and I’m not sure how feasible this is, is the ability to create a simulator for a specific device. For example, if I know that my app is crashing on the Nexus One for some reason, I’d love to be able to create a Nexus One simulator, so I can find and fix the issue. Currently, debugging and fixing device specific issues is not possible without access to the device in question. I know some large mobile shops purchase the devices they plan on supporting, for testing purposes. And, services like Device Anywhere allow you virtual access to a physical device, which would satisfy this need. But these options, especially the first one, are expensive. A developer trying to fix a device specific issue in their free, non ad subsidized application will not have the resources for either of these options.

Automated (unit and functional) Testing

Automated testing on the Android platform is PAINFUL. It’s hard to decide where to even begin. First off, the tests provided with the sample applications in the SDK are very bare bones. Most of them simply assert that an activity has been created, and that’s it. If there was sufficient documentation describing how to write tests for you application, this wouldn’t be that big of a deal. But, there isn’t. Luckily, guys like Diego Torres Milano are writing blog posts and giving presentations in an attempt to fill this gaping hole in documentation. The slides from Diego’s presentation at Droidcon 2009 are a great place to start. Standup Timer, which is open source, also has a complete set of unit and functional tests for you to check out if you so desire.

Another pain point regarding automated tests for Android applications is that the tests cannot be run inside a standard Java Virtual Machine. The implementation of the APIs in the android.jar file provided with the SDK simply throw exceptions. That jar file is only meant for building your application, not running it. So, all tests must be run on the device (or in a simulator). This dramatically slows down the execution of the tests. One alternative proposed by a non-Android group inside of Google is to mock out all of the Android components using a mocking library, enabling the execution of the tests in a standard virtual machine. Although this would speed up the test execution, it comes with its own drawbacks. Mainly, you end up having to mock out a TON of stuff. I think that mocking out too many components significantly reduces the value of your tests. If you mock out everything, what are you really testing?

The SDK contains a few base classes that you can extend for certain types of tests. In particular, the SDK provides support for functional tests (the testing of a particular activity, or screen) and unit tests (the testing of some underlying support code). Writing unit tests is pretty straight forward. With a little bit of investigation and some trial and error, I was able to create a set of tests for my database layer, which will execute against a test version of the application’s database.

Functional tests were much more problematic. In addition to executing much slower than unit tests, they are also more difficult to write. For functional tests, you write code to perform the operations a user would be performing on their phone (entering text, pressing buttons, etc). The problem is that it is not very straight forward to perform these operations via code. It took me quite some time to figure out how to simply enter some text in a text area and press a button via code. Even when I thought I had functional testing figured out, there was still a use case I was unable to write a test for. Most of the issues revolve around gaining access to the components you wish to interact with. For a simple text box or button, it is very easy. You can simply fetch the view by its id, the same way you do in your application. However, once you start nesting components, it becomes much more complicated. I tried several different ways to invoke the context menu (the long press menu) of a list item, inside of a list, which is inside of a tab. None of them worked.

Instead of writing functional tests for activities, I found it much easier to test activities using the ActivityUnitTestCase class with a mocked out instance of the activity. Test classes that extend ActivityUnitTestCase don’t have as much as access to the platform as functional tests. Several API methods will throw exceptions if called from within a ActivityUnitTestCase, and other API methods simply do nothing. However, this can be addressed with some simple tools that should be in every developers toolbox: mocking and stubbing. It is trivial to contain a method call that would throw an exception during testing within a method of its own. This method can then be overridden by a mock implementation of the activity. The mock can then optionally set a flag when the method is called, so the test can verify that some action was performed. One place I do this is where the activity under test tries to start another activity. Calling startActivity from within a ActivityUnitTestCase does nothing. So, I placed the call to startActivity by itself in a protected method, overrode that method in the mock activity class, set a flag in the mock method implementation, and checked to see if that flag was set in the test case to verify that the activity was started.

Deployment

Publishing to the Android Market

Publishing an application to the Android market could not be any easier. The process is very well documented on the Android developers website, and the Eclipse plugin makes creating and signing an installation package a point and click operation.

A fee of $25 is required to create a Android Developers account. This account gives you the ability to upload your application to the Android Market using the Android Developers website. Once published, the application can immediately be found in the Android Market. There is no approval process. Updating your application is just as easy. Simply upload the new version of your application, and it is available in the Android Market immediately.

It should be noted that there is no fee to install the application on your phone.

Summary

Android is a great platform. It is feature rich, and developer friendly. The devices that run it are powerful and easy to use. I also love the fact that it is built on open source technology, and that Google, for the most part, has continued this tradition of openness.

But, the platform is not without problems. Most of my complaints here are minor, and are very addressable. Android is still a young platform, so there is plenty of room to improve. What worries me most about the future of the Android platform is the complexity matrix I talk about in the Supporting Multiple Devices section.

Android was created to be a common platform for mobile application development. A platform where application developers could write a single application capable of running on numerous devices. However, this dream is at risk of becoming just that, a dream. Unless Google can find a way to minimize the impact of device specific issues, it risks losing more and more Android developers. Last year it was already known that 2010 was going to be a big year for Android. With several new devices slated to hit the market, industry analysts are expecting its market share to jump considerably. However, if each new device that hits the market brings with it its own set of device specific quirks, this could also cause the Android development community to quickly erode. Applications are a very large part of the mobile experience today, and they will continue to be in the future. But, without a development community, there will be no applications.

I really hope Google finds a way to address this very serious issue.

Standup Timer is free and open source. The source code can be found at http://github.com/jwood/standup-timer. Standup Timer can be found in the Android Market.

[Show as slideshow]

[View with PicLens]