Standup Timer is free and open source. The source code can be found at http://github.com/jwood/standup-timer. Standup Timer can be found in the Android Market.

First, Standup Timer was my first mobile app. Aside from a very small amount of time I’ve spent playing around with the Blackberry development kit (really too small to even mention), I have had no prior experience developing software for a mobile device. My background is largely web development and distributed applications development. Second, I have been doing Java development for quite some time, so I am very comfortable working in Java. Third, if you’re not familiar with Standup Timer, it is a very simple application. It only interacts with a few components of the Android platform, and only consists of a few screens. No GPS, no web, no multi-touch, etc. So, my tour of the Android SDK was far from complete.

Dealing with multiple devices

Supporting multiple screen sizes

The topic of supporting multiple screen sizes has been a popular one, especially when comparing Android development to iPhone development. Supporting multiple screen sizes in your application does add a little complexity to the development process, but for my app (and I’d imagine most apps), it is very manageable.

This issue did not take Google by surprise. They knew that if Android was to be successful, it would need support devices with a wide range of physical characteristics, including screen size. So, from the beginning, they made it possible for applications to support multiple screen sizes with little effort. The Android platform contains many features to help with this, including the ability to pre-scale images for phones with different resolutions, allowing you to specify your screen component sizes in density independent pixels (dips), and allowing you to easily center and stretch your components to fill the screen. These features, along with a set of best practices for supporting multiple screen sizes, are documented very well at http://developer.android.com/guide/practices/screens_support.html.

For applications that do not use the standard Android view components, or do custom graphics, it may be a different story. I have read blog posts from some Android game developers saying it is a larger issue, and some who say it is not. So, I’m not sure. However, for the vast majority of applications, the standard view components work just fine, making supporting multiple screen sizes a very manageable issue.

One thing that did bite me regarding multiple screen sizes is the fact that views will not scroll by default if they happen to flow off the screen. You need to anticipate this, and wrap any views that may flow off the screen with a ScrollView to enable scrolling behavior.

Supporting multiple devices

To me, this issue is much larger than the multiple screen size issue. Android is an open source project, which gives anybody the ability to modify the code however they wish. Google prevents carriers and cell phone manufacturers from abusing this by holding back a few key applications from the open source release, including the Android Market application. Without the Android Market application, phones would not have access to the 20,000 apps currently available for the Android platform.

However, phone manufacturers do tweak the platform to work for their specific hardware. Android 2.1 on one device is not necessarily the same as Android 2.1 on another device. For an example of this, compare 2.1 on the Motorola DROID with 2.1 on the Nexus One. These seemingly minor modifications can be very troublesome for application developers. The web has been flooded with reports of developers growing frustrated with the complexity matrix of Android versions and the exploding number of phones running them, all potentially containing their own tweaks to the platform. Browsing through the change logs of applications I have installed on my Motorola DROID, I can see that several apps have made changes to fix issues on specific devices.

This issue is forcing developers to not only test on every device they plan on supporting, but also to write device specific code to work around any known issues for a device. As Android continues to grow, this level of support will be unsustainable. Developing for a common platform should mean that your application will run fine on any device running that platform. But, because of these device specific tweaks, this is quickly turning out not to be the case for Android. I worry that if Google doesn’t find some way to control this, developers will continue to abandon the platform.

I have even run into what appear to be device related issues with Standup Timer, which is miniscule compared to the size and complexity some of the other apps available in the Android Market. Standup Timer uses an Android API that prevents the user’s screen from blacking out while a timer is in progress, letting you always see how much time is remaining for a meeting. Just the other day somebody left a comment in the Market indicating that this was not working on their device. So far, all of the other reviews have been positive, leading me to believe that this could be an issue with that user’s particular device. The commentator didn’t provide any information about the device they were using, or how they could be contacted. So, I’m not sure if I’ll ever be able to track down this issue. Even if I knew the device experiencing the issue, I’m not sure I would be able to help. Unless you have access the device in question, reproducing these issues is practically impossible. The Android simulator tool is great, but you cannot create a simulator for a specific Android device, running that device’s flavor of Android. You can only create simulators running the “generic” version of an Android release.

Development

Application life cycle

The Android platform primarily communicates with an application through a series of life cycle events. The platform will let the application know when an activity (a screen) has been created, paused (lost focus), resumed (regained focus), etc. The life cycle events are easy to understand, and are specific enough so that I never need to examine the state of my application or the platform within one of these callbacks. The fact that a specific life cycle method was called tells you what state your application is in.

My largest complaint about how the platform manages an application is how it destroys and re-creates the activity (the object that backs the screen) when the phone rotates from portrait mode to landscape mode, or visa versa. It seems to me that this would best be handled by making the appropriate life cycle callbacks to the same activity instance, giving it a chance to redraw the screen. Instead, you are forced to save all of your activity’s state, and then reload it when the new activity instance takes over. Forgetting to save the state of a particular variable means that value will be lost when the phone is rotated. This was the source of several bugs when writing Standup Timer.

Android APIs

I have no major issues with the Android APIs. They seem complete (for what I needed), and most importantly, behaved as expected. The database API seems a bit archaic, especially in today’s age of sophisticated O/R mapping tools. But, at least I didn’t have to catch SQLException after each operation, like the JDBC API.

However, the lack of decent documentation for some of the APIs is a problem. There were a few instances where I was forced into using trial and error to determine the purpose of a method parameter. I feel this level of unclarity is unacceptable for a public API.

UI Creation

The Android SDK offers developers two methods for creating UI screens: programatically using Java code or declaratively using XML. I did not attempt the programatic approach, as it brought back too many nightmares of building UIs for Java applets. The XML approach to creating screen layouts, which is the recommended approach, is very straight forward. Java developers, especially those working in the “enterprise”, have grown to hate XML over the past few years, due to its verbosity and proliferation. But, I think Android’s use of it is very tasteful. At no point did I consider myself to be in XML-hell. Looking back on the final XML that declares my user interface, it doesn’t seem overly verbose or complicated.

I did however struggle at times to figure out how to get the UI to look exactly the way I wanted it. Again, this seems to be the result of poor documentation of the XML elements and their corresponding attributes. Some more complex examples in the sample code included in the SDK would have also helped.

If you keep your screens simple, and don’t display too much information on any given screen, then you shouldn’t have much trouble dealing with how that screen looks in portrait or landscape mode. Screens with a list of items, for example, will usually look just fine when viewed in either portrait or landscape mode. When in landscape mode, the user will just see a little more blank space in the list. For more complicated screens, the Android platform allows you to specify an alternate UI layout for landscape mode. This allows you to completely restructure the elements on a screen to deal with the the wider, shorter screen size without affecting how the screen looks in portrait mode. Given how I struggled to get the views to look exactly the way I wanted, I tried my best to avoid the need to specify an alternate layout file for landscape views. Only two of the screens in Standup Timer actually needed a landscape specific layout file.

adb

The Android Debug Bridge (adb) is a nifty little tool that allows you to interact with an Android device or simulator from the command line. You can install/uninstall applications, interact with files on the device, or run a shell on the device. It also provides a series of commands that allow you to more easily create scripts to run on the device. Not only is this tool useful by itself, but it also enables the creation of sophisticated tools for Android development. Learning how to use adb is a wise investment of your time.

The Eclipse plugin

The Eclipse plugin for Android development is great. It provides tight integration between Eclipse and the Android SDK. Code completion is available for the Android APIs as well as the different types of XML files used by the Android platform. It has great support for Android resource files. You can easily start your application on a simulator with the click of a button, and running your tests is a breeze. I would not attempt to write an Android application without this plugin, or a similar plugin for a different IDE.

However, I did run into a few issues regarding how Eclipse interacts with the Android simulators. On more than a few occasions, when starting the application or running the tests, the Eclipse plugin started a simulator running a version of the Android platform that did not match my project settings (although, I have not completely ruled out user error here :) ). Also, the Eclipse plugin seems to have a few bugs around interacting with multiple simulators. adb allows you to specify the target device or simulator when issuing a command, which is useful when you have more than one simulator running at a time (which is almost always the case given the current state of the Android platform). However, the Eclipse plugin doesn’t always utilize this capability. Sometimes it will detect multiple simulators running, and ask you which one you’d like to target. Other times it will not. In addition to the simulator interaction issues, there are a few areas that could benefit from a bit of polish. The UI builders, for example, are difficult to work with. It became obvious very quickly that I would be better hand coding the XML for the UI than attempting to use the builders.

Testing

Simulators are awesome

The Android simulators are awesome. Once you have the proper versions of the SDK installed, it is trivial to create simulators with different screen sizes, different amounts of available storage, running different versions of the Android platform. And, as previously mentioned, adb is great for interacting with the simulators. While not perfect (you should always test your app on a real device before publishing it), they are pretty darn close.

The only thing missing, and I’m not sure how feasible this is, is the ability to create a simulator for a specific device. For example, if I know that my app is crashing on the Nexus One for some reason, I’d love to be able to create a Nexus One simulator, so I can find and fix the issue. Currently, debugging and fixing device specific issues is not possible without access to the device in question. I know some large mobile shops purchase the devices they plan on supporting, for testing purposes. And, services like Device Anywhere allow you virtual access to a physical device, which would satisfy this need. But these options, especially the first one, are expensive. A developer trying to fix a device specific issue in their free, non ad subsidized application will not have the resources for either of these options.

Automated (unit and functional) Testing

Automated testing on the Android platform is PAINFUL. It’s hard to decide where to even begin. First off, the tests provided with the sample applications in the SDK are very bare bones. Most of them simply assert that an activity has been created, and that’s it. If there was sufficient documentation describing how to write tests for you application, this wouldn’t be that big of a deal. But, there isn’t. Luckily, guys like Diego Torres Milano are writing blog posts and giving presentations in an attempt to fill this gaping hole in documentation. The slides from Diego’s presentation at Droidcon 2009 are a great place to start. Standup Timer, which is open source, also has a complete set of unit and functional tests for you to check out if you so desire.

Another pain point regarding automated tests for Android applications is that the tests cannot be run inside a standard Java Virtual Machine. The implementation of the APIs in the android.jar file provided with the SDK simply throw exceptions. That jar file is only meant for building your application, not running it. So, all tests must be run on the device (or in a simulator). This dramatically slows down the execution of the tests. One alternative proposed by a non-Android group inside of Google is to mock out all of the Android components using a mocking library, enabling the execution of the tests in a standard virtual machine. Although this would speed up the test execution, it comes with its own drawbacks. Mainly, you end up having to mock out a TON of stuff. I think that mocking out too many components significantly reduces the value of your tests. If you mock out everything, what are you really testing?

The SDK contains a few base classes that you can extend for certain types of tests. In particular, the SDK provides support for functional tests (the testing of a particular activity, or screen) and unit tests (the testing of some underlying support code). Writing unit tests is pretty straight forward. With a little bit of investigation and some trial and error, I was able to create a set of tests for my database layer, which will execute against a test version of the application’s database.

Functional tests were much more problematic. In addition to executing much slower than unit tests, they are also more difficult to write. For functional tests, you write code to perform the operations a user would be performing on their phone (entering text, pressing buttons, etc). The problem is that it is not very straight forward to perform these operations via code. It took me quite some time to figure out how to simply enter some text in a text area and press a button via code. Even when I thought I had functional testing figured out, there was still a use case I was unable to write a test for. Most of the issues revolve around gaining access to the components you wish to interact with. For a simple text box or button, it is very easy. You can simply fetch the view by its id, the same way you do in your application. However, once you start nesting components, it becomes much more complicated. I tried several different ways to invoke the context menu (the long press menu) of a list item, inside of a list, which is inside of a tab. None of them worked.

Instead of writing functional tests for activities, I found it much easier to test activities using the ActivityUnitTestCase class with a mocked out instance of the activity. Test classes that extend ActivityUnitTestCase don’t have as much as access to the platform as functional tests. Several API methods will throw exceptions if called from within a ActivityUnitTestCase, and other API methods simply do nothing. However, this can be addressed with some simple tools that should be in every developers toolbox: mocking and stubbing. It is trivial to contain a method call that would throw an exception during testing within a method of its own. This method can then be overridden by a mock implementation of the activity. The mock can then optionally set a flag when the method is called, so the test can verify that some action was performed. One place I do this is where the activity under test tries to start another activity. Calling startActivity from within a ActivityUnitTestCase does nothing. So, I placed the call to startActivity by itself in a protected method, overrode that method in the mock activity class, set a flag in the mock method implementation, and checked to see if that flag was set in the test case to verify that the activity was started.

Deployment

Publishing to the Android Market

Publishing an application to the Android market could not be any easier. The process is very well documented on the Android developers website, and the Eclipse plugin makes creating and signing an installation package a point and click operation.

A fee of $25 is required to create a Android Developers account. This account gives you the ability to upload your application to the Android Market using the Android Developers website. Once published, the application can immediately be found in the Android Market. There is no approval process. Updating your application is just as easy. Simply upload the new version of your application, and it is available in the Android Market immediately.

It should be noted that there is no fee to install the application on your phone.

Summary

Android is a great platform. It is feature rich, and developer friendly. The devices that run it are powerful and easy to use. I also love the fact that it is built on open source technology, and that Google, for the most part, has continued this tradition of openness.

But, the platform is not without problems. Most of my complaints here are minor, and are very addressable. Android is still a young platform, so there is plenty of room to improve. What worries me most about the future of the Android platform is the complexity matrix I talk about in the Supporting Multiple Devices section.

Android was created to be a common platform for mobile application development. A platform where application developers could write a single application capable of running on numerous devices. However, this dream is at risk of becoming just that, a dream. Unless Google can find a way to minimize the impact of device specific issues, it risks losing more and more Android developers. Last year it was alraedy known that 2010 was going to be a big year for Android. With several new devices slated to hit the market, industry analysts are expecting its market share to jump considerably. However, if each new device that hits the market brings with it its own set of device specific quirks, this could also cause the Android development community to quickly erode. Applications are a very large part of the mobile experience today, and they will continue to be in the future. But, without a development community, there will be no applications.

I really hope Google finds a way to address this very serious issue.

Standup Timer is free and open source. The source code can be found at http://github.com/jwood/standup-timer. Standup Timer can be found in the Android Market.

[View with PicLens]

We’ve all been there. Unfamiliar with the product, we are often unable to explain in detail what is going on, especially over the phone. And usually on the other end of the phone is an under paid customer support representative who simply reads from a script given to him by his manager. Often these representatives know very little about the product they are supporting. And, good luck if you happen to have a question that doesn’t appear on the script.

What happened next I will never forget. On the other end of the line appeared this friendly, knowledgeable technician. I described the problem I was having to him. He listened patiently, and asked a couple of targeted questions to help narrow down the issue. In no time at all, he knew exactly what was going on, and exactly how to fix it. Step by step, he clearly instructed me on what needed to be done to get my printer back online. Before you knew it, I was back in business. That technician, that single person, turned my experience around 180 degrees. I was once again a satisfied customer. Hell, I was more than satisfied.

You cannot underestimate the value of awesome customer support. I will gladly pay more for a product or service if I know that it will come with great support, as I know it will save me from headaches and grief down the road.

Typically, developers don’t serve on the front lines of customer support. However, not directly interacting with and supporting the users of your product means you are missing out on a great opportunity. An opportunity to build customer loyalty, an opportunity to understand how a customer is using your product or service, or simply an opportunity to help somebody.

I think developers should spend at least a few weeks a year directly supporting the users of their product. There are several reasons why.

You will learn how people use your product

You think you know how people are using your product? Think again. You think that user interface you designed makes perfect sense? Maybe it does to you and your co-workers, but odds are it is not as clear to your customers. When on customer support, you’ll see first hand how customers are using your product. If you find that your customers are fumbling around your product, thinking they are doing one thing but actually doing another, it’s a clear indication that your product isn’t as intuitive as it should be. Don’t write off these issues as “user errors”. Instead, get some feedback from your customer, and figure out how to make your product easier to use.

It will become obvious what features are missing, what needs to be improved, or what needs to be taken out

Your sales team, marketing team, and product team may have some great ideas regarding what direction to take your product. But, there is nothing quite like hearing it straight from the customer. This doesn’t mean that you should run out and implement every feature, and make every change requested by a customer. But, if you start to hear similar feedback while on support, it may be an indication of a legitimate need. Take note of it! Or, better yet, just do it!

Developers can fix problems…fast

Developers are in a great position to fix problems, fast. They usually know the product more intimately than anybody else on the team, have easy access to the code, and sometimes have the ability to release a patch. This gives developers the unique ability to provide amazing customer support. There have been times where I’ve fixed an issue while still on the phone with a customer, or shortly thereafter. Think about how you would feel if all of your problems were solved this quickly.

Summary

Happy customers become loyal customers. And, providing awesome customer support is one sure way to keep your customers happy. Providing customer support can be very time consuming, sometimes consuming the majority of a developer’s time (which is why I think a few weeks a year is enough). However, this should be thought of not as an expense, but an investment. An investment in the customer and the developer.

Standup Timer is a very simple application that helps keep your stand-up meetings focused, and on track. I’ve been participating in stand-up meetings of one form or another for a few years now. One thing that is constantly an issue with stand-up meetings is that they often run longer than they are supposed to. Instead of simply stating their status and moving on, participants will often start to ramble, or engage in conversations with other members of the team. Those who practice any form of the Agile software development methodology know that the value of stand-up meetings drops considerably if the meeting is constantly allowed to overflow its time constraints.

So, I created an app for that :) Standup Timer helps to keep your stand-up meetings on track by allotting each participant an equal share of time, and letting participants know when they are about to exceed, or have exceeded their time. The notification comes in the form of a bell ring for the warning, and an air horn when the time has run out. If any time is left after all participants have presented their status, then Standup Timer will keep the clock ticking, so the remaining meeting time can be used for an open discussion.

Standup Timer is easy to use. Simply provide the number of participants in the meeting, the length of the meeting, and press Start to start the timer. On the timer screen, press Next when a participant is finished presenting their status to reset the timer for the next participant. When the meeting is over, press Finished. Standup Timer will remember the number of participants and meeting length from the last time you used it. It also allows you to enable/disable sounds, and to specify when (how many seconds left in their share of time) to warn participants that they are about to exceed their share of time.

Standup Timer is free and open source. You can find the application in the Android Market, and the source code on GitHub at http://github.com/jwood/standup-timer.

[View with PicLens]

As it stands right now, Apple is king with consumers. Though not the first ones to market with an internet connected mobile device (the Blackberry has been around for a long time, and still holds the majority of the smartphone market share), Apple appears to be the first to really understand what the average, non-corporate consumer wants. A true mini computer. An open device that can play music, run applications, take photos, and provide a pleasant web browsing experience. And they did it in true Apple fashion, making the device extremely easy to use. As a result, the iPhone has become extremely popular with consumers, and is widely considered ”the device to have”.

With the majority of people still without a smartphone, much of the market remains up for grabs. Apple’s competitors are scrambling to catch up, trying to ensure that they they get a piece of the pie. But, one very important question lingers. What can Apple’s competitors offer that would give the average consumer a reason to buy their device instead of buying an iPhone? To me, the reasons are few, and becoming fewer.

(I’d love to hear your reasons in the comments. So please, chime in.)

A comparable feature set

This pretty much goes without saying. Any challenger to the iPhone crown must offer similar features to that of the iPhone. It is very unlikely that a competing device will lure anybody away from the iPhone if it is missing a feature that is now expected to be there. The device must be capable of running apps, taking photos, playing music, etc, for it even to be considered.

A better network

AT&T’s network leaves much to be desired. Having never been an AT&T customer, I can only relay the opinions of my friends and family who are AT&T customers. However, their opinions are one in the same. I’ve not heard a single word of praise when it comes to AT&T’s network. All of my friends and family with iPhones have expressed frustration that the device they love is frequently crippled by a network that is spotty and congested.

It’s no secret that Apple has an exclusive agreement with AT&T, and that agreement has an expiration date. Rumors have been circulating about a jump to another carrier, possibly Verizon, sometime next year. The more wireless carriers offering the iPhone, the less valid of a reason this will become for not purchasing one.

A comparable application ecosystem

Competing devices will need to have an application ecosystem that is at least comparable to the iPhone’s. This is no small task. There are over 100,000 applications in the App Store. Sure, several offer the same functionality, and many are of very poor quality. However, nobody can argue with Apple’s tag line of “There’s a app for that”. There really is an application, in most cases many, for everything you could possibly want to do with your iPhone.

Given their head start, beating Apple at this game will not be easy. Google’s Android OS currently stands the best chance of challenging Apple on this front, with over 10,000 applications already available. The Android OS is open, and capable of running on hardware from any manufacturer. In addition, applications written for Android are capable of running on any device that runs the OS (for the most part). Next year is going to be a big one for Android, with several new devices coming to market from many different manufacturers. Some analysts are even predicting that the number of Android devices in the hands of consumers will surpass the number of iPhones by 2012. This will no doubt attract more application developers to the platform.

However, Android has its own set of challenges awaiting. The fact that manufacturers are free to run Android on devices with very different hardware specifications (screen size, input controls, etc) poses a major challenge for application developers. Perhaps the risk of rendering thousands of existing Android applications useless by releasing a device with dramatically different hardware specs will be enough to convince manufacturers not to do it. Perhaps Google will provide a set of Android APIs that can help application developers deal with this issue. Perhaps a set of best practices will emerge as a guide for developers looking to tackle this issue. Perhaps we’ll see something similar to the PC application market in the mid-late 90’s (and the Blackberry application market today), where only certain devices will be capable of running certain applications. Only time will tell if these issues will prevent the development of the Android application ecosystem.

A killer feature

One wild card that is always in play is the killer feature. Apple’s competitors are only one, innovative, killer feature away from stealing the spotlight for themselves. By “killer feature”, I mean a feature so awesome that when you see it in action, you say to yourself, “I need one of those!”.

Version 2.0 of the Android OS took a stab at this with the introduction of Google Maps Navigation. A fantastic feature, Google Maps Navigation morphs your mobile device into a fully functional GPS unit, complete with a synthesized voice telling you where to go, real time traffic information, and several map overlays showing you the location of everything from ATMs to gas stations. But, is this a killer feature? Frankly, I’m not sure. But, its announcement was enough to cause a significant drop in the stock price of traditional GPS manufacturers, and it certainly has potential.

An incredibly easy to use device

Making devices that are intuitive and easy to use has always been one of Apple’s strengths. Look no further than the iPod for an example of this. Competing devices will need to be as easy to use as the iPhone is to appeal to the average consumer.

How do I get my music onto the device? How to I get the photos I take off? These operations should be simple and intuitive. Motorola’s new Android 2.0 device, the DROID, is seriously lacking in this area. Several steps are required to store data on or pull data off of the device:

• Attach the device to your computer
• Use the device’s menu system to instruct it to mount itself as an external drive
• Locate the files on your hard drive that you would like to store on the device
• Copy and paste the files from your hard drive onto the device
• Unmount the device

For the iPhone, the list of steps is much smaller.

• Attach the device to your computer, and let iTunes do the rest

Are the steps required to store data on the DROID too much to handle for an experienced computer user. No, of course not. But, there is still a large percentage of people out there who would struggle with completing those tasks. Believe me, I know. Many are family and friends of mine who I help complete “simple” tasks on their computers all of the time. These people make up a significant portion of the market. If you want them to buy your device, then you have to make it stupid simple to use.

Summary

Apple has set the bar high with the iPhone, very high. While I can think of several reasons why developers and techies would prefer a different device, I can’t think of many reasons why the average consumer would. And, there are a lot more average consumers than there are geeks.

But make no mistake, Apple’s competitors have the iPhone in their sights. The tide can shift very quickly in this market, especially since most people get a new phone every couple of years. Will the iPhone challengers be able make a dent in the iPhone’s market share? Or, will the iPhone be the de-facto standard for smart phones? Only time will tell.

What’s wrong with the relational database?

For 99% of the applications out there, absolutely nothing. The relational database has been the industry standard for data storage over the past 30+ years for good reason. It is an incredibly capable piece of software. Although it may not be the best tool for everything it is used for, it certainly satisfies the needs of the vast majority of applications just fine.

However, while not new, the class of applications mentioned above are becoming more common. These applications either handle enormous amounts of traffic, or deal with tremendous amounts of data. The relational database falls short in a few areas when trying to meet the demands of an application like this.

photo credit: bioxid

A single database server is usually not enough to support these requirements. Applications like this need a true database cluster, capable of adding storage space and processing power on the fly without the application even noticing. However, relational databases weren’t designed to operate in a cluster where all machines are capable of reading and writing data. This is largely due to the promises they make regarding data integrity. In order to fulfill these promises, the database needs easy, quick access to all of the data at all times to verify that duplicates aren’t being inserted, constraints aren’t being violated, etc. This quickly becomes a bottleneck when dealing with very large amounts of data.

There are techniques for scaling out relational databases, but they don’t address every concern. One popular technique is to use one or more slave databases for read requests, while continuing to funnel all write requests through the master database. The master database constantly synchronizes with the read only databases, so the data remains consistent between databases. This technique works great for read heavy applications, but does not help applications that perform just as many creates, updates, and deletes. Data sharding is another popular technique, which involves splitting the data up onto several different databases based on some criteria. But this pushes an extraordinary amount of complexity onto the application, as it is now responsible for determining which database to use for specific data sets. Master-master replication can be used to keep multiple master databases in sync, so any database server can perform read or write operations. However, for some applications there comes a point where the replication can’t keep up with the traffic.

Relational databases are also (intentionally) very strict when it comes to the structure of the data being stored. Data must be broken up into a series of rows and columns. Good object/relation mapping tools hide much of this awkwardness from us, but some applications deal with data that doesn’t map well into rows and columns. A simple key/value store is usually a better fit for applications like this.

How does the new breed address these problems?

The new breed of data stores, called NoSQL databases, make very few promises regarding data integrity. In this new model, data integrity becomes the application’s concern. By not having to enforce any complex data integrity rules, NoSQL databases can scale to levels way beyond that of a relational database. Adding more processing power or storage capacity can be as simple as adding a new machine to the cluster. The database can then store and process the data using any machine in the cluster.

In this model, the data being stored is self contained, and does not rely on any other data in the database. Therefore there is no need for one machine to know anything about the other machines in the cluster. This approach is quite different from the relational model, where data is broken up into multiple tables to eliminate duplicate data, and joined back together when being accessed.

Most of these databases subscribe to a theory called eventual consistency. In situations where duplicate information is scattered across different servers in the cluster, it is not feasible for the database to find all instances of that data and update it as a part of the original operation. Instead, the data will be replicated to the other database servers at a later time. Until that replication takes place, the application will be in an inconsistent state, where simultaneous queries fetching the same data could return different results. Although this sounds terrible, it turns out that in practice it is really not too big of a deal for most applications. Do all customers of an online retailer need to see the exact same set of product reviews 100% of the time? Probably not.

Also, because there are few promises regarding data integrity, NoSQL databases can offer data storage that is much more flexible. The database no longer has to enforce the uniqueness of a column, or ensure that the id of some referenced piece of data actually exists in the database. Some of these databases are true key/value data stores, where you can store just about anything. Others require a certain document format to be used (such as JSON or XML), but still allow you to freely change the contents of that document as you wish.

Still no one-stop-shop for persistence

Although NoSQL databases address some issues that can’t be addressed by relational databases, the opposite is true as well. The relational database offers an unparalleled feature set. While some of these features prevent it from serving the needs of the class of applications described above, they are absolutely required by other classes of applications. In some domains, data integrity is the number one concern. You need to look no further than the classic “try to withdraw money from the same account at the same time” example to justify the need for locks and transactions.

For the vast majority of applications out there, relational databases work great. There are a boat load of tools and libraries that support them, and software developers are very familiar with how to use them. It is safe to say that the relational database has secured its spot in IT departments and data centers around the world, and it isn’t going anywhere. It is far from dead.

Polyglot persistence

An increasing amount of case studies are appearing that describe how real world applications are needing the data integrity offered by the relational database in addition to the benefits offered by NoSQL databases. I believe this trend will continue, as companies are storing more data than ever, and processing that data in different ways than previously imagined.

To address these needs, some companies are beginning to run their relational database side-by-side with one or more of the NoSQL alternatives. Extremely large data sets that require scalable storage space and processing power are moved to a NoSQL database, while everything else, especially data that needs its integrity kept in-check, remains in the relational database. The term Polyglot Persistence has been used to describe the use of multiple databases within the same project.

The benefits of polyglot persistence

The benefits are somewhat obvious. By running a relational database side-by-side with a NoSQL database, you get the best of both worlds. Strict enforcement of data integrity from the relational database, and the scalability and flexibility provided by the NoSQL database. This allows you to use the best tool for the job, depending on your use case.

photo credit: jypsygen

There are a few scenarios where I’ve seen systems take advantage of polyglot persistence. The first scenario involves the need to perform some set of complex calculations on an extremely large data set. The data is either copied/moved from the relational database to the NoSQL database, or inserted directly into the NoSQL database by the application. The application can then use a cluster of NoSQL database servers can then divide the work, process the data, and aggregate the results. The more machines you have in your cluster, the less time the processing will take. The resulting data can either remain in the NoSQL database or be inserted into the relational database, depending on what needs to be done with the results.

The other scenario takes advantage of the schema-less nature of some NoSQL databases. While it is certainly possible to store a serialized data structure in a single column of a relational database, interacting with that data can be a bit more challenging than if that data were in a schema-less, document oriented database. This use case, after all, is what the documented oriented databases were designed for. These types of databases simply treat the data as a collection of key/value pairs, identified by a unique ID. The NoSQL databases provide ways in which you can add structure back into the document so the data inside the document can be queried. These databases are great for storing data that can be radically different from document to document, or data whose structure changes constantly.

The challenges of polyglot persistence

Polyglot persistence comes with its own set of challenges. While potentially getting the best of both worlds as far as features go, you get the complexity and hassle of dealing with not only multiple databases, but multiple databases models.

Determining which database to use to store certain data

With more than one database, you now have to decide where to store the data. It’s no longer a given. If you make the wrong decision, you could be looking at a painful migration from one database model to another as a result. To make this decision, you need to carefully examine how the data will be used.

Increased application complexity

Applications also face increased complexity as they now have to interface with two different (potentially very different) data stores. If done correctly, you should be able to isolate this complexity to the persistence layer of your application, freeing the rest of the application from having to know what database specific data is coming from. But, interfacing to multiple data stores could greatly increase the complexity of that data persistence layer. Your application will now need to know:

• How to connect to each of the databases
• What database to use for specific sets of data
• How to handle the different types of errors from each database
• How to map results from each database back to your application’s object model
• How to handle queries for information across databases
• How to mock out the different databases for testing
• Potentially, how to move data from one database to another

Addressing these concerns could result in a bunch of new application code, and with added code usually comes added complexity, and more bugs.

Increased deployment complexity

In addition to the increased application complexity, you will also face increased deployment complexity.

• Will you need to provision new hardware to host the new database?
• How will you backup the data in the new database?
• How will you manage and control changes to the configuration of the new database?

Training for developers and operational staff

Given that this database will likely be radically different from the relational database that your developers and operational staff are comfortable with, how will you bring them up to speed on how to use and manage this new database? And, given that the majority of the NoSQL databases are still very young, how will you keep your developers and operational staff up to speed with the latest developments on the project?

This is a big issue, especially in companies with large development and operations teams, and needs to be thought through carefully.

• Is there an expert you can hire to help you get up and running, and mentor your staff?
• Is there any training available that you can give to your staff?
• Who can you turn to for support when something goes wrong in production?

Summary

I’ve always been a big advocate of using the right tool for the right job. For the past 30 years, the relational database has been the de-facto standard for persistence. Creative people have managed to utilize and manipulate it to serve all sorts of different use cases, quite successfully. But just because you can fit a square peg through a round hole if you hit it with a big enough hammer doesn’t necessarily mean that you should.

NoSQL databases can be great tools for addressing data persistence cases that the relational database struggles with. In addition, each NoSQL database brings its own set of strengths and weaknesses to the table. They are becoming very important tools to have around, and I believe that our industry will see a steady increase in the adoption of these tools going forward.

Jumping into the Rails source, I found the offending line.

if configuration.frameworks.include?(:action_controller) &&
  ActionController::Base.session_store.name == 'ActiveRecord::SessionStore'

This code assumes that a session store is configured in your Rails app. However, this particular application has no need for a session, so we were disabling it in the configuration by specifying the following:

config.action_controller.session_store = nil

I poked around on the web for a while, trying to find another way to disable the session. No luck. It appeared that the only other way to disable the session was to properly configure a session store in your environment.rb file, and then disable it in your ApplicationController. That seemed lame. Why should I have to configure something that I want to disable?

So, I coded up a simple class to act as the session store for the application that simply raises an error if anybody tries to access the session.

class NilSessionStore < ActionController::Session::AbstractStore
  def get_session(env, sid)
    raise NotImplementedError, "NilSessionStore: No session configured"
  end

  def set_session(env, sid, session_data)
    raise NotImplementedError, "NilSessionStore: No session configured"
  end
end

I then configured the application to use this class as the session store.

config.action_controller.session_store = :nil_session_store

Nice and simple, and it keeps me from having to configure something I never plan to use.

<< Part 5: Application Changes

Addressing the remaining issues

We were almost there. After modifying the code to talk to CouchDB, TextMe was successfully pulling data from CouchDB in our development environments. There were just a few remaining issues that needed to be addressed before we could deploy CouchDB to production.

Reducing the view sizes on disk

As I mentioned in a previous post, the amount of disk space consumed by the views was a big problem. If we didn’t do something, we were sure to run out of disk space when migrating our 30 million row messages table to CouchDB.

We determined that it was not what we were emitting from our map functions that was killing us, but how many times we were emitting it. Each of the views emitted a key/value pair for every document in the database. At 30 million documents and 8 views, that ends up being a crap load of key/value pairs.

My colleagues Dave and Jerry took a detailed look at the problem, and came up with a solution. They determined that there was simply no need to be emitting data for each document in the database. While this would give us views that could report statistics by the second, our application only supported presenting statistics by the minute. Even if we were able to support statistics at this level of detail, we doubted our customers would even need it. It was simply not worth the disk space.

So, Dave and Jerry modified the import job described in the previous post to roll up several key statistics by the minute as it was building the documents. When the job finishes processing all of the documents for that minute, it creates a summary document containing all of the rolled up statistics, and adds it to the database. Then, they changed the map functions to only consider these summary documents.

This solution was able to dramatically reduce the sizes of the views on disk, while still supporting the current application functionality. Since we are still persisting all of the original documents to CouchDB, it is possible to add a new statistic to the summary documents should we ever need to.

Oh, and we also picked up two new terabyte database servers, just in case :)

Paginating records in CouchDB

Like many Rails applications, we were using the popular will_paginate gem to paginate results from the database. Given the size of our data sets, pagination was an absolute necessity to keep from using up every last bit of memory.

CouchRest has a Pager class that paginates over view results, but it is in the CouchRest Core part of the library and doesn’t integrate too well with the object model part of the library. It simply returns the view results as an array of hashes. We were hoping to see a solution that would give us back an array of the corresponding ExtendedDocument objects. We were also trying to keep our application from having to know about CouchDB outside of the classes described in the previous post. Having completely different pagination strategies for the two databases would make that more difficult.

So, I decided to write some new pagination code that supported the will_paginate interface and integrated a little better with the object model part of CouchRest. I had a quick solution that same day which fetched view results and handed back an array of the corresponding ExtendedDocument objects. I then spent some time over the next two weeks modifying the code to integrate a little better with CouchRest and add support for CouchRest views, which we weren’t using.

With the new code in place, we can now paginate over a set of contest entries without having to know what database they are coming from.

ContestCampaignEntryDelegate.contest_campaign_entries.paginate(
  :page => 1, :per_page => 50)

This pagination code eventually made it into CouchRest.

Going live

With the remaining issues addressed, it was time to start the production migration. One at a time, we manually started the jobs to move the data from MySQL to CouchDB. When one job completed, we would start the next. As I mentioned before, building the views is very resource intensive. We didn’t want to completely bog down the production machine we were using to do the migration by running multiple jobs at once.

Moving the archived data from MySQL to CouchDB and building all of the views took about a week (a day for this table, a couple of days for that table, etc). Overall, it was a fairly smooth process.

For the initial import, we did not purge any of the data from MySQL. Since we needed to wait until our CouchDB databases were fully populated with all views built before we could start using them, the application needed to continue working with the data in MySQL while the migration was in progress. In anticipation of the eventual switch from MySQL to CouchDB, I added a flag in the application configuration that told the application if it should pull archived data from CouchDB. Once all of the data had been imported and all of the views had been built, we flipped the switch.

With the pouring of a celebratory beer, we watched as our application began pulling data from CouchDB in production. It was time to relax :)

The results

I really wish we had taken the time to record how long our “troublesome” pages were taking to load before the move to CouchDB. Sadly, we did not. All I can say is that pages that used to occasionally time out were now loading in a few seconds. Since the migration, we have also implemented a few new features that would simply not have been possible without CouchDB due to database performance issues.

The database performance issues we set out to address seem to be a thing of the past. If new ones pop up, I’m confident that we could once again utilize CouchDB to address them.

What’s next

This project was focused on addressing database related performance issues that we were facing in production. With these issues out of the way, and our CouchDB infrastructure built-out and proven, we will soon be building even more reporting capabilities that would have simply killed our old database. TextMe customers will soon be able to view their data in more ways than they could have imagined.

I am also working on a project that takes advantage of CouchDB’s schema-less nature to let our customers store and utilize data they collect from their customers. Such a feature, which essentially lets customers define their own schema, would have been a challenge to implement in a relational database. With CouchDB, its just a document.

Thoughts about this project, and CouchDB

I learned a ton while working on this project. While vaguely familiar with “NoSQL” databases before this project, I have just recently become aware of all of the alternatives available. With the enormous amount of data companies are beginning to collect and process, I’m sure that CouchDB and its NoSQL friends will soon become a common component in the operational environments of most companies.

The CouchDB community has been great. The CouchDB and CouchRest mailing lists are extremely active, and have been very helpful. The committers on both of these projects are active, and always eager to help. I’d specifically like to call out Jan Lehnardt and Chris Anderson from the CouchDB project. Jan has commented on a few of these posts, encouraging me to keep writing. He also suggested a more efficient implementation of the CouchRest pagination code I wrote, which I quickly implemented. Chris left a comment on the first post in this series thanking me for writing about CouchDB, and offering his assistance if I needed it. I actually took Chris up on that offer when we were running into issues regarding the sizes of the views on disk. He was quick to reply, offering several suggestions. I’d like to thank Jan and Chris for their support and encouragement.

NoSQL databases are here to stay, and CouchDB is truly unique in this area. The way it handles views, and its support for replication/synchronization set it apart from the others. There are already several large projects, like Ubuntu One, that are relying on CouchDB to deliver what nobody else can. Because of this, I’m sure CouchDB has a very bright future ahead of it.

My wife Beth and I always wanted a dog, and knew that as soon as we got a house, a dog would be soon to follow. On a trip in to see my parents, we stopped by a local animal shelter to take a look at the dogs. We came upon this crate containing 5 puppies, the offspring of what appeared to be a German Shepherd and a Rottweiler. I liked German Shepherds, and Beth like Rotties. We quickly took to the sole black puppy in the litter, and started the paperwork necessary to take her home.

I remember the day I picked her up from the shelter like it was yesterday. I took the day off of work to go and pick her up. The little ball of fur weighed only 10 pounds, and was sleeping soundly in my arms as I filled out the last bits of paperwork. The first thing we did was drive up to see Beth at work. She whined the entire way. We named her Bacardi.

Childless for the first few years of our marriage, Bacardi was our baby. She was a first class member of the family. We did everything with that dog.

A more loyal friend you could not find. Whenever Beth was sick, Bacardi would refuse to leave her side. Those two had a connection like I’ve never seen. It was simply amazing to watch. An eternal puppy at heart, Bacardi was always playful, even in her older years. Affectionate like no other, she’d lick you until your skin came off. Her whip-like tail would start beating against the floor the minute she saw you. Even on my worst days, Bacardi was always able to put me in a good mood. She was always happy to see me, and always put a smile on my face.

When we welcomed our first son Dylan, we were repeatedly warned by friends and family not to let Bacardi too close. I know that many of these feelings stemmed from the fact that she was half Rottweiler, and Rottweilers don’t have the best reputation as a family dog. But Beth and I were confident that we had raised Bacardi to be kind and gentle. Needless to say, Bacardi proved them all wrong. She was both a friend and a guardian to our little boy, like she was for us. I remember one day when Dylan was lying in his bassinet, no more than a week or two old. The bassinet was just a bit taller than Bacardi, and she tried like hell to take a look inside, to see what was making all that noise. She was actually able to tilt the bassinet slightly with the side her head, and look at Dylan with her one eye.

In the following years, she could not have been a better family Dog. She let Dylan do anything to her. He’d ride her like a horse, lie on her side, pull her ears…everything you could imagine, and she just laid there. If she ever had enough, she would simply walk away. Dylan loved Bacardi as much as we did.

Three years later, Beth and I welcomed our daughter Chloe. Chloe was ill quite a bit the first year of her life. We attributed this to Dylan brining home “presents” from pre-school, but started to suspect something else when she was still getting sick over his long breaks from school. We found out that Chloe was an asthmatic, and allergic to Bacardi. We knew we had no choice; we had to find Bacardi a new home. Knowing that this was the right choice didn’t make it any easier. Luckily, Beth’s uncle who lives just 10 minutes from us agreed to take her in. His daughter had always adored Bacardi, and we knew that she would be treated well there. Shortly after agreeing to take her in, Beth’s uncle came to pick her up. That was a terrible day.

In the few short months Bacardi lived with Beth’s uncle, we continued to see her on a regular basis. She was always happy to see us. Even though we knew she was being treated great, a part of us always felt terrible when we left her behind. We wanted so badly to take her home with us, but we knew that we couldn’t. And, it didn’t help to see that she clearly wanted to come with us. Sometimes, it felt like losing her all over again.

Just this past Thursday, Beth’s cousin called to tell us that Bacardi was ill. Beth took her to the vet on Friday, and the vet said it was either a really bad infection or cancer. As Saturday and Sunday passed, her condition quickly got worse, and it became clear that it was not an infection. She was in the late stages of a battle with lymphoma. Early this morning, around 3am, we said goodbye to our friend, and put and end to her suffering.

In Bacardi’s 7 ½ years on this earth, she touched so many lives. We will always cherish the memories of the times we shared with her. She was a great friend, and one amazing dog.

Bacardi, I love you so much, and miss you terribly. Goodbye old friend.

[View with PicLens]

Compared to the challenges we faced with views, modifying TextMe to interact with CouchDB was very straight forward. This post describes how we changed the TextMe code in order to use CouchDB as an archive database. Since TextMe is a Ruby on Rails application, much of the content in this post references Ruby/Rails specific libraries and frameworks. However, I feel the general concepts could be applied to any development platform.

A quick recap

Before I dive into describing how we modified our application to work with CouchDB, I’d like to quickly recap exactly what we were trying to do (see the first post in this series for a more detailed overview). TextMe is a mobile marking application. You can use it to manage SMS powered voting campaigns, contests, subscription lists, and more. The majority of these campaigns have a pre-determined lifespan. Once the campaign is over, the data collected is primarily used to calculate statistics on the campaign. This data is very important to our customers.

A few of our database tables were getting quite large, and starting to affect the performance of the application. Tuning the queries didn’t seem to help much. So, we turned to CouchDB and its views to help us store and aggregate this large amount of data.

While a campaign is still active, we do more than simply calculate statistics on the data. For example, our contest campaign needs to ensure that a winner is properly selected. The winner could be the Nth entry, every Nth entry, the first N entries, etc, depending on how the campaign is configured. Selecting the wrong winner, or more winners than we are supposed to select would obviously be bad. So, we rely on the data integrity features provided by MySQL to help us do this correctly. However, once the campaign is over, the data is only used for statistics.

Based on these requirements, we decided to use CouchDB as an archive database. When a campaign completes, we could move the data out of MySQL into CouchDB. This would make the MySQL tables smaller and more efficient, and allow us to take advantage of CouchDB’s views to handle the statistics. But, this also meant that our application would have to interact two databases instead of one, and for the most part, be ignorant of which database the data was coming from.

Configure the application to access CouchDB

Before our application can talk to CouchDB, we need to tell it a little bit about our CouchDB installation. The CouchRest-Rails plugin aims to make this as easy as possible for Rails applications. CouchRest-Rails provides the necessary hooks that allow you to specify your CouchDB configuration in a couchdb.yml file, which serves the same purpose as the default database.yml file used by Rails. Simply update this file with your CouchDB connection information, and you’ll be able to easily connect to CouchDB from within your application.

CouchRest-Rails also provides a series of Rake tasks that help you manage your databases and views.

Define the documents

The very first thing you need to do when moving data to CouchDB is to figure out what your documents will look like. I talked about this in CouchDB: Databases and Documents, so I won’t cover it again here.

Write code to create documents from relational database backed data objects

Once you know what the documents are going to look like, you need to write some code that will convert your RDBMS backed objects into a document, and store it in CouchDB.

We decided to use CouchRest to help us interact with CouchDB. CouchRest consists of two main parts: code to interact directly with CouchDB via a set of APIs just above CouchDB’s HTTP API (known as CouchRest Core), and code that allows you to create an object model backed by CouchDB. The ExtendedDocument class is the cornerstone of the object model code. ExtendedDocument is like ActiveRecord::Base in Rails. It serves as the base class for CouchDB backed objects. It provides convenient ways to define document properties, access views, define life cycle callbacks, create documents, save documents, destroy documents, paginate view results, and more.

A class extending ExtendedDocument simply needs to define the properties that make up its document.

class ArchivedContestCampaignEntry < ExtendedDocument
  use_database :contest_campaign_entry_archive

  property :campaign_id
  property :user_id
  property :entry_number
  property :winner
end

Then, all it takes to save a document in CouchDB is to create an instance of this class, set its properties, and call the object's create method.

Determine how data will be moved to CouchDB

Now that we have code that can convert RDBMS objects into documents, we need to figure out how to actually get those documents into CouchDB. This step will likely be dependent on how you plan on using CouchDB. For us, we decided it would be best to “archive” records after their corresponding campaigns have been over for 48 hours or more. So, we created a nightly cron job to fetch all non-archived campaigns that have been over for 48 hours or more, and move their corresponding entries to CouchDB. When a campaign's entries have been moved, an “archive” flag is set on the campaign itself, so the application knows to fetch its entries from CouchDB instead of MySQL.

One important item to point out is that CouchDB supports a bulk save operation. This operation allows you to save a batch of documents with a single HTTP request. This is a big time saver, as executing one HTTP request is obviously much quicker than executing several thousand. Our archive cron job takes advantage of this. When archiving entries for a particular campaign, we will build one document for each entry record, and then toss that document into an array. When that array exceeds a certain size, 2,500 in our case, a single bulk save request is sent to CouchDB with the array of documents. This dramatically decreases the number of HTTP requests sent to CouchDB, and the amount of time required to add data to CouchDB.

In addition, our archive job will pause to rebuild all of the views in the database after 100,000 new documents have been inserted, as well as at the end of the job. The final view rebuild is necessary since all of the view queries done from within the application ask for stale data, which will not trigger a view update. We never did any research to determine if this was better or worse than waiting until the end of the job, which could produce over 500,000 new documents, to rebuild all of the views. This step was simply driven by the gut feelings of the three engineers working on the project. I'd be interested in hearing from you if you have done any research to determine if incremental view building is more or less efficient than a big bang view rebuild after a large import has completed.

Replacing SQL queries with CouchDB views

Next, we changed the application to support the substitution of SQL queries with CouchDB views. We did this in several steps.

Identify queries being performed on the data you want to move

The first step in replacing SQL queries with CouchDB views is identifying all of the queries being performed on the data you plan on moving to CouchDB. This took a few hours to do, but was not difficult by any means. We simply searched the code for all instances of the ActiveRecord class name and the MySQL table name for the tables with data being moved. We also tracked down all ActiveRecord associations that were made to that particular table. We then made a note of what the queries did, and how they were used.

Abstract away the query

After the queries had been identified, we moved the execution of all queries to a new class. This freed the rest of the application from having to know if the data being fetched lived in MySQL or CouchDB. The new class would make that decision, delegating to the ActiveRecord class if the data was in MySQL, or the ExtendedDocumenet class if the data was in CouchDB. To start off, we simply delegate to the ActiveRecord class since we have not yet implemented the CouchDB views.

class ContestCampaignEntryDelegate
  def self.find_all_by_campaign_id_and_winner(campaign_id, winner)
    # Delegate to the ActiveRecord object
    ContestCampaignEntry.find_all_by_campaign_id_and_winner(campaign_id, winner)
  end
end

Build views to replace the queries

Now that we have the complete list of queries performed on the data that we wish to archive, we can begin building the necessary CouchDB views to support those queries for archived campaigns. I wrote about CouchDB views in previous posts. See those posts for more information.

Add methods to the ExtendedDocument class to query the views

CouchRest gives you a few options when it comes to creating and accessing your views.

One option is to use the view_by method available on all classes that extend ExtendedDocument. view_by not only makes the views easily accessible via the code, but it will also take care of creating and storing the view in the database.

In its simplest form, view_by will generate the necessary map function based on the parameters you specify. This example from the CouchRest documentation shows the map function that will be generated when view_by :date is called inside a class named Post:

function(doc) {
  if (doc['couchrest-type'] == 'Post' && doc.date) {
    emit(doc.date, null);
  }
}

view_by also lets you specify compound keys (view_by :user_id, :date) and any parameters that you wish to be used when you query your view (:descending => true).

If you need to do something a little more complicated, view_by will also let you specify the map and reduce functions to use. Here's another example from the CouchRest documentation:

 # view with custom map/reduce functions
 # query with Post.by_tags :reduce => true
view_by :tags,
  :map =>
    "function(doc) {
      if (doc['couchrest-type'] == 'Post' && doc.tags) {
        doc.tags.forEach(function(tag){
          emit(doc.tag, 1);
        });
      }
    }",
  :reduce =>
    "function(keys, values, rereduce) {
      return sum(values);
    }"

Another option for creating and accessing views is to use CouchRest Core. CouchRest Core, as described above, is a raw, close to the metal set of APIs that let you interact with CouchDB. These APIs let you do basically anything, including creating and accessing views. This example from the CouchRest documentation shows how you can create and query a view using CouchRest Core:

@db = CouchRest.database!("http://127.0.0.1:5984/couchrest-test")
@db.save_doc({
  "_id" => "_design/first",
  :views => {
    :test => {
      :map =>
        "function(doc) {
          for (var w in doc) {
            if (!w.match(/^_/)) emit(w,doc[w])
          }
        }"
      }
    }
  })
puts @db.view('first/test')['rows'].inspect

For accessing our views, we decided to go with a hybrid approach. We didn't really feel comfortable storing our map and reduce functions inside the application code. Doing so made it less clear on how we could gracefully introduce new views or update existing views in production, keeping in mind that some of these views could take hours or days to be built for the first time. Instead, we stored our map and reduce code outside of the application, and used CouchRest-Rails to help us get those views into the database. This allows us to push new or updated views independent of the application, giving us time to build the views before anything tries to access them.

Since the views are already in the database, we decided to use CouchRest Core to access them. We created a class called ArchivedRecord to make working with CouchRest Core a little easier. ArchivedRecord contains methods that do type conversions, manage bulk save operations, incrementally regenerate the views, and more. It also contains a series of methods that help with executing views with similar behavior. For example, there are methods that will simply return the number of rows returned by a view, execute a view for a specific timeframe using the dates stored in the documents, etc. These abstractions also handle any errors that could pop up when accessing a view. Our application code uses the abstractions provided by ArchivedRecord to access the views.

Change the delegate class to call the ExtendedDocument class for archived data

Now that our views can be accessed via the application code, we can modify the delegate class to call the ExtendedDocument object's query method to fetch data for campaigns that have been archived.

class ContestCampaignEntryDelegate
  def self.find_all_by_campaign_id_and_winner(campaign_id, winner)
    campaign = ContestCampaign.find_by_id(campaign_id)
    if campaign.archived?
      ArchivedContestCampaignEntry.find_all_by_campaign_id_and_winner(campaign_id, winner)
    else
      ContestCampaignEntry.find_all_by_campaign_id_and_winner(campaign_id, winner)
    end
  end
end

Deal with the ActiveRecord associations

The last piece of the puzzle is to deal with the ActiveRecord associations. ActiveRecord associations are magic little things that make a record's associated data accessible via methods on an instance of the ActiveRecord class. For example, if I wanted to declare an association between a contest and its entries, I would simply declare the following at the top of my ContestCampaign class:

has_many :contest_campaign_entries

ActiveRecord takes care of joining the contest_campaign_entries table with the contest_campaigns table, and making all of the related campaign entries available via a call to some_contest_instance.contest_campaign_entries.

This will not work for us, as the contest_campaign_entries table will not contain any data for archived contests. So, we need to handle associations differently.

Instead of using the above code to create the association, we use the following:

has_many :active_contest_campaign_entries,
  :foreign_key => 'contest_campaign_id',
  :class_name => 'ContestCampaignEntry'

This more verbose version tells ActiveRecord that we want to setup an association, named active_contest_campaign_entries, on the class. Since we're circumventing the convention of naming the association after the foreign key to the associated data (which is in turned named after the associated table), we need to specify the foreign key to use, and the name of the class that backs that table.

To keep from breaking the existing code that uses the contest_campaign_entries method to obtain related entry data for a contest, we define a new method on the class with that name to fetch the associated data. The new method simply calls the corresponding method on the delegate class, which will pull the associated entries from MySQL or CouchDB, depending on if the campaign has been archived.

class ContestCampaign
…
  def contest_campaign_entries
    ContestCampaignEntryDelegate.contest_campaign_entries(self.id)
  end
…
end

class ContestCampaignEntryDelegate
…
  def self.contest_campaign_entries(campaign_id)
    campaign = ContestCampaign.find(campaign_id)
    if campaign.archived?
      ArchivedContestCampaignEntry.find_all_entries(campaign_id)
    else
      campaign.active_contest_campaign_entries
    end
  end
…
end

ActiveRecord supports other associations besides has_many. These other associations also add methods to the class that will fetch associated data from the database. In our case, some of this associated data is going to remain in MySQL. CouchRest will not (and should not) automatically fetch the corresponding data from MySQL, so we needed to handle this ourselves.

In our documents, we store the ids of the associated data still in MySQL (see campaign_id and user_id in the document snippet below). Because we have associations setup between the ContestCampaignEntry class and the ContestCampaign and User classes, ActiveRecord adds methods named campaign and user to ContestCampaignEntry that will fetch the associated objects. We need to do the same in our ExtendedDocument class.

class ArchivedContestCampaignEntry < ExtendedDocument
  use_database :contest_campaign_entry_archive

  property :campaign_id
  property :user_id
  property :entry_number
  property :winner

  def user
    @user ||= User.find_by_id(user_id)
  end

  def campaign
    @campaign ||= ContestCampaign.find_by_id(campaign_id)
  end
end

The user and campaign methods in the class above will take the ids stored in the document and fetch their corresponding objects from MySQL. In our case, these values will never change for an archived entry, so we hold on to the objects as instance variables to avoid doing additional queries when they are referenced again.

Make the ExtendedDocument class act like the ActiveRecord class

As I stated above, one of the goals was to make it so the application code does not need to know which database the data is coming from. Since the data can be returned as instances of two different classes, ContestCampaignEntry or ArchivedContestCampaignEntry, we need to make sure that both of these classes implement the same methods, and behave the same way. Failing to do so could result in hard to find bugs, or straight up exceptions.

One group of methods to pay extra attention to are the convenience methods that ActiveRecord adds to the class based on the attribute types in the database. An example of this is the attribute? accessor method that is added for boolean attributes. All attributes get an accessor named after the column in the database, but boolean attributes get an additional accessor, containing a “?” at the end. I personally use the “?” variation of the accessor method all of the time, as I feel it makes the code easier to read and understand.

CouchRest on the other hand is not able to determine in advance the data types of the properties you have stored, since CouchDB is a schema-less database. So, it is not able to do anything special for properties of a given type unless you specifically tell it what the type is. CouchRest does allow you to specify a type when you declare the property, but the current release (version 0.32) only uses this to cast property values into their proper type after they are fetched from the database. I've submitted a patch that will generate “?” accessor methods for properties with a type specified as :boolean. However, this is just one example of how your ExtendedDocument class could be subtlety different from the corresponding ActiveRecord class.

Summary

As I stated at the beginning of this post, changing the application to work with CouchDB was much more straightforward than getting the views to work as expected. Perhaps this is because I'm a developer, and not a DBA. But, great libraries like CouchRest and CouchRest-Rails certainly go a long way in helping to write clear and concise code that interacts with CouchDB. I can only hope that other programming languages have, or soon will have, libraries like these. The fact that CouchDB has a great API built on a protocol that everybody can talk, HTTP, certainly makes it possible.

In the previous post, I wrote about the many features of CouchDB views. In this post, I will describe the challenges we faced when replacing MySQL queries with CouchDB views.

Map/Reduce

One of the largest challenges with CouchDB views is simply wrapping your brain around the map/reduce model. If you’ve spent any significant amount of time in the relational model, this can be quite a task. Do not underestimate it. Give yourself plenty of time to make this adjustment. In my opinion, setting aside one or two full weeks to read about and experiment with views would not be excessive. It really is a whole new world. After several weeks, I’m still not 100% sure how to use the map/reduce model to its fullest potential. In fact, there were a few queries that I could not figure out how to implement as views. Because of that, I had to keep around an archive table in MySQL, and I run those few queries against that table.

Javascript

If you don’t know Javascript, you may want to tack on another week to the learning curve. Javascript is an incredibly powerful language. However, in its raw form it is fairly basic and can take some getting used to. It does help that you don’t need to write too much Javacript to implement most map and reduce functions. However, if you’re new to Javascript, get ready to do some research.

There are view servers available in other programming languages, and it is easy to configure CouchDB to use them. But, CouchDB is still young and under heavy development, and these alternative view servers are not supported by the CouchDB team. So, use them at your own risk.

SQL

As I mentioned in the previous post, views are powerful and flexible. But, views are not nearly as flexible as SQL. SQL has been in development for decades. Even today, it continues to evolve as a language. You can do a ton with SQL. As of right now, views simply cannot rival this flexibility. The CouchDB team continues to add built-in Javascript functions to help write map/reduce code, and there is even talk about supporting map/reduce/merge. But as of right now, the feature sets are not even close. It is very difficult for any new technology to enter the game with the same, or even a comparable feature set to such a battle-hardened veteran. And to be honest, I highly doubt that the CouchDB team is even trying to match SQLs feature set. After all, CouchDB is not meant to be a replacement for the relational database. However, this is an important point to consider if you think your current relational database backed application might be a good fit for CouchDB.

Multiple views, one design document

Views live in documents called “design documents”. Views within the same design document share a B-Tree data structure. This means that when one view in the design document is built, they all are built. So, careful planning is required to make sure unrelated views do not live in the same design document. You would not want the re-building of one view to delay the accessibility of another, totally unrelated view.

Building/Indexing views

Views can take a L…O…N…G time to build from scratch. The view building process is also very resource intensive. This becomes less of an issue once the views have been built, as views are updated incrementally. It really only comes into play when you are adding many, many documents to a CouchDB database between view builds. However, one place where this is an issue is ad-hoc queries. Every week or two, I’ll get a request from a customer for data that is not available via our web application. While we’ll throw that request onto the product backlog so it is eventually available via the application, it doesn’t change the fact that our customer needs that information now. We usually satisfy such requests by firing up the MySQL client, and running one or more ad-hoc queries. This simply is not feasible using CouchDB views, especially if you are working with a large database containing millions of documents. CouchDB does support “temporary views”, which are ad-hoc views that you can build and execute on the fly. However, temporary views are not recommended for production use, because they need to be built before they can get you the data you need. This could take hours, or days depending on the size of you database and the processing power of your database server. Temporary views are meant as a way to test new views in development which will eventually be saved into a design document, and not for running ad-hoc queries.

View sizes on disk

I’ve already mentioned that each design document is stored in its own B-Tree, completely separate from the B-Tree that holds the documents in the database. These data structures can become quite large, especially if you have a ton of documents in your database. A large database combined with several design documents can take up quite a bit of disk space. For example, our main messaging database consisting of around 30 million documents is 20GB on disk. The 8 design documents for that database combine for a total size of 35GB. This brings the grand total, for the documents and the views, to 55GB. That’s a whole lot of disk space. CouchDB sacrifices disk space for performance, which is a good tradeoff, as disk space is cheap and virtually limitless these days. But sadly, this is not the case for everyone. To add enough storage capacity to handle this database, the other large databases, a mirror/backup of these databases, and still have room to grow, we were looking at an additional several hundred bucks a month in charges from our hosting company for the additional disk space. That can be a lot to handle for a small company. Some larger companies use SANs or similar storage devices that offer unmatched redundancy and reliability. However, these devices will often have a fixed maximum amount of storage, and can be very expensive to upgrade one the storage capacity is maxed out. Justifying the use of so much disk space on such an expensive resource could prove difficult.

Summary

Views were without a doubt the hardest part of this project to get right. My colleagues Dave and Jerry are still hard at work trying to find creative ways to reduce the size of the views on disk. We’re very happy with the performance boots that we’ve seen using views in our testing environment. But unless we can find a way to reduce the disk usage, or find more affordable storage options, we may never see these performance boosts in production.

Views are what attracted us to CouchDB. If you’ve been reading the posts in this series, you already know that CouchDB is a document oriented database, and that documents themselves don’t have any official structure beyond the structure enforced by JSON. Views are what provide the necessary structure so that you can run queries on your data.

CouchDB has several strong points, including its efficient B-Tree data store implementation and replication/synchronization support. These strong points already set it apart from other, more traditional databases. However, we came for the views, because we saw views as the potential answer to our database performance woes.

CouchDB builds views using a map/reduce algorithm. When building a view, CouchDB will feed all documents that are new or have changed since the last time the view was built through a map function. The map function selects the documents of interest for that particular view. Then, optionally, a reduce function is run to calculate some aggregate statistics on the documents that have been selected (counts, sums, etc). There are several places you can go on the web for more information about how CouchDB views work.

A large part of the performance issues we are trying to address are being caused by repeatedly running the same database queries against very large tables, where the vast majority of the data in those tables has not changed since it was inserted. The last part of that statement is very important. The data has not changed since it was inserted, and due to the nature of these tables, it probably never will. It was very wasteful for us to keep running the same calculations on that old data.

This is where CouchDB views come in. When CouchDB generates a view, it stores the result of the view on disk in a B-Tree data structure, which is very efficient to access. CouchDB will only re-generate that view when documents that match the criteria specified in the map function are changed or added. And, CouchDB will only need to update the view for the changed/added documents. It will incrementally update the view’s index, so it doesn’t have to start from scratch every time. This makes views especially ideal for large data sets.

Using views, we can replace all of the queries we were performing on these tables, and the calculations would be performed once, and then stored. Accessing that data would be as simple as issuing a single HTTP request, which would efficiently pull the data from the view’s B-Tree. In other words, it would be fast, and very efficient.

CouchDB views are also very flexible. The output of the map function is a key/value pair. That key/value pair can be anything…data from the document, hard coded values, whatever. This flexibility allows you to create complex keys, such as a JSON array of values. Using the view API, you can specify ranges of keys when executing your query, fetching only the data that you want. You also have the ability to group complex keys by the first n elements of the key, and run the reduce function on those groups of data. This enables you fetch aggregate data on multiple levels, and allows you to support multiple queries with a single view. For example, we need to calculate the number of SMS messages sent by a particular account by minute, hour, day, month, year, etc. Using CouchDB’s view engine, we can have our map function emit a key of [account_id, year, month, day, hour, minute] and a value of 1 for each document in our messages database. Our reduce function simply sums all of the values for a matching key, using the provided sum function. Here is how simple the map/reduce code is for this view:

Map

function(doc) {
    datetime = doc.created_at_utc;
    year = parseInt(datetime.substr(0, 4));
    month = parseInt(datetime.substr(5, 2), 10);
    day = parseInt(datetime.substr(8, 2), 10);
    hour = parseInt(datetime.substr(11, 2), 10);
    minute = parseInt(datetime.substr(14, 2), 10);
    emit([doc.account_id, year, month, day, hour, minute], 1);
}

Reduce

function(keys, values, rereduce) {
    return sum(values);
}

Using the grouping feature of the view API, we can easily fetch message counts for this account by year, month, day, hour, or minute, by simply specifying how many levels of the key we would like to group together. For example, to get a breakdown of messages sent for a particular account on each day in May of 2009, we would simply include the following parameters in our URL when accessing the view: startkey=[1,2009,5]&endkey=[1,2009,5,{}]&group_level=4. These parameters tell the view that we only want to consider messages for account number 1 that were sent or received in May of 2009, and that we’d like the reduce results grouped by the 4th parameter in the key, which is the day of the month. This would return something like:

{"rows":[
{"key":[1,2009,5,1],"value":{"OK":{"MO":11,"MT":13}}},
{"key":[1,2009,5,2],"value":{"OK":{"MO":9,"MT":9}}},
{"key":[1,2009,5,3],"value":{"OK":{"MO":6,"MT":10}}},
{"key":[1,2009,5,4],"value":{"OK":{"MO":9,"MT":9}}},
{"key":[1,2009,5,5],"value":{"OK":{"MO":11,"MT":11}}},
{"key":[1,2009,5,6],"value":{"OK":{"MO":14,"MT":17}}},
{"key":[1,2009,5,7],"value":{"OK":{"MO":10,"MT":12}}},
{"key":[1,2009,5,8],"value":{"OK":{"MO":8,"MT":12}}},
{"key":[1,2009,5,9],"value":{"OK":{"MO":14,"MT":14}}},
{"key":[1,2009,5,10],"value":{"OK":{"MO":6,"MT":8}}},
{"key":[1,2009,5,11],"value":{"OK":{"MO":10,"MT":12}}},
{"key":[1,2009,5,12],"value":{"OK":{"MO":9,"MT":11}}},
{"key":[1,2009,5,13],"value":{"OK":{"MO":9,"MT":9}}},
{"key":[1,2009,5,14],"value":{"OK":{"MO":13,"MT":20}}},
{"key":[1,2009,5,15],"value":{"OK":{"MO":6,"MT":7}}},
{"key":[1,2009,5,16],"value":{"OK":{"MO":15,"MT":15}}},
{"key":[1,2009,5,17],"value":{"OK":{"MO":5,"MT":8}}},
{"key":[1,2009,5,18],"value":{"OK":{"MO":8,"MT":8}}},
{"key":[1,2009,5,19],"value":{"OK":{"MO":9,"MT":13}}},
{"key":[1,2009,5,20],"value":{"OK":{"MO":7,"MT":7}}},
{"key":[1,2009,5,21],"value":{"OK":{"MO":6,"MT":12}}},
{"key":[1,2009,5,22],"value":{"OK":{"MO":22,"MT":28}}},
{"key":[1,2009,5,23],"value":{"OK":{"MO":6,"MT":8}}},
{"key":[1,2009,5,24],"value":{"OK":{"MO":2,"MT":4}}},
{"key":[1,2009,5,25],"value":{"OK":{"MO":2,"MT":2}}},
{"key":[1,2009,5,26],"value":{"OK":{"MO":14,"MT":16}}},
{"key":[1,2009,5,27],"value":{"OK":{"MO":11,"MT":15}}},
{"key":[1,2009,5,28],"value":{"OK":{"MO":8,"MT":12}}},
{"key":[1,2009,5,29],"value":{"OK":{"MO":7,"MT":7}}},
{"key":[1,2009,5,30],"value":{"OK":{"MO":4,"MT":5}}},
{"key":[1,2009,5,31],"value":{"OK":{"MO":4,"MT":6}}}
]}


Views are re-built when they are accessed, and not when new documents are added to the database or existing documents are changed. However, you do have control over when views are built. If you specify stale=”ok” when accessing your view, CouchDB will not check to see if the view needs to be re-built. It will simply return results from the last time the view was built. We took advantage of this feature when writing the application code to access the views. In our case, data is only added to the database once a day, and it is added by a background job. When the job is finished inserting data into the CouchDB database, it triggers the views to re-build themselves by accessing all of the views in the database (a few at a time), without specifying the stale=”ok” flag. Since this background job takes on the responsibility of updating the views after it inserts new data, the rest of our application can always specify stale=”ok” when accessing the views. This keeps the queries executed by the application fast, even when views are in the process of being re-built.

Views are powerful, and offer a tremendous amount of flexibility. However, they come with their own set of challenges. In the next post, I will talk about some of the challenges we faced when attempting to replace our SQL queries against a MySQL database with CouchDB views.

CouchDB is a document oriented database. A document oriented database stores information as documents of related data. All of the data within a document is self contained, and does not rely on data in other documents within the database. This can be quite a shift if you’re used to working with a relational database, where data is broken up in to multiple rows existing in multiple tables, limiting (or eliminating) the duplication of data. Although radically different, the document oriented approach is a very good fit for many applications. For some applications, data integrity is not the primary concern. Such applications can work just fine without the restrictions provided by a relational database, which were designed to preserve data integrity. Instead, giving up these restrictions lets document oriented databases provide functionality that is difficult, if not impossible to provide with a relational database. For example, it is trivial to setup a cluster of document oriented databases, making it easier to deal with certain scalability and fault tolerance issues. Such clusters can theoretically provide you with limitless disk space and processing power. This is the primary reason why document oriented databases (or key/value pair databases) are becoming the standard for data storage in the cloud.

There are plenty of articles on the web describing the benefits of using a document oriented or key/value pair database, so I won’t re-hash any of that information here.

Databases

Creating a new database in CouchDB is a simple process, with no overhead. In fact, it’s as simple as issuing a single HTTP request.

curl -X PUT http://127.0.0.1:5984/my_database


There appears to be no “penalty” for hosting many databases within the same CouchDB server, as opposed to storing all of the documents within a single database. We took advantage of this when migrating data into CouchDB. Three very large tables were the focus of this migration, each containing between 3 to 50 million rows. We decided to store the data from each table in its own database. The data within these tables are completely unrelated, so we would never need to view data from one database combined with another. If they were related, we would have combined the tables into a single database, as CouchDB cannot create views across multiple databases. Also, storing each set of data in its own database provides additional flexibility. During the migration process, there were several points where we changed the structure of the documents. Having the ability to easily delete the affected database and re-populate it, without affecting any other document types, came in handy. With multiple databases, we have the flexibility to change the replication schedule for one database to be different from the others. It also makes it easy to move one or more databases to another server, should we ever choose to do so.

My colleague Dave made a few changes to CouchRest Rails to support connecting to multiple CouchDB databases within a single rails application. You simply specify the database server location in the configuration files, and then each model object can specify which database it is using.

Documents

CouchDB documents are very flexible. Documents are stored in JSON format, allowing you to take advantage of JSON arrays and dictionaries to represent collections of data. There is no external force dictating how a document should be structured, or what it should contain (as long as the document is valid JSON). Below is an example of what a document may look like for a blog post.

{
   “_id”: “CouchDB: Databases and Documents”,
   “_rev”: “1-704787893”,
   “author”: “John Wood”,
   “email”: “john_p_wood”,
   “post”: “CouchDB is a documented oriented database.  A document...”,
   “tags”: [“couchdb”, “couchdb case study”, “json”],
   “comments”: [
      {
         “email”: “joe@somewhere.com”,
         “comment”: “Thanks for the information”
      },
      {
         “email”: “kevin@xyz.com”,
         “comment”: “CouchDB sounds pretty interesting”
      }
   ]
}

Schema-less

Probably the best part about the document oriented approach is the ability to make each document different from the next. There is no schema to enforce that a document contains specific information. This makes CouchDB a great fit if your application needs to store data that can be wildly different between objects of the same type. In a relational database, this is usually handled by serializing the data in some format, writing the serialized data to the database, and de-serializing the data when it is read by the application. However, this is really nothing more than a hack. Querying the data in such columns can be a nightmare. And, the serialization/de-serialization process is just one more thing that can go wrong. In a document oriented database, there is no need for such a hack. You simply code your CouchDB views to account for the fact that certain fields may not be in the document, and act accordingly (either defaulting to some value, or simply move onto the next document in the database).

Self contained

The most important thing to remember about documents is that they are self contained. All of the data representing a particular concept is right there in the document. (This is a bit of a fabrication, as it is completely possible to establish relationships between documents by having one document store the unique id of another document. However, these links are not directly supported by CouchDB, and can be easily broken.) So if you are moving from a relational database to CouchDB, you should de-normalize your data as much as possible while defining the structure of your document. JSON arrays and dictionaries can help tremendously when de-normalizing relationships. If there is only one piece of information from the relationship worth storing in the document, then an array works great (see the “tags” property above). For relationships with more complex data structures, an array of dictionaries fits the bill quite nicely (see the “comments” property above).

The document id

Another important point to consider when designing your document structure is defining what you will use as the id of the document. The id must be unique not only in that database, but all instances of the same database if you happen to be running inside a database cluster. CouchDB uses document ids to replicate changes between servers. Auto-generated sequential keys are a poor fit for this. While wildly popular in relational databases, auto-generated sequential keys throw a wrench into the gears of the replication process. If each database in the cluster was responsible for generating its own sequential ids, it is highly likely that different documents on different servers could be assigned the same id, which would make CouchDB think that two distinct documents are the same document. Badness would surly ensue.

Instead, it is recommended that you use the data’s “natural key” as the id of your document. The natural key is some field, or combination of fields, in your document that uniquely identifies that document. In the example above, the title of the blog post is a good fit for a natural key. It is not very likely that I will be writing posts with the same title. If you happen to enjoy writing about the same stuff over and over, perhaps the title of the post combined with the date and time it was created would be a better fit. Either way, the id should be composed from data within the document.

If you do not provide an id, CouchDB will provide one for you. CouchDB uses an algorithm that makes it virtually impossible for multiple CouchDB instances to generate the same id. However, I have read articles on the web indicating that this is a very slow operation, so you may want to avoid letting CouchDB generate an id for you. Regardless, natural keys pulled straight from data within your document always make better ids, as they are easier to read, and more identifiable.

For a few of our documents, we used the sequential key generated by MySQL as the id :) I know how stupid this sounds, given the last few paragraphs. However, I think this was the best choice for an id in our case. The data contained in these documents are basically a collection of ids to rows that exist, and will remain in MySQL. None of the data within the document would be any more readable than the MySQL id. Also, since all of the keys were originally generated in a single MySQL database, they are guaranteed to be unique. As of right now, we always plan on creating the data in MySQL, and then “archiving” it to CouchDB at a later date, so this approach should continue to work just fine.

Supporting existing functionality

If you are migrating data from a relational database to CouchDB, there is another important item to think about. If your application needs to interact with CouchDB in the same way that it did with the relational database, then you need to make sure that the CouchDB views you build will be able to replace any SQL queries that are done against that data in the relational database. In order to make this happen, the CouchDB document will need to contain all of the necessary information for you to build views to replace those queries, if you intend on supporting the same functionality. Remember, there are no JOINs in CouchDB.

Summary

I think the way CouchDB handles databases and documents is very straight forward. Once you get used to the idea that there could be multiple instances of databases in a cluster, and that documents should be self contained, the rest is cake. The schema-less approach has the potential to open a lot of doors. I know that we’re already making plans to take advantage of it.

When I began looking into replacing some of TextMe’s large MySQL tables with CouchDB databases, one of the things I noticed right away was that pagination support was not quite there in CouchRest. I say “not quite there” because CouchRest does have the ability to fetch data from the database in paginated chunks, but the current support didn’t really fit too well with way the rest of the library interacts with CouchDB views. A helper class had to be used to fetch the data, and the data came back as hash instead of an instance of the appropriate class.

Pagination is a must for us, because these tables in particular are very large. That’s one of the main reasons why we’re moving them to CouchDB in the first place. Loading all of the data into memory at once would be troublesome to say the least.

CouchRest is still a very young library, currently on version 0.29. However, despite its age, it is already fully featured and off to a great start. So, I saw this as an opportunity to contribute to something that we have already greatly benefited from.

With a little inspiration from Rails, I decided to implement a proxy that would be created when a view was called to fetch data. The proxy would defer getting data from the database until that data was actually needed. I then implemented will_paginate style paginate and paginated_each methods on the proxy object. If either of these methods are called, only a chunk of data will be fetched from the database, and that data will be returned as an array of instances of the appropriate class. If any other method is called on the proxy, the proxy will fetch all of the data from the view, and forward the call on to the “real” array.

I decided to go with will_paginate style methods because the will_paginate gem is by far the most popular pagination solution for Rails. We use it extensively in TextMe. So, implementing the same methods would ensure that we could continue to use our existing pagination code, and the code wouldn’t have to know if it was dealing with a collection of ActiveRecord objects or a collection of CouchRest ExtendedDocument objects.

The new code also throws some methods onto the class itself that lets you paginate over instances of the class without having an instance of the proxy, or a view in your CouchRest ExtendedDocument object.

Here are some examples, pulled from the CouchRest tests:

Paginating using instance methods:

articles = Article.by_date :key => Date.today
articles.paginate(:page => 1, :per_page => 3).size.should == 3

articles = Article.by_date :key => Date.today
articles.paginated_each(:per_page => 3) do |a|
  a.should_not be_nil
end

Paginating via class methods:

articles = Article.paginate(:design_doc => 'Article',
  :view_name => 'by_date', :per_page => 3, :descending => true,
  :key => Date.today, :include_docs => true)
articles.size.should == 3

options = { :design_doc => 'Article', :view_name => 'by_date',
  :per_page => 3, :page => 1, :descending => true,
  :key => Date.today, :include_docs => true }
Article.paginated_each(options) do |a|
  a.should_not be_nil
end

Currently, the forked version of CouchRest containing this feature can be found on GitHub, at http://github.com/jwood/couchrest/tree/master. I’ve submitted a request to have this pulled into the main CouchRest repository.

Hopefully this will be helpful to others.

Part 2: Databases and Documents >>

The wall was quickly approaching. After only a few short years, several of our database tables had over a million rows, a handful had over 10 million, and a few had over 30 million. Our queries were taking longer and longer to execute, and our migrations were taking longer and longer to run. We even had to disable a few customer facing features because the database queries required to support them were too expensive to run, and were causing other issues in the application.

The nature of our business requires us to keep most if not all of this data around and easily accessible in order to provide the level of customer support that we strive for. But, it was becoming very clear that a single database to hold all of this information was not going to scale. Besides, it is common practice to have a separate, reporting database that frees the application database from having to handle these expensive data queries, so we knew that we’d have to segregate the data at some point.

Being a young company with limited resources, scaling up to some super-powered server, or running the leading commercial relational database was not an option. So, we started to look into other solutions. We tried offloading certain expensive queries onto the backup database. That helped a little, but the server hosting the backup database simply didn’t have enough juice to keep up with the load. We also considered rolling up key statistics into summary tables to save us from calculating those stats over and over. However, we realized that this was only solving part of the problem. The tables would still be huge, and summary tables would only replace some of the expensive queries.

It was about this time that my colleague Dave started looking into CouchDB as a possible solution to our issues. Up until this point, I had never heard of CouchDB. CouchDB is document oriented, schema-free database similar to Amazon’s SimpleDB and Google’s BigTable. It stores data as JSON documents and provides a powerful view engine that lets you write Javascript code to select documents from the database, and perform calculations. A RESTful HTTP/JSON API is used to access the database. The database boasts other features as well, such as robust replication, and bi-directional conflict detection and resolution.

The view engine is what peeked our interest. Views can be “rebuilt” whenever we determine it is necessary, and can be configured to return stale data. “Stale data? Why would I want stale data?”, you may be asking yourself. Well, one big reason comes to mind. Returning stale data is fast. When configured to return stale data, the database doesn’t have to calculate anything on the fly. It simply returns what it calculated the last time the view was built, making the query as fast as the HTTP request required to get the data. The CouchDB view engine is also very powerful. CouchDB views use a map/reduce approach to selecting documents from the database (map), and performing aggregate calculations on that data (reduce). The reduce function is optional. CouchDB supports Javascript as the default language for the map and reduce functions. However, this is extensible, and there is support out there for writing views in several other languages.

In our case, we are planning to use CouchDB as an archive database that we can move “old” data to once a night. Once the data is moved to the CouchDB database, it would no longer be updated, and would only be used for calculating statistics in the application. Since we would only be moving data into the database once a day, we only need to rebuild the views once a day. Therefore, all queries could simply ask for (and get) stale data, even when the views were in the process of being rebuilt. Also, moving all of the old data out of the relational database would dramatically reduce the size of the specific tables, improving the performance of the queries that hit those tables.

I’m really looking forward to this partial migration to CouchDB. The ability to add new views to the database without affecting existing views gives us the flexibility we need to grow the TextMe application to provide better, more specific, and more relevant statistics. In marketing, statistics are king. Since TextMe is a mobile marketing tool, we want it to be able to provide all of the data that our customers are looking for, and more. I feel that by moving to CouchDB, we will not only be able to re-activate those features that we had to disable due to database performance, but also add more features and gather more statistics that would have otherwise been impossible with our previous infrastructure.

The migration to CouchDB was not always straight forward. We faced several challenges, and learned many lessons over the past month. All of those challenges will be addressed here.

In the coming posts, I plan to talk about:

• Structuring your CouchDB databases, and the documents within them.
• More details about CouchDB views.
• The application code necessary to talk to CouchDB.
• Migrating parts of an existing application from a relational database backed by ActiveRecord to CouchDB.
• How the CouchDB security model differs from a traditional relational database.

Stay tuned!

A couple of options jumped out at me. I could pull the code I needed to use into a library that could be shared between the components, or implement some form of inter-process communication that would enable me to invoke the service from the one component, and have it processed by the component that already integrated with the service provider.

Pulling the code into a library would be the easier of the two to implement for sure. Like any project of reasonable size, we were already doing this for several other shared pieces of code. Adding one more to the list would be a piece of cake. The second option would require a bit more work. The component that integrates with the service provider runs as a daemon process, so using something straightforward like HTTP to handle the interprocess communication was out of the question. Instead, I’d likely have to utilize the queuing framework that we already had in place. What makes it more difficult is that the queuing library we use only handles asynchronous calls, and this would need to be a synchronous call. Not the end of the world by any means, but without a doubt more complicated than simply sucking the code into a library.

Even though option one was easier to implement, having two components in the architecture integrate with a 3rd party seemed like a bad idea. Sprinkling integration points throughout your application is usually a recipe for failure. Largely because it is only a matter of time before an integration point fails.

If we went with option one, we could have the library handle the failures. However, even if handled properly, failures like this usually have other consequences. For example, if the service never responded, it could cause requests to back up in the given component. Even if we implemented a timeout, it is likely that the timeout would be greater than the average response time, which means our system would take longer to process each request. If you had to deal with a lot of incoming requests at the time of the failure, you could be in for a world of hurt, especially if you had multiple components suffering from this issue.

With option two, we have a bit more control over the situation. First off, we would know there was one, and only one spot in our architecture that integrated with that particular service. This would allow us to better understand the potential impact of the failure, and the steps that needed to be taken to address it. Second, it would allow us to more easily implement a circuit beaker to prevent the failure from rippling across the system. If the circuit breaker was tripped, we could return an error, some sort of filler data, or queue the request up for processing at a later time. Third, we could potentially add resources to account for the situation. Since the work was being done in a completely different component, if it was simply a matter of increased latency on the part of our service provider, we could always spin up a few more instances of that component to account for the fact that some of the requests may be starting to back up.

In his fantastic book, Release It, Michael Nygard talks about integration points, along with a host of other topics regarding the deployment and support of production software. Any developer who writes code that will eventually be running in a production environment (which I hope is EVERY developer) should read this book. Regarding integration points, Michael says the following:

• Integration points are the number-one killer of systems.
• Every integration point will eventually fail in some way, and you need to be prepared for that failure.
• Integration point failures take several forms, ranging from various network errors to semantic errors.
• Failure in a remote system quickly becomes your problem, usually as a cascading failure when your code isn’t defensive enough.

However, even though integration points can be tough to work with, system’s without any integrations points are usually not that useful. So, integration points are a necessary evil. Our best tools to keep them in line are defensive coding, being smart about where you place the integration points in your system, and limiting the integration points in the system.

With the help of my colleague Doug Barth, we (mostly Doug) whipped up a synchronous client for the Ruby AMQP library. I then used this code to implement the synchronous queuing behavior I needed to keep the integration point where it belonged. Those interested can find the code in GitHub, at http://github.com/dougbarth/amqp/tree/bg_em.

Separating code that creates an object from the code that uses the object results in a much more flexible design, which is easier to change. Creating this separation is also very easy to do. By simply avoiding a call to the new operator in the “client” code for the particular object you wish to instantiate, you are able to evolve your code to adjust to a variety of changes, most of which require no changes in the code that uses the object. Let’s walk through an example.

Let’s say we have a logging class, named Logger, that we use to log messages from our application. The class is pretty simple, and looks something like this.

public class Logger {
    private static final String logFileName = "application.log";
    private FileWriter fileWriter;
    private Class from;

    public Logger(Class from) {
        this.from = from;

        try {
            fileWriter = new FileWriter(logFileName, true);
        } catch (IOException e) {
            throw new RuntimeException("Log file '" + logFileName +
                    "' could not be opened for writing.", e);
        }
    }

    public void log(String message) {
        try {
            fileWriter.write(
                from.getCanonicalName() + ": " + message + "\n");
            fileWriter.flush();
        } catch (IOException e) {
            System.err.println("Writing to the log file failed");
            e.printStackTrace();
        }
    }
}

In our application, we would typically use the Logger class like this:

Logger logger = new Logger(MyClass.class);
logger.log("Some message");

I think this is pretty typical, and seems to be the default pattern. Create the object that you need, and then use it. Simple and straightforward. However, the simplicity comes at the price of limited flexibility. For example, what if I wanted to limit the Logger class to only having one instance? Or, what if I wanted to start logging some messages to the database, and some to the file system? By combining the code that creates the object with the code that uses the object, we’ve greatly limited the ways in which we can evolve our design without affecting existing “client” code. Sure, we can work our way out of it, but since the Logger is a very popular class used by almost every other class in the system, it will require a lot of work to change.

So, how can we avoid this? How can we effectively encapsulate the creation of the object from the code that uses it? The very first “tip” in Effective Java, by Joshua Bloch, is to prefer static builder methods over constructors. Joshua suggests this for the same reasons Scott suggests separating code that creates the object from code that uses the object in Emergent Design. Instead of making your clients use the new operator to create instances of your object, provide them with a static builder method to do so.

    public static Logger getInstance(Class from) {
        return new Logger(from);
    }

    protected Logger(Class from) {
        this.from = from;

        try {
            fileWriter = new FileWriter(logFileName, true);
        } catch (IOException e) {
            throw new RuntimeException("Log file '" + logFileName +
                    "' could not be opened for writing.", e);
        }
    }

Note that I changed the scope of Logger’s constructor from public to protected. This will discourage other classes outside of the logging package from using it, while leaving the Logger class open for subclassing. With this new method in place, users of this class can now create an instance by doing the following.

Logger logger = Logger.getInstance(MyClass.class);
logger.log("Some message");

It seems silly to provide a method that simply calls new. But, doing so adds so much flexibility to the design, that Scott considers it a “practice”, or something he does every time without even thinking about it. Abandoning the constructor also opens a few doors. You are no longer required to return an instance of that specific class, giving you the freedom return any object of that type. You don’t always have to return a new instance, allowing you to implement a cache, or a singleton. You can use this flexibility to your advantage when evolving your design. Let’s see how.

Let’s say we get a request from our accounting department to log messages from code that deals with financial transactions (conveniently located in the net.johnpwood.financial package) to the database. This sounds like the birth of a new type of logger. Because clients are not using the new operator to create new instances of the Logger class, we can easily evolve Logger into an abstract class, keeping the static getInstance() method as the factory method for the Logger class hierarchy. After we have the abstract class, we can create two new subclasses to implement the individual behavior. All of this with no change to how the client uses the logging functionality.

Because the filesystem logger and the database logger don’t have too much in common, the Logger class has been slimmed down quite a bit. What remains is the interface for the Logger subtypes, defined via the abstract log() method, and a factory method to create the proper logger, which is implemented in getInstance().

public abstract class Logger {

    public static Logger getInstance(Class from) {
        if (from.getCanonicalName().startsWith(
                "net.johnpwood.financial")) {
            return DatabaseLogger.getInstance(from);
        } else {
            return FilesystemLogger.getInstance(from);
        }
    }

    protected Logger() {}
    public abstract void log(String message);
}

We now have two distinct classes that handle logging transactions. FilesystemLogger, which contains most of the old Logger code, and DatabaseLogger. FilesystemLogger should look pretty familiar.

public class FilesystemLogger extends Logger {
    private static final String logFileName = "application.log";
    private FileWriter fileWriter;
    private Class from;

    public static FilesystemLogger getInstance(Class from) {
        return new FilesystemLogger(from);
    }

    protected FilesystemLogger(Class from) {
        this.from = from;

        try {
            fileWriter = new FileWriter(logFileName, true);
        } catch (IOException e) {
            throw new RuntimeException("Log file '" + logFileName +
                    "' could not be opened for writing.", e);
        }
    }

    @Override
    public void log(String message) {
        try {
            fileWriter.write(
                from.getCanonicalName() + ": " + message + "\n");
            fileWriter.flush();
        } catch (IOException e) {
            System.err.println("Writing to the log file failed");
            e.printStackTrace();
        }
    }
}

DatabaseLogger is also pretty simple, since I didn’t bother to implement any of the hairy database code (doesn’t help to illustrate the point…and I’m lazy).

public class DatabaseLogger extends Logger {
    private Class from;

    public static DatabaseLogger getInstance(Class from) {
        return new DatabaseLogger(from);
    }

    protected DatabaseLogger(Class from) {
        this.from = from;
        establishDatabaseConnection();
    }

    @Override
    public void log(String message) {
        LoggerDataObject dataObject =
            new LoggerDataObject(from, message);
        dataObject.save();
    }

    private void establishDatabaseConnection() {
        // Connect to the database
    }
}

We’ve significantly changed how the Logger works, and the client is totally oblivious to the changes. The client code continues to use the Logger as it did before, and everything just works. Pretty sweet, eh?

As you can imagine, there are many other ways you can evolve your design if you have this separation of creation and use. If we need to create a MockLogger for testing purposes, it can be created in Logger.getInstance() along with the other Logger implementations. The client would never know that it is using a mock. If we ended up creating 10 different loggers, it would be trivial to have Logger.getInstance() delegate the creation of the proper Logger instance to a factory, moving the creation logic out of the Logger class. Again, no changes to the client.

Separating creation from use also allows you to easily evolve your class into a singleton (or any other pattern that controls the number of instances created). This doesn’t make much sense for Logger, since each unique Logger instance contains state. However, it does make sense for some classes. Evolving your class into a singleton simply requires a static instance variable on the class containing the instance of the singleton object, and an implementation of getInstance() that returns the singleton instance. If clients have already been using the getInstance() method to get an instance of the class, then no change would be required on their end. Here’s an example:

public class SomeOtherClass {
    private static SomeOtherClass instance = new SomeOtherClass();

    public static SomeOtherClass getInstance() {
        return instance;
    }

    private SomeOtherClass() {}
}

It is worth pointing out that static builder methods are not the only way to achieve this separation. Dependency injection frameworks like Spring and Guice do all of this for you. They take on the responsibility of creating the objects, and getting the instances to the code that uses them. If you are a disciplined developer, and never “cheat” by instantiating the objects directly, then all of the same benefits outlined above apply when using a dependency injection framework.

Like everything in life, there are cons that go along with the pros. Separating the code that creates an object from the code that uses the object is not the default pattern. It is not the norm. It will take time for you and your co-workers to get comfortable with this pattern. API documentation tools don’t “call out” static builder methods like they do constructors. This could have an effect on anybody using your library. Dependency injection frameworks take the creation of objects completely out of your code, moving it to some magical, mysterious land where things just happen, somehow. This also can take some time to get used to, especially for those new to the concept.

However, I feel that the benefits of separating creation from use far outweigh the drawbacks.

In our field, change is a constant. As a profession, we’re gradually learning to stop fighting change, and to start accepting it. This means designing for change. Doing so makes everybody’s life easier, from the customer to the developer. Separating creation from use is one, quick way we can increase the flexibility of our design, with very little up front cost.

Thanks to Mahesh Murthy for reviewing this post.

I think that it is a great idea for developers to have such an environment for themselves where they can try out new technologies, techniques, or processes. Setting up a sandbox now-a-days is pretty easy. Machines are cheap, or free if you’re not picky and keep your eyes and ears open. That 4 year old PC that your cousin is throwing away because he got a brand new Dell for Christmas will fit the bill just fine. Linux is free, runs great on older computers, and has the power and flexibility to host virtually any application. Sign up for a free DNS service such as DynDNS and poof, you’ve got yourself a little server that you can reach over the Internet. My sandbox is a dusty old dual Pentium III with 512 MB of RAM, which is running Ubuntu Linux. It fits the bill quite nicely if you ask me.

But a sandbox is only half the equation. For developers, we need an application that we can play with in the sandbox. Something we can poke and prod. A sandbox application so to speak. There are several reasons why you many want to create a sandbox application.

Platform for playing with new technologies

Perhaps the biggest reason for creating and maintaining your own sandbox application is that it can serve as a platform for trying out new technologies. Doing this at work can be tough. Your boss or client may not be thrilled to hear that you completely re-wrote part of the application to use the bleeding edge release of some hot new framework because you “thought it was cool”. But, there’s nothing stopping you from doing it with your own application. Even if you rely heavily on your application, there’s no reason why you can’t fork your code, and give the new technology a try on a separate branch. If it works out you can merge the code into the main branch, and if not you can always abandon the changes. Now, this approach will only work if you have your application under source control (which you should). If you don’t mind giving the world access to your code, GitHub will host your code for free. Otherwise, it’s very easy to setup your own source control system in your sandbox.

Material to blog about

Trying out new technologies, techniques, or processes can also give you plenty of material to blog about. If the technology/technique/process you are tinkering with is hot, it is very likely that many people will be interested in reading about your experience. If you blog frequently about topics that people are interested in, you’ll steadily increase readership. This could be very good for your career, as well known programmers generally don’t have a hard time finding work. Work usually finds them.

Create something that you will use

What’s the point in going through all of this trouble to create something if you never use it? I’ve created a few applications that I use on a regular basis. Not only did these applications address some need that I was currently facing, but having a sandbox application that you actually use means that you will be more likely to maintain and enhance it.

Looks great on a resume

Employers love to hire people who show an interest in their field outside of work. I’ve found that people passionate about their field are usually better at their jobs than those who show up at 9, work on the same ol’ stuff for 8 hours, and go home. Having a sandbox application shows people that you love what you do so much that you have dedicated time outside of work to create something that you care about. This especially holds true if you’ve put serious thought into your application, and are excited to show it off to anybody who asks to see it.

Release it, and potentially help others

One of my colleagues once said,

Your parents lied to you. You are not special. There’s millions of people out there just like you.

He wasn’t trying to be mean, this time :) He was simply pointing out that you are not alone. If you are facing a problem, odds are there are hundreds or thousands (or more) of people out there who are facing that same problem. Releasing your application could be helping all of these people. Perhaps it would help them so much that they would be willing to pay for it. Wouldn’t that be nice?

Open source your application

Open sourcing your application can be great for several reasons. Perhaps you’ve just finished migrating your application to the latest version some framework. Not only can you blog about your experience, but you can share the code with others so that they can see exactly how you did it. This could potentially help others looking to migrate their applications to the same framework. You could get feedback from the community about something you could be doing better, and learn something new. People looking at your code could spot a bug, giving you the opportunity to fix it before it affects you (especially if you use your application). Some employers ask to see source code samples as part of the interview process. What sort of reaction do you think you would get if you immediately spit out several repositories that you owned on GitHub for the prospective employer to browse at their leisure? I’m not sure about you, but I’d be pretty impressed.

However, there is a potential drawback in releasing your code to the world. Ironically, it’s the same as one of the benefits. The world can see your code. This can be a bad thing if your code is full of bugs, or sloppy. So if you plan on releasing your code, take the extra care necessary to ensure that the code reflects your best effort. Your code, and you as a developer, will benefit from the extra TLC.

The next big thing

You never know which crazy, off the wall idea will turn into the next big thing. Who would of thought that there was a real need for an application that lets you tell the world what you’re doing right now. If your idea for a sandbox application turns into something with real business value, then you never know where it will end up. Large companies will often spend big bucks to buy great ideas. I’d imagine it would be pretty cool to be on the receiving end of one of those deals.

Summary

Creating a sandbox, and a sandbox application is something that every serious developer should do. If nothing more, it will give you a place to tinker with new technologies and grow as a developer. There is little to no cost to set it up, and your imagination is the only limit.