It’s equally important to know what done means to the customer, and how it may conflict with your definition of done. If I’m hiring you to line a parking lot for my store, and the deadline is the day of the grand opening, done is when the paint is dry, not when the last stroke is sprayed (and still wet!). Since I don’t paint asphalt for a living, you surely can’t expect me to know how long it takes the paint to dry.

A Real World Example

I once worked for an international online retailer who’s idea of “meeting a deadline” was a little different than mine. We were scheduled to do a major release of the website consisting of a brand new design, and some additional features. It had a large supporting marketing campaign advertising it as a new generation of the store with super-awesome-o features. A hard launch date was set in the marketing material and community hype, so launching on that date was critical. The developers (through no fault of their own) ended up needing to work 20+ hour shifts in the days approaching the launch in order to make sure all of the promised features made the cut. After gallons of coffee and mountain dew, the launch was finally completed at 11:50pm local time on the day of the deadline.

In the coming days, the launch was touted as a total success, and the words “we met the deadline” were strewn about the meeting halls by the software development managers. I felt that expressing my opinion about the so called meeting of the deadline would create a lot of flak and would be taken as a slam to my fellow developers, so I decided to keep it to myself.

Quite frankly, this was not a satisfactory meeting of the deadline in my opinion. The expectation as a customer of the site was that they would log-on on the launch date and see a brand new website with pretty little features. Instead, we kept them waiting and wondering. Our blog site for the store was lit up with comments by loyal customers, wondering where the site is. Many had given up on it being launched that day, and only those 7 hours behind GMT (the time zone any international company should be paying attention to) or greater actually saw the website on the launch date. Yet, we were proud of meeting this tough deadline because “hey, we technically finish it on the date we promised!”.

The real issue in the online retailer case is the approach to the problem. The delivery date and the marketing campaign date should not have been the same. The deadline for software development should have been at least 1 day prior to the launch, giving the developers a deadline that could be approached in the manner it was, but still allowing the site to actually be launched and available to the customer when they expected it to be. The developers could push their efforts right up to the wire (as we did) without any negative impact.

Conclusion

You’ve probably heard the saying “If you’re 5 minutes early, you’re on time. If you’re on time, you’re late!” or some similar version of it. This saying definitely holds a lot of merit when it comes to delivering your products. Surprise your customers and clients by delivering “5 minutes early” instead of making them come looking. Don’t use the vagueness of your own deadline as an excuse to say you delivered on time. Save that for the cable company, and their “sometime between 9am and 5pm” appointments.

If you’re interested in seeing this component in Zend Framework soon, please follow progress at the Zend Framework Contributors Wiki, This blog, and on the project page at GitHub. Any feedback, suggestions, and patches you can provide will be nothing but useful. Send me an email, comment on the wiki, or This proposal is still in it’s early stages, but I hope to be knocking out a lot of the work in the next couple of weeks.

For me, learning the areas that I’ve yet to master has been a huge motivation to keep pushing the envelope of my own knowledge. I consider myself a very capable developer, and I don’t feel that my co-workers, clients, or associates would say otherwise. At the same time, I know I still have a lot to learn. In fact, anyone that says they’re a master of every craft of web and software development probably hasn’t learned enough to know what there is to know. This blog itself is a consequence of my desire to explore more technologies and techniques.

The best gift I’ve been given recently is a true evaluation of my level of mastery. This particular evaluation came interview-style from a Zend developer and client support specialist (meaning their salary is paid by Zend Technologies). Before I entered the conversation, I knew it would be highly technical. After all, you’re talking about the PHP company. I’ve been writing code in PHP for 7 years, am Zend Certified, ranked #3 on oDesk for the Advanced PHP Expert Rating test, placed top 10% in many other assessment tests on other freelance sites, yadda yadda yadda. While none of this equates to guru, it does equate to experience and track record with the language. Thus, I expected to be stumped occasionally, but stride through most of it as usual.

As you can imagine, that wasn’t the case.

Overall, I did OK with the evaluation. The questions that I was truly stumped on were things that only the best of the best would know off-hand (and even my evaluator admitted to not knowing the answer until he started working at Zend). Then, of course, there were the middle-ground questions that I knew most of the answer to, but there was a small edge-case that I didn’t think of, or that took me a bit of thinking to come up with. But what surprised me the most were the questions that I did know (or should have known) off hand because I encounter them every day, but have been numb to the rhyme-or-reason of them. I won’t even mention getting a question wrong that I obviously knew the answer to, but was to worried about it being a trick question to think it all the way through.

I won’t get into specifics of the questions that were asked, as I don’t want to provide a cheat sheet for the next person. But, the point is this — being able to academically and fluently explain something on the spot is the true test of how in-depth your knowledge is on a subject. Don’t settle for just being able to do it well, but understand why you’re doing it, how you did it, and what’s the best way to do it. The Pragmatic Programmers would call this programming by confidence, not coincidence.

Thanks for making me study harder, Zend

1. Preparing Your Application
2. A Basic Example
3. Running Your Tests
4. Extending the Testing Functionality
   1. Disposable Testing Models
   2. Authentication Support
   3. Putting It All Together
5. Writing Our Controller Test
6. Conclusion

Automated testing for your web applications is an important step in having the confidence to make changes to your application, and still be confident you’re delivering a quality, regression-free product. With Zend Framework’s testing framework (built with PHPUnit), you can build a thorough suite of test cases for your web application with very little hassle.

Part 1 will give you all of the basic information you need to start writing automated tests for your Zend Framework applications. In Part 2, I’ll provide more real-world example covering some other ways to write tests.

Lets get right to it.

For the examples below, I will be using an actual controller from one of my projects. This controller handles account activies, such as logins, logouts, registrations, and confirmations. We’ll be using a test database with a schema that clones our production database, and Doctrine to manage ORM (Sorry Zend_Db :(). I’ll assume that you’re using the prescribed Zend Framework (1.6+) project layout, and that you’re somewhat familiar with Zend_Config and using an Initializer controller plugin (created by default if you’re using Zend Studio for Eclipse 6.1).

Preparing Your Application

The first step to setting up automated testing is to prepare your application’s environment and settings appropriately. Depending no your setup, this can involve setting global variables, changing database connections, or reconfiguring paths. Fortunately for us, this capability is easy using Zend_Config and an Initializer controller plugin.

Zend_Config allows you to specify “sections”, which can inherit from another section. This allows you to modify configuration for different environments without duplicating settings across different files (and thereby helping us ensure we don’t forget to set something!). In our example project, we’ll need only to modify our database connection string, so we’re using the test database.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

< ?xml version="1.0" encoding="UTF-8"?>
<config>
 
<production>
  <db>
	<dsn>mysql://dbowner:password@localhost/maindb</dsn>
	<attributes>
		<model_loading>conservative</model_loading>
	</attributes>
  </db>
</production>
<test extends="production">
  <db>
	<dsn>mysql://dbowner:password@localhost/maindb_test</dsn>
  </db>
</test>
</config>

^{Notice how we can even inherit child attributes: Even though we specified a a node, we didn’t have to specify everything below the node}

Now that we have our configurations in order, we need something to manage this configuration for us, and switch off based on the environment we’re running in. That’s the role of our Initializer plugin, which accepts the environment to initialize as a constructor parameter. Showing the source of an Initializer is outside the scope of this article, but here’s a pastie for the curious.

A Basic Example

Lets start with the basic framework of a controller test. If you’re using Zend Studio for Eclipse, you can easily create this structure by right-clicking on your controller in PHP Explorer and selecting New > Zend Framework Item, and then selecting Zend Controller Test Case. Then, simply make sure the controller you want to test is chosen, and click finish.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

require_once 'Zend/Test/PHPUnit/ControllerTestCase.php';
require_once 'application/Initializer.php';
require_once 'application/default/controllers/IndexController.php';
 
class AccountControllerTest extends Zend_Test_PHPUnit_ControllerTestCase {
 
	/**
	 * Prepares the environment before running a test.
	 */
	protected function setUp() {
		$this->bootstrap = array ($this, 'appBootstrap' );
		parent::setUp ();
		// TODO Auto-generated FooControllerTest::setUp()
	}
 
	/**
	 * Prepares the environment before running a test.
	 */
	public function appBootstrap() {
		$this->frontController->registerPlugin ( new Initializer( 'test' ) );
	}
 
	/**
	 * Cleans up the environment after running a test.
	 */
	protected function tearDown() {
		// TODO Auto-generated FooControllerTest::tearDown()
		parent::tearDown ();
	}
 
	/**
	 * Constructs the test case.
	 */
	public function __construct() {
		// TODO Auto-generated constructor
	}
 
	/**
	 * Tests FooController->barAction()
	 */
	public function testIndexAction() {
		// TODO Auto-generated FooControllerTest->testBarAction()
		$this->dispatch ( '/index/index' );
		$this->assertController ( 'index' );
		$this->assertAction ( 'index' );
	}
}

You’ll see on line 20 where we’re using our initializer to setup the test environment before we run the tests. Before each test metod is run, PHPUnit will call our setup() method, which has been programmed to call our appBootstrap method. This ensures us that we’re using a clean configuration and environment before each test, as if each test was a separate process. When each test case is done, the tearDown() method is called. tearDown() is the place to put any code for removing resources or resetting any persistable changes that tests might make. We’ll make use of this in our advanced examples later.

Line 41 contains a bare-bones test case, which will ensure that dispatching to ‘/index/index’ results in the controller named ‘index’ and an action called ‘index’ are the last to be executed. This might seem trivial, but it helps detect errors with your controllers. If an uncaught exception is thrown, the controller assertion will fail, since your controller will be the last executed.

Running Your Tests

To keep this article focused, I’ve decided to remove this section and cover only writing the tests. If you need help creating Test Suites and running tests from the command line, check out PHPUnit Documentation, specifically the sections on the Command Line Test Runner and Organizing Test Suites. If you have any specific questions, feel free to shoot me an email.

Extending the Testing Functionality

Now that we covered the basics, lets get to fully testing our Accounts controllers. There are a couple of “outside the box” requirements we have for testing our accounts controller. First off, we need a way to test the full account creation process, as if a user was actually registering. When we’re done with a test, we want to get rid of that data whenever possible, so we can run tests as many times as we want and not worry about growing our test database. Secondly, We need a way to simulate an authenticated user, as well as checking for whether a user has been authenticated or not.

Because these operations are pretty general and an opportunity for reuse exists, Lets put our supporting logic in a parent class and let our test cases inherit them.

Disposable Testing Models

There are two ways to ensure the data you’re creating during a test is deleted once a test is complete. Some choose to create the database on the fly using seed data. Since I’m using Doctrine for my projects and working directly with models (no raw queries) in the tests, I decided that just deleting the data was the best approach. To do this, all we need to do is “schedule” our model for deletion after it has been created (or loaded).

1
2
3
4

protected function _setDisposable( Doctrine_Record $model )
{
    $this->_disposables[] = $model;
}

This function simply takes a reference to a model, and stores it in array, which will be handled by our tearDown() function later:

1
2
3
4
5
6
7
8
9
10
11
12
13

	protected function tearDown()
 
 
	{
	    parent::tearDown();
 
	    foreach ( $this->_disposables as $model ) {
	        if ( $model instanceof Doctrine_Record ) {
	            $model->delete();
	        }
	        unset( $model );
	    }
	}

We simply loop through all of our “scheduled” models and delete them. This must be done in tearDown() and not inside of any test method because it’s the only way to ensure it happens. Once an assertion fails or an unexpected exception occurs in a test, that method stops executing. IF we were to try to dispose our models after an assertion, it may never happen¹. Similarly, we obviously can’t dispose of a model before an assertion if that model is needed for the assertion (and why would it exist if you didn’t need it?).

Authentication Support

There are 3 things we must be able to do in order to fully test authentication.

• Create a fake identity
• Set our environment to a state equivalent to a user being logged in.
• Assert whether the environment has been changed to a logged in state.

Our example controller uses a database adapter for authentication and identity retrieval², so generating a fake identity for us means creating (or loading) a record in our accounts table, and returning the identity data we would normally get.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

/**
 * Generates a fake identity, usefull for simulating a logged in user
 *
 * @return StdClass an identity
 */
protected function _generateFakeIdentity()
{
	$identity = new stdClass();
 
        $account = new Account();
        $account->username     = 'AutoTest' . time();
        $account->emailAddress = 'autotest@example.org';
        $account->password     = md5( 'password' );
        $account->confirmed    = true;
        $account->enabled      = true;
        $account->save();
        $this->_setDisposable( $account );
 
	foreach( $account->toArray() as $key => $val ) {
	        $identity->$key = $val;
	}
	unset( $identity->password );
 
	return $identity;
}

Account is our model, a Doctrine_Record type. We’re just simply creating a random account, and returning it’s data as our idenity. Notice that we’re also scheduling this model for disposal (as we covered above.) Now, we just need a way to set our environment to the “logged in state” as this fake user.

1
2
3
4
5
6
7
8
9
10
11
12
13

/**
 * Sets the current state as if there is a logged in user
 *
 * @param object $identity the idenity to use, otherwise one is generated
 * @return void
 */
protected  function _doLogin( $identity = null )
{
    if ( $identity === null ) {
        $identity = $this->_generateFakeIdentity();
    }
    Zend_Auth::getInstance()->getStorage()->write( $identity );
}

In our example application, if Zend_Auth has an identity, a user must be logged in. Therefore, all we have to do is store an identity in Zend_Auth’s storage adapter, and call ourselves logged in. That makes asserting login as simple as checking for an identity.

1
2
3
4
5
6
7
8
9

public function assertNotLoggedIn()
{
    $this->assertFalse( Zend_Auth::getInstance()->hasIdentity(), 'Login assertion failed' );
}
 
public function assertLoggedIn()
{
    $this->assertTrue( Zend_Auth::getInstance()->hasIdentity(), 'Login assertion failed' );
}

These simple assertions ³ ensure that we’re logged in (or not logged in)

Putting It All Together

Putting it all together, we now have a base class which provides all of our test cases with the functionality we need.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

< ?php
 
require_once 'Zend/Test/PHPUnit/ControllerTestCase.php';
 
class BaseControllerTest extends Zend_Test_PHPUnit_ControllerTestCase
{
 
    /**
     * Contains models which should be destroyed on tear down
     *
     * @var array
     */
    protected $_disposables = array();
 
 
	protected function tearDown()
	{
	    parent::tearDown();
 
	    foreach ( $this->_disposables as $model ) {
	        if ( $model instanceof Doctrine_Record ) {
	            $model->delete();
	        }
	        unset( $model );
	    }
	}
 
	/**
	 * Sets a model as disposable, so teardown automatically deletes it
	 *
	 * @param Doctrine_record $model
	 */
	protected function _setDisposable( Doctrine_record $model )
	{
	    $this->_disposables[] = $model;
	}
 
	/**
	 * Sets the current state as if there is a logged in user
	 *
	 * @param object $identity the idenity to use, otherwise one is generated
	 * @return void
	 */
	protected function _doLogin( $identity = null )
	{
	    if ( $identity === null ) {
	        $identity = $this->_generateFakeIdentity();
	    }
 
	    Zend_Auth::getInstance()->getStorage()->write( $identity );
	}
 
	/**
	 * Generates a fake identity, usefull for simulating a logged in user
	 *
	 * @param boolean $unique
	 * @return StdClass an identity
	 */
	protected function _generateFakeIdentity( $unique = false )
	{
		$identity = new stdClass();
 
		$account = new Account();
	        $account->username     = 'AutoTest' . time();
	        $account->emailAddress = 'autotest' . time() . '@example.org';
	        $account->password     = md5( 'password' );
	        $account->confirmed    = true;
	        $account->enabled      = true;
	        $account->save();
	        $this->_setDisposable( $account );
 
	    	foreach( $account->toArray() as $key => $val ) {
	        	$identity->$key = $val;
	    	}
	    	unset( $identity->password );
 
	    	return $identity;
	}
 
	public function assertNotLoggedIn()
	{
	    $this->assertFalse( Zend_Auth::getInstance()->hasIdentity(), 'Login assertion failed' );
	}
 
	public function assertLoggedIn()
	{
	    $this->assertTrue( Zend_Auth::getInstance()->hasIdentity(), 'Login assertion failed' );
	}
 
}

Writing Our Controller Test

Now that our groundwork has been laid, we can finally start writing our controller test.

Our first set of test cases cover the requirement “when a user registers, they must confirm their email address before they can access their account”. To do this, we need to simulate a user posting their valid registration details to our controller, and verify that the account created is not set as confirmed. We then need to test that submitting login information for a non-confirmed account to our login action does not result in an authenticated user.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

public function testRegisterCreatesNewUnconfirmedAccount()
{
    $email = 'autotest' . time() . '@example.org';
    $data = array(
        'emailAddress'    => $email,
        'password'		  => 'testpassw0rd',
        'passwordconfirm' => 'testpassw0rd'
    );
 
    $_POST = $data;
 
    $this->dispatch( '/account/register' );
    //try to find the account record
    $table = Doctrine_Table::create( 'Account' ) ;
    $account = $table->findOneByEmailAddress( $email );
    $this->_setDisposable( $account );
    $this->assertNotNull( $account );
    $this->assertFalse( $account->confirmed, 'Account was not marked as unconfirmed' );
}
 
/**
 * Asserts that a user that hasn't been confirmed cannot login
 *
 */
public function testUnconfirmedUserCannotLogin()
{
    $email = 'autotest' . time() . '@example.org';
 
    $account = new Account();
    $account->username     = $email;
    $account->password     = md5( 'password' );
    $account->emailAddress = $email;
    $account->confirmed    = false;
    $account->enabled      = true;
    $account->save();
 
    $this->_setDisposable( $account );
 
    $_POST['username'] = $email;
    $_POST['password'] = 'password';
 
 
 
 
    $this->dispatch( '/account/login' );
    $this->assertFalse( Zend_Auth::getInstance()->hasIdentity() );
    $this->assertNotRedirect();
}

Our first test simply uses the $_POST global variable to simulate submitting our registration form with some test data. After dispatch(), we use Doctrine_Table to find the model created by AccountController::registerAction(), then assert that the record was found and that it was not marked as confirmed.

The second test works by manually inserting a record that’s not confirmed, and making sure no user is authorized when attempting to login with that user’s account information. As an added bonus, we also use assertNotRedirect() to make sure our controller didn’t redirect. Our controller should only redirect if login was successful, otherwise it would be confusing to a user.

Conclusion

Automated testing of your controllers is relatively simple using the combined power of PHPUnit and Zend Framework’s Zend_Test component. We can add additional functionality to allow our tests to simulate authentication, create fake identities, and even clean up our database after our tests have run. I showed you how you can put this all together to test a registration and confirmation process in your controllers.

In part 2, I’ll cover more areas of testing in our AccountsController, including testing our actions that require an authorized user in order to access them.

The first thing you must do is define the model you want to be hydratable. I’ll use “Article” as my model. This step is important because it gives Doctrine an object to hydrate when we perform operations against our Doctrine_Table object. It also, of course, allows Doctrine to do alias translation and automatically detect relationships (I.E., so you don’t have to manually perform joins.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

class Article extends Doctrine_Record
{
 
	public function setTableDefinition()
	{
 
		$this->hasColumn( 'title', 'string', 255,
			array( 'notblank' => true )
		);
 
		$this->hasColumn( 'author', 'string', 255,
			array( 'notblank' => true )
		);
 
		$this->hasColumn( 'body', 'clob' );
 
		$this->hasColumn( 'published', 'boolean', null,
			array( 'default' => false )
		);
	}
 
	public function setUp()
	{
		$this->hasMany( 'Comment', array(
			'local' => 'id',
            		'foreign' => 'article_id',
		);
	}
}

^{*It’s good practice to make all of your models Timestampable, but I left that out of this example for simplicity}

Now that our models have been defined, lets move on to our Peer, the Doctrine_Table. In it’s simplest form, our Peer only needs to extend Doctrine_Table. In order to harness the power of Doctrine out of the box, we need to make sure the 3rd parameter of the Doctrine_Table constructor is set to “true“. This allows Doctrine to bind our model (defined above) to the Table that this Doctrine_Table instance is representing. Without it, we would have to manually bind the model.

TIP: Instead of doing this for every Peer, create a base class which does it for you. I haven’t run into a case yet where I didn’t want to let Doctrine do it’s work for me. As long as the models exist, it should work fine.

1
2
3
4
5
6
7
8

class ArticlePeer extends Doctrine_Table
{
 
	public function __construct( Doctrine_Connection $oDbCon )
	{
		parent::__construct(  'Article', $oDbCon, true );
	}
}

Now that our peer is defined, we can save ourselves the trouble of writing any DQL. Doctrine_Table makes available some magic functions, allowing you to do simple queries against any field defined in the model you’re peering. For example, we can easily retrieve a collection of Articles by “A.J. Brown”:

1
2

$peer = new ArticlePeer( $connection );
$articles = $peer->findByAuthor( 'A.J. Brown' );

or all published articles:

1
2

$peer = new ArticlePeer( $connection );
$articles = $peer->findByPublished( true );

The advantage of this approach might not be obvious at first. After all, you could just as easily instanciate a new Doctrine_Table, passing the correct parameters, right?

Most applications won’t use the simple findBy methods made availble through the Doctrine_Table API. We’ll typically need to do lookups that will require custom DQL to return the objects we need. It’s a good practice to make these queries reusable, and place them at a lower level in your code. In fact, If you follow the “Fat Model, Skinny Controller” philosophy, you would never be writing DQL at any level of your application outside of the model / data access layer.

With that said, any query that hydrates Article models (or agregate data of Article models, such as counts and averages) should be performed within a method of our ArticlePeer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

< ?php
class ArticlePeer extends Doctrine_Table
{
 
	public function __construct( Doctrine_Connection $oDbCon )
	{
		parent::__construct(  'Article', $oDbCon, true );
	}
 
	public function findArticlesForReview()
	{
		return $this->createQuery()
			->where( 'Article.pendingReview = true' )
			->addWhere( "Article.status != 'draft' ")
			->orderBy( "Article.title" )
			->execute()
			;  
	}
 
}

Now our user code doesn’t have to know the query needed for the data, it only needs to know what the method returns. If your definition of an article pending review changes later, you won’t need to grep through all of your code for the related queries.

TIP: An added benefit of placing your DQL queries inside Doctrine_Table objects is caching. The ArticlePeer above makes a great place to implement Memcached caching, since it would be completely transparent to the user.

Happy Doctrine…ing?

For more basic information on using Doctrine, check the Doctrine ORM Online Manual.

BETWEEN clauses with relational alias operands do not parse, resulting in an exception “could not find short alias”.

Example

The following code will throw an exception. User hasOne Subscription.

1
2
3

$q1 = Doctrine_Query::create()
    ->select('u.id') ->from('User u') ->where("CURRENT_DATE() BETWEEN u.Subscription.begin AND u.Subscription.end")
    ->addWhere( 'u.id != 5' );

Since the between operands are relational aliases, and not literals or short aliases, Doctrine does not attempt to parse correctly. I modified the behavior to do extra parsing on the expressions when BETWEEN is encountered as the operation.

The Goods

• The Patch
• The Ticket

Guilherme from the team messaged me last night to let me know he’s going to commit the patch. I just checked and it’s not there yet, so if you need this fix immediately, you can apply the patch above.

In “Four Reasons to Love CouchDB, Part 1” I expressed how easy CouchDB is to implement and learn, despite being different than what you’d be used to using any of the many popular database platforms out there. I even suggested that you don’t even have to learn a server-side language to interact with it. Now, lets move on to our next reason:

Reason #2: It’s not a RDBMS

Relational database persist data in a very complex manner. One of my professors a while ago (do they call them that at Community College?) used an analogy of a garage to show why they don’t always make sense. “If your garage was a RDBMS, and you wanted to store your car in it, you would remove the engine, wheels, steering wheel, gas tank, etc and insert them individually in a neat manner. When you wanted your car back, you’d select all of the parts and reassemble them.” This works well if you ever want to quickly find the serial number and displacement of your engine, or want to share tires with another car, but doesn’t work well when you (more commonly) just want to drive your car. Lets not forget to mention the schema changes that would need to be made if you brought home a motorcycle!

CouchDB is a Document-Oriented Database

In MySQL, we’re often taught to normalize data to increase performance and reduce data duplication. This results in lots of “JOIN category ON category.id = article.id, JOIN author ON author.id = article.id” just to fetch all of the useful information for a single object. CouchDB is a Document-oriented database — All data for a specific object is self-contained, within the same document. This results in data being persisted in the same way that we think of it, with sacrifice for data duplication.

But that doesn’t mean that ALL data is duplicated of course. We can still retain the concept of “foreign keys”, but we use them minimally. The idea is to be semistructured.

A real world example.

We want to store articles, each with an article and collection of comments. We want to be able to show an article on a page, along with it’s comments. We also want to be able to get all comments across all posts for a “latest comments” page.

In MySQL, we would normally normalize all of this data into separate tables, and create foreign key references. There would be a single table for authors, a table for articles, and a table for comments. With this complex schema, we’d end up with queries such as the following

1

SELECT * FROM articles LEFT JOIN author ON author.id = article.author_id WHERE article.id = $articleId

To show the article information, and then a second query to show all comments for that article:

1

SELECT * FROM comments WHERE article_id = $articleId

In CouchDB, we can easily store all of this data within the same document, without hindering out ability to query it. Although schema-less, CouchDB does have the concept of “views”, which are similar to the concept of Views in RDBMS. Views are functions stored within special documents called “design documents”. They contain a field named “views” which is an array of JavaScript (the default language) functions used to filter (map) and sort (key) document data, and to generate aggregate results (reduce).

The following document represents all of the data we stored in our MySQL schema above, except that it’s all stored within one object.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

{
    "id" : "article_id_2",
    "type" : "article",
    "title" : "Some Title",
    "author" : {
          "name" : "A.J. Brown", 
          "email" : "nospam@ajbrown.org" 
    },
    . . . 
    "comments": [
        { 
            "name" : "John Smith", 
            "text" : "A.J. Brown is the best, ever",
            "created" : "2008-12-21 01:23:24 AM"
        },
        { 
            "name" : "Jane Applebees", 
            "text" : "I totally agree with John",
            "created" : "2008-12-21 02:55:24 AM"
        },
    ]
}

Although everything is stored within the structure of a single document, We can still get all of the data we need easily. We just need a view that will key our documents on the article id:

1
2
3
4
5

function map ( doc ) {
    if ( doc.type == 'article' ) {
          emit( null, doc ); 
    }
}

This function says “Show me all documents with a field named “type” that has a value of “article”, and sort and collate (key) them by their document id”. Now we can call the view and get all articles and their comments. By adding “?key=” to the URL for our view, we can retrieve only 1 post.

To retrieve only all comments of all posts, we’ll need to create a second view. This view will extract the comments from every document has a type=article, and collate them by them by the comment’s created field.

1
2
3
4
5

function map ( doc ) {
	for (var i in doc.comments) {
		emit(doc.comments[i].created, doc.comments[i]);
	}
}

Our resultset would look something like this: (note, view result sets have key and id information. I’ve left those out for simplicity sake):

1
2
3
4
5
6
7
8
9
10

        { 
            "name" : "John Smith", 
            "text" : "A.J. Brown is the best, ever",
            "created" : "2008-12-21 01:23:24 AM"
        },
        { 
            "name" : "Jane Applebees", 
            "text" : "I totally agree with John",
            "created" : "2008-12-21 02:55:24 AM"
        },

Since we keyed our view on the comments created field,we can easily narrow our results by date by specifying a startKey and an endKey in our request URI:

http://couchserver/database/_view/articles/comments?startkey=["2008-12-21 02:00:00 AM"]

This would only return “Jane Applebee’s” comment, since the specified start key come sequentially after the rest of the results.

Handling the custom fields is simple with CouchDB

Conclusion

Storing data this way makes a lot of sense in a lot of cases. For example, my newest project needs to be able to store system events. Every event has (and enforces) a standard set of fields, but there may be additional custom (unpredictable) fields as well, as defined by whatever process sends the event. Those custom fields must be available for querying by any program that does know of them. Handling the custom fields is simple with CouchDB — I just add them to the document being stored. A relational database schema for this data model would be complex and hard to follow.

RDMS’s are powerful, but sometimes they’re overkill for our applications. I’ve seen some hypothesis suggesting that CouchDB’s document oriented strategy would work for 90% of web applications today. Who knows how accurate that is, but it definitely makes you think. After all, you can’t get any more cutting edge than a database engine asking us to rethink how we model data which is written in a language that asks us to rethink parallel processing.

Check back tomorrow for Reason #3

Special thanks to the following blogs for helping me get started with CouchDb:

• about:cmlenz
• Ricky Ho
• Barking Ignuana

I’ve been sensing a lot of hype around CouchDB lately, and I recently decided to give it a try. After about ~30 hours of working with it, and a successful prototype of a project driven by CouchDB (and special shout-out to Zend Framework and Phly), I think I may have found a favorite new toy to share it with you.

This will be a multi-part series with each article touching on a single aspect of CouchDB. I enjoy MySQL and PostgreSQL as well, so I hope these posts aren’t taken as a knock on them, but more of a thumbs up to CouchDB.

What is CouchDB

Taken straight from their website, “Apache CouchDB is a distributed, fault-tolerant and schema-free document-oriented database accessible via a RESTful HTTP/JSON API.” It’s designed to drive today’s rich internet applications with scalability and even offline-operation in mind. If you’re interested in reading technical inforamtion about CouchDB, check out:

• Apache CouchDb’s Website
• Why CouchDB?
• Damien Katz, Creator and Project Lead

Reason #1: It’s Easy To Learn

My first experience with databases was Microsoft Access. I remember writing a Maxis fan site in ASP using an Access database which I manually updated every so often to store all of my content. I was able to leverage the “power” of Access (note: I was 13 or so at the time.), but it took a bit for me to really comprehend the features, and how my data should best be stored.

Fast forward to now. Companies are paying database administrators pretty good salaries for their database engineering skills. Even companies with rather simple data-structures need an experienced and talented database developer to keep their data schema scalable. It’s becoming a very specialized skill, such as the multi-hat of “Webmaster” has nearly vanished and split into “Graphics Designer”, “Web Programmer”, and “Systems Administrator” (in the simplest of cases). Writing queries can be the simplest thing you’ve ever done, or can cause you to pull out your hair, depending on the data needed. Lets not get into replication and clustering.

Database systems aren’t simple bread baskets.

CouchDB is very easy to learn. That’s actually a two-part reason, depending on your expertise. If you’re a developer or database expert, It’s not that MySQL or PostgreSQL are hard to learn, but rather that CouchDB is easy to pick up and run with despite being philosophically different. Thinking outside of the box of relational data that has been etched into my brain for the past 8 years was tough at first, but within a few hours of reading examples and playing around with it, I was able to start harnessing the full feature-suite.

Who needs a server-side language, anyway?

If you’re a front-end developer who’s never worked with a programming language other than JavaScript or Flex, there’s no SQL or complex protocol to learn. In fact, you don’t need any application layer whatsoever to start playing around with it: CouchDB operates completely over HTTP. It cuts out the “middle man” by serving data in JSON. Hell, the administration tools is completely AJAX based. Creating databases and views, querying data, and modifying data is done all through a RESTful interface. Imagine a world where Flex developers don’t even have to talk to PHP developers

I encourage you to start playing around with CouchDB. Keep in mind that CouchDB is still in an incubation phase, so it hasn’t even met its full potential yet. This project is very exciting to me, and I will continue to watch it’s progress.