Cloudant Blog

Search 2.0

2012-08-06T00:00:00-07:00

Background

Before I joined Cloudant, I worked in what we now simply call “Enterprise”. Not, you understand, the one where you go into space and meet aliens (and maybe learn a little about the human condition) but the other one, where you go into meetings and meet managers (and learn a lot about the human condition). It was there that I first encountered Lucene.

This was before the days of Solr, and way before the days of Elastic Search, a time when a Lucene index was an island, entire of itself. I was involved in building a proprietary clustered search platform and saw up close just how difficult that can be. Later, I was searching for some foundational technology to build a distributed archival system, having failed to create one of my own (a few times). Happily, this was around the dawn of “NoSQL”, as we no longer call it, and I found CouchDB. This database made so many good decisions that I was able to overlook that it was written in some arcane language I’d never heard of. I had already come to value the crash-only mentality that CouchDB enforces (tragically, by building a complicated system with elaborate and brittle shutdown code) as well as the append-only nature of disk writes and the all-in-one file format. The degree to which CouchDB’s API anticipates clustering was considerable too (albeit incomplete).

I could use this to build what I needed. To cut a long and winding story short, I didn’t. I joined Cloudant instead, who already had.

Before that, however, I had become an active CouchDB contributor and, a little later, a committer and community member. Further, I felt there was a gap that I was well-positioned to fill; the ability to index CouchDB databases with Lucene. This project, with the remarkably bland name, is, of course, couchdb-lucene.

Search 1.0

The first question to answer about Search 2.0 is “What was so wrong with Search 1.0?”. I could write a lot about this but much of it would be pedantry (not yet an Olympic sport, so my trophy shelf remains empty) but two important points remain. Firstly, the code base was large (relative to the feature set) and the original author had left. Secondly, and much more importantly, was the choice of on-disk format for the indexes.

For those familiar with Lucene, it suffices to say that Search 1.0 does not use the familiar index format you know. For those that don’t, or wouldn’t object to a reminder, Lucene’s on disk format is something splendid. It’s highly optimized for full-text search access patterns where you necessarily have to examine every document you have looking for the top-scoring matches. Specifically, it’s an inverted index where the keys are words (“cat”, for example) followed by a contiguous, ordered list of document id’s that contain that word (and where they contain it). A separate index enables you to efficiently find the index for a given word. There are many more things in there than I’ve said, and more optimizations, but this is the core of it.

Search 1.0 used the Lucene library for analysis but used clustered CouchDB views for the index. The idea, of course, being to reuse all of Cloudant’s considerable infrastructure around managing secondary indexes. It was anticipated that the less optimized data structure of the view would be compensated by the parallelism introduced from sharding.

This worked adequately but never spectacularly, and we wanted to do better.

What Search 2.0 Isn’t

Search 1.0 is written in Java and uses Jinterface to enable interprocess communication between the JVM and our Erlang nodes running everything else. Java is a verbose language in general, and profligately so from the Erlang perspective. Mindful that my coworkers have no Java history, I investigated other JVM languages for a better fit.

I didn’t choose Scala but I did choose Scalang (which happens to be written in Scala). Scalang provides a natural interaction with Erlang, notably because it follows the actor model for concurrency. Of course it was necessary to learn Scala in order to develop against Scalang and, indeed, to make some substantial contributions to the Scalang project (process monitoring being the main one).

Search 2.0 isn’t based on Apache Solr or Elastic Search. Why not? Well, Cloudant already has production quality solutions to the clustering problems we face; sharding, fail-over, read repair, and the like. Search 1.0 was right to use that infrastructure.

What Search 2.0 Is

This seems a good time to note that I don’t call this “Search 2.0” at all. What we’re branding as Search 2.0 is, internally, two components, Dreyfus and Clouseau. Dreyfus is about 800 lines of Erlang and deals with plugging into existing Cloudant technologies like Chttpd, Fabric and Mem3. Clouseau is about 1000 lines of Scala which provides access to the Lucene library via Scalang. For (a not entirely fair) comparison, Search 1.0 was a little under 7000 lines of Java.

Lucene is an excellent library and to use it well is to largely stay out of its way and not be stupid. Lucene 3.6, the latest at the time, is a much more refined release than when I started, back in the 1.4 days. Search 2.0 deliberately exposes only a part of the Lucene feature set so that we can make the core bullet-proof but there are quite a few exciting things I have my eye on, which will have to wait for a future blog post.

Readers familiar with couchdb-lucene will know that it uses Rhino for its javascript processing. Search 2.0 does not, it uses the same SpiderMonkey as the rest of Cloudant. This ensures we evaluate Javascript the same way in views, update handlers, and index functions and leverages our tools for managing and monitoring couchjs processes. Those readers will also notice that the Search 2.0 API is not the same as couchdb-lucene’s. I took the opportunity to design a simpler API, I hope you like it (Some details here). There are some planned enhancements to that API that will be deployed when the features they enable are ready for production.

Today, Search 2.0 provides a rich query language into your clustered indexes, including a minor tweak to the standard Lucene query syntax, namely that a range query where both ends look like numbers are transformed into a numerical range query. So queries for price ranges, or latitude/longitude ranges will work as you’d expect. Simon Metson kindly produced a nice demo of this here.

In addition to rich searching is a feature I’ve called “Bookmarks” for reasons I hope become apparent. We’re all familiar with the notion of “pagination” if only because of the Gooooooogle links we never click. In Search 2.0, every search result returns a “bookmark” value in addition to the number of matches we found and the first “page” of matches. Passing this value back in will return results on the next page. There’s a caveat, though. It’s quite possible that, between two searches, a new document (or an update to an existing document, or removal of a document) will alter the scoring such that you will see results from an earlier page.

We will be posting a full description of the API and all the features in the For Developers of our website.

We're on a Mission to Make it Easy

2012-08-01T00:00:00-07:00

Cloudant isn’t like other databases. So it goes that the content we present to developers in our newly minted For Developers section isn’t like other tech documentation; i.e. it’s not static. More on that in a minute.

First, a bit about the world we live in…

Cloudant is a hosted, distributed database service. Apps write to and read from Cloudant using our RESTful API. The payload is lightweight JSON. For all of that, you may be thinking it’d take a lot time to get Cloudant set up in order to play with it, and kick the tires. Nope. All it takes is the five or so seconds to fill out the sign up form.

And so…

So long, static

More than just docs, the For Developers section is a live demo of the data layer at work. Click a couple buttons to CRUD, get JSON via secondary indexes and search the sample database that you replicated into your account. It’s a breeze.

Technical references (including handy curl commands) run inline with each of the demos.

So long, friction

As a newb noob N00B who is developing his first application on Cloudant, it is no small comfort to me to play with live demonstrations alongside the reference material in here. Not only are the docs in-step with the product (whew), but also it’s clear right away that the thing just works. And fast.

Last night, I had the Olympics on in the background as I worked on my first Cloudant-driven app. When I picked up my head to look around, prime-time coverage was long over. Kayaking was on. It was 2am.

The hours that disappeared on me weren’t spent banging my simple brain against complex SQL statements, and writing and rewriting blocks of server-side code. The time was spent playing around with the data coming from my Cloudant database, moving it around the interface I’m working on. Which is a long way of saying that time flies when you’re having fun.

Getting started

I went from setting up my database to playing with the interface in minutes. Here’s how I got there.

Step 1 I put my data (JSON) into Cloudant with a bulk insert.

Step 2 I built my search index inside my Cloudant design doc (the same place I’ll keep my secondary indexes).

{
  "_id": "_design/people",
  "_rev": "13-24d8a5edfd54d61dacb0bc4230eee28a",
  "language": "javascript",
  "indexes": {
    "searchByName": {
      "index": "function(doc) {
        if(doc.docType=='person') {
          index('default', doc.last);index('default', doc.first);
        }
      }"
    }
  }
}

(My search index, searchByName, indexes the values of the first and last names of the people in my database so I can search for them by either of those or both in my application.)

Step 3 A little JavaScript to deliver it to the interface.

// Dependencies: JQuery and sag.js (http://www.saggingcouch.com/)
$(function() {
  // since I want to work locally (but use Cloudant data), I've set up a reverse proxy
  // more on reverse proxy: http://wiki.apache.org/couchdb/Apache_As_a_Reverse_Proxy
  // use sag.js to connect
  var user_db = 'my-Cloudant-database-name';
  var couch = sag.server('my-server'); // localhost
  couch.setDatabase(user_db);

  //  capture the search form submission
  $('form.search').submit(function(e){
    //  kill default action
    e.preventDefault();
    // get the value of the search input
    var query = $('#last').val();
    //  get data via sag.js
    couch.get({
      //  query my Cloudant search index (see above) and account for wildcards (partial string searches)
      url: '/_design/people/_search/searchByName?include_docs=true&q=' + query + '*',
      callback: function(resp, succ) {
        // vars for iterating over the results
        var rows = '', i;
        //  create the rows for my table
        for(i in resp.body.rows) {
          rows += '<tr><td>';
          rows += '<a href="#" class="view-detail data-cloudant-id="' + resp.body.rows[i].id + '">';
          rows += resp.body.rows[i].doc.first + ' ' + resp.body.rows[i].doc.last;
          rows += '</a></td></tr>';
        }
        //  drop the rows into a table and append it to the results container in my interface
        $('#results').html('<table id="people" class="table table-striped"><tbody>' + rows + ' </tbody></table>');
      }
    });
  });
});

The launch of the For Developers section is a small but important step for us. A first step toward a full-featured library of useful content for people like me and, we hope, people like you. To that end, if you’re a developer using Cloudant and have ideas about docs or demos you’d like to see, please reach out. We’d love to hear from you.

Big Science, Big Data

2012-07-23T00:00:00-07:00

Big Science, Big Data

Simon Metson started working for Cloudant in March 2012 as an “ecology engineer” and became a full time employee in July. He is based in the UK.

Before I joined Cloudant, I worked on the Compact Muon Solenoid (CMS), one of the two Large Hadron Collider (LHC) detectors that announced their discovery of the Higgs Boson earlier this month. I was a member of this collaboration for 10 years, from the start of my PhD through a bunch of post doc positions (all based in the UK at Bristol Uni), so was really excited and pleased to see the result being announced. It is the result of an amazing endeavor, where everything (hardware, software, wetware) has been pushed beyond known limits and, more importantly, delivered.

Candidate Higgs event - copyright CERN/CMS See the original

For most of my time with CMS, I was involved in one way or another with data and workflow management (DMWM); designing and delivering software that would be able to process, catalog and transfer the tens of petabytes of data that the detector would deliver each year, and for the last four and a half years I ran the DMWM project with a colleague from Fermi Lab.

CMS has a tiered, distributed computing system. Initial processing happens in the Tier 0 (at CERN). The resulting data is shipped to 7 national labs (Tier 1’) located in US, EU and Asia for second pass processing and offsite tape backup. Once on tape the data is made available to the collaboration; any CMS collaborator can request the transfer of any dataset to their local Tier 2 cluster. Last time I check we’d moved over 100PB of data around and had about 50PB of data active in the system.

CMS 2011 transfer link map - copyright Paul Rossman See the original

The DMWM project writes the software that manages these transfers, records and manages the metadata for all the datasets and submits and tracks all the processing jobs through the various batch and grid systems for both organized central processing and more ad hoc user analysis.

Getting the job done

Watching how the computing system of CMS evolved over the last ten years from something that required a lot of hand holding to something that could be relied upon to deliver, often well beyond specification, was a great experience; seeing the sites, organization, software and people mature has had a profound effect on how I think about the design, implementation and management of software projects.

The nature of these experiments is such that any and all tools are available for use. We have a complex problem to solve, which results in a complex solution. While we rationalized a lot of things (making sensible decisions can save you thousands of lines of code) we still ended up with quite a range of supported technologies. Our stack used (from memory and in no particular order):

Languages: Python, Perl, C++ and a bit of Java
Databases: Oracle, MySQL, CouchDB, MongoDB
Messaging: ZeroMQ, MonALISA
Grid middleware stacks: EGEE, LCG, OSG, and ARC
Batch schedulers: PBS, LSF, Torque, Maui, SGE, Condor
Distributed file systems: DPM, HDFS, CASTOR, dCache, Lustre, GPFS).

Wrangling that stack sure was entertaining…

I think CMS was the first experiment to start using NoSQL tools in production; HDFS is used at some of the T2 sites for their storage system, MongoDB is used as a smart cache aggregating data from various data services and CouchDB is used for various state machine and monitoring tasks. CouchDB’s replication feature was especially useful in CMS’s distributed computing system. Being able to quickly build aggregated views of monitoring or state information was great, as was having everything served via a REST HTTP API. Having it all in one application was even better!

From Higgs to Cloudant

I met Mike Miller at CERN in 2007 (well, in a fondue restaurant, I think), had followed Cloudant from its inception and we’d kept in touch after he left CMS to go look at neutrinos and dark matter at University of Washington, build a company and a set of desks. He contacted me at the end of 2011 to see if I’d be interested in joining Cloudant.

Like a number of my former colleagues, I felt that it was time to move on. As Mike once said to me “we’re builders”, and I wanted to be building new things again. I’d experienced working in a large (CMS has about 4000 people involved, and is spread over nearly 120 institutions), collaborative environment with all the benefits and issues that they bring and decided that seeing what life is like in a startup would be interesting.

Am I sad that I left CMS before the discovery announcement? Not really. The nature of these collaborations is that everyone who makes a contribution is recognized; the papers are signed by “the CMS collaboration” and I am a member for the next two years. While I wasn’t directly involved in the analyses that were presented on the 4th, the software that I was responsible for enabled that research. That’s something I’m quietly proud of.

What’s great about joining Cloudant now is that I get to see that whole evolution and scale up cycle again. Everyone in the team knows that we’re just at the beginning of a very interesting journey and is really excited to see where the company goes over the coming years. I’m really looking forward to helping the company scale up; the software, the organization and, importantly, the customers.

Cloudant Helps Light Up the 2012 London Olympics

2012-07-19T00:00:00-07:00

As much as we’d like to say Cloudant athletes will be lighting up the 2012 London Olympic track, playing fields, or pool…instead, Cloudant technology will be used to help light up…the Ferris wheel.

Tonight at 9PM local time in London, Energy of the Nation, the world’s first social-media-driven light show will launch at the EDF Energy London Eye. The nightly light show, and accompanying realtime Energy of the Nation infographic and “Energy2012” mobile app will measure the mood of the UK during the Olympics and Paralympics by performing sentiment analysis on Olympics-related tweets sourced from the UK’s 10 million Twitter users.

(source: Sosolimited.com)

The campaign was conceived and developed by the creative geniuses down the street from us in Boston, Sosolimited in conjunction with UK-based brand experience consultancy, Ignite. The project was sponsored by EDF Energy, which is a sponsor of both the London Eye and the Olympics.

How Energy of the Nation Works

Sosolimited has written software that captures Olympics- and Paralympics-related tweets in the UK. It has been capturing tweets since May 1st of this year. The tweets, and a vast library of emotional words, punctuation and emoticons, are stored using the Cloudant Data Layer service (hosted on SoftLayer cloud infrastructure in Amsterdam).

The Sosolimited software runs sentiment analysis algorithms on the Tweets in Cloudant to determine their emotional content. The analysis output reflects the rising and falling mood of the nation with respect to the games and is charted throughout the day on a large LED wall next to the London Eye. Every night at 9pm, that data is to be played back in a spectacular, 20-minute, low-carbon light show featuring full-color architectural lighting fixtures around the Eye and large, ground-based search beams, all powered by EDF Energy. There’s more good info on how this all works located on the EDF Energy website.

We’re psyched for the Olympic games to begin, and even more psyched that Sosolimited chose Cloudant to play a role in the event. If you haven’t seen Sosolimited’s catalog of interactive and multisensory design work, check it out!

Managing applications on Cloudant

2012-07-17T00:00:00-07:00

A fairly common question we get asked is “how do I manage my view code in insert your favorite version control system here?”. As people develop more complicated apps this question becomes increasingly important. Fortunately there are a set of tools that make transferring code from a version controlled directory into your database easy.

At the heart of Cloudant is Apache CouchDB. Applications that run out of CouchDB are known as “CouchApps”. At their most basic these applications are nothing more than a design document containing a view. There are a raft of tools that simplify the development of these applications that are served directly out of CouchDB. They each target slightly different use cases, from a minimal wrapper to upload code into the database to a fully featured development platform.

At the most basic level these tools enable developers to version control their views and other code, and “compile” and push them into the server. More advanced use enables development of “two-tier” applications, pulling in various libraries to build complex, dynamic applications; even ones available offline.

The tools

Below is a quick summary of the various tools that let you build and push code from your local system into the database. It’s by no means exhaustive, and I’d love any corrections/clarifications/additions.

CouchApp.py The original and reference implementation for CouchApp clients this python application is no longer under active development (Benoit has moved on to Erica) but it is still widely used and recieves occasional updates. It turns the structure of a design document into a filesystem layout, mapping the various functions of your application to files in a defined location on your disk.
Erica Erica is a CouchApp client built in erlang, effectively the next version of CouchApp.py. It has basically the same feature set as CouchApp.py.
Soca Soca takes a different approach to CouchApp.py/Erica, allowing any directory structure for the app and compiles the application into a single json file following a layout defined in a file, rather than the filesystem itself. It builds applications around Sammy.js by default.
node.couchapp.js node.couchapp.js is a node application for building CouchApps. It requires all the functions (views, lists etc) of the application be defined in a single json file, with attachments (images, html, external scripts) loaded from a defined location.
Reupholster Reupholster is a Java application that builds CouchApps in the same style as node.couchapp.js. It provides an end-to-end development tool, managing creation of the application, automated testing and pushing to the server.
Kanso Kanso is another node application, described as “the NPM for CouchApps”. It has a very comprehensive set of packages that enable easy management of various third party libraries and frameworks.
Situp situp.py is modeled after the python setup.py install script. It is a single self contained file with no dependencies. It’s the tool I use for my own applications that I develop on or for Cloudant, but it’s also written by me so you’d hope I would use it! It aims to support development of more complex CouchApps, and can pull in Kanso packages.
Mouch Mouch takes the metaphor of a build file a step further, resulting in a minimal application “for people who love purism. For people who keep complete control over their bytes. For people who love things speeding fast, keeping dependencies as minimal as possible.”

Summary

CouchDB has supported running applications out of the database since 2009, and the associated tools are in a reasonable state of maturity. It’s nice to see other databases getting similar functionality now; it can only be a benefit to developers creating two-tier, dynamic applications.

For very simple tasks (e.g. pushing a view into the database) there isn’t much to differentiate between the various tools; it’s a very subjective choice. As you go “further down the rabbit hole” you may find one tool fits your application or workflow better than another.

A Good Week for PHP and Cloudant

2012-06-30T00:00:00-07:00

Little known fact: about 20% of Cloudant Data Layer users are PHP developers. So, last week we were excited to announce that Cloudant is now available as an add-on at cloudControl PHP hosting platform, a cloud application platform for PHP developers (with Ruby and Python support coming soon).

The announcement follows several weeks of integration work and testing on both sides of the partnership, and we hope PHP developers can develop faster and scale their apps and databases easier as a result of the effort.

cloudControl account holders can add Cloudant to their service via the Cloudant Add-On page at cloudControl: https://www.cloudcontrol.com/add-ons/cloudant

Cloudant featured in a book on CouchDB & PHP Development Coincidentally, we also got a note from author Tim Juravich (@timjuravich) last week letting us know he had just published a book on developing PHP web applications on CouchDB. In the book, he walks developers through the creation and deployment of a simple social networking app using tools like PHP, CouchDB, Sag, Bootstrap, jQuery, PHPFog, and Cloudant! You can get the book here.

Lean and Technical-- How we chose and furnished an office the YCombinator way.

2012-06-26T00:00:00-07:00

Cloudant’s growing, and we recently opened offices in Seattle and the UK (in addition to our headquarters in Boston, near Cloudant’s MIT birthplace). As a YCombinator company, Cloudant is conditioned to run lean, but we also have an exceptionally technical team with an appreciation for good design. This made finding and furnishing our new office particularly challenging. Here’s the story of how we did it in Seattle, aka “Cloudant West”.

Location.

In his seminal 2005 article “How to Start a Startup”, Paul Graham writes, “for a startup location is very important.” We couldn’t agree more, and it’s why we surveyed five different neighborhoods in Seattle: Downtown, Pioneer Square, Capital Hill, Fremont/Ballard, and South Lake Union. Our criteria were loosely based on:

Environment. The energy of the neighborhood, what other companies (large and small) are nearby, and where do we like to spend time?
Food. Access to a diverse selection of good, cheap food is critical to a startup. Lunch is often where the best ideas are born, let’s make sure we step away from our laptops long enough to think and talk.
Space. Where can we find access to high ceilinged lofts with plenty of light and just enough partitioning into rooms?
Cost. Real estate costs vary significantly.
Connectivity. We wanted to be at most one switch from a fiber drop.
Commute. Seattle doesn’t have a subway and neighborhoods are separated primarily by water. A central location is key.

Environment concerns quickly narrowed down the list. Downtown is rather soulless during and after business hours. Fremont/Ballard and Capital Hill are amazing places to live and hang out, but they don’t have the necessary concentration of other companies. I wanted to love South Lake Union, primarily because it’s close to the lake and Amazon and biotech are absolutely transforming the neighborhood. Everything is shiny, new, and most intersections have their own cop to direct traffic (AMZ makes things happen). Unfortunately, aside from Techstars it doesn’t have many companies in our space. The kicker was the small selection of food and the surprisingly high costs. That left us with Pioneer Square, which luckily maps well onto space, cost, connectivity and commute. There are a multitude of lofts with just enough brick, priced well, and close to the all-important fiber. To boot, we have several customers around the corner, Future Advisor (YC10) across the street, easy access from three directions, and are right next to Century Link and Safeco fields.

Next step – find a good space.

Space.

Ben Anderson took the lead at scouting locations. I should say that commercial real estate is due for disruption. The fact that agents still control the listings and cost is absurd and creates an inefficient market. That said, we followed advice from several advisors and were introduced to a trusted agent that put plenty of good properties in front of us. We found it challenging to envision how we would lay out an office, so we called in the experts – my fiancee LJ. I delivered a blueprint to LJ and before I knew it she gave us a Google Sketchup model for layout and a video to send back east for the final sign-off. Here’s the video from the winning space; thanks LJ.

We decided that we could nicely lay this space out for 10 people without feeling crowded. We could get desks (more on that below) that would be easily re-arrangeable, and we could have space for sales and bizdev, meetings, open discussions, and door-closed, heads-down coding.

Furnishings.

This was hard. Our office in Boston came furnished. Since this time around we had a blank slate and some time to think, we decided to be creative. Ikea desks are actually OK, I've had many in my time. But, we wanted something just a little different, with superior function, fit/finish, and an imprint of the Cloudant brand: technical, well made, and aesthetically pleasing. We decided to build instead of buy, and to improve on Amazon's famous door desk design. Blueprints, specifications and instructions are available upon request. I'd like to say that I had a big hand in this, but really LJ led the charge (startups are a family affair). For the first prototype she selected a beautiful recycled door that she bought for $20. It required stripping and staining. Here's a look at the prototype in our living room in March. This door had recessed panels. We investigated simply covering the door with a sheet of glass but found that it was expensive, prone to finger prints, and sharp on the edges. Instead she filled the recessed portions with plywood. The inserts are held from below by screws and then the entire surface (door + insert) sanded to a uniform height and finish. Using multiple stains gave us a cool two-tone finish and then the entire door was coated with multiple layers of varnish.

This prototype turned out to be beautiful but labor intensive, so we instead chose to purchase solid core doors from Home Depot (~$40 each) and proceed to sand, stain, and varnish them all. We (meaning LJ) used the lumber purchased for the desk frame and legs to ingeniously stack doors so that we could varnish them in our tiny garage. It's critical to varnish the doors in a level horizontal position and in a particulate free environment. We recommend two coats of stain minimum and three coats of varnish for a uniform and smooth finish.

The support frame to turn the door into a desk was also a challenge. Several configurations were tried and we settled on running a single 2x4 on edge underneath the door running the length for support. That board is screwed into the underside of the door using small right angle metal tabs and provides a mount point for legs on each end of the door. The legs are cut and assembled from 2x4 pieces as well. The bonus is that all of the support structure can be cut, drilled, and assembled ahead of time. Assembly consists of screwing the support 2x4 into the underside of the door, connecting the legs to the support, and securing the legs to the underside of the door with right angle brackets.

We had quite a bit of fun making these desks. We got help from a longtime friend and advisor of Cloudant Ross Rosen and YCombinator's own Tony Wright (seen pulling quite a face below left). Tony, always the designer, augmented LJ's initial tapered leg design by rounding the corners (trying his best to make the table look like a big beveled "call to action" button). We ended up with a stack of gorgeously rounded legs, but they were a bit tricky to hold together, drill, and screw together without someone holding them in place with inventive yoga moves.

These desks are huge; you can comfortably fit 4 laptops if you use both sides, but we give one to each employee. It’s highly conducive to teamwork, as it’s easy to pull up a chair and work with somebody to solve a problem, review a design or test an interface. The rest of the office we furnished primarily from Costco business, and we’ve been particularly happy with the ProGrid chairs (highly adjustable, robust, comfortable) for under a third of their list value.

It took some work, but Cloudant West is a space that speaks to the ideals of YCombinator and Cloudant – build great things that people really like. I love our new colony in Seattle, and I really love my homemade desk.

Cloudant Data Layer Now Available on Windows Azure

2012-06-11T00:00:00-07:00

We are very excited to announce that the Cloudant Data Layer now runs on Azure. As a result, .NET, Node.js, PHP, Java, and Python developers can now enhance their fast-growing Azure apps with a scalable data layer that is co-located with their application servers in Azure data centers.

It’s great to see Windows Azure strengthen support for open source development technologies and hybrid clouds. The addition of Linux-based virtual machines made it possible for us to offer the Cloudant Data Layer service on Azure

We’re psyched to be active members of the Azure community now…the response to our announcement last week has been exciting. For those of you who are new to Cloudant, we offer a NOSQL database service (with an Apache CouchDB-compatible API.) A couple of things set Cloudant apart from other SQL and NOSQL database options available to Azure developers:

Cloudant is a service: we provision your data layer and then keep it running non-stop and scale it as you grow. Unlike scale-it-yourself DBs that you have to install and manage in the cloud, Cloudant’s back-end DB operations (provisioning, backup, scaling, re-partitioning, etc.) are taken care of for you by us, the experts.
Global data distribution: we push your data to the edge–to Azure data centers around the world–so it’s closer to users for low-latency data access.
Data Replication & Sync: enable offline data access & collection (especially to mobile devices) or use it to replicate data from the cloud to on-premise data centers, so you are never locked into Cloudant or Azure.

Cloudant on Azure is still in “preview” mode as we continue to tune it for performance and expand it to additional Azure data centers in North America, Europe, and Asia. Please let us know how it is working for you and what we can do to improve it.

To get a Cloudant Data Layer account running on Azure, please visit our signup page and choose “Lagoon” as your data layer location.

If you already have a Cloudant account and you want to move it to Azure, just follow these directions:

Log into Cloudant
Choose the ‘Account’ tab
Change the placement of your cluster to ‘LAGOON’
Cloudant will notify you after your account has been relocated to Azure

Cloudant Data Layer on Joyent Cloud

2012-04-11T00:00:00-07:00

We just added Joyent Cloud to the list of Cloudant Data Layer locations.

Joyent Cloud is a high-performance environment for running apps and data layer services in the cloud. Backed by Joyent SmartOS - an operating system optimized for multi-tenant cloud environments - and the ZFS file system, Joyent Cloud enables Cloudant to take advantage of specific caching and data backup features in the ZFS file system and deliver outstanding data layer performance and reliability Aside from Cloudant, Joyent Cloud also powers a growing number of cloud platform and software services providers including Nodejitsu, StackMob, and GameSalad.

Learn more about Joyent Cloud

Create a Free Cloudant Data Layer running on Joyent Cloud

Getting started is easy. Just create a free Cloudant account, and select the Joyent Cloud option “Paloma” from the list of locations, as shown below.

Or Move Your Data Layer to Joyent Cloud

If you already have a Cloudant account and you want to move it to Joyent Cloud, follow these instructions:

Log into Cloudant
Choose the ‘Account’ tab
Change the placement of your cluster to ‘Paloma - Joyent, US Southwest’
Cloudant will notify you after your account has been relocated to Joyent Cloud.

Cloudant CouchDB Summit Drinkup

2012-04-09T00:00:00-07:00

Cloudant is hosting the CouchDB Summit this week in Boston and we will be capping off the two-day affair with a public drinkup on Tuesday night. We invite everyone to come visit with the comitters and join us in wrapping up the summit. Just look for the folks in good lookin’ Cloudant swag. Details:

Tuesday April 10th
6 - 9pm
Lucky’s Lounge
355 Congress Street, Boston, MA.
All Beers on Cloudant’s Tab

The anti-sprint

2012-03-28T00:00:00-07:00

Every Monday morning the Cloudant engineering team jumps into a google hangout for our weekly “train” standup 1 where we swap stories from the weekend, I defend my hometown teams (MSU and Detroit), and decide what code to include in the next release – tomorrow. Yep, that’s right, we push new code every Tuesday (at the slowest 2.) This may not seem astounding to some, but we power applications for nearly 10,000 users and have an impressive amount of transactional data under management, spread globally across 11 data centers.

In my past life I once labored for literally more than a year to get a new release cut, tested and pushed. In that light, I find our weekly/daily release rhythm pretty amazing. More importantly, I think it’s an example of Jay Krep’s compelling argument that, when it comes to distributed systems, the true value is not solely, or even primarly, in the underlying software, but in the service provided by those that run it. That ethos lies at the very heart of all things Cloudant. It drives our technology choices, product vision, roadmap, release schedule, operations, support, and (perhaps most importantly) business model. We write software to scale your data and we run it. Below I briefly expand on the tools we use to constantly push new code, and why that is important to our customers.

Tools

Erlang. Building our core stack as a collection of Erlang OTP applications and using Chef for configuration management means that we upgrade our running database applications without ever taking them offline or even dropping active connections. If you haven’t ever witnessed the magic of hot code upgrades in Erlang, you haven’t lived. New and old code live side by side, processes are suspended, a new code path kicks in, state transformations are made on the fly, and voila, you’re running new code without ever skipping a beat. Don’t like it? Hit the roll-back button.
Github + CI + Tests. Hot code upgrades would be worthless if we didn’t have good code to push. Github’s tools have really matured over the last few years and it empowers a very efficient workflow (pull requests, code review, etc) for a distributed team like ours. Coupled with Jenkins for builds and automated testing (etap, etc) we can efficiently go from idea to tested implementation.

Value

Cloud. This is the promise of Cloud – Isolating non-core elements of your technical stack and finding the best software and people to deliver them faster, cheaper, and more reliably you can do yourself. Making a quantitative ROI argument has always been challenging because it misses a key point, “it’s not just the software, but who runs it.”
Response. On stage in 2009, one of our first customers (who was piping the twitter firehose into Cloudant) referred to our software as young and developing. Ouch. But he followed up by calling us some of the best problem solvers he had encountered. The distributed systems used to handle Big Data problems are relatively new. Our team has decades of experience in big science and enterprise IT, building and operating some of the world’s biggest computing systems, and still we are able to find novel failure modes. Our ability to handle these failures gracefully stems from our deep working knowledge of the code and it’s operational characteristics.
Priorities. There’s little worse than investing substantial time and money into a feature/product that is irrelevant in the market. Thus, while we have a clear product roadmap and development plan, we take the just-in-time ethos to heart. “Somebody is trying to create 100 million databases (true story). We should probably re-write that LRU module.”

Culture

Team. Perhaps most importantly, this perma-release / anti-sprint schedule suits us. We are lucky to have an expert team where developers push code, trouble shoot, and close tickets. Nobody on our team is above jumping into #cloudant on IRC and working through questions/issues with customers. If Kreps is right (hint: he is) and the only real meaningful metric is “continuous hours of successful production operations,” then any distributed data system is only as strong as the people that run it, that is, the people that ensure continuous operations. In Cloudant’s case, the builders (kocolosk: rnewson: bitdiddle: davisp: banjiewen: and others) are also the operators. We’ve been running these systems in production for our customers for more than three years.

Since we began working on Cloudant we have seen some major public failures of distributed big data systems. In nearly all those cases, the failure was not one of software but of operations. This seems like a natural consequence of putting up a wall between those that build code and those that run it. At Cloudant, that wall doesn’t exist, and this is what allows our customers to grow into their data layer, not out of it.

Slides and Presentation from Meteor Solutions and Softlayer

2012-03-16T00:00:00-07:00

On March 8th we hosted a webinar featuring presentations from Meteor Solutions and SoftLayer. Ben Straley, CEO of Meteor Solutions, gave a presentation on how his company developed their social audience marketing platform on the Cloudant data layer (powered by Softlayer). Afterward, Nathan Day, Chief Scientist at SoftLayer, elaborated on SoftLayer’s inner workings and high-performance network, which allows it to provide a global platform for web applications. Kicking things off I described some of the pertinent technology behind and features of the Cloudant Data Layer, and how, together with SoftLayer, we helped Meteor Solutions keep up with their ever growing data volumes and customer demands.

For those that missed it (or just want to view it again), we recorded the event, which you can view below. If you are interested in a copy of the slides, you will find those at the link below the video.

Download the slides here

Andy Palmer Joins Cloudant Board of Directors

2012-02-27T00:00:00-08:00

We are happy to announce that Andy Palmer, serial entrepreneur and Boston startup luminary, is joining the Cloudant Board of Directors. Andy has a remarkable track record of starting and building amazing companies in Boston. His experience with and deep knowledge of Big Data systems (which he recently wrote about for GigaOm) will be invaluable as we build out the Cloudant Data Layer. Having previous worked with Andy, I know first hand how committed he is to building great companies in New England. His energy and passion are second to none and we at Cloudant are privileged to have him join our board.

For more information about Andy and Cloudant see our press page here

Announcing CouchDB Summit Boston

2012-02-20T00:00:00-08:00

We are happy to announce today that Cloudant is hosting CouchDB Summit: Boston on April 9th and 10th. It promises to be a great, productive time and we are excited to introduce everyone to The @Cloudanator in person!

Here is the announcement e-mail that was sent to the Apache CouchDB user and developer mailing lists today.

Hello community,

For a few months now we at Cloudant have been talking about hosting a CouchDB Summit in Boston for the Apache CouchDB committers and some community members. This got pushed to the front burner due to recent events in the community, chief among them the purchase of a kegerator.

The goal is to give the CouchDB community the chance to be in the same room to discuss the direction of CouchDB over a two day period. Some admirable goals could include what CouchDB 2.0 should look like, how to get there, talking about auth, etc. Plus plenty of hacking time.

The results of the discussions will be posted to the community for decision making. Think RFC.

This is an invite only event with a max capacity of 20 to 30 people. We want to make sure that there is progress and flooding a room would inhibit that. The Apache CouchDB Committers are of course invited, plus some community members.

Apache CouchDB Commiters, please let me know if you would need financial assistance to attend. Cloudant is open to providing financial assistance for flights.

The particulars…

When: April 9 & 10

Where: Boston, MA, USA (venue to be announced)

$$$?: Cloudant is NOT charging for the event

Party?: Yes, there will be a PUBLIC drinkup for the community. Details to be posted separately.

RSVP: Invites are going out this and next week.

Let me know if you have any questions or concerns. We hope to make this a fun, positive experience for the community.

Cheers,

Sam Bisbee

Paul Davis Elected to the CouchDB PMC

2012-02-16T00:00:00-08:00

Congratulations and a hearty round of applause are in order. Noah Slater, speaking on behalf of the Apache CouchDB Project Management Committee, recently announced that our very own Paul Davis has been appointed to that very PMC. According to Noah’s email to the CouchDB user and dev lists:

Paul Joseph Davis, a long time contributor to the CouchDB project, has been appointed to the PMC. He’s an asset to this community, and we’re delighted to have him on board. Please take a moment to congratulate him and show your appreciation for all his hard work so far.

Paul is an outstanding engineer and a pillar of the CouchDB community. We consider ourselves extremely lucky to have lured him to Cloudant last fall. He continues to be a staggering force within Cloudant, working on all aspects of our product, and still has the energy to contribute heavily to CouchDB. It’s a pleasure to see him rewarded for his efforts.

I have no doubt that he will make a fantastic member of the CouchDB PMC. With Paul helping to steer the project, and with version 2.0 coming up this year, I believe CouchDB has a very bright 2012 ahead.

Multi-Tenant Cloudant in Europe

2012-02-13T00:00:00-08:00

I have a confession to make. We have been neglecting our friends in Europe. More than a year ago we launched our multi-tenant data layer with clusters on the east and west coasts of the US. Since then we have added shared clusters in Singapore and Japan and private clusters in numerous sites around the world. But, despite the requests from developers in Berlin, Amsterdam, Dublin, and more, we resisted opening up a shared cluster in Europe.

This was folly and it ends today. We are happy to announce the immediate availability of our brand-spanking-new EU-based cluster, “Jenever.” Jenever lives in SoftLayer’s new data center in Amsterdam just a few short milliseconds round trip from almost anywhere in the EU and UK. No longer will our European developer friends be forced to send requests or users across the Atlantic to hit the DB.

BUT WAIT THERE’S MORE! You can now choose the location of your data when you sign up or at any time. We maintain four shared clusters around the world, “Julep” in Northern Virginia, “Meritage” in San Jose, “Sling” in Singapore, and “Jenever” in in Amsterdam. When you sign up for a new account you will be asked where you want your data to live. If you ever want to change clusters, you can switch via your ‘accounts’ page. Note that switching clusters happens seamlessly but asynchronously. It takes some time to move an account so don’t expect this to happen instantly.

We at Cloudant believe that your data should live where your application lives, and your application should live where your users live. Our goal is for the Cloudant data layer to cover the globe, providing developers all around the world low-latency access to their data. With Jenever in Amsterdam, we take one step closer to this goal.

Announcing BigCouch 0.4

2012-01-26T00:00:00-08:00

It is a big day here at Cloudant HQ; we are announcing the release of BigCouch 0.4! This release, which brings BigCouch into API equivalence with Apache CouchDB 1.1.1, has been baking for a while, and we are excited that it is now ready for public consumption. Instructions for installing and using BigCouch 0.4 can be found on the BigCouch page. Users running Debian Squeeze, Ubuntu (LTS or newer) or RedHat / CentOS / Amazon Linux are welcome and encouraged to use our prebuilt distributions based on Erlang/OTP R14B01 and SpiderMonkey 1.8.5.

There are a whole slew of new features and performance improvements in this release. You can read the entire list here, but I wanted to highlight a few of our favorite features. These should convince you to upgrade if you haven’t already.

For my money, the killer addition to BigCouch 0.4 is support for ‘zones,’ which allows you control over where you place the individual copies of your data. Cloudant’s hosted service makes extensive use of this feature to protect against isolated failures in individual data centers. As an example, our west coast cluster “Meritage” is actually spread over two different data centers in two distinct geographic regions. We use zone-support to ensure that at least one copy of all of our customers’ data exists in both locations. That way, if one location falls into the ocean or is subject to nuclear attack, our customers’ data is still safe and available. With this feature now in BigCouch, you can extend clusters across multiple locations.

Another important improvement we added, taken from Apache CouchDB 1.1, is support for the replicator DB to help manage multiple replications. The BigCouch replicator DB works the same as the Apache CouchDB version. A nice writeup of usage can be found here.

We’d like to thank everyone in the CouchDB community that has contributed bug reports, patches, feature requests, etc. for BigCouch and to all the brave souls willing to try out the bleeding-edge releases. This release would not be nearly the upgrade it is without all of you. Special shout out goes to Benoit for his help with FreeBSD and native SSL support.

Finally, for those following along with recent events in the land of CouchDB, we’re dedicated to donating and integrating BigCouch to Apache CouchDB. This sort of integration won’t happen overnight, so we are still planning on making releases to BigCouch while we iron out the divergences in the two projects. We very much appreciate the support and enthusiasm we have recieved from the CouchDB community to date, and we look forward to working with that community to strengthen Apache CouchDB.

Cloudant SF Drinkup

2012-01-17T00:00:00-08:00

Cloudant is happy to be sponsoring Node Summit next week in San Francisco. To keep the party going an extra day we will be hosting a CouchDB drinkup Thursday night to touch base with our west coast crew. This is instead of Benoit and Randall's meetup which they kindly merged with us - community powers unite!

Just look for the folks in good lookin' Cloudant swag.

Thursday Jan 26th
5:30 - 8:30
21st Amendment Brewery
563 2nd St, San Francisco
All Beers on Cloudant's Tab

The Future of Apache CouchDB

2012-01-05T00:00:00-08:00

The future of CouchDB is Apache CouchDB.

Cloudant over the past 3 years has built and refined BigCouch, a fault-tolerant, horizontally scalable clustering framework purpose-built for CouchDB. Today we announce our intent to contribute this work back to the community. Working with the ASF and the CouchDB community, we hope to integrate the core capabilities of BigCouch into Apache CouchDB. Hopefully this will put to rest the tired (and false) “CouchDB doesn’t scale” meme. BigCouch forms the bedrock of a globally distributed, sophisticated technology stack that we’ve had in production operation for over two years at scale.

Those of you building applications with CouchDB know that its performance has been improving by leaps and bounds. We’ve achieved this through low-level implementations of performance-critical routines and smart refactoring of the Erlang codebase, but without sacrificing the rock-solid durability and stable REST API people have come to expect from CouchDB. The growing community of active committers includes passionate, stellar developers. More and more people are using CouchDB for projects, big and small, every day.

We, along with a host of other companies, strongly support the open source community in building CouchDB and we do not plan on stopping. We have been fortunate in our ability to attract outstanding engineers, investors, and customers. We intend to continue devoting resources to Apache CouchDB and offer our help in any way the community desires. The future of CouchDB is CouchDB.

Damien Katz has been an excellent founder and steward of CouchDB. It’s been an honor to work with him, and we wish him well in his new location and mission.

A Big Finish for 2011

2011-12-15T00:00:00-08:00

2011 has been quite an eventful year for Cloudant. In the spring we outgrew our tiny basement offices in Davis Square and moved Cloudant HQ into Boston proper. We added a number of fantastic engineers to the team and we said goodbye to some of our earliest hires.

Going into the new year, we’ve expanded our platform to 5 geographic zones in the US and 3 more around the globe. Most importantly we more than doubled the number of developers, applications, small businesses, and enterprises using our hosted service – and found quite a few independent BigCouch users out there along the way. We continue to be surprised and delighted by the passion our users show for our products. It is that passion that drives us to continuously improve BigCouch and our hosted service.

With all that has been going on here, the founders decided it was time to expand the team. So today we are happy to announce that Derek Schoettle has joined Cloudant as our new CEO. Yes, Derek has an MBA – don’t hold that against him. His passion is working within early-stage companies bringing great technology to market, most recently at Vertica, and numerous startups before that. Derek has been an advisor to Cloudant for most of 2011 and was a natural choice when the time came. We are very excited to have him on board helping Cloudant grow.

With Derek on board, Adam, Mike and I will have more time to focus on the things we are passionate about . For me, that means working closely with our customers and users, focusing on building a great product that solves their data problems. Mike will continue to focus on customer success and outreach, and will hopefully have more time ‘on the road,’ evangelizing BigCouch to a wider audience. Adam will continue to work deep in the guts of Apache CouchDB, BigCouch, and the Cloudant platform, making all of our crazy thought experiments into real features.

2011 was a big year for Cloudant, but we have even bigger goals for 2012 (assuming the world doesn’t end.) Stay tuned to this channel as we announce some of the big features and plans we have.

Alan (for Adam and Mike)

Co-Founder, Cloudant

Slides and Presentation from 2600hz

2011-11-16T00:00:00-08:00

On December 14th we hosted a webinar with our friends at 2600hz, the open-source VOIP platform. Darren Schreiber, CEO of 2600hz, gave a presentation on how his company uses the BigCouch data platform to better serve its customers. Darren gave some great examples of how moving to a scalable document store has helped his company move faster, save money, and avoid operational headaches.

In case you missed it, we recorded the webinar and you can view it below. If you just want the slides, follow the link below the video.

Download the slides here

Monsanto Chooses Cloudant

2011-10-12T00:00:00-07:00

Big day for Cloudant — and big news in the near future for all BigCouch users. We are happy to announce that biotech heavyweight and perennial Fortune 500 company Monsanto has chosen Cloudant to be the core of their new genome analysis platform.

We’ve been working with them for a few weeks now and we couldn’t be more thrilled with the partnership. We are no strangers to the analysis of the big bytes generated by the smallest of things. Our founders first devised our open-source platform BigCouch while at MIT, working on Large Hadron Collider experiments generating millions of data points per second from the collision of atomic particles. Turns out, Biotechnology & genome analysis is no different in this regard. As GigaOM summarized it: “innovation is rampant, but data growth is outpacing the ability to analyze it, making faster, cheaper and more scalable data systems integral to leveling the playing field”.

This is new ground for us — and for NOSQL as a whole. This Monsanto platform is not destined to toil away in one of their (many) R&D labs, only used by scientists with lab coats & beakers. We’re not powering the CMS of some administrative division or a minor part of their website. Cloudant’s BigCouch will be the core, for both storage and analysis of a new, company-wide platform powering a fundamental aspect of a Fortune 500 business: the analysis & identification of new traits & genomic combinations in agricultural crops. The data & reporting interfaces will be used across Monsanto and should be instrumental in the making of key business decisions.

Now, this also means big news for our users. Monsanto has been very eager to let us open-source many of the core enhancements we make to BigCouch for them. So while this won’t quite make it into the impending BigCouch 0.4 release (which is days away now), or on our cloudant.com clusters for another few months, you can definitely expect to see many Monsanto-driven improvements folded back into our offerings, such as support for more languages on view servers, new tools to model data workflows, as well as a flurry of improvements to our storage & retrieval algorithms.

At Cloudant, we’re all excited to see NOSQL platforms, including BigCouch, being used to power the next generation of business technology. So when announcements like this also translate to a more stable, performant & productive experience, for everybody else that uses a Cloudant product, we find it to be a big day indeed.

You can always contact us if you have any questions. There is also a press release available.

Optimizing your CouchDB Calls by 99%

2011-09-27T00:00:00-07:00

Earlier this month, I gave a presentation at GoGaRuCo about using CouchDB, and how many people do it in inappropriate or wasteful ways.

The centerpiece of my talk was a step-by-step example of best-practices for CouchDB interaction, leading to an improvement of 99% in the median time it takes to do a simple insert to CouchDB (or Cloudant). Admittedly, I start from a worst-case scenario, but the scary part is that it is not a totally unrealistic one; in fact, I’ve seen it replicated many times over.

After factoring my advice in, we have a performance improvement of 96% (if you stick to conversing with CouchDB through JSON, which is recommended for now), or 99% with my experimental patches to support MessagePack in CouchDB & CouchRest.

What’s more, none of the advice I give leads to a relaxation in consistency or durability. It’s plain optimization, no strings attached, just there for the taking. Many of the points I touch on are also valid no matter what language or library you are using (or are easily translatable to other ecosystems), so I would strongly encourage anybody using CouchDB and interested in performance optimization to look at the video.

Head on over over to Confreaks to see the video (the meat of the talk starts at the 5:10 mark).

Using Cloudant from Scala on CloudBees

2011-07-15T00:00:00-07:00

This blogpost originally appeared as “NoSQL: CouchDB with CloudBees and Cloudant” on the CloudBees Blog.

In this tutorial I will build a CouchDB based url shortener service (http://cloudbe.es) using CloudBees and the services/add-on platform with Cloudant. CouchDB is a popular NoSQL database - specifically you could called it a “document database” - i.e. it is all about storing documents which consist of sets of fields in a JSON document against a key. There is no schema - a document can consist of any JSON structure - there is no requirement that the same fields be present in each document unlike rows in a RDBMS. Of course CouchDB has a lot of the other benefits of NoSQL databases - high scalability and ease of use. Cloudant provides an excellent hosted CouchDB service. This takes care of all the management of CouchDB you would normally need to do - compaction, backups, and more.

Building a URL shortener

A url shortener is just that - takes a long messy URL and generates a token/shorter URL you can pass around.

Step 1: Get a cloudbees.com account, signup to a free RUN@cloud service

Step 2: Go to the subscription page — and choose Cloudant

Step 3: Create a new web app skeleton

For this I will use the Play! Framework - a simple framework for building RESTful web apps. I can use the command

play new urlshort --with scala

This will create an empty project using scala. To deploy to CloudBees- a handy way is to add cloudbees 0.2.1 to the dependencies file (don’t worry - source repo will be available to look at).

Step 4: Create an empty database

From grandcentral.cloudbees.com/services you can navigate to your Cloudant/CouchDB console. Any time you need to get back to the CouchDB console you can navigate from grandcentral.

This shows the “Futon” web interface to browse your data, quite a handy tool. Here you can create a new database - I called mine “urly”. Once you have a database, at the top left corner you can see an icon which will take you to the Cloudant management screen - click through to the newly created database, and you will be able to set permissions, and generate an api key:

The important bit is to note the api key details (a user name and password that you will use in your app). The user just generated should also get read/write access to your database (set it with the checkboxes).

Step 5: Set up credentials in your application config

In this case, as I am using the Play! framework - the configuration is in application.conf. The keys I set up were:

    #CloudBees details
    bees.api.key= key for deploying
    bees.api.secret= secret for deploying
    bees.api.domain=michaelvideo
    bees.api.name=urly

    #Cloudant details
    couch.user=api key user
    couch.password=api key password

I put the CloudBees details in there for easy deployment but technically you don’t have to (you can pass them to command line if you like).

Step 6: Talking to CouchDB

One of the really nice things about CouchDB is that you don’t need any drivers or dependencies to talk to it — https is a “native” api for it. So all that is needed is a simple http/REST/WS client.

Urls are how you address content - you can store a document in a nominated key for example by PUT’ing a JSON body to https://michaelvideo.cloudbees.cloudant.com/urly/ (urly is the name of the database). Updates to data require you to specify the current version tag of the data (field in the JSON body) as a form of optimistic locking.

Getting the data back is a simple GET ! (obviously there is a lot more you can read about if you need). In this case the keys are the randomly generated “slugs” that form the short URL.

Step 7: Wire up the application

Descending from the mountain top, a simple codebase which generates random url “slugs” and then does appropriate http redirects was carved in stone (also available on github):

Authentication with CouchDB is using http basic over SSL (using the api keys mentioned earlier).

Step 8: Deploy

(This will use the Play! CloudBees plugin - thanks Ivan !)

I then set up a domain and a record pointing to the host for the app, running at http://cloudbe.es/

Note that there was no need to migrate any data, or setup a schema (just create the database by name - which I could have also done programmatically).

The data is safely stored in CouchDB (below is browsing it with Futon):

MapReduce and Views

A further enhancement is to create views. CouchDB uses a Map Reduce model to provide views and aggregate calculations.

A view is similar to what relational databases offer and only requires a map function to be provided. Reducing is a more complicated topic and I won’t show it, but it could be used, for example, to calculate aggregates (sum or average over some range of documents). The advantage of a Map Reduce model is that the calculation can be distributed and run in parallel with the data - good for very large sets of data.

The Map function above simply takes a document and emits another document which is made up of the slug and the agent string (each browser provides an agent string). The key in this case is the url we stored - this means we can (if we want) lookup slugs based on URL (the reverse of what we were doing before). Note that as the fields in each document are optional - even if there is no agent field the above view will still work (just will have no agent field in the view output).

The functions are written in javascript - and are themselves stored as documents (with a special naming convention). The results of these views are accessed the same as the normal documents you write data to.

Hopefully from this you get a taste of the convenience and power of CouchDB and Cloudant.

You can sign up for both CloudBees (if you don’t have an account) and the Cloudant service on the CloudBees website.

A Technical Look at Cloudant Search

2011-07-14T00:00:00-07:00

As we just announced, Cloudant has added fault-tolerant, federated full-text search to our document database offering.

Cloudant Search builds upon our existing database to offer full-text indexing and search. It makes use of the Apache Lucene query syntax and large portions of the Lucene framework, but uses a new data storage model entirely within the CouchDB database framework. The inverted index (the map of terms to documents in which they appear) is calculated using our incremental map-reduce framework. This means the database is always available for document insertion and reading even if the indexing task backs up - other search solutions only allow documents to be added as quickly as they can be indexed. Newly indexed documents are immediately searchable – there is no latency while the inverted indices are rebuilt.

Documents within Cloudant are distributed to multiple nodes using consistent hashing in a ring like structure. The inverted index data associated with each document is saved on the same node as the document itself. Searches are done at the shard level, and results are combined in a federation layer. If a particular node in the cluster is offline, other copies of the shard are used. Cloudant utilizes the same underlying distribution and federation technology from the open-source Bigcouch project.

Important features of Cloudant Search include:

Text analysis can utilize existing Java based Lucene analyzer. Java code can be loaded into the database and stored along with the search indices.
Support for indexing in many languages (Java, Javascript, python, erlang, ruby, etc.) User indexing code is also stored in database.
Ability to run ad-hoc queries against multiple independent map-reduce views of data.
Full support for all JSON types during indexing (Strings, Numbers, JSONObjects). Pure Lucene search solutions only support String tokens.
Support for following query types: Boolean, Range (text, numerical, date, and compound key ranges), Phrase, Sorted, Prefix, Dates
Ability to query a “stale” snapshot of the database in case where indexing has not caught up to document insertion
Horizontal scalability through addition of database nodes
HTTP based search API consistent with CouchDB view API.

We see several distinct use cases where Cloudant Search will be superior to existing products. Since Cloudant runs as both a low cost hosted service and as a dedicated deployment for the largest customers, developers can build products around Cloudant Search and know that they will scale to terabytes of indexed data and high IOPS rates. An ideal use case would be mobile applications that require real-time geographical bounding box search coupled with text queries. Cloudant Search’s native support for numerical index types allows this. Cloudant Search’s reliance on CouchDB views also opens up unique possibilities. Multiple algorithms can be used simultaneously to search, score and sort documents.

To get started with Cloudant search, you first need to specify how your database is indexed by uploading a new CouchDB design document (indexing instructions). Then you can use the search API. There is also a full description of the query syntax.

Announcing Cloudant Search

2011-07-14T00:00:00-07:00

I’ve always strongly felt that using NOSQL wasn’t so much a choice as a necessity. That most successful NOSQL deployments start with the intimate knowledge that your set of requirements — from speed & availability to operational considerations and budget — cannot be met with a relational database, coupled with a deep understanding of the tradeoffs you are making. Among those, perhaps no tradeoff has been felt more deeply by NOSQL users worldwide, than the eponymous loss of a natural, instantaneous way of accessing your data through a structured query language. We all came up with our own remedies; more often than not, that substitute was based on MapReduce: Google’s novel, elegant way of explicitly parallelizing computation over distributed, unstructured data. But as the joke goes, it’s always been a non-starter for the more novice users out there, and where suits & ties are involved.

CouchDB Views (as our brand of MapReduce is called) come with additional concerns, as they are pre-computed and written to disk. While this is fine — and actually, extremely useful — for the use-cases and small scales a lot of Apache CouchDB deployments reside at (single instances working off a limited dataset), this behavior is somewhere North of nagging and South of suicidal for the data sizes & use-cases most Cloudant customers have to deal with. Part of the promise of our industry is — or should be, anyway — to make your life & business easier, no matter how much data you have. And so, while CouchDB Views have been, and will undoubtedly remain, an essential tool to index, filter & transform your data, once you know what to do with it; and while its various weaknesses (explicitly parallelized syntax, lengthy computation, heavy disk usage) are also the source of its most meaningful strengths (distributed processing, high performance on repeated queries, persistent transformations), we at Cloudant saw a clear opportunity to offer a novel, complementary way to interact with your data.

A way that would allow you to interact with your data instantaneously; wouldn’t force you to mess around with MapReduce jobs or complex languages; a way that would not require you to set up a third-party, financially or operationally expensive solution.

We call this way Cloudant Search. And today, we’re proud to announce its immediate availability, as a public beta.

Want to easily find all the documents that contain the word “bieber”? This is the Cloudant Search query you have to write:

bieber

Want to find all the records that have “my world” in their title? Just write:

title:"my world"

How about only finding all the artists who were born in Canada in February & March 1994?

type:artist country:canada dob<date>:[1994-02-01 TO 1994-03-31]

How about people who are fan of either Justin Bieber or Justin Timberlake, that live in JBiebz’ hometown in Ontario — wait, what’s it called, Stratwood? Stratburg? Strat- something. Oh, I’ll just search it:

type:person fan-of:(justin (bieber OR timberlake)) city:strat* state:ontario

And that’s just scratching the surface and you can see more query examples in our knowledge base. Of course, all those queries are also served instantly, and new records you put in your database will be indexed in real time.

But Cloudant Search goes beyond ease of access for developers, analysts & end-users.

For the IT & DevOps teams out there, and for our own sake (since, as maintainers of hosting service cloudant.com, we have the distinction of being the only NOSQL company tasked with operating its own product around the clock), we worked tirelessly to make it capable of tackling terabytes and easy to maintain under load. It is baked in directly into the Cloudant codebase and leverages existing, proven technology, such as the Lucene indexers, CouchDB’s view indexing logic and of course, Cloudant’s own Dynamo-inspired distribution algorithms. Compared to external search & analytics solutions such as Hadoop, SAS, Solr, Sphinx or ElasticSearch, this integration & simplicity effectively translates to more flexibility to address ever-changing use-cases, a drastically reduced operational complexity and, at the end of the day, a much lower total cost of ownership for start-ups and established businesses alike.

Finally, for the power-users out there, we’ve made Cloudant Search extremely extensible. As our documentation details, you can write your own indexing algorithms, leverage Lucene tokenizers or even, customize the Cloudant Search indexer itself, with a bit of Java code.

Now, I’d be engaging in doublespeak if I didn’t make it clear that Search is still bound by some of the tradeoffs that epitomize our industry. Like some of our esteemed colleagues, we strive to make our users & customers aware of the adjustments that, yes, define our weaknesses but more importantly, are also the source of our strengths. Among those, I should mention that Cloudant Search will require an initial, one-time indexing and this single index itself will take up disk space. To limit resource consumption and self-induced podiatric harm, wildcard (“som*thing”) or suffix (“*thing”) searches are not allowed (only prefix searches like “some*”). And yes, some of its more advanced features like the dynamic sorting of results, are still memory-bound and as such, should be used carefully.

There are no silver bullets in this industry. That is something that the Cloudant team has kept in mind through the months we spent developing Cloudant Search. But more than an admission of eventual defeat, we’ve always taken it as a challenge to innovate at the pain-points. To explore the areas beyond the boundaries of our existing trade-offs. To find tasteful solutions to gritty problems. Today, we’re happy to bring you the fruits of the labor, share our response to this problem, and in a way, pass the challenge on to you: we can’t wait to see what you will do with it. We’re confident that Cloudant Search will open new doors, from user-facing analytics dashboards, to efficient Object-Document Mappers, to killer apps we haven’t even dared to think of.

So go ahead and read more about the technology, peruse the docs, activate Search on a database through your cloudant.com dashboard or contact us to see how this technology can be deployed in your datacenter.

.NET + CouchDB = ♥: Announcing the Cloudant Add-On for AppHarbor

2011-06-28T00:00:00-07:00

The NOSQL movement originally found its community roots in some the “edgier” platform & languages of this world; CouchDB in particular, has always felt strongly backed by a certain kinship with the Ruby and Rails communities, amongst others. But as one of the most mature NOSQL projects out there, CouchDB now also enjoys energetic support in the more established communities such as PHP, Java or even .NET.

As it turns out NOSQL isn’t the only movement kindling a relationship with the .NET framework. The “Cloud” has also taken root in that community, and a service called AppHarbor has arisen to meet the demand for reliable, elastic & pay-as-you-grow .NET hosting.

I’d be beating around the bush if I introduced AppHarbor as anything else but “Heroku for .NET”. That description may sound catchy but in and of itself, does not a great service make. No, what really caught our eye about our fellow ex-YCombinator brethren Michael, Rune & Troels, is their flawless execution of the concept and their tactful tweaks to the Platform-as-a-Service model. By giving you access to the Common Language Runtime and the .NET framework, AppHarbor opens more doors for developers than just single-language execution. You instantly gain access to the years of loving care, performance tuning and functional extensions that Microsoft has poured into its platform, as well as support for a flurry of languages.

So today, we’re happy to announce the Cloudant add-on for AppHarbor. It will give you instant provisioning of our hosted CouchDB solutions, for all your hosted .NET apps. Your Cloudant databases will be co-located in the same datacenter as AppHarbor, so you can enjoy the same reliability & performance you’ve come to expect from Cloudant, with the absolute minimal latency possible. And on the business side, you’ll also get the accounting convenience of a single monthly bill for both your app & data hosting expenses. (Don’t forget to put a coversheet on that TPS report before running it by the boss, though!)

So go on ahead and get started with one of the many CouchDB libraries available for the .NET framework, such as Divan, Relax, Hammock or EasyCouchDB. Or peruse this recent introduction to CouchDB for .NET developers. (And please send us an email to show us what you’ve built with AppHarbor + Cloudant!)

What are you still doing here? Go sign up for AppHarbor… You’re in good hands with Michael, Rune & Troels.

Paul Davis is Joining Cloudant!

2011-06-27T00:00:00-07:00

We are thrilled to announce that noted Apache CouchDB community member & committer Paul Davis will be joining Cloudant in July.

Paul had been on our “must-hire” list for a while now, and he will be joining a technical team that now includes a third of the currently active committers to Apache CouchDB! We’ve always tried to be faithful stewards and dedicated supporters of the Open Source community behind CouchDB, and this commitment will only grow stronger with Paul’s inclusion in Team Cloudant.

Paul’s background in large distributed systems & data analytics, combined with his expertise in Erlang and hardware optimization will undoubtedly come in handy as he helps us solidify our quality of service, add features to our platform and get BigCouch ever closer to 1.0.

You can learn more about Paul through his github, twitter or website.

Bottoms up, Paul. We couldn’t be happier to have you working with us.

Mike Miller Talks Cloud Databases at GigaOM Structure

2011-06-24T00:00:00-07:00

Yesterday, Cloudant Chief Scientist (and VP of powder-blue shirts) sat on a panel at GigaOm’s Structure Conference. Along with representatives from Xeround, ParAccel, and NimbusDB, Mike discussed the ins and outs of running databases in a Cloud environment. As the other three panelists represented more traditional relational and data-warehousing tchnologies, they turned to Mike to represent NOSQL databases as uniquely suited for high performance, elasticity, and operational stability.

We at Cloudant are delighted to have been chosen for this panel. This is the third year in a row we have sent delegates to Structure and we have always found the speakers, panels, and hallway conversations to be very enlightening.

You can watch a video of the panel, or read PCWorld’s writeup.

CouchDB & Java in the Cloud, Now Easier than Ever with CloudBees

2011-06-21T00:00:00-07:00

Today, Cloudant is proud to be a launch partner of the CloudBees Ecosystem. Effective immediately, our line of hosted CouchDB cloud solutions is available directly through CloudBees’ Java platform, with a simple API, unified billing and integrated account management.

CloudBees started out in 2010, with the goal of giving Java developers & organizations a platform, that not only lets them run their applications online easily, but also gives them specific tools to develop, test & integrate them. This twist in the usual PaaS model means they’re the only platform out there that aims to cover the entire lifecycle of traditional Java application development. So today, we’re thrilled to be part of this vision, by helping them launch an add-on program that also features New Relic, SauceLabs, Sonar & JFrog.

Considering the recent increase in CouchDB adoption in the Java community (bolstered by the existence of several drivers & persistence layers, such as our own fork of couchdb4j, ektorp, jcouchdb and jrelax), we’re more than happy we answered CloudBees call, integrated deeply into their platform, and we can’t wait to see what you will build with us!

So go on ahead and check out the details of their announcement or sign up for the CloudBees platform right away!

A Closer Cloudant? Now Accepting Reservations for a European Cluster!

2011-05-24T00:00:00-07:00

One of the hardest parts of running a global business like Cloudant isn’t necessarily on the technical side of things. While we certainly got better at developing & operating a fleet of clusters that now spans both US coasts & Asia, we’ll admit to still being somewhat under-skilled at tackling some of our marketing concerns, such as estimating latent demand in foreign markets.

In recent months, our support team has been receiving more & more inquiries about provisioning Cloudant accounts in Europe. But instead of trying to guess who else is interested, run complex spreadsheet models or hire a team of consultants, we figured we’d just ask you. (Yes, you: Time magazine’s 2006 Person of the Year.) To limit the noise in this poll, we’ll ask you to put your money where your mouth is: we’ll only accept reservations from paying customers (i.e., users on an “Argon” plan or above).

Having your data located a cluster close to your application layer (or other consumers of your CouchDB docs) has a big primary benefit: latency. With a cluster in Europe, your median read & write latencies to Europe should go down by approximately 200ms (or by ~500ms for SSL access), when compared to our existing East Coast cluster .

So go on ahead to our reservation page. You’ll be asked to sign up for Cloudant if you don’t already have an account; or upgrade to a paid account if you’re currently on our “Oxygen” plan.

We’ll let you know how the process is going with a handy reservation progress bar. We’ll set up our European cluster and load it with the accounts that reserved a spot, as soon as that bar hits 100%!

Go ahead and tell us you’re interested!

Search Indexing in Javascript

2011-05-19T00:00:00-07:00

In order to leverage existing Lucene analyzers, our Search feature (currently in beta) makes use of Java views, rather than the traditional Javascript views you’re used to in CouchDB. Recently, we wanted to do some performance comparisons between the two; would the Java view server be significantly faster (or slower) than the Javascript view server on typical indexing tasks? To test this, we wrote a simple Javascript view (in the form of a CouchApp) that creates a Cloudant Search-compatible inverted index. The code is available on github here.

To use it, clone the git repository and use the couchapp tool to push it to your database:

$ git clone git@github.com:cloudant/jsindexer.git
$ cd jsindexer
$ couchapp push http://<user>.cloudant.com:5984/<db_or_couchapp_you_want_to_search>

With this simple push, you gain the ability to do a search against this inverted index. For example, if you want to search all the docs in that DB whose “comments” field contains “cloudant is awesome” and whose “rating” value is between 80 and 100:

$ curl 'http://<user>.cloudant.com:5984/<db_or_couchapp_you_want_to_search>/_search?q=comments:"cloudant is awesome" AND rating:[80 TO 100]&index=_design/jsindexer/_view/whitespace'

Obviously, that’s just scratching the surface. Our knowledge base has several articles about Search. You can go there to read our introduction to Search, learn more about the Search API or how to write your own Indexer.

As for the results of the performance comparison, our full text indexing speed is limited by the writing of the data to the CouchDB B-Tree, so Javascript and Java perform the same. So there you have it: full-text search from Javascript, a guilt-free geeky pleasure. Go ahead and have at it.

Fukushima from Afar

2011-04-02T00:00:00-07:00

In the past month I had the all too rare opportunity to do something that was both terrifically exciting and, I believe, deeply meaningful. On Friday, March 11, I awoke in Seattle to the news of the devastating earthquake and tsunami that struck Japan. As the day developed the sheer magnitude of the true devastation emerged and it was clear that sobering stories would continue to unfold for quite awhile.

I was doubly surprised by the emerging media coverage of the Fukushima reactor complex. The Japanese are absolutely top-tier in both their science and engineering and the very fact that Tokyo had so little damage from a 9.0 earthquake (imagine if that happened here!) suggested that they were equally prepared for the aftermath. To a large extent, they were — just not for a series of events that were so powerful that within days it was clear that the Fukushima situation had changed dramatically.

In this era of real-time data, it occurred to me just how hard it was for the educated public to gain access to the right data. What was the level of radiation being released by Fukushima? Was it enough to be dangerous to the people in Japan? Was it enough to impact our sensitive (and expensive!) physics experiments here in Seattle? Was it enough to be a public health concern in the US?

These questions, plus a little prodding from senior faculty at UW, put the wheels in motion for me. If information was going to continue to be so hard to find, then maybe we could do something about it. We are physicists, after all. Emails were sent, grad students were diverted from thesis writing and qualifying exam preparation, and by Monday we had worked to convert my lab at UW from Dark Matter detector R&D to an air radioactivity monitoring lab, containing some very expensive, one-of-a-kind equipment, along with the requisite quantities of duct tape and zip-ties holding everything together. This was no small feat, and it never would’ve happened without the work of a superb team and the right tools.

We knew that any radiation that was released would be heavily diluted in transit to the west coast, so we had to (i) find a way to sample enormous quantities of air (nearly 150,000 cubic meters per day) and (ii) protect our radiation counter and air samples from the copious amounts of normal background activity. A little known (or appreciated) fact is that natural radiation is part of your daily life, from radon decay products to bananas, and natural radiation would likely drown out any faint signal from Fukushima. By Wednesday we started recording data, and then it suddenly hit me — we were going to be producing a lot of data, and it was already coming in. The data had to be catalogued, processed, condensed, distributed, data models modified, reprocessed, redistributed, and we were learning as we went along. Not only was the data was important to us and our experiments, but the public was hungry for realtime results (that had to be correct!), so we didn’t have the luxury of writing a spec, reviewing it, building it, prototyping, or any of the typical processes. We already were running TV crews through the lab daily, time was short, and so I turned to the tool that I happened to know best: open source BigCouch, and specifically the free hosted version on Cloudant.

We began putting together a real-time BigCouch workflow that ultimately enabled us to complete complex processing (data transformation and aggregation, signal processing, statistical analyses and the like), real-time data monitoring and visualization, collaborative analyses, and ultimately to serve the results directly to the public from the database. At the end of this post I will briefly review my BigCouch experiences, both good and bad.

On Friday, March 18, I had just placed the overnight air filter sample into our monitoring station and was adding some additional signal processing to our workflow. After 20 minutes of counting I happened to look at the computer screen to check the realtime data acquisition histogram, and there it was, clear as day, the first arrival of 131-I in the Seattle air. I spent about 10 minutes double checking, looking for other characteristic fission products in the spectra, and then realized we had something very unique.

Racing down the hall, I gathered a team and the next week is an absolute blur. That Friday night, around 10, we knew that we had unambiguously detected fission fragments from Fukushima, and we had a good handle on the total level of the radioactivity levels. While results were far below any local health concerns (roughly 1000 times below the EPA limits), it was higher than I had personally expected, and we suddenly knew that the situation in Japan was indeed serious. Our thoughts go out to those that live near Fukushima, as well as the heroic workers at the plant as they fight to contain the incident.

As the week progressed, we saw the radiation levels peak, slowly decline, and peak again as we detected events that occurred roughly 7 days after they happened in Japan. Within a day of our initial sighting we had posted the numbers and plots online, and they continued to update in real-time as the situation unfolded. By the middle of the week we posted the first publication draft online, and are now witnessing the radiation levels drop below the limit of detectability in our experiment. If you are interested in the details, the pieces in the Washington Post, Nature News, and NY Times were particularly well written, in my opinion.

Before closing, I want to briefly review the pros and cons of using BigCouch/Cloudant for this critical application.

Pros

Flexibility. The ability to smoothly handle both structured and unstructured data was critical. The document philosophy allowed us to be agile with data types (metadata + binary data side-by-side) and date models (sometimes changing by 20x per day).
Cloud. Not having to install and maintain my own database server for this purpose was a huge win.
Scalability. This application isn’t the world’s biggest data producer, but it’s no slouch either. We were able to easily store and process Terabytes of data without breaking a sweat, and it was nice to know that we wouldn’t be limited by scaling concerns in the future.
Real-time Analytics. We were able to easily insert our domain specific libraries/application into an incremental MapReduce workflow to provide up-to-the-minute results.
Collaborative. The database is a web server, so we were immediately able to collaborate as a team. The fact that the db is also a web server that speaks JSON over HTTP enables simple integration with other standard tools for client-side visualization, etc.
Thin client applications. I was able to write ultra-lightweight client-side javascript applications for data monitoring and presentation. These applications were typically around 100 lines of code and served directly from the cloud to the browser, which means I didn’t have to set up my own web-tier stack, nor learn Rails, Django, etc. This has been on my list of “things to learn” for a year, and this pushed me to finally do it.

Now the things that can definitely be improved

Analytics integration. Like Hadoop, building a BigCouch analytics workflow is a bit raw and has a substantial learning curve. This experience makes me realize that there is tremendous low-hanging fruit for tools that bridge that gap for the average developer and/or analyst.
CouchApp. The client-side JS model is a huge win here, and the couchapp tool was a big help in turning my JS and HTML files immediately into an application in the cloud. However, I had to strip out a significant amount of unused baggage to make sense of it all. Further, there is still some work to be done to allow for rich applications that securely allow not just data reads from the client(which currently works well), but also writes.

In summary, BigCouch was a key enabling technology, and I will definitely use it for future large scale experiments. In this case, it helped maximize the value in a very time-sensitive big data problem.

Our thoughts go to the people nearest Fukushima. We are fortunate that the radioactivity in the US is so low. I feel both proud and humbled to have been able to contribute in some way, by bringing scientific information to the public in a time of great uncertainty.

Rise & Shine

2011-03-20T00:00:00-07:00

You wouldn’t know it from looking out your window here in Boston but spring is supposed to start tomorrow. And so the Cloudant team once again spends a lazy March Sunday, shut indoors, hoping for milder spells.

But it’s already Monday in Japan and it seems like the tech community there has had a busy weekend.

In an effort spearheaded by the local branches of Google, Amazon, Yahoo!, Microsoft and many others, more than 200 developers have been sharing ideas, teaming up and designing apps that will help speed up Japan’s recovery after the disastrous earthquake, tsunami, nuclear scare and 632+ aftershocks they have been experiencing since Friday, March 11.

The endeavor is dubbed Hack For Japan and while the English language details are still sparse (see the idea poll or the ongoing Twitter stream), we immediately knew we wanted to help out in whatever way we could.

The Japanese CouchDB community is 150+ strong, has organized several “relaxons”, is writing an impressive community translation of “CouchDB: the Definitive Guide” and has a very active leader in Yohei Sasaki. Cloudant in particular, has enjoyed some traction in Japan for a while and we set up a cluster dedicated to serving that community last year.

Today, we’re moved by the resilience, kindness and positive outlook of the Japanese tech community. So we’re announcing we’re donating our spare Asia-Pacific cluster capacity to any Hack For Japan project that may need it. It should be enough for several hundred Argon or a few Krypton accounts. The details are here.

That’s just our way of saying thanks to the Japanese community, for always having our back, for keeping the lost, the injured & the stranded in their hearts, and for showing that you can always count on your fellow geeks to unite, build & help rebuild.

After all, it’s already the season of love, life & new hopes, over there.

“Using Incentive Prizes to Crowdsource Brilliance”

2011-03-09T00:00:00-08:00

I can’t take credit for the title, it’s a direct quote from a fantastic talk that I just saw by Peter Diamandis, the force behind the X Prize Foundation. Peter is a superb speaker. The room was packed and he didn’t disappoint. He is passionate, eloquent and flat-out inspiring. Now, I’ve been incredibly lucky to be surrounded by inspiration at MIT, Cloudant and Y-Combinator, but… Something about this talk really struck home.

From our time in YC and beyond, Paul Graham has always been pounding home a single message: make something people want. By now I’m sure everyone has seen the t-shirts and knows the slogan. Something about this motto has always struck me as just a bit, well, incomplete. Yes, I want to make something people want. Yes, that’s as good of a guiding principle as any on the path to success. However, following that mantra to its logical end can actually lead to sub-optimal outcomes on a whole. I’d bet a decent chunk of entrepreneurs (PG clearly included) actually want to make the world a better place while following their passions. Most of us at Cloudant spent the better parts of our young careers trying (sometimes in vain) to understand what the universe is made of and how it behaves! And somehow I’ve always felt this tension between those two poles: what the market wants and what actually makes the world a better place. And that’s when Peter’s message finally hit home.

Peter recognizes that sometimes free market forces alone don’t converge on the global minimum. Sometimes they need a nudge to escape a local minimum, and Peter has made it his life mission to use incentive prizes to nudge the market as needed.

The X Prize Foundation is awesome. They identify hard problems with high stakes impact on society as a whole (water, medicine, energy, enviornment) and create an innovative/entrepreneurial environment to solve those problems. For profit. Yes the X Prize Foundation is strictly philanthropic, but the contestants are anything but. The foundation only gives prize money to the winning team, but that doesn’t matter – they’ve just created a new market from scratch, and a big one to boot! In the case of the first X Prize, for space flight (actually 100 km elevation, to be precise), they’ve created a billion dollar industry, nearly overnight¹. To an entrepreneur that is simply amazing: the prize and the ecosystem it spawned ameliorate the risk. You don’t have to win the prize to be successful. Great team, great gumption, and clarity of vision: you’ve got a fighting chance to do anything. Great team, great gumption, clarity, revolutionary space, and guaranteed market‽ Now you’re really talking!

Finally, Peter opened and closed with a question: “What are you passionate about?” This is a big one. At Cloudant, we are passionate about innovation and it’s heartening to see the explosion of not just new companies and technology, but also forces such as YCombinator and the X Prize Foundation that are changing both the way innovation happens as well as what it targets. PG and Peter D. — keep it up!

Ok, we can quibble about the market’s existence prior to the 1st prize (hell, Boeing is in the news daily here in Seattle). But.. I walked away convinced that there was at least $1B of new market created, in addition to anything pre-existing.
↩

New Cloudant HQ (with pictures!)

2011-02-23T00:00:00-08:00

The number 402 is following me, you could call it my lucky number. And apparently I’m sharing the luck with my cofounders. I should probably take our investors’ money to Vegas and bet on 402. Back when I first moved to the east coast my first apartment was at 402 Broadway in Cambridge. My office at MIT, where I first met and worked with my cofounders was in Building 26, office 402. When we started Cloudant we set up shop in a small basement office at 402 Highland Ave. in Somerville. Starting to get creepy right? Well, this week we finally moved into our new office space in the south end. We had outgrown the place in Davis Sq. and as much as we are going to miss its horror-movie ambiance (no really, when we moved in that office looked like the set of “Saw”) we couldn’t be more excited about our new digs. Where are they? 580 Harrison Avenue, Suite 402. If I were a man that believed in such things, I would think the universe was trying to tell me something. My colleagues tell me that 402 is an obscure HTTP error code for “Payment Required.” Maybe the universe is calling in some IOUs, or maybe it’s just suggesting that we need to charge more.

Anyway, here are a few pictures of the team working in the new space. The new office has exposed brick wall, large windows without bars on them, and enough room for more than three desks. All of these amenities were lacking in the old office.

It’s a big upgrade from our previous place. If you have the time and the inclination, please stop by to say hi. We’re always happy to chat about CouchDB, startups, complex data problems, NOSQL, and our favorite Twilight Zone episodes. Oh yeah, and if all this sounds enticing (and how could it not?) we’re looking for a crack devops engineer to join the team. If you are serious about keeping the servers well-oiled, building distributed systems, and providing a great cloud service, apply here

BigCouch 0.3

2011-02-11T00:00:00-08:00

Time sure flies when you’re having fun. It’s been a little over three months since the first BigCouch release so we thought it might be a good time to package and release a new version. A lot of folks forked this first release and started using it, providing valuable feedback, logging issues and bug reports. We greatly appreciate the effort and hopefully have covered most of the issues. This release sports a major new feature in addition to lots of consolidation and bug fixes.

BigCouch has several simply architected layers that provide the core functionality. These are now broken out into separate sub-projects that are managed as dependencies by rebar, along with ibrowse, mochiweb and the other dependencies. This allows these projects to evolve independently and enables BigCouch developers to switch between various branches as needed. We’ve fixed several bugs and added specs and docs to fabric, which together with rexi provides our distributed CouchDB API. Rexi is actually a very nice RPC like layer that can be easily used standalone if needed.

We’ve done a lot to tighten up propagation of errors through the system, reducing the number of dead_shards errors that occur. For the most part these errors are harmless, often the result of a bad Javascript function in view code. At the suggestion of our operations team we’ve tagged each request with a unique id which is returned in the HTTP headers but is also written in the logs. This helps us track errors better as we’re able to map specific stack traces to API calls. We’ve also created an in-memory buffer in the rexi server that is accessible on the back end and maintains a configurable number of errors and stack traces.

The major feature addition in this release is shard synchronization in the mem3 layer. By proactively triggering replications between database shard copies we reduce the amount of time required to restore the proper replica level for all documents after a node outage. Towards this end we’ve implemented a new lightweight replicator customized for use with shards, and integrated it into mem3. We saw that this replicator could also be used to replicate the nodes and shard_db databases which are used to manage the cluster information, thereby allowing us to not need the heavier weight CouchDB replicator internally. It’s still new and has some kinks to work out but we wanted to get it into developers hands as soon as we could. We encourage your feedback.

Last, but surely not least, we’ve added Robert Newson to the BigCouch team. He’s hit the ground running and has already fixed some of the trickier issues with which we’ve been grappling.

These are just a few of the items covered in this release; all specific changes are enumerated in the release notes. You can always build BigCouch straight from the source, but for your convenience BigCouch 0.3 is now available via precompiled RPM and .deb packages on the downloads page at Github and as a cookbook for installation via the Opscode Chef platform.

Thanks for using BigCouch!

Some Thoughts on the Couchbase Announcement

2011-02-08T00:00:00-08:00

That will teach me to go to sleep before midnight on a weekday.

I woke up this morning to find a shakeup in the world of NOSQL. CouchOne has merged with Membase to form Couchbase, “the most comprehensive data management solution for scalable web and mobile applications.” I’ll add mine to the many voices wishing luck to the new company and the communities it supports. We’re excited to see more and more resources being poured into the NOSQL space. Congratulations to Damien, James, and everyone at both companies.

We’ve been asked if this changes anything at Cloudant. The answer is no. We’ll continue to offer Cloudant’s hosted, managed, scalable data platform. Apache CouchDB remains an important part of our technology offering; it forms the core data store around which we’ve built our wider platform. We have always been advocates, contributors, and leaders in the CouchDB community and we see no reason to change that. We’ll continue our support for both CouchDB and BigCouch. We’ll continue to provide our customers and partners with a robust database, search, and analytics platform to power their mission-critical business applications – as we’ve done for the last two years.

I do want to take issue with one thing said in the press release for Couchbase. They say,

“Couchbase becomes the only document database capable of safely storing your data
whether stored on a single server, or spread across hundreds.”

Some of our customers have billions of documents stored safely on dozens of nodes in datacenters around the world. It’s too soon to say what Couchbase will become, but if you need a safe, scalable, and easy-to-use document storage platform, our technology already provides that today: cloudant.com/solutions.

Alan Hoffman

Co-founder and CEO, Cloudant

SearchApp – Add a Searchbox to your CouchApps and CouchDB Databases in 10 Minutes

2011-02-03T00:00:00-08:00

Recently we released the developer preview of our full-text indexing and search for Cloudant’s hosted CouchDB. Here we will explain how to use a simple CouchApp to add search to your existing CouchApps and CouchDB databases. The goal is to make your data searchable in 10 minutes are less. Start your timers.

To get this running, you’ll need 4 things:

Git version control system
The couchapp command line tool is used to generate code templates in your application and to push your changes to an instance of CouchDB. This has been tested on couchapp version 0.6.1 and higher.
Free Cloudant account
An existing CouchDB database, CouchApp, or a data importer.

(Note: CouchApps are JavaScript and HTML5 applications served directly from CouchDB. couchapp is a command line tool used to create and manage CouchApps.)

If your CouchDB database is not already hosted at Cloudant, you’ll need to replicate it to Cloudant. I’m assuming your CouchDB database or CouchApp (named mydb) is on your local CouchDB instance (http://localhost:5984/mydb), and that your Cloudant username/password is <user>/<password>:

$ curl http://localhost:5984/_replicate -H 'Content-Type: application/json' -d '{ "source": "mydb", "target": "https://<user>:<pass>@<user>.cloudant.com/mydb", "create_target":true}'

Now, you’ll need to clone the SearchApp. To do this, use the git clone command:

$ git clone git@github.com:cloudant/SearchApp.git
$ cd SearchApp

For convenience, you can make your database the default target for your SearchApp. Modify the file .couchapprc as follows:

{"env":{"default":{"db":"http://<user>:<pass>@<user>.cloudant.com:5984/mydb"}}}

Now for the last step. Push your SearchApp on top of your database:

$ couchapp push

If you want to push to a different location that specified in the .couchapprc file, do:

$ couchapp push http://<user>:<pass>@<user>.cloudant.com:5984/mydb

You should see the following message:

2011-02-01 11:49:36 [INFO] Visit your CouchApp here:
http://<user>.cloudant.com:5984/mydb/_design/searchapp/index.html

Visit that page and you’ll have a working search box. Now check your watch – has it been more that 10 minutes? Your searches might come up blank at first if you have a large database. Your database needs to be indexed before it can be searched. The indexing is a distributed Map-Reduce view. You can check the status of indexing using the Cloudant dashboard at:

https://cloudant.com/#!/dashboard/<user>/mydb/stats

(be sure to insert your username in the url).

For a more details on configuring SearchApp, please look at the README file. To configure the indexing strategy, check out the search indexing page.

MapReduce from the basics to the actually useful (in under 30 minutes)

2011-01-13T00:00:00-08:00

Recently, Tim Anglade culminated his NOSQL World Tour with the release of the NOSQL Tapes, a collection of live interviews collected globally. Not only do I find that site aesthetically pleasing (courtesy of our own Steadicat), but there is a tremendous amount of solid content, from technical discussions to opinions and conjectures from many of the young leaders in the field. I was fortunate to get a bit of Tim’s time when he stopped in Seattle and we recorded a longish video on MapReduce in my lab at UW. While I love the old school feel of a chalk-talk, I must admit I’m pretty embarrassed at how rambling and fragmented my explanations are, not to mention my shoddy use of available board space. Therefore the Physics Professor in me feels obligated to try again, albeit offline this time.

Brief NOSQL Taxonomy

To set the MapReduce stage, I’m going to offer a very naive taxonomy of NoSQL data stores (apologies to the experts). I have yet to meet a database that isn’t a key/value store, so let’s focus on some higher level distinction. In particular, let’s focus on the difference between big tables (BigTable, HBase, Cassandra, etc.) and document stores (CouchDB, MongoDB). All of these solutions are finding solid acceptance in the wild, so what – if any – difference is there? My one-word answer is flexibility. Huh, isn’t all NoSQL flexible? Well… yes and no. In my eyes, BigTable and its derivatives excel at one thing in particular: structuring your data for the efficient I/O access that is necessary to get blazing read speeds for large data sets. Somewhat orthogonal, document stores, especially CouchDB, prioritize flexibility. In particular, there is no requirement to define columns, column families, or super columns up front. You simply encapsulate your data as documents and push them into the store. How then, you may ask, can you get efficient queries? MongoDB treats the problem with blazingly fast in-memory table scans and document introspection for index builds, whereas CouchDB takes a different approach with something called incremental MapReduce. I’m going to briefly explain the latter and then show you how it works in action.

MapReduce Primer

MapReduce is an old design pattern that was recently made famous by Google. There are many different implementations, and I won’t even try to address them all. It suffices to say that MapReduce is all about giving programmers an efficient way to consume data without needing to know how or where it is actually stored. Further, MapReduce excels at traversing datasets that live on more than one machine. It is simple, a bit restrictive, but extremely powerful. I’m going to show you how it works in CouchDB (well, actually in BigCouch hosted on Cloudant.com).

Broadly speaking, CouchDB (and Cloudant’s BigCouch) use MapReduce as a tool to let you introspect your data and build persistent ‘views’ (indices) for fast query responses. And, oh yeah, it does it incrementally. Let’s examine the pieces in detail:

View builds. The process of scanning over your data to create indices that make queries fast. This step also often performs data normalization.
MapReduce. The framework by which user code gets executed for index builds.
View queries. How you get the information out of your indices
Incremental. When new documents are added or existing documents are modified/removed, you don’t need to rescan your entire dataset. Instead the view engine only rescans the new/modified documents, and deletes the contributions from the deleted/modified documents. It’s a complex implementation, but one that really separates the Couch MapReduce model from, e.g., Hadoop’s or Riak’s implementations.

For full details on Cloudant’s implementation of MapReduce, take a look at Dynamo and CouchDB Clusters and Adam’s O’Reilly Webcast. So what do you actually have to do? Well, you simply need to write one or two functions. The ‘map’ function is required, and the ‘reduce’ is optional. For simple sorting problems (e.g. building a SEO reverse index) map alone suffices. For data aggregation, reduce comes into the picture. CouchDB allows you to write these functions in any language you choose, although JavaScript is the most common. Let’s put some meat on the bones with examples. Below I’m going to show you how to do three of the most common things people do with CouchDB’s MapReduce:

sort documents based on arbitrary information
calculate aggregate statistics
perform a time-series analysis

Our Example Problem

We are going to load and analyze FAA plane crash data from data.gov using Cloudant. If you want to follow along at home, you have two choices.

Use CouchDB replication to copy my db into yours via:

$ curl 'http://<usr>:<pwd>@<usr>.cloudant.com/_replicate' -Hcontent-type:application/json -d '{"source":"http://mlmiller.cloudant.com/planes","target":"http://<usr>:<pwd>@<usr>.cloudant.com/planes","create_target":true}'

Repeat from scratch by getting the code from Github.

(1) is simpler and doesn’t require any special software installed on your machine. All you need is to have a free account on Cloudant. (2) is a bit more effort but should work with any CouchDB server. If you go with (1), the best way to check the progress is to login to your account, click on your user dashboard (top right) and then click on the newly created ‘planes’ database. If you examine the Stats tab and refresh periodically, you will see the db size increasing, as well as some MapReduce ‘views’ running as the data imports:

That bar that registers 31.7% is telling you that MapReduce is being used to index your documents in real time without you having to write any code at all. The ‘2839 of 8952’ on the right represents the fraction of freshly added/updated documents that is processed in this current pass, and if you click on ‘show details’ you can see the results of the MapReduce processes on each of the cluster nodes that contain any chunk of your data. That’s just how easy it is to get your code running, in parallel, in realtime!

Data Import

I’ve chosen to import the FAA crash data using my favorite tools: Python and Benoit’s wonderful Couchdbkit library. If you use the Python Boto module for AWS, you will feel right at home. In upload.py, I use the very handy DictReader class from Python’s csv module to parse the AviationData.txt.gz. DictReader slurps a line of a CSV (or similarly structured text file) with column headings and gives you back a dictionary. It’s a one-liner to serialize that Python dictionary and push it into Cloudant. If you want to try this yourself the README file has instructions on installing the Couchdbkit module and executing the upload.py script, but it just boils down to:

python upload.py AviationData.txt.gz 'http://<username>:<password>@<username>.cloudant.com' planes

My inported data lives at http://mlmiller.cloudant.com/planes. One thing to note: you can see that I don’t take any steps to discover the type of the data on import. Instead, we just throw it all into Cloudant and we can sort it out later using MapReduce (_flexible_). Once the data is fully imported, I can login into cloudant.com and view the ‘planes’ database in the dashboard:

I’ve also gone into the Permissions tab and granted read access to the data:

Each document contains quite a bit of data, too much to fit onto a single screen shot, in fact:

But the important thing to note is that we have data for nearly 70,000 domestic accidents, spanning the range 1988–2010. If you want to look at a single document from your command line, you can via:

$ curl -X GET 'http://<usr>:<pwd>@<usr>.cloudant.com/planes/_all_docs?limit=1&include_docs=true'

and if you want it pretty printed, my favorite is:

$ curl -X GET 'http://<usr>:<pwd>@<usr>.cloudant.com/planes/_all_docs?limit=1&include_docs=true' | python -m json.tool

Since I’ve granted read permissions on mlmiller/planes, you can execute this yourself via:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_all_docs?limit=1&include_docs=true' | python -m json.tool
{
    "offset": 0,
    "rows": [
        {
            "doc": {
                "": "",
                "Accident Number": "LAX08CA100",
                "Air Carrier": "",
                "Aircraft Category": "",
                "Aircraft Damage": "Substantial",
                "Airport Code": "U42",
                "Airport Name": "Salt Lake City Muni 2",
                "Amateur Built": "Yes",
                "Broad Phase of Flight": "",
                "Country": "United States",
                "Engine Type": "Reciprocating",
                "Event Date": "04/12/2008",
                "Event Id": "20080513X00662",
                "FAR Description": "",
                "Injury Severity": "Non-Fatal",
                "Investigation Type": "Accident",
                "Latitude": "40.619445",
                "Location": "Salt Lake City, UT",
                "Longitude": "-111.992777",
                "Make": "Questair, Inc.",
                "Model": "Venture",
                "Number of Engines": "1",
                "Publication Date": "05/28/2008",
                "Purpose of Flight": "Personal",
                "Registration Number": "N36V",
                "Report Status": "Probable Cause",
                "Schedule": "",
                "Total Fatal Injuries": "",
                "Total Minor Injuries": "2",
                "Total Serious Injuries": "",
                "Total Uninjured": "",
                "Weather Condition": "VMC",
                "_id": "0fedbabcd21f15785750457f83000947",
                "_rev": "1-3e9637bb5214c82fc7b29bb4a75e765b"
            },
            "id": "0fedbabcd21f15785750457f83000947",
            "key": "0fedbabcd21f15785750457f83000947",
            "value": {
                "rev": "1-3e9637bb5214c82fc7b29bb4a75e765b"
            }
        }
    ],
    "total_rows": 68388
}

Example 1: Sort

Now that we have the data, let’s get down to business. Key/Value is great, but I want to be able get find all the documents for a given brand of plane, say Cessna. This is really an inverted index, and is in the absolute sweet spot for MapReduce. We can accomplish this with a ‘map-only’ view that looks like:

function(doc) {
    emit(doc.Make, null);
}

The MapReduce code is actually stored in the database as part of a view in a design document (see the CouchDB Book for more details). My favorite method is to use a client-side tool such as couchapp or couchdbkit that allows me to organize map.js and reduce.js files according to a filesystem hierarchy and then automatically upload them to the database as a design document. I actually snuck that two-liner into upload.py, where I have:

loader = FileSystemDocsLoader('_design/')
loader.sync(db, verbose=True)

Those two lines unpack the contents of the _design directory and build a design document. That directory looks like:

This gives me a design document with the name example; that contains two views, one called date and one called make. You can see the entire design document from the command line via:

$ curl -X GET http://mlmiller.cloudant.com/planes/_design/example

or you can just browse the ‘_design’ code in the file _design/example/views/make/map.js. The code looks like this:

function(doc) {
    emit(doc.Make, 1);
}

The emit() function generates a key/value pair that gets sorted according to CouchDB’s deterministic collation order. These key/value pairs are either stored directly in the view index, or passed through the reducer first (see below). Don’t worry about the second argument to the emit method, that’s for the the reduce phase and we don’t need that for a basic sort. The moment that we upload that design document to Cloudant, the index is kept up to date automatically. That’s quite different than the default CouchDB behavior, but it’s a nice optimization to make sure that the query response is snappy.

Query (finally!). Let’s find all of the documents (‘rows’) that have doc.Make=="Cessna". Since we emitted doc.Make in our map code, doc.Make is therefore the ‘key’ for this index, and that means we get snappy responses to queries like:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make?reduce=false&key="Cessna"&limit=10'
{"total_rows":68387,"offset":13905,"rows":[
{"id":"0fedbabcd21f15785750457f8300324c","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f83003a2f","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f830060ec","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f83006bfb","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f8300a5ae","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f8300e15d","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f830162b7","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f8301a3a4","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f8301a54e","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f83058740","key":"Cessna","value":1}
]}

The full list of query options is given here, but really there are only a handful of options I typically use. For this query, if you remove the limit argument you will get the entire list of crashes with a Cessna involved. If you want to do some fun statistics, you can try:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make?reduce=false&key="Airbus"' | wc -l 15
$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make?reduce=false&key="Boeing"' | wc -l 773

Well that’s interesting! But wait, isn’t there a more efficient way to calculate statistics than slurping the entire set of Boeing lines to the client and doing a client-side count? You bet, and that’s where reduce comes in.

Example 2: Data Aggregation

Now we want to go beyond simple sorts and aggregate to find the total number of crashes for different types of planes. Reduce is perfect for this, and that’s why we did emit(doc.Make,1) in map.js. If you look at the corresponding reduce.js file for that view you’ll see nothing more than: _sum.

What the heck is that? Well there are certain aggregation tasks that are simply so common (count, sum, average, min, max, etc.) that we have coded them up in erlang and pushed them into the database. Cloudant has an ever growing number of built-in reduces (_builtins_), and using a builtin is always faster and less error prone than writing it yourself. As an avid user, I can admit that I haven’t used a non-builtin reduce in at least 9 months. So the _sum that we use here is simply going to add up the numbers that we emitted as the 2nd argument to emit() in the map.js function. Now we can answer the Boeing vs. Airbus question without any client side work via:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make?key="Boeing"'
{"rows":[
{"key":null,"value":771}
]}
$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make?key="Airbus"'
{"rows":[
{"key":null,"value":13}
]}

Check that out! Hmm, now suppose I want to find the total number of accidents, I can then just do:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make'
{"rows":[
{"key":null,"value":68387}
]}

Very cool, and if I want to make a graph of how they plot out vs manufacture I can add the group=true option:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make?group=true'
{"rows":[
{"key":"","value":53},
{"key":"107.5 Flying Corporation","value":1},
{"key":"1200","value":1},
{"key":"1977 COLFER-CHAN","value":1},
{"key":"1ST FTR GP","value":1},
{"key":"2000 McCoy","value":1},
{"key":"2001 MCGIRL","value":1},
{"key":"2003 Nash","value":1},
{"key":"2007 Savage Air LLC","value":1},
{"key":"67 FLYING DUTCHMAN","value":1},
{"key":"85 MANISTA","value":1},
{"key":"A Pair of Jacks","value":1},
{"key":"A. H. GETTINGS","value":1},
{"key":"A. LE FRANCOIS","value":1},
…

(truncated because it’s a looong list!)

Ok, so aggregation seems pretty straightforward. Now let’s go the last step to time-series.

Example 3: Time Series Analysis

Let’s get a bit more numerical and ask, what’s the distribution of crashes and fatalities vs time. That’s what the date view is for. If you inspect any of our crash documents you will see that they have a two other important fields: Event Date and Total Fatal Injuries. Let’s create an index that allows us to answer time-based queries about both the number of accidents and the number of fatalities. While a bit morbid, it’s good to know what the odds are! This example is a bit more complex because we are going to use an array for both arguments of emit(). Take a look at the code in the map.js:

function(doc) {
    var then = new Date(Date.parse(doc['Event Date']));
    var fatalities = 0;
    if (doc['Total Fatal Injuries']!="") {
        fatalities = parseInt(doc['Total Fatal Injuries']);
    }
    emit([then.getFullYear(), then.getMonth()], [1, fatalities]);
}

This is a bit more complex because we do some processing of the data. In line #2 we unpack the Event Date variable into a JavaScript Date object, very handy. I find myself using new Date(Date.parse(some_weird_timestamp)) all the time. Line #6 simply casts our fatalities info into an integer. The magic is in emit, where we use JavaScript arrays [] for both the key and the value. Then, for reduce we use good ol’ _sum. One of Cloudant’s additional builtins is a _sum that works on arrays, very handy. Now to query this index we get rows that are sorted by year, then month. The index looks like:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/date?group_level=2' | more
{"rows":[
{"key":[1982,0],"value":[207,185]},
{"key":[1982,1],"value":[232,96]},
{"key":[1982,2],"value":[280,95]},
{"key":[1982,3],"value":[303,113]},
{"key":[1982,4],"value":[387,133]},
{"key":[1982,5],"value":[349,106]},
{"key":[1982,6],"value":[433,283]},
{"key":[1982,7],"value":[399,96]},
{"key":[1982,8],"value":[332,119]},
{"key":[1982,9],"value":[239,129]},
{"key":[1982,10],"value":[224,134]},
{"key":[1982,11],"value":[208,96]},
{"key":[1983,0],"value":[199,102]},
{"key":[1983,1],"value":[210,72]},
{"key":[1983,2],"value":[264,91]},
…

So, e.g., 1982 month 0 (January) had 207 crashes and 185 fatalities. What was that group_level command? Let’s try changing it:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/date?group_level=1' | more
{"rows":[
{"key":[1982],"value":[3593,1585]},
{"key":[1983],"value":[3556,1273]},
{"key":[1984],"value":[3457,1229]},
{"key":[1985],"value":[3096,1648]},
{"key":[1986],"value":[2880,1180]},
{"key":[1987],"value":[2828,1237]},
…

Well that’s interesting! group_level=1 vs 2 has a big effect: with group_level=2 we get the full granularity/dimensionality of the array we used for the key (first argument to emit). But we can, at query time, reduce even further by integrating over month! Very cool, and the gory details are few: group_level runs between 0 and length(key), so in this case valid values are 0, 1, 2. If we specify group_level=0, we’ll get sum statistics over the entire date range:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/date?group_level=0'
{"rows":[
{"key":null,"value":[68387,38652]}
]}

Awesome. And therein lies the power of using MapReduce for an efficient index build. With Cloudant you don’t need to install any software, you simply use the cloud hosted cluster to run your code. If you think about it, all we really had to do was write 9 lines of javascript for the map method, and no original code for the reduce method. We can use this same exact index to query for a given date range, say 1994:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/date?group_level=2&startkey=\[1994\]&endkey=\[1995\]'
{"rows":[
{"key":[1994,0],"value":[116,60]},
{"key":[1994,1],"value":[142,42]},
{"key":[1994,2],"value":[191,67]},
{"key":[1994,3],"value":[179,72]},
{"key":[1994,4],"value":[224,74]},
{"key":[1994,5],"value":[223,98]},
{"key":[1994,6],"value":[267,131]},
{"key":[1994,7],"value":[260,101]},
{"key":[1994,8],"value":[212,200]},
{"key":[1994,9],"value":[169,129]},
{"key":[1994,10],"value":[131,79]},
{"key":[1994,11],"value":[143,130]}
]}

or if we want to do can turn off the reduce in the query and get all of the crashes from a given year/month combo:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/date?key=\[1994,4\]&reduce=false' | more
{"total_rows":68387,"offset":34959,"rows":[
{"id":"88601d815897032516f87f0c510c7c60","key":[1994,4],"value":[1,1]},
{"id":"88601d815897032516f87f0c510c87b7","key":[1994,4],"value":[1,0]},
{"id":"88601d815897032516f87f0c510c951a","key":[1994,4],"value":[1,2]},
{"id":"88601d815897032516f87f0c510c9b0a","key":[1994,4],"value":[1,0]},
{"id":"88601d815897032516f87f0c510c9bbb","key":[1994,4],"value":[1,0]},
{"id":"88601d815897032516f87f0c510c9c15","key":[1994,4],"value":[1,1]},
{"id":"88601d815897032516f87f0c510c9d6d","key":[1994,4],"value":[1,0]},
{"id":"88601d815897032516f87f0c510ca925","key":[1994,4],"value":[1,0]},
…

Wow, handy! And if you want to really blow your mind, add the query-string parameter ?include_docs=true to get the full documents themselves.

Summary

There’s a lot of power in MapReduce, and I hope I’ve given you enough to get your hands dirty running it an an extremely simple way using a hosted service. If you have more questions, there are lots of examples on the CouchDB wiki or on the (Cloudant support center). But the rule of thumb is:

sort: map only
aggregate: use a _sum reduce + group_level
time-series: use an array of [year, month, day, …] + group_level

In the next post, we’ll show you how to link Cloudant MapReduce views with builtin full-text/numerical search for killer data visualizations.

Developer Preview: Cloudant Search for CouchDB

2011-01-11T00:00:00-08:00

CouchDB is a compelling choice for database-backed web apps and analytics applications. Cloudant’s hosted CouchDB service adds scaling, redundancy, and fault-tolerance to CouchDB. Today we are releasing a new feature for developers to try out: full text indexing and search.

There are several products that implement search for CouchDB as external services (couchdb-lucene and ElasticSearch). Like these, our full text search uses Lucene query syntax and is based on Lucene. For Cloudant search, however, the search application is directly integrated into the Cloudant CouchDB database. This has several distinct advantages over other Lucene based search products:

Real-time Search. Search results reflect state of database at moment it is searched. Little to no latency on document additions, changes, or deletions.
Indexing and Search using all valid JSON types. In Lucene, the search terns are required to be strings. This is especially problematic for date range and numerical range searches. With Cloudant search, the tokens can be any valid JSON type (string, number, object, array, boolean, null).
Fault-tolerance and redundancy. The search indices are stored on the same nodes as their parent documents.
Little to no setup and configuration. For basic search, the procedure is upload documents, enable search, search documents using REST interface.
Scalability. Cloudant Core is designed to be horizontally scalable document store. Cloudant Search is a horizontally scalable search product.

Under the hood, our search consists of two parts. The first step is the creation of a reverse index, or posting list, for your documents. This reverse index maps terms to the documents in which they appear. We have implemented document indexing as a MapReduce view using couchjava, our Java language view server for CouchDB. This allows for usage of custom application-specific Lucene analyzers written in Java. In fact, your existing Lucene analyzers can be imported directly into Cloudant. If you prefer, you can write your document indexer in Javascript as long as you write a MapReduce view using the correct format (in a later blog post, we’ll show how our search application can be used for ad-hoc queries of your CouchDB database).

The second part of Cloudant search is a RESTful API that takes your search query and performs the necessary boolean logic to return the most relevant documents.

What is Cloudant search good for? Here are some applications that are easy (and cheap) to build using Cloudant full text indexing and search:

Searchable Real Time Catalog. If you are starting the next Ebay, build it using Cloudant. We combine full text search and numerical range queries.
Document Discovery by the Hour. If you have a collection of documents that you wish to search through but don’t need to archive, you can upload them to Cloudant, enable search, run your queries, and delete the database.

Currently, the beta version of Cloudant search is available to all hosted customers with paid accounts (including users of the Cloudant Heroku Addon). The version of the search product we are rolling out today is designed for smaller databases (less that a few gigabytes). An enhanced version that uses distributed search for larger databases will be available soon. The feature is free, but the search index counts against your data storage quota and the search queries count against your query quotas.

A technical description of the search API can be found on the Search API page. Instructions for making your databases searchable can be found on the Search Indexing page. Please submit your questions and suggestions to our discussion forum. If your application requires advanced indexing or search capabilities, please contact us for a quote on a custom solution. Please contact us for further information.

Over the next weeks, we’ll be posting tutorials on various ways the search can be used for web applications and ad-hoc analytics. Please stay tuned.

Bob Newson and Tim Anglade Join Cloudant!

2011-01-05T00:00:00-08:00

It’s a huge day here Cloudant HQ; we are very excited to announce the two newest additions to our team: Bob Newson and Tim Anglade.

Tim Anglade certainly needs no introduction, at least in the world of NOSQL. I first met Tim here in Boston back in early 2009 when he gave the keynote at the “NOSQL Live” conference we cosponsored with 10Gen. Since that time, Tim has been a tireless promoter, interlocutor, and sometimes skeptic of all things NOSQL. This past year, he organized the NOSQL Summer reading group that had participation from more than 35 cities in 20 countries around the world. Since then he has been globetrotting on his “NOSQL World Tour,” organizing and promoting events large and small, and working on the NoSQL Tapes, a series of conversations with domain experts, thought-leaders, trailblazers, and Cloudant founders. Tim will be moving from his home country of France to join us here at Cloudant HQ in Boston (in January! buy a thick parka Tim.) Tim will be taking on various challenges at Cloudant including developer advocacy, community development, customer outreach and support, technical writing, and directing our social media efforts. We’re basically piling onto Tim all the stuff that we have never been able to do well. This means that the blog you’re reading is about to become much more exciting and useful. Please join us in welcoming Tim by directing all your questions and complaints to tim@cloudant.com. You should also follow him on Twitter: @timanglade.

Many of you already know Bob Newson from his prolific work with CouchDB. Bob has been a contributor on the project since 2008. Among his countless contributions to that project is the CouchDB-lucene extension, which provides full-text indexing capabilities. Bob is an expert in distributed systems, big data processing, and erlang. His official title at Cloudant will be ‘Senior Database Engineer,’ but unofficially we’re already referring to him as the resident ‘super-hard-problem solver’. He’ll be putting his software expertise to good use building new features for Cloudant and BigCouch, and streamlining our systems to make them faster, stronger, and more reliable. For now, Bob will continue to live and work from Windsor in his native land of England (God Save The Queen!) which is nice, because he’ll be able to provide early morning support while those of us in The States are still sleeping soundly. In his free time, Bob enjoys reading about Paxos and drinks with sparklers in them. Welcome to the team Bob! We couldn’t be more excited to have you on board.

P.S. – The pop-culture nerd in me would be remiss if I didn’t point out that both Bob Dionne and Bob Newson are now working with Cloudant. This means that I now have the chance to make endless Office Space allusions (“I have a meeting with The Bobs,” “Hey Bob! Bob!” etc.) which brings me no end of joy. It also gives me an excuse to put this picture up and have a good laugh.

“What would you say, you do here?”

Scaling CouchDB with BigCouch

2010-10-25T00:00:00-07:00

On Friday Adam gave an free webcast on BigCouch. I, in my completely biased opinion, think that he killed it. The discussion in the chat room and the questions were fantastic. I hope the community got a lot of this. I’m posting the slides above. The good people at O’Reilly have told us that they will put up a full video of the talk (thanks Kathryn!) When that goes up, I encourage everyone to see it, you’ll definitely get more out of that than just reading the slides. Thanks to all those who attended and made it such a great presentation. Hopefully we’ll get to do another one soon.

P.S. – We’d love your feedback on BigCouch and Cloudant. Help us out by picking one of the choices on our survey page cloudant.com/feedback. You could win a free upgrade to an Argon or Helium plan for a year.

Welcome Bob Dionne to Team Cloudant!

2010-10-07T00:00:00-07:00

The Cloudant team just got a little bit bigger. We’re super excited to welcome Bob Dionne (@bitdiddle) as our new Senior Software Engineer. Bob has been developing software for more than 25 years, and spent a significant chunk of that time at IBM. Since leaving he has built his own software company and, more recently, founded the consulting firm Dionne Associates. Needless to say, the man has serious chops when it comes to writing code, which is good because we’re going to be throwing a lot of hard problems at him. Bob is a true New Englander (with a wicked awesome accent to boot) and though he’ll be mostly working remotely, we’re looking forward to him joining us at the Boston office a couple times a month.

We (or I should say, I) first met Bob at the NoSQL Live Boston event. I was aware of his stellar work with CouchDB, where he focused some of his efforts on the etap testing framework. I was struck by his quiet thoughtfulness, and his readily-apparent passion for the project and the community. I feel privileged to have him join our team. So far, he’s been digging into BigCouch learning the codebase and helping us build up our testing framework. Over the coming months I’m sure he’ll be building and testing features for BigCouch and Cloudant and working very closely with Adam (our CTO) to make sure our hosted offering is a solid as possible.

Please join me in welcoming Bob to the team!

Java Language MapReduce Views

2010-09-08T00:00:00-07:00

One of the core principles at Cloudant is push code to data. In the era of big data and open source software, it’s much easier to move your code around than to move the data. Code is not tied to machine by software licenses, and the code is generally smaller than the data one wishes to analyze. When I first heard the concept push code to data, the retained marketing synapse in my brain immediate fired with write once run anywhere, the Sun corporation marketing slogan for Java.

The Cloudant Hosted CouchDB service uses MapReduce views to analyze data. Code is contained in a design document. The design document is passed to a view server along with each document in the corresponding database. CouchDB view servers have been written for many interpreted languages (see the CouchDB wiki for a list), but until now there has not been a Java Language View Server for CouchDB.

Today we are releasing the Java Language MapReduce View Interface for Cloudant’s Hosted CouchDB service. This interface defines the protocol for writing MapReduce views in Java that can be run on our hosted CouchDB platform. The Java language has several features that make it extremely useful for large scale analytics within CouchDB:

Compile-Time Type Checking. CouchDB views are generally written in weakly typed interpreted languages (javascript, python). Anyone who has worked extensively with CouchDB, however, knows that debugging complicated views can be very challenging. Sometimes it is easier to let the compiler find the syntax errors.
Libraries, Libraries, and More Libraries. Java is the most widely used computer language at the moment. As such, there is a library for almost everything. CouchJava allows for the re-use of existing Java langauge libraries within the CouchDB framework. Libraries can be packaged in jar files and uploaded directly to the view server.

When might you use this? Let’s assume you want to use a neural network to analyze financial options data in real time. You can directly import your favorite neural network library(there are many written for Java)directly into CouchDB along with the weights that you have calculated during training (the weights will be configuration parameters in the design document). Each new piece of data will be automatically analyzed in real-time. You can update the weights by modifying the design document. In fact, one could build the unsupervised learning cycle directly into CouchDB.

The Java view server works differently than a standard CouchDB view server. The design document does not contain code. Instead, the design document specifies which class should be called for the Map and Reduce steps. The code (a jar) is attached to the design document in the form of a binary attachment. This jar contains both user defined classes and external libraries that are needed. This paradigm (libraries as binary attachments) is a non-standard extension of the CouchDB view server API.

The interface and instructions for writing Java Language MapReduce views are now available on Github. Let us know what you think.

Cloudant’s BigCouch is open-source

2010-08-30T00:00:00-07:00

The Cloudant team is pleased to make available its ‘BigCouch’ software project as open source software under the Apache 2.0 license. We have been the beneficiary of countless open source projects while constructing this system, so it is only fitting that we share our efforts in hopes that people will not only find utility, but also assist us in making it better.

It has taken us a while to get to this point. The version we are opening has benefitted from Cloudant operating our systems in production for our customers for almost a year. The recent (and heavy) refactoring effort has encompassed our learning over this time, and we believe it is now time to share this system with the world. We will now be focusing our efforts on documenting and ease of use, as is the case with most newly opened projects.

What does it do? Think of BigCouch as a set of Erlang/OTP applications that allow you to create a cluster of CouchDBs that is distributed across many nodes/servers. Instead of one big honking CouchDB, the result is an elastic data store which is fully CouchDB API-compliant. The top is a picture of the standalone CouchDB setup that is common today. When you download BigCouch and build with make dev (see the README.md file), the result is the bottom picture, a three-node development cluster.

The clustering layer is most closely modeled after Amazon's Dynamo, with consistent hashing, replication, and quorum for read/write operations. CouchDB view indexing occurs in parallel on each partition, and can achieve impressive speedups as compared to standalone serial indexing.

Also contained within BigCouch is a tailor-made RPC server application called Rexi. Rexi better fits the needs of this distributed data store by dropping some unneeded overhead in rex, the RPC server that ships with Erlang/OTP. Rexi is optimized for the case when you need to spawn a bunch of remote processes. Cast messages are sent from the origin to the remote rexi server, and local processes are spawned from there, which is vastly more efficient than spawning remote processes from the origin.

So, let us know what you think. We will be hanging around Freenode IRC in the #cloudant channel. Pull the code, build it, install, and begin playing. Document, enhance, give back, benchmark, blog, tweet, …and enjoy.

Dynamo and CouchDB Clusters

2010-08-13T00:00:00-07:00

For a while, CouchDB was described as a “distributed, fault-tolerant and schema-free document-oriented database accessible via a RESTful HTTP/JSON API.” The story about CouchDB’s ‘distributed’ description has always involved its multi-master replication. In this sense, it is not truly a horizontally scalable database, as noted here. With the availability of Cloudant’s new hosted service, a new option has entered the scene. Our clustering is similar to Voldemort, Cassandra, or Riak](http://github.com/basho/riak), as it implements a version of Amazon’s Dynamo. We’d like to tell you a bit about how it works.

Clustering in a Ring

Cloudant makes use of partitions for sharding data. Picture a ring of partitions that each are responsible for a range of the data. The partitions, sometimes referred to as virtual nodes or vnodes, are placed on separate physical nodes as the primary means of distribution. Partitioning databases is nothing new, and with Dynamo systems, a keyspace is usually employed. We use CRC32, so from 0 to 2^32. We’ve seen MD5, fnv, and SHA-1 used, and previous versions of our software used SHA-1. When creating a database, users may provide Q, a constant that determines how many partitions the database will be spread across. Let’s check out an example:

$ curl -X PUT http://username.cloudant.com/dbname?q=10

For a Q of 10, there will be 10 partitions put across all the nodes in the cluster. Each partition will be responsible for a range of 2^32/Q keys.

Any Node can Handle a Request

Each request to the cluster will most likely go through a round-robin load balancer, because each of the nodes knows how to find the data in O(1) lookup time. To do this, the document id is hashed, and the resulting hash is compared against the database’s partition map, as we discussed above. If you’re referring to the shiny picture on this page, you will see that our example hashes ‘doc_id’ to the E partition. In reality, this could be something like hashing a value to c34b and that falls between c000 and cfff hex values, so we call that range the E partition. The partition map for that database knows which node or nodes have copies of the document, and requests are routed via distributed Erlang messages to those nodes.

Quorum system (N, R, W)

The durability and consistency settings of Dynamo systems are governed by quorum constants. As you can see from the diagram, N=3. This means that three replica copies of ‘doc_id’ document will be stored in the cluster. When the ‘write’ arrives at Node 1 and the consistent hashing algorithm tells us the doc will live on the E partition, the system attempts three asynchronous writes, one to each of the nodes that house the E partition replicas. You can be reasonably certain of the writes succeeding, because the request will wait for W (in this case, W=2) responses. When two successful responses return to Node 1, the response of 201 Created is returned to the client. The same thing occurs on ‘reads.’ When a GET request arrives at a node, N ‘read’ requests go out to the nodes which house that document’s partition. If R=2, then the system waits for two agreed-upon values to be returned, and then replies to the client with a 200 OK and the document’s contents.

You can set the defaults of N, R, and W for your cluster, but you may also override them at request time. N may be overridden at database creation time, just like Q above. R and W may be overridden for each request individually. A higher W value means more durable writes, where lower W value means more write throughput. High R values mean consistent values returned, while low R values mean first read wins, and higher read throughput.

Masterless (no SPOF)

Server failure is handled transparently. The load balancer will not route requests to a downed node. With consistent hashing, the other nodes are capable of serving requests. Also, when a node is down, other partitions will provide the data that is located on the downed node. If N=3, W=2, R=2 as in our example, a node may be down but read and write quorums are still satisfied. When the node returns, operations return to normal. The node is placed back into service by the load balancer, and a ‘hinted-handoff-like’ system is used to replicate the writes that the downed node may have missed. The other nodes holding data for the partitions of the downed node replicate the updates continuously after the node returns.

Horizontally Scalable

When the cluster gets closer to capacity, or could use more nodes to crunch data in a more parallel fashion, extra nodes may be added to the cluster. When this happens, partitions are moved from existing nodes to the new nodes. The system has the flexibility to move partitions on a database level, and can split large partitions if required. This is an evolving portion of the code base, and we are attempting to do partition merging and splitting all while the database remains online, i.e. other partitions serve data while the work is being done.

Above, there was mention of partition merging. This may be required as the cluster shrinks. Or, it is also possible to re-partition an over-partitioned database into fewer shards. The flexibility exists to shrink a cluster by removing nodes, just as it may be grown, providing a truly elastic data store.

Transparent to the application

All of these features – distributed, horizontally scalable, durable, consistent – happen with little or no change required in applications that have been written for CouchDB. A cluster looks just like a stand-alone CouchDB, and API compliance has been our goal from the beginning. Granted, there are a few extra options like overriding quorum constant defaults and there are a few vagaries, like views always performing rereduce due to the views being distributed. But on the whole, the extras in Cloudant are transparent to the application.

Feel free to comment away, email us, or join us on Freenode IRC in the #cloudant room. We’d love to hear from you.

Using Cloudant’s Heroku Add-on

2010-08-05T00:00:00-07:00

Heroku has recently jumped with both feet into the NoSQL world, providing add-ons for many different NoSQL data stores to complement it’s easy ruby platform. We’re very excited to be able to offer CouchDB 1.0 to Heroku users via the Cloudant add-on. Cloudant is a distributed CouchDB service, with the added benifits of data redundancy, clustering, and automatic maintenance (view builds and compaciton). You can use it just like you would standalone CouchDB but you have the advantage of your data being distributed over multiple machines. The following is a (very short) guide to help you get started with Cloudant and Heroku. It is by no means comprehensive. If you have quesitons, feel free to get in touch. You can read more about the add-on at the Heroku Blog.

Local setup

Since CouchDB and Cloudant have the same REST API you can use them the same way. You just need a http client to interface with the database and a JSON parser.

In the simplest case you can just use the rest-client and json gems:

$ sudo gem install rest-client json

If you don’t already have CouchDB on your local machine:

OSX:

$ brew install couchdb

Ubuntu:

$ sudo apt-get install couchdb

Using from Sinatra

Installing the Cloudant add-on gives you an environmental variable CLOUDANT_URL. This contains your API URL and credentials and is how you access your database(s) at Cloudant:

$ heroku config --long CLOUDANT_URL
https://USER:PASSWORD@APP.heroku.cloudant.com

When you first install the add-on you need to create a database:

Using cURL:

$ curl -X PUT https://USER:PASSWORD@APP.heroku.cloudant.com/DATABASENAME

Using rest-client:

$ RestClient.put("https://USER:PASSWORD@APP.heroku.cloudant.com/DATABASENAME", "")

You could also do this in your code by doing something like:

begin
  RestClient.put("#{ENV['CLOUDANT_URL']}/SOMEDATABASE", "")
rescue
  puts "database already created"
end

The following code will allow you to GET a document from the music database and then print it to the page:

DB = "#{ENV['CLOUDANT_URL']}/music"
get "/doc/:doc" do
  doc = RestClient.get("#{DB}/#{params[:doc]}")
  @result = JSON.parse(doc)
  haml :doc_id
end

The view would look something like:

%h1 A Doc from CouchDB!
- @result.each do |k,v|
  %b=k
  %em=v
  %br

To request this page by pointing your browser to:

http://SOMEAPP.heroku.com/doc/YOURDOCUMENT

Deploying to Heroku

To use Cloudant on Heroku, install the Cloudant add-on:

$ heroku addons:add cloudant:basic

Cloudant’s Own to Speak at SurgeCon

2010-07-27T00:00:00-07:00

As Cloudant’s ops guy and utility infielder, I (@williamsjoe) will be speaking at the Surge conference in September. Surge is a conference presented by OmitTI that focuses on infrastructure and web operations. As such I will be discussing how we have built Cloudant’s database platform on top of Amazon’s EC2. The conference will include keynotes from the always excellent John Allspaw (of Flickr and Etsy fame) and Theo Schlossnagle (OmniTI).

Cloudant and SETI: Crowdsourcing the search for E.T.

2010-07-21T00:00:00-07:00

Going to OSCON? SETI needs your help!

Cloudant is teaming up with the SETI Institute to help engage citizen-scientists in the search for intelligent life elswhere in the universe. With some help from Amazon, we’ve built a ‘croudsourced data analytics platform’ where ordinary citizens can search and analyze data from the Allen telescope array. Are you a citizen-scientist? Do you know about signal processing? Are you interested in helping in the search? Come find out how!

Thursday, July 22nd at OSCON, our own Dr. Dave Hardtke will be demonstrating SETICloud. He’ll be describing the Cloudant platform and how we’re bringing the wisdom of the crowds to SETI’s search.

Time: 12:20–1:20 pm

Place: Room E143/144

SETI is hacking on CouchDB! Come find out how you can help.

And don’t miss Dr. Jill Tarter’s keynote talk on “Open SEITQuest” Thursday morning at OSCON.

Learning to Relax

2010-06-28T00:00:00-07:00

Over the weekend I traveled to Chicago to give a talk on ‘CouchDB for Beginners’ at the WindyCityDB conference. You can read the slides above. The conference exceeded my expectations and many of the talks, both long and short, were stellar. Big thanks to Ray Hightower and all the people who helped put WindyCityDB together. I very much appreciate the opportunity to give this talk and I hope everyone enjoyed it.

Cloudant Nationwide Tour

2010-06-16T00:00:00-07:00

It turns out that next week (the week of June 21st) is a big week for tech conferences. A bunch of the Cloudant people will be hitting up various conferences and giving a couple talks. If anyone is interested in meeting up over beers to talk Cloudant, CouchDB, NoSQL, Startups, etc. get in touch (info at cloudant dot com is a good place to start.) Here are the tour dates:

Structure Conference. June 23–24. San Francisco, CA: Both Adam Kocoloski (CTO) and David Hardtke (Search) will be milling about this conference, whose tagline is “Put Cloud Computing To Work.” On the 23rd in the afternoon, Cloudant will be taking part in the Launchpad! event. Adam will be working the crowd, drumming up some love for Cloudant, giving a short pitch about our service. Check it out.
Velocity Conference. June 22–24. Santa Clara, CA: No Cloudant talk at this conference but Joe Williams (Operations and Utility Infeilding) will be in attendence. No one I know gets more pumped up about server uptime, configuration management, and monitoring than Joe. If you’re the same way, say hi.
Momentum Summit. June 23. Cambridge, MA: This one day conference focuses on how to turn a startup into big business. CEO Alan Hoffman (that’s me) will be in attendence. Looking forward to hearing from Boston startup gurus like Steve Kaufer and Scott Griffith. I’m always eager to talk with other local startup types so come find me.
Windy City DB. June 26. Chicago, IL: I’m lucky enough to be heading back to my hometown to attend and speak at this one day event on the south side of Chicago. My talk is entitiled “Learning to Relax: CouchDB for Beginners” and it is about, uh, CouchDB for beginners. It’s my first long-format NoSQL talk so if you’re in attendance, please no heckling. I’m happy to meet up after the conference; if you’re looking for me on Friday afternoon, I’ll be at U.S. Cellular Field watching the White Sox.

Just Opensourced: Gaff and Deckard

2010-06-04T00:00:00-07:00

Today we released two open source projects that have been in use internally at Cloudant for some time now, Gaff and Deckard.

All of our infrastructure is in the cloud and as such we need a way for disperate systems to all request resources, this is where Gaff comes in. Gaff is a pubsub daemon for asynchronously talking to cloud APIs using AMQP. Currently it supports a subset of the Dynect (DNS), Slicehost and EC2 APIs and uses @geemus’s awesome fog Ruby library. The basic workflow for Gaff is to send JSON-RPC formated messages to an AMQP exchange with a routing key corresponding to the API you are talking to, you could be sending these messages from a web application or another service. Each message gets routed to an API specific queue and is picked up by Gaff and turned into the appropriate API call, starting, stopping, modifying your servers on EC2 or elsewhere.

We have a lot of CouchDB instances to keep tabs on to do this we wrote Deckard. Deckard is a HTTP check monitoring system based on CouchDB. Yo dawg! What better than to monitor CouchDB with CouchDB (and some Ruby)? Deckard supports basic HTTP content checks, email alerts, SMS alerts (via email) for on-call rotations, basic maintenance scheduling, replication latency alerts (between two Couches) and even has EC2 Elastic IP support for failover between two EC2 instances. Best of all since it’s based on Couch you get an API for free, just PUT a doc in the HTTP checks database and you get a new HTTP check the next time Deckard runs.

Checkout these and my other projects on Github and follow @cloudant and myself on Twitter.

Ode to a Utility Infielder

2010-02-17T00:00:00-08:00

Today marks the one year anniversary of Joe Williams (@williamsjoe) becoming a member of Team Cloudant. Those of us who are married will recall that one year is the paper anniversary so it’s fitting that Joe will be receiving paper (in the form of vested stock options) from Cloudant today. I thought I would take a moment on this occasion to thank Joe for his work over the past year.

We first hired Joe on a contract basis to help us build an internal infrastructure tool we call The Deployer. This is the tool that allows us to quickly provision and configure resources from a number of different cloud providers. We can, with the push of one button, ‘inflate’ an X-node Cloudant database cluster, and have it configured and available in a matter of minutes, all because of Joe’s ongoing efforts.

Since joining full-time, Joe has acted as our ‘utility infielder.’ All the problems in the operations and infrastructure realm fell to Joe. He built and mantains our monitoring system, our logging system, and our build system. That alarm system we have where we get a text message in the middle of the night when something goes wrong, Joe did that (thanks Joe!). Joe, margarita in hand, has pushed hot code updates to live systems from a Mexican karaoke bar.

Needless to say Joe has been and continues to be a staggering presence at Cloudant. We feel privileged to work with him on a daily basis. I only hope I’m able to write a similar post on his 5 year anniversary. Plus, the man has excellent mutton chops.

Thanks Joe! Here’s to the big things (and fewer alarms) coming in the next year.

NoSQL Live From Boston

2010-01-27T00:00:00-08:00

We’re very excited to be joinin up with our friends at 10gen to sponsor the NoSQL Live conference right here in Boston, our backyard. The official press release can be found here.

The event will take place on March 11th at the John Hancock hotel. A number of us from Cloudant will be at the conference in both official (panel discussions, lightning talks) and unofficial (milling about, drinking) capacities. If you are in the New England area or plan on coming to the New England area, you should register and attend the conference. You can register here.

From the 10gen events site:

About:

A one-size-fits-all approach to databases no longer applies. Relational databases have worked well - and will continue to - for highly transactional systems. But today’s web applications require enormous scalability and performance. This has spurred the growth of a new class of databases that trade off some of the features of relational databases to offer high performance, ease of programming, high availability & the ability to scale in cloud environments. They are collectively called NoSQL or non-relational databases.

NoSQL Live from Boston is a full-day interactive conference that brings together the thinking in this space. Picking up where other NoSQL events have left off, NoSQL Live goes beyond understanding the basics of these databases to how they are used in production systems. It features panel discussions on use cases, lightning talks, networking session, and a NoSQL Lab where attendees can get a practical view of programming with NoSQL databases.

Interested in presenting or sponsorship opportunities? Contact meghan@10gen.com.

Welcome Dr. Dave Hardtke

2010-01-06T00:00:00-08:00

The Cloudant team is growing!

We’re excited to welcome Dr. Dave Hardtke who has joined Cloudant as Director of Search. Like many of us here at Cloudant, Dave began his career as an experimental particle physicist. In over a decade of experience Dave illuminated the secrets of the universe at various experiments such as STAR (Brookhaven National Lab), ICECUBE (Antarctica), and NA44 (CERN). Dave’s seminal work lead to the discovery of the Quark Gluon Plasma, the 2005 physics story of the year. More recently Dave has turned his talents in algorithms and analyses to search, where he has built his own startup stinkyteddy.com, a general purpose search engine combining multiple keyword-driven search feeds and intelligent semantic analysis to deliver the most timely search results regardless of source and content types. Prior to that Dave served as Chief Scientist at Surf Canyon, a startup focusing on personalization of real-time search.

Dave is accelerating Cloudant Search. As we add customers it has become clear that they are hungry not just for a flexible way to store thier data, but a powerful way to “discover” their data. Search is a critical component in nearly all data driven applications. We are therefore expanding our efforts to integrate a robust search platform into our CouchDB-based, cloud database service. Dave brings his expertise in internet and enterpise search technology to the project.

Please join me in welcoming Dave to the team.

Cloudant.com Refresh: CouchDB in the Cloud

2009-12-08T00:00:00-08:00

Our new front page just went live on Cloudant.com. The main goal of this redesign is to give more information about the service we are currently rolling out. While we are still in private beta, we are starting to accept more people onto our platform and therefore we are ready to tell you more about what we are doing. We are currently aiming at existing CouchDB users, and to them we are offering professional hosting of CouchDB in the cloud.

The new design follows the AIDA principle, according to which a succesful marketing flow involves four main phases: attention (A), interest (I), desire (D), and action (A).

The “CouchDB in the Cloud” headline is designed to attract the attention of CouchDB users. The screenshot behind it is meant to evoke interest by giving a peak to the features available in our dashboard. Underneath it, the three “features” listed aim to create desire. They link our offering to CouchDB, by stating that we provide all the benefits of the database that our potential customers already love, but also highlight what we have to offer on top of that: a scalable and managed infrastructure and the chance to use it for free. Finally, the two buttons, placed in what the Gutenberg Rule calls the “terminal area”, provide options for action.

Let us know what you think about this new design.

Changing the World, One Password at a Time

2009-12-01T00:00:00-08:00

Since launching the beta we’ve received a bunch of feedback along the lines of “Uh… I can’t change my password, wtf?” Yes, wtf indeed. We knew we would be building this feature eventually but we didn’t want to hold up the release for it. Well, now we have it on our new Account Options page.

You can access it by selecting ‘account options’ from the user account pulldown on the top right of the page. Thanks for all the feedback and keep it coming.

API Keys: No Really, We’re Listening

2009-11-05T00:00:00-08:00

Since Monday we’ve been sending out invites to the private beta (if you haven’t gotten yours yet, be patient, it’s coming). And we’ve gotten a lot of great feedback on how to make the service better. One suggestion that came from a few users was to allow database access to non-Cloudant users through API keys that can be shared with anyone. Well, your requests did not fall on deaf ears. For any of your databases you can now navigate to the permissions tab and create a database-specific API key and password. You can then set permissions for that key as if it were a Cloudant user. The screenshot below shows what that should look like. Thanks to everyone for the great suggestion. You keep the feature requests coming and we’ll get to them as fast as we can.

Benchmarking CouchDB with Baracus

2009-11-04T00:00:00-08:00

Baracus is a new project built specifically for benchmarking CouchDB. Using trusty httperf, Baracus makes it easy to create and perform a battery of tests based on simple configuration files. In this post I’ll go into the details of a Baracus run and show you how to run your own benchmarks.

First, let’s install Baracus. It depends on httperf so install it first either from source (check the README for compile options for httperf) or from your distribution. Note that Baracus uses gemcutter for gem hosting.

gem sources -a [http://gemcutter.org](http://gemcutter.org)
gem install baracus

Next, let’s examine a simple Baracus config file.

httperf: "/usr/bin/httperf"
host: "localhost"
port: "5984"
user:
password:
db: "testdb"
report_url: "http://localhost:5984/bench"
name: "test123"
config:
  timeout: 30
  sessions: 100
  rate: 100
  doc_size: 100
  writes: 10
  reads: 10
  batchok: false
info:
  storage: "raid10, 4 disks, ebs"
  anything: "stuff"
  more: "foo"

This configuration file specifies where the httperf executable lives, the host, port, user, password, and database to use for the test. Baracus puts all of the results and wsesslog files back into CouchDB for later consumption, the report_url is where you define this database.

As I mentioned, Baracus uses httperf to perform benchmarks. Specifically it uses httperf’s wsesslog feature. wsesslog is a simple file format that describes what httperf should do each session. Here’s an example of an entry generated with Baracus that does an HTTP GET:

/testdb/a12bf8572f788824dfd5169e2eb496ab method=GET

Baracus creates two wsesslogs for each run, one for writes (POST) and one for reads (GET). Baracus generates these files based on a few items in your configuration. Specifically, the number of sessions, document size and how many reads or writes to do per session. Additionally in the configuration you can specify at what rate httperf should perform the operations within each wsesslog file. Baracus creates the wsesslog for writing with random data and creates the read file with random docs from alldocs. Since Baracus uses alldocs, it performs all the writes first. The above config will create 100 sessions with 10 writes and reads each at a rate of 100 per second. Each document will be roughly 100 bytes.

The info section at the end of the config contains any additional information you may want to have tag along with your results. You can put anything you want in there; I generally use this to describe the system I just ran the tests on.

Once you have a configuration you are happy with it’s very simple to run your tests.

$ baracus baracus.yml

This will print the standard httperf output as well as a tab delimited version. If you want to checkout the wsesslog files they will be written to the directory you ran Baracus in as well as to the results document in the reporting database. Here’s what the results look like in Couch.

Baracus is meant to make repeatable benchmarking easy. If you have any questions, bugs or would like to contribute check out Baracus on Github.

Cloudant at No:SQL (east)

2009-08-31T00:00:00-07:00

Cloudant’s own Brad Anderson is helping to organize No:SQL (east), a conference centered around the latest nonrelational database systems. The conference will be held at the Georgia Tech Research Institute in Atlanta on October 28–30. The description from the website reads:

NoSQL East, a conference of non-relational data stores, aims to present the experiences of different companies using so-called ‘NoSQL’ solutions in production.

The reasons for using these new systems are varied, but often involve scale that relational solutions cannot achieve. We have a new tool in our toolbelts, and as you will see, some people are using them quite effectively.

Join us in Atlanta as we discuss ‘Big Data’ and the systems that are completely transforming how people look at data.

The organizers have a great venue picked out (courtesy of Georgia Tech) and an exciting group of speakers. A number of nonrelational systems will be covered, and Cloudant co-founder Mike Miller will be giving a talk on CouchDB. Anyone interested in the NoSQL space should really try to make it down to Atlanta to attend.

Cloudant is one of two primary sponsors of the event, the other being GTRI. Many, if not all, of us will be attending the conference. We’re always happy to talk about NoSQL, CouchDB, JQuery, The Bears winning the Superbowl, etc. You should come say hi.

More information about No:SQL (east) can be found at nosqleast.com. Check it out.

Update (10/22/09): Since posting the above, a couple other great companies, including Basho and Rackspace Cloud, have also joined on as sponsors for the event.

Well, how did I get here? – Talking Heads

2009-07-08T00:00:00-07:00

Dharmesh tells me that I’m supposed to write a blog post explaining how we got to where we are. I’ll try my best, but sometimes I feel like the guy in that Talking Heads song, looking around saying, “This is not my beautiful house. This is not my beautiful wife.” This company formed sometime between 2004 and 2008, but definitely around one of the rather large lunch tables on the first floor of the Stata Center at MIT. Adam, Mike, and I all worked in the same group in the physics department and every day we ate lunch around one of those oddly shaped tables. By January of 2008 came to the conclusion that we couldn’t leave MIT without starting a company. We didn’t really know what we wanted to do, but we did know 1) We were all fairly bright and industrious guys, 2) we enjoyed working together, and 3) we didn’t want to go look for “real” jobs.

Mike’s original idea was for an automatic tamale maker. I’m not kidding. That one got nixed when we found out we wereway behindalready.

Eventually we decided to play to our strengths, which were in large-scale data processing and analysis. Our high-energy physics experiments generated some of the largest data sets in the world. It was our job to sift through the terabytes and petabytes to find rare particles and strange interactions. This is what we did for fun. Turns out that we’re living in a data-centric world (more so every day) and many of the problems we had faced were popping up in the business world. What we wanted was a way to bring physics-style data analysis tools to the masses. All we had to do was build it.

Toward the end of March someone sent me Paul Graham’s essay “A Student’s Guide to Startups.” I had never heard of PG before, but his essay resonated with me, especially the part about getting to stop treading water. I stumbled upon YCombinator literally by clicking random links on Graham’s website. As it turned out, this YCombinator Company was pretty well known and respected, and they were asking for applications for startups that needed funding. Hey! We were a startup that needed funding and we were already in Boston. So we applied, almost on a whim. And to be honest, we were a little bit floored when we got accepted.

After a number of back-and-forths with PG and other YC companies, we decided to focus on building a database designed for the specific needs of the web (hence our tagline “A database for the web,” clever huh?) One that would scale horizontally, could run on commodity hardware in a fault tolerant fashion, with a simple interface, and a powerful built-in analysis engine. We stumbled on Apache CouchDB, which at the time was in its relative infancy, and were very impressed with what they had built, especially the natural JSON-over-HTTP interface. We decided to take Couch and build it into a true ‘cloud-ready’ database.

It’s been more than a year since we started working on Cloudant. Since then a huge community has sprung up around CouchDB. Our own Adam Kocoloski was tapped to be an official committer on the project. We were able to raise significant additional funding late last year, just as the economy was collapsing – a true testament to the vision of our amazing VC. The Cloudant Team has grown from 3 to 6 as we were able to convince a few top-notch hackers to help us build our vision.

We’ll soon be releasing a beta version of Cloudant: CouchDB-in-the-cloud. It seems as though we were on to something when we began tossing ideas around in the Stata Center 18 months ago. The NoSQL movement, which is gaining steam, indicates the need for a new type of database, one built for the specific needs of web applications. At Cloudant we hope to fill that need. If you want to learn more or sign-up to be notified of our impending release, click here.

Cloudant Blog

Search 2.0

Background

Search 1.0

What Search 2.0 Isn’t

What Search 2.0 Is

We're on a Mission to Make it Easy

So long, static

So long, friction

Getting started

Big Science, Big Data

Big Science, Big Data

Getting the job done

From Higgs to Cloudant

Cloudant Helps Light Up the 2012 London Olympics

How Energy of the Nation Works

Managing applications on Cloudant

The tools

Summary

A Good Week for PHP and Cloudant

Lean and Technical-- How we chose and furnished an office the YCombinator way.

Location.

Space.

Furnishings.

Cloudant Data Layer Now Available on Windows Azure

Cloudant Data Layer on Joyent Cloud

Cloudant CouchDB Summit Drinkup

The anti-sprint

Tools

Value

Culture

Slides and Presentation from Meteor Solutions and Softlayer

Andy Palmer Joins Cloudant Board of Directors

Announcing CouchDB Summit Boston

Paul Davis Elected to the CouchDB PMC

Multi-Tenant Cloudant in Europe

Announcing BigCouch 0.4

Cloudant SF Drinkup

The Future of Apache CouchDB

A Big Finish for 2011

Slides and Presentation from 2600hz

Monsanto Chooses Cloudant

Optimizing your CouchDB Calls by 99%

Using Cloudant from Scala on CloudBees

A Technical Look at Cloudant Search

Announcing Cloudant Search

.NET + CouchDB = ♥: Announcing the Cloudant Add-On for AppHarbor

Paul Davis is Joining Cloudant!

Mike Miller Talks Cloud Databases at GigaOM Structure

CouchDB & Java in the Cloud, Now Easier than Ever with CloudBees

A Closer Cloudant? Now Accepting Reservations for a European Cluster!

Search Indexing in Javascript

Fukushima from Afar

Rise & Shine

“Using Incentive Prizes to Crowdsource Brilliance”

New Cloudant HQ (with pictures!)

BigCouch 0.3

Some Thoughts on the Couchbase Announcement

SearchApp – Add a Searchbox to your CouchApps and CouchDB Databases in 10 Minutes

MapReduce from the basics to the actually useful (in under 30 minutes)

Brief NOSQL Taxonomy

MapReduce Primer

Our Example Problem

Data Import

Example 1: Sort

Example 2: Data Aggregation

Example 3: Time Series Analysis

Summary

Developer Preview: Cloudant Search for CouchDB

Bob Newson and Tim Anglade Join Cloudant!

Scaling CouchDB with BigCouch

Welcome Bob Dionne to Team Cloudant!

Java Language MapReduce Views

Cloudant’s BigCouch is open-source

See Also

Github Project Page

Cloudant Developer Resources

Dynamo and CouchDB Clusters

Clustering in a Ring

Any Node can Handle a Request