Just a little Python

Moving along with PyMongo

2012-01-31T14:08:00.000-05:00

In a previous post, I introduced MongoDB and its Python driver pymongo. Here, I continue in the same vein, describing in more detail how you can become productive using pymongo.

Connecting to the Database

In the last post, I gave a brief overview of how to connect to a database hosted on the MongoLab service using pymongo, so I won't go into detail on that here. Instead, I will mention a few options you'll want to be aware of.

Some of the connection options set the default policy for the safety of your data:

safe MongoDB by default operates in a "fire-and-forget" mode where all data-modifying operations are optimistically assumed to succeed. Turning on safe mode changes this, waiting for a response from the database server indicating that an operation has succeeded or failed before returning from a data-modifying operation.
journal In version 1.8, MongoDB introduced journaling to provide single-server durability. MongoDB's flexible approach to balancing safety and performance, however, means that if your application wants to make sure its data has really been saved, you need to wait for the journal to be written.
fsync This is the really, really safe option. With or without a journal, this will wait until your data is on the physical disk before it returns from update operations.
w Before journaling, MongoDB used (and still uses) replication to ensure the durability of your data. The w parameter allows you to control how many servers (or which set of servers) your update has been replicated to before returning from a data-modifying operation. Be aware that this parameter can significantly slow down your updates, particularly if you are requiring them to be replicated to different datacenters.
read_preference This option allows you to specify how you'd like to handle queries. By default, even in a replica set configuration, all your queries will be routed to the primary server in the replica set to ensure strong consistency. Using read_preference you can change this policy, allowing your queries to be distributed across the secondaries of your replica set for increased performance at the cost of moving to "eventual consistency."

One thing that's nice about the pymongo connection is that it's automatically pooled. What this means is that pymongo maintains a pool of connections to the mongodb server that it reuses over the lifetime of your application. This is good for performance since it means pymongo doesn't need to go through the overhead of establishing a connection each time it does an operation. Mostly, this happens automatically. you do, however, need to be aware of the connection pooling, however, since you need to manually notify pymongo that you're "done" with a connection in the pool so it can be reused.

Now that all that is out of the way, the easiest way to connecto to a MongoDB database from python is below (assuming you are running a MongoDB server locally, and that you have installed ipython and pymongo):

In [1]: import pymongo

In [2]: conn = pymongo.Connection()

Inserting Documents

Inserting documents begins by selecting a database. To create a database, you do... well, nothing, actually. The first time you refer to a database, the MongoDB server creates it for you automatically. So once you have your database, you need to decide which "collection" in which to store your documents. To create a collection, you do... right - nothing. So here we go:

In [3]: db = conn.tutorial

In [4]: db.test
Out[4]: Collection(Database(Connection('localhost', 27017), u'tutorial'), u'test')

In [5]: db.test.insert({'name': 'My Document', 'ids': [1,2,3], 'subdocument': {'a':2}})
Out[5]: ObjectId('4f25bcffeb033049af000000')

One thing to note here is that the insert command returned us an ObjectId value. This is the value that pymongo generated for the _id property, the "primary key" of a MongoDB document. We can also manually specify the _id if we want (and we don't have to use ObjectIds):

In [6]: db.test.insert({'_id': 42, 'name': 'My Document', 'ids': [1,2,3], 'subdocument': {'a':2}})
Out[6]: 42

A note on document structure

A note here is order about the kind of documents you can insert into a collection. Technically, the type of document is described by the BSON spec, but practically you can think of it as JSON plus a few extra types. Some of the types you should be aware of are:

primitive types Python ints, strings, and floats are automatically handled appropriately by pymongo.
object Objects are represented by pymongo as regular python dicts. Technically, in BSON, objects are ordered dictionaries, so if you need or want to rely on the ordering you can use the bson module, included with pymongo, to create such objects. Objects have strings as their keys and can have any valid BSON type as their values.
array Arrays are represented by pymongo as Python lists. Again, any valid BSON type can be used as the values, and the values in an array do not need to be of the same type.
ObjectId ObjectIds can be thought of as globally unique identifiers. They are often used to generate "safe" primary keys for collections without the overhead of using a sequence generator as is often done in relational databases.
Binary Strings in BSON are stored as UTF-8 - encoded unicode. To store non-unicode data you must use the bson.Binary wrapper around a Python string.

Documents in MongoDB also have a size limit (16MB in the latest version), which is enough for many use-cases, but is something you'll need to be aware of when designing your schemas.

Batch inserts

MongoDB and pymongo also allow you to insert multiple documents with a single API call (and a single trip to the server). This can significantly speed up your inserts, and is useful for things like batch loads. To perform a multi-insert, you simply pass a list of documents to the insert() method rather than a single document:

In [7]: db.test.insert([{'a':1}, {'a':2}])
Out[7]: [ObjectId('4f25c0aceb033049af000001'), ObjectId('4f25c0aceb033049af000002')]

You may have noticed that the structure of the documents inserted in this snippet differ (significantly!) from the documents inserted before. MongoDB does not make any requirements that documents share structure with one another. Analogous to Python's dynamic typing, MongoDB is a dynamically typed database, where the structure of the document is stored along with the document itself. Practically, I've found it useful to group similarly structured documents into collections, but it's definitely not a hard-and-fast rule.

Querying

OK, now that you've got your data into the database, how do you get it back out again? That's the function of the find() method on collection objects. With no parameters, it will return all the documents in a collection as a Python iterator:

In [8]: db.test.find()
Out[8]: <pymongo.cursor.Cursor at 0x7f2910068b90>

In [9]: list(db.test.find())
Out[9]: 
[{u'_id': ObjectId('4f25bce9eb033049a0000000'),
  u'ids': [1, 2, 3],
  u'name': u'My Document',
  u'subdocument': {u'a': 2}},
 {u'_id': ObjectId('4f25bcffeb033049af000000'),
  u'ids': [1, 2, 3],
  u'name': u'My Document',
  u'subdocument': {u'a': 2}},
 {u'_id': 42,
  u'ids': [1, 2, 3],
  u'name': u'My Document',
  u'subdocument': {u'a': 2}},
 {u'_id': ObjectId('4f25c0aceb033049af000001'), u'a': 1},
 {u'_id': ObjectId('4f25c0aceb033049af000002'), u'a': 2}]

Most of the time, however, you'll want to select particular documents to return. You do this by providing a query as the first parameter to find. Queries are represented as BSON objects as well, and are similar to query-by-example that you may have seen in other database technologies. For instance, to retrieve all documents that have the name 'My Document', you would use the following query:

In [13]: list(db.test.find({'name':'My Document'}))
Out[13]: 
[{u'_id': ObjectId('4f25bce9eb033049a0000000'),
  u'ids': [1, 2, 3],
  u'name': u'My Document',
  u'subdocument': {u'a': 2}},
 {u'_id': ObjectId('4f25bcffeb033049af000000'),
  u'ids': [1, 2, 3],
  u'name': u'My Document',
  u'subdocument': {u'a': 2}},
 {u'_id': 42,
  u'ids': [1, 2, 3],
  u'name': u'My Document',
  u'subdocument': {u'a': 2}}]

MongoDB also allows you to 'reach inside' embedded documents using the dot notation. Here are some examples of how you can use that:

In [22]: db.testq.insert([
   ....:         { 'a': { 'b': 1 }, 'c': [{'d': 1}, {'d':2}, {'d':3}]},
   ....:         { 'a': { 'b': 2 }, 'c': [{'d': 2}, {'d':3}, {'d':4}]},
   ....:         { 'a': { 'b': 3 }, 'c': [{'d': 3}, {'d':4}, {'d':5}]}
   ....:         ])
Out[22]: 
[ObjectId('4f25c89beb033049af000009'),
 ObjectId('4f25c89beb033049af00000a'),
 ObjectId('4f25c89beb033049af00000b')]

In [23]: # reach inside objects

In [24]: list(db.testq.find({'a.b': 1}))
Out[24]: 
[{u'_id': ObjectId('4f25c89beb033049af000009'),
  u'a': {u'b': 1},
  u'c': [{u'd': 1}, {u'd': 2}, {u'd': 3}]}]

In [25]: list(db.testq.find({'a.b': 2}))
Out[25]: 
[{u'_id': ObjectId('4f25c89beb033049af00000a'),
  u'a': {u'b': 2},
  u'c': [{u'd': 2}, {u'd': 3}, {u'd': 4}]}]

In [26]: # find objects where *any* value in an array matches 

In [27]: list(db.testq.find({'c': {'d':1}}))
Out[27]: 
[{u'_id': ObjectId('4f25c89beb033049af000009'),
  u'a': {u'b': 1},
  u'c': [{u'd': 1}, {u'd': 2}, {u'd': 3}]}]

In [28]: # reach into an array

In [29]: list(db.testq.find({'c.d': 2}))
Out[29]: 
[{u'_id': ObjectId('4f25c89beb033049af000009'),
  u'a': {u'b': 1},
  u'c': [{u'd': 1}, {u'd': 2}, {u'd': 3}]},
 {u'_id': ObjectId('4f25c89beb033049af00000a'),
  u'a': {u'b': 2},
  u'c': [{u'd': 2}, {u'd': 3}, {u'd': 4}]}]

In [30]: list(db.testq.find({'c.1.d': 2}))
Out[30]: 
[{u'_id': ObjectId('4f25c89beb033049af000009'),
  u'a': {u'b': 1},
  u'c': [{u'd': 1}, {u'd': 2}, {u'd': 3}]}]

One other thing that's important to be aware of is that you can return only a subset of the fields in a query. (By default, _id is always returned unless you explicitly suppress it.) Here is an example:

In [31]: list(db.testq.find({'c.1.d':2}, {'c':1}))
Out[31]: 
[{u'_id': ObjectId('4f25c89beb033049af000009'),
  u'c': [{u'd': 1}, {u'd': 2}, {u'd': 3}]}]

In [32]: # we can also reach inside when specifying which fields to return

In [33]: list(db.testq.find({'c.1.d':2}, {'a.b':1}))
Out[33]: [{u'_id': ObjectId('4f25c89beb033049af000009'), u'a': {u'b': 1}}]

We can also restrict the number of results returned from a query by skipping some documents and limiting the total number returned:

In [66]: list(db.testq.find())
Out[66]: 
[{u'_id': ObjectId('4f25c89beb033049af000009'),
  u'a': {u'b': 1},
  u'c': [{u'd': 1}, {u'd': 2}, {u'd': 3}]},
 {u'_id': ObjectId('4f25c89beb033049af00000a'),
  u'a': {u'b': 2},
  u'c': [{u'd': 2}, {u'd': 3}, {u'd': 4}]},
 {u'_id': ObjectId('4f25c89beb033049af00000b'),
  u'a': {u'b': 3},
  u'c': [{u'd': 3}, {u'd': 4}, {u'd': 5}]}]

In [67]: list(db.testq.find().skip(1).limit(1))
Out[67]: 
[{u'_id': ObjectId('4f25c89beb033049af00000a'),
  u'a': {u'b': 2},
  u'c': [{u'd': 2}, {u'd': 3}, {u'd': 4}]}]

The query and advanced query pages in the MongoDB docs provide the full query syntax, which includes inequalities, size and type operations, and more.

Indexing

Like any database, MongoDB can only perform so well by scanning collections for matches. To provide better performance, MongoDB can use indexes on its collections. The normal method of specifying an index in MongoDB is by "ensuring" its existence, in keeping with the pattern of having things spring into existence when they're needed. To create an index on our 'test' collection, for instance, we would do the following:

In [13]: db.test.drop()

In [14]: db.test.ensure_index('a')
Out[14]: u'a_1'

In [15]: db.test.insert([
   ....: {'a': 1, 'b':2}, {'a':2, 'b':3}, {'a':3, 'b':4}])
Out[15]: 
[ObjectId('4f28261deb033053bc000000'),
 ObjectId('4f28261deb033053bc000001'),
 ObjectId('4f28261deb033053bc000002')]

In [16]: db.test.find({'a':2})
Out[16]: <pymongo.cursor.Cursor at 0x24f7b90>

In [17]: db.test.find_one({'a':2})
Out[17]: {u'_id': ObjectId('4f28261deb033053bc000001'), u'a': 2, u'b': 3}

Ok, well that's not actually all that exciting. However, MongoDB provides an explain() method that lets us see whether our index is getting used:

In [18]: db.test.find({'a':2}).explain()
Out[18]: 
{u'allPlans': [{u'cursor': u'BtreeCursor a_1',
   u'indexBounds': {u'a': [[2, 2]]}}],
 u'cursor': u'BtreeCursor a_1',
 u'indexBounds': {u'a': [[2, 2]]},
 u'indexOnly': False,
 u'isMultiKey': False,
 u'millis': 0,
 u'n': 1,
 u'nChunkSkips': 0,
 u'nYields': 0,
 u'nscanned': 1,
 u'nscannedObjects': 1}

There are several important things to note about the result here. The most important is the cursor type, BtreeCursor a_1. This means that it's using an index, and in particular the index named a_1 that we created above. If the field is not indexed, we get a different query plan from MongoDB:

In [19]: db.test.find({'b':2}).explain()
Out[19]: 
{u'allPlans': [{u'cursor': u'BasicCursor', u'indexBounds': {}}],
 u'cursor': u'BasicCursor',
 u'indexBounds': {},
 u'indexOnly': False,
 u'isMultiKey': False,
 u'millis': 0,
 u'n': 1,
 u'nChunkSkips': 0,
 u'nYields': 0,
 u'nscanned': 3,
 u'nscannedObjects': 3}

Here, MongoDB is using a BasicCursor. For you SQL experts out there, this is equivalent to a full table scan, and is very inefficient. Note also that when we queried the indexed field, nscanned and nscannedObjects were both one, meaning that it only had to check one object to satisfy the query, whereas in the case of our unindexed field, we had to check every document in the collection.

MongoDB has an extremely fast query that it can use in some cases where it doesn't have to scan any objects, only the index entries. This happens when the only data you're returning from a query is part of the index:

In [36]: db.test.find({'a':2}, {'a':1, '_id':0}).explain()
Out[36]: 
...
 u'indexBounds': {u'a': [[2, 2]]},
 u'indexOnly': True,
 u'isMultiKey': False,
...

Note here the indexOnly field is true, specifying that MongoDB only had to inspect the index (and not the actual collection data) to satisfy the query.

Another thing that's nice about the MongoDB index system is that it can use compound indexes (indexes that include more than one field) to satisfy some queries. In this case, you should specify the direction of each field since MongoDB stores indexes as B-trees. An illustration is probably best. First, we'll drop our a_1 index and ensure a new index on a and b, both ascending:

In [44]: db.test.drop_index('a_1')

In [45]: db.test.ensure_index([('a', 1), ('b', 1)])
Out[45]: u'a_1_b_1'

Now, let's see what happens when we query by just a:

In [55]: db.test.find({'a': 2}).explain()
Out[55]: 
...
 u'cursor': u'BtreeCursor a_1_b_1',
 u'indexBounds': {u'a': [[2, 2]],
  u'b': [[{u'$minElement': 1}, {u'$maxElement': 1}]]},
...

MongoDB's optimizer here is "smart" enough to use the compound (a,b) index to query just the a value. What if we query just the b value?

In [56]: db.test.find({'b': 2}).explain()
Out[56]: 
{u'allPlans': [{u'cursor': u'BasicCursor', u'indexBounds': {}}],
 u'cursor': u'BasicCursor',
...

Oops! That doesn't work because the index is sorted with keys (a,b). Key order also becomes important when we want to sort our results:

In [64]: db.test.find().sort([ ('a', 1), ('b', 1)]).explain()
Out[64]: 
...
 u'cursor': u'BtreeCursor a_1_b_1',
 u'indexBounds': {u'a': [[{u'$minElement': 1}, {u'$maxElement': 1}]],
  u'b': [[{u'$minElement': 1}, {u'$maxElement': 1}]]},
...

In [65]: db.test.find().sort([ ('a', 1), ('b', -1)]).explain()
Out[65]: 
{u'allPlans': [{u'cursor': u'BasicCursor', u'indexBounds': {}}],
 u'cursor': u'BasicCursor',
...

So if we sort in the same order as our index, we can use the B-tree index to sort the results. If we sort in a different order, we can't, so MongoDB has to actually load the entire result set into RAM and sort it there. (MongoDB can actually use an index for the exact reverse of our sort order as well, so [(a, -1), (b, -1)] would have worked just fine.) In a collection of 3 documents, this isn't important, but as your data grows, this can become quite slow, in some cases actually returning an error because the result set is too large to sort in RAM.

Deleting data

Deleting data in MongoDB is fairly straightforward. All you need to do is to pass a query to the remove() method on the collection, and MongoDB will delete all documents from the collection that match the query. (Note that deletes can still be slow if you specify the query inefficiently, as they use indexes just like queries do.)

In [72]: list(db.testq.find())
Out[72]: 
[{u'_id': ObjectId('4f25c89beb033049af000009'),
  u'a': {u'b': 1},
  u'c': [{u'd': 1}, {u'd': 2}, {u'd': 3}]},
 {u'_id': ObjectId('4f25c89beb033049af00000a'),
  u'a': {u'b': 2},
  u'c': [{u'd': 2}, {u'd': 3}, {u'd': 4}]},
 {u'_id': ObjectId('4f25c89beb033049af00000b'),
  u'a': {u'b': 3},
  u'c': [{u'd': 3}, {u'd': 4}, {u'd': 5}]}]

In [73]: db.testq.remove({'a.b': {'$gt': 1}})

In [74]: list(db.testq.find())
Out[74]: 
[{u'_id': ObjectId('4f25c89beb033049af000009'),
  u'a': {u'b': 1},
  u'c': [{u'd': 1}, {u'd': 2}, {u'd': 3}]}]

Updating data

In many cases, an update in MongoDB is as straightforward as calling save() on a python dict:

In [76]: doc = db.testq.find_one({'a.b': 1})

In [77]: doc['a']['b'] = 4

In [78]: db.testq.save(doc)
Out[78]: ObjectId('4f25c89beb033049af000009')

In [79]: list(db.testq.find())
Out[79]: 
[{u'_id': ObjectId('4f25c89beb033049af000009'),
  u'a': {u'b': 4},
  u'c': [{u'd': 1}, {u'd': 2}, {u'd': 3}]}]

save() can also be used to insert documents if they don't exist yet (this check is done by checking the dict to be saved for the _id key).

Unlike some other NoSQL solutions, MongoDB allows you to do quick, in-place updates of documents using special [modifier operations][mongodb-modifier]. For instance, to set a field to a particular value, you can do the following:

In [81]: db.testq.update({'a.b': 4}, {'$set': {'a.b': 7}})

In [82]: list(db.testq.find())
Out[82]: 
[{u'_id': ObjectId('4f25c89beb033049af000009'),
  u'a': {u'b': 7},
  u'c': [{u'd': 1}, {u'd': 2}, {u'd': 3}]}]

MongoDB provides several different modifiers you can use to update documents in place, including the following (for more details see updates):

$inc Increment a numeric field (generalized; can increment by any number)
$set Set certain fields to new values
$unset Remove a field from the document
$push Append a value onto an array in the document
$pushAll Append several values onto an array
$addToSet Add a value to an array if and only if it does not already exist
$pop Remove the last (or first) value of an array
$pull Remove all occurrences of a value from an array
$pullAll Remove all occurrences of any of a set of values from an array
$rename Rename a field
$bit Bitwise updates

There's a lot more that I could cover, but hopefully that whets your appetite to learn more about MongoDB. In future posts, I'll discuss how to use GridFS (a "filesystem" on top of MongoDB) and MongoDB's various aggregation options, as well as how you can use Ming to simplify certain operations.

Getting Started with MongoDB and Python

2012-01-18T13:07:00.000-05:00

If you've been following this blog for a while, you've seen me mention MongoDB more than once. One exciting thing for me is that I'll be co-teaching a tutorial at PyCon this year on Python and MongoDB that will cover MongoDB, PyMongo, and Ming. So to hopefully whet your appetite for learning more at the tutorial, I thought I'd write a few posts covering MongoDB, PyMongo, and Ming from a beginner's perspective.

What is MongoDB?

From MongoDB.org:

MongoDB (from "humongous") is a scalable, high-performance, open source NoSQL database.

Well, that's not all that enlightening, so I'll expand a bit here on MongoDB's features...

MongoDB is a document database

MongoDB is a document database, which means that instead of storing "rows" in "tables" like you do in a relational database, you store "documents" in "collections." Documents are basically JSON objects (technically BSON. This is to be distinguished from other NoSQL-type databases such as key-value stores (e.g. Tokyo Cabinet), column family stores (e.g. Cassandra) or column stores (e.g. MonetDB).

MongoDB has a flexible query language

This is one thing that makes MongoDB a pleasure to work with, particularly if you come from another NoSQL database where querying is either restrictive (key-value stores which can only be queried by key) or cumbersome (something like CouchDB that requires you to write a map-reduce query). MongoDB has a BSON-based query language that's a bit more restrictive than SQL, that you can still use to get a lot done.

Here's an example of a simple MongoDB query that we use at SourceForge to find all the blog posts for a project:

blog_post.find({'state':'published','app_config_id':{'$in':app_config_ids}})

There are also several other operators like '$lt', '$nin', '$not', and '$or' that allow you to construct quite complex queries, though you are somewhat restricted from what you can do in SQL (even with a single table).

MongoDB is fast and scalable

A single MongoDB node is able to comfortably serve 1000s of requests per second on cheap hardware. When you need to scale beyond that, you can use either replication (keeping several copies of the data on different servers) or sharding (partitioning the data across servers). MongoDB even includes logic to automatically load-balance your shards as your database and load increase.

Getting Started with MongoDB

While MongoDB is fairly straightforward to install on (64-bit) systems, there are also a couple of companies that provide a free tier of MongoDB hosting, MongoLab and MongoHQ that are great for getting started. I've been using, for no particular reason, MongoLab for my own things and I can recommend them, and it's what I have experience with, so that's what I'll cover here.

Let's assume you sign up for a MongoLab account. Once you've done this, you can create a database using their web-based control panel and click on it, you'll note the connection info at the top of the page:

(Your server name and port number may be different.) At this point, most tutorials would tell you to install and launch the 'mongo' command-line tool to begin exploring your database. We'll skip that here and use the python driver PyMongo directly. I like to use virtualenv myself and ipython, so that's the approach I'll take here:

$ virtualenv mongo
... install messages ...
$ source mongo/bin/activate
(mongo) $ pip install pymongo ipython
... install messages ...
(mongo) $ ipython
... banner message ...

Now that we're in ipython, we'll go ahead and connect to the database and create a document.

In [1]: import pymongo

In [2]: conn = pymongo.Connection('mongodb://tutorial-test:u3ZYh136@ds029187.mongolab.com:29187/tutorial-test')

In [3]: db = conn['tutorial-test']

In [4]: db.test_collection.insert({})
Out[4]: ObjectId('4f16f5c7eb03306a92000000')

In [5]: db.test_collection.find()
Out[5]: <pymongo.cursor.Cursor at 0x7fbb9006f350>

In [6]: list(db.test_collection.find())
Out[6]: 
[{u'_id': ObjectId('4f16f5c7eb03306a92000000')}]

Well, that's it for now. I'll be posting several followup articles in this series that will go into more detail on how to do various queries and updates using PyMongo, the MongoDB python driver, as well as how to effectively use Ming, so stay tuned!

MongoDB's Write Lock

2011-12-31T15:22:00.008-05:00

MongoDB, as some of you may know, has a process-wide write lock. This has caused some degree of ridicule from database purists when they discover such a primitive locking model. Now per-database and per-collection locking is on the roadmap for MongoDB, but it's not here yet. What was announced in MongoDB version 2.0 was locking-with-yield. I was curious about the performance impact of the write lock and the improvement of lock-with-yield, so I decided to do a little benchmark, MongoDB 1.8 versus MongoDB 2.0.

Page Faults and Working Sets

MongoDB uses the operating system's virtual memory system to handle writing data back to disk. The way this works is that MongoDB maps the entire database into the OS's virtual memory and lets the OS decide which pages are 'hot' and need to be in physical memory, as well as which pages are 'cold' and can be swapped out to disk. MongoDB also has a periodic process that makes sure that the database on disk is sync'd up with the database in memory. (MongoDB also includes a journalling system to make sure you don't lose data, but it's beyond the scope of this article.)

What this means is that you have very different behavior depending on whether the particular page you're reading or writing is in memory or not. If it is not in memory, this is called a page fault and the query or update has to wait for the very slow operation of loading the necessary data from disk. If the data is in memory (the OS has decided it's a 'hot' page), then an update is literally just writing to system RAM, a very fast operation. This is why 10gen (the makers of MongoDB) recommend that, if at all possible, you have sufficient RAM on your server to keep the entire working set (the set of data you access frequently) in memory at all times.

If you are able to do this, it turns out that the global write lock really doesn't affect you. Blocking reads for a few nanoseconds while a write completes turns out to be a non-issue. (I have not measured this, but I suspect that the acquisition of the global write lock takes significantly longer than the actual write.) But this is all kind of hand-waving at this point. I did end up measuring what happens to your query rate as you do more and more (non-faulting) writes and found very little effect.

2.0 to the Rescue?

So the real problem happens when you're doing a write, you acquire the lock, and then you try to write and fault while holding the lock. Since a page fault can take around 40,000 times longer than a nonfaulting memory operation, this obviously causes problems. In MongoDB version 2.0 and higher, this is addressed by detecting the likelihood of a page fault and releasing the lock before faulting. This allows other reads and writes to proceed while the data is brought in from disk. This is what I'm really trying to measure here; what is the effect of a so-called "write lock with yield" as opposed to a plain old "write lock?"

Benchmark Setup

First off, I didn't want to worry about all the stuff I normally run on my laptop to get in the way, and I'd like to make the benchmark as reproducible as possible, so I decided to run it on Amazon EC2. (I have made the code I used available here - particularly fill_data.py, lib.py, and combined_benchmark.py if you'd like to try to replicate the results. Or you can just grab my image ami-63ca1e0a off AWS.) I used an m1.large instance size instance running Ubuntu server with ephemeral storage used for the database store. I also reformatted the ephemeral disk from ext3 to ext4 and mounted it with noatime for performance purposes. MongoDB is installed from the 10gen .deb repositories as version 1.8.4 (mongodb18-10gen) and 2.0.2 (mongodb-10gen).

The database used in the test contains one collection of 150 million 112-byte documents. This was chosen to give a total size significantly in excess of the 7.5 GB of physical RAM allocated to the m1.large instances. (The collection size with its index takes up 21.67 GB in virtual memory). Journalling has been turned *off* in both 1.8 and 2.0.

What I wanted to measure is the effect of writes on reads that are occurring simultaneously. In order to simulate such a load, the Python benchmark code uses the gevent library to create two greenlets. One is reading randomly from its working set of 10,000 documents as fast as possible, intended to show non-faulting read performance. The second greenlet attempts to write to a random document of the 150 million total documents at a specified rate. I then measure the achieved write and read rate and create a scatter plot of the results.

Results

First, I wanted to show the effect of non-faulting writes on non-faulting reads. For this, I restrict writes to occur only in the first 10,000 documents (the same set that's being read). As expected, the write lock doesn't degrade performance much, and there's little difference between 1.8 and 2.0 in this case.

The next test shows the difference between 1.8 and 2.0 in fairly stark contrast. We can see an immediate drop-off in read performance in the presence of faulting writes, with virtually no reads getting through when we have 48 (!) writes per second. In contrast, 2.0 is able to maintain around 60% of its peak performance at that write level. (Going much beyond 48 faulting writes per second makes things difficult to measure, as you're using a significant percentage of the disk's total capacity just to service your page faults).

Conclusion

The global write lock in MongoDB will certainly not be mourned when it's gone, but as I've shown here, its effect on MongoDB's performance is significantly less than you might expect. In the case of non-faulting reads and writes (the working set fits in RAM), the write lock degrades performance only slightly as the number of writes increases, and in the presence of faulting writes, the lock-with-yield approach of 2.0 mitigates most of the performance impact of the occasional faulting write.

MongoDB Rocks My World

2011-10-31T12:30:00.000-04:00

I am delighted to announce that I'll be speaking at several MongoDB-related events over the next couple of months. So if you're in or near Dallas, Seattle, or Silicon Valley, I'd love it if you could make it to one of my talks. 10gen (the makers of MongoDB) do a great job of putting on regional conferences that are extremely reasonably priced (typically $30 student and $50 early bird), and there's usually a nice after-party for each one, so there's no excuse not to go.

So anyway, here are the conferences I'll be speaking at:

November 17: MongoDallas
December 1: MongoSeattle
December 9: MongoSV (Santa Clara, CA)

I also wanted to say a few words on why I enjoy using MongoDB and why it is such a good fit for a lot of the problems I face...

MongoDB is Flexible

You probably expected me to say something about MongoDB being web-scale, and while that's true, it's not the biggest reason I use it. What I like the most is that you can organize your data the way you want to without a lot of restrictions forced on you by the DBMS you're using. For those who don't know, MongoDB is what's called a 'document-oriented' database. Rather than storing "rows" in "tables" like you do in a relational database, you store "documents" in "collections."

What is a document? Well, in MongoDB is basically a JSON object. (I'm actually lying. It's technically called BSON because it's a binary format and you get a couple of extra data types for 'free' in MongoDB including datetimes, ObjectIds, and regular expressions, but it's basically JSON.) So if your RDBMS-based application is doing some queries, and every query has a one-to-many type join (or worse yet you're doing several queries to grab all the data you need to render a page), in MongoDB you would just store all the data you need in a single document, which can be retrieved by a single query. Fast and flexible.

Another way MongoDB is flexible is that it allows you to use ad-hoc queries and indexes. In some NoSQL solutions, you need to define views or map-reduce jobs to get at your data. Rather than forcing you to define all that up-front (or pay a penalty at runtime), MongoDB includes a BSON-based query language and indexes. What this means is that you don't have to clutter your data model with every possible path someone might want to use someday to access your data. For me, MongoDB strikes a sweet spot between explicit query models like Hadoop or CouchDB and a full query compiler like SQL databases.

MongoDB is Fast

Ok, so there is a little bit about "web-scale." MongoDB supports both replication and sharding, which means that it scales really well over several servers. But that's not really what I'm talking about. What I'm talking about it single-server speed. MongoDB uses a nifty little trick (mmap) to map its entire database into memory, where it can treat reads and writes to the database as reads and writes to RAM. For many common operations, this is really fast. Like tens of thousands of writes per second fast. This means that for a site like SourceForge, which is getting millions of hits a day, we're comfortable serving the majority of requests off a single MongoDB server. And by setting up replication (which we have now) and journalling (which we plan on adding Real Soon Now) with MongoDB, you can be quite safe as well. Replication also means that backups are a snap and don't require our server to go through any downtime.

So great, it's fast in production. But it's also fast (and easy to set up) for development. The MongoDB server is just a single binary application that you can run yourself from the command line, and it keeps all its data in a single directory. This is really nice for a development setup like mine, where I'm running everything needed for an Allura (the SF.net platform) on a VMWare virtual machine. It's just plain simple to set up a single node, and setting up larger systems with replication and sharding isn't *that* much harder.

MongoDB is Simple

While there are many parts of MongoDB that I'm sure took a good bit of complexity in the server to get right (shard migration, journalling, and the new aggregation framework come to mind), the basic ideas are easy to get your mind around. Some of the other NoSQL solutions do some really mind-stretchingly awesome stuff (Cassandra's write scaling and CouchDB's multi-master replication come to mind). MongoDB has some nice and interesting features, as well, but its real strength is that it does what it does really well. It's fast (performance wise and development wise), flexible, stable, and scalable. And because a) the 10gen engineers are awesome and b) it's open source, when there is a feature or technique enough people want/need, MongoDB grows that feature. But historically, it has grown that feature in a way that fits in really well with the existing feature set and doesn't require an enormous cognitive leap to grasp.

10gen

MongoDB's creators are an awesome group of folks. They're throwing conferences every 6 weeks or more frequently somewhere, they're incredibly responsive to support questions, they're actually really smart, and they've developed an incredible community around MongoDB. And just for fun, they've decided to single-handedly support the coffee mug industry. Seriously, though, the support they provide for MongoDB is nothing short of stellar, from n00b to Mongo master, and that's probably the biggest reason MongoDB has seen the kind of uptake it has.

So if you haven't ever had the chance to use MongoDB, I'd urge you to grab a copy and start playing around with it. And if you're curious, try to make it out to one of the MongoDB conferences or user groups. All the folks I've met at these are quite friendly and love to help newer users along. So yeah, I guess I'm a MongoDB fanboy. Think I've gone off the deep end? Any other features of MongoDB you love? Please let me know in the comments. I'd love to hear from you!

MongoDB Atlanta User Group

2011-08-10T21:25:00.003-04:00

So last night I had the opportunity to speak about Zarkov at the Atlanta MongoDB user group (not to be confused with the MongoATL conference). It was a good crowd, and we had a good time munching on pizza beforehand and talking about MongoDB afterward. Anyway, I did a little realtime demo combining some stuff I've been doing with Socket.io, Zarkov, and Flot and promised I'd make the code available. A mostly-functional version is available at http://sf.net/u/rick446/mongodb-atl/.

Also, I figured I'd use this post to collect all the presentation paraphernalia from last night, so below are the slides and somewhat poor quality video I recorded. Hope you manage to enjoy it anyway!

Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ

View more presentations from Rick Copeland

Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ

WebSockets to Socket.io

2011-08-07T16:46:00.000-04:00

In a previous blog post, I showed how you can use Gevent, ZeroMQ, WebSockets, and Flot to create a nice asynchronous server that graphs events in real time to multiple web consumers. Unfortunately, Chrome is just about the only browser that my demo worked in due to some issues with the WebSockets standard. So I've updated the example to use Socket.io, a WebSockets-like Javascript library that will work across multiple browsers.

Prerequisites

The best place to start is my last post; you'll need all the libraries I mentioned there. In addition, you need to install gevent-socketio for Socket.io support in gevent.

Changes from WebSockets server

The first thing we need to change is we need to create a SocketIOServer to replace our WSGIServer with a WebSocketHandler. While we're at it, we'll go ahead and merge our little static file server with the SocketIOServer. Our new main method now looks like this:

def main(): 
    '''Set up zmq context and greenlets for all the servers, then launch the web 
    browser and run the data producer''' 
    context = zmq.Context() 
 
    # zeromq: tcp to inproc gateway 
    gevent.spawn(zmq_server, context) 
    # WSGI server: copies inproc zmq messages to websocket 
    sio_server = SocketIOServer( 
        ('', 8000), SocketIOApp(context), 
        resource="socket.io") 
    # Start the server greenlets 
    sio_server.start() 
    # Open a couple of webbrowsers 
    webbrowser.open('http://localhost:8000/graph.html') 
    webbrowser.open('http://localhost:8000/graph.html') 
    # Kick off the producer 
    zmq_producer(context)

Now let's take a look at our SocketIOApp:

class SocketIOApp(object): 
    '''Funnel messages coming from an inproc zmq socket to the socket.io''' 
 
    def __init__(self, context): 
        self.context = context 
        self.static_app = paste.urlparser.StaticURLParser( 
            os.path.dirname(__file__)) 
 
    def __call__(self, environ, start_response): 
        if not environ['PATH_INFO'].startswith('/socket.io'): 
            return self.static_app(environ, start_response) 
        socketio = environ['socketio'] 
        sock = self.context.socket(zmq.SUB) 
        sock.setsockopt(zmq.SUBSCRIBE, "") 
        sock.connect('inproc://queue') 
        while True: 
            msg = sock.recv() 
            socketio.send(msg)

What we've done here is overlay the StaticURLParser provided by paste with our own little ZeroMQ-to-Socket.io gateway. So if a request hits any path starting with /socket.io, it gets routed to our gateway. Otherwise it gets handled as a request for a static resource. We could have had a separate WSGIServer dedicated to handling static requests, but it seemed simpler (to me) to go ahead and combine them here.

Client Updates

Now, for our client, we've made a couple of changes as well. For one, we need to include the socket.io.js library. I've included the one from the socket.io CDN to simplify development:

<script src="http://cdn.socket.io/stable/socket.io.js"></script>

In our javascript, I've also made a few changes to handle the Socket.io protocol better as well as make the graph update a little less choppy. First, we'll set up our socket and some basic variables we'll use later:

$(function() { 
    socket = new io.Socket('localhost'); 
    socket.connect(); 
 
    var $placeholder = $('#placeholder'); 
    var datalen = 100; 
    var plot = null; 
    var series = { 
        label: "Value", 
        lines: { 
            show: true, 
            fill: true 
        }, 
        data: [] 
    };

Next, we need to handle the protocol events from socket.io to update our status:

socket.on('connect', function() { 
        $('#conn_status').html('<b>Connected: ' + socket.transport.type + '</b>'); 
    }); 
    socket.on('error', function() { 
        $('#conn_status').html('<b>Error</b>'); 
    }); 
    socket.on('disconnect', function() { 
        $('#conn_status').html('<b>Closed</b>'); 
    });

Finally, we will handle the actual messages coming from the server:

socket.on('message', function(msg) { 
        var d = $.parseJSON(msg); 
        series.data.push([d.x, d.y]); 
        while (series.data.length > datalen) { 
            series.data.shift(); 
        } 
        if(plot == null && series.data.length > 10) { 
            plot = $.plot($placeholder, [series], { 
                xaxis:{ 
                    mode: "time", 
                    timeformat: "%H:%M:%S" 
                }, 
                yaxis: { min: 0, max: 5 }, 
                hooks: { 
                    draw: function(plot, canvascontext) { 
                        // Redraw the graph in 50ms 
                        setTimeout(function() { 
                            plot.setData([series]); 
                            plot.setupGrid(); 
                            plot.draw();}, 50); 
                    } 
                } 
            }); 
        } 
    }); 
});

What we're doing here is pushing the value from the server on an array, shifting off the old values. Then we set up a hook for Flot to redraw the graph every 50ms when drawing completes. Altogether, there aren't too many changes from our WebSockets demo, and that demo itself wasn't too complex. If you'd like to see all the code, you can at my SourceForge Repo. Hope you've enjoed these posts as much as I've enjoyed playing around with these libraries. Happy hacking!

Extending Zarkov's Map/Reduce

2011-07-26T15:04:00.001-04:00

Since I've been working on Zarkov, I've been writing a few
map/reduce jobs. One of the things I noticed about map/reduce, at
least as I've implemented it in Zarkov, is that it's pretty
inefficient if you want t o generate several aggregate views of the
same data (like say an event stream). In order to meet Zarkov's
performance requirements without making my operations team
too angry, I decided to "extend" map/reduce, as it's
implemented in Zarkov. I'm not terribly creative, so I called this
command xmapreduce. Here' I'll briefly describe
map/reduce, show the problem with it, and explain the solution
implemented in Zarkov.

Map/Reduce

The map/reduce algorithm is nice because a) it's
extremely scalable and b) it's pretty easy to grasp the basics. The
idea is that you write two functions, map and
reduce which will be applied to a large dataset to get
yourself some interesting data. For our purposes here, I'll
illustrate using Zarkov map/reduce.
Let's say you want to see how many timestamped objects in some
collection exist for each date. Here's how you'd write your Zarkov
map/reduce functions:

def map(objects): 
    for object in objects: 
        yield object['timestamp'].date(), 1 
 
def reduce(key, values): 
    return sum(values)

The map/reduce framework will take these functions and do
(basically) the following (but efficiently distributed over several
workers):

import operator, itertools 
 
def map_reduce(input_collection, query, output_collection, map, reduce): 
    objects = input_collection.find(query) 
    map_results = list(map(objects)) 
    map_results.sort(key=operator.itemgetter(0)) 
    for key, kv_pairs in itertools.groupby(map_results, operator.itemgetter(0)): 
        value = reduce(key, [ v for k,v in kv_pairs ]) 
        output_collection.save({"_id":key, "value":value})

And that's pretty much all there is to it (conceptually).

The Problem with Map/Reduce

The problem arises when you have several output collections that
all depend on a single input collection (or even depend on the same
documents in the input collection). Map/reduce can get you
there, bit it'll execute the query once for each output collection,
and it needs to distribute the data to the map()
functions once for every output collection. This isn't too
efficient.
Let's extend our example above by tracking the date, month, and
year of each object. In classic map/reduce, we'd need three
map functions:

def map_date(objects): 
    for object in objects: 
        yield object['timestamp'].date(), 1 
 
def map_month(objects): 
    for object in objects: 
        yield object['timestamp'].date().replace(day=1), 1 
 
def map_year(objects): 
    for object in objects: 
        yield object['timestamp'].date().replace(month=1,day=1), 1 
 
def reduce(key, values): 
    return sum(values)

Now, if we treat these three jobs separately, we get a lot of
duplicated effort. Particularly, Zarkov must query the input
collection three times and transfer the data to map workers three
times. (There's also duplicate serialization/deserialization of the
input objects, though the Python bson library is quite
fast.)

XMapReduce to the Rescue

The solution Zarkov uses for this case is to allow the
map function to return a target collection
along with key and value. What xmapreduce does is
transform the map functions above into one xmap
function, and tweaking our reduce function just a bit
to take the collection as an input:

def xmap(objects): 
    for object in objects: 
        yield { 
            'c':'by_date', 
            'k':object['timestamp'].date(), 
            'v':1 } 
        yield { 
            'c':'by_month', 
            'k':object['timestamp'].date().replace(day=1), 
            'v':1 } 
        yield { 
            'c':'by_year', 
            'k':object['timestamp'].date().replace(day=1, month=1), 
            'v':1 } 
 
def xreduce(collection, key, values): 
    return sum(values)

The xmapreduce algorithm is just a slight bit more complex than
the map/reduce algorithm:

import operator, itertools 
 
def xmap_reduce(input_collection, query, output_db, map, reduce): 
    objects = input_collection.find(query) 
    map_results = list(map(objects)) 
    keyfunc = lambda doc:(doc['c'], doc['k']) 
    map_results.sort(key=keyfunc) 
    for (c,k), docs in itertools.groupby(map_results, keyfunc): 
        value = reduce(c,k, [ doc['v'] for doc in docs ])span> 
        output_db[c].save({"_id":key, "value":value})

Now, we can invoke a single job to calculate all three
aggregates, with close to a 3x reduction in data transfer and a
significant speedup. Assuming that a) your jobs can be
combined into a single xmapreduce job and b) you find it acceptable
to code the target collection in your map jobs, xmapreduce should
give you a significant speedup over classic map/reduce. In our
case, we started with 12 map/reduce jobs and ended up with 4
xmapreduce jobs, with a tremendous speedup, but of course your
mileage may vary. Happy hacking!

Zarkov is a Web Service

2011-07-25T12:51:00.000-04:00

Over the past few weeks I've been working on a service in Python that I'm calling, in the tradition of naming projects after characters in Flash Gordon, Zarkov. So what exactly is Zarkov? Well, Zarkov is many things (and may grow to more):

Zarkov is an event logger
Zarkov is a lightweight map-reduce framework
Zarkov is an aggregation service
Zarkov is a webservice

In my first post, I discussed Zarkov as an event logger. Next, I explored Zarkov as a map-reduce framework. After that, I talked about using Zarkov's aggregation definitions. Today, I'll focus on how you can get those aggregates out of Zarkov using Zarkov's built-in JSON web service.

Basic usage

In order to expose Zarkov's webservice, you'll need to add a key to your development.yaml telling it what port to listen on:

web_port: 8081

Once that's set up, you need to start the Zarkov server:

(zarkov) $ zcmd -y development.yaml serve

Now we need some data to be put into Zarkov. Luckily, Zarkov comes with a 'perftest' command that we can use to fill the data:

(zarkov) $ zcmd -y development.yaml init
(zarkov) $ # zcmd perftest <url> <num_events> <concurrency> <rate> 
(zarkov) $ zcmd -y development.yaml perftest tcp://localhost:6543 1000 1 100
(zarkov) $ zcmd -y development.yaml aggregate perftest

Now you can request data from Zarkov by using simple HTML form requests, receiving the results in JSON:

(zarkov) $ curl 'http://127.0.0.1:8081/q?p=perftest.&c=sum_by_second' 
{"v": [["00", 100], ["01", 101], ["02", 101], ["03", 101], ["04", 101], ["05", 101], ["06", 101], ["07", 101], ["08", 101]]} 
(zarkov) $ curl 'http://127.0.0.1:8081/q?p=perftest.&c=sum_by_second&b=03&e=07' 
{"v": [["03", 101], ["04", 101], ["05", 101], ["06", 101], ["07", 101]]}

Integrating with Flot

The data format returned by the Zarkov web service is a list of (key, value) pairs for every value calculated in the aggregate. If you compute your aggregates using ISO timestamps as part of the key, Zarkov will helpfully convert these timestamps to Flot-acceptable timestamps for graphing:

(zarkov) $ curl 'http://127.0.0.1:8081/q?p=perftest.&c=sum_by_second_ts' 
{"v": [[1311626220000.0, 100], [1311626221000.0, 101], [1311626222000.0, 101], [1311626223000.0, 101], [1311626224000.0, 101], [1311626225000.0, 101], [1311626226000.0, 101], [1311626227000.0, 101], [1311626228000.0, 101]]}

In order to use the automatic timestamp feature, you'll need to make sure that your key ends with an ISO formatted datetime prefixed by a slash ("/") character. You can set the definition of the perftest task in zarkov.helpers to see how this can be done. Also note that you can specify a prefix argument to your query. This is prepended onto the collection name. If it's not provided, Zarkov will assume you mean the zagg. prefix.

To demonstrate the flot integration, let's set up a tiny webapp in /tmp/zarkov. First we'll create the directory structure and grab a copy of flot:

(zarkov) $ mkdir /tmp/zarkov
(zarkov) $ mkdir /tmp/zarkov/static
(zarkov) $ pushd /tmp/zarkov/static
(zarkov) $ wget http://flot.googlecode.com/files/flot-0.7.tar.gz
(zarkov) $ tar xzf flot-0.7.tar.gz

Now we'll create an index page in /tmp/zarkov/index.html:

<html> 
  <head> 
    <title>Zarkov Graph Test</title> 
  </head> 
   <div id="placeholder" style="width:600px;height:300px"></div> 
    <script type="text/javascript" src="flot/jquery.min.js"></script> 
    <script type="text/javascript" src="flot/jquery.flot.min.js"></script> 
  <!--[if lte IE 8]><script language="javascript" type="text/javascript" src="/static/flot/excanvas.min.js"></script><![endif]--> 
  <script type="text/javascript" src="index.js"></script> 
  </body> 
</html>

and the corresponding index.js:

(function() { 
    var $placeholder = $('div#placeholder'); 
    var plot_options = { 
        xaxes: [ {mode:'time' }]}; 
    var params = { 
        p: "perftest.", c:"sum_by_second_ts" }; 
    $.getJSON( 
        '/ws/q', 
        $.param(params), 
        function(result) { 
            var series = []; 
            $.each(result, function(key, value) { 
                series.push( {label:key, data:value } ); 
            }); 
            $.plot($placeholder, series, plot_options); 
        }); 
}());

And here's our little webapp. You'll need to pip install paste if you don't already have it.

from gevent.pywsgi import WSGIServer 
from paste.urlparser import StaticURLParser 
from paste.urlmap import URLMap 
from zarkov.web import WebApp 
 
app = URLMap() 
app['/static'] = StaticURLParser('/tmp/zarkov/static') 
app['/ws'] = WebApp() 
 
web_server = WSGIServer(('', 8081), app) 
web_server.serve_forever()

Now if we run the webapp with the Zarkov script command and visit http://localhost:8081/static/index.html, we can see a pretty graph:

(zarkov) $ zcmd -y development.yaml script /tmp/zarkov/test.py

Hope you've enjoyed this brief intro to Zarkov. If you've got any questions, comments, or suggestions please let me know either in the comments, Zarkov's discussion forums, or Zarkov's ticket tracker (all conveniently located at http://sf.net/p/zarkov). Happy hacking!

Gevent, ZeroMQ, WebSockets, and Flot FTW!

2011-07-23T16:17:00.002-04:00

As part of the work I've been doing on Zarkov I've had the opportunity to play around with a lot of cool technologies, among which are gevent, ZeroMQ, WebSockets, and Flot. It took a while to get to the point where I could actually get things done, but once I was there, things were incredibly simple. In this post I'll show you how to use these three techologies together to build a simple web app with realtime server push data.

Prerequisites

Before you can run the example program, you'll need to install the following Python libraries:

pyzmq
gevent
gevent-zeromq
gevent-websocket
paste (for a simple WSGI http static file server)

The easiest way to do this is to:

pip install pyzmq gevent gevent_zeromq paste

You may see messages about missing system libraries; you'll need to have libzmq-dev 2.1+ and libevent-dev installed to build the Python packages.

You'll also need to download Flot JQuery plugin and untar it in your project directory.

Greenlets

One of the things you'll need to understand before running the code is the concept of green threads or greenlets. These behave just like threads except for two caveats:

they never run concurrently with one another
they only yield to one another at specified points

What this ends up meaning is that they're very lightweight to create compared to threads and you can actually be a little sloppier with shared data between greenlets since you can deterministically know when a greenlet might yield to another greenlet. The gevent library provides our greenlet support, and makes greenlets even more useful by turning blocking system calls into yield points so that another greenlet can run while one is waiting on data.

ZeroMQ

ZeroMQ sounds like it's a message queue like RabbitMQ, but it's really not. ZeroMQ is a library that provides many message queue-like features, but requires no broker and very little setup. We'll be using ZeroMQ pub/sub sockets to manage communication in this demo. Pub/sub sockets have the nice feature that published messages get distributed to all subscribers currently connected, or if no subscribers are connected, the message simply gets dropped.

ZeroMQ was designed to work with threads rather than greenlets, so we'll also need to use the excellent gevent_zeromq library to "greenletize" ZeroMQ. For this example, we'll write a loop that pushes (x,y) point values out on a zmq.PUB socket:

def zmq_producer(context): 
    '''Produce a nice time series sine wave''' 
    socket = context.socket(zmq.PUB) 
    socket.connect('tcp://127.0.0.1:5000') 
 
    while True: 
        x = time.time() 
        y = 2.5 * (1 + math.sin(x / 500)) 
        socket.send(json.dumps(dict(x=x, y=y))) 
        gevent.sleep(0.1)

The context being passed in is created in our main() function with the call zmq.Context(). The context is just a place for ZeroMQ to store some global state; you generally only create one per application.

OK, now that we have our producer, let's look at the ZeroMQ server. What we'll do here is just relay messages received on an incoming tcp zmq.SUB socket and publish them on an outgong inproc zmq.PUB socket:

def zmq_server(context): 
    '''Funnel messages coming from the external tcp socket to an inproc socket''' 
    sock_incoming = context.socket(zmq.SUB) 
    sock_outgoing = context.socket(zmq.PUB) 
    sock_incoming.bind('tcp://*:5000') 
    sock_outgoing.bind('inproc://queue') 
    sock_incoming.setsockopt(zmq.SUBSCRIBE, "") 
    while True: 
        msg = sock_incoming.recv() 
        sock_outgoing.send(msg)

The only thing to note here is that we have to tell ZeroMQ that we want to subscribe to all messages on the zmq.SUB socket.

Gevent WSGI Servers

Gevent provides a fast implementation of the WSGI standard in the gevent.pywsgi module. We'll be using two instances of the server (greenlets are lightweight) in this demo. One will simply serve up static files, while the other one will provide our WebSocket connectivity. Here's the code to set up these two servers:

# websocket server: copies inproc zmq messages to websocket 
ws_server = gevent.pywsgi.WSGIServer( 
    ('', 9999), WebSocketApp(context), 
    handler_class=WebSocketHandler) 
# http server: serves up static files 
http_server = gevent.pywsgi.WSGIServer( 
    ('', 8000), 
    paste.urlparser.StaticURLParser(os.path.dirname(__file__)))

Pretty simple; just specify the bind address for the server, the WSGI app to call, and in the case of the WebSocket server, the WebSocket handler class. The http_server above is mostly uninteresting, as it just serves up static files via paste. The ws_server has a (slightly) more interesting implementation:

class WebSocketApp(object): 
    '''Funnel messages coming from an inproc zmq socket to the websocket''' 
 
    def __init__(self, context): 
        self.context = context 
 
    def __call__(self, environ, start_response): 
        ws = environ['wsgi.websocket'] 
        sock = self.context.socket(zmq.SUB)/span> 
        sock.setsockopt(zmq.SUBSCRIBE, "") 
        sock.connect('inproc://queue') 
        while True: 
            msg = sock.recv() 
            ws.send(msg)

What's happening here is that when we get a connection to the websocket address (port 9999 in our example), we do the following:

Subscribe to the inproc socket that our zmq_server is publishing messages to
Grab the websocket from the environ
Relay messages from the zmq socket to the websocket

Client-side

So the server-side stuff is straightforward; what about the client-side? Well, it's pretty easy as well. Here's the (static) HTML page we'll use:

<html> 
    <head> 
        <title>ZWS Example</title> 
        <script type="text/javascript" src="/flot/jquery.min.js"></script> 
        <script type="text/javascript" src="/flot/jquery.flot.min.js"></script> 
        <script type="text/javascript" src="graph.js"></script> 
    </head> 
    <body> 
        <h1>ZMQ - WebSocket Example</h1> 
        <div id="conn_status">Not Connected</div> 
        <div id="placeholder" style="width:600px;height:300px;"></div> 
    </body> 
</html>

Here, all we're doing is pulling in the JQuery and Flot libraries as well as our custom graph.js and setting up a couple of placeholders. The Javascript is also pretty straightforward. I've tried to provide inline commentary:

$(function() { 
    // Open up a connection to our server 
    var ws = new WebSocket("ws://localhost:9999/"); 
    // Save our plot placeholder 
    var $placeholder = $('#placeholder'); 
    // Maximum # of data points to plot 
    var datalen = 100; 
    // This will be the plot object 
    var plot = null; 
    // Set up some options on our data series 
    var series = { 
        label: "Value", 
        lines: { 
            show: true, 
            fill: true 
        }, 
        points: { 
            show:true 
        }, 
        data: [] 
    }; 
    // What do we do when we get a message? 
    ws.onmessage = function(evt) { 
        var d = $.parseJSON(evt.data); 
        series.data.push([d.x, d.y]); 
        // Keep the data series a manageable length 
        while (series.data.length > datalen) { 
            series.data.shift(); 
        } 
        if(plot) { 
            // Create the plot if it's not there already 
            plot.setData([series]); 
            plot.setupGrid(); 
            plot.draw(); 
        } else if(series.data.length > 10) { 
            // Update the plot 
            plot = $.plot($placeholder, [series], { 
                xaxis:{ 
                    mode: "time", 
                    timeformat: "%H:%M:%S", 
                    minTickSize: [2, "second"], 
                }, 
                yaxis: { 
                    min: 0, 
                    max: 5 
                } 
            }); 
            plot.draw(); 
        } 
    } 
    // Just update our conn_status field with the connection status 
    ws.onopen = function(evt) { 
        $('#conn_status').html('<b>Connected</b>'); 
    } 
    ws.onerror = function(evt) { 
        $('#conn_status').html('<b>Error</b>'); 
    } 
    ws.onclose = function(evt) { 
        $('#conn_status').html('<b>Closed</b>'); 
    } 
});

Putting it all together

To put everything together, here's the main function I used:

def main(): 
    '''Set up zmq context and greenlets for all the servers, then launch the web 
    browser and run the data producer''' 
    context = zmq.Context() 
 
    # zeromq: tcp to inproc gateway 
    gevent.spawn(zmq_server, context) 
    # websocket server: copies inproc zmq messages to websocket 
    ws_server = gevent.pywsgi.WSGIServer( 
        ('', 9999), WebSocketApp(context), 
        handler_class=WebSocketHandler) 
    # http server: serves up static files 
    http_server = gevent.pywsgi.WSGIServer( 
        ('', 8000), 
        paste.urlparser.StaticURLParser(os.path.dirname(__file__))) 
    # Start the server greenlets 
    http_server.start() 
    ws_server.start() 
    # Open a couple of webbrowsers 
    webbrowser.open('http://localhost:8000/graph.html') 
    webbrowser.open('http://localhost:8000/graph.html') 
    # Kick off the producer 
    zmq_producer(context)

For fun, I threw a couple of webbrowser calls at the end so you can see the data getting distributed to all the clients that connect to our server. If you'd like to see the full program including flot et. al., here's a download. Hope you enjoy playing around with gevent, ZeroMQ, and WebSockets as much as I did!

Zarkov is an aggregation service

2011-07-22T10:55:00.003-04:00

Zarkov is an event logger
Zarkov is a lightweight map-reduce framework
Zarkov is an aggregation service
Zarkov is a webservice

In my first post, I discussed Zarkov as an event logger. Next, I explored Zarkov as a map-reduce framework. Today, I'll focus on Zarkov as an aggregation service suitable for doing near real-time analytics.

Aggregation Definitions

In Zarkov, aggregations are defined in the agg_def collection. Here's the Ming schema for that collection:

# Aggregations defined over (by default) the 'event' collection 
agg_def = collection( 
    'agg_def', doc_session, 
    Field('_id', S.ObjectId()), 
    # Name for the aggregation 
    Field('name', str), 
    # Input collection to aggregate 
    Field('object', str, if_missing='event'), 
    # Pointer to python Aggregation subclass 
    Field('task_name', str), 
    # Timestamp of last aggregation 
    Field('last_run', datetime, if_missing=datetime(1970,1,1))) 
 
class AggDef(object): 
    # some helper methods 
 
orm_session.mapper(AggDef, agg_def)

AggDefs basically define an input collection ('object'), a meaningful 'name', and a 'task_name ' to define the logic behind the aggregation. task_name is actually a dotted name that's suitable for use in Python's import statement that points to a 'Task' subclass. There is also a 'last_run' timestamp that is used to restrict which documents in the object collection (by default zarkov.event) are examined for aggregation. When you request an aggregation from Zarkov, several things happen:

Zarkov loads the AggDef from MongoDB
Zarkov imports the Task subclass defined in the AggDef
Zarkov calls the 'run' method of the Task subclass with a REQ socket connected to the Zarkov map-reduce router

Task Definitions

Normally, you won't override the 'run' method yourself, instead overriding the _start_jobs method:

def _start_jobs(self, socket, full): raise NotImplementedError, '_start_jobs'

If "full" is true, you'll need to treat this task as a "full recompute" rather than an incremental update. Usually, this just means that you'll be overwriting the output collection rather than reducing your results into it. There are a couple of things that the Task class does for you before calling _start_jobs:

"New" object documents have been identified. This is done by querying the object collection for documents whose 'aggregates' field does not contain this AggDef's name. In a "full" aggregation, this is done by querying the entire collection. If it is not a "full" aggregation, the query is further restricted to documents that have been inserted into zarkov since the last run of this AggDef.
"New" object documents have been marked for this via their 'jobs' field. All the documents that may require processing can thus be queried with {"jobs":self._lock_id}, where self is the Task instance.

In your _start_jobs method, then, you'll mostly be starting Zarkov map-reduce jobs. The Task parent class provides a start_mr() method to help with that:

def start_mr(self, socket, collection, full, filter=None, map=None, reduce=None, 
             map_name=None, reduce_name=None):

The arguments are similar to the fields in the map-reduce request message covered in the previous post. If 'full' is true, the out_mode will be set to 'replace', otherwise it will be set to 'reduce'. start_mr(), in addition to actually kicking off the map-reduce job, notes the job ID in the Task instance so that the run() method can wait until all the started map-reduce jobs are complete before returning.

So how do you actually invoke an aggregation? Let's assume we have the following task definition in a module 'my.tasks':

from zarkov.task import Task 
from zarkov import model as M 
 
class MyTask(task): 
 
    def _start_jobs(socket, full): 
        db = M.doc_session.db 
        self.start_mr(socket, db.zagg.my_agg, full, 
            map=map, reduce=reduce) 
 
def map(objects): 
    for object in objects: 
        yield object['foo'], 1 
 
def reduce(key, values): 
    return sum(values)

You can tell Zarkov about the agg via the following code (probably via zcmd shell or zcmd script:

from ming.orm import session 
from zarkov import model as M 
 
M.AggDef(name='my_agg_def', task_name='my.tasks.MyTask') 
session(M.AggDef).flush()

Running aggregations

Now to trigger the aggregation, just make sure your Zarkov server and/or workers are running and execute the following command:

(zarkov) $ zcmd -y development.yaml aggregate my_agg_def

Wait a bit, and when the command exits, you should hve a brand new collection zagg.my_agg with the counts you requested. If you run the aggregate command multiple times, it should reduce only the new events added since the last aggregation run. And that's all there is to it.

Zarkov is a Lightweight Map-Reduce Framework

2011-07-21T10:10:00.005-04:00

Zarkov is an event logger
Zarkov is a lightweight map-reduce framework
Zarkov is an aggregation service
Zarkov is a webservice

In my previous post, I discussed Zarkov as an event logger. While this may be useful (say for logging to a central location from several different servers), there's a bit more to Zarkov. Today I'll focus on the map-reduce framework provided by Zarkov. If you want instructions on setting up Zarkov or getting events into it, please see my previous post.

Why Another Map-Reduce Framework?

Good question. Currently there exist two big frameworks of which I'm aware: Hadoop and Disco. Hadoop is a framework written in Java that includes all sorts of goodness to make sure that your larger-than-life job gets executed relatively efficiently on a large cluster. Disco has the same kinds of things. Both of them require a good bit of setup to get running. Both of them provide a distributed filesystem that they really like to get jobs from. Neither has tight MongoDB integration.

I wanted something smaller. Something easy to configure, with tight MongoDB integration. "But wait," I hear you say, "MongoDB has built-in map-reduce!" In fact, Zarkov began life as a front-end for MongoDB's own map-reduce. Unfortunately, there are a few problems with the built-in mapreduce command in MongoDB:

it saves its intermediate results in MongoDB collections (lots of locking)
it is restricted to a single thread per mongod process (thanks a lot, Spidermonkey :-p )
it doesn't allow scaling compute power independent of shards

So I set out to write my own....

Setting up a single-node Zarkov map-reduce (zmr) cluster

This is actually pretty simple. This is how you do it:

(zarkov) $ # Create the default directory in which zmr stores its temp files 
(zarkov) $ mkdir /tmp/zmr
(zarkov) $ zcmd -y development.yaml serve

Yes, that's the same command as the event logger. You specify in the development.yaml file how many worker processes you want to keep "locally", and these will start up automatically with your router. For more complex setups, or to run the zmr framework without the Zarkov event logger, you can do the following:

(zarkov) $ # Start "router" -- this will also start local worker procs 
(zarkov) $ zcmd -y development.yaml zmr-router &

Then on your worker nodes:

(zarkov) $ zcmd zmr-worker tcp://worker-ip:5555 &

Note that there's actually no need to specify the config file for the worker process; workers will connect to the router to retrieve their configuration. You do want to make sure that you start all your workers before you start your job in order to correctly load-balance between them.

Running a job on a zmr cluster

Running a job is actually pretty simple. You need to connect to the zmr request port (by default this is 5555) on the router using a ZeroMQ REQ socket and submit a BSON message of the following format:

database: what MongoDB database to read/write to
collection: what collection forms the input to the job?
query: a bson.Binary() containing a BSON-encoded query object
map: python source code string containing a map function
reduce: python source code string containing a reduce function
map_name: name of the function in the "map" param
reduce_name: name of the function in the "reduce" param
out: collection in which to store the result of the job
out_type: replace, merge, or reduce, corresponding to the MongoDB mapreduce command options

The map function has the following format:

def map(objects): 
    for object in objects: 
        output = do_something_to(object) 
        yield new_key, value

Basically, it's a generator that takes a sequence of dicts and yields zero or more (key, dict) pairs. The reduce function has the following format:

def reduce(key, values): 
    return one_value_from_many(values)

So for instance, if we wanted to count the number of objects, grouped by the field 'foo', you could do the following:

def map(objects): 
    for object in objects: 
        yield object['foo'], 1 
 
def reduce(key, values): 
    return sum(values)

The result would be a collection with its objects' _id fields equal to the 'foo' values from the input collection and its 'value' fields equal to the number of objects with that 'foo' value.

Zarkov is an event logger

2011-07-20T15:29:00.003-04:00

Zarkov is an event logger
Zarkov is a lightweight map-reduce framework
Zarkov is an aggregation service
Zarkov is a webservice

In the next few posts, I'll be going over each of the components of Zarkov and how they work together. Today, I'll focus on Zarkov as an event logger.

Technologies

So there are just a few prerequisite technologies you should know something about before working with Zarkov. I'll give a brief overview of these here.

ZeroMQ: ZeroMQ is used for Zarkov's wire and buffering protocol all over the place. Generally you'll use PUSH sockets to send data and events to Zarkov, and REQ sockets to talk to the Zarkov map-reduce router.
MongoDB: Zarkov uses MongoDB to store events and aggregates, so you should have a MongoDB server handy if you'll be doing anything with Zarkov. We also use Ming, an object-document mapper developed at SourceForge, to do most of our interfacing with MongoDB.
Gevent: Internally, Zarkov uses gevent's "green threads" to keep things nice and lightweight. If you're just using Zarkov, you probably don't need to know a lot about gevent, but if you start hacking on the source code, it's all over the place (as well as special ZeroMQ and Ming adapters for gevent). So it's probably good to have at least a passing familiarity.

Installation

In order to install Zarkov, you'll need to be able to install ZeroMQ and gevent, which probably means installing the zeromq and libevent development libs. In Ubuntu, I had to install zeromq2-1 from source (which isn't too tough):

$ wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
$ tar xzf zeromq-2.1.7.tar.gz
$ cd zeromq-2.1.7
$ ./configure --prefix=/usr/local && make
$ sudo make install
$ # if you're on ubuntu, this next line will work 
$  sudo apt-get install libevent-dev
$ # otherwise you need to 
$ wget http://monkey.org/~provos/libevent-1.4.13-stable.tar.gz
$ tar xzf libevent-1.4.13-stable.tar.gz
$ cd libevent-1.4.13-stable
$ ./configure --prefix=/usr/local && make
$ sudo make install

Now you should be able to do a regular pip install to get everything else:

$ virtualenv zarkov
$ source zarkov/bin/activate
(zarkov) $ pip install Zarkov

Next, you should customize your development.yaml file. Here's a convenient example we use in testing:

bson_bind_address: tcp://0.0.0.0:6543 
json_bind_address: tcp://0.0.0.0:6544 
web_port: 8081 
backdoor: 127.0.0.1:6545 
mongo_uri: mongodb://localhost:27017 
mongo_database: zarkov 
verbose: true 
incremental: 0 
zmr: 
        req_uri: tcp://127.0.0.1:5555 
        req_bind: tcp://0.0.0.0:5555 
        worker_uri: tcp://0.0.0.0 
        local_workers: 2 
        job_root: /tmp/zmr 
        map_page_size: 250000000 
        map_job_size: 10000 
        outstanding_maps: 16 
        outstanding_reduces: 16 
        request_greenlets: 16 
        compress: 0 # compression level 
        src_port: 0 # choose a random port 
        sink_port: 0 # choose a random port 
        processes_per_worker: null # default == # of cpus

Zarkov defines a format for an event stream which tries to be fairly generic (though our main use-case is logging SourceForge events for later aggregation). A Zarkov event is a BSON object containing the following data:

timestamp (datetime) : when did the event occur?
type (str): what is the type of event?
context (object): in what context did the event occur? On SourceForge, this includes the project context, the user logged in, the IP address, etc.
extra (whatever): this is purely up to the event generator. It might be a string, integer, object, array, whatever. (It should be supported by BSON, of course.)

The Zarkov events are stored in a MongoDB database (again with the Flash Gordon references). Assuming you've already installed Zarkov, to run the server you'd execute the following command::

(zarkov) $ zcmd -y development.yaml serve

Now to test, you can use the file zsend.py (included with Zarkov) to send a message to the server::

(zarkov) $ echo '{"type":"nop"}' | zsend.py tcp://localhost:6543

To confirm it got there, you can use the 'shell' subcommand from zcmd:

(zarkov) $ zcmd -y development.yaml shell

Then, in the shell you're given, execute the following commands:

In [1]: ZM.event.m.find().all() 
Out[1]: 
[{'_id': ObjectId('4e2723eeb240217416000001'), 
  'aggregates': [], 
  'context': {}, 
  'jobs': [], 
  'timestamp': datetime.datetime(2011, 7, 20, 18, 52, 30, 272000), 
  'type': u'nop'}]

(Your _id value will probably be different). To actually use Zarkov as an event logger, you'll probably want to actually send the ZeroMQ messages yourself. Zarkov includes a client to do just that. From the zcmd shell:

In [1]: from zarkov import client 
In [2]: conn = client.ZarkovClient('tcp://localhost:6543') 
In [3]: conn.event('nop', {'sample_context_key': 'sample_context_val'}) 
In [4]: ZM.event.m.find().all() 
Out[4]: 
[{'_id': ObjectId('4e2723eeb240217416000001'), 
  'aggregates': [], 
  'context': {}, 
  'jobs': [], 
  'timestamp': datetime.datetime(2011, 7, 20, 18, 52, 30, 272000), 
  'type': u'nop'}, 
 {'_id': ObjectId('4e2725a8b240217483000001'), 
  'aggregates': [], 
  'context': {u'sample_context_key': u'sample_context_val'}, 
  'extra': None, 
  'jobs': [], 
  'timestamp': datetime.datetime(2011, 7, 20, 18, 59, 52, 756000), 
  'type': u'nop'},

If you want to customize things further, the ZarkovClient code is actualy quite
short:

'''Python client for zarkov.''' 
import zmq 
 
import bson 
 
class ZarkovClient(object): 
 
    def __init__(self, addr): 
        context = zmq.Context.instance() 
        self._sock = context.socket(zmq.PUSH) 
        self._sock.connect(addr) 
 
    def event(self, type, context, extra=None): 
        obj = dict( 
            type=type, context=context, extra=extra) 
        self._sock.send(bson.BSON.encode(obj)) 
 
    def event_noval(self, type, context, extra=None): 
        from zarkov import model 
        obj = model.event.make(dict( 
                type=type, 
                context=context, 
                extra=extra)) 
        obj['$command'] = 'event_noval' 
        self._sock.send(bson.BSON.encode(obj)) 
 
    def _command(self, cmd, **kw): 
        d = dict(kw) 
        d['$command'] = cmd 
        self._sock.send(bson.BSON.encode(d))

MongoDC: Rapid, Scalable Web Development with MongoDB, Ming, and Python

2011-06-28T08:49:00.000-04:00

Here are the slides from my #mongodc talk for those interested....

Rapid, Scalable Web Development with MongoDB, Ming, and Python

View more presentations from Rick Copeland

MongoSF Slides

2011-05-25T13:30:00.000-04:00

Here are the slides from my talk at MongoSF about Allura and how we're using MongoDB to build the next generation SourceForge developer tools. Enjoy!

Allura - an Open Source MongoDB Based Document Oriented SourceForge

View more presentations from Rick Copeland

Allura Sprint-orials

2011-03-31T20:57:00.005-04:00

So with our initial announcement at PyCon and the sprints afterwards, the newly open source SourceForge project hosting platform Allura has gotten a bit of attention. One thing we want to do at SourceForge is to be responsive and open to the community, to make Allura much more than a "throw it over the wall" open source project. To that end, we've been looking at holding a sprint-orial series.

So what exactly is a "sprint-orial"? It's kind of like a sprint and a tutorial combined. If you don't feel like you know enough to contribute to an Allura sprint, you're welcome to attend to get help setting up Allura or learning where to start hacking on it (the "tutorial" aspect). If you want to start contributing to the project immediately, it's a chance to get together with other contributors in a focused setting. We find that we get a lot more done and have a lot more fun when we're all in a room together (even if it's just an IRC channel). If you're somewhere in between, you're also welcome to come and learn or contribute as your desire and expertise direct you. If you attend , you'll have access to SourceForge developers for answers to questions, help installing, or direction in ways to contribute.

So that's what a sprint-orial is. What we don't know is when and where to do the sprinting. My first thought is that we could do monthly Saturday sprint/tutorials. I can personally commit to this level of involvement. I live north of Atlanta and would be happy to host the sprinting, but I'm also willing to travel within reason, and SourceForge has tentatively (and quite generously) offered to foot the bill for my travel. So if there are groups of Allura hackers who would like a visit, we can schedule one or more of the sprints for your neck of the woods.

So what happens if the sprint isn't in your neck of the woods? Well, the good news is that you can still join us in IRC in #allura on FreeNode during the sprint (or most other times as well, we usually have quite a few SF.net engineers hanging around in there on weekdays).

So what do you think? Anyone out there interested in a sprint-orial? Please let me know in the comments, or on twitter at @rick446. Thanks!

Allura, the Open Source Forge

2011-03-09T14:53:00.004-05:00

So in kind of a soft launch, SourceForge released the project I've been working on (Allura) under the Apache license last month. The project happens to be the actual software underlying the SourceForge.net 'beta' tools, and we're hoping to get lots of community involvement, starting at PyCon.

First off, Mark Ramm will be presenting a couple of talks, one focused on how we're using Python at Sourceforge (Scaling Python Past 100). I will also be presenting a poster at the poster session on Sunday (Using the Allura Platform to Create Your Own Forge). While this is nice and all, the thing that has me really excited is that we'll be hosting a sprint on the Allura platform next Monday through Thursday where we'd love to have people come and help SourceForge to become the forge you'd like it to be.

If you want to check out Allura, feel free to register a SourceForge Beta project and play around with it. Or you can just fork the Allura repository into your own user project area on SourceForge. Or feel free to clone it locally and play around with it. We tie together some cool technologies that I'll be blogging about in future articles, including MongoDB, Solr, and RabbitMQ, to give you a unified set of tools (right now including Repositories, Wiki, Tracker, and Forums) to manage your project.

So if you're interested in contributing, or you're just curious about Allura and want to learn more, please come by our sprints. We'll probably also host an open spaces Allura/SourceForge BOF during the conference proper. As always, feedback is welcome and encouraged, and we'll try to be as responsive as possible (most of our engineers will actually be at PyCon this year). See you there!

MongoATL Video

2011-02-22T15:05:00.002-05:00

Here's the video from my talk at MongoATL a few weeks ago. Enjoy!

http://www.10gen.com/video/mongoatlanta2011/sourceforge

MongoATL Slides

2011-02-08T15:41:00.003-05:00

Here are the slides from my MongoATL presentation for those who might be interested.

MongoATL: How Sourceforge is Using MongoDB

View more presentations from Rick Copeland.

MongoATL!

2011-02-03T14:56:00.003-05:00

Quick note to let anyone in the ATL area know that I'll be speaking on how SourceForge is using MongoDB at MongoATL on February 8th (next Tuesday). The registration cost is only $100 ($30 if you're a student), and it's a great way to learn about one of the most popular NoSQL databases on the scene today.

Review: Grok 1.0 Web Development

2010-03-23T18:29:00.007-04:00

Those who know me technically know that I'm big into the TurboGears Python web development framework. Despite my affinity for al things TurboGearish, it's often useful to spend some time studying other frameworks to see the different design decisions they made, and to maybe just get ideas on how you can improve your own chosen framework. It was with that mindset that I sat down to read Grok 1.0 Web Development by Carlos de la Guardia.

For those who are unfamiliar with Grok, Grok is a Python web framework built on top of the Zope 3 toolkit that endeavors to make Zope more accessible to those who are not already experts in the deep magic of Zope. One of the problems with a layer like Grok is that sometimes Zope leaks through into your code. In such cases, it's nice to have a guide for where Grok starts, where Zope begins, and how they work together. This book can provide just that, as well as giving someone who may be new to Python web development a pretty good tutorial and head start on Grok development in particular, and Python web development in general.

tl;dr Summary

This is a good book. It's not "OMG I need to drop all my TurboGears work right now and rewrite everything in Grok" good, but it gives you a good feel for the Grok framework and the Zope toolkit underneath it without being too overwhelming. If you want to learn Grok or Zope web development, or if you just want to see how someone else did it, I'd say this is a good book to read. If, however, you want an architectural description of Zope, or if you're brand-new to Python, HTML, CSS, and Javascript, this probably isn't the book for you.

Synopsis

This book follows a running example of building a to-do list manager, which I think is a great way to make the daunting world of Zope more accessible to us mortals. By building the app from its barely-on-the-web-no-persistence-at-all roots to a multi-user, multi-project list manager with full security, a heterogenous database backend, etc., the book introduces Grok and Zope concepts in a logical and easy-to-follow way.

The book starts in chapter 1 with a high-level overview of what makes Grok Grok. This is nice if only to give you your bearings before diving into the more tutorial-oriented remainder of the book. Chapter 2 leads your through Grok's installation process including just a bit about Buildout, Zope's packaging and distribution system. Chapter 3 launches into Grok views, including Zope Page Templates (Grok's templating language), form handling, and static resources.

In chapter 4, the Zope Object Database (ZODB) is introduced, as you add persistence to your to-do list application. Some of the "magic" of Grok is also revealed, including how Grok magically knows what views go with what models and what templates. More importantly, it lets you know how you can override this default behavior. Chapter 5 returns to the topic of forms and introduces the grok.EditForm, Grok's solution to automatic form generation and validation.

Chapter 6 introduces the Catalog, which provides search capabilities to Grok applications. Chapter 7 moves on to security, including authorization and authentication. In this chapter, the author even leads you through adding your own authentication module to the todo-list application.

Chapter 8 presents the Grok Viewlet architecture, a set of tools that let you easily build complex templates from simple parts. This is something that in my experience very few frameworks get right. I will definitely be stealing some ideas from here for TurboGears ;-).

Chapters 9 and 10 return to the topic of persistence, with a more detailed discussion of the ZODB in chapter 9 and a description of how you can seamlessly integrate SQLAlchemy and relational databases into a Grok application, a topic near and dear to my heart.

Chapters 11 and 12 go into a good bit of detail of the fundamental concepts behind Grok and Zope, including Zope interfaces and "grokkers" (the things that read the little Grok decorators that make everything so much easier than straight Zope). If you want to know which chapters you can skip when getting started with Grok, it's these two.

Chapter 13 describes Grok's support fortesting and debugging which, like so much of Zope, is really well though out and cohesive once you "get it." Chapter 14 concludes the book with a discussion of deployment and scaling Grok applications, including deployment behind Apache and ZEO (load-balancing) deployments. Again, for the beginner, this is a chapter to save for later.

Conclusion

Overall, I think this is a well-written book. There are a few typos in some of the examples, but it's easy enough to make the corrections in your head, and I didn't find them distracting at all. The book flows well, and was written at a pace that I enjoyed. (For an excerpt to judget for yourself, here's Chapter 5: Forms Again, I'm not going to drop all my TG stuff quite yet, but I do think Grok is worth a second look.

Ming 0.1 Released - Python Library for MongoDB

2009-12-07T17:11:00.006-05:00

One of the things that's been nice about working with SourceForge for the last few months is the chance I get to work with new open source technology and contribute something back. Well, the first (in a long series, I hope) of projects we're putting out there as the result of recent work is a little library we wrote called Ming. It's been available over git for a while, but I just finished publishing our first "official" release, with a tutorial, over at http://merciless.sourceforge.net and http://pypi.python.org/pypi/Ming/0.1. Ming gives you a way of enforcing schemas on a MongoDB database in a Python application. We also throw in automatic, lazy schema migration for good measure.

If you haven't heard of it yet, MongoDB describes itself as on its website as follows:

MongoDB (from "humongous") is a scalable, high-performance, open source, schema-free, document-oriented database.

There are lots of trade-offs to consider when using a non-relational database like MongoDB. On the plus side, MongoDB is simple to develop for (no SQL), extremely fast, and models hierarchical relationships really nicely. On the downside, we give up relational integrity constraints and transactional behavior.

With all the flexibility provided by MongoDB, however, comes some danger. When the database is schema-free, you have to spend extra code making sure that the data you put in and the data you take out is valid for your application. That's where Ming comes in. A play off the character of Ming the Merciless who ruled the planet of Mongo with an iron fist in Flash Gordon, the Ming library provides a succinct way to specify the requirements your application has for the data it produces and consumes. Ming also supports lazy migration of documents across schema revisions. To give you a flavor for how easy this can be, here is a sample schema with migration from the tutorial:

from ming.datastore import DataStore
from ming import Session
from ming import Document, Field, schema

bind = DataStore('mongo://localhost:27017/tutorial')
session = Session(bind)

class OldWikiPage(Document):

    class __mongometa__:
        session = session
        name = 'wiki_page'

    _id = Field(schema.ObjectId)
    title = Field(str)
    text = Field(str, if_missing='')
    metadata = Field(dict(
            tags=[str],
            categories=[str]))

class WikiPage(Document):

    class __mongometa__:
        session = session
        name = 'wiki_page'
        version_of = OldWikiPage
        def migrate(data):
            result = dict(
                data,
                tags=data['metadata']['tags'],
                categories=data['metadata']['categories'],
                version=1)
            del result['metadata']
            return result

    _id = Field(schema.ObjectId)
    version = Field(1)
    title = Field(str)
    text = Field(str, if_missing='')
    tags = Field([str])
    categories = Field([str])

Hopefully, that's enough to whet your appetite for now. Rather than duplicating the entire tutorial, I'll direct you to the docs at http://merciless.sourceforge.net for more information. Let me know what you think!

MetaPython 0.2.2 with Hygienic Macros

2009-05-03T13:40:00.005-04:00

In my ever-expanding quest to, as @jgustak recently tweeted, "introduce evil to Python to prevent even scarier evil," I have released MetaPython 0.2.2 Once again, if you aren't familiar with MetaPython, a good place to start is the tutorialwhich walks you through the construction of a macro-ized collections.namedtuple from the Python 2.6 standard library. If you already know MetaPython, here's the stuff that's new in 0.2.2.

One issue that macro implementors must eventually face is whether they want their
macro system to be hygienic or not. According to wikipedia,

Hygienic macros are macros whose expansion is guaranteed not to cause collisions with existing symbol definitions. They are a feature of programming languages such as Scheme and Dylan.

So what's all that mean? The best way to explain is probably by showing the bad things unhygienic macros bring you. Here is something you could do in MetaPython 0.2:

def vadd(result, a, b):
   defcode foo:
       for i, (aa,bb) in enumerate(zip($a, $b)):
           $<result>[i] = aa+bb
   return foo

...

$vadd(?result, ?a, ?b)

Calling this macro $vadd(?z, ?x, ?y) then gives you something like:

for i,(aa,bb) in enumerate(zip(x, y)):
   z[i] = aa+bb

Which is fine, as long as you weren't planning on using the i, aa, or bb variables for anything important in the surrounding code. This is especially bad because macros can introduce names that conflict with names in the context where they are expanded surprisingly, without any indication to the macro user that they are going to do so.

In MetaPython 0.2.1 and 0.2.2, you declare the block as defcode foo(): which tells MetaPython that the foo block should not capture any variables from its context when it is expanded. In this case, MetaPython detects that the variables i, aa, and bb are assigned in the block and replaces them with "known unique" names (names that should not exist in the surrounding block, wherever they are expanded. In MetaPython 0.2.1 and 0.2.2, you get this expansion:

for _mpy_0,(_mpy_1,_mpy_2) in enumerate(zip(x, y)):
   z[_mpy_0] = _mpy_1 + _mpy_2

So as long as you avoid using names starting with _mpy, you should be fine.

Now sometimes, you actually want to capture a value from the context into which the macro is being defined. One example is when you're writing a class factory like namedtuple, covered in the tutorial. Your goal is to generate a new class, and if that class has some weird _mpy_* name, it's pretty useless. So MetaPython lets you specify, via the arguments to defcode, which names should be captured. In the namedtuple example, for instance, the defcode declaration looks like defcode result(typename):, where typename is a variable containing the name of the class being created. When MetaPython expands the code block, then, any names mentioned in the argument list will not be auto-renamed by the "sanitizer."

MetaPython 0.2.1 and 0.2.2 both had this ability to hygienically expand macros, but 0.2.2 added the ability to use variable arguments to the defcode block. One time when you might want to do this is in a domain-specific language context. Say you wanted to specify that a class contained certain properties, and that those properties should be accessed via the property builtin function. For instance, say you wanted to have a class Foo with properties a, b, c, and d implemented by semi-private instance variables _a, _b, _c, and _d. You might write something like this:

class Foo(object):
   $has_properties(?a, ?b, ?c, ?d)

The implementation of has_properties, then, is the following:

def has_properties(*props):
   str_props = (str(p) for p in props)
   gen = ( (p,
            '_' + p,
            '_get_' + p,
            '_set_' + p)
           for p in str_props )
   defcode result(*props):
       $for pub, pri, getter, setter in gen:
           def $<getter>(self):
               return self.$pri
           def $<setter>(self, value):
               self.$pri =  value
           $pub = property($getter, $setter)
   return result

The final expanded version of Foo is then:

class Foo (object ):
  def _mpy_3 (self ):
    return self ._a

   def _mpy_7 (self ,value ):
       self ._a =value

   a =property (_mpy_3 ,_mpy_7 )
   def _mpy_1 (self ):
       return self ._b

   def _mpy_5 (self ,value ):
       self ._b =value

   b =property (_mpy_1 ,_mpy_5 )
   def _mpy_2 (self ):
       return self ._c

   def _mpy_6 (self ,value ):
       self ._c =value

  c =property (_mpy_2 ,_mpy_6 )
  def _mpy_4 (self ):
       return self ._d

  def _mpy_8 (self ,value ):
       self ._d =value

  d =property (_mpy_4 ,_mpy_8 )

Note in particular that the getters and setters were sanitized, while the actual property names a, b, c, and d were skipped.

So MetaPython now has what I think is a workable hygienic macro system with appropriate escapes for "non-hygienic" operation. Any comments, questions, or criticisms are welcome, as well as ideas for how you are using or might use MetaPython. Let me know what you think!

MetaPython 0.2 Release

2009-04-17T09:34:00.004-04:00

For those intrepid souls who are interested in generating Python code from the macros and code quoting facilities of MetaPython, I have spun a new release. If you aren't familiar with MetaPython, a good place to start is the tutorial, which walks you through the construction of a macro for generating a macro-ized version of collections.namedtuple from the Python 2.6 standard library. If you are already familiar with MetaPython, I will try to summarize the changes and ideas in this release here.

MetaPython 0.1 was mainly a proof of concept implementation to show that it was possible to generate a moderately useful macro facility in a short period of time. I did, however, make one really questionable decision: to use Jinja2 as the templating language for code quotes.

Don't get me wrong; Jinja2 is an awesome templating language. But if I am generating a language extension (MetaPython) that allows you to change the text of a module as just before it gets imported, wouldn't it be nice to also be able to change the text of a code quote using the same syntax? Hence version 0.2.

There was also the issue of ugly syntax with the ?, $, and {%...%} operators. Although ? and $ are still there, I have tried to normalize their use a bit. $ always introduces a construct that should be executed or evaluated "earlier" than the surrounding code. (Inside a defcode...: block, this means at block construction time, otherwise it means at import time.) The ? operator now has a much more limited use as an inline code quoting operator.

Other than syntax changes, one of the main things you'll notice in MetaPython 0.2 is the introduction of import-time control statements ($for..., $if..., etc.) These allow the conditional or repeated expansion of code blocks. These constructs basically obviate the need for another template language (Jinja2) inside defcode... blocks.

Another big change and move toward normalization is that macro calls are now just import-time function calls. This means that their arguments are evaluated before they are called, not passed through as code objects. In order to get the old behavior, you can simply quote the arguments you wish to be sent through unevaluated. For instance, in the old syntax, creating a named tuple was accomplished via ?namedtuple(Point, x, y), whereas in the new syntax, you would type $namedtuple(?Point, ?x, ?y). This makes it substantially easier to write macros which need non-code arguments, and follows the Python Zen of "Explicit is better than implicit."

There have also been significant reworking of the internal MetaPython parser and code construction machinery, although this should not be user-visible. So try it out, let me know what you think, and have fun!

MetaPython Presentation

2009-04-10T13:23:00.002-04:00

Last night at the Python Atlanta meetup I gave a brief talk on MetaPython, including the motivations for doing something so profane as adding macros and code quoting to Python. The video is on blip.tv and you can find the slides on the MetaPython.org Enjoy!

Announcing MetaPython - Macros for Python

2009-03-19T09:11:00.006-04:00

As I mentioned in my last post, I have been considering writing some version of macros for Python and was looking for use cases. Well, having gotten the use cases I so desired from my wonderful commenters, I went ahead and put together an import hook and Google Code project that I'm calling MetaPython — all just in time for PyCon! (I have no talks, but I will be there, and would love to have a MetaPython Open Space if anyone's interested.)

So what's all the excitement about? MetaPython introduces some hooks to allow you to modify module code just before it is seen by the Python interpreter (at what I'm calling "import time"). The import-time syntax is pretty simple, and is (almost) all denoted by a question mark ? prefix (question marks are currently syntax errors in regular Python). Here is a trivial example that defines an import-time function (which as we will see can be used as a macro) that will conditionally remove a function call (and the evaluation of its associated arguments). Suppose the following text is saved in a file "cremove.mpy":


def cremove(debug, expr):
    debug = eval(str(debug))
    defcode empty_result:
        pass
    if debug:
        result = expr
    else:
        result = empty_result
    return result

The idea here is that cremove will be called with two metapython.Code values, debug and expr. cremove will convert debug to its Python code representation by calling str() and then evaluate the result. If debug is true, then expr will be returned. Otherwise a pass statement will be returned (defined using the MetaPython import-time construct defcode which defines a code template). To actually call cremove as a macro, we will need to define another MetaPython module, say "test_cremove.mpy":


import logging
logging.basicConfig(level=logging.DEBUG)
log = logging.getLogger(__name__)

?from cremove import cremove

def do_test():
    ?cremove(True, log.debug('This statement will be logged'))
    ?cremove(False, log.debug('This statement will not be logged'))

Here, we do an import-time import (the ?from... import business). This makes the module we're importing available at import-time (a regular import would be seen just as another line of Python code at import-time). To actually call cremove as a macro, we just need to prefix it with the "?" as shown.

Now, to actually test this, we'll need to install MetaPython and fire up an interpreter. MetaPython is available from the CheeseShop, so to get it just run easy_install MetaPython. Once it's installed, we can test our MetaPython code as follows:


>>> import metapython; metapython.install_import_hook()
>>> import test_cremove
>>> test_cremove.do_test()
DEBUG:test_cremove:This statement will be logged

Since macro expansion can get pretty complex and it's always tricky debugging code you've never seen, the fully-expanded module is available as the __expanded__ attribute of the MetaPython module:


>>> print test_cremove.__expanded__
import logging 
logging .basicConfig (level =logging .DEBUG )
log =logging .getLogger (__name__ )


def do_test ():
    log .debug ('This statement will be logged')
    pass

There's a lot more to MetaPython, but hopefully this has whetted your appetite. There's a short tutorial available that shows how you can implement the collections.namedtuple class factory using a macro. It also shows how you can use Jinja2 syntax along with the defcode construct to dynamically produce Python code. Have fun, and let me know what you think!