Once you've enabled Developer Mode, you can startup a Chrome instance with WebGL enabled. Run "shell" from within crosh (ctl+alt+t) and then run:

/opt/google/chrome/chrome --enable-webgl

Then try loading up a WebGL demo: Cooliris Wall

YMMV

Permalink | Leave a comment  »

See the full gallery on Posterous

Permalink | Leave a comment  »

p69.mov Watch on Posterous

Permalink | Leave a comment  »

Permalink | Leave a comment  »

How do you guarantee consistency with replica sets in a sharded environment? Only read from the master.

And we all know single server durability is not yet possible.

ACID fail

Permalink | Leave a comment  »

Permalink | Leave a comment  »

Seems the theme of the day is: redundant data

Permalink | Leave a comment  »

Permalink | Leave a comment  »

Since the dawn of AJAX, web developers have longed for persistent server-side connections. For a while Comet was hailed as the bastion of “server push”, but deep down we knew it was just a hack. Now finally, years later, we have an API and a protocol being standardized for socket connections between the browser and the server – aptly named, WebSockets.

WebSockets are bi-directional communication channels that run on single TCP sockets allowing communication between the client and the server. Since they behave like regular INET sockets, we should be able to easily implement them with existing tools. However, when I was looking for example implementations in Python, I didn’t find anything that quite satisfied me.

Python sockets module

Now don’t make the mistake of thinking I’m a systems programmer. I have never written a low-level network application like this, and in fact this is my first time playing with sockets or select in Python. The root of all of this is the WebSocket itself, which is just a socket.

import socket
websocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
websocket.bind(("localhost", 9999))
websocket.listen(5)

That’s all you need to get the WebSocket up and running. Granted, it’s not very useful since you can’t connect to it (no handshake), but it’s a WebSocket nonetheless. When a client connects to the socket, it initiates the handshake with the following

GET / HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: localhost:9999
Origin: file://
Sec-WebSocket-Key1: x   d3L703 2  {63 k  L1( 90
Sec-WebSocket-Key2: ^    14   +40Z7R<12om I8  0[

??????????????

And expects a response in a similar form:

HTTP/1.1 101 Web Socket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
WebSocket-Origin: file://
WebSocket-Location: ws://localhost:9999/
Sec-Websocket-Origin: file://
Sec-Websocket-Location: ws://localhost:9999/

??????????????

The “?” are random bits used in the challenge/response part of the handshake. Interesting note: In addition to failing to do the Challenge/Response, Chrome looks for the “Websocket-X” headers, while Safari (correctly) looks for the “Sec-Websocket-X” headers.

Here’s my full standalone WebSocket server: http://gist.github.com/512987

I won’t delve into the details of the implementation, namely because I’m sure it’s suboptimal. I was pretty happy with Challenge/Response piece. I read the spec from IETF and implemented it, nice and simple. Aren’t open standards great? I ended up having to do the handshake because Safari 5 won’t let you use a WebSocket otherwise.

Stay tuned for everyone’s favorite asynchronous demo: a chat program!

-David

Permalink | Leave a comment  »

One thing that kept puzzling me about pymongo was that I couldn’t a serialize a document as JSON. Aren’t these things just fancy JSON objects on the backend? Well, they are – but ObjectId is part of mongoDB extension of JavaScript so there is no JSON-equivalent. And since Python only knows about the standard JSON spec, it won’t know what to do with ObjectId instances. When attempting to encode a Python dictionary which has an ObjectId as one of its values, I get a TypeError saying ObjectId “is not JSON serializable”.

My solution is to extend the JSONEncoder included in Python’s json module (in 2.6 or later)

1
2
3
4
5
6
7
8
9

from json import JSONEncoder
from pymongo.objectid import ObjectId

class MongoEncoder(JSONEncoder):      
    def _iterencode(self, o, markers=None):
        if isinstance(o, ObjectId):
            return """ObjectId("%s")""" % str(o)
        else:
            return JSONEncoder._iterencode(self, o, markers)

It adds a special case to handle encoding an ObjectId into a literal “ObjectId” in the encoded JSON.

Custom JSON encoders can be used when issuing json.dump or json.dumps by specifying cls

json.dump(obj, cls=MongoEncoder)

E.g.,

>>> import json
>>> from pymongo.objectid import ObjectId
>>> from mongoencoder import MongoEncoder
>>> x = {'a':1,'b':"foo",'c':ObjectId()}
>>> print x
{'a': 1, 'c': ObjectId('4c4f4f5e2554c813e4000001'), 'b': 'foo'}
>>> print json.dumps(x)
Traceback (most recent call last):
[...]
TypeError: ObjectId('4c4f4f5e2554c813e4000001') 
  is not JSON serializable
>>> print json.dumps(x, cls=MongoEncoder)
'{"a": 1, "c": ObjectId("4c4f4f5e2554c813e4000001"), "b": "foo"}'

Viola! Enjoy

Permalink | Leave a comment  »

Seems like no one can agree on the best way to structure documents in mongoDB. The consensus seems to be: do what works for you. The nice folks at 10gen offer some guidance on laying out your documents, and they seem to sit in the camp of “redundant data over references”. Redundancy over references is fine for some things, but it can be a real pain in the ass for certain situations. E.g., if your would-be embedded documents are updated frequently, you’re talking about a ridiculous amount of effort to make all the right changes in the right places. Write situations like this make me nervous about data consistency, but that’s another story.

For a little prototype I was working on this past week, I was using ObjectIds to reference documents instead of going the embedded document route. One big disadvantage of this approach with mongoDB is that there is no capacity for JOIN-like operations (it’s part of their NoSQL philosophy). I think this is somewhat bullshit, so I took it upon myself to find a workaround. The goal, get some super basic JOIN-like functionality that I can use from a client library (such as pymongo).

Let’s begin. Suppose I’ve got document class Person that looks like

{
    _id : ObjectId(...),
    name : "string",
    school: ObjectId(...)
}

And document class School that looks like

{
    _id : ObjectId(...),
    name : "string",
}

With the reference document approach (called DBRef by the 10gen folks), the dereferencing takes place on the client side meaning a call back to the server for each document that needs dereferencing. That’s a lot of churn on the wire just for a little bit of data. My solution was to do the dereferencing on the database using JavaScript and db.eval().

var deref = function (field, collection) {
    // C-C-C-Closure!!
    return function (doc) {
        return _deref(doc, field, collection);
    };
}
var _deref = function (doc, field, col) {
    var oid = ObjectId(doc[field]);
    delete doc[field];
    doc[field] = db[col].findOne({_id:oid});
    return doc;
}

Once you have saved these functions on the server, you can use them in MapReduce, $where, or db.eval() calls. Here’s an example call using pymongo (the collection name is “people”):

>>> db.eval("db.people.find({}).map(deref('school','people'))")

Now instead of an ObjectId as the ‘school’ field, you get the document whose ‘_id’ is that ObjectId. The deref function takes in a field and collection so it knows which field contains the reference Id and where it should look for that document. N.B., calling map on a cursor will unroll that cursor (so use skip() and limit() accordingly). Also, db.eval calls will block (though I don’t think it should be problematic since findOne is cheap).

The code as a Gist: http://gist.github.com/477121

Permalink | Leave a comment  »

Or should that be "Schema Free"? Lately, I've been digging into the features of several NoSQL systems, and each time I read the bullet points I see something to the effect of "schema-free" or "unstructured documents". This is often touted as one of the features that makes document databases so great - you are released of the bonds of relational databases: no more key constraints, no more type checking, you are essentially free to insert whatever the hell you want. And how. On the flip side, one of the shitty things about document databases is that since you are so free, it is very difficult to code in the oh-so-familiar OO paradigm if you don't know what the data looks like. There have been many efforts to mitigate this including a plethora of frameworks which let you define structured models in your application (with types and everything!) so that the documents are homogeneous in the database. Ok, order is restored.

Permalink | Leave a comment  »

I won't promise or even suggest that I'll update this more than previous incarnations of my blog. Seriously, no promises.

Permalink | Leave a comment  »

Permalink | Leave a comment  »

Warning pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

import hmac
import hashlib
import base64
from cPickle import dumps
 # The unsigned pickled data
string_to_sign = dumps({'foo':"bar",'spam':"eggs",'the answer':42})
 # The signature object
signature = hmac.HMAC(key="my application's super secret key",
    msg= string_to_sign, digestmod=hashlib.sha256)
 # The signed string: store this
signed_string = string_to_sign + base64.encodestring(signature.digest())

import hmac
import hashlib
import base64
from cPickle import loads
 # Break up the signed string into message and signature
signature = signed_string[-45:]
message = signed_string[:-45]
 # Calculate the signature of the message
msg_sig = hmac.HMAC(key="my application's super secret key",
    msg= message, digestmod=hashlib.sha256)
 # See that it matches the given signature
assert base64.encodestring(msg_sig.digest()) == signature

Permalink | Leave a comment  »

import urllib2
from collections import defaultdict
from xml.dom import minidom
from xml import xpath
class TestUserNoun:
        def get_xml_doc(self,url_params):
                url = "http://api.example.com/users/show?"
                url += "max_count=%(max_count)s&order_by=%(order_by)s"
                url_p = urllib2.urlopen( url % defaultdict(str,url_params) )
                doc = minidom.parseString( url_p.read() )
                url_p.close()
                return doc

class TestUserNoun:
        ...
        def test_user_count(self):
                # Test several values of max_count
                counts = (5,10,15,20)
                def count_users(n):
                        # Test that the number of results returned is less than or equal to n
                        doc = self.get_xml_doc({'max_count':n})
                        user_count = len( xpath.Evaluate('/xpath/expr',doc.documentElement) )
                        assert user_count 
And you get the idea - one can write tests ad nauseum (although I'm not sure if there's such a thing as too many tests). Of course neither of these tests will work since the XPath expressions are not valid - I didn't really feel like spelling out a whole XML schema just for this example. There are plenty of good XPath tutorials out there. The basic idea here is you want to test all of your request parameters for the API to see a number of things: 

* Does the controller handle the requests properly? What about missing/extra parameters?
* Are errors handled properly?
* Is the resulting XML valid? This is implicitly done by parsing the XML document
* Does the resulting data correspond to the request parameters? This one will require the most tests to be written - don't forget about generative tests!

A powerful test suite means a robust application. When you have a nice set of tests, you can push your code with confidence - and believe me, that is a very rewarding and relieving feeling. Writing this API has been an extremely rewarding experience, and probably the most educational thing I've done programming-wise since I wrote a cross-browser javascript event library like 5 years ago.

So go forth, programmer - embrace testing and empower yourself.

-David

Permalink | Leave a comment  »

class Client:
"Here's the class schema"
        def get(self, key)
        def put(self, key, value, time_to_expire=3600, replace=False)
        def update(self, key, value, time_to_expire=3600)
        def delete(self, key)

CloudCached Benchmarks (10 runs)
        --------------------------------------------------------
        Test                                  |        Average (s)                | Total (s)  
        --------------------------------------------------------
        GET integer                         |        0.0283360004425        | 0.283360004425
        GET string (32 byte) |        0.0315794944763        | 0.315794944763
        GET string (512KB)         |        0.1265994787220        | 1.265994787220
        PUT integer                         |        0.0650457143784        | 0.650457143784
        PUT string (32 byte) |        0.0563205003738        | 0.563205003738
        PUT string (512KB)         |        0.1773290872570        | 1.773290872570
        --------------------------------------------------------

Permalink | Leave a comment  »

Permalink | Leave a comment  »

Permalink | Leave a comment  »

Permalink | Leave a comment  »