Chris Moyer

Creating Sequences using SDB

2010-02-25T15:56:00.000-08:00

Thanks to the new functionality in Amazon's SimpleDB we can now choose to have consistent DBs instead of highly scalable ones. Previously if we wanted to have a database that was guaranteed to be consistent right away, our only choice was to use our own SQL database, or use RDS.

But why would you want to trade in performance and availability for consistency? It's quite simple, if you've ever tried to generate sequential numbers for any reason (typically because people don't like using random UUIDs), then you know the ONLY way to do this is by using a locking mechanism. Since SDB was previously only eventually consistent, this made it impossible to use such a database for that purpose.

Thanks largely to Mitch Garnaat's post about how to create Counters, I've been able to create a "Sequence" object for boto that will allow you to persist a SequenceGenerator into SDB, and use it reliably across multiple locations, threads, and processes. This new functionality is now in boto.

Using this new sequence object is relatively simple. First, if you have a [DB] section already in your boto.cfg, it's easy to set up a default domain for your sequences. The Sequence object will look first for a key "sequence_db", and if that doesn't exist it will fall back to "db_name" which is used by the rest of the boto.sdb.db module as well. A sample config section would look like this:



[DB]

db_name = default

sequence_db = sequences

Next it's time to launch up python and start playing around.

>>> from boto.sdb.db.sequence import Sequence

>>> s = Sequence() # Note that we can pass in an optional name

>>> s.id # but if we don't it just uses a UUID

'1ce3eb7b-3fdd-4c60-b243-ec33019090bd'

>>> s.val # The value is set to the first value in our set

0

>>> s.next() # Lets get the next value in this set

1

>>> s2 = Sequence(s.id) # Lets load up this set in another object

>>> s2.val # The value should be the same, even if this was somewhere else

1

>>> s.next() # We increment our first object

2

>>> s2.val # And when we look at the second object it's also incremented

2

>>> s.delete()



So that's all fine and dandy if we're just using a simple sequence, but what if we want to do something more complicated like a fibonacci sequence? Lucky for us this is built into our sequence module:







>>> from boto.sdb.db.sequence import fib

>>> s = Sequence(fnc=fib)

>>> s.val

1

>>> s.next()

1

>>> s.next()

2

>>> s.next()

3

>>> s.next()

5

>>> s.next()

8

>>> s.delete()



But what is this "fnc" argument you ask? Quite simply the sequence object allows you to pass in a custom function that determines how to get the next value in the sequence. This function is passed in both the current value, and the previous value in the sequence. The fibonacci function, which you could have made yourself, simply looks like this:


def fib(cv=1, lv=0):
    """The fibonacci sequence, this incrementer uses the
    last value"""
    if cv == None:
        cv = 1
    if lv == None:
        lv = 0
    return cv + lv




The important things here to remember is that the first value in the sequence must be returned if both the first and second values passed into the function are "None". The first value passed into this function is the "current" value of the sequence, and the second value passed in is the "last" or "previous" value that was in the sequence just before our current value. 





So this is great if you're dealing with integers, but what if I want to increment a string, or a double, or for that matter any random sequence? Lucky for us the cast type is determined automatically, so whatever types you have in your sequence will be the types that come back out of it. So, for example, if you have a string sequence that you want to increment easily, you can use the "increment_string" function:







>>> from boto.sdb.db.sequence import increment_string

>>> s = Sequence(fnc=increment_string)
>>> s.val
'A'
>>> s.next()
'B'
>>> s.next()
'C'
>>> s.val = "Z"
>>> s.next()
'AA'
>>> s.delete()


So what's the magic of this "increment_string" function? Let's take a look:


increment_string = SequenceGenerator("ABCDEFGHIJKLMNOPQRSTUVWXYZ")


What's this SequenceGenerator stuff? Quite simply you can pass in either a string or a list and it'll use that to determine what the next value in the sequence should be. You can also pass in an optional value called "rollover" which will prevent the sequence from "rolling over" and instead just make it return back to the initial value, so instead of going from "Z" to "AA", it would go back to "A":


>>> from boto.sdb.db.sequence import SequenceGenerator
>>> s = Sequence(fnc=SequenceGenerator("ABC", True))
>>> s.val
'A'
>>> s.next()
'B'
>>> s.next()
'C'
>>> s.next()
'A'
>>> s.delete()


With this new SequenceGenerator, and the Sequence object now available in boto.sdb.db, you should now be able to generate any sort of sequence kept in SDB that you could think of.

OSX, Vim, and Python

2010-01-05T14:37:00.000-08:00

I've been using python and vim for the past few years as my sole development environment since I hate GUIs, and I love how simple and powerful vim really is. I started off with a simple tutorial, and then expanded from there to develop my own custom IDE. I recently discovered that vim7 has support for running python commands inline. This only further's my love of vim.

I also recently discovered a nice little checker for vim called pyflakes. This program allows you to check python scripts for simple errors that are detected at compile time in languages that support that sort of thing. It includes being able to check for un-imported modules, typos, and even things that you imported that you didn't use within that module.

The problem is that I'm using Mac OSX, which comes with vim that isn't compiled with "--enable-pythoninterp", so the pyflakes vim plugin doesn't work. A simple workaround is on the tutorial page I found, by simply installing MacVim though you only get it compiled against python2.5, not 2.6 which is standard now in Snow Leopard. Additionally, the syntax highlighting in MacVim is horrible for python compared to the built in one that ships with OSX, and I still prefer to run vim in the console.

Solving this issue involves a few steps:

1) Build MacVim from source. Of course you'll need the 10.6 XCode to do this, but after that it's pretty straight forward. The only thing to watch is that you have to run "./configure --enable-pythoninterp --with-macsdk=10.6" instead of just "./configure". Other then that you can follow the instructions from the MacVim site for compiling your own version.

2) Copy the MacVim.app folder into your /Applications folder

3) add [ -x "/Applications/MacVim.app/Contents/MacOS/Vim" ] && alias vim=/Applications/MacVim.app/Contents/MacOS/Vim To your shell rc (for me it's .zshrc but for most people it's probably .tcshrc). I noticed that simply copying the Vim binary to your PATH, or sym linking it doesn't work, not quite sure why but this works well enough for me.

4) Copy the python syntax file from /usr/share/vim/vim72/syntax/python.vim to ~/.vim/syntax/python.vim

5) Download and install the pyflakes.vim plugin

6) Open up a new terminal and edit an already created python source file.

After you've done all of that you should see that anything that would normally show up in an error while running pyflakes now automatically get's highlighted as an error.

I've also been using the pyflakes binary included with the pyflakes.vim plugin to check everything in my modules before building and uploading them to our servers. The only issue I've noticed is that sometimes you need to import a module even though you don't need to use it (for example with boto this is used in the sdb.db module to enable reverse-references). I'm still not quite sure how to ignore those or get around that.

One other thing that doesn't work is doing conditional imports, for example to get JSON support the manuals tell you to do:



try:

   import json

except ImportError:

   import simplejson as json

But in pyflakes, this will result in an error.

Aeron Chairs

2009-12-18T11:16:00.000-08:00

Newstex has a tradition of giving all of it's employee's something related to the business we do (Authoritative Content Aggrigation) every christmas. One year it was a Kindle, once an Apple TV, and once an iPhone. This year, however, Newstex bought the most expensive desk chairs ever produced.

It took a while to adjust this chair, about 400 different settings to change, it can be relatively comfortable, but it's one of those things that we all look at yet never want to buy (who spends $850 on a chair?). In any event, this is the exact same chair as we use to have at the old RIT DataCenter when we had to stay up late running scripts which should have been handeld by a good crontab.

I've realized today that I've moved quite a bit away from that original job in the DCO. Back then I was running scripts by hand that should have easily been automated, and today I work for a company automating processes that most people would consider far to complex for a computer to handle. Now that we've migrated our entire operations into Amazon Web Services, we have the scalability and availability of the largest companies, without any big investments in hardware, or having to worry about hardware upgrades at all. Just today I performed a live upgrade on our websites without a single second of downtime.

Paging SDB results in boto

2009-12-12T18:05:00.001-08:00

Boto can be a great tool if you're querying against SDB, and it helps you out by managing paging automatically for you so you don't have to keep querying it for the next set of results. If you're dealing with a web-based application, however, you have to deal with your own paging and simply iterating forever over a large result set will eventually time out your connections. To solve this, you can use the built-in paging system provided by boto.

Everytime you query using "db.select" in boto, you get back a result set. Most people probably just think of this as an iterator, since it does all the magic behind-the scenes and only queries when you start iterating. It also stores that magical "next_token" within itself so it can query for the next page of results from SDB. Normally, you wouldn't even notice this attribute, but if you're dealing with a service that needs to return in a short amount of time, it can be quite useful.

Additionally, there are two important keyword arguments you can specify to the "select" command on any domain. These are max_items, and next_token. The max_items keyword tells boto to return after it has yielded that number of results, instead of simply handling the paging automatically for you. It's also quite important to add the limit SDB command to your query or boto will return in the middle of the result set and you will lose those middle results!

Ok, now to the code:

>>> import boto

>>> sdb = boto.connect_sdb()

>>> db = sdb.get_domain("default")

>>> rs = db.select("SELECT * FROM `default` LIMIT 10", max_items=10)

Notice that we set "LIMIT" and "max_items" both to 10.

Also note that "rs" is the result set of your select query, but only runs after you start iterating, rs.next_token should be blank now

>>> rs.next_token

>>> for i in rs:

... print i

Your first 10 results will print out, now rs.next_token is set:

>>> rs.next_token

u'r........'

Now you can pass that next_token back to the SAME select, it must be the EXACT same query for next_token to work:

>>> rs2 = db.select("SELECT * FROM `default` LIMIT 10", max_items=10, next_token=rs.next_token)

>>> for i in rs2:

... print i

Your next 10 results will print out

After you get to this point, it's a simple process to rinse-repeat. Once you run out of results, rs.next_token will be empty.

Snow Leopard and lxml

2009-09-07T14:39:00.000-07:00

As many of you might already know, I've been dealing a lot with XML and python recently. I also recently upgraded to Mac OS 10.6 Snow Leopard. To my surprise, after the upgrade I noticed that I was now being pushed from python 2.5 to python 2.6. At first I was a little concerned since everything I had been doing was in 2.5, and all of our servers are running 2.5, but I soon found that I could continue to build for 2.5 and upload my eggs, so I welcomed the change.

After a while, I became aware that subversion was also upgraded. Building pysvn again proved to be a difficult task. I upgraded from python 2.5.1 to python 2.5.2 in order to try to also install the development libraries required to build pysvn from source. I had many issues there and then decided to move on and just install subversion 1.6.1 and the python 2.6 egg from the pysvn website.

I then discovered that lxml wasn't installed properly, and every time I tried to install the egg it told me that lxml was a pre-requisite of lxml... a bit of a confusing error message, but basically the change came about because 10.6 builds for 64 bit by default. Eventually, after attempting to build lxml for quite a while, I discovered the issue was that it wasn't working with gcc-4.2, the default compiler. If you get an error like this:

/Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory

...

lipo: can't figure out the architecture type of: /var/folders/....

Then you have to do two things.

First of all, make sure you install the new XCode including the XCode 10.4 SDK (an optional install that is unchecked by default). For some reason most of the python modules I've found seem to rely on the 10.4 SDK and they even back-port the install for 10.3.

Secondly, you have to change your default compiler from gcc 4.2 to gcc 4.0. This can be done by removing the symlink file in /usr/bin/gcc and re-linking that to gcc-4.0. This can be done by doing:

sudo rm /usr/bin/gcc

sudo ln -s /usr/bin/gcc-4.0 /usr/bin/gcc

After all that is done, you should be able to download lxml and build it using the standard Mac build command:

python setup.py install --static-deps

REST Method calling

2009-07-17T08:25:00.001-07:00

For a few years now, I've been strongly encouraging usage of REST interfaces when building any sort of application. Providing a simple REST interface on top of your application means that you can provide a service and have someone else deal with creating the interfaces. This allows you to have a simple JavaScript client, a Dashboard widget, an iPhone client, and even a Command line client all operating on the same back-end.

The one problem I've found with REST interfaces is that they only allow you to operate on resources with 4 methods, GET, POST, PUT, and DELETE. These methods as stated by most, allow you to Read, Create, Update, and Delete objects ONLY. In working with my applications, I often find the need to let the client perform other operations directly on the object. One possible solution to this problem is to overload the POST method with an optional "ACTION" parameter associated with it. This, however, always seemed hack-ish to me and more of something you'd see in SOAP then REST.

My proposed solution was to use HTTP to it's fullest and allow arbitrary HTTP verbs to be used on objects and collections. For example, if you had a user object at:

/users/moyer

REST says you can do:

GET /users/moyer

To perform a "GET" (which would read the object) operation on the object, or

DELETE /users/moyer

To perform a "DELETE" operation on the object, so why not

RESETPW /users/moyer

To perform a "RESETPW" operation on the object? This turns a simple REST interface into a fully developed remote procedure calling system built directly off of the HTTP specifications.

I have tested this usage using the python httplib standard library, proxying through apache and hitting a CherryPy and Paste backend server. All of my tests suggest that the client libraries and server libraries all will support arbitrary extensions of the HTTP specification. This is the natural flow of progression to extend REST to support more methods on an individual object.

OSX WebDav Client

2009-07-07T15:23:00.000-07:00

In working on my custom WebDAV server, I discovered an interesting note about the OSX built-in WebDAV client. First of all, if you use finder to copy files (instead of cp in terminal), it sends all files over in chunked transfer-encoding. This is entirely allowed within the WebDAV specification, but it doesn't help that they also send it using the WRONG header. They send the encoding method as "Chunked" instead of "chunked".

While working with Python Paste, I found that chunked encoding wasn't supported at all, so I switched over to using CherryPy. After testing locally and getting everything to work (CherryPy apparently ignores case), I deployed this system to production behind an apache server to add in SSL support (Yes, I plan to use Nginx in the future, but we have apache set up here already). After setting up the proxy, I noticed that I could no longer send files to my WebDAV server and a strange message was appearing in the apache error.log:

[error] proxy: Chunked Transfer-Encoding is not supported

[error] [client xxx.xxx.xxx.xxx] Handler for proxy-server returned invalid result code 22

After a lot of googling, I found a single page (written in german), which describes the issue:

http://www.stresser.de/blog/category/server/&ei=0clTSsHICYO0NvnDtewI&sa=X&oi=translate&resnum=1&ct=result&prev=/search%3Fq%3D%2522Handler%2Bfor%2Bproxy-server%2Breturned%2Binvalid%2Bresult%2Bcode%2B22%2522%26hl%3Den%26client%3Dsafari%26rls%3Den-us

The end result was that I had to enable mod_headers, and place this line in my apache config under the virtual host:

RequestHeader edit Transfer-Encoding Chunked chunked early

This fixes the header before mod_proxy gets to it and changes the "Chunked" to "chunked" which tells apache how to handle the request body.

Now that's service

2009-03-23T15:43:00.000-07:00

After fighting with my mac pro all week, I finally took it in on Friday to get the graphics card replaced. I had deduced this was the problem, but still I expected to get some heat from the apple store. After bringing the computer up to the store, an apple rep quickly came out to meet me and carry the beast the rest of the way to the table. He pushed up my appointment by half an hour, and then I even got them to look at it even earlier because someone hadn't shown up yet.

Not only did they not ask me a dozen questions, they didn't even plug the thing in. I said "The graphics card is dead" and they said "Ok, we'll get you a new one".... Thank you apple for not treating me like an idiot.

They had to order it, which took a total of a day, less then 24 hours later I got a phone call saying I could pick it up. They even took it out to my car for me... and all of this was under warranty so I didn't pay anything. Now that's what I call service. Yes, apple's may be more expensive, but this is what you're paying for and it's WELL worth it.

RAWSUG Meeting

2009-03-14T19:54:00.000-07:00

This tuesday at 6:00 pm will be our next Rochester Amazon Web Services User Group (RAWSUG) meeting.

We will be going over some of the basics of cloud infrastructure, and discussing the interesting solutions we have each come up with to develop high-demand and highly-scalable/reliable applications. If you've ever had any desire to learn about AWS or Cloud Computing in general, I highly suggest you come check it out. Most of the meeting will be a forum-based discussion with a few presentations in the beginning. If you have any specific requests please email me beforehand and I'll try to get it lined up. All presentation notes will be posted on http://rawsug.org after the meeting.

It will be hosted at RIT in the IT conference room on the second floor of GCCIS. For more information visit http://rawsug.org

Life lessons

2009-01-05T07:32:00.001-08:00

Over the years I've developed a few theories about people that I thought I'd share with everyone.

People are idiots

In general, people are idiots. Even if you're not an idiot, you'll have idiotic moments. Everyone does something stupid at least once a day that someone else sees. The only difference between idiots and intelligent people is that the idiots don't notice it.

Stereotypes exist for a reason

Stereotypes came from somewhere, we really can't make this crap up. No, not everyone fits them, but there's at least 80% that do. Have you ever met a straight white person that can dance?

In the end, everything comes down to greed

Every single thing that anyone does is for selfish reasons. Even if you're helping someone else out of what you would think of as pure reasons, you'll expect some sort of self-gratification, either from a thank you or something else. Nothing is ever done out of pure selflessness.

How RIT reverses your education.

2008-12-11T16:42:00.000-08:00

So, I'm finally a Senior at RIT, doing my last year, which basically consists of one class that spans two quarters, known as Senior Project. On the first day we were shown a pretty powerpoint that discussed how we're suppose to use all of our knowledge and training over the past 4 years (yes, it's a 5 year program), and incorporate everything into one big project.

Sounds great right? Well not so much. For this class we're giving a real-world customer and a product they want us to create. We start out by gathering requirements from the customer, then we have to design, architect, build, and, finally, deploy the product. It all starts off great, we meet the customer, start gathering requirements, and respond back with some decent offers for what we can do.

Next, we start talking about how we can actually go about doing this, and what type of software we can build in the next 22 weeks. Since we're only a team of 5, and we have 22 weeks to build the entire system from scratch, scope is very important. We limit down what we're going to do and present a counter-proposal to the customer for what we think we can get done given our time constraints.

During the meeting, the client keeps talking about how he wants a nice user interface, we explain that we're focused on the core system, but we can help his people develop a decent one, and provide a very simple interface. We discuss our plans to architect the system in such a way that it will be expandable later on and should easily scale and be able to run on multiple systems. We're designing the system to be a proper 3-tier Presentation, Application, Data, system (web service). After the meeting, our faculty advisor turns to us and says "You don't need to worry about architecting the system like that, just give him a nice GUI and he'll be happy, we don't care if it's scalable".

As it turns out, RIT is really only interested in making the customer happy for the time-being, and not really producing decent software. I once again show my distain for RIT's policies. Instead of actually teaching us things and then testing us on them as they said, they want to throw us into the world and tell us "Just forget about your training and try to make the other guy happy."

Good job RIT.