I have an API I need to talk to. It’s pure XML-over-HTTP (henceforth XHTTP). Actually, the first issue I had was just dealing with XHTTP. All of the APIs I’ve dealt with in the past were either XMLRPC, SOAP, which have ready-made libraries ready to use, or they used GET parameters and the like, which are pretty simple to deal with. I’ve never had to actually construct an HTTP request and just POST raw XML.

It’s as easy as it should be, really. I code in Python, so once you actually have an XML message ready to send, you can use urllib2 to send it in about 2 lines of code.

The more interesting part is putting the request together. I thought I had this beat. I decided to use the xml.dom.minidom module and create my Document object by going through the DOMImplementation object, because it was the only thing I found that allowed me to actually specify a DTD programmatically instead of hackishly tacking on hard-coded text. No big deal.

Now that I had a document, I needed to add elements to it. Some of these XML queries I’m sending can get too long to write a function that manually creates all the elements, then adds them to the document, then creates the text nodes, then adds them to the proper element… it’s amazingly tedious to do. I really wish Python had something like PHP’s XMLWriter, which lets you create an element and its text in one line of code.

Tedium drives me nuts, so rather than code this all out, I decided to create a dictionary that mirrored the structure of my query, with the data for the text nodes filled in by variables.

query_params = {'super_special_query':
                   {
                      'credentials': {'username': user, 'password': password, 'data_realm': realm},
                      'result_params': {'num_results': setsize, 'order_by': order_by},
                       query_type: query_dict
                    }
                }

def makeDoc():
    impl = getDOMImplementation()
    dt = impl.createDocumentType("super_special", None, 'super_special.dtd')
    doc = impl.createDocument(None, "super_special", dt)
    return doc

def makeQuery(doc, query_params, tag=None):
    """
        @doc is an xml.minidom.Document object
        @query_params is a dictionary structure that mirrors the structure of the xml.
        @tag used in recursion to keep track of the node to append things to next time through.

    """

    if tag is None:
        root = doc.documentElement
    else:
        root = tag

    for key, value in query_params.iteritems():
        tag = doc.createElement(key)
        root.appendChild(tag)
        if isinstance(value, dict):
            makeQuery(doc, value, tag)
        else:
            root.appendChild(tag)
            tag_txt = doc.createTextNode(value)
            tag.appendChild(tag_txt)

    return doc.toxml()

doc = makeDoc()
qxml = makeQuery(doc, query_params)

This is simplistic, really. I don’t need to deal with attributes in my queries, for example. But it is generic enough that if I need to send different types of queries, all that’s required is creating another dictionary to represent it, and passing that through the same makeQuery function to create the query.

Initial testing indicated success, but that’s why you can’t rely on only simple initial tests. Switching things up immediately exposed a problem: The API server validated my query against a DTD that enforced strict ordering of the elements, and Python dictionaries do not have the same notion of “order” that you and I do.

So there’s the code. If nothing else, it’s a less-contrived example of what you might actually use recursion for. Tomorrow I have to figure out how to enforce the ordering. One idea is to have a separate list to consult for the ordering of the elements. It requires an extra outer loop to go through the list, get the name of the next tag, then use that value to ask the dictionary for any related values. Seemed like a good way to go, but I had a bit of difficulty figuring out how to make that work at all. Maybe with fresh eyes in the AM it’ll be more obvious — that happens a lot, and is always a nice surprise.

Ideas, as always, are hereby solicited!

I’ve had a lot of time to myself this weekend, and decided to spend a few solid 1-hour blocks playing guitar. Oh, I still play around the house here and there, but usually if I can get in 20 minutes I feel privileged. Learning more kids songs is helping

But this weekend I wanted to learn a couple of songs in particular. I wanted to learn “Why Georgia” and “My Stupid Mouth” from John Mayer’s “Room for Squares” album. Although since that album’s release Mr. Mayer has gone on to do more produced and (to a guitarist) less interesting things on his commercial albums, Room for Squares contained really wonderful textures and sounds while somehow keeping the guitar front-and-center.

John also uses a lot of cool chords and fingerpicking techniques that I knew I’d have fun with if I ever had the time to sit down with the songs and my guitar at the same time. I’ve been playing for almost 25 years, and knew a lot of the chords and techniques, but it’d be great to be able to use them in songs that people under 30 recognize rather than having to explain (for example) who James Taylor is.

How I Learn Songs: First Phase

I learn songs by first listening to them either with headphones on, or at a relatively high volume, about 20 times. I want to become intimate, at a subconscious level, with every detail of the song. I want to feel the tension build, recognize cues from all of the instruments, and understand how they’re playing off each other. This is wildly important to a one-man musician, because often I try to embellish the final piece to try to put hints of the entire song into the final piece — not just the guitar part.

Of course, I *start* with the guitar part, because it’s a bridge to the rest of the song. I learn it as close to note-for-note as possible (where there are solos, the same goes for the solo. I’m sick that way).

Once I can play the song through along with the recording and not hear anything going awry, I practice it that way several times. Maybe 10 times.

Then the fun begins.

Having Fun Learning: Second Phase

When I was a kid, my mom and her sisters liked to go to flea markets. I was not a big fan of these huge collections of what looked like garage sales, until I came across a group of tables with what must’ve been 1000 cassette tapes (that was the popular audio medium of the day). When I scanned the collection, I noticed an unusually large collection of Jimi Hendrix tapes. I figured they were compilations, but after picking one up, I realized it was actually a random collection of live and unreleased recordings.

Over the course of months I collected as many of these as I could, because Hendrix was an artist that always sent me running to the guitar. All told I had about 6-8 versions of Red House, a classic 12-bar blues jam. I had 3-4 versions of Foxy Lady, another 3 of Purple Haze, and lots more. Going through and picking out differences between the live and recorded versions was a great primer on how you can alter and embellish while staying within the same basic framework of a song without totally destroying it.

From then on, whenever I learned new songs or started studying a particular guitarist, I tried to get as many different versions of different songs as I could, to see how the guitarist thought about the piece. Guitarists on the road forced to play the same songs night after night get bored, and often try to make things interesting by altering different parts of the song, or even by changing the guitar’s role in different parts of the song. Fascinating stuff.

Making it Your Own: Third Phase

I would learn lots of this stuff, again, note-for-note to the best of my ability. I was blessed with a fantastic ear and good sense of rhythm, and that along with practice and constant exercising of those talents made me able to pick out and play even some rather complex stuff in no time. By the time I was 14, I was able to play simple things like Led Zeppelin’s “The Lemon Song” in about an hour (not counting the solo), and most of the stuff on the radio I could begin playing along with accurately before the song was even over.

Even better, there were plenty of “solo acoustic” variants of songs, or songs where the live recording happened to pick up the guitar and drums more than the bass and keyboards, or it emphasized the bass and drums more than the guitar. These imperfections are absolute gems for learning more about what’s going in the song… or what *could* be going on when you play it!

Once I learn a song and work in some of the original artist’s own embellishments, some of my own embellishments, and some of my musical personality and influences, it more or less becomes “my own”. I can still play a variant faithful to the recording, but it seems boring when you’ve heard all that can be done with it.

What’s Better Now?

Well, it’s not like I used to pick a song, spend a weekend with it, and all of a sudden it was my own. While I could easily learn decently solid renditions of songs in a weekend, it would take me months to find different recordings of a particular song, to read magazine articles and sometimes books about guitarists, scouring the words for descriptions of how they played things and thought about them. It would be an ongoing process that sometimes also involved figuring out the players’ influences and tracking down *their* recordings… ugh.

Today?

*Literally* today, I decided to learn “Why Georgia” by John Mayer (I learned a decently solid version of “My Stupid Mouth” yesterday). While I was listening to the song in iTunes, I also checked out YouTube and searched for the song there. Boom! 5000 hits. In about an hour, I had heard (AND SEEN!!!) about 10 different performances by John Mayer himself, and I spent about another 20 minutes browsing videos others have posted either covering the song, or providing a tutorial on how to play the song!

Of course, some of the covers and tutorials are horribly wrong, but it’s easier to pick out stuff that’s not going to help you on YouTube than it is actually having to play through badly written tablature to figure out it’s wrong (and the internet, I promise you, is absolutely flooded with bad tablature).

The point is, I was able to learn this song, and work in Mayer’s embellishments as I went. Watching him play it in live performances, I was able to rely in visual cues and his body language to help me figure out how he thought about the song as he was playing it. Seeing him play in solo acoustic settings vs. live with a whole band was enlightening. In the end, I have learned a version of the piece, in one day, that would probably have taken me months to put together in the 80’s.

Thanks, Internet!

While I really like writing code and like it more using Python, no language is without its quirks. Let me say up front that I don’t consider these quirks bugs or big hulking issues. I’m not trying to bash the language. I’m just trying to help folks who trip over some of these things that I found to be slightly less than obvious.

Python’s Cmd Module and Handling Arguments

Using the Python Cmd module lets you create a program that provides an interactive shell interface to your users. It’s really simple, too. You just create a class that inherits from cmd.Cmd, and define a bunch of methods named do_<something>, where <something> is the actual command your user will run in your custom shell.

So if you want users to be able to launch your app, be greeted with a prompt, type “hello”, and have something happen in response, you just define a method called “do_hello” and whatever code you put there will be run when a user types “hello” in your shell. Here’s what that would look like:

import cmd

class MyShell(cmd.Cmd):
   def do_hello(self):
      print "Hello!"

# Kick off the shell
shell = MyShell()
shell.cmdloop()

Of course, what’s a shell without command line options and arguments? For example, I created a shell-based app using Cmd that allowed users to run a ‘connect’ command with arguments for host, port, user, and password. Within the shell, the command would look something like this:

> connect -h mybox -u jonesy -p mypass

Note that the “>” is the prompt, not part of the command.

The idea here is that you pass the arguments to the option flags, and you can set sane defaults in the application for missing args (for example, I didn’t provide a port here — I’m leaning on a default, but I did provide a host, since the default might be ‘localhost’).

Passing just one, single-word argument with Cmd is dead easy, because all of the command methods receive a string that contains *everything* on the line after the actual command. If you’re expecting such an argument, just make sure your ‘do_something’ method accepts the incoming string. So, to let users see what “hello” looks like in Spanish, we can accept “esp” as an argument to our command:

class MyShell(cmd.Cmd):
   def do_hello(self, arg):
      print "Hello! %s" % arg

The problems come when you want more than one argument, or when you want flags with arguments. For example, in the earlier “connect” example, my “do_connect” method is still only going to get one big, long string passed to it — not a list of arguments. So where in a normal program you might do something like:

class MyShell(cmd.Cmd):
   def do_connect(self, host='localhost', port='42', user='guest', password='guest'):
      #...connection code here...

In a Cmd method, you’re just going to define it like we did the do_hello method above: it takes ’self’ and ‘args’, where ‘args’ is one long line.

A couple of quick workarounds I’ve tried:

Parse the line yourself. I created a method in my Cmd app called ‘parseargs’ that just takes the big long line and returns a dictionary. My specific application only takes ‘name=value’ arguments, so I do this:

         d = dict([arg.split('=') for arg in args.split()])

And return the dictionary to the calling method. My connect method can then check for keys in the dictionary and set things up. It’s longer an a little more arduous, but not too bad.

Use optparse. You can instantiate a parser right inside your do_x methods. If you have a lot of methods that all need to take several flags and args, this could become cumbersome, but for one or two it’s not so bad. The key to doing this is creating a list from the Big Long Line and passing it to the parse_args() method of your parser object. Here’s what it looks like:

class MyShell(cmd.Cmd):
   def do_touch(self, line):
      parser = optparse.OptionParser()
      parser.add_option('-f', '--file', dest='fname')
      parser.add_option('-d', '--dir', dest='dir')
      (options,args) = parser.parse_args(line.split())

      print "Directory: %s" % options.dir
      print "File name: %s" % options.fname

This method is just an example, so don’t scratch your head looking for “import os” or anything

This is probably the more elegant solution, since it doesn’t require you to restrict your users to passing args in a particular way, and doesn’t require you to come up with fancy CLI argument parsing algorithms.

Using urllib2 for Pure XML Over HTTP

I wrote a web service client this week that does pure XML over HTTP to send queries to a service. I’ve written things like this before using Python, but it turns out, after looking back at my code, I was always either using XMLRPC, SOAP, or going through some wrapper that hid a lot from me in an effort to make my life easier (like the Google Data API). I’ve never had to try to send a pure XML payload over the wire to a web server.

I figured urllib2 was going to help me out here, and it did, but not before going through some pain due mainly to an odd pattern in various sources of documentation on the topic. I read docs at python.org, effbot.org, a couple of blogs, and did a Google search, and everything, everywhere, seems to indicate that the urllib2.Request object’s optional “data” argument expects a urlencoded string. From http://docs.python.org/library/urllib2.html?highlight=urllib2.request#urllib2.Request

data should be a buffer in the standard application/x-www-form-urlencoded format

The examples on every site I’ve found always pass whatever ‘data’ is through urllib.urlencode() before adding it to the request. I figured urllib2 was no longer my friend, and almost started looking at implementing an HTTPSClient object. Instead I decided to try just passing my unencoded data. What’s it gonna do, detect that my data wasn’t urlencoded? Maybe I’d learn something.

I learned that all of the documentation fails to account for this particular edge case. Go ahead and pass whatever the heck you want in ‘data’. If it’s what the server on the other end expects, you’ll be fine.

Decorators

I found myself in dark, dusty corners when I had to decide how and where inside of a much larger piece of code to implement a feature. I really wanted to use a decorator, and still think that’s what I’ll wind up doing, but then how to implement the decorator isn’t as straightforward as I’d like either.

Decorators are used to alter how a decorated function operates. They’re amazingly useful, because instead of implementing some bit of code in a bunch of methods that themselves live inside a bunch of classes across various modules, or creating an entire class or mixin to inherit from when you only need the code overhead in a couple of edge cases, you can just create a decorator and apply it only to the proper methods or functions.

The lesson I learned is to try very hard to make one solid decision about how your decorator will work up front. Will it be a class? That’s done somewhat differently than doing it with a function. Will the decorator take arguments? That’s handled differently in both implementations, and also requires changes to an existing decorator class that didn’t used to take arguments. I don’t know why I expected this to be more straightforward, but I totally did.

If you’re new to decorators or haven’t had to dig into them too deeply, I highly recommend Bruce Eckel’s series introducing Python decorators, which walks you through all of the various ways to implement them. Part I (of 3) is here.

I know this is my geek blog, but geeks have kids too, so first I want to announce the birth of our second daughter, Sadie, who was born on September 15th. She’s now over a month old. This is the first time I’ve stayed up late enough to blog about her. Everyone is healthy, if slightly sleep-deprived

New Job!

The day before Sadie’s birth, I got a call with an offer for a job. A *full-time* job, as a Senior Operations Developer for MyYearbook.com. After learning about the cool and very geeky things going on at MyYearbook during the interview process, I couldn’t turn it down. I started on October 5, and it’s been a blast digging into all of the cool stuff going on there. While I’m certainly doing my fair share of PHP code review, maintenance, and general coding, I’m also getting plenty of hours in working out the Python side of my brain. I’m finding that while it’s easier switching gears than I had anticipated, I do make some really funny minor syntax errors, like using dot notation to access object attributes in PHP ;-P

What I find super exciting is something that might turn some peoples’ stomachs: at the end of my first week, I sat back and looked at my monitors to find roughly 15 tabs in Firefox open to pages explaining various tools I’d never gotten to use, protocols I’ve never heard of, etc. I had my laptop and desktop both configured with 2 virtual machines for testing and playing with new stuff. I had something north of 25 terminal windows open, and 8 files open in Komodo Edit.

Now THAT, THAT is FUN!

The projects I’m working on run the gamut from code cleanups that nobody else has had time to do (a good tool for getting my brain wrapped around various parts of the code base), to working on scalability solutions and new offerings involving my background in coding *and* system administration. It’s like someone cherry-picked a Bay Area startup and dropped it randomly 30 minutes from my house.

My own business is officially “not taking new clients”. I have some regular clients that I still do work for, so my “regulars” are still being served, but they’ve all been put on notice that I’m unavailable until the new year.

New Car!

I’m less excited about the new car, really. I used to drive a Jeep Liberty, and I loved it. However, in early September, before Sadie’s arrival, it became clear to me that putting two car seats in that beast wasn’t going to happen. The Jeep is great for drivers, and it has some cargo space. It’s not a great vehicle for passengers, though.

At the same time, I was running a business (this was before the job offer came along), and I was finding myself slightly uncomfortable delivering rather serious business proposals in a well-used 2003 Jeep. So, I needed something that could fit my young family (my oldest is 2 yrs), and that was presentable to clients. So, I got a Lexus ES350.

I like most things about the car, except for the audio system. It seems schizophrenic to me to have like 6 sound ‘zones’ to isolate the audio to certain sets of speakers, but then controls like bass and treble only go from 0 to 5. Huh? And the sound always sounds like it’s lying on the floor for some reason. It’s not at all immersive. The sound system on my Jeep completely kicked ass. I miss it. A lot.

Other News

I’ve submitted an article to Python Magazine about my (relatively) recent work with Django and my (temporarily stalled) overhaul of LinuxLaboratory.org, and my experiences with various learning resources related to Django. If you’re looking to get into Django, it’s probably a good read.

I’ve been getting into some areas of Python that were previously dark, dusty corners, so hopefully I’ll be writing more about Python here, because writing about something helps me to solidify things in my own brain. Short of that, it serves as a future reference point in case it didn’t get solidified enough

My sister launched The Dance Jones, a blog where she talks about fitness, balance, dance, and stuff I should probably pay much more attention to (I’m close to declaring war on my gut). Also, if you ever wanted to know how to shoulder shimmy (and who hasn’t wanted to do that?), you should check it out

Me: Live in Princeton, NJ area. Over 10 years experience with UNIX/Linux administration, databases and data modeling, and PHP/Perl. About 3 years experience using Python for back-end scripting and system automation, and less than a year of Django experience. Former Director of Technology for AddThis.com (it was bought out), Infrastructure Architect at cs.princeton.edu, and systems consultant/trainer. Creator of Python Magazine, former Editor in Chief of both php|architect *and* Python Magazine, and co-author of “Linux Server Hacks, volume 2″ (O’Reilly).

You are one of these:

• Web graphic designer who has worked on several web-based projects for clients in various industries, understands current best practices and standards, has the tools and experience necessary to create custom graphics, and has some familiarity (secondarily) with PHP and/or Python, Javascript and Ajax. If you regularly make use of table-based web designs or ActiveX controls, this isn’t you.
• Hardcore web developer with at least 6 years experience doing nothing but web-based projects using Javascript and (at some point) *both* PHP and Python, and has worked with or has an interest in Django, Cake, and other frameworks, and understands that client needs often don’t coincide with the religion of fanboyism. Knowledge of Javascript, Ajax, web standards and security is essential here. If your last “big project” was volunteer work to build a website for your kid’s soccer team, this isn’t you.
• A generalist webmaster (sysadmin/db admin/scripter) with at least 6 years experience working in production *nix environments with good familiarity in the areas of high availability, web servers (specifically Apache), proxy servers and monitoring, and has worked with/supported users like the ones mentioned above on web-based projects. If you have to look at the documentation to figure out how to implement a 301 redirect, this probably isn’t you.

Experience working on a team in larger projects with multiple people would be good. Note that I’m looking for people to partner with on projects, I’m not hiring full time employees. Future partnership in a proper business is certainly a possibility, but… baby steps! I do have a couple of domains that would be great for use with this kind of project if it ever progresses that far

I know that other people are out there looking for people to partner with on projects, but there doesn’t appear to be a common place for them to interact. Maybe that can be a project we undertake together   — if there *is* a place where people meet up for this kind of thing, let me know!

Let’s have fun, and take over the world! Shoot me an email at “bkjones” @ Google’s mail domain.