Darrell Hawley: Home Page

Why I do Agile: The Swamp King's Dilemna

2010-10-21T10:22:00.000-07:00

The King of the Swamp People is not happy. Living in a swamp is really unpleasant. There’s mosquitoes and flies, alligators are always a danger and there is no dry ground to put buildings . So the King decides something must be done.

Managing the Project in a Traditional Way

The King gathers his workforce of cartographers, explorers and engineers around and tells them that the Swamp People are moving to the mountain. Why the mountain? Because the King has heard it’s dry and there aren’t any mosquitoes.

The cartographers immediately research the route using their library and begin drawing detailed maps to be compiled into a book. Once completed, the book is handed over to the explorers who begin blazing the trail to the mountain, faithfully following the book. It’s not long before the explorers encounter their first obstacle: a river. The cartographers knew the river was there and prepared for it. They immediately send the engineers to build the bridge they've carefully planned to handle the Swamp People and their belongings. The explorers find a rope bridge about a mile down river and use it to cross the river.

While the bridge is being finished, the explorers encounter their next obstacle: a desert. This isn’t on any map in the book so they send for the cartographers. The cartographers insist that there is no desert and the explorers must have gone in the wrong direction. This naturally makes the explorers angry because going in the right direction is what they are exceptionally good at doing. So one of the cartographers is sent to put everyone back on track. When he gets to the desert, the cartographer is silent. Clearly the explorers were right. He tells the explorers to stay there until the cartographers can develop a new plan back in the swamp. On his way back, the cartographer tells the engineers to stop work since they may not need the bridge after all.

The cartographers meet and decide they should build boats and attempt to reach the mountain via the river. They know they should first create all new maps before the explorers or engineers do anything, but the desert has really set them back. They tell the explorers to go downstream and the engineers to build boats.

As the explorers make their way down river they discover a road paralleling the river. They know they should tell the engineers about this but feel it’s more important for them to continue. When they get to the mouth of the river they find that it empties into an ocean. They send for the cartographers who once again insist that they’ve done something wrong. But of course, when a cartographer arrives, they immediately know the explorers are right. The cartographer tells the explorers to stay where they are and then informs the engineers they won't need boats since there is a road.

The cartographers confer and decide to have the engineers build ships and the explorers find their way to the mountain (forget about creating any new maps for them). The engineers, having never built an ocean going vessel before, head back to the library to learn how. The explorers, meanwhile, follow the beach to get to the mountain.

Despite their outwardly calm appearance, the cartographers are worried. The plan they originally laid out isn’t working and the Swamp king has no idea there have been any problems. They inform a very unhappy King that getting to the mountain is going to take longer than they originally anticipated.

Construction of a shipyard finally begins. The engineers, understanding that they are stepping out of their field of expertise, are still confident they can build ships that will transport their people and belongings. Unfortunately, however, the explorers have reached their destination and the news isn’t good. Sure the mountain is dry, but there is no drinking water. There are no mosquitoes, but the wind is fierce. The mountain is simply inhospitable.

The King is furious at the news. He’s spent a huge amount of resources and has nothing to show for it. He banishes the senior members of his workforce from the swamp.

An Agile Approach

The Swamp King gathers everyone around and begin discussing where the Swamp People should move. Many ideas are put forth but most people think that the mountain is a likely place to live. Being a wise leader, the King acknowledges that even though the mountain seems a good idea now, it may not be the best solution for his people. His declares his initial goal is to gather as much information as possible as quickly as possible. The King begins by selecting a cartographer, an explorer and an engineer to help him identify and explore a likely route to the mountain. The group picks a route to explore and departs.

When the team encounters the river, they immediately head back to the swamp to share their findings. The cartographer shows a rough map of their route, the explorer talks about the terrain and what supplies a migrating people may need and the engineer shares his thoughts on how they might build a bridge if it becomes necessary.

Two more teams are formed using members of the original team and some new people. One team is tasked with heading upriver and the other exploring downriver. Before they leave, an outpost is constructed on the shore of the river where supplies can be kept and messages sent. This also serves as an experiment in living in a different environment than a swamp. The cartographer from the original team stays at the outpost to refine his map. Because of the time he spent with the explorer and engineer, he clearly defines sources of food, water, and building materials as well as obstacles.

Not long after departing, the down river team discovers a rope bridge and immediately sends a message back to the outpost. A new team is formed to explore across the river.

Meanwhile, the upriver team returns to report that there is a town several miles away with a bridge suitable for moving a large number of people and their belongings. Envoys are sent to the town to begin negotiating use of their bridge.

The downriver team returns to report that the river empties into an ocean. To get to the mountain using this route, ships will have to be built. The engineers are concerned that they have no experience with constructing sea-worthy vessels. When the cross river team returns and reports they’ve run into a desert, the King realizes it may not be possible to get to the mountain., However, he has seen that it is possible to move his people to the river. The outpost has proven to be self-sufficient; the river provides plenty of food and water and the surrounding trees provide lumber. Sure there are mosquitoes, but there isn’t nearly as many as back in the swamp. It may not be the perfect solution, but it would substantially improve his people’s quality of life. The teams all agree. New teams are created to begin building the new town and to prepare the Swamp People for the migration to the river.

Observations

In the traditional model, the most important thing was that each group of people were asked to solve predefined problems. In the agile model, each group was asked to find problems and suggest possible solutions.
The second model accomplished much more in less time than the first and used fewer resources. In roughly the same amount of time it took the first model to find the desert and build part of a bridge, the second model found the desert, the ocean and the town, built an outpost and sent envoys the town.
It seems that the second model is waterfall since everyone is working toward building documentation, but that’s not really true. A map is simply an artifact of a team’s experiences during their travel. I would equate these missions with programming spikes.
In the first model, the King banishes his most senior workers leaving behind an inexperienced workforce further hurting his chances of improving his people's situation.
In the second model, innovation happened. The cartographer saw what was important to the explorer and the engineer and created a way of showing that on a map. The only way this could happen is if the cartographer was working side by side with the explorer and the engineer.
The outpost in the second model was clearly intended to be a temporary structure. It was meant to help the teams complete their work and learn something of the area, not to serve as part of the migration of the Swamp People.
The number of people involved in the migration initially was small for the second model: the King, the cartographer, the explorer and the engineer. When the scope of the project increased and more people were required, teams were reformed with a mixture of old and new workers.
In the first model, the cartographers did not trust what the explorers told them. This was not true of the second model.
The amount of waste in the Traditional model is significant. The engineers designed and started a bridge that would never be used, the cartographers drew maps of areas that were later proven to be completely inaccurate, both the explorers and the engineers spent a lot of downtime while they waited for the cartographers to plan a new strategy, the engineers built boats they would never use and then later began work on a shipyard that was never needed.

Note to Self 14: Blogger FTP Problems

2010-03-30T09:21:00.001-07:00

Way back in January, Google announced that they will no longer be supporting FTP publishing. Unfortunately for me, I was apparently one of only 0.5% of all people who actually relied on this offering. I finally buckled down and solved the problem. Here’s what I learned:

The migration tool they offer (see the yellow box at the top of your blogger dashboard) is not going to work for you if you want to use a custom domain. You’ll have to use a manual process.
Though the process is not difficult, you’ll want to do a bit of reading before diving into this. Making a mistake could cause you more grief than you really want. Here’s an article that I found particularly helpful.
If you have downloads or images on your site that you want to maintain, they won’t be part of the migration. You’ll have to setup a missing files host. A missing files host is just another place to look in the event your page contains broken links. Blogger requires that your missing file host is a subdomain. I’d recommend you read this article to learn more.
In the Blogger Publishing Settings, I set my domain name to www.darrellhawley.com so that I could maintain my original urls while my missing file host was set to www2. I was concerned that this might cause my problems, but I was pleasantly surprised.
Repointing my domains is something that I always dread and this was no different though it appears I had nothing to worry about. I pointed my www2 subdomain at my original host and my naked domain (i.e., darrellhawley.com as opposed www.darrellhawley.com) at the IP addresses provided by Blogger.

Note to Self 13 – Whole Lot of Django

2010-02-26T13:52:00.001-08:00

Testing Django can be somewhat challenging. Instead of simply running a unittest module, you need to run the manage.py script in your Django app passing it the “test” parameter. This sets up a test database for your model, runs your tests against that model and then destroys your database. In effect, it turns your unit tests into functional tests.
Another item regarding testing that bothers me is the hoops you have to jump through in order to use a testing framework other than unittest. We want to use Nose for our project and getting it to run in the Django environment requires that we install a Django App (django-nose) in our Project, and then make a couple of changes in our project settings file. I admit it doesn’t sound too bad, but when you compare it to the ease of simply running “nosetest” the story is not nearly as compelling.
Putting testing aside for the moment, the rest of Django has been a pretty good experience. Both the templating language and the models are simple to learn and start using right away. Even though Django favors configuration over convention, the amount of configuration necessary is relatively small and hasn’t been much of a burden.
The main configuration hassle has been mapping urls. The process can be simple, but leaves the door open for something much more elaborate. This is one of those things where I wish they would looked closely at Rails and adopt some of their practices: use conventions until it configurations become necessary.
Going back to the manage.py file for a moment, I should say there are some cool things that you can do here. For those Rails developers out there, some of the functionality of Rake is contained in manage.py though certainly not all of it. Though the most common use of manage.py is syncing the database with your model (syncdb) and starting your development server (runserver), I find the shell command to be extremely useful for test driving models I’ve created.
Though not specifically Django related, I feel Nosy is worth mentioning. I’ve become addicted to this script (go here for the original post) written by Jeff Winkler a few years ago. Like many others, I’ve taken the original script and modified it suit the needs of my project. In short, Nosy is a script that runs Nose anytime a python file in a given directory changes. It’s essentially like having a mini-ci server running on your desktop. I’ve found it very freeing to simply write code and be instantly notified if something breaks. No switching windows to run a test.
As I mentioned before, the testing story for Django is not ideal. But this doesn’t mean you can’t improve the situation. I’ve made sure that the code going into our views.py file (which interestingly enough is really a controller) as close to pure framework code as I can. Most of my complexity is stored in a separate module that can be easily tested with Nose. To allow for even better testing, I make sure that all of the methods in the extra module accept a model as a parameter making mocking the models possible.
In summation, Django absolutely makes creating websites much easier. Though testing is challenging, you can improve upon it if you put some though into it. Would I recommend Django? If you’re already comfortable in Python, than absolutely. If not, I’m not so sure. Rails is a compelling option considering it’s preference of convention over configuration. Keep in mind the required configurations in Django are very obvious whereas “stepping off the Rails” in RoR takes a bit knowledge of the framework.

Django at the Ann Arbor Software Development Study Group

2010-01-24T13:21:00.001-08:00

I was supposed to lead a jam last December at the Ann Arbor Software Development Study Group, but a client project postponed it. On Tuesday, February 2, we’ll try one more time. The topic is “Zero to Django: Writing Web Apps with Python Using the Django Framework”. By the end of the session, you will be collecting form data and redisplaying it on a webpage.

NOTE: This exercise is meant to get you collecting data quickly and I’ll be skipping a lot of explanations. Feel free to ask questions or research any areas that you may find particularly tricky.

On with the jam!

Pre-Requisites:

Though there is only one requirement for this session - installing Django - you'll want to get this done ahead of time. It's not difficult, but there are a few steps and it will take enough time that it could keep you from completing all the exercises. To install Django, go to http://docs.djangoproject.com/en/dev/topics/install for a detailed description of what you need to do. Before following that link, here's a couple of suggestions:

Be sure to install Python 2.5 (Python 2.6 will probably be fine). Django will not run with the 3.x series of Python.
You can skip installing Apache and mod-wsgi. This session is only about how to use the framework which can be done completely within the development environment provided by Django.
You can also skip the section on getting a database running. We'll be using SQLite which is baked into Python 2.5 and later versions.
Be sure to install an official release of Django. Under "Installing an official release", click on "download page" in line item 1. Use the version under "Option 1: Get the latest official version".

The Exercise:

Create a folder where all of your Django apps can you live. I'm using Windows and I'm putting all of my Django projects in the C:\django directory.

From the command line, navigate to the directory you just created
from the command line, run "python django-admin.py startproject people". You may have to explicity give the path of the django-admin.py file which can be found in python25/Lib/site-packages/[django dir]. If you copied the file as suggested by the installation directions, just use that version of the file instead. Also, you may get a "permission denied" message. If that's the case, check your permissions on the folder.
If you successfully completed step 3, you should now have a "people" directory. On my machine, the path is c:\django\people.
Navigate to the directory mentioned in Step 4 and run "python manage.py runserver"
Open up a browser and navigate to "http://localhost:8000/people". You should see a message congratulating you on "your first Django-powered Page"

Now that you’ve created a page, let’s configure our site so that we can do something useful.

Django Projects contain one or more Apps (OK, they don’t have to contain Apps, but if you want to connect to a database they do). To create an app, run “python manage.py startapp dataentry”. The “manage.py” file can be found in the root of the people directory you created earlier. On my machine, the path is C:\django\people\manage.py.
In the “settings.py” file contained within the project root, configure the database. For this exercise we’ll use SQLite since it comes bundled with Python 2.5 and all later versions:

DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = 'peopledb'
DATABASE_USER = ''
DATABASE_PASSWORD = ''
DATABASE_HOST = ''
DATABASE_PORT = ''

In the settings file, you need to add your app to the INSTALLED_APPS section. Your INSTALLED_APPS section should look like the following (note the “people.dataentry” on the last line):
INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.admin',
    people.dataentry',
)
Again in the settings file, add “"/templates",” to the TEMPLATE_DIRS setting.

Don’t worry about the extra lines. Those go beyond the scope of this session. If they really bother you, you can remove them. Just be aware that you’ll need add the extra comma (,) at the end so Python understands that this is a single item tuple.

Now that the configuration is done, it’s time to setup your database. To do that, we’ll need to open up model.py file inside your application directory. On my machine, that’s “c:\django\people\dataentry\model.py”. Your model should like the following:

from django.db import models

class Person(models.Model):
    T_SHIRT_SIZE_CHOICES = (
        ("WS","Women's Small"),
        ("WM","Women's Medium"),
        ("WL","Women's Large"),
        ("S","Small"),
        ("M","Medium"),
        ("L","Large"),
        ("XL","X-Large"),
        ("2XL","2X-Large"),
        ("3XL","3X-Large"),
    )
    name = models.CharField(max_length=30)
    t_shirt_size = models.CharField(max_length=10,
        choices=T_SHIRT_SIZE_CHOICES)
    special_dietary_concerns = models.BooleanField(
        default=False)
    special_dietary_concerns_comments = models.TextField(null=True,
        blank=True)

    def __unicode__(self):
        return "<Person %s %s %s>" % (
            self.name,
            self.t_shirt_size,
            self.special_dietary_concerns)

Now that your model is in place, let’s make sure that it’s reflected in the database.

From the command line in the directory where the model.py file lives (“c:\django\people\dataentry\model.py” on my machine), run “python manage.py syncdb”. A series of tables will be created.
You will be prompted to create a superuser. Type “yes” and continue through the prompts.

That’s it! The Person class is now in the database. But how do you use it.? To test it out, run “python manage.py shell” from the command line. You’ll be placed into Python’s Interactive Console. Now run the following commands

from dataentry.models import Person
p = Person()
p.name = “Me”
p.t_shirt_size = “L”
p.save()
Person.objects.get(id=1)

That last command should have printed “<Person Me L False>” in the console. Assuming it did, how do we get this form on a webpage? Before we do that, let’s make a webpage first.

Create a new folder called “templates” in the “c:\django\people\dataentry” directory.
In the “c:\django\people\dataentry\templates” directory, create a new file called “template.html”
In the template.html file, paste the following snippet “<h1>{{ hello }}</h1>” . Save the file.

You’ve just created a very basic template. To use it, make sure that your views.py file looks like the following:

from django.shortcuts import render_to_response

def hello(request):
hello = "Hello, World"
return render_to_response("template.html", locals())

Open “c:\django\people\urls.py” and make sure the following code is in it:

from django.conf.urls.defaults import *
from people.dataentry import views

urlpatterns = patterns('',
(r'^hello/$', views.hello)
)

All the pieces should be in place for you to actually view your webpage. From the command line, run “python manage.py runserver”. Open your favorite web browser and navigate to http://localhost:8000/hello. You should be rewarded with “Hello, World” in a very large font. That’s all well and good, but what about that model we created? Let’s put it to good use right now. Let’s create a list.html in our template directory that looks something like the following:

<html>
    <head>
        <title>{{ title }}</title>
    </head>
    <body>
        <table>
            <tr>
                <th>Name</th>
                <th>T Shirt Size</th>
                <th>Special Diet</th>
                <th>Special Diet Comments</th>
            </tr>
        {% for person in people %}
            <tr>
                <td>{{ person.name }}</td>
                <td>{{ person.t_shirt_size }}</td>
                <td>{{ person.special_dietary_concerns }}</td>
                <td>{{ person.special_dietary_concerns_comments }}</td>
            </tr>
        {% endfor %}
    </body>
</html>

Now let’s create a new method in our view so that our views.py file looks like the following:

from django.shortcuts import render_to_response
from people.dataentry.models import Person

def hello(request):
    hello = "Hello, World"
    return render_to_response("template.html", locals())

def list_people(request):
    title = "List of People"
    people = Person.objects.all()
    return render_to_response("list.html", locals())

Switch to the urls.py file and wire a url to our new method. Our updated urls.py file should look like the following:

from django.conf.urls.defaults import *
from people.dataentry import views

urlpatterns = patterns('',
(r'^hello/$', views.hello),
(r'^list/$', views.list_people),
)

From your web browser, navigate to http://localhost:8000/list. You should see the person you entered in the Interactive Console. Cool! By now you should see a pattern on how to create a new page in Django: create a template, create a new method in the view and then edit the urls.py file. Let’s add a bit more complexity by creating a form. Create a new forms.py file in the “C:\django\people\dataentry” directory. Put the following code block in that new file:

from django import forms
from models import Person

class PersonForm(forms.ModelForm):
class Meta(object):
model = Person

Create a new template in the template folder called “form.html” and put the following html inside it:

<html>
    <head>
        <title>Adding a Person</title>
    </head>
    <body>
        <form method="post">
            <table>
                {{ form.as_table }}
            </table>
            <input type='submit'>
        </form>
    </body>
</html>

Switch over to your view.py file and modify it so that it looks like the following:

from django.shortcuts import render_to_response, HttpResponseRedirect
from people.dataentry.models import Person
from people.dataentry.forms import PersonForm

def hello(request):
    hello = "Hello, World"
    return render_to_response("template.html", locals())

def list_people(request):
    title = "List of People"
    people = Person.objects.all()
    return render_to_response("list.html", locals())

def add_person(request):
    title = "Add Person"
    if 'name' in request.POST:
        form = PersonForm(request.POST)
        if form.errors:
            errors = form.errors
        else:
            form.save()
            return HttpResponseRedirect("../list")
    else:
        form = PersonForm()
    return render_to_response("form.html", locals())

Now all you have to do is wire up the url in the urls.py file. Now when you successfully add a new person using the form you will be redirected back to the list of people where you will see your entry at the bottom of the list.

IterHelper Released

2009-12-29T09:53:00.001-08:00

I’ve mentioned IterHelper in my last few posts and I’m glad to say that I finally released a version of it that I feel is solid. I’ve run my unit tests on Python versions 2.5 and 2.6 and also on IronPython 2.0 and 2.6.

It doesn’t do anything earth-shattering at this point, but it does have some functions I find particularly useful such as a couple of filter methods and a number of “skip” and “take” methods. For a complete rundown of what it does, just run “help(IterHelper)”

I’ve used LaunchPad to manage my project and the experience has been good thus far. If you want to download it, you’ll need to do so from the trunk. I’m eventually moving to Python eggs, but until I work out the particulars, you’ll have to do include it in your project manually.

Euler Problem 1 Revisited

2009-11-29T17:22:00.001-08:00

The other day at the Software Development Study Group in Ann Arbor, Chris Marinos was demonstrating F#, a functional programming language targeting the .NET framework. Of course, this got me thinking about programming styles in Python. So what better way to compare and contrast programming styles than with Euler problems? As you probably don’t recall, the first Euler problem is:

If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23.

Find the sum of all the multiples of 3 or 5 below 1000.

The traditional approach to this might look something like the following:

def traditional_approach():
    total = 0
    for i in range(1,1000):
        if i % 5 == 0 or i % 3 == 0:
            total += i
    return total

It’s not a bad approach, but there are more concise approaches. Here is my original solution using a list comprehension:

isValid = lambda x: x % 5 == 0 or x % 3 == 0

def with_list_comprehension():
filteredValues = [x for x in range(1,1000) if isValid(x)]
return sum(filteredValues)

I like list comprehensions, but this feature does introduce a lot of noise characters. If you use list comprehensions, you probably like this solution, otherwise you’re looking for something else. The next approach uses the itertools module (and the isValid lambda from the previous example).

def with_itertools():
filteredValues = itertools.ifilter(isValid, range(1,1000))
return sum(filteredValues)

This approach removes a couple of noise characters and is structured in a way familiar to most developers. My last solution uses a module called IterHelper that I proposed in a previous post. As I mentioned in that post, IterHelper sits on top of the itertools module and implements some of the recipes from the official itertools documentation. It’s largely inspired by my desire to use LINQ from both Python and IronPython. Here’s my solution to Euler Problem 1 using IterHelper:

def with_iterhelper():
    return IterHelper(range(1,1000))\
        .where(isValid)\
        .sum()

Thanks to the Python continuation character, a backslash, I can put discreet units of functionality on their own line, making it easier for me to consume.

If you decide to download and try out IterHelper, be aware that I’m not even considering it alpha yet. It’s still very much in the R & D phase and I’m planning on changing some of the method names.

Note to Self 12 – Lessons Learned from Our Latest Project

2009-10-28T14:14:00.001-07:00

I know I’m supposed to be releasing my IterHelper tool right now, but I’ve been on a bit of death march lately and have had no time. Though the project hasn’t ended yet, there are some items that I need to store away for future use before they’re forgotten.

On small teams, non-development related responsibilities should be divided among the team. Then the single non-dev responsibility (hopefully it’s the only one) should always take precedence over development responsibilities. It sounds backwards, but no matter what my priorities, I’m always going to find a way to write code. The same isn’t true of project management.
I can’t say enough about having a non-developer - who I’ll refer to as the Customer Advocate – on the project team. Things we had problems making time for – client meetings and acceptance testing for instance – fell to the Customer Advocate which takes a lot of pressure off everyone else.
Story points are great, but in order to get value out of them you actually have to use them – obvious, I know. Estimating tasks and then tracking how many points were completed daily, turned an out of control project into a manageable one.
Speaking of scoring, I wish we would have re-scored each task after we completed it. It would have taken minimal effort while at the same time providing us with enough information to improve our estimates.
We started using Agile Zen at the beginning of our project, but eventually abandoned it in favor of physical notecards. This introduced a lot of challenges, but it did allow us to grow our process exactly the way we wanted it. The added benefit is we’re in a much better position to evaluate the various kanban applications for our next project. To be clear, Agile Zen is a great product it just wasn’t what we needed at the time.
We didn’t use Selenium or Watir to run acceptance tests. Even if we couldn’t run all the tests we wanted, it would have been a huge time saver. A costly mistake that won’t be repeated.

Does IronPython need LINQ?

2009-09-10T15:17:00.001-07:00

Before I get started, I think I need to clear up some misconceptions that the Python community might have about LINQ. From the posts I’ve read, Python developers have dismissed LINQ as only a series of language enhancements that allow for SQL-like querying of lists. Though it is true that both C# and VB.NET have made changes to their respective languages for working with lists, you can still use LINQ without the aforementioned language enhancements. Python developers would be better served to think of LINQ as wrappers around the map, filter and reduce functions. In fact, LINQ is not much different than the “recipes” found in the official itertools documentation.

So the question still stands: does IronPython need LINQ? In my last post, I described how to use LINQ from an IronPython application and implied that I didn’t think it was the best idea. Don’t get me wrong, I really enjoy writing LINQ statements, I just don’t think LINQ fits well with IronPython. Why not? The first problem I have with the IronPython/LINQ combination is put on prominent display in this code sample.

print Enumerable.Average[object](
list, Func[object, int](lambda x:x))

If you are a static language aficionado, you probably are wondering what the problem is. If you’re a Python developer, however, the fact that I’m explicitly naming the “object” and “int” types is sending up giant, red flags. Python folks don’t really care what type of object it is they’re using, they only want to know what the object can do and what it contains (i.e., duck-typing). It’s much easier – and more readable - to simply write a completely Python version of the Average function. Consider the following chunk of LINQ code from C#:

List<string> values = new List<string>("the quick brown fox jumped over the lazy dogs".Split());

double avgLengthOfLongWords = values
                .Select(x => x.Length)
                .Where(x => x > 3)
                .Average();

Console.WriteLine(avgLengthOfLongWords)

To convert that to IronPython, we’d be forced to write something like the following:

import clr
clr.AddReferenceToFileAndPath("c:\users\darrell\desktop\System.Core.dll")

import System.Linq
from System import Func
from System.Linq import Enumerable

stuff = "the quick brown fox jumped over the lazy dogs".split()

def Select(col, func):
    return Enumerable.Select[object, object](col, Func[object, object](func))
def Where(col, func):
    return Enumerable.Where[object](col, Func[object, bool](func))
def Average(col):
    return Enumerable.Average[object](
    col, Func[object, float](lambda x:x))

avgLengthOfLongWords = Average(Where(Select(stuff, lambda x:len(x)), lambda x:x > 3))
print avgLengthOfLongWords

Before I’m even able to write useable code, I need to add a reference to a dll, import a namespace and a couple of classes and then write some helper functions; all so that I can write LINQ statements without the benefits of chaining. That’s right – I have to nest my LINQ statements. No chaining. That’s because when we use IronPython, the Select and Where don’t return an IEnumerable or an IQueryable. Instead they returna WhereSelectEnumerableIterator.

But do you know what really rankles me about using LINQ in IronPython? It contributes absolutely nothing to the Python community. The only way you can use LINQ is if you are using IronPython, not CPython, not Jython. I want to engage the entire Python community, not just my corner of the world. If ever there was a case where implementing the principle trumps migrating the implementation, this is it.

Enter the itertools module. Itertools is a Python standard library module containing tools for working with iterators. It also will serve as the cornerstone for my implementation of LINQ. When you visit the official itertools documentation at python.org, you’re even presented with code “recipes” that could be described as LINQ-ish. The only problem is that the recipes only accomodate nesting of statements. Why is this a problem? Consider the differences in readability between nested and chained in the following example:

stuff = "the quick brown fox jumped over the lazy dogs".split()

getLength = lambda x:len(x)
isNotTooLong = lambda x:x<6
isNotTooShort = lambda x:x>3

nested = ifilter(isNotTooShort, ifilter(isNotTooLong, imap(getLength, stuff)))

chained = IterHelper(stuff).select(getLength).where(isNotTooLong).where(isNotTooShort)

As always is the case with nesting methods, you have to read the statement populating nested backwards. That is, the outermost function call is the last to be run and you work your way inwards from there to find the origin. This is not a big deal to most of us since this is something we had to get used to very early in our careers. But when you compare that to the statement populating chained, I know I can read it from left to right. My eyes aren’t jumping back and forth to understand what is happening in what should be a relatively simple statement.

So what does IterHelper look like?

class IterHelper(object):
    def __init__(self, someIterable):
        self.someIterable = someIterable
    def __iter__(self):
        return self.someIterable
    def next(self):
        return self.someIterable.next()
    def select(self, func):
        return IterHelper(itertools.imap(func, self.someIterable))
    def where(self, predicate):
        return IterHelper(itertools.ifilter(predicate, self.someIterable))

This extremely simplistic version of IterHelper is really only a wrapper around functions found in the itertools module. This is great news since any application written using Python 2.3 or later can make use of this pattern with little effort. There’s no special libraries AND everyone is invited regardless of your flavor of Python.

My plan is to release IterHelper as an open source project by sometime by mid-October. I have a well-tested, basic implementation now, but I still have questions regarding licensing, hosting and deployment. I will be doing a bit of research on these topics and be posting what I’ve learned in my Notes to Self series.

IronPython and LINQ: A Devlink Correction

2009-08-15T20:19:00.001-07:00

During my “Anatomy of an IronPython Application” presentation at Devlink, I mistakenly said that LINQ can be consumed from IronPython easily. I said this because I was thinking that IronPython 2.6 was built using .NET 3.5 when in fact it’s built with .NET 2.0. Fortunately, if you want to use LINQ, you still can because .NET 3.5 is really .NET 2.0 plus extensions; you just have a bit of extra work to do. First on the todo list is to copy the System.Core.dll for 3.5 to some other location (I was able to get to it from it’s home in the c:\windows\assembly directory) and then use the “clr.AddReferenceToFileAndPath” in your IP code to gain access. Then, with a simple import of the Enumerable or Queryable type, you magically have access to LINQ.

Still, the story doesn’t end there. You have to remember that IronPython does NOT support extension methods and since LINQ is pretty much a collection of extensions methods, there’s a problem. Why this design decision? Because in order to support extension methods, the IronPython team would have had to make changes to the core language which would have, of course, been rather un-Pythonic. This means that the code for consuming LINQ is going to be somewhat messy. The following snippet gives an example of consuming the Average function of the Enumerable object.

clr.AddReferenceToFileAndPath(r"C:\folder\System.Core.dll")
from System.Linq import Enumerable
from System import Func

list = [1,2,3,4,5]

print Enumerable.Average[object](
list, Func[object, int](lambda x:x))

I have a lot of concerns as soon as I see “object” and “int” in brackets. At this point I’m going to hold save those concerns until my next post. Instead I’m going to leave this as an example of consuming LINQ from IronPython.

I should point out that Harry Pierson’s post on IronPython and Linq-to-XML really got me started on this topic. I also have to give a shout-out to Chris Smith who was kind enough to help me out and relay a couple of CodeMash stories at the same time.

Speaking at the Lansing Day of .NET on Saturday

2009-07-31T07:31:00.001-07:00

If you’re going to be at the Lansing Day of .NET this Saturday, stop by and catch my Anatomy of an IronPython Application presentation. I’ve completely rewritten the talk so if you’ve seen it before, this new version will be covering completely different topics including testing and mocking. I’ll also be giving away a copy of IronPython in Action, courtesy of Manning Publications, at the end of my talk.

Note to Self 11 – Patterns in Python, Process is Everything, Selenium

2009-07-25T05:01:00.001-07:00

While studying a particular portion of IronPython in Action, I came across a footnote to a blog post entitled Patterns in Python. By the end of the post, I found myself mentally tweaking a lot of my code. Though quite old (the page was posted February of 2003), the content remains quite relevant.
I’m fortunate to be working on a project where everyone involved is committed to being agile (note the lack of a capital “A”). We’ve done enough things right that the items I would do differently are very minor. With that said, the imbalance in hours spent by each team member seems to be causing a problem…
Another interesting observation – the project that I feel very fortunate to be working on is not using the technology of my choice. It’s a Java project. In fact, almost every facet of the project requires some specific technology in which I almost know nothing, yet the development process makes the experience a joy.
Configuring my Selenium tests to work with Selenium RC and IE8 has not been an easy task. Some things I learned along the way…
- Easiest way to create a new Selenium test is to use the Selenium IDE plugin for FireFox. It records everything you click and do and allows you to convert the recording to C#, Java, Python, PHP or Ruby. From there, further customizations are a snap.
- Selenium RC allows you to run your Selenium tests in browsers other than FireFox, such as IE, Chrome and Safari.
- Before running your tests in one of these “other” browsers, you’ll need to start the Selenium Server. Selenium Server comes parceled with the Selenium RC download and can be started from the command line with “java -jar selenium-server.jar”.
- To run my Selenium tests with IE8, I had to use the *iexploreproxy option even though I’m not using a proxy. Otherwise I kept getting redirected to a phantom file supposedly hidden away in my user directory.

Book Review: IronPython in Action

2009-07-14T08:24:00.001-07:00

IronPython in Action (IPIA) gets the distinction of being the first book published on IronPython. As the first book, it has the responsibility of giving developers their first impression of IronPython. IPIA does exactly that by taking numerous shallow dives into various technologies and techniques. To be clear, this book is aimed at professional developers preferably with either .NET or Python experience. Pythonistas will find the book offers quick introductions to a number of Microsoft technologies such as Windows Forms, WPF and ASP.NET. .NET developers will get the chance to see technologies they’re already familiar with expressed in a completely new fashion.

IPIA avoids the trap of simply being a brochure of Microsoft technologies using “IronPython” in place of “C#” and “VB.NET”. Instead, the authors mix in chapters on development techniques ranging from the familiar - such as unit testing and mocking – to the more exotic such as metaprogramming. As .NET developers wind their way through the book, they’ll begin to appreciate the differences between the static and dynamic worlds – a crucial step toward wide ranging acceptance.

If you’re curious about what IronPython can do for you, IronPython in Action is well worth your time.

Note to Self 10 – JQuery Selectors and Custom Functions and a thought on JQuery

2009-06-28T09:01:00.001-07:00

After using Python, diving into jQuery has proven exceptionally difficult. Even though it solves the not-so-small problem of commonizing the DOM across multiple browsers, I find it very difficult to read. I know with time I’ll get used to it and maybe even find some beauty in my jQuery code, but it will never stop me from wishing for a cleaner syntax like that of Python or Ruby.

With that said, here’s a bit of what I learned about jQuery recently.

Selectors in jQuery are a syntax for selecting an html element. Selectors can return a single element or several (or none if nothing matching is found).
$(“#[elementID]”) – finds an element with an id attribute equal to “elementID”.
$(“[name=’elementName’]”) – gets all elements with a name attribute equal to “elementName”
$(“[name^=’element’]”) – gets all elements with a name attribute that begins with “element”
$(“[name$=’Name’]”) – gets all elements with a name attribute that ends with “Name”
$(“input:checkbox”) – gets all checkboxes
$(“#blah > input:checkbox”) – gets all checkboxes within a parent element of id “blah”
Creating a custom function in jQuery:

$.fn.CustomFunctionName({function()({
//code here
})

$.fn.exclaim = function(message){
alert(message)
}
$.fn.exclaim("hello, world")

Note to Self 9 – Exploring SharpDevelop and Going off Topic

2009-05-29T07:05:00.001-07:00

It’s been too long since I last blogged which typically means I have too many competing ideas to settle on just one. So welcome to Note to Self 9!

SharpDevelop is an open source IDE for the .NET platform. Is it new? Hardly. Based on the history of news releases from their website, It’s been around since December of 2000. Considering that the .NET platform was publicly announced only 6 months earlier, SharpDevelop seems only that much more mature.
If you’re comfortable with the basics of Visual Studio, SharpDevelop should feel like home to you. In fact, the best way to describe SharpDevelop is “Visual Studio without a lot of stuff I don’t want and a few thid-party items I do”.
SharpDevelop uses NUnit instead of MSTest. That’s right. No third party plugins to make your IDE work with NUnit. It simply works.
It’s time for me to confess. It’s all about IronPython. The reason I started investigating SharpDevelop is because of my frustration with Visual Studio’s lack of support for the language. Specifically, I want Intellisense when I write IronPython. To be sure, you only get Intellisense for “non-DLR code”, but surprisingly that doesn’t seem to be as much of a handicap as you might think.
Speaking of first class citizens, SharpDevelop comes stock with C#, Visual Basic, Boo (I can hear Jay Wren cheering now), IronPython and F# (now a number of SRT folks are cheering).
If you’re frightened of the consequences that may come from changing your development environment, don’t be. You can open one of your existing Visual Studio solutions with SharpDevelop and start coding. Switching back is as simple as opening up your now modified project/solution in Visual Studio.
For clarity, SharpDevelop is new to me. I haven’t used SharpDevelop for any projects yet so I can’t tell you about any of its nuances. I plan on putting it to the test, however, over the next several weeks. I’ll be posting about my experiences.
Now for the off-topic part. I’ve been listening to The History of Rome, a FANTASTIC podcast series on, well, the history of Rome. Mike Duncan, creator and narrator, is not only an engaging story teller, but also has the technical savvy to put together a professional podcast. Though it’s not the sort of podcast typically on a geek’s menu, I think there are enough parallels to some of the challenges we are facing in the software industry to make it worthwhile.

Anatomy of an IronPython Application at AADND

2009-04-07T20:00:00.001-07:00

I'm going to be talking IronPython at the Ann Arbor .NET Developers Group at SRT Solutions tomorrow night (April 8) at 6:00. I'll be showing you how you might introduce IronPython into your existing .NET environment through a lot of sample code and very few PowerPoint slides. I hope to see you there.

Anatomy of an IronPython Application

You've heard about IronPython and maybe have even taken the time to write the obligatory "Hello, World" application. Now what? The buzz around IronPython continues but there's still little guidance on tools and development processes. This presentation focuses on the basics of interacting with C# and VB.NET libraries using IronPython while introducing useful tools and techniques to get you started. Though experience with .NET is necessary, the samples and discussions are understandable to even beginning Python developers.

Writing an IronPython WCF Host

2009-03-15T06:19:00.001-07:00

This is a simple WCF host written in IronPython that consumes a library I wrote in C#.

import clr
clr.AddReferenceByPartialName("System.ServiceModel")
clr.AddReferenceByPartialName("CSharpClass")

from System.ServiceModel import ServiceHost, BasicHttpBinding
from System.ServiceModel.Description import ServiceMetadataBehavior
from CSharpClass import PersonService, IPersonService
from System import Array, Uri

uri = Uri("http://localhost:1234/people")
uriArray = Array[Uri]([uri]) # creates a generic array of type Uri
serviceHost = ServiceHost(PersonService, uriArray)

binding = BasicHttpBinding()

serviceHost.AddServiceEndpoint(IPersonService, binding, "http://localhost:1234/people")

behavior = ServiceMetadataBehavior()
behavior.HttpGetEnabled = True
serviceHost.Description.Behaviors.Add(behavior)

serviceHost.Open()

raw_input("Service has been started. Press Enter to exit...")

Running nose with IronPython

2009-03-06T11:28:00.001-08:00

I just spent the past week at the 2009 Microsoft MVP Summit where I got a chance to talk to the some of the crew behind the dynamic languages at Microsoft. When I mentioned that I had gotten nose, a Python unit testing framework, working with IronPython, I was strongly encouraged to blog about it. Without further adieu, here's my code.

import sys
sys.path.append(r"C:\Python25\Lib\site-packages\nose-0.10.4-py2.5.egg")
sys.path.append(r"C:\python25\lib")
sys.path.append(r"C:\Python25\Lib\email")
sys.path.append(r"C:\Python25\Scripts")
import nose
nose.main()

I dropped this script into a directory with a couple of bogus unit tests and ran it with Python 2.5.4. Results were consistent with my expectations - successful tests passed and failing tests threw exceptions. When I ran the same suite against IronPython 2.0.0, I still received the correct results but the script took 1 minute and 20 seconds to complete. Compare that to less than a second (estimated) for CPython. For the sake of completeness, I ran the same test with IronPython 2.0.1 and received the same result.

The root of the problem appears to be the large number of GeneratorExitExceptions being thrown. I'm not sure why they're being thrown, but I'll continue to investigate and report my findings. If you're interested in the gory details, here's the stack trace of the first exception:

at IronPython.Runtime.PythonGenerator.ThrowThrowable()
at IronPython.Runtime.PythonGenerator.CheckThrowable()
at IronPython.Runtime.PythonGenerator.
CheckThrowableAndReturnSendValue()
at IronPython.Runtime.Operations.PythonOps.
GeneratorCheckThrowableAndReturnSendValue(Object self)
at S$11.lambda_method$312(Closure , Int32& state, Object& current) in C:\python25\lib\types.py:line 52
at Microsoft.Scripting.Runtime.GeneratorEnumerator`1.
System.Collections.IEnumerator.MoveNext()
at IronPython.Runtime.PythonGenerator.MoveNextWorker()
at IronPython.Runtime.PythonGenerator.System.
Collections.IEnumerator.MoveNext()
at IronPython.Runtime.PythonGenerator.throw(Object type, Object value, Object traceback)
at IronPython.Runtime.PythonGenerator.throw(Object type)
at IronPython.Runtime.PythonGenerator.close()

IronPython Exception: [A]Person cannot be cast to [B]Person

2009-02-20T12:54:00.001-08:00

Had an interesting problem while using the Hosting API the other day. Here's the exception message.

[A]CSharpClass.Person cannot be cast to [B]CSharpClass.Person. Type A originates from 'CSharpClass, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' in the context 'LoadNeither' at location '[project directory]\bin\Debug\CSharpClass.dll'. Type B originates from 'CSharpClass, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' in the context 'Default' at location '[temp directory]\CSharpClass.DLL'.

I was struggling with this for a while until I found this link. In order to get my IronPython scripts to communicate with my CLR-based code, I was using the clr module's AddReference methods. These methods require you to point at the appropriate copy of the library you want to consume. I do stress "copy". Let's say you point your IronPython script at the bin\debug folder of a C# library project you're consuming in a console application. When you run the application, you'll get the exception message above. To make this example work, you'd have to point your IronPython script at the bin\Debug folder of your console application instead. Clearly, this is a maintenance nightmare. Instead, insert the following chunk of code into your hosting application (in the example described above, your console application) to avoid the problem all together.

Assembly assembly = typeof (CSharpClass.Person).Assembly;
runtime.LoadAssembly(assembly);

Note to Self 8 - IronPython2, Hosting, DLR

2009-02-04T19:03:00.001-08:00

The latest DLR code and documentation can be found at www.codeplex.com/dlr
For the examples below, here is the Python script I'm using

class Person(object):
    def __init__(self, name, phone):
        self.name = name
        self.phone = phone
    def TalkToCSharp(self):
        val = {"name":self.name, "phone":self.phone}
        return val

def GetPerson():
    person = Person("darrell", "555-4321")
    return person.TalkToCSharp()

def GetPersonWithParams(name, phone):
    person = Person(name, phone)
    return person.TalkToCSharp()

theNumberFive = 5
myPerson = GetPerson()

Simplest thing you can do is execute a Python script.

static void ExecutePythonScript(string path)
{
ScriptRuntime runtime = Python.CreateRuntime();
runtime.ExecuteFile(path);
}

Sometimes you need to get a variable after a script has run.

static void GetDoubleVariable(string path)
{
    ScriptRuntime runtime = Python.CreateRuntime();
    ScriptScope scope = runtime.ExecuteFile(path);
    object x = scope.GetVariable("theNumberFive");
    Console.WriteLine(x);
}

What happens if you're trying to return a custom class defined in the script? The easiest - and IMHO the best - way to go about this is to convert the object to a dictionary before passing it to your C# code.

static void GetInstanceCustomClass(string path)
{
    ScriptRuntime runtime = Python.CreateRuntime();
    ScriptScope scope = runtime.ExecuteFile(path);
    PythonDictionary dictionary =
        scope.GetVariable("myPerson")
        as PythonDictionary;
    foreach (var property in dictionary)
        Console.WriteLine(property.Key +
            " == " + property.Value);
}

Getting properties is definitely useful, but you'll probably want to call a function.

static void CallingAFunction(string path)
{
    ScriptRuntime runtime = Python.CreateRuntime();
    ScriptScope scope = runtime.ExecuteFile(path);
    ObjectOperations op =
        scope.Engine.CreateOperations();
    ObjectHandle handle =
        scope.GetVariableHandle("GetPerson");
    var x = op.Call(handle, new object[0]);
    PythonDictionary dictionary =
        x.Unwrap() as PythonDictionary;
    foreach (var property in dictionary)
        Console.WriteLine(property.Key +
            " == " + property.Value);
}

Here's an example of passing parameters to a Python function.

static void CallingAFunctionWithParameters(string path)
{
    ScriptRuntime runtime = Python.CreateRuntime();
    ScriptScope scope = runtime.ExecuteFile(path);
    ObjectOperations op =
        scope.Engine.CreateOperations();
    ObjectHandle handle =
        scope.GetVariableHandle("GetPersonWithParams");
    var x = op.Call(handle, "Larry", "555-1234");
    PythonDictionary dictionary =
        x.Unwrap() as PythonDictionary;
    foreach (var property in dictionary)
        Console.WriteLine(property.Key +
            " == " + property.Value);
}

Note to Self 7 - Naming Python Modules, SciTE, IronPython and name

2009-01-29T08:14:00.001-08:00

Be careful naming your Python modules. If you name a Python file the same as an existing file in your Python install directory, you may get the following exception message: "'module' object has no attribute '[some-attribute-name]'".
I'm really loving SciTE. It's a dead simple code editor that offers syntax highlighting for an impressive number of languages and displays the results of the run in it's own output window.
More SciTE: If you decide to use SciTE, you'll probably want to customize how it works. Here's a nice quick reference to supplement the SciTE documentation to help guide you through the config files.
If you want to use SciTE as your IronPython editor, you'll need to modify the existing python.properties file. Select "Open python.properties" from the "Options" menu. Search for "command.go" and modify the value so that it points at your IronPython directory.
All Python modules have a special variable called "__name__". When the code in your module is executed with CPython, __name__ is set to the file name of your module minus the "py" extension ("mymodule" if the file name of your module is "mymodule.py", for example), except when the module is executed directly. If that's the case, __name__ is equal to "__main__". Unfortunately, this is not the case if you execute it with IronPython. The special variable __name__ will always be based on the file name. UPDATE: The above is true only when you compile your IronPython code. If you launch a *.py file with ipy.exe, the __name__ variable will be "__main__".

Note to Self 6 - IronPython, pyc.py and SciTE

2009-01-19T20:09:00.001-08:00

The pyc.py file can compile you IronPython code into .NET executables.
The pyc.py file is part of the sample code released with IronPython. It can be found here. Look for the "IronPython-2.0-Samples.zip" link.
Typing out "[IP directory]\ipy [code directory]pyc.py" explicitly everytime you want to use the tool is somewhat annoying. Consider creating a script to do this for you. Here's a simple batch file.

cd C:\data\FeClass
c:\fepython\ipy.exe pyc.py class.py /main:main.py

Note that you'll have to copy the pyc.py file to the same directory as your code for this to work.
IronPython Studio just isn't up to snuff. You're better off to use a text editor and a build script
I like SciTE as my IronPython editor even more so than Notepad++. I really like that I can run my Python or IronPython script from SciTE and the output the normally would be displayed in the command window is captured in SciTE.

SQL Alchemy ORM Basics

2009-01-11T20:25:00.001-08:00

In my previous post, Learning SQL Alchemy, I focused on the SQL-like aspect of SQL Alchemy. You'll need to review the sample code in that post before understanding the code below. On with the code.

class User(object): pass
mapper(User, user_table)

Believe it or not, we just created a class with an ID, Name and Password. Simply create an empty class and pass it to the the mapper along with the user_table we had created. How do we use it? First off we have to create a Session object. Note that when we call the sessionmaker, it returns a Session class which you use to create the session.

Session = sqlalchemy.orm.sessionmaker()
session = Session()

All have we to do now is create a new User object, set the appropriate fields and pass it to the "save" method.

user = User()
user.name = "Fred Flintstone"
user.password = "yaba daba doo"
session.save(user)

But how do we get our user back from the database?

query = session.query(User)

We can now iterate through the query like any list.

for user in query:
print user.id, user.name, user.password

You can see the new record we inserted along with the other records in the table.

2 angie password2
3 nate awesomeness
4 Fred Flintstone yaba daba doo

The important thing to note is that the query object is actually a list of the very same User type I defined at the beginning of this post. Of course we probably don't want everything in the table, so here's an example of getting back a single object.

for user in query.filter_by(name = "angie"):
print user.id, user.name, user.password

The result is just what we would expect.

2 angie password2

But let's say I want to change Angie's password.

user.password = "a much more secure passphrase"

for user in query.filter_by(name = "angie"):
print user.id, user.name, user.password

The result?

2 angie a much more secure passphrase

Learning SQL Alchemy

2009-01-07T17:58:00.001-08:00

I spent a portion of my holidays studying the basics of SQL Alchemy. My first lesson is that SQL Alchemy is NOT an ORM for Python but instead a SQL abstraction tool with an ORM built-into it. At first I was having trouble understanding what this really meant, but after seeing an example it sunk in. Let's start by building a table.

from sqlalchemy import Table, Column, Integer, String

meta = sqlalchemy.MetaData("sqlite:///")

user_table = Table('userTable', meta,
    Column('id', Integer, primary_key=True),
    Column('name', String, unique=True, nullable=False),
    Column('password', String, nullable=False))

meta.create_all()

First thing we need is a MetaData object which holds all of our database schema information as well as our engine for executing queries. By passing "sqlite:///" in the MetData constructor we create an engine pointing at an in-memory SQLite database. To create a table, we just declare an instance of the Table object and pass in the necessary parameters: table name, our MetData instance and whatever columns we need in the table. The constructor for the Column class not only takes the column name and type, but constraints as well. Then all we have to do is execute the statement by using the create_all() method of the MetaData class.

Did you note the user_table variable we created? That comes in really handy when we want to start manipulating our new table.

insert = user_table.insert()
insert.execute(name='darrell', password='password1')
insert.execute(name='angie', password='password2')
insert.execute(name='nate', password='password3')
insert.execute(name='drew', password='password4')

See how easy inserting data is? There are methods just like the insert method above for select, insert and delete. Let's take a look at the select method.

select = user_table.select()
results = select.execute()

for row in results:
print row.name, row.password

The result are just as you would expect.

darrell password1
angie password2
nate password3
drew password4

That's great, but how do we change Nate's password and then return only his record?

update = user_table.update(user_table.c.name=='nate')
update.execute(password='awesomeness')

select = user_table.select(user_table.c.name=='nate')
results = select.execute()

for row in results:
print row.name, row.password

It's very simple and readable code with one caveat: what is "user_table.c.name"? It's a convenience attribute containing a collection of all the columns in userTable. So what our update statement is saying in terms of SQL is "UPDATE userTable SET ????? WHERE name = 'nate'". The only thing we don't know at this point is what to change. That happens when we execute the statement on the next line. If you understand the update statement above, the select statement works exactly the same way. Our result is below.

nate awesomeness

We've seen select, update and insert, but what about delete?

delete = user_table.delete(user_table.c.name.like("d%"))
delete.execute()
select = user_table.select()
results = select.execute()

I couldn't help myself - I had to confuse the issue by using "LIKE". In my defense, now you know how to introduce a "LIKE" comparison into your code AND that the "c" attribute is more than just a collection of strings. Beyond that, the delete works exactly the way you would expect it to work: call the delete method by passing the constraining expression as a parameter and then execute the result.

Now I understand why SQL Alchemy is an SQL abstration tool. All I did in the examples above were execute very basic SQL statements; the ORM was never used. Why is this so powerful? Imagine being able to write the same code for a SQL Server database as you did for a MySQL server and having the confidence that it's going to work. That's the power of SQL Alchemy.

Congratulations 2009 Microsoft MVP!

2009-01-02T06:26:00.001-08:00

That's the subject line of an email I received yesterday. No one is more surprised than me. Considering who the other MVPs are, I'm honored to be among them. I'm not entirely sure what I did to deserve this, but I'm grateful all the same. Thank you, Microsoft, for noticing me and here's to a productive 2009. Cheers!

Note to Self 5 - Inherit and Override

2008-12-27T08:37:00.001-08:00

I've been reading Working Effectively with Legacy Code by Michael C. Feathers. It's a trove of useful information for turning your unreadable and unreliable code into the complete opposite.

Here's a useful technique for dealing with dependencies that have problematic methods for testing. Mark the offending method as virtual and inherit from the class. Now you can use this new derivative class in your test methods in place of the original. Here's an example in C#.

public class MyClass
{
    public MyClass()
    {
        Console.WriteLine("MyClass has been initialized");
    }

    public virtual void DifficultMethod()
    {
        Console.WriteLine(
                "Difficult method makes testing difficult.");
    }
}

public class MyClassDerived : MyClass
{
    public MyClassDerived() : base()
    {
    }

    public override void DifficultMethod()
    {
        Console.WriteLine("It's testable!");
    }

}

public class ClassToTest
{
    public ClassToTest(MyClass myclass)
    {
        Console.WriteLine("Initializing ClassToTest");
        myclass.DifficultMethod();
    }
}

The ClassToTest has a constructor that takes a MyClass parameter. Unfortunately, MyClass has the DifficultMethod which is a very...well, difficult method. By marking DifficultMethod as virtual we can override it in a derived class which can then be used to pass into the ClassToTest constructor. Handy.
Here's the same idea in Python

class MyClass(object):
    def __init__(self):
        print "MyClass Initialized"
    def difficult_method(self):
        print "this makes things hard"

class MyClassDerivative(MyClass):
    def __init__(self):
        MyClass.__init__(self)
    def difficult_method(self):
        print "MUCH better"

class ClassToTest(object):
    def __init__(self, myClass):
        print "ClassToTest has been initialized"
        myClass.difficult_method()