Darrell Hawley: Home Page

Note to Self 12 – Lessons Learned from Our Latest Project

2009-10-28T14:14:00.001-07:00

I know I’m supposed to be releasing my IterHelper tool right now, but I’ve been on a bit of death march lately and have had no time. Though the project hasn’t ended yet, there are some items that I need to store away for future use before they’re forgotten.

On small teams, non-development related responsibilities should be divided among the team. Then the single non-dev responsibility (hopefully it’s the only one) should always take precedence over development responsibilities. It sounds backwards, but no matter what my priorities, I’m always going to find a way to write code. The same isn’t true of project management.
I can’t say enough about having a non-developer - who I’ll refer to as the Customer Advocate – on the project team. Things we had problems making time for – client meetings and acceptance testing for instance – fell to the Customer Advocate which takes a lot of pressure off everyone else.
Story points are great, but in order to get value out of them you actually have to use them – obvious, I know. Estimating tasks and then tracking how many points were completed daily, turned an out of control project into a manageable one.
Speaking of scoring, I wish we would have re-scored each task after we completed it. It would have taken minimal effort while at the same time providing us with enough information to improve our estimates.
We started using Agile Zen at the beginning of our project, but eventually abandoned it in favor of physical notecards. This introduced a lot of challenges, but it did allow us to grow our process exactly the way we wanted it. The added benefit is we’re in a much better position to evaluate the various kanban applications for our next project. To be clear, Agile Zen is a great product it just wasn’t what we needed at the time.
We didn’t use Selenium or Watir to run acceptance tests. Even if we couldn’t run all the tests we wanted, it would have been a huge time saver. A costly mistake that won’t be repeated.

Does IronPython need LINQ?

2009-09-10T15:17:00.001-07:00

Before I get started, I think I need to clear up some misconceptions that the Python community might have about LINQ. From the posts I’ve read, Python developers have dismissed LINQ as only a series of language enhancements that allow for SQL-like querying of lists. Though it is true that both C# and VB.NET have made changes to their respective languages for working with lists, you can still use LINQ without the aforementioned language enhancements. Python developers would be better served to think of LINQ as wrappers around the map, filter and reduce functions. In fact, LINQ is not much different than the “recipes” found in the official itertools documentation.

So the question still stands: does IronPython need LINQ? In my last post, I described how to use LINQ from an IronPython application and implied that I didn’t think it was the best idea. Don’t get me wrong, I really enjoy writing LINQ statements, I just don’t think LINQ fits well with IronPython. Why not? The first problem I have with the IronPython/LINQ combination is put on prominent display in this code sample.

print Enumerable.Average[object](
list, Func[object, int](lambda x:x))

If you are a static language aficionado, you probably are wondering what the problem is. If you’re a Python developer, however, the fact that I’m explicitly naming the “object” and “int” types is sending up giant, red flags. Python folks don’t really care what type of object it is they’re using, they only want to know what the object can do and what it contains (i.e., duck-typing). It’s much easier – and more readable - to simply write a completely Python version of the Average function. Consider the following chunk of LINQ code from C#:

List<string> values = new List<string>("the quick brown fox jumped over the lazy dogs".Split());

double avgLengthOfLongWords = values
                .Select(x => x.Length)
                .Where(x => x > 3)
                .Average();

Console.WriteLine(avgLengthOfLongWords)

To convert that to IronPython, we’d be forced to write something like the following:

import clr
clr.AddReferenceToFileAndPath("c:\users\darrell\desktop\System.Core.dll")

import System.Linq
from System import Func
from System.Linq import Enumerable

stuff = "the quick brown fox jumped over the lazy dogs".split()

def Select(col, func):
    return Enumerable.Select[object, object](col, Func[object, object](func))
def Where(col, func):
    return Enumerable.Where[object](col, Func[object, bool](func))
def Average(col):
    return Enumerable.Average[object](
    col, Func[object, float](lambda x:x))

avgLengthOfLongWords = Average(Where(Select(stuff, lambda x:len(x)), lambda x:x > 3))
print avgLengthOfLongWords

Before I’m even able to write useable code, I need to add a reference to a dll, import a namespace and a couple of classes and then write some helper functions; all so that I can write LINQ statements without the benefits of chaining. That’s right – I have to nest my LINQ statements. No chaining. That’s because when we use IronPython, the Select and Where don’t return an IEnumerable or an IQueryable. Instead they returna WhereSelectEnumerableIterator.

But do you know what really rankles me about using LINQ in IronPython? It contributes absolutely nothing to the Python community. The only way you can use LINQ is if you are using IronPython, not CPython, not Jython. I want to engage the entire Python community, not just my corner of the world. If ever there was a case where implementing the principle trumps migrating the implementation, this is it.

Enter the itertools module. Itertools is a Python standard library module containing tools for working with iterators. It also will serve as the cornerstone for my implementation of LINQ. When you visit the official itertools documentation at python.org, you’re even presented with code “recipes” that could be described as LINQ-ish. The only problem is that the recipes only accomodate nesting of statements. Why is this a problem? Consider the differences in readability between nested and chained in the following example:

stuff = "the quick brown fox jumped over the lazy dogs".split()

getLength = lambda x:len(x)
isNotTooLong = lambda x:x<6
isNotTooShort = lambda x:x>3

nested = ifilter(isNotTooShort, ifilter(isNotTooLong, imap(getLength, stuff)))

chained = IterHelper(stuff).select(getLength).where(isNotTooLong).where(isNotTooShort)

As always is the case with nesting methods, you have to read the statement populating nested backwards. That is, the outermost function call is the last to be run and you work your way inwards from there to find the origin. This is not a big deal to most of us since this is something we had to get used to very early in our careers. But when you compare that to the statement populating chained, I know I can read it from left to right. My eyes aren’t jumping back and forth to understand what is happening in what should be a relatively simple statement.

So what does IterHelper look like?

class IterHelper(object):
    def __init__(self, someIterable):
        self.someIterable = someIterable
    def __iter__(self):
        return self.someIterable
    def next(self):
        return self.someIterable.next()
    def select(self, func):
        return IterHelper(itertools.imap(func, self.someIterable))
    def where(self, predicate):
        return IterHelper(itertools.ifilter(predicate, self.someIterable))

This extremely simplistic version of IterHelper is really only a wrapper around functions found in the itertools module. This is great news since any application written using Python 2.3 or later can make use of this pattern with little effort. There’s no special libraries AND everyone is invited regardless of your flavor of Python.

My plan is to release IterHelper as an open source project by sometime by mid-October. I have a well-tested, basic implementation now, but I still have questions regarding licensing, hosting and deployment. I will be doing a bit of research on these topics and be posting what I’ve learned in my Notes to Self series.

IronPython and LINQ: A Devlink Correction

2009-08-15T20:19:00.001-07:00

During my “Anatomy of an IronPython Application” presentation at Devlink, I mistakenly said that LINQ can be consumed from IronPython easily. I said this because I was thinking that IronPython 2.6 was built using .NET 3.5 when in fact it’s built with .NET 2.0. Fortunately, if you want to use LINQ, you still can because .NET 3.5 is really .NET 2.0 plus extensions; you just have a bit of extra work to do. First on the todo list is to copy the System.Core.dll for 3.5 to some other location (I was able to get to it from it’s home in the c:\windows\assembly directory) and then use the “clr.AddReferenceToFileAndPath” in your IP code to gain access. Then, with a simple import of the Enumerable or Queryable type, you magically have access to LINQ.

Still, the story doesn’t end there. You have to remember that IronPython does NOT support extension methods and since LINQ is pretty much a collection of extensions methods, there’s a problem. Why this design decision? Because in order to support extension methods, the IronPython team would have had to make changes to the core language which would have, of course, been rather un-Pythonic. This means that the code for consuming LINQ is going to be somewhat messy. The following snippet gives an example of consuming the Average function of the Enumerable object.

clr.AddReferenceToFileAndPath(r"C:\folder\System.Core.dll")
from System.Linq import Enumerable
from System import Func

list = [1,2,3,4,5]

print Enumerable.Average[object](
list, Func[object, int](lambda x:x))

I have a lot of concerns as soon as I see “object” and “int” in brackets. At this point I’m going to hold save those concerns until my next post. Instead I’m going to leave this as an example of consuming LINQ from IronPython.

I should point out that Harry Pierson’s post on IronPython and Linq-to-XML really got me started on this topic. I also have to give a shout-out to Chris Smith who was kind enough to help me out and relay a couple of CodeMash stories at the same time.

Speaking at the Lansing Day of .NET on Saturday

2009-07-31T07:31:00.001-07:00

If you’re going to be at the Lansing Day of .NET this Saturday, stop by and catch my Anatomy of an IronPython Application presentation. I’ve completely rewritten the talk so if you’ve seen it before, this new version will be covering completely different topics including testing and mocking. I’ll also be giving away a copy of IronPython in Action, courtesy of Manning Publications, at the end of my talk.

Note to Self 11 – Patterns in Python, Process is Everything, Selenium

2009-07-25T05:01:00.001-07:00

While studying a particular portion of IronPython in Action, I came across a footnote to a blog post entitled Patterns in Python. By the end of the post, I found myself mentally tweaking a lot of my code. Though quite old (the page was posted February of 2003), the content remains quite relevant.
I’m fortunate to be working on a project where everyone involved is committed to being agile (note the lack of a capital “A”). We’ve done enough things right that the items I would do differently are very minor. With that said, the imbalance in hours spent by each team member seems to be causing a problem…
Another interesting observation – the project that I feel very fortunate to be working on is not using the technology of my choice. It’s a Java project. In fact, almost every facet of the project requires some specific technology in which I almost know nothing, yet the development process makes the experience a joy.
Configuring my Selenium tests to work with Selenium RC and IE8 has not been an easy task. Some things I learned along the way…
- Easiest way to create a new Selenium test is to use the Selenium IDE plugin for FireFox. It records everything you click and do and allows you to convert the recording to C#, Java, Python, PHP or Ruby. From there, further customizations are a snap.
- Selenium RC allows you to run your Selenium tests in browsers other than FireFox, such as IE, Chrome and Safari.
- Before running your tests in one of these “other” browsers, you’ll need to start the Selenium Server. Selenium Server comes parceled with the Selenium RC download and can be started from the command line with “java -jar selenium-server.jar”.
- To run my Selenium tests with IE8, I had to use the *iexploreproxy option even though I’m not using a proxy. Otherwise I kept getting redirected to a phantom file supposedly hidden away in my user directory.

Book Review: IronPython in Action

2009-07-14T08:24:00.001-07:00

IronPython in Action (IPIA) gets the distinction of being the first book published on IronPython. As the first book, it has the responsibility of giving developers their first impression of IronPython. IPIA does exactly that by taking numerous shallow dives into various technologies and techniques. To be clear, this book is aimed at professional developers preferably with either .NET or Python experience. Pythonistas will find the book offers quick introductions to a number of Microsoft technologies such as Windows Forms, WPF and ASP.NET. .NET developers will get the chance to see technologies they’re already familiar with expressed in a completely new fashion.

IPIA avoids the trap of simply being a brochure of Microsoft technologies using “IronPython” in place of “C#” and “VB.NET”. Instead, the authors mix in chapters on development techniques ranging from the familiar - such as unit testing and mocking – to the more exotic such as metaprogramming. As .NET developers wind their way through the book, they’ll begin to appreciate the differences between the static and dynamic worlds – a crucial step toward wide ranging acceptance.

If you’re curious about what IronPython can do for you, IronPython in Action is well worth your time.

Note to Self 10 – JQuery Selectors and Custom Functions and a thought on JQuery

2009-06-28T09:01:00.001-07:00

After using Python, diving into jQuery has proven exceptionally difficult. Even though it solves the not-so-small problem of commonizing the DOM across multiple browsers, I find it very difficult to read. I know with time I’ll get used to it and maybe even find some beauty in my jQuery code, but it will never stop me from wishing for a cleaner syntax like that of Python or Ruby.

With that said, here’s a bit of what I learned about jQuery recently.

Selectors in jQuery are a syntax for selecting an html element. Selectors can return a single element or several (or none if nothing matching is found).
$(“#[elementID]”) – finds an element with an id attribute equal to “elementID”.
$(“[name=’elementName’]”) – gets all elements with a name attribute equal to “elementName”
$(“[name^=’element’]”) – gets all elements with a name attribute that begins with “element”
$(“[name$=’Name’]”) – gets all elements with a name attribute that ends with “Name”
$(“input:checkbox”) – gets all checkboxes
$(“#blah > input:checkbox”) – gets all checkboxes within a parent element of id “blah”
Creating a custom function in jQuery:

$.fn.CustomFunctionName({function()({
//code here
})

$.fn.exclaim = function(message){
alert(message)
}
$.fn.exclaim("hello, world")

Note to Self 9 – Exploring SharpDevelop and Going off Topic

2009-05-29T07:05:00.001-07:00

It’s been too long since I last blogged which typically means I have too many competing ideas to settle on just one. So welcome to Note to Self 9!

SharpDevelop is an open source IDE for the .NET platform. Is it new? Hardly. Based on the history of news releases from their website, It’s been around since December of 2000. Considering that the .NET platform was publicly announced only 6 months earlier, SharpDevelop seems only that much more mature.
If you’re comfortable with the basics of Visual Studio, SharpDevelop should feel like home to you. In fact, the best way to describe SharpDevelop is “Visual Studio without a lot of stuff I don’t want and a few thid-party items I do”.
SharpDevelop uses NUnit instead of MSTest. That’s right. No third party plugins to make your IDE work with NUnit. It simply works.
It’s time for me to confess. It’s all about IronPython. The reason I started investigating SharpDevelop is because of my frustration with Visual Studio’s lack of support for the language. Specifically, I want Intellisense when I write IronPython. To be sure, you only get Intellisense for “non-DLR code”, but surprisingly that doesn’t seem to be as much of a handicap as you might think.
Speaking of first class citizens, SharpDevelop comes stock with C#, Visual Basic, Boo (I can hear Jay Wren cheering now), IronPython and F# (now a number of SRT folks are cheering).
If you’re frightened of the consequences that may come from changing your development environment, don’t be. You can open one of your existing Visual Studio solutions with SharpDevelop and start coding. Switching back is as simple as opening up your now modified project/solution in Visual Studio.
For clarity, SharpDevelop is new to me. I haven’t used SharpDevelop for any projects yet so I can’t tell you about any of its nuances. I plan on putting it to the test, however, over the next several weeks. I’ll be posting about my experiences.
Now for the off-topic part. I’ve been listening to The History of Rome, a FANTASTIC podcast series on, well, the history of Rome. Mike Duncan, creator and narrator, is not only an engaging story teller, but also has the technical savvy to put together a professional podcast. Though it’s not the sort of podcast typically on a geek’s menu, I think there are enough parallels to some of the challenges we are facing in the software industry to make it worthwhile.

Anatomy of an IronPython Application at AADND

2009-04-07T20:00:00.001-07:00

I'm going to be talking IronPython at the Ann Arbor .NET Developers Group at SRT Solutions tomorrow night (April 8) at 6:00. I'll be showing you how you might introduce IronPython into your existing .NET environment through a lot of sample code and very few PowerPoint slides. I hope to see you there.

Anatomy of an IronPython Application

You've heard about IronPython and maybe have even taken the time to write the obligatory "Hello, World" application. Now what? The buzz around IronPython continues but there's still little guidance on tools and development processes. This presentation focuses on the basics of interacting with C# and VB.NET libraries using IronPython while introducing useful tools and techniques to get you started. Though experience with .NET is necessary, the samples and discussions are understandable to even beginning Python developers.

Writing an IronPython WCF Host

2009-03-15T06:19:00.001-07:00

This is a simple WCF host written in IronPython that consumes a library I wrote in C#.

import clr
clr.AddReferenceByPartialName("System.ServiceModel")
clr.AddReferenceByPartialName("CSharpClass")

from System.ServiceModel import ServiceHost, BasicHttpBinding
from System.ServiceModel.Description import ServiceMetadataBehavior
from CSharpClass import PersonService, IPersonService
from System import Array, Uri

uri = Uri("http://localhost:1234/people")
uriArray = Array[Uri]([uri]) # creates a generic array of type Uri
serviceHost = ServiceHost(PersonService, uriArray)

binding = BasicHttpBinding()

serviceHost.AddServiceEndpoint(IPersonService, binding, "http://localhost:1234/people")

behavior = ServiceMetadataBehavior()
behavior.HttpGetEnabled = True
serviceHost.Description.Behaviors.Add(behavior)

serviceHost.Open()

raw_input("Service has been started. Press Enter to exit...")

Running nose with IronPython

2009-03-06T11:28:00.001-08:00

I just spent the past week at the 2009 Microsoft MVP Summit where I got a chance to talk to the some of the crew behind the dynamic languages at Microsoft. When I mentioned that I had gotten nose, a Python unit testing framework, working with IronPython, I was strongly encouraged to blog about it. Without further adieu, here's my code.

import sys
sys.path.append(r"C:\Python25\Lib\site-packages\nose-0.10.4-py2.5.egg")
sys.path.append(r"C:\python25\lib")
sys.path.append(r"C:\Python25\Lib\email")
sys.path.append(r"C:\Python25\Scripts")
import nose
nose.main()

I dropped this script into a directory with a couple of bogus unit tests and ran it with Python 2.5.4. Results were consistent with my expectations - successful tests passed and failing tests threw exceptions. When I ran the same suite against IronPython 2.0.0, I still received the correct results but the script took 1 minute and 20 seconds to complete. Compare that to less than a second (estimated) for CPython. For the sake of completeness, I ran the same test with IronPython 2.0.1 and received the same result.

The root of the problem appears to be the large number of GeneratorExitExceptions being thrown. I'm not sure why they're being thrown, but I'll continue to investigate and report my findings. If you're interested in the gory details, here's the stack trace of the first exception:

at IronPython.Runtime.PythonGenerator.ThrowThrowable()
at IronPython.Runtime.PythonGenerator.CheckThrowable()
at IronPython.Runtime.PythonGenerator.
CheckThrowableAndReturnSendValue()
at IronPython.Runtime.Operations.PythonOps.
GeneratorCheckThrowableAndReturnSendValue(Object self)
at S$11.lambda_method$312(Closure , Int32& state, Object& current) in C:\python25\lib\types.py:line 52
at Microsoft.Scripting.Runtime.GeneratorEnumerator`1.
System.Collections.IEnumerator.MoveNext()
at IronPython.Runtime.PythonGenerator.MoveNextWorker()
at IronPython.Runtime.PythonGenerator.System.
Collections.IEnumerator.MoveNext()
at IronPython.Runtime.PythonGenerator.throw(Object type, Object value, Object traceback)
at IronPython.Runtime.PythonGenerator.throw(Object type)
at IronPython.Runtime.PythonGenerator.close()

IronPython Exception: [A]Person cannot be cast to [B]Person

2009-02-20T12:54:00.001-08:00

Had an interesting problem while using the Hosting API the other day. Here's the exception message.

[A]CSharpClass.Person cannot be cast to [B]CSharpClass.Person. Type A originates from 'CSharpClass, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' in the context 'LoadNeither' at location '[project directory]\bin\Debug\CSharpClass.dll'. Type B originates from 'CSharpClass, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' in the context 'Default' at location '[temp directory]\CSharpClass.DLL'.

I was struggling with this for a while until I found this link. In order to get my IronPython scripts to communicate with my CLR-based code, I was using the clr module's AddReference methods. These methods require you to point at the appropriate copy of the library you want to consume. I do stress "copy". Let's say you point your IronPython script at the bin\debug folder of a C# library project you're consuming in a console application. When you run the application, you'll get the exception message above. To make this example work, you'd have to point your IronPython script at the bin\Debug folder of your console application instead. Clearly, this is a maintenance nightmare. Instead, insert the following chunk of code into your hosting application (in the example described above, your console application) to avoid the problem all together.

Assembly assembly = typeof (CSharpClass.Person).Assembly;
runtime.LoadAssembly(assembly);

Note to Self 8 - IronPython2, Hosting, DLR

2009-02-04T19:03:00.001-08:00

The latest DLR code and documentation can be found at www.codeplex.com/dlr
For the examples below, here is the Python script I'm using

class Person(object):
    def __init__(self, name, phone):
        self.name = name
        self.phone = phone
    def TalkToCSharp(self):
        val = {"name":self.name, "phone":self.phone}
        return val

def GetPerson():
    person = Person("darrell", "555-4321")
    return person.TalkToCSharp()

def GetPersonWithParams(name, phone):
    person = Person(name, phone)
    return person.TalkToCSharp()

theNumberFive = 5
myPerson = GetPerson()

Simplest thing you can do is execute a Python script.

static void ExecutePythonScript(string path)
{
ScriptRuntime runtime = Python.CreateRuntime();
runtime.ExecuteFile(path);
}

Sometimes you need to get a variable after a script has run.

static void GetDoubleVariable(string path)
{
    ScriptRuntime runtime = Python.CreateRuntime();
    ScriptScope scope = runtime.ExecuteFile(path);
    object x = scope.GetVariable("theNumberFive");
    Console.WriteLine(x);
}

What happens if you're trying to return a custom class defined in the script? The easiest - and IMHO the best - way to go about this is to convert the object to a dictionary before passing it to your C# code.

static void GetInstanceCustomClass(string path)
{
    ScriptRuntime runtime = Python.CreateRuntime();
    ScriptScope scope = runtime.ExecuteFile(path);
    PythonDictionary dictionary =
        scope.GetVariable("myPerson")
        as PythonDictionary;
    foreach (var property in dictionary)
        Console.WriteLine(property.Key +
            " == " + property.Value);
}

Getting properties is definitely useful, but you'll probably want to call a function.

static void CallingAFunction(string path)
{
    ScriptRuntime runtime = Python.CreateRuntime();
    ScriptScope scope = runtime.ExecuteFile(path);
    ObjectOperations op =
        scope.Engine.CreateOperations();
    ObjectHandle handle =
        scope.GetVariableHandle("GetPerson");
    var x = op.Call(handle, new object[0]);
    PythonDictionary dictionary =
        x.Unwrap() as PythonDictionary;
    foreach (var property in dictionary)
        Console.WriteLine(property.Key +
            " == " + property.Value);
}

Here's an example of passing parameters to a Python function.

static void CallingAFunctionWithParameters(string path)
{
    ScriptRuntime runtime = Python.CreateRuntime();
    ScriptScope scope = runtime.ExecuteFile(path);
    ObjectOperations op =
        scope.Engine.CreateOperations();
    ObjectHandle handle =
        scope.GetVariableHandle("GetPersonWithParams");
    var x = op.Call(handle, "Larry", "555-1234");
    PythonDictionary dictionary =
        x.Unwrap() as PythonDictionary;
    foreach (var property in dictionary)
        Console.WriteLine(property.Key +
            " == " + property.Value);
}

Note to Self 7 - Naming Python Modules, SciTE, IronPython and name

2009-01-29T08:14:00.001-08:00

Be careful naming your Python modules. If you name a Python file the same as an existing file in your Python install directory, you may get the following exception message: "'module' object has no attribute '[some-attribute-name]'".
I'm really loving SciTE. It's a dead simple code editor that offers syntax highlighting for an impressive number of languages and displays the results of the run in it's own output window.
More SciTE: If you decide to use SciTE, you'll probably want to customize how it works. Here's a nice quick reference to supplement the SciTE documentation to help guide you through the config files.
If you want to use SciTE as your IronPython editor, you'll need to modify the existing python.properties file. Select "Open python.properties" from the "Options" menu. Search for "command.go" and modify the value so that it points at your IronPython directory.
All Python modules have a special variable called "__name__". When the code in your module is executed with CPython, __name__ is set to the file name of your module minus the "py" extension ("mymodule" if the file name of your module is "mymodule.py", for example), except when the module is executed directly. If that's the case, __name__ is equal to "__main__". Unfortunately, this is not the case if you execute it with IronPython. The special variable __name__ will always be based on the file name. UPDATE: The above is true only when you compile your IronPython code. If you launch a *.py file with ipy.exe, the __name__ variable will be "__main__".

Note to Self 6 - IronPython, pyc.py and SciTE

2009-01-19T20:09:00.001-08:00

The pyc.py file can compile you IronPython code into .NET executables.
The pyc.py file is part of the sample code released with IronPython. It can be found here. Look for the "IronPython-2.0-Samples.zip" link.
Typing out "[IP directory]\ipy [code directory]pyc.py" explicitly everytime you want to use the tool is somewhat annoying. Consider creating a script to do this for you. Here's a simple batch file.

cd C:\data\FeClass
c:\fepython\ipy.exe pyc.py class.py /main:main.py

Note that you'll have to copy the pyc.py file to the same directory as your code for this to work.
IronPython Studio just isn't up to snuff. You're better off to use a text editor and a build script
I like SciTE as my IronPython editor even more so than Notepad++. I really like that I can run my Python or IronPython script from SciTE and the output the normally would be displayed in the command window is captured in SciTE.

SQL Alchemy ORM Basics

2009-01-11T20:25:00.001-08:00

In my previous post, Learning SQL Alchemy, I focused on the SQL-like aspect of SQL Alchemy. You'll need to review the sample code in that post before understanding the code below. On with the code.

class User(object): pass
mapper(User, user_table)

Believe it or not, we just created a class with an ID, Name and Password. Simply create an empty class and pass it to the the mapper along with the user_table we had created. How do we use it? First off we have to create a Session object. Note that when we call the sessionmaker, it returns a Session class which you use to create the session.

Session = sqlalchemy.orm.sessionmaker()
session = Session()

All have we to do now is create a new User object, set the appropriate fields and pass it to the "save" method.

user = User()
user.name = "Fred Flintstone"
user.password = "yaba daba doo"
session.save(user)

But how do we get our user back from the database?

query = session.query(User)

We can now iterate through the query like any list.

for user in query:
print user.id, user.name, user.password

You can see the new record we inserted along with the other records in the table.

2 angie password2
3 nate awesomeness
4 Fred Flintstone yaba daba doo

The important thing to note is that the query object is actually a list of the very same User type I defined at the beginning of this post. Of course we probably don't want everything in the table, so here's an example of getting back a single object.

for user in query.filter_by(name = "angie"):
print user.id, user.name, user.password

The result is just what we would expect.

2 angie password2

But let's say I want to change Angie's password.

user.password = "a much more secure passphrase"

for user in query.filter_by(name = "angie"):
print user.id, user.name, user.password

The result?

2 angie a much more secure passphrase

Learning SQL Alchemy

2009-01-07T17:58:00.001-08:00

I spent a portion of my holidays studying the basics of SQL Alchemy. My first lesson is that SQL Alchemy is NOT an ORM for Python but instead a SQL abstraction tool with an ORM built-into it. At first I was having trouble understanding what this really meant, but after seeing an example it sunk in. Let's start by building a table.

from sqlalchemy import Table, Column, Integer, String

meta = sqlalchemy.MetaData("sqlite:///")

user_table = Table('userTable', meta,
    Column('id', Integer, primary_key=True),
    Column('name', String, unique=True, nullable=False),
    Column('password', String, nullable=False))

meta.create_all()

First thing we need is a MetaData object which holds all of our database schema information as well as our engine for executing queries. By passing "sqlite:///" in the MetData constructor we create an engine pointing at an in-memory SQLite database. To create a table, we just declare an instance of the Table object and pass in the necessary parameters: table name, our MetData instance and whatever columns we need in the table. The constructor for the Column class not only takes the column name and type, but constraints as well. Then all we have to do is execute the statement by using the create_all() method of the MetaData class.

Did you note the user_table variable we created? That comes in really handy when we want to start manipulating our new table.

insert = user_table.insert()
insert.execute(name='darrell', password='password1')
insert.execute(name='angie', password='password2')
insert.execute(name='nate', password='password3')
insert.execute(name='drew', password='password4')

See how easy inserting data is? There are methods just like the insert method above for select, insert and delete. Let's take a look at the select method.

select = user_table.select()
results = select.execute()

for row in results:
print row.name, row.password

The result are just as you would expect.

darrell password1
angie password2
nate password3
drew password4

That's great, but how do we change Nate's password and then return only his record?

update = user_table.update(user_table.c.name=='nate')
update.execute(password='awesomeness')

select = user_table.select(user_table.c.name=='nate')
results = select.execute()

for row in results:
print row.name, row.password

It's very simple and readable code with one caveat: what is "user_table.c.name"? It's a convenience attribute containing a collection of all the columns in userTable. So what our update statement is saying in terms of SQL is "UPDATE userTable SET ????? WHERE name = 'nate'". The only thing we don't know at this point is what to change. That happens when we execute the statement on the next line. If you understand the update statement above, the select statement works exactly the same way. Our result is below.

nate awesomeness

We've seen select, update and insert, but what about delete?

delete = user_table.delete(user_table.c.name.like("d%"))
delete.execute()
select = user_table.select()
results = select.execute()

I couldn't help myself - I had to confuse the issue by using "LIKE". In my defense, now you know how to introduce a "LIKE" comparison into your code AND that the "c" attribute is more than just a collection of strings. Beyond that, the delete works exactly the way you would expect it to work: call the delete method by passing the constraining expression as a parameter and then execute the result.

Now I understand why SQL Alchemy is an SQL abstration tool. All I did in the examples above were execute very basic SQL statements; the ORM was never used. Why is this so powerful? Imagine being able to write the same code for a SQL Server database as you did for a MySQL server and having the confidence that it's going to work. That's the power of SQL Alchemy.

Congratulations 2009 Microsoft MVP!

2009-01-02T06:26:00.001-08:00

That's the subject line of an email I received yesterday. No one is more surprised than me. Considering who the other MVPs are, I'm honored to be among them. I'm not entirely sure what I did to deserve this, but I'm grateful all the same. Thank you, Microsoft, for noticing me and here's to a productive 2009. Cheers!

Note to Self 5 - Inherit and Override

2008-12-27T08:37:00.001-08:00

I've been reading Working Effectively with Legacy Code by Michael C. Feathers. It's a trove of useful information for turning your unreadable and unreliable code into the complete opposite.

Here's a useful technique for dealing with dependencies that have problematic methods for testing. Mark the offending method as virtual and inherit from the class. Now you can use this new derivative class in your test methods in place of the original. Here's an example in C#.

public class MyClass
{
    public MyClass()
    {
        Console.WriteLine("MyClass has been initialized");
    }

    public virtual void DifficultMethod()
    {
        Console.WriteLine(
                "Difficult method makes testing difficult.");
    }
}

public class MyClassDerived : MyClass
{
    public MyClassDerived() : base()
    {
    }

    public override void DifficultMethod()
    {
        Console.WriteLine("It's testable!");
    }

}

public class ClassToTest
{
    public ClassToTest(MyClass myclass)
    {
        Console.WriteLine("Initializing ClassToTest");
        myclass.DifficultMethod();
    }
}

The ClassToTest has a constructor that takes a MyClass parameter. Unfortunately, MyClass has the DifficultMethod which is a very...well, difficult method. By marking DifficultMethod as virtual we can override it in a derived class which can then be used to pass into the ClassToTest constructor. Handy.
Here's the same idea in Python

class MyClass(object):
    def __init__(self):
        print "MyClass Initialized"
    def difficult_method(self):
        print "this makes things hard"

class MyClassDerivative(MyClass):
    def __init__(self):
        MyClass.__init__(self)
    def difficult_method(self):
        print "MUCH better"

class ClassToTest(object):
    def __init__(self, myClass):
        print "ClassToTest has been initialized"
        myClass.difficult_method()

Using Lambdas to Avoid Gnarly IF Statements

2008-12-16T20:20:00.001-08:00

Do you have a function in your C# code somewhere that takes a parameter - the type doesn't matter - and then executes another function based on that parameter's value?

public void what_to_do(string param)
{
    if (param == "this")
        DoThis();
    else if (param == "that")
        DoThat();
    else
        DoSomethingElse();
}

Whenever I see "else if", my spider-sense starts tingling. Adding another "else if" is just too easy and the result can be cumbersome. I do use "else if" (and "switch" - it's just as vulnerable to this idea), but I don't always like it. The good news is that there is a fix - if you're willing to think "functionally". The bad news is it took me so long to piece it all together.

When did this epiphany come? At the latest MichiPUG meeting, while Mark Ramm was demonstrating a Python library for implementing generics, he displayed an interesting technique for avoiding the Python version of the above code. Here's my rendition of what he showed us:

whatToDoDictionary = {'this':do_this, 'that':do_that}

def what_to_do(param):
    if param in whatToDoDictionary:
        whatToDoDictionary[param]();
    else:
        do_something_else();

The key to this bit of Python is the values in the whatToDoDictionary. Instead of your standard string or integers, the values are functions. When we call the what_to_do method, we look to see if the parameter is present as a key in the dictionary. If it is, we simple execute the value portion of the key/value pair. Voilà! No more "else if"!

Can we do this in C#? Sure. Let's do it using lambdas.

Dictionary<string, Action> dict =

     new Dictionary<string, Action>();

public Lambda()
{
    dict.Add("this", () => do_this());
    dict.Add("that", () => do_that());
}

public void WhatToDo(string param)
{
    if (dict.ContainsKey(param))
        dict[param]();
    else
        do_something_else();
}

The concept is exactly the same. Build a dictionary of lambdas (functions in Python), see if the parameter passed to the WhatToDo(string) is in the dictionary and then execute the lambda if it is.

Note to Self 4 - Python Editors, Debuggers, stdout and stderr

2008-12-13T13:11:00.001-08:00

PyScripter has been IDE of choice for Python, but I've run into some problems with it. I would have continued to use it, but I noticed that the latest anyone did any work on it was October of 2006 - or at least that's what I thought. It turns out that early this year PyScripter is now maintained in Google Code and supports Python 2.6. You can download it here.
I've started using Notepad++ for editing my Python. Though it's only an editor, it uses syntax highlighting, supports macros and allows me to define hot keys to run external programs. This last point makes it possible for me to launch my code in Python with a simple keypress.
Want to know how to use Notepad++ to launch the file you're currently editing in Python? Type "[%Python directory%]\python.exe "$(FULL_CURRENT_PATH)" into the Notepad++ Run dialog and then save it to the keystroke combination of you choice.
If you took my previous bullet point to heart, you'll find that Python launches in a command window and then promptly closes after your tests run losing any output you may have had. The only way that I've been able to solve this is by pointing the stdout and stderr to a file. I would prefer to override the these when I launch Python, but instead I have to put some addtional code into my application. Bummer.
WinPDB is a debugging tool for Python. The only tool I'd used for debugging in the past is the one embedded into PyScripter. When I started using Notepad++ as my editor, I started trying to debug from the command line using the PDB module. No fun. That's when I came across WinPDB. It's worked well for the past few days that I've been using it, but it still isn't as nice as having your debugger integrated into your environment.

Note to Self 3 - CodeMash Screen Casting Options

2008-11-17T10:41:00.001-08:00

I've been learning a lot about Screencasting this week...

I've updated my podcasting links at http://delicious.com/dwhawley/podcasting with a couple of new items.
I reviewed a couple of USB devices to capture video and audio (USB 28000D DVD Maker and the Diamond VC500 One Touch Video Capture Device) and I wasn't happy with any of the reviews. This really seemed to be the simplest avenue but seems to be a dead-end.
The solution I'm searching for assumes that I'll only have laptops available. There are a lot of video capture cards out there for a desktop machine but I'm avoiding this one for now.
I like the VNC option the best, but a major problem with this is that we'll need to rely on speakers installing something on their machine. Jason Follas pointed this out to me and he's right. I think the only way to resolve this is ask the community. If they don't like the solution I'll know to keep looking. Besides, someone might have a really good idea.
Jing is cool. Jay Wren gave me quick demo earlier today and I was impressed with it's simplicity. One of the things that I really like about Jing is it's social spirit. I was looking at Camtasia, another offering by TechSmith , but I think Jing may be a better fit.

Note to Self 2 - Capturing Multiple Conference Sessions Simultaneously

2008-11-04T11:00:00.001-08:00

I'm the A/V Coordinator for CodeMash 2009 and as such I have a lot of sessions to capture. Here are some notes.

I'm storing links related to podcasting at http://delicious.com/dwhawley/podcasting.
According to Wikipedia, "Virtual Network Computing (VNC) is a graphical desktop sharing system which uses the RFB protocol to remotely control another computer. It transmits the keyboard and mouse events from one computer to another, relaying the graphical screen updates back in the other direction, over a network." In short, the VNC server displays what's on the screen of the VNC client.
Every presenter will require a VNC client of some sort on their machine. The client will connect to a VNC server where the actual video capturing will take place. I'm assuming one server per client with each hard-wired to a switch, but I'm hoping for more 2 or more clients per server.
Without knowing much about it, I had been using VNC for quite some time. I've used both GoToMeeting and NetMeeting with excellent results.
Because this is CodeMash, there will be three different families of Operating Systems represented: Windows, Mac and Linux. Theoretically, my Windows based VNC Server should have no problems seeing a Linux- or Mac-based client, but I've heard this sort of thing before. I will be setting up a test network in the very near future.
Audio will have to be recorded separately since it isn't part of the RFB spec (communications protocol between VNC client and server). I know some tools cover this for you, but I can't count on it in platform agnostic environment.

Note to Self 1

2008-11-03T13:57:00.001-08:00

I haven't been blogging much lately, which is no surprise to anyone who follows my blog. To those three readers, I'd like to tell you that writing - to me - is a very much a love/hate relationship. When I'm inspired, writing becomes an obsession I can't possibly put aside. Even if what I produce is not noteworthy among my colleagues, the process of putting my thoughts into words helps me clarify personal thoughts like no other learning experience could. It confirms truths. It tears-down misconceptions. It's the highest form of human expression. It's REALLY exhausting. So If I don't have the time or energy to write, how do I effectively maintain a blog?

The title of this post sums up my solution. If I don't have the time or energy to write the kind of post I want, just write short bullet points on the things that I've been exploring. What I really like about this approach is that I do a lot more in the course of a day than I could possibly convey through my blog. For example, since I last blogged, I've experimented with jQuery, VMWare Player, Virtual PC, Python and SQL Alchemy. Even at the best of times, there is no way that I'm going to cover all of those topics before forgetting one or more of them. With that said, time to write some notes.

jQuery rules, though I've had problems debugging using FireBug. Apparently it requires a bookmarklet though that still doesn't seem to work for me.
The mapper class in SQL Alchemy - Python ORM - gets rid of having to manually set properties of derived classes when the corresponding base class is already known (i.e., Employee.Name = Person.Name). This is the sort of code that really annoys me and I'm glad there's a way to avoid it.
Virtualization rocks, but it gets tough on a laptop. My host machine is Vista and my guests are all XP. I've tried Vista as guest before, but it was so slow I couldn't really do anything with it. Some of my virtualization links can be found at http://delicious.com/dwhawley/virtual.
I've been using both VMWare Player and Virtual PC. VMWare Player works with more than just Windows allowing me to periodically play with Linux, but doesn't have the really cool pause feature that Virtual PC has. Virtual PC seems a bit snappier to me, but it lacks support for USB. No excuses for that.
Azure is really cool though you should keep in mind that this is in direct competition with Google Apps. Either way, I'll be using both.
Of all of the features I've heard about Windows 7, I'm most excited about it's smaller footprint.

A Visit from Redmond - Catching up on Visual Studio Team Edition with Mark Mydland

2008-10-27T13:54:00.001-07:00

On November 12th, Mark Mydland, Group Manager for the Visual Studio Team Edition for Software Testers, will be speaking at the Ann Arbor .NET Developers Group. This is your opportunity to glimpse the future of - possibly even influence - the tool you spend the most time using, so clear you calendar!

The phrase “drive quality upstream” has been abused so badly by ALM software vendors that it has to be relegated to the platitude junk pile along with such all time favorites as “work smarter not harder,” “think outside the box,” and “synergistic leveraging of code reuse.” Before we drive quality anywhere, we need to give quality a seat at the table. VSTS Rosario release will do this by automatically gathering critical information about the project and code and making that data available when, where and to whom it is needed. During this discussion we will examine how Rosario impacts quality across the application lifecycle by:
* Allowing testers to provide developers with details about what the code did instead of just providing the details about what the tester did.
* Allowing development leads and architects to visualize and understand their current code (not the code they wish they had, but the code they really have) so that they can minimize the impact of changes; and
* Allowing developers to understand the impact of their changes in terms of affected tests, concurrency and bounds checking.
The Rosario release of VSTS will bring all project stakeholders together to allow richer information to be shared across every role to make software quality accessible and achievable.

About Mark

Mark Mydland is a Group Manager for the Visual Studio Team Edition for Software Testers product at Microsoft. In the past 12 years, Mark has worked as a developer and consultant across a wide variety of applications and industries. Mark first joined Microsoft in 2001 working as a member of the Natural Interactive Services Division (NISD). During his time in that group, Mark was the development manager for a team focused on analytics for assessing the efficacy of natural language interpreters with a particular emphasis on driving authoring simplification and relevance quality for user assistance. Based on this work, Mark filed numerous patents and coauthored a paper for the SIGIR journal. In 2004, Mark left Microsoft to work as a Director of Development at Getty Images where he led a change in process from a traditional waterfall methodology to a scrum-based agile approach which brought the release frequency from 12-18 months down to 1 month. Since Getty made extensive use of VSTS, it seemed a natural fit for Mark to join VSTS on his return to Microsoft in 2006. Mark received his B.S. from West Point in 1991. He has also held positions with USWeb/marchFirst and Andersen Consulting/Accenture.