My Brain on Python

Slower than I'd hoped, and ...

2012-01-14T14:24:00.000-08:00

Well, I've had a few more days to ponder on things, and my work situation, while *very* nice/neat/cool, just isn't going to allow me to continue at anywhere near the pace I'd been maintaining before the change in jobs happened. In addition, I'm actually writing Python code at work, and though I'm doing a lot of really neat things with it, I'm not sure whether I can share any of it at all (I certainly can't without approval from my boss).

I'll try to ask around and see if any of what I'm doing is OK to share here (I hope so, since I've written a couple thousand lines of code so far), but until I know for sure, I'm gonna err on the side of caution, and not post anything unless I *know* it's mine, unencumbered.

Interlude: Enumerations and TypedContainers

2012-01-09T04:30:00.001-08:00

So, while I ponder my next step(s), it occurred to me that I have some old code lying around that could stand to be polished and/or brought up to my current standards. One item from that mix is an attempt to implement an Enumerated Type, much like the types provided in Java and C#. Although it isn't, technically, a strongly-typed container like TypedDictionary, TypedList or TypedTuple are, it has enough similar characteristics that dropping the Enumeration class into the TypedContainers module felt reasonable. Consider:

An enumeration is a collection of values, like dictionaries, lists and tuples;
Also like dictionaries, lists, and tuples, enumerations are (or at least can be considered as a type);
Value-membership is also a shared characteristic. That is, all of the established types and enumerations both should allow a developer to determine whether some arbitrary value is a member of the instance.
Unlike those types, however, an enumeration has a fixed, immutable set of values.

Whether the member-values of an enumeration need to be of the same type or not is an open question, to my mind. If they are, so be it. But, ultimately, the purpose that an enumeration serves (mainly, that I'm aware of) is to collect some fixed set of values, which feels like a type, regardless.

So that (I hope) justifies the decision to put Enumeration into TypedContainers (shared at dl.dropbox.com/u/1917253/site-packages/TypedContainers.py).

By preference, I'd like to be able to use code structures like:

SomeStuff = Enumeration(
    'This is an enumeration of values for some purpose', 
    Good=1,
    Bad=-1,
    Indifferent=0
    )

# Dot-notation should be an option:
goodValue = SomeStuff.Good

# For comparison/membership-detection, "in" should be used:
if( goodValue in SomeStuff ):
    # do something

The creation of an enumeration (lines 1-6) should allow for the generation of an instance doc-string (though, to be truthful, I'm not sure that it'll ever be of much use). At present, this doc-string won't appear anywhere, but I'll eventually get my head together on the documentation side of things to work out a way to retrieve it for API documentation, so it might as well be available now, rather than having to come back and implement it later. The generation of member-names and values should be as simple as providing them suring the construction of the instance, and they should not be limited in number or type.

Use of dot-notation for specific member-names/-values should be allowed (line 9), and detection of whether some arbitrary value is present within the Enumeration should be as simple as possible (I prefer the in keyword, lines 12-13).

With all of these requirements/preferences in mind, here's the (updated) code that I generated to meet them:

class Enumeration( object ):
    """Provides a "formal" enumeration-type, allowing a single class-instance to be
used to provide dot-notation enumeration values."""

    ##################################
    # Class Attributes               #
    ##################################

    __isLocked = False
    _iterableItems = []
    _key = 0

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetMembers( self ):
        """Gets the members of the Enumeration as a Dictionary value."""
        return self.__members

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetMemberNames( self ):
        """Gets the names of the Enumeration's members."""
        return self.__members.keys()

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetMemberValues( self ):
        """Gets the values of the Enumeration's members."""
        return self.__members.values()

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    ##################################
    # Class Properties               #
    ##################################

    Members = property( _GetMembers, None, None, _GetMembers.__doc__ )
    MemberNames = property( _GetMemberNames, None, None, _GetMemberNames.__doc__ )
    MemberValues = property( _GetMemberValues, None, None, _GetMemberValues.__doc__ )

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'docString', None, 'The documentation-string for the Enumeration.' )
    @DocumentArgument( 'keyword', 'members', None, 'The members of the Enumeration, in the form "MemberName=MemberValue".' )
    @DocumentException( NotImplementedError, 'used as a base class' )
    @DocumentException( TypeError, 'not supplied with a keyword-argument set' )
    @DocumentException( KeyError, 'an enumeration member-name is specified that is used internally by the class (see it\'s properties)' )
    def __init__( self, docString, **members ):
        """Object constructor."""
        # Nominally final: Don't allow any class other than this one
        if self.__class__ != Enumeration:
            raise NotImplementedError( 'Enumeration is (nominally) a final class, and is not intended to be derived from.' )
        if type( members ) != types.DictType:
            raise TypeError( "Enumeration expected a keyword-list of member names and values." )
        self.__isLocked = False
        self._key = 0
        self.__members = {}
        forbiddenKeys = dir( self )
        for theMember in members:
            if theMember in forbiddenKeys:
                raise KeyError( "An Enumeration cannot override the %s key-name: It's reserved by the object." % ( theMember ) )
            self.__dict__[ theMember ] = members[ theMember ]
            self.__members[ theMember ] = members[ theMember ]
        if docString != None:
            self.__doc__ = docString + (' %s' % ( self.__members.keys() ) ).replace( "'", '' )
        self._iterableItems = self.__members.values()
        self.__isLocked = True

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def __setattr__( self, name, value ):
        """Intercepts attempts to set attribute-values, and prevents them from being set if the instance has been locked."""
        # Allow changes to self._key for iteration purposes:
        if not self.__isLocked or name == '_key':
            return object.__setattr__( self, name, value )
        raise AttributeError( 'Enumerations are immutable after instantiation.' )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def __iter__( self ):
        """Standard iterator handle."""
        self._key = 0
        return copy.copy( self )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentException( StopIteration, 'the end of an iteration against an instance is reached (per specifications for an iterable object)' )
    def next( self ):
        """Standard iteration mechanism."""
        try:
            thisKey = self._key
            self._key += 1
            return self._iterableItems[ thisKey ]
        except IndexError:
            self._key = 0
            raise StopIteration

__all__ += [ 'Enumeration' ]

Commentary

Line(s)

9-11

Provides a default value for the __isLocked attribute of the instance. Without this, the __setattr__ call made implicitly during object-construction to set it's value to False fails, since no attribute exists to check. _key is similarly set, to avoid it being similarly constrained.

17-30, 44-46

My typical property-getter and property-declaration structure, for read-only properties.

52-77

Object constructor:

63-64: Checks that the members argument is a dictionary, which is the default type for a keyword argument list. This may not be strictly necessary, but given that I anticipate the use of constructor-calls along the lines of MyEnumeration = Enumeration( **someDictionary ), it felt safer to leave it in place than to remove it.
65-68: Sets the default values for various properties, then gets every member-name of the instance (with dir()) in order to have a list of forbidden member-names for later checking/comparison.
69-73: Iterates through the provided member names/values dictionary, checks each key-name therein against the forbidden keys (raising a KeyError if there's a collision), and if everything's OK, it attaches the member name/value to the instance's __dict__ to make it available in a dot-notation syntax later. This is also where (line 72) the internal __members dictionary gets populated so that the various Member... properties and getters defined previously have something to work with.
74-75: Sets the __doc__ of the instance, making sure to include the provided member-names.
76: Sets the _iterableItems array up, so that it's available for use by the iteration methods later.
77: Locks the object so that any future use of __setattr__ will know that the object is locked, and should not be modified.

88-93

Leverages the built-in __setattr__ method to prevent the addition of new attributes or modification of existing ones once the instance is locked.

96-99, 101-111

The __iter__ method is Python's standard mechanism for iterating over iterator-types - and since I wanted Enumeration to support the in keyword to determine membership, it needs to be iterable. In this particular implementation, all it does is return a shallow copy of the original instance to be iterated over with next. Similarly, the next method is a standard iteration-mechanism, which returns the next item from the container/collection.

There are some new/unusual/interesting items in the unit-tests, I think, so I'll share those as well.

    class EnumerationDerived( Enumeration ):
        def __init__( self, docString, **members ):
            Enumeration.__init__( self, docString, **members )
    
    class testEnumeration( unittest.TestCase ):
        """Unit-tests the Enumeration class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass

        def getGoodValues( self ):
            return [
                {'name1':'value1', 'name2':'value2'},
                {'name1':True, 'name2':False},
                {'name1':-1, 'name2':0, 'name3':1},
                ]
        
        def testFinal( self ):
            """Testing final nature of the Enumeration class."""
            try:
                testObject = EnumerationDerived( 'docstring', Name='value' )
                self.fail( 'Enumeration is nominally a final class, and should not be extensible.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Attempting to instantiate a class derived from Enumeration should raise NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
    
        def testConstruction( self ):
            """Testing construction of the Enumeration class."""
            testValues = self.getGoodValues()
            for testValue in testValues:
                testObject = Enumeration( 'docstring', **testValue )
                self.assertEquals( testValue.keys(), testObject.MemberNames, 'An Enumeration instance created with %s as members should have %s as member names, but %s was returned instead.' % ( testValue, testValue.keys(), testObject.MemberNames ) )
                self.assertEquals( testValue.values(), testObject.MemberValues, 'An Enumeration instance created with %s as members should have %s as member values, but %s was returned instead.' % ( testValue, testValue.values(), testObject.MemberValues ) )
                self.assertEquals( testValue, testObject.Members, 'An Enumeration instance created with %s as members should have %s as member values, but %s was returned instead.' % ( testValue, testValue, testObject.Members ) )
            # Construction should fail with bad docstrings...
            badDocstrings = [ True, 1, object() ]
            for testValue in badDocstrings:
                try:
                    testObject = Enumeration( testValue, One=1, Two=2 )
                    self.fail( '%s should not be a valid docstring for an Enumeration' % ( testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Passing %s as a docstring should raise TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )
    
        def testPropertyCountAndTests( self ):
            """Testing the properties of the Enumeration class."""
            items = getMemberNames( Enumeration )[0]
            actual = len( items )
            expected = 3
            self.assertEquals( expected, actual, 'Enumeration is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the Enumeration class."""
            items = getMemberNames( Enumeration )[1]
            actual = len( items )
            expected = 1
            self.assertEquals( expected, actual, 'Enumeration is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

        # Test properties

        def testMemberNames( self ):
            """Tests the MemberNames property of the Enumeration class."""
            #Tested in construction
            pass

        def testMemberValues( self ):
            """Tests the MemberValues property of the Enumeration class."""
            #Tested in construction
            pass

        def testMembers( self ):
            """Tests the Members property of the Enumeration class."""
            #Tested in construction
            pass

        # Test methods

        def testnext( self ):
            """Tests the next method of the Enumeration class."""
            #Tested in iterability tests
            pass

        # Test iterabilility
        def testIterability( self ):
            """Tests the various iteration-dependent functions of the Enumeration class."""
            testValues = self.getGoodValues()
            for testValue in testValues:
                testObject = Enumeration( 'docstring', **testValue )
                for member in testValue:
                    memberValue = testValue[ member ]
                    self.assertTrue( memberValue in testObject, 'Test of %s being in the Enumeration\'s values should return true.' % ( memberValue ) )
            testObject = Enumeration( 'docstring', Zero=0, One=1, Two=2, Three=3 )
            testValue = 0
            for value in testObject:
                self.assertEquals( testValue, value, 'The iterated value should match' )
                testValue += 1

        # Test immutability after instantiation
        def testImmutability( self ):
            """Tests the immutability of the members/values of an Enumeration after it's been instantiated."""
            testObject = Enumeration( None, Good=1, Bad=-1, Indifferent=0 )
            try:
                testObject.Ugly = 2
                self.fail( 'Once instantiated, the members of an Enumeration should be immutable.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Once instantiated, modification of the members of an Enumeration should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__ ) )
            try:
                testObject.Good = 2
                self.fail( 'Once instantiated, the members of an Enumeration should be immutable.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Once instantiated, modification of the members of an Enumeration should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__ ) )

        # Test dot-notation access
        def testDotAccess( self ):
            """Tests dot-notation access to member values by name."""
            testObject = Enumeration( None, Good=1, Bad=-1, Indifferent=0 )
            self.assertTrue( hasattr( testObject, 'Good' ), 'Member names should be accessible as attributes of the Enumeration instance.' )
            self.assertTrue( hasattr( testObject, 'Bad' ), 'Member names should be accessible as attributes of the Enumeration instance.' )
            self.assertTrue( hasattr( testObject, 'Indifferent' ), 'Member names should be accessible as attributes of the Enumeration instance.' )
            self.assertEquals( testObject.Good, 1, 'Member values accessed through dot-notation should match the values supplied at creation.' )
            self.assertEquals( testObject.Bad, -1, 'Member values accessed through dot-notation should match the values supplied at creation.' )
            self.assertEquals( testObject.Indifferent, 0, 'Member values accessed through dot-notation should match the values supplied at creation.' )

        # Test reserved-name creation - all should raise errors
        def testReservedNames( self ):
            """Tests that the reserved names (existing attributes) aren't allowed as member-names."""
            names = [ '_key', 'Members', 'MemberNames', 'MemberValues' ]
            for name in names:
                members = { name:True }
                try:
                    testObject = Enumeration( 'docstring', **members )
                    self.fail( 'Creation of an Enumeration with a member-name of %s should raise KeyError' % ( name ) )
                except KeyError:
                    pass
                except Exception, error:
                    self.fail( 'Creation of an Enumeration with a member-name of %s should raise KeyError but %s was raised instead:\n  %s' % ( name, error.__class__.__name__, error ) )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testEnumeration ) )

Line(s)
14-19: While poking around at some proof-of-concept PHP code at work over the last few weeks, it occurred to me that a test-case class could be extended upon to provide at lease some of the typical functionality I find myself using. One of those typical items is the generation of lists of good and bad property-values for various cases. In this case, rather than repeating the "good" cases at least twice (see lines 33, 99), which could lead to the values being tested for those methods getting out of sync and leading to inconsistent testing, I'm calling one method to return all of the good values.; I may very well refactor the UnitTestUtilities module to make use of this concept (and while I'm there, maybe attach the oft-repeated testMethodCountAndTests and testPropertyCountAndTests methods so that I'll have a unit-test class that just does that automatically.
31-48: Apart from the fact that it makes use of the good-values method noted above, this is pretty typical for construction-testing. Note that the test-method tests both good and bad values.
70-83, 87-90: These test-methods probably look a bit odd, since they are required, but they have no useful implementation. The useful tests all exist elsewhere, so the required test-methods simply note that this is the case as a comment, and pass.
93-105: This marks the first test-method of this set that tests an overall characteristic of the class - in this case, it's iterability. The method tests both whether the in keyword works as expected/desired (line 100) for each member, and the "overall" iterability of instances (102-105).
108-124: This method tests that instances are immutable after creation (another "characteristic"-based test-method).
127-135: This method tests that member-names specified at construction are available as attributes of the instance by name, and that the values behind those attribute-names match what was supplied.
138-149: This test-method was almost missed... I realized just before I was first scheduling this for publication on the blog that I hadn't tested whether using any of the "reserved" names at construction would raise the errors I wanted. I could've gone back and added these tests to the constructor tests instead of making a new method specifically for them, but since it's kinda critical that class members not get destroyed, it seemed apropos to call the tests out in their own method.

And that wraps up the Enumeration...

Where to go from here?

2012-01-06T04:30:00.000-08:00

With MySQLConnector (and the supporting development that led up to it) complete, I'm at something of a loss as to where to go from here. My daily routine hasn't settled down yet, and though I'd hoped to be able to get some serious code-blogging time in over the week off from work between Christmas and New Year's, that hasn't happened. One complete computer rebuild, which pointed out a critical flaw in my home-system backup-plan, correcting that and trying like hell to back things up before the main drive on the old machine died completely. The random chaos of the holiday season. The demands of family while I'm home and they're awake. The list goes on and on. I did at least manage to get the data-connectors module (up through MySQLConnector) completed, which I'd promised myself I'd get done.

I have a list of possibles to pick from, some of which are dependent on other items on that list.

But the simple fact of the matter is that I'm kinda stuck here. If any readers have a preference what I should tackle next, please comment and let me know. Otherwise, I'll poke around a bit and see what, if anything, strikes my fancy (though I have to say that I'm leaning towards the MiniAppServer idea at present).

The MySQLConnector module

2012-01-04T04:30:00.000-08:00

Happy New Year, readers! The bulk of today's code is shared at dropbox.com/u/1917253/site-packages/MySQLConnector.py.

So the last leg of our journey into Python-database connectivity is (finally) the MySQLConnector class. There's been a lot of what I'd call "support" development to get to this point:

Three interfaces (IsDataConnector, IsQuery, and IsResultSet);
One abstract class (BaseDataConnector); and
Three concrete classes (Query, ResultSet and Record).

These took considerably longer to write about than I'd expected, though I'm glad that they provided enough material for the number of posts that they generated. There's also a fair amount of code that had to be written: With unit-tests, the DataConnectors module is just barely closer to 2,400 lines of code than 2,300, in about a 40:60 split between "real" code and test-code.

One (hopefully significant) advantage of this approach is that the creation of the MySQLConnector class is going to be crazy-simple. As a derivative of both BaseDataConnector and IsConfigurable, there's really not a whole lot of code that's specific to the MySQLConnector to be written, and other similar classes (say, a hypothetical PostGreSQLConnector) would require a similarly small level of effort. Consider that MySQLConnector has the following properties and methods:

The Configure method, from IsConfigurable, which may need specific implementation, but that was implemented at a basic level in BaseDataConnector.
The Connect method, required by IsDataConnector, which will need implementation.
The Database property, from BaseDataConnector, that is complete and tested until or unless we need to make implementation-specific changes.
The Execute method, required by IsDataConnector, which will need implementation.
The Host property, from BaseDataConnector, also complete and tested, barring implementation-specific changes.
The Password property, from BaseDataConnector, also complete and tested, barring implementation-specific changes.
The User property, from BaseDataConnector, also complete and tested, barring implementation-specific changes.
And, of course, unit-tests for the new class. Ideally, there should also be some system tests to test/prove out functionality against an actual database.

As I was working on MySQLConnector, I discovered that I'd never finished figuring out how to unit-test the Configure method of it, and that I was missing a Configuration class:

Configuration (from the Configuration module):

class Configuration( ConfigParser.ConfigParser, object ):
    """Represents configuration data derived from a text-file."""

    #################################
    # Class Attributes              #
    #################################

    #################################
    # Class Property-Getter Methods #
    #################################

    def _GetConfigFile( self ):
        """Gets the config-file-path specified at object creation."""
        return self._configFile

    def _GetSections( self ):
        """Gets the available configuration-sections."""
        if self._configSections == None:
            try:
                self.read( self._configFile )
            except Exception, error:
                raise AttributeError( '%s.Sections: Could not read configuration from %s. Original error:\n\t%s' % ( self.__class__.__name__, self._configFile, error ) )
            self._configSections = {}
            for theSection in self.sections():
                self._configSections[ theSection ] = {}
                for theItem in self.items( theSection, 1 ):
                    itemName = theItem[0]
                    itemValue = theItem[1]
                    if itemValue[0] in [ '[', '{', '(' ]:
                        # Evaluate the value instead... Ugly, but viable...
                        itemValue = eval( itemValue )
                    self._configSections[ theSection ][ itemName ] = itemValue
        return self._configSections

    #################################
    # Class Property-Setter Methods #
    #################################

    #################################
    # Class Properties              #
    #################################

    ConfigFile = property( _GetConfigFile, None, None, _GetConfigFile.__doc__ )
    Sections = property( _GetSections, None, None, _GetSections.__doc__ )

    #################################
    # Object Constructor            #
    #################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'filePath', None, '(String) The path to the configuration-file.' )
    def __init__( self, filePath ):
        """Object constructor."""
        ConfigParser.ConfigParser.__init__( self )
        self._configSections = None
        self._configFile = filePath

    #################################
    # Object Destructor             #
    #################################

    #################################
    # Class Methods                 #
    #################################

__all__ += [ 'Configuration' ]

This will likely need to be refactored at some point, since there are other configuration mechanisms than the ConfigParser base here. For now, though, I'll leave it be. Configuration instances are basically just wrappers around ConfigParser instances, that provide ConfigFile and Sections properties. The ConfigFile is probably pretty straightforward - it's a pointer to the configuration-file that the instance uses. Sections might be a little harder to understand, though - I wanted to have a simple access mechanism for the configuration-sections of a given configuration-file, in order to be able to retrieve section-items without having to make a function-call every time.

With that in place, I could finish the unit-tests for BaseDataConnector:

        def testConfigure( self ):
            """Unit-tests the Configure method of the BaseDataConnector abstract class."""
            # First, we need to build out a configuration-file
            hostName = 'hostName'
            databaseName = 'databaseName'
            userName = 'userName'
            userPassword = 'userPassword'
            configString = """[Default Section]
SectionName1: SectionValue1
SectionName2: SectionValue2

[Data Access]
host: %s
database: %s
user: %s
password: %s
""" % ( hostName, databaseName, userName, userPassword )
            fp = open( 'test.cfg', 'w' )
            fp.write( configString )
            fp.close()
            # Now we need to create the configuration object and read in the configuration values
            configParser = Configuration( 'test.cfg' )
            # Now, finally, we can test the configuration
            testObject = BaseDataConnectorDerived()
            testObject.Configure( configParser, 'Data Access' )
            self.assertEquals( testObject.Host, hostName, 'The host name should be retrievable from configuration' )
            self.assertEquals( testObject.Database, databaseName, 'The database name should be retrievable from configuration' )
            self.assertEquals( testObject.User, userName, 'The user name should be retrievable from configuration' )
            self.assertEquals( testObject.Password, userPassword, 'The user password should be retrievable from configuration' )
            os.unlink( 'test.cfg' )

All this is doing is creating a hard-coded (and very simple) configuration-file, with a Data Access section that contains the host, database and user names for the connector, and the password that it should use. Looking at lines 22 and 24-25, creation of a BaseDataConnector-derived instance should always be feasible with this sort of code-structure: Create an instance with no arguments, create a configuration instance, then call the Configure method of the data-connector instance.

It may bear noting that MySQLConnector doesn't have any sort of direct support for cursors - that may be something that I'll add in to BaseDataConnector at some point in the future, if there's support for it across MySQL (yes), PostgreSQL (not sure), and ODBC (also not sure), which are the three data-connection types I expect this to support in the long run.

MySQLConnector (Nominal final class):

#####################################
# Convenience imports               #
#####################################

# See To-do list on MySQLConnector.__init__

#####################################
# Classes defined in the module     #
#####################################

class MySQLConnector( BaseDataConnector, IsConfigurable, object ):
    """Class doc-string."""

    ##################################
    # Class Attributes               #
    ##################################

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetConnection( self ):
        """Gets the connection to the specified MySQL database."""
        if self._connection == None:
            self.Connect()
        return self._connection

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, 'The name of the database the connection will be made to.' )
    def _SetDatabase( self, value ):
        if self._connection != None:
            raise AttributeError( '%s.Database cannot be reset once a connection is established' % ( self.__class__.__name__ ) )
        BaseDataConnector._SetDatabase( self, value )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, 'The name or IP address of the host that the database being connected to lives on.' )
    def _SetHost( self, value ):
        if self._connection != None:
            raise AttributeError( '%s.Host cannot be reset once a connection is established' % ( self.__class__.__name__ ) )
        BaseDataConnector._SetHost( self, value )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, 'The password that will be used to connect to the database.' )
    def _SetPassword( self, value ):
        if self._connection != None:
            raise AttributeError( '%s.Password cannot be reset once a connection is established' % ( self.__class__.__name__ ) )
        BaseDataConnector._SetPassword( self, value )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, 'The user-name that will be used to connect to the database.' )
    def _SetUser( self, value ):
        if self._connection != None:
            raise AttributeError( '%s.User cannot be reset once a connection is established' % ( self.__class__.__name__ ) )
        BaseDataConnector._SetUser( self, value )

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _DelConnection( self ):
        self._connection = None

    ##################################
    # Class Properties               #
    ##################################

    Connection = property( _GetConnection, None, None, 'Gets the connection to the specified MySQL database.' )
    Database = property( BaseDataConnector._GetDatabase, _SetDatabase, BaseDataConnector._DelDatabase, BaseDataConnector._GetDatabase.__doc__ )
    Host = property( BaseDataConnector._GetHost, _SetHost, BaseDataConnector._DelHost, BaseDataConnector._GetHost.__doc__ )
    Password = property( BaseDataConnector._GetPassword, _SetPassword, BaseDataConnector._DelPassword, BaseDataConnector._GetPassword.__doc__ )
    User = property( BaseDataConnector._GetUser, _SetUser, BaseDataConnector._DelUser, BaseDataConnector._GetUser.__doc__ )

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'keyword', 'parameters', 'host', 'The name or IP-address of the host that the database resides upon.' )
    @DocumentArgument( 'keyword', 'parameters', 'database', 'The name of the database that the connection will be made to.' )
    @DocumentArgument( 'keyword', 'parameters', 'user', 'The user-name used to connect to the database.' )
    @DocumentArgument( 'keyword', 'parameters', 'password', 'The password used to connect to the database.' )
    @ToDo( 'Figure out what items from DataConnectors should be imported for convenience in using the MySQLConnector class.' )
    def __init__( self, **parameters ):
        """Object constructor."""
        # Nominally final: Don't allow any class other than this one
        if self.__class__ != MySQLConnector:
            raise NotImplementedError( 'MySQLConnector is (nominally) a final class, and is not intended to be derived from.' )
        self._connection = None
        BaseDataConnector.__init__( self, **parameters )

    ##################################
    # Object Destructor              #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def __del__( self ):
        self._Close()

    ##################################
    # Class Methods                  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _Close( self ):
        if self._connection:
            self._connection.close()

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def Connect( self ):
        """Connects to the database specified by Database, residing on Host, using User and Password to connect."""
        self._connection = MySQLdb.connect( 
            host=self._host, 
            db=self._database, 
            user=self._user, 
            passwd=self._password
            )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'query', None, 'The IsQuery object that will be executed against the object\'s Connection.' )
    def Execute( self, query ):
        """Executes the provided IsQuery object's query against the object's connection, returning one or more results."""
        if not isinstance( query, IsQuery ):
            raise TypeError( '%s.Execute expects an instance of IsQuery as it\'s query argument.' % ( self.__class__.__name__ ) )
        self.Connection.query( query.Sql )
        resultSets = []
        resultSet = []
        results = self.Connection.store_result()
        while results:
            row = list( results.fetch_row( 1, 1 ) )
            while row:
                resultSet += row
                row = results.fetch_row( 1, 1 )
            if self.Connection.next_result() == 0:
                resultSets.append( resultSet )
                resultSet = []
                results = self.Connection.store_result()
            else:
                break
        resultSets.append( resultSet )
        return resultSets

__all__ += [ 'MySQLConnector' ]

Line(s)

5

This is a fine-tuning item that will almost certainly evolve in an obvious way as I start writing code that actually uses MySQLConnector: There will almost certainly be some (small?) set of imports from the base DataConnectors module that will be needed frequently enough to warrant importing them as part of the MySQLConnector import. Off the top of my head, I'd expect at least Query and maybe ResultSet to be needed, but I'll wait to see what actually crops up as I start using the module...

23-27

An almost-typical property getter method, the main difference being that the underlying MySQLdb Connection object isn't instantiated until it's needed.

33-59

Overridden property-setter methods. The only additional functionality that they provide is to prevent the modification of any connection-control property (the host, database- and user-name, and password used for the connection) once the connection has been instantiated. The rationale for this is, basically, that once a connection has been established, changes to that connection should not be allowed. Chances are good that an error of some type would be thrown anyway, but since I'd rather raise errors as close to their "real" source as possible, I'm explicitly causing that to happen here.

73-77

The actual property declarations. Since most of these are a mix of the functionality from BaseDataConnector (for the _Get... methods) and local implementations (_Set... methods), there's a mix of complete external method-references and "normal" local references to accommodate.

101-103, 109-112

I'm not sure that a formal object-destructor's actually needed, since the underlying MySQLdb Connection object probably closes the database connection all on it's own, but to be safe, I'm providing one that calls to the _Close method to make sure that the connection is closed cleanly when the object is destroyed.

The _Close method is necessary because when the destructor fires, it doesn't recognize object properties. I think I mentioned this same pattern somewhere before, but I cannot recall where, exactly.

114-122

A public method to open the database-connection with the instance's parameters.

124-144

The method that actually queries against the database, returning a list of lists of dictionaries, ready to be passed to ResultSet constructors.

128-129

Typical type-checking of the supplied query argument.

130

Run the provided Query's SQL against the database, readying the results for formatting/digestion.

131, 132

Prep the lists of all results, and the first list of rows for use.

133

Get (and store) the first result-set returned.

This uses store_result instead of use_result, in the belief that a well-designed application (and database) should rarely return so many records that storing them wouldn't provide faster throughput than keeping the database-connection open and retrieving result-sets one at a time.

134

For so long as there are results:

135: Get the first row of the current result-set
136-137: For so long as there's a valid row, add the row to the current result-set, and get the next row.
139-142: Once the current result's row-set has been processed, if there is another result-set after the current one, append the current results to the list to be returned, reset the row-list, and get/store the next result-set.
143-144: If there are no other result-sets, then exit the loop.

145

Make sure that the last set of rows gets attached to the results to be returned before returning them.

Unit-tests

Some of the unit-tests (the ones that are very basic system-tests, all in the testExecute method) require some actual tables on the test_database database:

CREATE TABLE `test_database`.`test_table_1` (
  `id` INT UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'The record-id',
  `first_name` VARCHAR(30)  NOT NULL COMMENT 'A first name',
  `last_name` varchar(30)  NOT NULL COMMENT 'A last-name',
  PRIMARY KEY (`id`)
)
ENGINE = MyISAM
COMMENT = 'A test-table';

INSERT INTO test_table_1 ( first_name, last_name )
VALUES ( 'Brian', 'Allbee' ), ( 'Joe', 'Smith' );

CREATE TABLE `test_database`.`test_table_2` (
  `id` int UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'Record-ID',
  `name_id` int  NOT NULL 
      COMMENT 'ID of the record from test_table_1 that relates to this record',
  `email` varchar(60)  NOT NULL COMMENT 'An email address',
  PRIMARY KEY (`id`)
)
ENGINE = MyISAM
COMMENT = 'Another test table';

INSERT INTO test_table_2 ( name_id, email )
VALUES ( 1, 'brian.allbee@somedomain.com' ), ( 2, 'joe.smith@somedomain.com' );

    class MySQLConnectorDerived( MySQLConnector ):
        def __init__( self, **parameters ):
            MySQLConnector.__init__( self, **parameters )
        def __del__( self ):
            try:
                MySQLConnector.__del__( self )
            except:
                pass
    
    class testMySQLConnector( unittest.TestCase ):
        """Unit-tests the MySQLConnector class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass

        def getLiveConnection( self ):
            """Returns a MySQLConnector instance pointing to a common local test-database."""
            return MySQLConnector( host='localhost', database='test_database', user='test_user', password='test_password' )
        
        def testFinal( self ):
            """Testing final nature of the MySQLConnector class."""
            try:
                testObject = MySQLConnectorDerived()
                self.fail( 'MySQLConnector is nominally a final class, and should not be extendable.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Instantiating a derivation of MySQLConnection should raise NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
    
        def testConstruction( self ):
            """Testing construction of the MySQLConnector class."""
            testObject = MySQLConnector()
            self.assertTrue( isinstance( testObject, MySQLConnector ), 'Instances of MySQLConnector should be instances of MySQLConnector.' )
            self.assertEquals( testObject.Host, None, 'With no host supplied at construction, it\'s property value should be None' )
            self.assertEquals( testObject.Database, None, 'With no database supplied at construction, it\'s property value should be None' )
            self.assertEquals( testObject.User, None, 'With no user supplied at construction, it\'s property value should be None' )
            self.assertEquals( testObject.Password, None, 'With no password supplied at construction, it\'s property value should be None' )
            testHost = 'testHost'
            testDatabase = 'database'
            testUser = 'testUser'
            testPassword = 'testPassword'
            testObject = MySQLConnector( host=testHost, database=testDatabase, user=testUser, password=testPassword )
            self.assertEquals( testObject.Host, testHost, 'The host supplied at construction should be present in the object\'s properties' )
            self.assertEquals( testObject.Database, testDatabase, 'The database supplied at construction should be present in the object\'s properties' )
            self.assertEquals( testObject.User, testUser, 'The user supplied at construction should be present in the object\'s properties' )
            self.assertEquals( testObject.Password, testPassword, 'The password supplied at construction should be present in the object\'s properties' )
    
        def testPropertyCountAndTests( self ):
            """Testing the properties of the MySQLConnector class."""
            items = getMemberNames( MySQLConnector )[0]
            actual = len( items )
            expected = 5
            self.assertEquals( expected, actual, 'MySQLConnector is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the MySQLConnector class."""
            items = getMemberNames( MySQLConnector )[1]
            actual = len( items )
            expected = 3
            self.assertEquals( expected, actual, 'MySQLConnector is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

        # Unit-test properties

        def testConnection( self ):
            """Unit-tests the Connection property of the MySQLConnector class."""
            testObject = MySQLConnector( host='localhost', database='no_such_database', user='test_user', password='test_password' )
            try:
                testObject.Connect()
                self.fail( 'Trying to connect to a non-existant database should raise OperationalError.' )
            except MySQLdb.OperationalError:
                pass
            except Exception, error:
                self.fail( 'Trying to connect to a non-existant database should raise OperationalError, but %s was raised instead:\n  %s.' % ( error.__class__.__name__, error ) )

        def testDatabase( self ):
            """Unit-tests the Database property of the MySQLConnector class."""
            testObject = self.getLiveConnection()
            oldDatabase = testObject.Database
            testObject.Database = 'ook'
            self.assertEquals( testObject.Database, 'ook', 'Before a connection is made, the object should allow modification of the Database property.' )
            testObject.Database = oldDatabase
            testObject.Connect()
            try:
                testObject.Database = 'ook'
                self.fail( 'Once a connection is established, the Database property should be immutable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Modifying the Database property after connection is establilshed should raise AttributeError, bu %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testHost( self ):
            """Unit-tests the Host property of the MySQLConnector class."""
            testObject = self.getLiveConnection()
            oldHost = testObject.Host
            testObject.Host = 'ook'
            self.assertEquals( testObject.Host, 'ook', 'Before a connection is made, the object should allow modification of the Host property.' )
            testObject.Host = oldHost
            testObject.Connect()
            try:
                testObject.Host = 'ook'
                self.fail( 'Once a connection is established, the Host property should be immutable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Modifying the Host property after connection is establilshed should raise AttributeError, bu %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testPassword( self ):
            """Unit-tests the Password property of the MySQLConnector class."""
            testObject = self.getLiveConnection()
            oldPassword = testObject.Password
            testObject.Password = 'ook'
            self.assertEquals( testObject.Password, 'ook', 'Before a connection is made, the object should allow modification of the Password property.' )
            testObject.Password = oldPassword
            testObject.Connect()
            try:
                testObject.Password = 'ook'
                self.fail( 'Once a connection is established, the Password property should be immutable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Modifying the Password property after connection is establilshed should raise AttributeError, bu %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

#        def testQueue( self ):
#            """Unit-tests the Queue property of the MySQLConnector class."""
#            pass

        def testUser( self ):
            """Unit-tests the User property of the MySQLConnector class."""
            testObject = self.getLiveConnection()
            oldUser = testObject.User
            testObject.User = 'ook'
            self.assertEquals( testObject.User, 'ook', 'Before a connection is made, the object should allow modification of the User property.' )
            testObject.User = oldUser
            testObject.Connect()
            try:
                testObject.User = 'ook'
                self.fail( 'Once a connection is established, the User property should be immutable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Modifying the User property after connection is establilshed should raise AttributeError, bu %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        # Unit-test methods

        def testConfigure( self ):
            """Unit-tests the Configure method of the MySQLConnector class"""
        def testConfigure( self ):
            """Unit-tests the Configure method of the BaseDataConnector abstract class."""
            # First, we need to build out a configuration-file
            hostName = 'localhost'
            databaseName = 'test_database'
            userName = 'test_user'
            userPassword = 'test_password'
            configString = """[Default Section]
SectionName1: SectionValue1
SectionName2: SectionValue2

[Data Access]
host: %s
database: %s
user: %s
password: %s
""" % ( hostName, databaseName, userName, userPassword )
            fp = open( 'test.cfg', 'w' )
            fp.write( configString )
            fp.close()
            # Now we need to create the configuration object and read in the configuration values
            configParser = Configuration( 'test.cfg' )
            # Now, finally, we can test the configuration
            testObject = MySQLConnector()
            testObject.Configure( configParser, 'Data Access' )
            self.assertEquals( testObject.Host, hostName, 'The host name should be retrievable from configuration' )
            self.assertEquals( testObject.Database, databaseName, 'The database name should be retrievable from configuration' )
            self.assertEquals( testObject.User, userName, 'The user name should be retrievable from configuration' )
            self.assertEquals( testObject.Password, userPassword, 'The user password should be retrievable from configuration' )
            os.unlink( 'test.cfg' )

        def testConnect( self ):
            """Unit-tests the Connect method of the MySQLConnector class."""
            pass # Tested in the various property-tests above...

        def testExecute( self ):
            """Unit-tests the Execute method of the MySQLConnector class."""
            testObject = self.getLiveConnection()
            # Single-table, single row, single field
            MyQuery = Query( testObject, "SELECT first_name FROM test_table_1 LIMIT 1;" )
            testResults = testObject.Execute( MyQuery )
            self.assertEquals( len( testResults ), 1, '%s should return a result-set' % ( MyQuery.Sql ) )
            self.assertEquals( len( testResults[0] ), 1, '%s should return a single row' % ( MyQuery.Sql ) )
            self.assertEquals( testResults[0][0].keys(), [ 'first_name' ], '%s should return only the first_name field' % ( MyQuery.Sql ) )
            # Single-table, single row, all fields
            MyQuery = Query( testObject, "SELECT * FROM test_table_1 LIMIT 1;" )
            testResults = testObject.Execute( MyQuery )
            self.assertEquals( len( testResults ), 1, '%s should return a result-set' % ( MyQuery.Sql ) )
            self.assertEquals( len( testResults[0] ), 1, '%s should return a single row' % ( MyQuery.Sql ) )
            self.assertEquals( len( testResults[0][0] ), 3, '%s should return three fields' % ( MyQuery.Sql ) )
            self.assertEquals( testResults[0][0].keys(), [ 'first_name', 'last_name', 'id' ], '%s should return first_name, last_name and id fields' % ( MyQuery.Sql ) )
            # Single-table, all rows, one field
            MyQuery = Query( testObject, "SELECT first_name FROM test_table_1;" )
            testResults = testObject.Execute( MyQuery )
            self.assertEquals( len( testResults ), 1, '%s should return a result-set' % ( MyQuery.Sql ) )
            self.assertEquals( len( testResults[0] ), 2 )
            self.assertEquals( testResults[0][0].keys(), [ 'first_name' ] )
            self.assertEquals( testResults[0][0].keys(), testResults[0][1].keys(), 'Field-names in different rows of the result-set should be identical' )
            # Single-table, all rows, all fields
            MyQuery = Query( testObject, "SELECT * FROM test_table_1;" )
            testResults = testObject.Execute( MyQuery )
            self.assertEquals( len( testResults ), 1, '%s should return a result-set' % ( MyQuery.Sql ) )
            self.assertEquals( len( testResults[0] ), 2, '%s should return two rows' % ( MyQuery.Sql ) )
            self.assertEquals( len( testResults[0][0] ), 3, '%s should return three fields' % ( MyQuery.Sql ) )
            self.assertEquals( testResults[0][0].keys(), [ 'first_name', 'last_name', 'id' ], '%s should return first_name, last_name and id fields' % ( MyQuery.Sql ) )
            self.assertEquals( testResults[0][0].keys(), testResults[0][1].keys(), 'Field-names in different rows of the result-set should be identical' )
            # Multiple tables (joined), all fields
            MyQuery = Query( testObject, "SELECT t1.*, t2.* FROM test_table_1 t1 LEFT JOIN test_table_2 t2 ON t1.id=t2.name_id;" )
            testResults = testObject.Execute( MyQuery )
            self.assertEquals( len( testResults ), 1, '%s should return a result-set' % ( MyQuery.Sql ) )
            self.assertEquals( len( testResults[0] ), 2, '%s should return two rows' % ( MyQuery.Sql ) )
            self.assertEquals( testResults[0][0].keys(), ['first_name', 'last_name', 't2.id', 'email', 'name_id', 'id'] )
            self.assertEquals( testResults[0][0].keys(), testResults[0][1].keys(), 'Field-names in different rows of the result-set should be identical' )
            # Multiple tables, multiple result-sets
            MyQuery = Query( testObject, "SELECT * FROM test_table_1;SELECT * FROM test_table_2;" )
            testResults = testObject.Execute( MyQuery )
            self.assertEquals( len( testResults ), 2, '%s should return two result-sets' % ( MyQuery.Sql ) )
            self.assertEquals( len( testResults[0] ), 2, '%s should return two rows in it\'s first result-set' % ( MyQuery.Sql ) )
            self.assertEquals( len( testResults[1] ), 2, '%s should return two rows in it\'s second result-set' % ( MyQuery.Sql ) )
            self.assertEquals( testResults[0][0].keys(), [ 'first_name', 'last_name', 'id' ], '%s should return first_name, last_name and id fields' % ( MyQuery.Sql ) )
            self.assertEquals( testResults[0][0].keys(), testResults[0][1].keys(), 'Field-names in different rows of the result-set should be identical' )
            self.assertEquals( testResults[1][0].keys(), ['name_id', 'id', 'email'], '%s should return name_id, id and email fields' % ( MyQuery.Sql ) )
            self.assertEquals( testResults[1][0].keys(), testResults[1][1].keys(), 'Field-names in different rows of the result-set should be identical' )
            # Query that returns no results
            MyQuery = Query( testObject, "SELECT * FROM test_table_1 WHERE first_name='ook';" )
            testResults = testObject.Execute( MyQuery )
            self.assertEquals( len( testResults ), 1 )
            self.assertEquals( len( testResults[0] ), 0 )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testMySQLConnector ) )

Line(s)
1-8: An almost-typical derived-test class. The main difference here is the inclusion of an explicit object-destructor that cannot throw errors.; Unit-testing the object-destructor in any sort of meaningful way has eluded me still - I don't like not having such a test, but I'm recdonciled to having to take it on faith that the destructor is doing what it's supposed to do...
19-21: A convenience method to return a valid MySQLConnector instance pointing to the test database.
23-31: Standard final-class testing.
33-49: Object-construction testing. Note that the properties being passed as arguments to the constructor are being explicitly verified, though they shouldn't have to be as long as MySQLConnector remains true to the interface of BaseDataConnector. These are in place as much to make sure that potential future breaking changes underneath the MySQLConnector class would be caught before introducing bugs to a production system.

With the exception of the testConfigure and testExecute methods, most of the rest of the tests provided here are pretty typical. The testConfigure method is almost identical to the same method in the test-suite for BaseDataConnector, but is provided as a separate test-set in case of future modification to the Configure method of either BaseDataConnector or MySQLConnector.

testExecute, on the other hand, is an entirely different flavor of unicorn. This is really more of a system/integration test than a "real" unit-test, though I think it may straddle the line to some degree. It's necessary because the Execute method of MySQLConnector exists, but at the same time, there's no good way to really test it without making a connection to the database, and running queries against that connection. I'll hammer out some additional (and more detailed) integration tests that will eventually be added to the test-suite for MySQLConnector, but for now, this feels pretty good: Given the two tables of two rows present in the test database, I'm pretty confident that I've hit all of the meaningful variations of number of results, rows and fields that could come up against that data-set.

Taking a break...

2011-12-23T04:30:00.000-08:00

With the changes in my routine after my recent job-change, I'm going to take a week off from posting, and try to get more backlog built up during the week off from work.

Check back in on January 4th - I should have material enough written by then to resume my normal posting-schedule, I hope... And, at the very least, I'll have finished MySQLConnector, and started in on some system/integration testing of it.

Defending Class Finality in Python

2011-12-21T04:30:00.000-08:00

Back when Record was defined in DataConnectors (part 4), I had an inkling that a nominally final class would be viewed as "not Pythonic." I still struggle with what, exactly "pythonic" really means, though there are several results that look promising for various searches on Google, including pythonic code. The reason that the precise definition of Pythonic is a sticking point for me (at least right now) is that when I was looking to see if there was a standard mechanism for making final (non-extendable) classes, I ran across several mail-list items, blog-posts, and other resources on the web that all seemed to present more or less the same reason for discouraging class finality in any way:

Who are you to say that a class cannot be extended?

Having given it some thought, I have a two-part response (not necessarily an answer) to that question:

I am the person who created the class in the first place, who thus arguably knows (and at least should know) what all of the design- and implementation-decisions were for it, and who is the most likely to know what is going to break if a haphazard extension of that class is made.

Given the very nature of Python, and (at least for my part) the intention of releasing the code rather than just the executable in good open-source tradition, by making a class nominally final as I have in a few cases so far, I am not saying that the class cannot be extended. Anyone who feels the need to extendit has access to the code, and can remove the nominally-final constraint I have put in place. What I am saying is that the class wasn't intended to be extended, and if you want to extend it, you'd better at least look at my code first.

As to what "Pythonic" means...? Well, this seems to be a popular (and detailed) explanation. I'm not very far into it just yet, but at the very least, it looks to be an enjoyable read...

The DataConnectors module (part 6)

2011-12-19T04:30:00.001-08:00

So, the last generic-but-concrete class that I was working on, you may remember, was the Query class. It was stalled for a time because I wasn't sure how I really wanted the process of executing a query against a database to work. My primary concern was which object was going to be responsible for what part of the result-acquisition.

In retrospect, this feels to me now like the entire question is tied to making a decision about where the line should be drawn for the logical encapsulation-points for the relevant classes, and to some extent, it's also partly a question of how tightly coupled the classes themselves are. Actually, there are several principles that I'm trying to find a good balance between for this part of the object-stack, so I'll list them out, and examine different scenarios against them:

Encapsulate what varies: "When designing software, look for the portions most likely to change and prepare them for future expansion by shielding the rest of the program from that change. Hide the potential variation behind an interface. Then, when the implementation changes, software written to the interface doesn't need to change. This is called encapsulating variation." (from blogs.msdn.com/b/steverowe). There may be other methods than hiding things behind an interface, but that's likely the most common (and best?) practice.
Program to an interface, not to an implementation: Any two objects that conform to the same interface should be functionally interchangeable, so that other objects that care only about that interface then don't have to know or care about the implementation underneath it.
The Single Responsibility Principle: Any given object should have a single responsibility, that responsibility should be entirely encapsulated within the class, and it's properties and methods should be narrowly aligned with that responsibility.

I've listed these in the order of priority that I think makes the most sense for this particular problem, my argument being that of all of these, the Single Responsibility Principle is the most flexible and I'm all but certain that encapsulation of what varies the least flexible. Programming to an interface rather than to an implementation will, I think, be implicit with any reasonable design for the problem, but should be kept in mind as a higher priority than the class' responsibility.

Scenario 1: Query handles everything (almost)

The first scenario centers around the concept that the Query is responsible for the storage of the SQL to be executed, the execution of that SQL against the data-source, and the handling and storage of results returned from that execution. In this scenario, the Query must have (or be provided) a data-source object (derived from BaseDataConnector) which must have a method to execute an arbitrary SQL string against the database it represents, and that returns results to the Query. In keeping with the idea that the Query holds as much relevant responsibility as possible, those results would probably not be in ResultSet/Record instances.

Encapsulating what varies

Since the most likely point of variation is the specific mechanism that executes a SQL string against a data-source, and what structure/format those results come back in, encapsulating that in the BaseDataConnector-derived data-source feels pretty good.

Programming to an interface, not to an implementation

This scnario also feels pretty good in this respect, since Query doesn't need to know (or care) what kind of BaseDataConnector it's data-source is, just that it is a BaseDataConnector.

The Single Responsibility Principle

Assuming that the BaseDataConnector returns the simplest possible structure/format of it's results (most likely a list of dictionary-instances), and that Query is responsible for taking that raw result-set and converting them into ResultSet/Record structures, the responsibilities for each class might be stated something like so:

BaseDataConnector: Manages internal and/or low-level connection to and interaction with a back-end data-source.
Query: Provides user/developer-level mechanisms to interact with a back-end database.

Other Considerations

I'm pretty sure that I noted, some few posts ago, that for performance reasons it would be desirable to be able to queue up queries in order to minimize the number of connections made against the back-end database. This approach doesn't prevent that sort of capability, but I suspect that it would be more difficult to implement. It would almost certainly require a change to the responsibility of the BaseDataConnector-derived classes, if only to provide some mechanism to be able to associate an arbitary number of ResultSet instances with a particular Query object, but since creation of a ResultSet and it's associated Record instances to indicate the beginning of a query-association can be done in code, not as part of the "real" data-access, this may not be too onerous.

An example of code using this structure would look something like this:

# Create the data-source, where DataConnector is derived from BaseDataConnector
dataSource = DataConnector( host='host_name', database='database_name', user='user_name', password='password' )
# Create the Query instance, with SQL that returns two result-sets
sql = """SELECT somefield 
    FROM sometable 
    ORDER BY somefield;
SELECT someotherfield
    FROM sometable
    ORDER BY someotherfield;"""
myQuery = Query( dataSource, sql )
someResults, someOtherResults = myQuery.Results

On reflection, the idea of queueing up multiple queries is, perhaps, not as critical as I'd thought. Consider that this example returns multiple result-sets, and is managed in a single Query instance. The performance concerns about multiple database-connections could be mitigated easily enough by taking this sort of approach, provided that the number of result-sets returned by any given chunk of SQL were known and didn't change based on input parameters. This approach is feeling better and better to me the more I think on it.

Scenario 2: BaseDataConnector handles everything (almost)

This scenario's concept is diametrically opposed from the first: Here, the BaseDataConnector-derived objects handle as much of the interaction as possible, while Query becomes little more than a dumb data-object, containing the SQL to be executed, if it exists at all.

Encapsulating what varies

The encapsulation of implementation-variations is not significantly different here from the first scenario - it's still centered around the potential differences between different database-engines, how their connection and query-execution works, and what the results-structure looks like.

Programming to an interface, not to an implementation

Again, not significantly different from the first scenario - the specifics of what the interface (at the BaseDataConnector level) provides change, but we're still keeping the implementation hidden from the rest of the world.

The Single Responsibility Principle

This scenario does make drastic changes to the responsibilities of the objects, though. Fundamentally, there's no real need for any Query objects at all, since the actual SQL could just be passed to a Query method in a data-source object, and the results retrieved directly from there. If we assume that there's no need to have a Query object at all, the responsibilities break down to:

BaseDataConnector: Manages all interaction with a back-end data-aource, including execution of queries against it.

Other Considerations

This scenario, assuming that there is no Query object in the mix at all, pretty much prohibits any sort of queueing of queries. If we assume that a single SQL execution can be unpacked to a number of result-set variables in a manner similar to the example code for the first scenario, maybe this consideration is moot. The only thing we lose in this scenario is the ability to have state-data associated with the SQL of the query.

An example of code using this structure would look something like this:

# Create the data-source, where DataConnector is derived from BaseDataConnector
dataSource = DataConnector( host='host_name', database='database_name', user='user_name', password='password' )
# execute a query against the data-source
sql = """SELECT somefield 
    FROM sometable 
    ORDER BY somefield;
SELECT someotherfield
    FROM sometable
    ORDER BY someotherfield;"""
someResults, someOtherResults = dataSource.Query( sql )

I'm going to stop with these two scenarios, at least for now, because the desire for state-data/SQL association is a point woth more detailed consideration. When I originally conceived the DataConnectors module, it was, admittedly, with a specific eye towards a structure more along the lines of the structure that I'd gotten accustomed to as a CFML programmer. That included some query- or result-set-specific properties being available, including:

Data-source type (through a data-source association);
Authentication and authorization (also through a data-source association, but easily converted into a host/database/user/password structure);
Timeout control (though I don't know that I ever used this functionality);
Caching control;
Association of results to variables within the code;
Execution-time information; and
Paging of results;

Of these, the first two can be converted into object-constructor arguments easily enough, by passing the authentication/authorization values as simple values, and the data-source type as a class derived from BaseDataConnector. They don't require any sort of state-data at a Query-object level at all. Timeout control, if it's even something that can be usefully implemented, doesn't feel like it would require any local state-data.

Caching control might, though, depending on how that caching were implemented. If it's implemented outside the database (say, by picking a result-set and writing it to a file), the query would potentially need to be able to keep track of how long the cached data-sets were valid. Since data-source-side caching may or may not be available for any given database-engine, that's a point of variation that should be encapsulated somewhere (though where is still an open question).

So long as we assume that the unpacking of results (as shown in the previous code-examples) will work, the association of results to variables is moot.

Execution-time information would require state-data somewhere, and the Query object seems like the only place that makes sense. It can't be usefully associated as a ResultSet level, since the number of result-sets from a given query will vary - particularly if multiple result-sets are expected with any frequency.

Paging of results is probably something that makes more sense at the ResultSet level, so it can be safely disregarded for the purpose of discussion of BaseDataConnector and Query responsibilities.

With all of this in mind, my decision is to make the Query object as smart as possible, and the BaseDataConnector-derived objects pretty simple (though not completely dumb). I believe that this will afford the best development experience in the long run, and it keeps the possibility of query-state-dependent capabilities open on a longer-term basis, should there be a need for them somewhere down the line. With all of that in mind, then, the IsDataConnector interface loses it's Query method in favor of an Execute method, with changes made accordingly to it's unit-tests:

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'query', None, 'The IsQuery object that will be executed against the object\'s Connection.' )
    def Execute( self, query ):
        """Executes the provided IsQuery object's query against the object's connection, returning one or more results."""
        raise NotImplementedError( '%s.Query is not implemented as required by IsDataConnector.' % ( self.__class__.__name__ ) )

        def testExecute( self ):
            """Unit-tests the Execute method of the IsDataConnector interface."""
            testObject = IsDataConnectorDerived()
            try:
                testObject.Execute( None )
                self.fail( 'Execute should raise a NotImplementedError.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Execute should raise a NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__mame__, error ) )

The BaseDataConnector inherits that abstract Execute method, so it's unit-tests need to be updated accordingly:

        def testExecute( self ):
            """Unit-tests the Execute method of the BaseDataConnector abstract class."""
            self.assertEquals( BaseDataConnector.Execute, IsDataConnector.Execute, 'BaseDataConnector should not define the Execute method.' )

...and now Query can be defined. There are some commented-out properties and methods in this code that I've left in place for later, mostly centering around caching of query results in the future.

Query (Nominal final class):

class Query( IsQuery, object ):
    """Represents a query (and it's results) against a back-end data-source."""

    ##################################
    # Class Attributes               #
    ##################################

    ##################################
    # Class Property-Getter Methods  #
    ##################################

#    def _GetCacheExpiresAt( self ):
#        """Gets the time at which the query's cached results will expire."""
#        return self._cacheExpiresAt

    def _GetDatasource( self ):
        """Gets the Datasource used to execute the query's SQL."""
        return self._datasource

    def _GetResults( self ):
        """Gets the results from the execution of the query's SQL against the data-source."""
        if not self._results:
            if self._sql:
                self.__Execute()
        return self._results

    def _GetSql( self ):
        """Gets the SQL to be executed against the data-source to generate the results."""
        return self._sql

    ##################################
    # Class Property-Setter Methods  #
    ##################################

#    def _SetCacheExpiresAt( self, value ):
#        """Sets the time at which the query's cached results will expire."""
#        self.self._cacheExpiresAt = value

    def _SetDatasource( self, value ):
        """Sets the Datasource used to execute the query's SQL."""
        if not isinstance( value, IsDataConnector ):
            raise TypeError( '%s.Datasource expects an instance of IsDataConnector.' % ( self.__class__.__name__ ) )
        if self._results != None:
            raise AttributeError( '%s.Datasource cannot be reset once results have been retrieved.' % ( self.__class__.__name__ ) )
        self._datasource = value

    def _SetResults( self, value ):
        """Sets the results from the execution of the query's SQL against the data-source."""
        self._results = value

    def _SetSql( self, value ):
        """Sets the SQL to be executed against the data-source to generate the results."""
        if type( value ) != types.StringType:
            raise TypeError( '%s.Sql expects a non-empty string value.' % ( self.__class__.__name__ ) )
        if value == '':
            raise ValueError( '%s.Sql expects a non-empty string value.' % ( self.__class__.__name__ ) )
        if self._results != None:
            raise AttributeError( '%s.Sql cannot be reset once results have been retrieved.' % ( self.__class__.__name__ ) )
        self._sql = value

    ##################################
    # Class Property-Deleter Methods #
    ##################################

#    def _DelCacheExpiresAt( self ):
#        """Deletes the time at which the query's cached results will expire - uncaches the results in the process."""
#        self._cacheExpiresAt = 0

    def _DelDatasource( self ):
        """Deletes the Datasource used to execute the query's SQL."""
        self._datasource = None

    def _DelResults( self ):
        """Deletes the results from the execution of the query's SQL against the data-source."""
        self._results = None

    def _DelSql( self ):
        """Deletes the SQL to be executed against the data-source to generate the results."""
        self._sql = None

    ##################################
    # Class Properties               #
    ##################################

#    CacheExpiresAt = property( _GetCacheExpiresAt, _SetCacheExpiresAt, None, IsQuery.CacheExpiresAt.__doc__ )
    Datasource = property( _GetDatasource, _SetDatasource, None, IsQuery.Datasource.__doc__ )
    Results = property( _GetResults, None, None, IsQuery.Results.__doc__ )
    Sql = property( _GetSql, _SetSql, None, IsQuery.Sql.__doc__ )

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'datasource', None, 'The Datasource used to execute the query\'s SQL.' )
    @DocumentArgument( 'argument', 'sql', None, 'The SQL to be executed against the data-source to generate the results.' )
#    @DocumentArgument( 'keyword', 'parameters', 'cacheFor', '(Non-negative integer, optional) The number of seconds that the Query\'s results should be cached for.' )
#    @DocumentArgument( 'keyword', 'parameters', 'cacheUntil', '(Time, optional) A time that the Query\'s results will be cached until.' )
#    def __init__( self, datasource, sql, **parameters ):
    def __init__( self, datasource, sql ):
        """Object constructor."""
        # Nominally final: Don't allow any class other than this one
        if self.__class__ != Query:
            raise NotImplementedError( 'Query is (nominally) a final class, and is not intended to be derived from.' )
        # Set default values:
        self._DelDatasource()
        self._DelResults()
        self._DelSql()
        # Set properties from arguments
        self._SetDatasource( datasource )
        self._SetSql( sql )

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

#    def Cache( self ):
#        """Caches the results of the Query in a manner yet to be determined."""
#        raise NotImplementedError( 'Query.Cache has not been implemented yet.' )

    def __Execute( self ):
        """Executes the query's SQL against it's specified Datasource, retrieves the results, and builds out a list of ResultSet objects in the Query's Results."""
        self._results = TypedList( [ ResultSet ] )
        rawResults = self.Datasource.Execute( self )
        for theResult in rawResults:
            self._results.append( ResultSet( theResult ) )

#    def Uncache( self ):
#        """Uncaches the results of the Query."""
#        raise NotImplementedError( 'Query.Uncache has not been implemented yet.' )

__all__ += [ 'Query' ]

Given my typical patterns, and assuming that you've been following along for any length of time, there's not a whole lot out of the ordinary here. The Datasource and Sql properties are the exception, in that they allow modification until the query's results are retrieved.

Similarly, with the exception of the FixedResultsConnector class, the unit-tests are pretty typical, though again, they are explicitly testing the mutability of the Datasource and Sql properties before and after results are acquired. The Results property tests may feel a bit on the sparse side also, but bear in mind that all we care about at this level is that we get back a TypedList of ResultSet instances - ResultSet is already tested, so that's all we need to know...

    class QueryDerived( Query ):
        def __init__( self, datasource, sql ):
            Query.__init__( self, datasource, sql )
    
    class FixedResultsConnector( BaseDataConnector ):
        def __init__( self, **parameters ):
            BaseDataConnector.__init__( self, **parameters )
        def Execute( self, query ):
            return [
                    # First result-set
                    [
                        {'results1row':1, 'results1field1':'results1row1field1', 'results1field2':'results1row1field2' },
                        {'results1row':2, 'results1field1':'results1row2field1', 'results1field2':'results1row2field2' },
                    ],
                    # Second result-set
                    [
                        {'results2row':1, 'results2field1':'results2row1field1'},
                    ],
                ]
        def ResultsCount( self ):
            return length( self.Execute( self, '' ) )
        def ResultsFields( self ):
            results = self.Execute( self, '' )
            keySets = []
            for theResult in results:
                keysets.append( theResult[0].keys() )
            return keySets
            

    class testQuery( unittest.TestCase ):
        """Unit-tests the Query class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
        def testFinal( self ):
            """Testing final nature of the Query class."""
            testDataSource = FixedResultsConnector()
            testSql = 'SELECT * FROM sometable;'
            try:
                testObject = QueryDerived( testDataSource, testSql )
                self.fail( 'Query is nominally a final class, and should not be instantiable.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Attempting to instantiate an object derived from Query should raise NotImplemetedError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            testObject = Query( testDataSource, testSql )
            self.assertTrue( isinstance( testObject, Query ), 'Creating a Query instance should result in a Query instance.' )
    
        def testConstruction( self ):
            """Testing construction of the Query class."""
            pass
    
        def testPropertyCountAndTests( self ):
            """Testing the properties of the Query class."""
            items = getMemberNames( Query )[0]
            actual = len( items )
            expected = 3
            self.assertEquals( expected, actual, 'Query is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the Query class."""
            items = getMemberNames( Query )[1]
            actual = len( items )
            expected = 0
            self.assertEquals( expected, actual, 'Query is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

        # Test properties

        def testDatasource( self ):
            """Tests the Datasource property of the Query class."""
            testDataSource = FixedResultsConnector()
            testOtherDataSource = FixedResultsConnector()
            testSql = 'SELECT * FROM sometable;'
            testObject = Query( testDataSource, testSql )
            self.assertEquals( testObject.Datasource, testDataSource, 'The Datasource property should be set and gettable after creation.' )
            testObject.Datasource = testOtherDataSource
            self.assertEquals( testObject.Datasource, testOtherDataSource, 'The Datasource property should be settabe and gettable after creation if results haven\'t been retrieved.' )
            testResults = testObject.Results
            try:
                testObject.Datasource = testDataSource
                self.fail( 'A Query\'s Datasource property should not be modifiable after results have been retrieved' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Attempting to change a Query\'s Datasource should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testResults( self ):
            """Tests the Results property of the Query class."""
            testDataSource = FixedResultsConnector()
            testSql = 'SELECT * FROM sometable;'
            testObject = Query( testDataSource, testSql )
            queryResults1, queryResults2 = testObject.Results
            resultSet1, resultSet2 = testDataSource.Execute( '' )
            self.assertEquals( resultSet1[0].keys(), queryResults1.FieldNames )
            self.assertEquals( len( resultSet1 ), len( queryResults1 ) )
            self.assertEquals( resultSet2[0].keys(), queryResults2.FieldNames )
            self.assertEquals( len( resultSet2 ), len( queryResults2 ) )

        def testSql( self ):
            """Tests the Sql property of the Query class."""
            testDataSource = FixedResultsConnector()
            testSql = 'SELECT * FROM sometable;'
            testOtherSql = 'SELECT * FROM someothertable;'
            testObject = Query( testDataSource, testSql )
            self.assertEquals( testObject.Sql, testSql, 'The Sql property should be set and gettable after creation.' )
            testObject.Sql = testOtherSql
            self.assertEquals( testObject.Sql, testOtherSql, 'The Sql property should be settabe and gettable after creation if results haven\'t been retrieved.' )
            testResults = testObject.Results
            try:
                testObject.Sql = testSql
                self.fail( 'A Query\'s Sql property should not be modifiable after results have been retrieved' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Attempting to change a Query\'s Sql should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testQuery ) )

The DataConnectors module (part 5)

2011-12-16T04:30:00.000-08:00

Looking at the requirements (both functional and logical) for ResultSet, most of them are already accounted for simply by making ResultSet derive from TypedList, and hard-setting the member-types to Record. Consider:

A result-set is a sequence of records - Check;
A result-set can be iterated over, and the fields of any individual row or record can be accessed using a syntax along the lines of result[ i ].FieldName, or within an iteration over results, results.FieldName - Check;
The individual rows, once populated, are immutable - Check;

But there are other requirements, more based on the logical expectation(s) of a record or row from a query:

Each row or record of a result-set should have the same structure (e.g., the same field-names as attributes), even if they have a null or None value;
Once populated, the sequence of rows cannot be altered, including adding new rows, removing existing ones, or changing any row-values, either singly or as part of a slice;

These requirements haven't been dealt with yet, since they reside at the ResultSet level of the stack (and they may carry back up to the IsResultSet nominal interface as well). But given the functionality we know of in TypedList, and the object-locking-pattern that was used in Record, the basic mechanisms aren't difficult to implement. Since there will be overrides of many (all?) of the TypedList methods in order to check for lock-state on the record-set object, as well as to check the field-structure of the records, each of these methods has to be tested at the ResultSet level, just as they were for TypedList, though... And if you recall, there was a substantial amount of testing that had to happen there, for all of the various list-behaviors...

ResultSet (Nominal final class):

class ResultSet( IsResultSet, TypedList, object ):
    """Class doc-string"""

    ##################################
    # Class Attributes               #
    ##################################

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    def _GetFieldNames( self ):
        return self.__fieldNames

    def _GetRowCount( self ):
        return len( self )

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    ##################################
    # Class Properties               #
    ##################################

    FieldNames = property( _GetFieldNames, None, None, IsResultSet.FieldNames.__doc__ )
    RowCount = property( _GetRowCount, None, None, IsResultSet.RowCount.__doc__ )

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'results', None, '(iterable collection of dictionaries) The sequence of rows to populate in the result-set.' )
    def __init__( self, results ):
        """Object constructor."""
        # Nominally final: Don't allow any class other than this one
        if self.__class__ != ResultSet:
            raise NotImplementedError( 'ResultSet is (nominally) a final class, and is not intended to be derived from.' )
        TypedList.__init__( self, [ Record ] )
        self.__fieldNames = []
        self.__resultsLocked = False
        for item in results:
            if not isinstance( item, Record ):
                item = Record( **item )
            if self.__fieldNames == []:
                self.__fieldNames = item.FieldNames
            else:
                if self.__fieldNames != item.FieldNames:
                    raise AttributeError( '%s Error: Expected a row with %s fields, but a row with %s fields was present instead' % ( self.__class__.__name__, self.__fieldNames, item.FieldNames ) )
            self.append( item )
        self.__resultsLocked = True

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

    ##################################
    # TypedList Methods overridden   #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'items', None, '(List or derived, required) The list of items to add to the current list.' )
    def __add__( self, items ):
        """Called when adding items to the list with a "+" (e.g., TL + items)."""
        if self.__resultsLocked:
            raise AttributeError( '%s __add__: the %s is locked and cannot be modified' % ( self.__class__.__name__, self.__class__.__name__ ) )
        # check items
        for item in items:
            if item.Keys != self.__fieldNames:
                raise ValueError( '%s __add__: %s is not a Record instance with fields %s' % ( self.__class__.__name__, item, self.__fieldNames ) )
        TypedList.__add__( self, items )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'item', None, '(List or derived, required) The list of item to add to the current list.' )
    def __delitem__( self, item ):
        """Called when deleting an item from the list."""
        if self.__resultsLocked:
            raise AttributeError( '%s __delitem__: the %s is locked and cannot be modified' % ( self.__class__.__name__, self.__class__.__name__ ) )
        TypedList.__delitem__( self, item )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'items', None, '(List or derived, required) The list of items to append to the current list.' )
    def __iadd__( self, items ):
        """Called when adding items to the list with a "+=" (e.g., TL += item)."""
        if self.__resultsLocked:
            raise AttributeError( '%s __iadd__: the %s is locked and cannot be modified' % ( self.__class__.__name__, self.__class__.__name__ ) )
        # check items
        for item in items:
            if item.Keys != self.__fieldNames:
                raise ValueError( '%s __iadd__: %s is not a Record instance with fields %s' % ( self.__class__.__name__, item, self.__fieldNames ) )
        TypedList.__iadd__( self, items )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'index', None, '(Integer, required) The index-location to set the item in.' )
    @DocumentArgument( 'argument', 'item', None, '(Any, required) The item to set in the specified list-location.' )
    def __setitem__( self, index, item ):
        """Called when setting the value of a specified location in the list (TL[n] = item)."""
        if self.__resultsLocked:
            raise AttributeError( '%s __setitem__: the %s is locked and cannot be modified' % ( self.__class__.__name__, self.__class__.__name__ ) )
        if item.FieldNames != self.__fieldNames:
            raise ValueError( '%s __setitem__: %s is not a Record instance with fields %s' % ( self.__class__.__name__, item, self.__fieldNames ) )
        TypedList.__setitem__( self, index, item )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'start', None, '(Integer, required) The starting position of the slice to set the supplied items in.' )
    @DocumentArgument( 'argument', 'end', None, '(Integer, required) The ending position of the slice to set the supplied items in.' )
    @DocumentArgument( 'argument', 'items', None, '(List or derived, required) The list of items to set in the specified slice-location.' )
    def __setslice__( self, start, end, items ):
        """Called when setting the value(s) of a slice-location in the list (TL[1:2] = item)."""
        if self.__resultsLocked:
            raise AttributeError( '%s __setslice__: the %s is locked and cannot be modified' % ( self.__class__.__name__, self.__class__.__name__ ) )
        # check items
        for item in items:
            if item.FieldNames != self.__fieldNames:
                raise ValueError( '%s __setslice__: %s is not a Record instance with fields %s' % ( self.__class__.__name__, item, self.__fieldNames ) )
        TypedList.__setslice__( self, start, end, items )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'item', None, '(Any, required) The item to append.' )
    def append( self, item ):
        """Called when appending items to the list."""
        if self.__resultsLocked:
            raise AttributeError( '%s error: the %s is locked and cannot be modified' % ( self.__class__.__name__, self.__class__.__name__ ) )
        if item.FieldNames != self.__fieldNames:
            raise ValueError( '%s append: %s is not a Record instance with fields %s' % ( self.__class__.__name__, item, self.__fieldNames ) )
        TypedList.append( self, item )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'items', None, '(List or derived, required) The list of items to extend to the current list.' )
    def extend( self, items ):
        """Called when extending items to a the end of the list."""
        if self.__resultsLocked:
            raise AttributeError( '%s error: the %s is locked and cannot be modified' % ( self.__class__.__name__, self.__class__.__name__ ) )
        # check items
        for item in items:
            if item.FieldNames != self.__fieldNames:
                raise ValueError( '%s error: %s is not a Record instance with fields %s' % ( self.__class__.__name__, item, self.__fieldNames ) )
        TypedList.extend( self, items )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'index', None, '(Integer, required) The index-location to insert the item in.' )
    @DocumentArgument( 'argument', 'item', None, '(Any, required) The item to insert in the specified list-location.' )
    def insert( self, index, item ):
        """Called when inserting items to a specified location in the list (TL[1] = item)."""
        if self.__resultsLocked:
            raise AttributeError( '%s error: the %s is locked and cannot be modified' % ( self.__class__.__name__, self.__class__.__name__ ) )
        if item.FieldNames != self.__fieldNames:
            raise ValueError( '%s append: %s is not a Record instance with fields %s' % ( self.__class__.__name__, item, self.__fieldNames ) )
        TypedList.insert( self, index, item )

__all__ += [ 'ResultSet' ]

Line(s)

12-16, 30-31

Standard read-only property set-up...

37-56

Object constructor:

42-43

Enforcing a nominally-final class for the object.

44

Making sure that TypedList functionality is available for instances.

45-46

Setting instance-values for the two new attributes of the class.

47-55

Populating the instance with the results provided:

48-49: The class should, ideally, be able to handle results as either an iterable of Record instances, or an iterable of dictionary instances - if the item is a dictionary, it will need to be made into a Record instance.; I feel like I've neglected type-checking the item at this point, making sure that it's a dictionary, but I don't know that it would really make a difference. Instantiation of a Record with anything other than a dictionary or a keyword argument list raises an error on it's own, so I don't think explicitly checking makes enough of a difference to implement that. At least not yet.
50-54: This is where the ResultSet's field-names are set, on the first pass through the loop, and where the structural consistency of subsequent records is checked.
55: If no errors were raised by the time that we reach this point, then the item is appended to the underlying list.
56: Locks the object, preventing futher modification.

70-158

The various built-ins and methods here all follow more or less the same pattern:

Check to see if the object is locked, and raise AttributeError if it is;
Check for structural consistency with the established field-names for the item or items supplied;
If no errors are raised, call the parent method from TypedList to perform the actual operation.

Ultimately, they're all just checking to see if the operation is allowed before performing it - nothing too fancy, though some cases may look a bit more convoluted than is perhaps typical.

82-88

Since the records in a ResultSet should be immutable, deletion has to be prevented as well.

This is something that I overlooked in Record, so I'll need to get that in place there as well.

Unit-tests

class ResultSetDerived( ResultSet ):
        def __init__( self, results ):
            ResultSet.__init__( self, results )
    
    class testResultSet( unittest.TestCase ):
        """Unit-tests the ResultSet class."""

        def getBadRecord( self ):
            return { 'Number1':3, 'Name':'ook eek', 'IsValid':True }

        def getGoodRecord( self ):
            return { 'Number':3, 'Name':'ook eek', 'IsValid':True }

        def getBadValues( self ):
            return [ 1, 'ook', True, None, {} ]

        def getGoodValues( self ):
            return [
                { 'Number':1, 'Name':'ook', 'IsValid':True },
                { 'Number':2, 'Name':'eek', 'IsValid':False },
                { 'Number':None, 'Name':None, 'IsValid':None },
                ]

        def getInconsistentValues( self ):
            return [
                { 'Number':1, 'Name':'ook', 'IsValid':True },
                { 'Number':1, 'IsValid':True },
                { 'Number':2, 'Name':'eek' },
                { 'Name':'eek', 'IsValid':False },
                { 'Number':None },
                { 'IsValid':None },
                ]

        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
        def testFinal( self ):
            """Testing final nature of the ResultSet class."""
            try:
                testObject = ResultSetDerived( self.getGoodValues() )
                self.fail( 'ResultSet is nominally a final class, and should not be extendable.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Instantiating an object derived from ResultSet should raise NotImplementedError, by %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testConstruction( self ):
            """Testing construction of the ResultSet class."""
            testObject = ResultSet( self.getGoodValues() )
            # TODO: Test values...
            # Test inconsistent values - all should raise errors
            inconsistentValues = self.getInconsistentValues()
            # Test creation with an iterable
            try:
                testObject = ResultSet( inconsistentValues )
                self.fail( 'Construction with inconsistent record-structures should raise an error.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Construction with inconsistent record-structures should raise AttributeError, but %s was raised instead:\n  %s.' % ( error.__class__.__name__, error ) )

        def testPropertyCountAndTests( self ):
            """Testing the properties of the ResultSet class."""
            items = getMemberNames( ResultSet )[0]
            actual = len( items )
            expected = 3
            self.assertEquals( expected, actual, 'ResultSet is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the ResultSet class."""
            items = getMemberNames( ResultSet )[1]
            actual = len( items )
            expected = 4
            self.assertEquals( expected, actual, 'ResultSet is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

        # Test list behavior
        def testListBehavior( self ):
            """Unit-tests list-like behavior of ResultSet."""
            # Test good values - all should raise errors
            goodValues = self.getGoodValues()
            testObject = ResultSet( goodValues )
            validRecord = Record( **self.getGoodRecord() )
            # Test L = L + i structure
            try:
                testObject + [ validRecord ]
                self.fail( 'Once a ResultSet has been populated, it should be immutable.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Attempting to modify a ResultSet once it\'s been populated should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            # Test L = L += i structure
            try:
                testObject += [ validRecord ]
                self.fail( 'Once a ResultSet has been populated, it should be immutable.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Attempting to modify a ResultSet once it\'s been populated should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            # Test L.append equivalent
            try:
                testObject.append( validRecord )
                self.fail( 'Once a ResultSet has been populated, it should be immutable.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Attempting to modify a ResultSet once it\'s been populated should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            # Test L.extend equivalent
            try:
                testObject.extend( [ validRecord ] )
                self.fail( 'Once a ResultSet has been populated, it should be immutable.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Attempting to modify a ResultSet once it\'s been populated should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            # Test L.insert equivalent
            try:
                testObject.insert( 0, validRecord )
                self.fail( 'Once a ResultSet has been populated, it should be immutable.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Attempting to modify a ResultSet once it\'s been populated should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            # Test L[index] = value equivalent
            try:
                testObject[ 0 ] = validRecord
                self.fail( 'Once a ResultSet has been populated, it should be immutable.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Attempting to modify a ResultSet once it\'s been populated should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            # Test L[slice] = [ i, j ] equivalent
            try:
                testObject[ 0:1 ] = validRecord
                self.fail( 'Once a ResultSet has been populated, it should be immutable.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Attempting to modify a ResultSet once it\'s been populated should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            # Test del L[i]
            try:
                del testObject[ 0 ]
                self.fail( 'Once a ResultSet has been populated, it should be immutable.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Attempting to modify a ResultSet once it\'s been populated should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        # Test Properties

        def testMemberTypes( self ):
            """Unit-tests the MemberTypes property of the ResultSet class."""
            self.assertEquals( TypedList.MemberTypes, BaseTypedCollection.MemberTypes, 'TypedList should inherit the MemberTypes property of BaseTypedCollection.' )

        def testFieldNames( self ):
            """Unit-tests the FieldNames property of the ResultSet class."""
            goodValues = self.getGoodValues()
            testObject = ResultSet( goodValues )
            expected = Record( **goodValues[ 0 ] ).FieldNames
            self.assertEquals( testObject.FieldNames, expected, 'The field-names of a ResultSet should be the same as the field-names of it\'s Record-instances.' )

        def testRowCount( self ):
            """Unit-tests the RowCount property of the ResultSet class."""
            goodValues = self.getGoodValues()
            testObject = ResultSet( goodValues )
            self.assertEquals( testObject.RowCount, len(goodValues), 'The number of rows in RowCount should be the same as the number of items presented at construction.' )

        # Test Methods

        def testappend( self ):
            """Unit-tests the append method of the ResultSet class."""
            pass # This is tested in testListBehavior, above.

        def testextend( self ):
            """Unit-tests the extend method of the ResultSet class."""
            pass

        def testinsert( self ):
            """Unit-tests the insert method of the ResultSet class."""
            pass # This is tested in testListBehavior, above.

        def testIsValidMemberType( self ):
            """Unit-tests the IsValidMemberType method of the ResultSet class."""
            self.assertEquals( TypedList.IsValidMemberType, BaseTypedCollection.IsValidMemberType, 'TypedList should inherit the IsValidMemberType method of BaseTypedCollection.' )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testResultSet ) )

I'm going to skip commenting on the unit-tests this time around - they are all either very straightforward (I believe), or repeats of previous test-structures. The only new thing here, really, is the use of a couple of methods (getBadRecord, getGoodRecord, getGoodValues, and getInconsistentValues) used to generate results for creation of RecordSet instances for the various test-methods.

The unit-tests for RecordSet also revealed a flaw in Record: I'd forgotten to initialize a local __fieldNames attribute, and as a result, the value being set in that property for the first instance was shared across all instances (it was behaving as a static attribute). The changes to Record and it's unit-tests are:

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'keyword', 'properties', 'variable', 'Key/value pairs providing the record\'s field-names and values of those fields.' )
    def __init__( self, **properties ):
        """Object constructor."""
        # Nominally final: Don't allow any class other than this one
        if self.__class__ != Record:
            raise NotImplementedError( 'Record is (nominally) a final class, and is not intended to be derived from.' )
        self.__fieldNames = []
        for theKey in properties:
            self.__setattr__( theKey, properties[ theKey ] )
            if not theKey in self.__fieldNames:
                self.__fieldNames.append( theKey )
        self.__recordIsLocked = True

# break

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'name', None, '(String, required) The name of the attribute to be set.' )
    @DocumentArgument( 'argument', 'value', None, '(Any type, required) The value to be set in the named attribute.' )
    def __setattr__( self, name, value ):
        """Object-attribute setter."""
        if name == '_Record__recordIsLocked' and self.__recordIsLocked != False:
            raise AttributeError( 'The record is locked, and cannot be unlocked' )
        if self.__recordIsLocked:
            raise AttributeError( 'The record has been locked, and the "%s" attribute cannot be set.' % ( name ) )
        if hasattr( self, name ) and not name in [ '_Record__recordIsLocked', '_Record__fieldNames' ]:
            raise AttributeError( 'The "%s" attribute already exists, and cannot be reset.' % ( name ) )
        return object.__setattr__( self, name, value )

The changes here are:

Line(s)
8: self.__fieldNames is being explicitly set to an empty list
26: Explicitly checking the property-name against both _Record__recordIsLocked and _Record__fieldNames, in order to allow either to be set if the object isn't locked yet.; This feels like it may not be complete, so I may very well come back to it later...

        def testFieldNames( self ):
            """Unit-tests the FieldNames property of the Record class."""
            testProperties = { 'Property1':1, 'Property2':'ook', 'Property3':True }
            testProperties2 = { 'Property':1 }
            testObject = Record( **testProperties )
            self.assertEquals( testObject.FieldNames, testProperties.keys() )
            try:
                testObject.Keys = 'ook'
                self.fail( 'A Record\'s FieldNames cannot be altered.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Changes to a record\'s FieldNames should raise AttributeError, bu %s was raised instead:\n  %s ' % ( error.__class__.__name__, error ) )
            testObject = Record( **testProperties2 )
            self.assertEquals( testObject.FieldNames, testProperties2.keys(), 'The field-names for each record should be local to the instance' )

The change here is the addition of the last two lines - since each Record instance should have set it's FieldNames property independently, the second testObject should have the same keys as the testProperties2 dictionary. If the attribute were not explicitly set locally, it would have the keys from testProperties instead.

And that's the last of the "simple" concrete classes that weren't waiting on research. At least until the need for another one rears it's head. Next (finally), I'll get back to defining where query-execution takes place - you may remember that I set that aside to do some research in my previous post. I've done that research, and I can move on to implementation now...

The DataConnectors module (part 4)

2011-12-14T04:30:00.000-08:00

So, before I start digging into the code for the concrete classes of DataConnectors, I want to do a quick review of how they are all supposed to fit together:

An instance of BaseDataConnector (which implements IsDataConnector) will provide the basic connectivity to the back-end data-store, as well as the mechanism(s) for executing queries against that data-store;
Queries are objects implementing IsQuery, that store the SQL to be executed, as well as the results of that execution once it's run in one or more ResultSet objects;
The ResultSet instances (which implement IsResultSet, and are TypedLists keep track of the fields returned, row-by-row.

There's a piece missing from the most recent class-diagram: how the records are going to be represented... We need to add a Record class to handle that, like so:

Ideally, what I'd prefer from a code-structure standpoint, is to be able to iterate over a ResultSet instance, and use object-/dot-notation for the fields. For example, something like:

for record in results:
    print record.FirstName, record.LastName

Providing that capability is easy, since Python's objects can have dynamically-assigned attributes and values. The downside to taking that very basic approach, though is that those records, and the fields within them, are mutable in the code, and for a database-record that's read-only, that's not a good thing. There is a way to work around that, though, and I'll demonstrate that a bit later.

Another thing that hasn't really been thrashed out yet is when and where query-execution occurs. I've been contemplating two basic approaches, and trying to weight between their merits and drawbacks:

Execution happens at the Query:

In this model, the Query uses it's Datasource to connect to the back-end database, executes the SQL, and generates the result-set.

Advantages

It feels to me like query-objects preserve the single responsibility principle better this way - really, at least in theory, all they need is a connection-object, supplied by the data-connector, and away they go.
I suspect that this will allow for easier implementation of "one-off" queries - I don't know how many times in my career a project has crossed my desk where the work needed essentially boiled down to "run another query, and add the results to the display." I believe that optimizing the data-connector approach for this sort of one-off case wouldn't be difficult, but I'm not completely sure about that at this point.
It feels like this approach would lend itself well (or at least better) to lazy instantiation of query-results: that is, create as many query-objects as even might be necessary, but don't actually get the results from the database until (or even unless) they are specifically requested. At least for basic fetch-and-display processes, this could be very advantageous.

Drawbacks

There is a strong potential, when (not if) this module is expanded to include back-end databases other than MySQL, that query-objects may need to be customized to each individual database-engine. I don't have a strong feeling at this point for just how likely this really is, but since the logical point of encapsulation for any database-engine-specific capabilities is at the data-connector level, not at the query level, this feels like it would be less than optimal the first time that this kind of database-engine variation happens. It's not future-proof, and could easily require a lot of new work or rework, and I feel that's a major concern.
If the assumption that one-off queries would be easier to implement in this model holds true, that could tend to reinforce that sort of design/solution in applications. On the surface, this doesn't sound all that bad, but if each one-off query represents some (hopefully small) amount of database-connection time, the more this sort of design is propagated, the more performance could suffer as a result. I'm not a big fan of making things easier in the shorter term at the expense of longer-term stability or maintainability.
It doesn't feel like this approach would facilitate building up a list of queries, then executing all of them at once - at least not as easily.

Execution happens at the BaseDataConnector:

In this model, the connector (a derivative of BaseDataConnector) connects to the datasource, then runs the query or queries specified, handing the results back to the Query object(s) that supplied the SQL in the first place so that they can keep track of them.

Advantages

The major advantage is almost certainly that keeping the actual execution of the query encapsulated in the data-connector (which is the place where it might vary) will protect us from the potential need to have to spin off a Query-variant for each and every data-connector-type that we might eventually want. As I noted above, I don't have a strong feeling for how likely this really is, so more research on that topic is probably in order. It should be noted, however, that even when that research is complete, and even if it proves that the connector-types that we're planning aren't going to be affected, that does not mean that they won't change, or that some new, hot-topic database-engine won't come out that would require distinct Query classes...
A major advantage to this approach is, I think, that queries could be queued up and deferred until the developer knows that the code will need them.

Drawbacks

There might be significant design changes needed for BaseDataObject-derived objects in order to take advantage of query-queueing capabilities. Or it may not even be feasible to reconcile those objects' data-needs with a queueable query-stack, in which case a mixed model might be necessary (which is more up-front work).

So, it feels like more research is needed, and possibly some rework on the designs of BaseDataConnector, and maybe some minor changes to Query will be needed as well. In the interests of moving this post along (while I do my research in the background), I'm not going to work on Query just yet. That class, and any changes to BaseDataConnector will be the topic of the next post, and I'll attack the two remaining general-purpose concrete classes left: Record and ResultsSet.

Instances of the Record class represent single records/rows returned from a query executed against the database. They are the core member-type for a ResultSet instance (which in turn represent the collections of records from a query). Record-objects should (for the time being) be immutable once instantiated: we don't want to allow changes to a record's data inside an application without having some way to re-synchronize that data with the original record, which is why the BaseDataObject abstract class exists. For similar reasons, ResultSet instances should also be immutable once instantiated. Both are variations of a "dumb" data-object concept (though ResultSet is not completely dumb, since it has all of the functionality of a TypedList): objects whose sole purpose is to represent some data-structure.

Since I'm expecting a Record instance to be generated from a dictionary data-structure, where the dictionary's keys are the column-names and the values are the field-values, the first challenge to overcome is to find a way to pass a dictionary structure into an object at creation time, without having to go through the typical property2=value2, property2=value2 structure that we might expect given the code I've shown so far. These names/values will vary from one record-structure to another, and having to code that structure into a record's constructor would mean having to generate a Record-derived class for every data-type that needs to be represented. That, obviously, is not sustainable. Fortunately, Python provides a mechanism to pass a dictionary as a keyword-argument list. We've already seen the recieving-end structure (see BaseDataConnector.__init__ for an example), though I don't see that I've provided an example of the "sending" structure yet. To illustrate that, here's a quick little script:

class DictProperties( object ):
    """A test-class, created to see how dictionary arguments can be 
pushed into an object's properties."""
    def __init__( self, **properties ):
        """Object constructor."""
        for theKey in properties:
            self.__setattr__( theKey, properties[ theKey ] )

properties = {
    'Prop1':1,
    'Prop2':'ook',
    'Prop3':True,
    }

testObject = DictProperties( **properties )

print 'testObject.Prop1 ... %s' % ( testObject.Prop1 )
print 'testObject.Prop2 ... %s' % ( testObject.Prop2 )
print 'testObject.Prop3 ... %s' % ( testObject.Prop3 )

When this script is run, it prints out:

testObject.Prop1 ... 1
testObject.Prop2 ... ook
testObject.Prop3 ... True

This is what we want: The elements of the dictionary (properties) are being set as "real" attributes in the object-instance. But how does it work? The secret is in lines 4, 6-7, and 15:

Line(s)
4: The **properties argument is a keyword-argument list. Internally, that is a dictionary that can be iterated over in line 6.
6-7: As the properties keyword/dictionary is being iterated over (line 6), we take advantage of a Python built-in: __setattr__. When present, __setattr__ is called when an attribute assignment is attempted, instead of simply storing the value in the object's instance dictionary. Remember this, because it's going to be a key part of the immutability of a Record later on...
15: This is the part that allows an arbitrary dictionary to be passed into the object's constructor (it would work with any other method as well). Instead of supplying Prop1=1, Prop2='ook', Prop3=True as arguments to the constructor, which would tie each instantiation to a known structure, we pass what is essentially a keyword-argument-reference to a dictionary (**properties) instead. The constructor recognizes it as a keyword-arguments structure (which it's expecting), and handles it just like it would if each keyword/value were supplied directly.

So an arbitrary data-structure can be passed to an object during creation. That's the first hurdle passed.

The next hurdle is to make those values immutable after they've been set. Getting that to work takes a bit of modification that's easier to explain by showing the code, so here's a modified copy of the same script above:

class DictProperties( object ):
    """A test-class, created to see how dictionary arguments can be 
pushed into an object's properties."""
    # Reserved attributes
    __objectIsLocked = False
    # Object constructor
    def __init__( self, **properties ):
        """Object constructor."""
        for theKey in properties:
            self.__setattr__( theKey, properties[ theKey ] )
        self.__objectIsLocked = True
    def __setattr__( self, name, value ):
        """Object-attribute setter."""
        if name == '__objectIsLocked' and self.__objectIsLocked:
            raise AttributeError( 'The object is locked, and cannot be unlocked' )
        if self.__objectIsLocked:
            raise AttributeError( 'The object has been locked, and the "%s" attribute cannot be set.' % ( name ) )
        return object.__setattr__( self, name, value )

properties = {
    'Prop1':1,
    'Prop2':'ook',
    'Prop3':True,
    }

testObject = DictProperties( **properties )

print 'testObject.Prop1 ... %s' % ( testObject.Prop1 )
print 'testObject.Prop2 ... %s' % ( testObject.Prop2 )
print 'testObject.Prop3 ... %s' % ( testObject.Prop3 )

try:
    testObject.Prop3 = False
except AttributeError:
    print 'testObject.Prop3 ... Reset failed as expected'
try:
    testObject.Ook = False
except AttributeError:
    print 'testObject.Ook ..... New-attribute set failed as expected'

When this is run, the results are:

testObject.Prop1 ... 1
testObject.Prop2 ... ook
testObject.Prop3 ... True
testObject.Prop3 ... Reset failed as expected
testObject.Ook ..... New-attribute set failed as expected

The magic that makes this work is in the overridce of __setattr__ (lines 12-18). Since __setattr__ is being overridden, the local method gets called instead of the built-in that exists in object. It checks to see if the private __objectIsLocked attribute is being set, and if it's already been set to True (which would lock the object's attributes, as we'll see shortly), and raises an error if this is the case. If the attribute being set is not __objectIsLocked, control passes to the next if on line 16, where a general check of __objectIsLocked is performed. If __objectIsLocked is True, then the object's attributes are locked, and an error is raised, otherwise control passes on, and the results of the base object's __setattr__, with the same arguments, is returned.

That's pretty much all we need, really, for Record:

Record (Nominal final class):

class Record( object ):
    """Represents an immutable record as retrieved from a query against a database."""

    ##################################
    # Class Attributes               #
    ##################################

    __fieldNames = []
    __recordIsLocked = False

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    def _GetFieldNames( self ):
        """Gets the field-names of the Record object created at instantiation."""
        return self.__fieldNames

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    ##################################
    # Class Properties               #
    ##################################

    FieldNames = property( _GetFieldNames, None, None, 'Gets the field-names of the Record object created at instantiation.' )

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'keyword', 'properties', 'variable', 'Key/value pairs providing the record\'s field-names and values of those fields.' )
    def __init__( self, **properties ):
        """Object constructor."""
        # Nominally final: Don't allow any class other than this one
        if self.__class__ != Record:
            raise NotImplementedError( 'Record is (nominally) a final class, and is not intended to be derived from.' )
        for theKey in properties:
            self.__setattr__( theKey, properties[ theKey ] )
            if not theKey in self.__fieldNames:
                self.__fieldNames.append( theKey )
        self.__recordIsLocked = True

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'name', None, '(String, required) The name of the attribute to be set.' )
    @DocumentArgument( 'argument', 'value', None, '(Any type, required) The value to be set in the named attribute.' )
    def __setattr__( self, name, value ):
        """Object-attribute setter."""
        if name == '_Record__recordIsLocked' and self.__recordIsLocked != False:
            raise AttributeError( 'The record is locked, and cannot be unlocked' )
        if self.__recordIsLocked:
            raise AttributeError( 'The record has been locked, and the "%s" attribute cannot be set.' % ( name ) )
        if hasattr( self, name ) and name != '_%s__recordIsLocked' % ( self.__class__.__name__ ):
            raise AttributeError( 'The "%s" attribute already exists, and cannot be reset.' % ( name ) )
        return object.__setattr__( self, name, value )

__all__ += [ 'Record' ]

Line(s)

8-9

Since Record is nominally a final class, and cannot be extended, the __keys and __recordIsLocked can be safely made private. They are also defined as class-level attributes so that they will already exist when __setattr__ is defined, helping to protect those attributes from modification if a Record object is created with "__keys" or "__recordIsLocked" fields.

15-17, 31

A typical, though read-only property for the record, providing the field-names of the record.

37-48

The object constructor:

42-43: My standard check to assure that classes extending Record, so long as they call Record.__init__, will raise an error.
44-47: Reads in the keyword arguments, and calls __setattr__ to set the supplied property names and values in the object.; One caveat about this format is that construction of a Record must use a syntax like aRecord = Record( **fields ). If a call like aRecord = Record( fields ) is attempted, a TypeError will be raised, even if the argument is a dictionary.
48: Locks the record, preventing addition or modification of properties in the object.

58-69

Overrides the default __setattr__ provided by the base object to prevent modification of the object's state-data once the object's been locked:

63-64

Allows the record to be locked, but not unlocked, by specifically checking for the private attribute __recordIsLocked, and allowing it to be changed only if it's False.

This could be changed, if extension of Record was needed, by changing the check to look something like this:

if name == '_%s__recordIsLocked' % ( self.__class__.__name__ ) and self.__recordIsLocked != False:
    # do the rest

65-66

Checks to see whether the object is locked, regardless of the attribute-modification being attempted, and raises an error if it is.

67-68

Checks specifically for attempts to re-set an existing attribute. I don't think this would ever come into play, since trying to duplicate key-names in either a dictionary or a keyword-argument list will raise errors, but since I can't categorically rule the possibility out, I checked for it anyway. I cannot think of a way to test it, though...

Unit-tests

    class testRecord( unittest.TestCase ):
        """Unit-tests the Record class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
        def testFinal( self ):
            """Testing final nature of the Record class."""
            try:
                testObject = RecordDerived()
                self.fail( 'Record is nominally a final class, and should not be extendable.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Instantiating an object derived from Record should raise NotImplementedError, by %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
    
        def testConstruction( self ):
            """Testing construction of the Record class."""
            testObject = Record()
            self.assertTrue( isinstance( testObject, Record ), 'Instances of Record should be instances of Record... Duh!' )
            testProperties = { 'Property1':1, 'Property2':'ook', 'Property3':True }
            testObject = Record( **testProperties )
            for property in testProperties:
                testValue = eval( 'testObject.%s' % property )
                self.assertEquals( testValue, testProperties[ property ], '' )
    
        def testPropertyCountAndTests( self ):
            """Testing the properties of the Record class."""
            items = getMemberNames( Record )[0]
            actual = len( items )
            expected = 1
            self.assertEquals( expected, actual, 'Record is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the Record class."""
            items = getMemberNames( Record )[1]
            actual = len( items )
            expected = 0
            self.assertEquals( expected, actual, 'Record is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

        # Test properties

        def testFieldNames( self ):
            """Unit-tests the FieldNames property of the Record class."""
            testProperties = { 'Property1':1, 'Property2':'ook', 'Property3':True }
            testObject = Record( **testProperties )
            self.assertEquals( testObject.FieldNames, testProperties.keys() )
            try:
                testObject.Keys = 'ook'
                self.fail( 'A Record\'s FieldNames cannot be altered.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Changes to a record\'s FieldNames should raise AttributeError, bu %s was raised instead:\n  %s ' % ( error.__class__.__name__, error ) )
                

        def testImmutability( self ):
            """Unit-tests the immutability of records."""
            testProperties = { 'Property1':1, 'Property2':'ook', 'Property3':True }
            testObject = Record( **testProperties )
            try:
                testObject.NewProperty = 'ook'
                self.fail( 'Once created, a record should not allow the creation of a new attribute/field.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Changes to a record\'s data should raise AttributeError, bu %s was raised instead:\n  %s ' % ( error.__class__.__name__, error ) )
            try:
                testObject.Property1 = 'ook'
                self.fail( 'Once created, a record should not allow the alteration of an existing attribute/field.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Changes to a record\'s data should raise AttributeError, bu %s was raised instead:\n  %s ' % ( error.__class__.__name__, error ) )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testRecord ) )

Most of the unit-tests are pretty typical, so there's not a lot to comment on:

Line(s)

25

Note the object-construction argument: **testPropertiesm not testProperties. THis is an example of the syntax noted above for creation of a Record.

64-81

Tests the immutability of a Record:

68-74: Testing that creation of a new attribute on a Record instance isn't allowed.
75-81: Testing that alteration of an existing attribute is not allowed.

Though this is only one of the two concrete classes I said I was going to hit, this post is pretty long, so I'll pick up next time with ResultSet.

Disruption to my publication-cycle...

2011-12-12T04:30:00.000-08:00

As I was finishing up the previous post, and getting ready to start the next one (the next part of the DataConnectors module), I was also preparing for a major job-change, leaving one company for another, for the first time in nearly seven years, and only the second time in the last 22.

As a result of this change, my normal routine for writing these posts may be disrupted for a while, and I'm not sure how (or if) it will recover. I'll still be making an attempt to keep at least a few items in the blog's scheduled backlog, but I can't guarantee that I'll be able to swing that just yet.

I've also reached a major design-decision point in DataConnectors, and I'm having to thrash through a lot of alternatives about how I want to implement things. In at least one major case, I'm also having to give a lot of thought to why a given implementation should be chosen.

So, if you're following my writing/coding, please bear with any disruptions to my publication schedule for at least a little while. I'm hoping that I'll be able to either resume my normal schedule, or come up with a new one over the next couple of weeks, but I simply don't know what that'll look like yet, and until I get through the current design contemplation, even if I have the time, I may not be able to produce much for a while.

Awkward timing, I guess... Such is the stuff that life in the glamorous world of software design and development is made of.

On the plus-side, though, by the time this post is published, I'll be just a two weeks away from a week-long break for the Christmas/New Year's holiday-week, and I definitely should be able to get more backlog going by then.

Stay tuned!

The DataConnectors module (part 3)

2011-12-09T04:30:00.000-08:00

So, now that the interface-definitions are out of the way, I'm going to tackle the nominal abstract classes. There are only two, BaseDataConnector that will provide structure for concrete data-connector classes like MySQLConnector (and others down the line, potentially), and BaseDataObject, which will provide CRUD capabilities for objects whose state is going to be persisted in a back-end data-store.

BaseDataConnector provides standard connection parameters (host, database, user and password) as properties for connector objects. It does not define the connection property, or the mechanism used to connect to the database, however, since those will vary from database-engine to database-engine. A typical usage is expected to look something like the code below, assuming that Connector is a class derived from BaseDataConnector:

connection = Connector( 
    host='host', 
    database='database', 
    user='user', 
    password='password'
    )
# Optionally, the "Connect" method can be called before executing a query:
# connection.Connect()
results = connection.Query( 'some SQL code' ).Results

This looks pretty simple, though there's a few things I still need to think out (specifically, what I want the items in the results to look like). But even at this point, there's a lot going on "under the hood," as it were, and some possibilities that I'm contemplating for the future:

The creation of the Connector handles all of the connection-information for the back-end datasource - that much we pretty much have to assume.
Whether the code written against the Connector explicitly calls Connect or not, there has to be an active connection before queries can be run, so the first call to a query's Results property would perhaps have to tell the connector-object that it's ready to query, and to create the connection.
- I have to admit that I like the idea of lazily-instantiating queries, results, etc. - Provided that some care is taken to keep things logical, I think that using a lazy instantiation would potentially save some database-connection time.
- I also contemplated the idea of having one or more queues of queries associated with the connector-object: One for queries to be executed at object destruction, and one to allow queries to be generated and stored before they were actually needed, which would allow the first "real" need for a query's results to run all of the pending queries in one connection-batch, maybe. I'm not sure how much (if any) difference this would actually make, and it's potentially more complicated than I want to deal with right this minute, though, so I'm filing that away for future reference/use.
The results returned could/should be allowed to support multiple assignments (e.g., results1, results2 = connection.Query( 'some SQL code' ).Results) as well as single results. I'm not quite sure how I'm going to handle that just yet...

At the nominal-abstract-class level, there's not a lot that can realistically be implemented - mostly standard properties and the like - so the code is pretty short:

BaseDataConnector (Nominal abstract class):

class BaseDataConnector( IsDataConnector, IsConfigurable, object ):
    """Provides baseline functionality, interface requirements and type-identity for objects that can represent a connection to a back-end database."""

    ##################################
    # Class Attributes               #
    ##################################

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetDatabase( self ):
        """Gets the name of the database that the connection will be made to."""
        return self._database

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetHost( self ):
        """Gets the name or IP-address of the host that the database resides upon."""
        return self._host

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetPassword( self ):
        """Gets the password used to connect to the database."""
        return self._password

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetUser( self ):
        """Gets the user-name used to connect to the database."""
        return self._user

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, '(String, required) The name of the database that the connection will be made to.' )
    def _SetDatabase( self, value ):
        """Sets the name of the database that the connection will be made to."""
        if type( value ) != types.StringType:
            raise TypeError( '%s.Database expects a single-line, non-empty string value that contains to tabs.' % ( self.__class__.__name__ ) )
        if value == '' or '\n' in value or '\r' in value or '\t' in value:
            raise ValueError( '%s.Database expects a single-line, non-empty string value that contains to tabs.' % ( self.__class__.__name__ ) )
        self._database = value

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, '(String, required) The name of the Host that the connection will be made to.' )
    def _SetHost( self, value ):
        """Sets the name of the Host that the connection will be made to."""
        if type( value ) != types.StringType:
            raise TypeError( '%s.Host expects a single-line, non-empty string value that contains to tabs.' % ( self.__class__.__name__ ) )
        if value == '' or '\n' in value or '\r' in value or '\t' in value:
            raise ValueError( '%s.Host expects a single-line, non-empty string value that contains to tabs.' % ( self.__class__.__name__ ) )
        self._host = value

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, '(String, required) The password that the connection will be made with.' )
    def _SetPassword( self, value ):
        """Sets the password that the connection will be made with."""
        if type( value ) != types.StringType:
            raise TypeError( '%s.Password expects a single-line, non-empty string value that contains to tabs.' % ( self.__class__.__name__ ) )
        if value == '' or '\n' in value or '\r' in value or '\t' in value:
            raise ValueError( '%s.Password expects a single-line, non-empty string value that contains to tabs.' % ( self.__class__.__name__ ) )
        self._password = value

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, '(String, required) The name of the User that the connection will be made with.' )
    def _SetUser( self, value ):
        """Sets the name of the User that the connection will be made with."""
        if type( value ) != types.StringType:
            raise TypeError( '%s.User expects a single-line, non-empty string value that contains to tabs.' % ( self.__class__.__name__ ) )
        if value == '' or '\n' in value or '\r' in value or '\t' in value:
            raise ValueError( '%s.User expects a single-line, non-empty string value that contains to tabs.' % ( self.__class__.__name__ ) )
        self._user = value

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _DelDatabase( self ):
        """Deletes the name of the database that the connection will be made to."""
        self._database = None

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _DelHost( self ):
        """Deletes the name or IP-address of the host that the database resides upon."""
        self._host = None

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _DelPassword( self ):
        """Deletes the password used to connect to the database."""
        self._password = None

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _DelUser( self ):
        """Deletes the user-name used to connect to the database."""
        self._user = None

    ##################################
    # Class Properties               #
    ##################################

    Database = property( _GetDatabase, _SetDatabase, _DelDatabase, 'Gets, sets or deletes the name of the database that the connection will be made to.' )
    Host = property( _GetHost, _SetHost, _DelHost, 'Gets, sets or deletes the name or IP-address of the host that the database resides upon.' )
    Password = property( _GetPassword, _SetPassword, _DelPassword, 'Gets, sets or deletes the password used to connect to the database.' )
    User = property( _GetUser, _SetUser, _DelUser, 'Gets, sets or deletes the user-name used to connect to the database.' )

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'keyword', 'parameters', 'host', 'The name or IP-address of the host that the database resides upon.' )
    @DocumentArgument( 'keyword', 'parameters', 'database', 'The name of the database that the connection will be made to.' )
    @DocumentArgument( 'keyword', 'parameters', 'user', 'The user-name used to connect to the database.' )
    @DocumentArgument( 'keyword', 'parameters', 'password', 'The password used to connect to the database.' )
    def __init__( self, **parameters ):
        """Object constructor."""
        # Nominally abstract: Don't allow instantiation of the class
        if self.__class__ == BaseDataConnector:
            raise NotImplementedError( 'BaseDataConnector is (nominally) an abstract class, and is not intended to be instantiated.' )
        self._DelDatabase()
        if parameters.has_key( 'database' ):
            self._SetDatabase( parameters[ 'database' ] )
        self._DelHost()
        if parameters.has_key( 'host' ):
            self._SetHost( parameters[ 'host' ] )
        self._DelPassword()
        if parameters.has_key( 'password' ):
            self._SetPassword( parameters[ 'password' ] )
        self._DelUser()
        if parameters.has_key( 'user' ):
            self._SetUser( parameters[ 'user' ] )

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

    ##################################
    # IsConfigurable Methods         #
    ##################################

    @ToDo( 'Figure out how to approach unit-testing of this method.' )
    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'configProvider', None, 'A configuration provider object, an instance of one of IsConfigurable.ConfigurationTypes' )
    @DocumentArgument( 'argument', 'configSection', None, 'String, required) The configuration-section name that the object\'s configuration-state should be retrieved from.' )
    @DocumentConfiguration( 'Section Name', 'host', 'The name or IP-address of the host that the database resides upon.' )
    @DocumentConfiguration( 'Section Name', 'database', 'The name of the database that the connection will be made to.' )
    @DocumentConfiguration( 'Section Name', 'user', 'The user-name used to connect to the database.' )
    @DocumentConfiguration( 'Section Name', 'password', 'The password used to connect to the database.' )
    def Configure( self, configProvider, configSection ):
        """Configures the object using configuration data from the specified section of the specified provider."""
        parameters = configProvider[ configSection ]
        if parameters.has_key( 'database' ):
            self._SetDatabase( parameters[ 'database' ] )
        if parameters.has_key( 'host' ):
            self._SetHost( parameters[ 'host' ] )
        if parameters.has_key( 'password' ):
            self._SetPassword( parameters[ 'password' ] )
        if parameters.has_key( 'user' ):
            self._SetUser( parameters[ 'user' ] )

    ##################################
    # IsDataConnector Methods        #
    ##################################

    # Leaving Connect method abstracted from IsDataConnector

    # Leaving Query method abstracted from IsDataConnector

__all__ += [ 'BaseDataConnector' ]

Line(s)
12-30: Typical (by now) property-getter methods.
36-74: Typical (by now) property-setter methods.
80-98: I'm generating explicit Deleter methods as well, since I want the properties to be "deletable," but without allowing actual deletion - instead I want them to be set to None. Partly this is so that I've got an easy way to set them to their default values during construction or configuration, but mostly it's because I anticipate that in actual implementations I'll need to check whether they're set or not, and I'd rather that they always have at least a None value, rather than raising AttributeErrors if they get accidentally deleted.
104-107: The "real" property declarations.
113-134: The object constructor. Note that we're using a keyword-list style argument, documenting the recognized names, and using the class' _Set... methods to actually set the values. We could use the properties that those methods map to, but by going directly to the methods themselves, if the "settability" of the property is changed, that won't introduce errors.
148-166: Object configuration. To keep things simple, it uses the same names as the object constructor.; I'm not sure how I want to approach the unit-testing of this method just yet (as noted in the ToDo on 148). I'll have to give that some thought, and I'll likely generate a "clean-up" post once everything else is done that will address that and any other items that seem relevant, but for now, I'm going to leave this untested.

Unit-tests

    class BaseDataConnectorDerived( BaseDataConnector ):
        def __init__( self, **parameters ):
            BaseDataConnector.__init__( self, **parameters )
    
    class testBaseDataConnector( unittest.TestCase ):
        """Unit-tests the BaseDataConnector class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
        def testDerived( self ):
            """Testing abstract nature of the BaseDataConnector class."""
            try:
                testObject = BaseDataConnector()
                self.fail( 'BaseDataConnector is nominally abstract, and should not be instantiable.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'BaseDataConnector is nominally abstract, and should raise NotImplementedError when instantiated, but %s was raised instead\n  %s' % ( error.__class__.__name__, error ) )
            testObject = BaseDataConnectorDerived()
            self.assertTrue( isinstance( testObject, BaseDataConnector ) , 'BaseDataConnector-derived objects should be instances of BaseDataConnector' )
    
        def testPropertyCountAndTests( self ):
            """Testing the properties of the BaseDataConnector class."""
            items = getMemberNames( BaseDataConnector )[0]
            actual = len( items )
            expected = 5
            self.assertEquals( expected, actual, 'BaseDataConnector is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the BaseDataConnector class."""
            items = getMemberNames( BaseDataConnector )[1]
            actual = len( items )
            expected = 3
            self.assertEquals( expected, actual, 'BaseDataConnector is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

        # Unit-test properties

        def testConnection( self ):
            """Unit-tests the Connection property of the BaseDataConnector abstract class."""
            self.assertEquals( BaseDataConnector.Connection, IsDataConnector.Connection, 'BaseDataConnector implements IsdataConnector, but should not define the Connection property.' )

        def testDatabase( self ):
            """Unit-tests the Database property of the BaseDataConnector abstract class."""
            goodValues = [ 'databasename', 'database_name', 'a' ]
            for testValue in goodValues:
                testObject = BaseDataConnectorDerived()
                testObject.Database = testValue
                self.assertEquals( testObject.Database, testValue, 'The database property, if set, should get the set value.' )
                testObject = BaseDataConnectorDerived( database=testValue )
                self.assertEquals( testObject.Database, testValue, 'The database property, if set, should get the set value.' )
            badTypes = [ None, True, 0, [] ]
            for testValue in badTypes:
                testObject = BaseDataConnectorDerived()
                try:
                    testObject.Database = testValue
                    self.fail( 'Setting Database to %s should raise a TypeError.' )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Setting Database to %s should raise a TypeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
                try:
                    testObject = BaseDataConnectorDerived( database=testValue)
                    self.fail( 'Creating a BaseDataConnector with a database value of %s should raise a TypeError.' )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Creating a BaseDataConnector with a database value of %s should raise a TypeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            badValues = [ '', 'bad\nvalue', 'bad\tvalue', 'bad\rvalue' ]
            for testValue in badValues:
                testObject = BaseDataConnectorDerived()
                try:
                    testObject.Database = testValue
                    self.fail( 'Setting Database to %s should raise a ValueError.' )
                except ValueError:
                    pass
                except Exception, error:
                    self.fail( 'Setting Database to %s should raise a ValueError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
                try:
                    testObject = BaseDataConnectorDerived( database=testValue)
                    self.fail( 'Creating a BaseDataConnector with a database value of %s should raise a ValueError.' )
                except ValueError:
                    pass
                except Exception, error:
                    self.fail( 'Creating a BaseDataConnector with a database value of %s should raise a ValueError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testHost( self ):
            """Unit-tests the Host property of the BaseDataConnector abstract class."""
            goodValues = [ 'Hostname', 'Host_name', 'a' ]
            for testValue in goodValues:
                testObject = BaseDataConnectorDerived()
                testObject.Host = testValue
                self.assertEquals( testObject.Host, testValue, 'The Host property, if set, should get the set value.' )
                testObject = BaseDataConnectorDerived( host=testValue )
                self.assertEquals( testObject.Host, testValue, 'The Host property, if set, should get the set value.' )
            badTypes = [ None, True, 0, [] ]
            for testValue in badTypes:
                testObject = BaseDataConnectorDerived()
                try:
                    testObject.Host = testValue
                    self.fail( 'Setting Host to %s should raise a TypeError.' )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Setting Host to %s should raise a TypeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
                try:
                    testObject = BaseDataConnectorDerived( host=testValue)
                    self.fail( 'Creating a BaseDataConnector with a Host value of %s should raise a TypeError.' )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Creating a BaseDataConnector with a Host value of %s should raise a TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )
            badValues = [ '', 'bad\nvalue', 'bad\tvalue', 'bad\rvalue' ]
            for testValue in badValues:
                testObject = BaseDataConnectorDerived()
                try:
                    testObject.Host = testValue
                    self.fail( 'Setting Host to %s should raise a ValueError.' % ( testValue ) )
                except ValueError:
                    pass
                except Exception, error:
                    self.fail( 'Setting Host to %s should raise a ValueError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
                try:
                    testObject = BaseDataConnectorDerived( host=testValue)
                    self.fail( 'Creating a BaseDataConnector with a Host value of %s should raise a ValueError.' )
                except ValueError:
                    pass
                except Exception, error:
                    self.fail( 'Creating a BaseDataConnector with a Host value of %s should raise a ValueError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )

        def testPassword( self ):
            """Unit-tests the Password property of the BaseDataConnector abstract class."""
            goodValues = [ 'password', 'pass_word', 'a' ]
            for testValue in goodValues:
                testObject = BaseDataConnectorDerived()
                testObject.Password = testValue
                self.assertEquals( testObject.Password, testValue, 'The Password property, if set, should get the set value.' )
                testObject = BaseDataConnectorDerived( password=testValue )
                self.assertEquals( testObject.Password, testValue, 'The Password property, if set, should get the set value.' )
            badTypes = [ None, True, 0, [] ]
            for testValue in badTypes:
                testObject = BaseDataConnectorDerived()
                try:
                    testObject.Password = testValue
                    self.fail( 'Setting Password to %s should raise a TypeError.' )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Setting Password to %s should raise a TypeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
                try:
                    testObject = BaseDataConnectorDerived( password=testValue )
                    self.fail( 'Creating a BaseDataConnector with a password value of %s should raise a TypeError.' % ( testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Creating a BaseDataConnector with a password value of %s should raise a TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )
            badValues = [ '', 'bad\nvalue', 'bad\tvalue', 'bad\rvalue' ]
            for testValue in badValues:
                testObject = BaseDataConnectorDerived()
                try:
                    testObject.Password = testValue
                    self.fail( 'Setting Password to %s should raise a ValueError.' % ( testValue ) )
                except ValueError:
                    pass
                except Exception, error:
                    self.fail( 'Setting Password to %s should raise a ValueError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
                try:
                    testObject = BaseDataConnectorDerived( password=testValue)
                    self.fail( 'Creating a BaseDataConnector with a password value of %s should raise a ValueError.' )
                except ValueError:
                    pass
                except Exception, error:
                    self.fail( 'Creating a BaseDataConnector with a password value of %s should raise a ValueError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testUser( self ):
            """Unit-tests the User property of the BaseDataConnector abstract class."""
            goodValues = [ 'User', 'user_name', 'a' ]
            for testValue in goodValues:
                testObject = BaseDataConnectorDerived()
                testObject.User = testValue
                self.assertEquals( testObject.User, testValue, 'The User property, if set, should get the set value.' )
                testObject = BaseDataConnectorDerived( user=testValue )
                self.assertEquals( testObject.User, testValue, 'The User property, if set, should get the set value.' )
            badTypes = [ None, True, 0, [] ]
            for testValue in badTypes:
                testObject = BaseDataConnectorDerived()
                try:
                    testObject.User = testValue
                    self.fail( 'Setting User to %s should raise a TypeError.' )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Setting User to %s should raise a TypeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
                try:
                    testObject = BaseDataConnectorDerived( user=testValue )
                    self.fail( 'Creating a BaseDataConnector with a User value of %s should raise a TypeError.' % ( testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Creating a BaseDataConnector with a User value of %s should raise a TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )
            badValues = [ '', 'bad\nvalue', 'bad\tvalue', 'bad\rvalue' ]
            for testValue in badValues:
                testObject = BaseDataConnectorDerived()
                try:
                    testObject.User = testValue
                    self.fail( 'Setting User to %s should raise a ValueError.' % ( testValue ) )
                except ValueError:
                    pass
                except Exception, error:
                    self.fail( 'Setting User to %s should raise a ValueError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
                try:
                    testObject = BaseDataConnectorDerived( user=testValue)
                    self.fail( 'Creating a BaseDataConnector with a User value of %s should raise a ValueError.' )
                except ValueError:
                    pass
                except Exception, error:
                    self.fail( 'Creating a BaseDataConnector with a User value of %s should raise a ValueError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        # Unit-test methods

        def testConfigure( self ):
            """Unit-tests the Configure method of the BaseDataConnector abstract class."""
            # Not sure how I want to test this yet. See ToDo on Configure.
            pass

        def testConnect( self ):
            """Unit-tests the Connect method of the BaseDataConnector abstract class."""
            self.assertEquals( BaseDataConnector.Connect, IsDataConnector.Connect, 'BaseDataConnector should not define the Connect method.' )

        def testQuery( self ):
            """Unit-tests the Query method of the BaseDataConnector abstract class."""
            self.assertEquals( BaseDataConnector.Query, IsDataConnector.Query, 'BaseDataConnector should not define the Query method.' )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testBaseDataConnector ) )

With the exception of the method-tests, this should be pretty straightforward by now - it's my typical structure...

The method-tests, though, may deserve some explanation. I noted above that I wasn't sure how I wanted to test the Configure method (as much as anything else because it, ideally, should be tested against real configuration-files, and I'm not sure I want to generate the sheer number of files that would be needed). For now, I'm going to leave that method untested, though I'll have to get back to that fairly soon to maintain my code-coverage goals for code tied to my posts.

The tests for Connect and Query are pretty simple, but what they do may not be readily apparent. Each of these is really just making sure that there is no local method (e.g., that BaseDataConnector doesn't define them, but inherits the nominally-abstract stubs from IsDataConnector).

The next nominally abstract class to tackle is BaseDataObject. The intention behind BaseDataObject is to provide common CRUD functionality for derived objects, as noted previously. In order to try to keep the actual read/write/update/delete process as simple as possible, creation of and update to a state-data record is going to be wrapped in a Save method that pays attention to various flag-value properties:

IsDeleted: Indicates whether an object's state-data record is pseudo-deleted. It's not uncommon, at least in my experience, to save even "deleted" records in a system for any of several reasons, practical and legal alike. This flag exists in order to assure that data-objects can be saved while being easily flagged as deleted.
IsDirty: Indicates whether an object's state-data record needs to be updated in the back-end data-store. This means that objects deriving from BaseDataObject will have to make sure that when a persistent state value changes, the entire object is flagged as "dirty".
IsNew: Indicates whether an object's state-data record needs to be created in the back-end data-store. When a record is created, the unique idenitifier of the record will need to be returned in some fashion, and stored in the object's state if at all possible.

These properties have implications on the design of database-tables that store object state-data: They should ideally have a unique identifier (which is generally desirable anyway), and a flag-field that indicates whether the state-record is "deleted" (or is active, if you prefer that approach). Other common fields (creation date, last modified date) could be added easily enough to BaseDataObject, but they feel less... globally used, I guess, to me.

The IsDirty and IsNew flags raise some questions as well. Ideally, I think, no given object should ever be considered both New and Dirty, since one implies that the object-record hasn't been created, while the other implies that it has, and on top of that, that it's been modified since it was created. An object that is created in an application and that will be saved to a database should, therefore, be considered "new" until it has been saved (at which point both flags should be False). Changes to that object's state before it's saved should also make the object "dirty," logically, but we don't want both a creation and an update to happen on the same object when it's being saved. I believe that all of this should be handled at the Save-method level, with logic that would look something like this:

When an object is created, it is new and not dirty; Saving it should create a state-record.
When that object is modified, it is both new and dirty; Saving it should create a state-record.
When that object is pseudo-deleted, it is both new and deleted; Saving it should create a state-record, and that record should indicate that it is "deleted".
When an object is retrieved from the database, it is not new, nor is it dirty; Saving it should do nothing, if only to avoid unneeded database connections/interaction.
When a retrieved object is modified, it is dirty but still not new; Saving it should cause an update.
When a retrieved object is pseudo-deleted, it is both dirty and deleted; Saving it should cause an update.
Any of these changes should be accounted for when the object is destroyed, in order to assure that state-data is correctly persisted.

BaseDataObject (Nominal abstract class):

class BaseDataObject( object ):
    """Provides baseline functionality, interface requirements and type-identity for objects whose state can be persisted in a back-end database."""

    ##################################
    # Class Attributes               #
    ##################################

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetDataSource( self ):
        """Gets the data-source (an IsDataConnector instance) that the data-object will use to perform it's queries against."""
        return self._dataSource

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetId( self ):
        """Gets the unique ID of the object's state-data-record in the back-end database."""
        return self._id

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetIsDeleted( self ):
        """Gets the flag indicating whether the object's state-data record is "deleted"."""
        return self._isDeleted

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetIsDirty( self ):
        """Gets the flag indicating whether the object's state-data record is "Dirty"."""
        return self._isDirty

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetIsNew( self ):
        """Gets the flag indicating whether the object's state-data record is "New"."""
        return self._isNew

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, 'The data-source (an IsDataConnector instance) that the data-object will use to perform it\'s queries against.' )
    def _SetDataSource( self, value ):
        """Sets the data-source (an IsDataConnector instance) that the data-object will use to perform it's queries against."""
        if self._dataSource:
            raise NotImplementedError( '%s.DataSource cannot be changed after it has been set.' % ( self.__class__.__name__ ) )
        if not isinstance( value, IsDataConnector ):
            raise TypeError( '%s.DataSource expects an instance of IsDataConnector.' % ( self.__class__.__name__ ) )
        self._dataSource = value

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, 'The unique ID of the object\'s state-data-record in the back-end database.' )
    def _SetId( self, value ):
        """Sets the unique ID of the object's state-data-record in the back-end database."""
        self._id = value

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, 'The flag indicating whether the object\'s state-data record is "deleted".' )
    def _SetIsDeleted( self, value ):
        """Sets the flag indicating whether the object's state-data record is "deleted"."""
        if value != True and value != False and value != 1 and value != 0:
            raise TypeError( '%s.IsDeleted expects a True, False, 1 or 0 value.' % ( self.__class__.__name__ ) )
        self._isDeleted = value

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, 'The flag indicating whether the object\'s state-data record is "dirty" (in need of an update).' )
    def _SetIsDirty( self, value ):
        """Sets the flag indicating whether the object's state-data record is "dirty" (in need of an update)."""
        if value != True and value != False and value != 1 and value != 0:
            raise TypeError( '%s.IsDirty expects a True, False, 1 or 0 value.' % ( self.__class__.__name__ ) )
        self._isDirty = value
        if value:
            self._isNew = False

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, 'The flag indicating whether the object\'s state-data record is "New".' )
    def _SetIsNew( self, value ):
        """Sets the flag indicating whether the object's state-data record is "New"."""
        if value != True and value != False and value != 1 and value != 0:
            raise TypeError( '%s.IsNew expects a True, False, 1 or 0 value.' % ( self.__class__.__name__ ) )
        self._isNew = value

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _DelDataSource( self ):
        """Deletes the data-source that the data-object will use to perform it's queries against."""
        self._dataSource = None

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _DelId( self ):
        """Deletes the data-source that the data-object will use to perform it's queries against."""
        self._id = None

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _DelIsDeleted( self ):
        """Deletes the flag indicating whether the object's state-data record is "Deleted"."""
        self._isDeleted = None

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _DelIsDirty( self ):
        """Deletes the flag indicating whether the object's state-data record is "dirty" (in need of an update)."""
        self._isDirty = None

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _DelIsNew( self ):
        """Deletes the flag indicating whether the object's state-data record is "New" (in need of creation)."""
        self._isNew = None

    ##################################
    # Class Properties               #
    ##################################

    DataSource = property( _GetDataSource, _SetDataSource, None, 'Gets or sets the data-source (an IsDataConnector instance) that the data-object will use to perform it\'s queries against.' )
    Id = property( _GetId, None, None, 'Gets the unique identifier of the object\'s state-data record.')
    IsDeleted = property( _GetIsDeleted, _SetIsDeleted, None, 'Gets or sets the flag indicating that the object\'s state-data record is pseudo-deleted.' )
    IsDirty = property( _GetIsDirty, _SetIsDirty, None, 'Gets or sets the flag indicating that the object\'s state-data record is "dirty," and in need of an update.' )
    IsNew = property( _GetIsNew, None, None, 'Gets or sets the flag indicating that the object\'s state-data record is "New," and needs to be created.' )

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def __init__( self ):
        """Object constructor."""
        # Set new and dirty flags before checking for instantiability, in order to 
        # avoid raising errors at destruction.
        self._isDirty = None
        self._isNew = None
        # Nominally abstract: Don't allow instantiation of the class
        if self.__class__ == BaseDataObject:
            raise NotImplementedError( 'BaseDataObject is (nominally) an abstract class, and is not intended to be instantiated.' )
        self._DelDataSource()
        self._DelId()
        self.IsDeleted = False
        self.IsDirty = False
        self._isNew = True

    ##################################
    # Object Destructor              #
    ##################################

    def __del__( self ):
        """Object destructor.

Saves the object's state-data before destruction."""
        self.Save()

    ##################################
    # Class Methods                  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _Create( self ):
        """(Nominal abstract method) Creates a record in the back-end database for the object's state-data, updating the object's Id property with the identity of the created record."""
        raise NotImplementedError( '%s._Create has not been implemented as required by BaseDataObject.' % ( self.__class__.__name__ ) )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def Delete( self ):
        """(Nominal abstract method) Performs a physical record-deletion of a specific object's state-data record from the back-end database."""
        raise NotImplementedError( '%s.Delete has not been implemented as required by BaseDataObject.' % ( self.__class__.__name__ ) )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'id', None, 'The unique identifier for the state-data record of the object to be retrieved from the database.' )
    def Read( self, id ):
        """(Nominal abstract method) Reads a specific object's state-data record from the back-end database, and returns an object-instance with the retrieved state-data."""
        raise NotImplementedError( '%s.Read has not been implemented as required by BaseDataObject.' % ( self.__class__.__name__ ) )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def Save( self ):
        """Creates or updates the object's state-data record based on whether the object's IsNew or IsDirty flag is set."""
        if self._isNew:
            self._Create()
        elif self._isDirty:
            self._Update()
        self.IsDirty = False
        self._isNew = False

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _Update( self ):
        """(Nominal abstract method) Updates the object's state-data record in the back-end database."""
        raise NotImplementedError( '%s._Update has not been implemented as required by BaseDataObject.' % ( self.__class__.__name__ ) )

__all__ += [ 'BaseDataObject' ]

Commentary

Line(s)

13-35

Typical property-getter structures.

42-49

Standard property-setter structure.

51-55

The property-setter for the Id property is slightly different than the typical pattern I've shown in previous code, in that it just sets the value, performing no type- or value-checking. Id could, arguably, be left abstract at the BaseDataObject level, but there doesn't seem to me to be any significant advantage to doing so - I'd rather have a single point of definition, and (if necessary) override the setter method in derived classes to provide type- and/or value-checking when or if necessary. I have a lingering suspicion that I'll regret that design decision somewhere along the line, but for the time being, I'll hold to it, and seee what happens.

58-63, 66-72, 75-80

Since the properties behind these setters are boolean values, there's no real need to do a type-check them - a simple check of all of the allowed values should suffice.

71-73: Since setting the object's state to "dirty" should reset it's "new" status, we're doing that here, but only if the "dirty" status is True (or equivalent).

88-110

Typical property-deleter method-structure.

116-120

Typical property definitions.

146-150

By default, any object derived from BaseDataObject should save it's state before it is destroyed. Since the _Create and _Update methods are still abstract (and thus need to be defined by the derived classes), but still exist at the BaseDataObject level, the destructor can take advantage of the methods' existence, define the Save method, and call it in the object destructor.

157-159, 162-164, 168-170, 183-185

The nominally-abstract methods for state-data-record creation, deletion, read/retrieval and update, respectively. These must be abstract at this level, because there is no way to anticipate exactly how they will work, but we know that they need to exist.

173-180

The concrete method that creates or updates a state-data record for the derived object, based on the object's "dirty" or "new" state.

For the most part, the unit-tests are pretty much what you'd expect, particularly if you've been following this blog for any length of time:

    class BaseDataObjectDerived( BaseDataObject ):
        def __init__( self ):
            BaseDataObject.__init__( self )
        def __del__( self ):
            pass
    
    class BaseDataObjectImplemented( BaseDataObject ):
        def __init__( self ):
            BaseDataObject.__init__( self )
        def __del__( self ):
            BaseDataObject.__del__( self )
        def _Create( self ):
            fp = open( '/tmp/create', 'w' )
            fp.write( '_Create' )
            fp.close()
        def _Update( self ):
            fp = open( '/tmp/update', 'w' )
            fp.write( '_Update' )
            fp.close()
    
    class testBaseDataObject( unittest.TestCase ):
        """Unit-tests the BaseDataObject class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            try:
                os.unlink( '/tmp/create' )
            except:
                pass
            try:
                os.unlink( '/tmp/update' )
            except:
                pass
        
        def testDerived( self ):
            """Testing abstract nature of the BaseDataObject class."""
            try:
                testObject = BaseDataObject()
                self.fail( 'BaseDataObject is nominally an abstract class, and should not be instantiable.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Attempting to instantiate a BaseDataObject should raise NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            testObject = BaseDataObjectDerived()
            self.assertTrue( isinstance( testObject, BaseDataObject ), 'Objects derived from BaseDataObject should be instances of BaseDataObject.' )

        def testDestruction( self ):
            """Testing destruction of the BaseDataObject class."""
            testObject = BaseDataObjectImplemented()
            del testObject
            try:
                fp = open( '/tmp/create', 'r' )
                fp.close()
            except IOError:
                self.fail( 'Destruction of a new object should result in it\'s _Create method being called.' )
            except Exception, error:
                self.fail( 'Destruction of a new object should result in it\'s _Create method being called, but %s was raised"\n  %s' % ( error.__class__.__name__, error ) )
            testObject = BaseDataObjectImplemented()
            testObject.IsDirty = True
            del testObject
            try:
                fp = open( '/tmp/update', 'r' )
                fp.close()
            except IOError:
                self.fail( 'Destruction of a modified object should result in it\'s _Update method being called.' )
            except Exception, error:
                self.fail( 'Destruction of a modified object should result in it\'s _Update method being called, but %s was raised"\n  %s' % ( error.__class__.__name__, error ) )
    
        def testPropertyCountAndTests( self ):
            """Testing the properties of the BaseDataObject class."""
            items = getMemberNames( BaseDataObject )[0]
            actual = len( items )
            expected = 5
            self.assertEquals( expected, actual, 'BaseDataObject is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the BaseDataObject class."""
            items = getMemberNames( BaseDataObject )[1]
            actual = len( items )
            expected = 3
            self.assertEquals( expected, actual, 'BaseDataObject is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

        # Test properties

        def testDataSource( self ):
            """Unit-tests the DataSource property of the BaseDataObject nominal abstract class."""
            testValue = IsDataConnectorDerived()
            testObject = BaseDataObjectDerived()
            testObject.DataSource = testValue
            self.assertEquals( testObject.DataSource, testValue, 'Setting the DataSource should return the set value on a get.' )
            badValues = [ None, True, 1, 'ook' ]
            for testValue in badValues:
                try:
                    testObject = BaseDataObjectDerived()
                    testObject.DataSource = testValue
                    self.fail( 'Setting DataSource to a value of %s should raise a TypeError' % ( testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Setting DataSource to a value of %s should raise a TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )
            testObject = BaseDataObjectDerived()
            testValue = IsDataConnectorDerived()
            testValue2 = IsDataConnectorDerived()
            testObject.DataSource = testValue
            try:
                testObject.DataSource = testValue2
                self.fail( 'Changing an established DataSource of a BaseDataObject should not be allowed' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Changing an established DataSource of a BaseDataObject should raise NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testId( self ):
            """Unit-tests the Id property of the BaseDataObject nominal abstract class."""
            # There really isn't any useful test that can be done here without
            # mandating an ID type or structure, so this test really doesn't do
            # anything except get counted as a testable property.
            pass

        def testIsDeleted( self ):
            """Unit-tests the IsDeleted property of the BaseDataObject nominal abstract class."""
            testObject = BaseDataObjectDerived()
            goodValues = [ True, False, 1, 0 ]
            for testValue in goodValues:
                testObject.IsDeleted = testValue
                self.assertEquals( testObject.IsDeleted, testValue, 'Setting the IsDeleted value should returnt he same value when gotten.' )
            badValues = [ None, 2, 'ook', -1, object() ]
            for testValue in badValues:
                try:
                    testObject.IsDeleted = testValue
                    self.fail( 'Setting IsDeleted to a value of %s should raise TypeError' % ( testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Setting IsDeleted to a value of %s should raise TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )

        def testIsDirty( self ):
            """Unit-tests the IsDirty property of the BaseDataObject nominal abstract class."""
            testObject = BaseDataObjectDerived()
            goodValues = [ True, False, 1, 0 ]
            for testValue in goodValues:
                testObject.IsDirty = testValue
                self.assertEquals( testObject.IsDirty, testValue, 'Setting the IsDirty value should returnt he same value when gotten.' )
            badValues = [ None, 2, 'ook', -1, object() ]
            for testValue in badValues:
                try:
                    testObject.IsDirty = testValue
                    self.fail( 'Setting IsDirty to a value of %s should raise TypeError' % ( testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Setting IsDirty to a value of %s should raise TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )

        def testIsNew( self ):
            """Unit-tests the IsNew property of the BaseDataObject nominal abstract class."""
            testObject = BaseDataObjectDerived()
            self.assertEquals( testObject.IsNew, True )
#            self.assertEquals( testObject.IsNew, True, 'A default BaseDataObject-derived object should start with IsNew of True' )

        # Test methods

        def testDelete( self ):
            """Unit-tests the Delete method of the BaseDataObject nominal abstract class."""
            testObject = BaseDataObjectDerived()
            try:
                testObject.Delete()
                self.fail( 'BaseDataObject should not implement Delete.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'BaseDataObject.Delete should raise NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testRead( self ):
            """Unit-tests the Read method of the BaseDataObject nominal abstract class."""
            testObject = BaseDataObjectDerived()
            try:
                testObject.Read( None )
                self.fail( 'BaseDataObject should not implement Read.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'BaseDataObject.Read should raise NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testSave( self ):
            """Unit-tests the Save method of the BaseDataObject nominal abstract class."""
            testObject = BaseDataObjectImplemented()
            testObject.Save()
            try:
                fp = open( '/tmp/create', 'r' )
                fp.close()
            except IOError:
                self.fail( 'Save called on a new object should result in it\'s _Create method being called.' )
            except Exception, error:
                self.fail( 'Save called on a new object should result in it\'s _Create method being called, but %s was raised:\n  %s.' % ( error.__class__.__name__, error ) )
            testObject.IsDirty = True
            testObject.Save()
            try:
                fp = open( '/tmp/create', 'r' )
                fp.close()
            except IOError:
                self.fail( 'Save called on an updated object should result in it\'s _Create method being called.' )
            except Exception, error:
                self.fail( 'Save called on an updated object should result in it\'s _Create method being called, but %s was raised:\n  %s.' % ( error.__class__.__name__, error ) )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testBaseDataObject ) )

Line(s)

1-5, 7-19

Since there's a need to test both the default, unimplemented methods as well as whether those methods get called as expected, there are actually two derived test-classes that are used. The first, BaseDataObjectDerived is used to make sure that any unimplemented abstract methods raise the exceptions that they should (a key part of making sure that the nominally-abstract class will respond with expected abstract-like behavior). The second, BaseDataObjectImplemented, provides implementations for _Create and _Update that write to files that the test-methods can access - these are a fair, but low-weight representation of the basic kind of data-persistence process that would be implemented (storing data to some external source), though in this case, it's file-based instead of being tied to a database (which we'd want to abstract away anyway).

27-35

Since some of the test-methods are writing files, and a failure in a test-method could leave a written file that would generate a false positive result in/for later tests, the tearDown method is used to make sure that if the expected files exist, they are deleted.

The potential for false positives in this test is particularly worrisome, since it would be possible for a bad test-run one day to leave an undeleted test-result file that another test-run could pick up minutes, hours, or days later. The fact that the files in question are being generated in /tmp would potentially help, but not prevent this from happening.

37-51

Fairly typical construction/derivation

49-69

Tests the destruction of a BaseDataObject-derived class with _Create and _Update implemented.

51: Creates a new test-object, with an expected IsNew of True and IsDirty of False. This is the default state for such a derived object.
52: Deletes the newly-created object, which should fire the _Create method.
53-59: Checks for the existence of the expected file that _Create should create
60-69: With the exception of setting IsDirty on the test-object (which should cause the _Update method to fire on destruction) and the file expected, the tests for a "dirty" object

With the exception of the unit-test for the Id property, which really can't usefully test the property anyway, the property- and method tests are, I hope, what would be expected given the typical structure/pattern of such tests from previous posts.

Test-driven Development: Implementing TypedList (part 3)

2011-12-07T04:30:00.000-08:00

I was also going to undertake some research into the methods that TypedList needed to override or otherwise implement in order to make it work the way it needs to. That research leads me to believe that there are a total of 8 methods that I need to be concerned with:

__init__: the object constructor has to check any items supplied to it for type-restrictions, but if there are no invalid types provided, it can call list.__init__ and return whatever comes back from that;
__add__: called when concatenating two lists (in this case). I have to assume that any items provided are suspect until proven otherwise. I also couldn't determine how many times this method gets called in cases of multiple additions;
__iadd__: called when concatenating with +=, same assumptions as for __add__;
__setitem__: called when assigning a value to a specific position within the list. The value is treated as a non-iterable (e.g., passing a list results in a list in the position of the original list);
__setslice__: Called when assigning values to a slice, items being set need to be type-checked;
append: Called when appending values to the list, items added need to be type-checked;
insert: Called when inserting a value to a specific position in the list, the value needs to be type-checked;
extend: Called to extend items to the end of the list, same drill;

This list of methods was mostly generated by looking at the documentation-sources noted in the last post, but since I wasn't sure exactly what some of the operational parameters for them were (whether they could handle single items only, iterable items, etc., and how they would deal with unexpected types), I also hacked up a quick little test-script to provide some visibility into what exactly happened when various operations and list-methods were called:

class MyList( list, object ):

    def __init__( self, iterable=None ):
        print "MyList.__init__( %s )" % ( iterable )
        list.__init__( self, iterable )
    def __add__( self, items ):
        print "MyList.__add__( %s )" % ( items )
        return list.__add__( self, items )
    def __iadd__( self, items ):
        print "MyList.__iadd__( %s )" % ( items )
        return list.__iadd__( self, items )
    def __setitem__( self, index, item ):
        print "MyList.__setitem__( %s, %s )" % ( index, item )
        return list.__setitem__( self, index, item )
    def __setslice__( self, start, end, items ):
        print "MyList.__setslice__( %s, %s, %s )" % ( start, end, items )
        return list.__setslice__( self, start, end, items )
    def append( self, item ):
        print "MyList.append( %s )" % ( item )
        return list.append( self, item )
    def insert( self, index, item ):
        print "MyList.insert( %s, %s )" % ( index, item )
        return list.insert( self, index, item )
    def extend( self, items ):
        print "MyList.extend( %s )" % ( items )
        return list.extend( self, items )

tester = MyList( ['eek'] )                # uses __init__
tester += [ 'ook' ]                       # uses __iadd__
tester.append( 'eek' )                    # uses append
tester.insert( 1, 'OOOOK!' )              # uses insert
tester[1:2] = [ 'spam', 'eggs' ]          # uses __setslice__
tester += [ 'oooo', 'ooooo!' ]            # uses __iadd__
tester[2] = 'bork!'                       # uses __setitem__
tester.extend( [ 'eeee', 'eeee' ] )       # uses extend
tester = tester + [ 'mooo!' ] + [ 'me' ]  # uses __add__
print
print tester

This little script provided a lot of insight, actually - Though I'd seen some of the documentation for __add__ and __iadd__, for example, it never really stuck in my head that these were the methods that were called when + and += operators were used.

So, with our unit-tests set, and a solid feel for what the code needed to override from list, TypedList was pretty easily written:

TypedList (Nominal final class):

class TypedList( BaseTypedCollection, list ):
    """Represents a strongly-typed List."""

    ##################################
    # Class Attributes               #
    ##################################

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    ##################################
    # Class Properties               #
    ##################################

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'memberTypes', None, '(Iterable, required) An iterable collection of item-types to be allowed as members in the list.' )
    @DocumentArgument( 'argument', 'iterable', None, '(Iterable, required) An iterable collection of item-values to populate the TypedList with at creation.' )
    def __init__( self, memberTypes, iterable=None ):
        """Object constructor."""
        # Nominally final: Don't allow any class other than this one
        if self.__class__ != TypedList:
            raise NotImplementedError( 'TypedList is (nominally) a final class, and is not intended to be derived from.' )
        self._SetMemberTypes( memberTypes )
        list.__init__( self )
        if iterable != None:
            for item in iterable:
                self.append( item )

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

    ##################################
    # Methods overridden from list   #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def __add__( self, items ):
        """Called when adding items to the list with a "+" (e.g., TL + items)."""
        if not isinstance( items, list ):
            raise TypeError( 'TypedList.__add__ expects a list (or list-derived) instance of values.' % ( self._memberTypes ) )
        for item in items:
            if not self.IsValidMemberType( item ):
                raise TypeError( 'TypedList.__add__ expects a list (or list-derived) instance of values of any of %s. %s is not valid.' % ( self._memberTypes, items ) )
        return list.__add__( self, items )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def __iadd__( self, items ):
        """Called when adding items to the list with a "+=" (e.g., TL += item)."""
        if not isinstance( items, list ):
            raise TypeError( 'TypedList.__iadd__ expects a list (or list-derived) instance of values.' % ( self._memberTypes ) )
        for item in items:
            if not self.IsValidMemberType( item ):
                raise TypeError( 'TypedList.__iadd__ expects a list (or list-derived) instance of values of any of %s. %s is not valid.' % ( self._memberTypes, items ) )
        return list.__iadd__( self, items )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def __setitem__( self, index, item ):
        """Called when setting the value of a specified location in the list (TL[n] = item)."""
        if not self.IsValidMemberType( item ):
            raise TypeError( 'TypedList.__setitem__ expects an instance of any of %s.' % ( self._memberTypes ) )
        return list.__setitem__( self, index, item )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def __setslice__( self, start, end, items ):
        """Called when setting the value(s) of a slice-location in the list (TL[1:2] = item)."""
        for item in items:
            if not self.IsValidMemberType( item ):
                raise TypeError( 'TypedList.__setslice__ expects an iterable of instances of any of %s.' % ( self._memberTypes ) )
        return list.__setslice__( self, start, end, items )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def append( self, item ):
        """Called when appending items to the list."""
        if not self.IsValidMemberType( item ):
            raise TypeError( 'TypedList.append expects an instance of any of %s.' % ( self._memberTypes ) )
        return list.append( self, item )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def extend( self, items ):
        """Called when extending items to a the end of the list."""
        if not isinstance( items, list ):
            raise TypeError( 'TypedList.extend expects a list (or list-derived) instance of values.' % ( self._memberTypes ) )
        for item in items:
            if not self.IsValidMemberType( item ):
                raise TypeError( 'TypedList.extend expects a list (or list-derived) instance of values of any of %s. %s is not valid.' % ( self._memberTypes, items ) )
        return list.extend( self, items )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def insert( self, index, item ):
        """Called when inserting items to a specified location in the list (TL[1] = item)."""
        if not self.IsValidMemberType( item ):
            raise TypeError( 'TypedList.insert expects an instance of any of %s.' % ( self._memberTypes ) )
        return list.insert( self, index, item )

__all__ += [ 'TypedList' ]

Ultimately, as with TypedDictionary earlier, there's not a huge amount of code to be written. Python's built-ins and the methods specific to adding items to or replacing items in a list were the only methods that had to be overridden, and there were only two patterns for those modifications:

If the method expected a list-type (e.g., __add__, __iadd__, __setslice__ and extend):
- The supplied items are checked to make sure that they are derived from a list. This allows list-types and extensions of list types (like Typedlist) to be used.
- Once the basic supplied type is successfully checked, the items in it have to be individually type-checked (using IsValidMemberType), and if any of them are invalid, raise an error.
- If everything is good, then call (and return) the parent list-class method with the supplied value(s).
If the method expects a non-list instance value, then all that need be done is type-check the value (again, with IsValidMemberType):
- If it's invalid, raise a TypeError.
- Otherwise, call (and return) the parent list-class method with the supplied value.

Line(s)
34-35: Treat TypedList as a final class, and don't allow extension.
36: Set the member-types allowed by the instance.
37: Makes sure that the instance calls a basic list-constructor. We can (and should) ignore any supplied instance-values in the iterable argument when calling the parent constructor, though, because we want to use the overridden append method to build out the instance's list of values.
38-40: Checks to see if the constructor is being passed an iterable collection of values to populate the instance with, loops over them if they are supplied, and appends them to the instance's values one by one.; I could've just as easily looped over any supplied iterable values in the constructor, raising an error at the first bad type supplied, and then called the parent constructor with the iterable value. I have a gut-level feeling, however, that taking that approach could raise issues in the future, and since I'd've had to iterate over the values anyway, it just seemed safer to implement the way I did. I can't justify it, but I'm going to trust my instincts.
55-62, 65-72, 82-87, 97-104: Uses the list-checking pattern noted above to override __add__, __iadd__, __setslice__ and extend, respectively.
75-81, 90-94, 107-111: Uses the value-checking pattern noted above to override __setitem__, append and insert, respectively.

Test-driven Development: Implementing TypedList (part 2)

2011-12-05T04:30:00.000-08:00

The next step, if we approach this in a TDD fashion, is probably to define our unit-tests. Technically, this step probably should've been undertaken earlier, perhaps even before starting to track down the methods that we'd need to override, but since I expected that there were going to be a lot of them (or at least a lot to evaluate), I wanted to get that started fairly early. The unit-tests need to encompass all of the expected uses that we can put a TypedList to, and need to include both passing and failing (negative) tests in order to assure that they're covering the code sufficiently.

At the minimum, that means that we need to test each of the following cases:

Instantiation of a TypedList with values presented to the constructor;
Appending items to a TypedList with +;
Appending items to a TypedList with +=;
Appending items to a TypedList with append;
Appending (or extending) items to a TypedList with extend;
Setting the value in a specific TypedList location with TypedList[n] = value;
Setting one or more values in a slice of a TypedList with TypedList[i:j] = value;
Inserting a value into a specific TypedList location with insert

Given that we need to test all of these both positively (to prove that basic list behavior for allowed types is preserved) and negatively (to raise errors when they are presented with an invalid type), there are a lot of assertions that have to be made in the testListBehavior test-method. Most of the rest of the test-methods are pretty typical of the pattern I've followed thus far, so I won't spend a lot of time commenting on those.

Unit-tests

    class TypedListDerived( TypedList ):
        def __init__( self, memberTypes, iterable=None ):
            TypedList.__init__( self, memberTypes, iterable )
    
    class testTypedList( unittest.TestCase ):
        """Unit-tests the TypedList class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass

        def testFinal( self ):
            """Testing final nature of the TypedList class."""
            try:
                testObject = TypedListDerived( [ types.IntType ] )
                self.fail( 'TypedList is nominally a final class, and is not intended to be extended.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'TypedList is expected to raise NotImplementedError if a derived class is instantiated, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            testObject = TypedList( [ types.IntType ] )
            self.assertEquals( testObject, [], 'A Typedlist constructed with no values should equal an empty list.' )

        def testPropertyCountAndTests( self ):
            """Testing the properties of the TypedList class."""
            items = getMemberNames( TypedList )[0]
            actual = len( items )
            expected = 1
            self.assertEquals( expected, actual, 'TypedList is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the TypedList class."""
            items = getMemberNames( TypedList )[1]
            actual = len( items )
            expected = 4
            self.assertEquals( expected, actual, 'TypedList is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

        # Test list behavior
        def testListBehavior( self ):
            """Unit-tests list-like behavior of TypedList."""
            # Test good values - all should pass
            goodValues = [ -1, 0, 1, 2, -1L, 0L, 1L, 2L, -1.0, 0.0, 1.0, 2.0 ]
            # Test creation with an iterable
            testObject = TypedList( [ types.IntType, types.LongType, types.FloatType ], goodValues )
            self.assertEquals( testObject, goodValues, 'Values set in a TypedList at creation should act like values set in a normal list at creation.' )
            # Test L = L + i structure
            testObject = TypedList( [ types.IntType, types.LongType, types.FloatType ] )
            compareList = []
            for testValue in goodValues:
                compareList = compareList + [ testValue ]
                testObject = testObject + [ testValue ]
                self.assertEquals( testObject, compareList, 'Values added to a TypedList should act like values added to a normal list.' )
            # Test L = L += i structure
            testObject = TypedList( [ types.IntType, types.LongType, types.FloatType ] )
            compareList = []
            for testValue in goodValues:
                compareList += [ testValue ]
                testObject += [ testValue ]
                self.assertEquals( testObject, compareList, 'Values added to a TypedList should act like values added to a normal list.' )
            # Test L.append equivalent
            testObject = TypedList( [ types.IntType, types.LongType, types.FloatType ] )
            compareList = []
            for testValue in goodValues:
                compareList.append( testValue )
                testObject.append( testValue )
                self.assertEquals( testObject, compareList, 'Values appended to a TypedList should act like values appended to a normal list.' )
            # Test L.extend equivalent
            testObject = TypedList( [ types.IntType, types.LongType, types.FloatType ] )
            compareList = []
            for testValue in goodValues:
                compareList.extend( [ testValue ] )
                testObject.extend( [ testValue ] )
                self.assertEquals( testObject, compareList, 'Values extended to a TypedList should act like values extended to a normal list.' )
            # Test L.insert equivalent
            testObject = TypedList( [ types.IntType, types.LongType, types.FloatType ] )
            compareList = []
            for testValue in goodValues:
                compareList.insert( 0, testValue )
                testObject.insert( 0, testValue )
                self.assertEquals( testObject, compareList, 'Values inserted to a TypedList should act like values inserted to a normal list.' )
            # Test L[index] = value equivalent
            testObject = TypedList( [ types.IntType, types.LongType, types.FloatType ], [ 0 ] )
            compareList = [ 0 ]
            for testValue in goodValues:
                compareList[0] = testValue
                testObject[0] = testValue
                self.assertEquals( testObject, compareList, 'Values set in a position in a TypedList should act like values set in a position in a normal list.' )
            # Test L[slice] = [ i, j ] equivalent
            testObject = TypedList( [ types.IntType, types.LongType, types.FloatType ], [ 0, 0, 0 ] )
            compareList = [ 0, 0, 0 ]
            goodValues = [ [ -1, 0 ], [ 1, 2 ] ]
            for testValue in goodValues:
                compareList[0:1] = testValue
                testObject[0:1] = testValue
                self.assertEquals( testObject, compareList, 'Values set in a slice position in a TypedList should act like values set in a slice position in a normal list.' )
            # Test bad values - all should raise errors
            badValues = [ None, 'string', [ 'ook' ] ]
            testObject = TypedList( [ types.IntType, types.LongType, types.FloatType ] )
            # Test L = L + i structure
            for testValue in badValues:
                try:
                    testObject = testObject + testValue
                    self.fail( 'Adding %s with "+" to a TypedList that doesn\'t allow that type should raise a TypeError' % ( testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Adding %s with "+" to a TypedList that doesn\'t allow that type should raise a TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )
            # Test L = L += i structure
            for testValue in badValues:
                try:
                    testObject += testValue
                    self.fail( 'Adding %s with "+=" to a TypedList that doesn\'t allow that type should raise a TypeError' % ( testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Adding %s with "+=" to a TypedList that doesn\'t allow that type should raise a TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )
            # Test L.append equivalent
            for testValue in badValues:
                try:
                    testObject.append( testValue )
                    self.fail( 'Appending %s to a TypedList that doesn\'t allow that type should raise a TypeError' % ( testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Appending %s to a TypedList that doesn\'t allow that type should raise a TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )
            # Test L.extend equivalent
            for testValue in badValues:
                try:
                    testObject.extend( [ testValue ] )
                    self.fail( 'Extending %s to a TypedList that doesn\'t allow that type should raise a TypeError' % ( testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Extending %s to a TypedList that doesn\'t allow that type should raise a TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )
            # Test L.insert equivalent
            for testValue in badValues:
                try:
                    testObject.insert( 0, testValue )
                    self.fail( 'Inserting %s to a TypedList that doesn\'t allow that type should raise a TypeError' % ( testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Inserting %s to a TypedList that doesn\'t allow that type should raise a TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )
            # Test L[index] = value equivalent
            for testValue in badValues:
                try:
                    testObject[ 0 ] = testValue
                    self.fail( 'Setting %s to a position in a TypedList that doesn\'t allow that type should raise a TypeError' % ( testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Setting %s to a position in a TypedList that doesn\'t allow that type should raise a TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )
            # Test L[slice] = [ i, j ] equivalent
            badValues = [ [ None, 'string' ], [ 'string', None ] ]
            testObject = TypedList( [ types.IntType, types.LongType, types.FloatType ], [ 0, 0, 0 ] )
            for testValue in badValues:
                try:
                    testObject[ 0:1 ] = testValue
                    self.fail( 'Setting %s to a slice-position in a TypedList that doesn\'t allow that type should raise a TypeError' % ( testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Setting %s to a slice-position in a TypedList that doesn\'t allow that type should raise a TypeError, but %s was raised instead:\n  %s' % ( testValue, error.__class__.__name__, error ) )

        # Test Properties
        def testMemberTypes( self ):
            """Unit-tests the MemberTypes property of the TypedList class."""
            self.assertEquals( TypedList.MemberTypes, BaseTypedCollection.MemberTypes, 'TypedList should inherit the MemberTypes property of BaseTypedCollection.' )

        # Test Methods

        def testappend( self ):
            """Unit-tests the append method of the TypedList class."""
            pass # This is tested in testListBehavior, above.

        def testextend( self ):
            """Unit-tests the extend method of the TypedList class."""
            pass

        def testinsert( self ):
            """Unit-tests the insert method of the TypedList class."""
            pass # This is tested in testListBehavior, above.

        def testIsValidMemberType( self ):
            """Unit-tests the IsValidMemberType method of the TypedList class."""
            self.assertEquals( TypedList.IsValidMemberType, BaseTypedCollection.IsValidMemberType, 'TypedList should inherit the IsValidMemberType method of BaseTypedCollection.' )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testTypedList ) )

Line(s)
48: Most of these tests will need a simple baseline list to work with, or to draw test-values from. The TypedList instance I'm going to work with will allow integers, floats amd long integers, so all of the "good" values must fall into one of those types. I also want to make sure that it handles positive, zero and negative values correctly, and it never hurts to test for even/odd differences either, which is why there's a value of 2 of each of those types as well.
50, 53, 60, 67, etc.: In order to make sure that each test has a fresh, clean value, I'm creating a new TypedList instance each time.
54, 61, 68, etc.: All of the "good" value tests are based around the concept that if we perform the same actions/operations (with the same value[s]) on a TypedList instance as we do on an equivalent list instance, the end results will be identical. If that proves not to be the case, then the code has broken core list-functionality. Again, in order to assure that there's a fresh comparison-list copy each time, a new list is created.
50, 51: Standard lists can be initialized with an iterable set of values, so a TypedList needs to be able to do that as well.
55-58: Tests the "+" operator as it applies to TypedLists
62-65: Tests the "+=" operator as it applies to TypedLists
69-72: Tests the append method - this could be broken out into the dedicated test-method created specifically for it (testappend), but it feels better to keep this test-code here along with the other list-behavior tests.
76-79: Tests the extend method - this could be broken out into the dedicated test-method created specifically for it (testextend), but it feels better to keep this test-code here along with the other list-behavior tests.
83-86: Tests the insert method - this could be broken out into the dedicated test-method created specifically for it (testinsert), but it feels better to keep this test-code here along with the other list-behavior tests.
90-93: Tests the ability to set a value at a specific index-location in the TypedList.
97: For the slice-value test, we need a slightly different good-values structure, so we reset it here before proceeding.
98-101: Tests the ability to set values to a slice-location in the TypedList.
102-169: These are the negative tests for the same TypedList functionality and capabilities that have been positively tested in the previous code. They are somewhat simpler, in that there is no comparison that needs to be made, though it might not be a bad idea to check at each iteration that the values in the underlying list haven't changed, but it doesn't feel necessary at this point - if they do, that feels to me like there's something greviously wrong at a level in the language itself that I won't be able to fix anyway.

That's a fair amount of code for today, even if it is all unit-tests, so I'll stop here, and finish up with the "real" development in my next post. Bear in mind that we still don't have the actual TypedList functionality built at all yet - and that the unit-tests will fail, because that functionality doesn't exist. At this point in time, that's OK - when all of the "real" code is writtem and all of our tests pass, we're done! That ought to go pretty quickly, though.

Test-driven Development: Implementing TypedList (part 1)

2011-12-02T04:30:00.001-08:00

As I've been contemplating the next steps in the DataConnectors module, I've been waffling back and forth on how best to implement certain aspects of it. That's usually a sign, for me at least, that I need to take a step back and let things percolate for a while. I mentioned that I'd want/need to implement the TypedList before I got to the concrete classes of DataConnectors anyway, so:

In today's post, I'm going to tackle the creation of the TypedList class in the TypedContainers module. The shared code can be found at dl.dropbox.com/u/1917253/site-packages/TypedContainers.py, as noted when I posted the TypedDictionary some time back. I'm going to approach this from a slightly different angle, though - I'm going to start with as little as I can on the class-definition, build out the unit-tests that it needs in order to do what it needs to do, and write code until the tests pass. If this sounds familiar to you, then you're probably already using Test-Driven Development (TDD).

Part of the reason I'm taking this approach is because the standard unit-test structure that I use (specifically the testPropertyCountAndTests and testMethodCountAndTests test-methods that've been part of all of the recent test-code I've shown make the discovery of the underlying methods and their signatures fairly easy, though it can also be a bit on the tedious side when there's a significant number of properties or methods to be discovered. Another option, and one that might be considerably faster, is to just build a stub-class extending the base that we're starting with (a list in this case), then take a look at it's members with dir() or with pydoc. Use whatever works best, or is easiest for you.

So, to start with, we'll define the TypedList class with as little to it as we can manage. That means that it must inherit from the Python list, and, since we're going to be using the MemberTypes functionality provided by BaseTypedCollection, it needs to inherit from that as well. The bare-bones starting-point, then looks something like this:

class TypedList( BaseTypedCollection, list ):
    """Represents a strongly-typed List."""

    ##################################
    # Class Attributes               #
    ##################################

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    ##################################
    # Class Properties               #
    ##################################

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'memberTypes', None, '(Iterable, required) An iterable collection of item-types to be allowed as members in the list.' )
    def __init__( self, memberTypes ):
        """Object constructor."""
        # Nominally final: Don't allow any class other than this one
        if self.__class__ != TypedList:
            raise NotImplementedError( 'TypedList is (nominally) a final class, and is not intended to be derived from.' )
        self._SetMemberTypes( memberTypes )
        raise NotImplementedError( 'TypedList is not implemented yet.' )

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

__all__ += [ 'TypedList' ]

Next, and more importantly, I'm going to generate the unit-tests for the Typedlist class. I'm going to start with only my standard unit-test stub-code, in order to show how it can be used to reveal the underlying properties and methods of other objects. It's starting-point, then is very simple:

    class TypedListDerived( TypedList ):
        def __init__( self ):
            pass
    
    class testTypedList( unittest.TestCase ):
        """Unit-tests the TypedList class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
#        def testDerived( self ):
#            """Testing abstract nature of the TypedList class."""
#            pass
    
#        def testFinal( self ):
#            """Testing final nature of the TypedList class."""
#            pass
    
        def testConstruction( self ):
            """Testing construction of the TypedList class."""
            pass
    
        def testPropertyCountAndTests( self ):
            """Testing the properties of the TypedList class."""
            items = getMemberNames( TypedList )[0]
            actual = len( items )
            expected = 0
            self.assertEquals( expected, actual, 'TypedList is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the TypedList class."""
            items = getMemberNames( TypedList )[1]
            actual = len( items )
            expected = 0
            self.assertEquals( expected, actual, 'TypedList is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testTypedList ) )

With that basic test-stub in place, I'm going to run my module, and see what the unit-tests reveal (I'll trim the detailed code-dumps, though, and just show the results):

Running unit-tests
###############################################################################
Unit-tests failed (see items below)
###############################################################################
Failures
#-----------------------------------------------------------------------------#
AssertionError: TypedList is expected to have 0 methods to test, but 1 were 
dicovered by inspection.
#-----------------------------------------------------------------------------#
AssertionError: TypedList is expected to have 0 properties to test, but 1 were 
dicovered by inspection.
#-----------------------------------------------------------------------------#
###############################################################################

So, immediately, we can see that there's at least one property and one method that need to be added to our test-cases. My next step is to see what those items are, and in order to do that, I'll change the expected values in testPropertyCountAndTests and testMethodCountAndTests to match what the test-output results are expecting, then re-run the tests:

Running unit-tests
###############################################################################
Unit-tests failed (see items below)
###############################################################################
Failures
#-----------------------------------------------------------------------------#
AssertionError: There should be a test for the IsValidMemberType method 
(testIsValidMemberType), but none was identifiable.
#-----------------------------------------------------------------------------#
AssertionError: There should be a test for the MemberTypes property 
(testMemberTypes), but none was identifiable.
#-----------------------------------------------------------------------------#
###############################################################################

So, already, the standard unit-test structure has shown that we need to create testIsValidMemberType to test the IsValidMemberType method that TypedList inherits from BaseTypedCollection, and the MemberTypes property, also from BaseTypedCollection.

Since those are actually implemented in BaseTypedCollection, all we really need to test for is that they are the inherited members, though:

        # Test Properties
        def testMemberTypes( self ):
            """Unit-tests the MemberTypes property of the TypedList class."""
            self.assertEquals( TypedList.MemberTypes, BaseTypedCollection.MemberTypes, 'TypedList should inherit the MemberTypes property of BaseTypedCollection.' )

        # Test Methods
        def testIsValidMemberType( self ):
            """Unit-tests the IsValidMemberType method of the TypedList class."""
            self.assertEquals( TypedList.IsValidMemberType, BaseTypedCollection.IsValidMemberType, 'TypedList should inherit the IsValidMemberType method of BaseTypedCollection.' )

Once those test-methods are in place, we re-run the tests:

###############################################################################
Running unit-tests
###############################################################################
All unit-tests passed! Ready for build/copy!
 + Copied TypedContainers.py to ~/.local/lib/python2.6/site-packages
###############################################################################

Et viola! We now have the members inherited from BaseTypedCollection confirmed. In retrospect, these same two tests should also be added to TypedDictionary, as well as the testMethodCountAndTests and testPropertyCountAndTests test-methods, which I'm going to do now, before I forget. They didn't cause any test-failures, so that was an easy addition...

That doesn't even begin to cover the members (public or otherwise) of a list, though... If we take a look at a list-instance in a terminal, we see that there are actually a lot of members (I actually used a quick little hacked-up script to generate this list so I could show what types of members they all are):

__add__ ............ <method-wrapper '__add__'...
__class__ .......... <type 'list'>
__contains__ ....... <method-wrapper '__contains__'...
__delattr__ ........ <method-wrapper '__delattr__'...
__delitem__ ........ <method-wrapper '__delitem__'...
__delslice__ ....... <method-wrapper '__delslice__'...
__doc__ ............ list() -> new empty list
list(iterable) -> new list initialized from iterable's items
__eq__ ............. <method-wrapper '__eq__'...
__format__ ......... <built-in method __format__...
__ge__ ............. <method-wrapper '__ge__'...
__getattribute__ ... <method-wrapper '__getattribute__'...
__getitem__ ........ <built-in method __getitem__...
__getslice__ ....... <method-wrapper '__getslice__'...
__gt__ ............. <method-wrapper '__gt__'...
__hash__ ........... None
__iadd__ ........... <method-wrapper '__iadd__'...
__imul__ ........... <method-wrapper '__imul__'...
__init__ ........... <method-wrapper '__init__'...
__iter__ ........... <method-wrapper '__iter__'...
__le__ ............. <method-wrapper '__le__'...
__len__ ............ <method-wrapper '__len__'...
__lt__ ............. <method-wrapper '__lt__'...
__mul__ ............ <method-wrapper '__mul__'...
__ne__ ............. <method-wrapper '__ne__'...
__new__ ............ <built-in method __new__...
__reduce__ ......... <built-in method __reduce__...
__reduce_ex__ ...... <built-in method __reduce_ex__...
__repr__ ........... <method-wrapper '__repr__'...
__reversed__ ....... <built-in method __reversed__...
__rmul__ ........... <method-wrapper '__rmul__'...
__setattr__ ........ <method-wrapper '__setattr__'...
__setitem__ ........ <method-wrapper '__setitem__'...
__setslice__ ....... <method-wrapper '__setslice__'...
__sizeof__ ......... <built-in method __sizeof__...
__str__ ............ <method-wrapper '__str__'...
__subclasshook__ ... <built-in method __subclasshook__...
append ............. <built-in method append...
count .............. <built-in method count...
extend ............. <built-in method extend...
index .............. <built-in method index...
insert ............. <built-in method insert...
pop ................ <built-in method pop...
remove ............. <built-in method remove...
reverse ............ <built-in method reverse...
sort ............... <built-in method sort...

The reason that the various public members from append through sort at the end of this list don't raise errors in the methods-count test is because they are built-ins, rather than "normal" methods. It would be possible to add a testBuiltinCountAndTests test-method, using the same pattern as testMethodCountAndTests and testPropertyCountAndTests, but that would require modification of UnitTestUtilities that I don't want to get into right now (I'll note it for later addition, though, don't worry...).

Instead, I'm going to take a look at the pydoc output for my class, and/or dig around a bit in the Python list documentation and Emulating container types documentation for some guidance on what all of these members do. As a fall-back, in case I don't have network-access to check the online documentation, I could also create a simple list-extension class, override all of it's methods to print out that they were called (and with what arguments) before calling the parent list-methods, and see what happens when I do normal list-manipulation. I'll need to look at those normal list-manipulations anyway in order to build out unit-tests for basic list capabilities for the TypedList, so that has benefits beyond just figuring out what each method does. But that'd be time-consuming, with 45(!) items to look into, so I'm going to check the documentation first...

And that seems like a good spot to stop for today.

The DataConnectors module (part 2)

2011-11-30T04:30:00.000-08:00

The next two items tho work through are the IsQuery and IsResultSet nominal interfaces. With the sort of structure established with IsDataConnector in the last post, all that really need be done for either of these is to figure out what functionality (and properties) we're eventually going to want in the classes and objects derived from these interfaces - so it's design time again! This will fill out two more items in the class diagram.

Without going into too much detail about how the queries, results and connectors actually interact, the process for retrieving data with a query using the DataConnectors stack will look something like this:

A connector object (derived from BaseDataConnector, and in turn from IsDataConnector) will be created, pointing at some back-end database.
A query-object (derived from IsQuery) will be created with the SQL to be executed against the datasource, and the datasource itself. Whether the query-object actually executes the SQL when instatiated, or when the results are first requested is still to be determined, but it will have a Results property that will allow access to the results of the query against the database.
When the results are requested from the query-object, they will be returned as a list of dumb data-objcets, each having an attribute for each field in the rows returned for the result-set, named after the field-name, and containing the value retrieved. These attributes will be immutable.

This structure should allow a good balance between flexibility, ease of use, and ease of maintaining data-access code.

Code-share: dl.dropbox.com/u/1917253/site-packages/DataConnectors.py

IsQuery (Nominal interface):

class IsQuery( object ):
    """Class doc-string."""

    ##################################
    # Class Attributes               #
    ##################################

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    ##################################
    # Class Properties               #
    ##################################

    Datasource = property( None, None, None, 'Gets the data-source (an IsDataConnector instance) that the query was or will be executed against.' )
    Results = property( None, None, None, 'Gets the results returned by execution of the query against the object\'s Datasource.' )
    Sql = property( None, None, None, 'Gets the SQL that has been executed against the object\'s Datasource.' )

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'datasource', None, '(IsDataConncetor, required) The IsDataConnector instance that the query\'s SQL will be executed against.' )
    @DocumentArgument( 'argument', 'sql', None, '(String, required) The SQL that will be executed against the supplied datasource.' )
    def __init__( self, datasource, sql ):
        """Object constructor."""
        # Nominally abstract: Don't allow instantiation of the class
        if self.__class__ == IsQuery:
            raise NotImplementedError( 'IsQuery is (nominally) an interface, and is not intended to be instantiated.' )

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

__all__ += [ 'IsQuery' ]

There really isn't much to say about IsQuery: it's simple, it has only a few properties, and there aren't any method-stubs to it. In fact, the only significant change here is that the properties have been documented in the interface. We'll use this documentation-string later, when we define the Query class. The unit-tests are equally straightforward:

    class IsQueryDerived( IsQuery ):
        def __init__( self, datasource, sql ):
            IsQuery.__init__( self, datasource, sql )
    
    class testIsQuery( unittest.TestCase ):
        """Unit-tests the IsQuery class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
        def testDerived( self ):
            """Testing abstract nature of the IsQuery class."""
            try:
                testObject = IsQuery( None, None )
                self.fail( 'Instantiation of an IsQuery object should raise a NotImplementedError.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Instantiation of an IsQuery object should raise a NotImplementedError, but %s was raised instead:\n  %s.' % ( error.__class__.__name__, error ) )
            testObject = IsQueryDerived( None, None )
            self.assertTrue( isinstance( testObject, IsQuery ), 'Objects derived from IsQuery should be instances of IsQuery.' )

        def testPropertyCountAndTests( self ):
            """Testing the properties of the IsQuery class."""
            items = getMemberNames( IsQuery )[0]
            actual = len( items )
            expected = 3
            self.assertEquals( expected, actual, 'IsQuery is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the IsQuery class."""
            items = getMemberNames( IsQuery )[1]
            actual = len( items )
            expected = 0
            self.assertEquals( expected, actual, 'IsQuery is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

        # Test properties

        def testDatasource( self ):
            """Tests the datasource property of the IsQuery interface."""
            testObject = IsQueryDerived( None, None )
            try:
                x = testObject.Datasource
                self.fail( 'IsQuery.Datasource should not be usable as a getter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsQuery.Datasource, used as a getter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                testObject.Datasource = None
                self.fail( 'IsQuery.Datasource should not be usable as a setter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsQuery.Datasource, used as a setter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                del testObject.Datasource
                self.fail( 'IsQuery.Datasource should not be usable as a deleter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsQuery.Datasource, used as a deleter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testResults( self ):
            """Tests the Results property of the IsQuery interface."""
            testObject = IsQueryDerived( None, None )
            try:
                x = testObject.Results
                self.fail( 'IsQuery.Results should not be usable as a getter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsQuery.Results, used as a getter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                testObject.Results = None
                self.fail( 'IsQuery.Results should not be usable as a setter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsQuery.Results, used as a setter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                del testObject.Results
                self.fail( 'IsQuery.Results should not be usable as a deleter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsQuery.Results, used as a deleter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testSql( self ):
            """Tests the Sql property of the IsQuery interface."""
            testObject = IsQueryDerived( None, None )
            try:
                x = testObject.Sql
                self.fail( 'IsQuery.Sql should not be usable as a getter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsQuery.Sql, used as a getter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                testObject.Sql = None
                self.fail( 'IsQuery.Sql should not be usable as a setter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsQuery.Sql, used as a setter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                del testObject.Sql
                self.fail( 'IsQuery.Sql should not be usable as a deleter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsQuery.Sql, used as a deleter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

I didn't provide any commentary on the similarly-structure unit-tests in my last post, for IsDataConnector, so let me go over some of the new items that are common to it and to IsQuery alike:

Line(s)
14-24: The testDerived test-methods, in both cases, are really just intended to prove that the nominal interfaces behave as expected when directly instantiated and when implemented. Since there's not much in the way of "real" functionality associated with either, they shouldn't be instantiated (it would do no good to anyway, since instances would just be dumb objects with properties and methods that raise errors).
26-33, 35-42: The standard testPropertyCountAndTests and testMethodCountAndTests make sure that all properties and method have test-methods associated with them, like usual.
46-69: The testDatasource test-method makes certain that the interface's Datasource property exists, but is not implemented, raising an AttributeError when a get, set or delete operation is attempted against it.
71-94, 96-119: Test-methods, using the same structure as testDatasource, but for the Results and Sql properties of the interface.

Overall, pretty simple, I hope. The interface isn't very complex, so the tests against it shouldn't be too complex either.

The next nominal interface I'm going to delve into is IsResultSet. Before I do, though, I'd like to show the class diagram again, because when I drew it up, there was something that I did that cannot, I think, be done in languages that provide formal interface declarations. Take a moment and look at it again, if you would, paying close attention to the relationships associated with IsResultSet (upper right area):

The tricky bit here, at least as presented on the class diagram, is that IsResultSet, a nominal interface, is derived from TypedList, which is a concrete class. I'm not going to keep that structure (and I'll alter the class diagram accordingly), but this idea is, possibly, worth some discussion.

If that inheritance-relationship were kept, what we would end up with is a nominal interface that contained some actual, concrete implementation. If you're used to interfaces as they're handled in most of the languages that support formal interface declarations, this may set your eye to twitching. But, really, it might not be as bad as you'd think at first blush. In this particular case, since I'm not expecting to need more than one ResultSet class, it almost makes sense to leave the relationship - anything that implemented IsResultSet would then also be, inherently, a TypedList, which is exactly what is needed if we assume that any result-set object will be a list of rows with some additional properties.

Since that's what I'm aiming to accomplish, that doesn't actually feel too bad.

Yes, there are potential encapsulation implications, and it could certainly have implications in how unit-tests are written for IsResultSet and anything that derived from it, but even if it was discovered somewhere down the line that the TypedList/IsResultSet joint inheritance was causing problems, it'd be a simple matter of changing the inheritance there, removing some (now-superfluous) unit-tests for IsResultSet, and moving them to the derived class' unit-tests. That process might be a bit tedious, but it wouldn't be difficult (and I think it would actually be a quick change to make).

If I were 100% confident that there wouldn't be a need to have more than one ResultSet-like class in the stack, I think I'd leave the relationship and it's... odd... inheritance. But I'm not that sure that I won't want or need to have a custom implementation similar to ResultSet somewhere down the line, so I'll break it out.

So, the revised class-diagram looks like this:

Another (minor) advantage to this is that I don't have to stop and implement TypedList right here and now, though I think I'll take care of that soon, since I will need that to be in place before we get to implementation of ResultSet.

IsResultSet (Nominal interface):

class IsResultSet( object ):
    """Class doc-string."""

    ##################################
    # Class Attributes               #
    ##################################

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    ##################################
    # Class Properties               #
    ##################################

    CurrentRow = property( None, None, None, 'Gets the current row of the result-set.' )
    FieldNames = property( None, None, None, 'Gets the names of the fields in each row of the results.' )
    RowCount = property( None, None, None, 'Gets the number of rows in the result-set.' )
    RowIndex = property( None, None, None, 'Gets the index of the current row of the result-set in an iteration.' )

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def __init__( self ):
        """Object constructor."""
        # Nominally abstract: Don't allow instantiation of the class
        if self.__class__ == IsResultSet:
            raise NotImplementedError( 'IsResultSet is (nominally) an interface, and is not intended to be instantiated.' )

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

__all__ += [ 'IsResultSet' ]

The only items of note here are the property-definitions (lines 24-27), which are defined the same way the properties for IsQuery were, above.

Unit-tests

    class IsResultSetDerived( IsResultSet ):
        def __init__( self ):
            pass
    
    class testIsResultSet( unittest.TestCase ):
        """Unit-tests the IsResultSet class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
        def testDerived( self ):
            """Testing abstract nature of the IsResultSet class."""
            try:
                testObject = IsResultSet()
                self.fail( 'Instantiation of an IsResultSet object should raise a NotImplementedError.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Instantiation of an IsResultSet object should raise a NotImplementedError, but %s was raised instead:\n  %s.' % ( error.__class__.__name__, error ) )
            testObject = IsResultSetDerived()
            self.assertTrue( isinstance( testObject, IsResultSet ), 'Objects derived from IsResultSet should be instances of IsResultSet.' )

        def testPropertyCountAndTests( self ):
            """Testing the properties of the IsResultSet class."""
            items = getMemberNames( IsResultSet )[0]
            actual = len( items )
            expected = 4
            self.assertEquals( expected, actual, 'IsResultSet is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the IsResultSet class."""
            items = getMemberNames( IsResultSet )[1]
            actual = len( items )
            expected = 0
            self.assertEquals( expected, actual, 'IsResultSet is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

        # Test properties

        def testCurrentRow( self ):
            """Tests the CurrentRow property of the IsResultSet interface."""
            testObject = IsResultSetDerived()
            try:
                x = testObject.CurrentRow
                self.fail( 'IsResultSet.CurrentRow should not be usable as a getter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsResultSet.CurrentRow, used as a getter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                testObject.CurrentRow = None
                self.fail( 'IsResultSet.CurrentRow should not be usable as a setter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsResultSet.CurrentRow, used as a setter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                del testObject.CurrentRow
                self.fail( 'IsResultSet.CurrentRow should not be usable as a deleter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsResultSet.CurrentRow, used as a deleter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testFieldNames( self ):
            """Tests the FieldNames property of the IsResultSet interface."""
            testObject = IsResultSetDerived()
            try:
                x = testObject.FieldNames
                self.fail( 'IsResultSet.FieldNames should not be usable as a getter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsResultSet.FieldNames, used as a getter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                testObject.FieldNames = None
                self.fail( 'IsResultSet.FieldNames should not be usable as a setter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsResultSet.FieldNames, used as a setter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                del testObject.FieldNames
                self.fail( 'IsResultSet.FieldNames should not be usable as a deleter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsResultSet.FieldNames, used as a deleter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testRowCount( self ):
            """Tests the RowCount property of the IsResultSet interface."""
            testObject = IsResultSetDerived()
            try:
                x = testObject.RowCount
                self.fail( 'IsResultSet.RowCount should not be usable as a getter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsResultSet.RowCount, used as a getter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                testObject.RowCount = None
                self.fail( 'IsResultSet.RowCount should not be usable as a setter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsResultSet.RowCount, used as a setter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                del testObject.RowCount
                self.fail( 'IsResultSet.RowCount should not be usable as a deleter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsResultSet.RowCount, used as a deleter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testRowIndex( self ):
            """Tests the RowIndex property of the IsResultSet interface."""
            testObject = IsResultSetDerived()
            try:
                x = testObject.RowIndex
                self.fail( 'IsResultSet.RowIndex should not be usable as a getter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsResultSet.RowIndex, used as a getter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                testObject.RowIndex = None
                self.fail( 'IsResultSet.RowIndex should not be usable as a setter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsResultSet.RowIndex, used as a setter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                del testObject.RowIndex
                self.fail( 'IsResultSet.RowIndex should not be usable as a deleter.' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'IsResultSet.RowIndex, used as a deleter, should raise AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

These unit-tests are also essentially identical to the ones generated for IsQuery.

A side-note, however, on these unit-tests: Since I had stubbed out all of the interfaces, abstract classes and classes I knew of when I started writing the code and posts for the DataConnectors module, that included class- and unit-test stubs for ResultSet. As soon as the properties of IsResultSet were defined, the testPropertyCountAndTests test-method on ResultSet started failing - it now had properties that were visible through inspection, and no tests for those properties!

In order to allow the tests for the module to pass (for now), with an eye towards raising error once actual implementation in ResultSet was available, I added the following test-methods to the tests for ResultSet:

        # Test Properties

        def testCurrentRow( self ):
            """Unit-tests the CurrentRow property of the ResultSet class."""
            self.assertEquals( ResultSet.CurrentRow, IsResultSet.CurrentRow, 'Until CurrentRow is implemented in ResultSet, it should be inherited from IsResultSet.' )

        def testFieldNames( self ):
            """Unit-tests the FieldNames property of the ResultSet class."""
            self.assertEquals( ResultSet.FieldNames, IsResultSet.FieldNames, 'Until FieldNames is implemented in ResultSet, it should be inherited from IsResultSet.' )

        def testRowCount( self ):
            """Unit-tests the RowCount property of the ResultSet class."""
            self.assertEquals( ResultSet.RowCount, IsResultSet.RowCount, 'Until RowCount is implemented in ResultSet, it should be inherited from IsResultSet.' )

Since I know that there will be implementation in ResultSet, but it isn't there yet, I want these tests to pass until ResultSet has it's own implementation of those properties...

Basically, all I did here was to assert that the properties of the ResultSet class and IsResultSet interface were the same - so these will start breaking as soon as ResultSet has local implementation of those properties.

And I think that's enough for today. I'm not sure whether I'll go and implement TypedList next, or progress on to the nominal abstract classes of DataConnetors... I'll have to noodle on which makes the most sense.

The DataConnectors module (part 1), and some ruminations on interfaces

2011-11-28T04:30:00.001-08:00

Perhaps one of the reasons that I've been struggling with picking DBFSpy back up is that it has a lot of code in it that's only secondarily related to it's core purpose. That may sound kind of strange on the surface (I know I struggled with coming to the conclusion, and I'm not completely sure that it's correct, but for now, that's my story and I'm sticking to it).

Consider, for example, that ultimately, it should be something configurable (which I have accounted for with my last post), and that it has database-connectivity functionality. Neither of those is directly related to the FUSE-based-filesystem aspect of the project, though, to be fair, it couldn't do what it's intended to do without at least having the database-connection capabilities. That does not mean that the database-connectivity functionality needs to be a part of the core DBFSpy code, though. So, in the interests of both keeping the code modular, and of simplifying my future efforts on DBFSpy, I'm going to break the database-functionality out into it's own module: DataConnectors.

Code-share: dl.dropbox.com/u/1917253/site-packages/DataConnectors.py

Let's start with some design first. By my reckoning, it needs to have the ability to connect to, query against, and return results from a back-end database. Since my database-focus (for now) is centered around MySQL, I'll stop when I get to a functional MySQL connection with querying capabilities, though I want to make sure that it's extensible enough to allow use with, say, ODBC data-sources, PostgreSQL, or whatever else is technically possible. Additionally, there needs to be some sort of common mechanism to allow objects to persist their state-data as needed, and to track that their state data has changed or has been created, and thus needs to be updated or created in the database. In my mind, then, the class-structure will look something like the class diagram below.

The Class Diagram

In today's post, I'm only going to cover one of the nominal interfaces of this module. There's a substantial amount of stuff going on even there, and I also wanted to share some of my thoughts (dare I call them insights?) on the creation of explicit interfaces.

In most languages that directly support explicit interfaces, those interface definitions do not allow the specification of anything other than abstract methods. They don't allow properties to be specified, nor do they allow any sort of true functionality to be defined for any methods - everything is an abstract method-stub.

That seems pretty straightforward.

But taking a step back, a question occurs to me. Let me lead into it with some other thoughts/questions, though. First off, what purposes are served by specifying an interface? One of them, certainly, is to provide a contract, right? When a class is defined that implements a particular interface, it's promising that it will provide the functionality specified by that interface. Somewhere along the line, whether as part of a compilation process, or at run-time, a failure of a class to make good on that promise should raise some kind of error (at run-time is, perhaps, less than optimal, but can be worked around with discipline, I believe). A secondary, but related purpose is to provide what I've been calling "type-identity" in my class' doc-strings. If a class implements an interface, and assuming that the contract associated with it is enforced in any fashion, the derivation from that interface allows our code to know that any given instance is of a given type.

My question, then, is "why not allow interfaces to specify properties?" Properties, even in a language like Python, where they have no set type, and can be defined on the fly, could just as easily be part of a contract, couldn't they?

The only arguments I've been able to find thus far to justify properties not being part of interface definitions all seem to be variations on "properties represent specific implementations of data representations, and would break encapsulation." I think I'd argue that, provided that properties didn't specify anything more than that they exist, and maybe what type they are (if applicable), that I'd disagree. But maybe I'm missing something.

The reason for most of the pondering above is that as I was thinking out some of the details for the interfaces in the DataConnectors module, it occurred to me that some of my nominal interfaces could easily specify/require that specific properties would at least exist for derived classes. It would even have been possible to force specific types, perhaps, though I suspect that it would've been more trouble than it would be worth (and may not have been possible without a lot of nominal interface clutter).

As a case in point, consider the IsDataConnector nominal interface.

IsDataConnector (Nominal interface):

class IsDataConnector( object ):
    """Provides interface definition and type-identity for objects that can represent connectors to a back-end data-source (typically a relational database engine)."""

    ##################################
    # Class Attributes               #
    ##################################

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    ##################################
    # Class Properties               #
    ##################################

    Connection = property( None, None, None, None )
    Database = property( None, None, None, None )
    Host = property( None, None, None, None )
    Password = property( None, None, None, None )
    User = property( None, None, None, None )

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def __init__( self ):
        """Object constructor."""
        # Nominally abstract: Don't allow instantiation of the class
        if self.__class__ == IsDataConnector:
            raise NotImplementedError( 'IsDataConnector is (nominally) an interface, and is not intended to be instantiated.' )

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def Connect( self ):
        """Connects to the database specified by Database, residing on Host, using User and Password to connect."""
        raise NotImplementedError( '%s.Connect is not implemented as required by IsDataConnector.' % ( self.__class__.__name__ ) )

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'query', None, 'The IsQuery object that will be executed against the object\'s Connection.' )
    def Query( self, query ):
        """Executes the provided IsQuery object's query against the object's connection, returning one or more results."""
        raise NotImplementedError( '%s.Query is not implemented as required by IsDataConnector.' % ( self.__class__.__name__ ) )

__all__ += [ 'IsDataConnector' ]

Commentary

Line(s)

24-28

Here is where I've defined the "required" properties for classes implementing the interface.

The practical upshot of these is that each property has been defined, but has no implementation associated with them (they have no getter, setter or deleter methods). The more I think about this approach, the more I like it. Consider:

Each property is now defined here, but has no implementation, and will raise errors when tested. In combination with the normal unit-testing processes (from previous posts) that requires a test-method for each property, this feels pretty safe.
That effectively requires (provided that discipline is maintained at the unit-testing level) that all derived classes will have to implement the properties defined here.

I also didn't specify a documentation-string for them, though I'm toying with the idea of putting the documentation string in at this level in the object/interface tree, which is why they aren't just property() calls. This also feels like another justification for explicitly defining properties using the property() function, rather than using the decorator-structure.

24

The Connection property will contain the actual database-connector object in a derived instance.

25

Database will contain the name of the database being connected to.

26

Host will contain the name or IP-address of the host machine that the Database-database resides on.

27

Password will contain the password used to connect to the database.

28

User will contain the user-name used to connect to the database.

49-52

Requires a common method/mechanism to connect to the database that the derived object represents a connection to.

55-60

Requires a common method/mechanism to execute a query against the database.

Looking at this definition, then, and applying it to BaseDataConnector and MySQLConnector (and any other connectors that may be derived from BaseDataConector), it should hopefully be obvious that they will have:

A Connection property;
A Database property;
A Host property;
A Password property;
A User property;
A Connect method;
A Query method; and

There are no implementation details, and it's not possible to determine (yet) where these properties and methods will actually be realized. I expect that the items that aren't specific to a given database-engine - most of the properties, possibly all of them, and the AppendToQueue method - will be implemented in BaseDataConenctor. That's the lowest common denominator for actual implementations of various database-connector objects, at least, so that would make the most sense, but we'll have to see how that works out later.

I'm going to stop here for today, while this post sinks in, but before I do, here are the unit-tests for the interface.

    class IsDataConnectorDerived( IsDataConnector ):
        def __init__( self ):
            pass
    
    class testIsDataConnector( unittest.TestCase ):
        """Unit-tests the IsDataConnector class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
        def testAbstract( self ):
            """Testing abstract nature of the IsDataConnector class."""
            try:
                testObject = IsDataConnector()
                self.fail( 'IsDataConnector is a nominal interface, and should not be instantiable.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Attempting to instantiate an IsDataConnector should raise a NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            testObject = IsDataConnectorDerived()
            self.assertTrue( isinstance( testObject, IsDataConnector ), 'Objects derived from IsDataConnector should be instances of it.' )
    
        def testPropertyCountAndTests( self ):
            """Testing the properties of the IsDataConnector class."""
            items = getMemberNames( IsDataConnector )[0]
            actual = len( items )
            expected = 5
            self.assertEquals( expected, actual, 'IsDataConnector is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the IsDataConnector class."""
            items = getMemberNames( IsDataConnector )[1]
            actual = len( items )
            expected = 2
            self.assertEquals( expected, actual, 'IsDataConnector is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

        # Unit-test properties

        def testConnection( self ):
            """Unit-tests the Connection property of the IsDataConnector interface."""
            testObject = IsDataConnectorDerived()
            try:
                testObject.Connection = None
                self.fail( 'The Connection property of IsDataConnector is nominally abstract, and should not be settable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to set the Connection property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                testValue = testObject.Connection
                self.fail( 'The Connection property of IsDataConnector is nominally abstract, and should not be gettable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to get the Connection property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                del testObject.Connection
                self.fail( 'The Connection property of IsDataConnector is nominally abstract, and should not be deletable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to delete the Connection property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testDatabase( self ):
            """Unit-tests the Database property of the IsDataConnector interface."""
            testObject = IsDataConnectorDerived()
            try:
                testObject.Database = None
                self.fail( 'The Database property of IsDataConnector is nominally abstract, and should not be settable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to set the Database property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                testValue = testObject.Database
                self.fail( 'The Database property of IsDataConnector is nominally abstract, and should not be gettable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to get the Database property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                del testObject.Database
                self.fail( 'The Database property of IsDataConnector is nominally abstract, and should not be deletable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to delete the Database property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testHost( self ):
            """Unit-tests the Host property of the IsDataConnector interface."""
            testObject = IsDataConnectorDerived()
            try:
                testObject.Host = None
                self.fail( 'The Host property of IsDataConnector is nominally abstract, and should not be settable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to set the Host property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                testValue = testObject.Host
                self.fail( 'The Host property of IsDataConnector is nominally abstract, and should not be gettable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to get the Host property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                del testObject.Host
                self.fail( 'The Host property of IsDataConnector is nominally abstract, and should not be deletable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to delete the Host property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testPassword( self ):
            """Unit-tests the Password property of the IsDataConnector interface."""
            testObject = IsDataConnectorDerived()
            try:
                testObject.Password = None
                self.fail( 'The Password property of IsDataConnector is nominally abstract, and should not be settable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to set the Password property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                testValue = testObject.Password
                self.fail( 'The Password property of IsDataConnector is nominally abstract, and should not be gettable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to get the Password property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                del testObject.Password
                self.fail( 'The Password property of IsDataConnector is nominally abstract, and should not be deletable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to delete the Password property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testUser( self ):
            """Unit-tests the User property of the IsDataConnector interface."""
            testObject = IsDataConnectorDerived()
            try:
                testObject.User = None
                self.fail( 'The User property of IsDataConnector is nominally abstract, and should not be settable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to set the User property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                testValue = testObject.User
                self.fail( 'The User property of IsDataConnector is nominally abstract, and should not be gettable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to get the User property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                del testObject.User
                self.fail( 'The User property of IsDataConnector is nominally abstract, and should not be deletable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Trying to delete the User property of IsDataConnector should raise an Attribute error, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        # Unit-test methods

        def testConnect( self ):
            """Unit-tests the Connect method of the IsDataConnector interface."""
            testObject = IsDataConnectorDerived()
            try:
                testObject.Connect()
                self.fail( 'Connect should raise a NotImplementedError.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Connect should raise a NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__mame__, error ) )

        def testQuery( self ):
            """Unit-tests the Query method of the IsDataConnector interface."""
            testObject = IsDataConnectorDerived()
            try:
                testObject.Query( None )
                self.fail( 'Query should raise a NotImplementedError.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Query should raise a NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__mame__, error ) )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testIsDataConnector ) )

Not posting today...

2011-11-25T04:30:00.001-08:00

It's Thanksgiving here in the US, so I'm taking the day off.

Come back on Monday for the next installment, when I'll start looking at a data-connectors module.

The Configuration module

2011-11-23T04:30:00.000-08:00

One of the requirements for the project I mentioned that spawned the idea of the MiniAppServer project was the need to be "configurable," at least for some value of "configurable." In pondering what I've posted over the past couple of days for MiniAppServer, it occurred to me that the original "configurable" requirement would, logically, carry through here as well, and that I hadn't addressed it at all thus far.

And, whether I proceed with DBFSpy, restart it, or shelve it in favor of MiniAppServer for the time being, I'll likely want configuration capabilities.

So. Python's got at least one configuration-related module in it's default installation: ConfigParser. It's aimed at providing configuration-capabilities based on what I think of as a typical configuration-file format (I'm pretty sure I've seen it in Windows and Linux environments over the years, at least), that looks something like this:

#########################################################
# "#" indicates the beginning of a comment that         #
# runs to the end of the line                           #
#########################################################
[Section Name]                # Configuration-section name. 
                              # Spaces are allowed!
keyname:            value     # keynames can be mixed-case, 
                              # but come back as lower-case
keyname2:           value2    # values are returned as 
                              # strings by default.

Since the configuration types, section-names, values, etc., will all vary significantly from one configurable type to another, there's not a whole lot that needs to be coded: An interface and an abstract class, to my thinking, cover most of it, since the specific implementations will have to dig out and format and/or convert the sections, key-names and values. The classes and their unit-tests are pretty simple as a result.

The complete code is shared at dl.dropbox.com/u/1917253/site-packages/Template.py

Stripping the items in question down to just the classes and their unit-tests, I get:

IsConfigurable (Nominal interface):

class IsConfigurable( object ):
    """Provides interface requirements and type-identoty for objects that can retrieve 
state-data from a configuration file (at present, only ConfigParser is supported)."""

    ##################################
    # Class Attributes               #
    ##################################

    ConfigurationTypes = [ ConfigParser.ConfigParser ]

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    ##################################
    # Class Properties               #
    ##################################

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def __init__( self ):
        """Object constructor."""
        # Nominally abstract: Don't allow instantiation of the class
        if self.__class__ == IsConfigurable:
            raise NotImplementedError( 'IsConfigurable is (nominally) an interface, and is not intended to be instantiated.' )

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'configProvider', None, 'A configuration provider object, an instance of one of IsConfigurable.ConfigurationTypes' )
    @DocumentArgument( 'argument', 'configSection', None, 'String, required) The configuration-section name that the object\'s configuration-state should be retrieved from.' )
    @DocumentConfiguration( 'Section Name', 'keyname', 'value' )
    @DocumentConfiguration( 'Section Name', 'keyname2', 'value2' )
    def Configure( self, configProvider, configSection ):
        """Configures the object using configuration data from the specified section of the specified provider."""
        raise NotImplementedError( '%s.Configure has not been implemented' % ( self.__class__.__name__ ) )

__all__ += [ 'IsConfigurable' ]

Note the use of the @DocumentConfiguration decorator on lines 49 and 50. Documentation of this sort of configuration information is what that decorator was designed for, and will provide documentation that looks like this:

Configure(self, configProvider, configSection)
    Configures the object using configuration data from the specified 
    section of the specified provider.
     
    ###################
    ## Arguments     ##
    ###################
     + self ................ The object-instance that the method is called against. 
     + configProvider ...... A configuration provider object, an instance of one of 
                             IsConfigurable.ConfigurationTypes 
     + configSection ....... String, required) The configuration-section name that 
                             the object's configuration-state should be retrieved 
                             from. 
     
    ###################
    ## Configuration ##
    ###################
     
    The relevant configuration-file section(s) and values(s) are:
    [Section Name]
    keyname ..................... value 
    keyname2 .................... value2

The relevant unit-tests:

    class IsConfigurableDerived( IsConfigurable ):
        def __init__( self ):
            IsConfigurable.__init__( self )
    
    class testIsConfigurable( unittest.TestCase ):
        """Unit-tests the IsConfigurable class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
        def testDerived( self ):
            """Testing abstract nature of the IsConfigurable class."""
            try:
                testObject = IsConfigurable()
                self.fail( 'IsConfigurable is nominally an interface, and should not be instantiable.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Instantiating IsConfigurable should raise a NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            testObject = IsConfigurableDerived()
            self.assertTrue( isinstance( testObject, IsConfigurable ), 'Objects implementing IsConfigurable should be instances of IsConfigurable.' )
    
        def testPropertyCountAndTests( self ):
            """Testing the properties of the IsConfigurable class."""
            items = getMemberNames( IsConfigurable )[0]
            actual = len( items )
            expected = 0
            self.assertEquals( expected, actual, 'IsConfigurable is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the IsConfigurable class."""
            items = getMemberNames( IsConfigurable )[1]
            actual = len( items )
            expected = 1
            self.assertEquals( expected, actual, 'IsConfigurable is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

        # Unit-test properties

        # Unit-test methods

        def testConfigure( self ):
            """Unit-tests the Configure method of the IsConfigurable class."""
            testObject = IsConfigurableDerived()
            try:
                testObject.Configure( ConfigParser.ConfigParser(), 'configSection' )
                self.fail( 'An IsConfigurable instance that has not overridden Configure should raise a NotImplementedError when it is called.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'An IsConfigurable instance that has not overridden Configure should raise a NotImplementedError when it is called, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testIsConfigurable ) )

HasConfiguration (Nominal abstract class):

class HasConfiguration( object ):
    """Provides a configuration property (allowing an instance of any of the types in 
IsConfigurable.ConfigurationTypes)."""

    ##################################
    # Class Attributes               #
    ##################################

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetConfiguration( self ):
        """Gets the object's associated configuration object."""
        return self._configuration

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, '(Any of %s, required) The configuration-provider object that will contain the object\'s configuration-state data.' % ( IsConfigurable.ConfigurationTypes ) )
    def _SetConfiguration( self, value ):
        """Sets the object's associated configuration object."""
        if not value.__class__ in IsConfigurable.ConfigurationTypes:
            raise TypeError( '%s.Configuration expects an instance of one of %s' % ( self.__class__.__name__, IsConfigurable.ConfigurationTypes ) )
        self._configuration = value

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _DelConfiguration( self ):
        """Deletes the object's associated configuration object."""
        self._configuration = None

    ##################################
    # Class Properties               #
    ##################################

    Configuration = property ( _GetConfiguration, None, None, 'Gets the object\'s associated configuration object.' )

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @ToDo( 'Document class.' )
    def __init__( self ):
        """Object constructor."""
        # Nominally abstract: Don't allow instantiation of the class
        if self.__class__ == HasConfiguration:
            raise NotImplementedError( 'HasConfiguration is (nominally) an abstract class, and is not intended to be instantiated.' )
        # Set default property value(s)
        self._DelConfiguration()

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

__all__ += [ 'HasConfiguration' ]

The relevant unit-tests:

    class HasConfigurationDerived( HasConfiguration ):
        def __init__( self ):
            HasConfiguration.__init__( self )
    
    class testHasConfiguration( unittest.TestCase ):
        """Unit-tests the HasConfiguration class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
        def testDerived( self ):
            """Testing abstract nature of the HasConfiguration class."""
            try:
                testObject = HasConfiguration()
                self.fail( 'HasConfiguration is nominally an abstract class, and should not be instantiable.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Instantiating HasConfiguration should raise a NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            testObject = HasConfigurationDerived()
            self.assertTrue( isinstance( testObject, HasConfiguration ), 'Objects implementing HasConfiguration should be instances of HasConfiguration.' )
            self.assertEquals( testObject.Configuration, None, 'A default instance of HasConfiguration should have a None value for it\'s Configuration property.' )
    
        def testConstruction( self ):
            """Testing construction of the HasConfiguration class."""
            pass
    
        def testPropertyCountAndTests( self ):
            """Testing the properties of the HasConfiguration class."""
            items = getMemberNames( HasConfiguration )[0]
            actual = len( items )
            expected = 1
            self.assertEquals( expected, actual, 'HasConfiguration is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the HasConfiguration class."""
            items = getMemberNames( HasConfiguration )[1]
            actual = len( items )
            expected = 0
            self.assertEquals( expected, actual, 'HasConfiguration is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

        # Unit-test properties

        def testConfiguration( self ):
            """Unit-tests the Configuration property of the HasConfiguration class."""
            testObject = HasConfigurationDerived()
            goodValues = IsConfigurable.ConfigurationTypes
            badValues = [ None, 'string', 1, 1L, 1.0, True ]
            for testClass in goodValues:
                testValue = (testClass)()
                testObject._SetConfiguration( testValue )
                self.assertEquals( testObject.Configuration, testValue, 'The property set (%s) should be retrievable by a get.' % ( testValue ) )
            for testValue in badValues:
                try:
                    testObject._SetConfiguration( testValue )
                    self.fail( 'Setting the configuration property to an item that is not an instance of %s (%s) should raise a TypeError' % ( IsConfigurable.ConfigurationTypes, testValue ) )
                except TypeError:
                    pass
                except Exception, error:
                    self.fail( 'Setting the configuration property to an item that is not an instance of %s (%s) should raise a TypeError, but %s was raised instead:\n  %s' % ( IsConfigurable.ConfigurationTypes, testValue, error.__class__.__name__, error ) )

        # Unit-test methods

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testHasConfiguration ) )

Design for the MiniAppServer idea

2011-11-21T04:30:00.000-08:00

The design for MiniAppServer feels like it ought to be fairly simple, given what I'm thinking is a typical development-case:

Create a MiniAppServer instance, with various properties, including:
- Host name;
- Port;
- A document-root, if desired;
- A top-level request-routing mechanism (something that can handle application-ish endpoints and/or static-file requests/responses);
- Any application-ish endpoint-handlers needed;
- Any error-handling endpoint-handlers needed;
If necessary, add/register any non-standard endpoint-handlers;
If necessary, add/register any standard handlers for errors, etc.;
Call MiniAppServer.serve_forever()

When a request is recieved by the MiniAppServer instance, it gets handled as follows:

The request (plus it's socket and whatever other information/objects are needed by a BaseRequestHandler instance) are passed to the application's top-level routing object.
That routing object examines the request made, looking for a match based on:
1. Registered application-ish endpoints;
2. Static-file endpoints;
3. Other endpoints? I can't think of any that make sense, but I can't rule out the possibility...
If a match is found, the request (and everything associated) is passed to a new instance of the appropriate endpoint class.
The endpoint-handler does whatever it needs to do, generating whatever output it needs to generate, and returns that response.
- If an error is encountered, then the request (etc.) should be passed to the registered error-handling endpoint-handler, if there is one;
- If there is not a specific error-handling endpoint, a simple, raw response should be returned with the appropriate HTTP code);

Given all of this, what I'm expecting (at least as of this writing) is an object-structure something like so:

IsApplication (nominal interface): Provides interface requirements and type-identioty for objects that can provide MiniAppServer-like functionality. This is specifically called out becuse there would otherwise be circular references between MiniAppServer and HasApplication.
HasApplication (nominal abstract class): Provides an Application property for derived objects, which should be an instance implementing IsApplication.
IsEndpoint (nominal abstract class): Provides baseline functionality, interface requirements and type-identity for objects that can act as endpoints for a MiniAppServer instance. Endpoint definitions should be derived from this and SocketServer.BaseRequestHandler, though I'm not sure whether I want to make IsEndpoint a subclass of SocketServer.BaseRequestHandler directly, or just have it check to make sure that instances derived from it are also derived from SocketServer.BaseRequestHandler.
BaseRouter (nominal abstract class), or
Router (class, extensible): Ultimately, I'm expecting that a running MiniAppServer instance will require a router-object: something that the recieved requests are handed off to, that then spins off an appropriate IsEndpoint-derived instance to handle the request and return results to the client. Whether this should be a class in and of itself, or a nominal abstract class that developers should derive their own classes from, I'm not sure yet. There are potential advantages to both, and I'm very much on the fence as to which way to go with it.
ErrorEndpoint (class, extensible, derives from IsEndpoint and HasApplication): An IsEndpoint-derived class that will be handed control over an incoming request when/if an error occurs.
FileEndpoint (class, final, derives from IsEndpoint and HasApplication): An IsEndpoint-derived class that will handle looking up static files for responses to requests that don't get caught by a functional IsEndpoint instance first.
MiniAppServer (class, extensible, implements IsApplication): The class that's actually responsible for generating and starting a server instance.

Working my way from the top of this list down seems a rational approach, at least for now...

Requirements for the MiniAppServer idea

2011-11-18T04:30:00.000-08:00

So, while I'm trying to re-familiarize myself with the DBFSpy stack, I'm gonna capture some of the requirements ideas for the free-standing, Python-based mini-web-app-server idea that I mentioned in my last post. That entire string (free-standing, Python-based mini-web-app-server) is a bit much to type over and over again, it doesn't lend itself well to a meaningful acronym or abbreviation, and all of the logical names I can think of are already taken (Servelets, Applets, etc.), so I'm going to call it MiniAppServer, at least for now...

I mentioned that the idea came to me while I was working on a Python project at work. Without going into too much detail on that project (since I'm not sure if I can, legally), it basically boiled down to a free-standing Python-based service that would periodically query information from a remote logging service, potentially perform some sort of examination of the data returned, and respond if certain criteria were met (sending an email if our 404-count exceeded some threshold, for example).

One of the requirements for that project was that it be "configurable." There wasn't much in the way of definition for what "configurable" meant, so I started with the default of using the ConfigParser module's offerings to retrieve configuration information from a fairly typical configuration-file. While on the bus back home one night, it suddently occurred to me that if configuration-changes needed to be made, that would entail stopping the service, changing the configuration, and re-starting it. Admittedly, it might not happen in that order; given the shape of the code I came up with, altering the config-file, then restarting the service would have worked as well. But that started me thinking "what if there were a way to have the service also generate a web-based configuration-interface. Something that would allow IT staff to simply hit some URL and make changes to the configured values in real-time.

That was the impetus for MiniAppServer.

Whether the original idea for it's use becomes valid or not (the project at work was effectively cancelled, and I mothballed all of the code for it, following my previous recommendations as much as I could), I can certainly come up with some uses for such a thing for my own ends, so I'm adding it to my stack of projects.

So, what are MiniAppServer's requirements?

It needs to be able to run on any machine (with a full Python installation being the only requirement);
URLs should be (for anyting that isn't a static-file request/response) something along the lines of http://host:port/ObjectType/Action[/Format], where:
ObjectType

is the name of an object-type (perhaps a class, perhaps not)

Action
is some sort of action to be taken with the object-type. Some likely candidates include:
- List - show a list of available items of the applicable object-type, probably as links to a View for the individual items;
- View - Show a detailed view of a specific instance of an object-type;
- Edit or Update - make and submit changes to one instance of a given object-type;
- Create or New - create a new instance of an object-type (which may or may not be substantially different that the Edit form/view);
- Delete - Delete a single instance of an object-type
Format

is (maybe) a format to return - XML, JSON, or whatever else. I cannot come up with a real use for this that justifies it to my satisfaction, and perhaps allowing a query-string (e.g., ?format=xml) instead would be better. I'll noodle on this for a while, I think;
Some examples might include (for a User object-type):
- http://localhost:8192/User/List
- http://localhost:8192/User/View (this would need some sort of ID to be passed, maybe as an additional path element [.../View/USERID] or as a query-string)
- http://localhost:8192/User/Edit (same comment as above)
- http://localhost:8192/User/Create
- http://localhost:8192/User/Delete
It has to handle GET and POST requests, in order to support complete HTML form-interaction;
- It needs to be able to handle "simple" POST submissions (e.g., text-only);
- It will likely need to be able to handle binary upload-capable POST submissions as well, though that may not be an immediate requirement.
It needs to be as small as possible - there's no point in having a whole lot of overhead for an application-server of this sort when a LAMP stack could just as easily be built with mod-python;
At the same time, it needs to be highly extensible;
It must be able to serve static files as well (for things like images, stylesheets and JavaScript);
It should be something that could be set up to run as a service on a machine, or used as a local instance of an application.

Some "nice-to-have" features include:

It should support as much of a Model-view-controller architecture/structure as is feasible;
It would be neat if the entire application-site (if there are any non-Python files associated, at least) could be packaged up as a single zip-file or tarball;

How to mothball a project

2011-11-16T04:30:00.000-08:00

Part of the reason that the last several posts I've made since I returned from my hiatus have not been obviously related to the DBFSpy project that I started this blog with is that in the intervening couple of months, I've forgotten a lot of what I'd intended to do with it.

Fortunately, I had documented at least a sequence of classes, interfaces, etc., that I'd expected to cover, so it's not like I'm completely lost. At the same time, though, I hadn't gotten even as far as stubbing out properties and methods for the remaining items before things got out of control, and I had no time to continue posting.

When projects are in danger of needing to be mothballed, I like to try and spend as much time as I can get away with making sure that I capture as much of my thought-process as I can before I have to set it aside. I don't always get as much of this kind of time as I'd like, though, even when the project in question is my own.

Case in point: All I really have on DBFSpy is a list of classes and interfaces I was expecting, and some indication that they were nominally complete (though maybe not with full unit-tests, etc.):

BaseDatabaseConnection (abstract class, complete)
BaseDataObject (abstract class, complete)
BaseFileSystemItemCache (abstract class, complete)
IFilesystem (interface, complete)
IFilesystemItem (interface)
IFileystemOperators (interface)
MySQLDatasource (class)
MySQLFileItem (class)
MySQLFilesystem (class)
Unit-tests for classes, etc.
dbfspy (executable)

Looking at older posts here, it looks like I was about on track with this list. I also managed to create and post the UML diagram, which I hope will refresh my memory somewhat as I start digging into that project again. Though now I suspect I'll need to review my progress (it looks like I had to break off after IFilesystem, bu before IFilesystemItem). With some luck, given my new insights into testing, and the discovery of properties and methods that it facilitates, I'll be able to recapture a lot of my previous momentum, and bang things out quickly. If not, I guess I'll have to think about either setting it aside again (with the expectation that it'll be so full of holes by the time I pick it up again that I'll be starting from scratch), or just biting that bullet now, and re-starting.

So, chalk this up to another lesson learned - take the time to document and comment as you go. You never know when you may have to set a project aside, and if there isn't time to gracefully (and thoroughly) mothball it, it may very well be full of holes by the time you get back to it.

What I think I'd advocate is (at the very least):

Make sure all decisions are documented;
If possible, make sure that any class-diagrams are complete as of the time the project had to be stopped;
Make sure that the design is thorough, documented, and annotated as needed to keep track of what the design intended to achieve, and what modifications had to be made, if any.

There may be other items as well, but these are, I hope, the major items...

Another part of the reason I spent the last few posts going over new stuff (apart from my enthusiasm for the topic therein), was that as I was working through a Python project at work, it occurred to me that it might be a cool idea to generate a small, freestanding, Python-based web-application server framework. Nothing huge, you understand, but rather something that could be incorporated into other Python projects, to allow for things like administrative interfaces for services and the like.

More on that topic later, I think...

The TypedContainers Module

2011-11-14T04:30:00.000-08:00

Code-sharing: dl.dropbox.com/u/1917253/site-packages/TypedContainers.py

One of the other items I came up with during my month-or-so-long hiatus from blogging about My Brain on Python was spurred by a perceived need to have some container objects that were, for lack of a better term, strongly typed as far as it's values were concerned. Python (relying on duck-typing, as previously noted) doesn't provide such things out of the box, but since it's a very capable dynamic language, it proved to be fairly simple to implement these.

My initial item in this realm was a strongly-typed Dictionary equivalent - something that would allow a developer to define a dictionary-like data-structure that would allow only items that were of a give type or one of a group of types to be added to it. It's certainly possible to perform that kind of type-checking through a class-method before adding an item to a dictionary property, but if the dictionary itself is publicly accessible, there's no good way that I can see to prevent code from being written that would skip any such type-checking, and, for example, add a string member to a dictionary that was intended to contain only numbers.

A Python purist might very well (and with some justification, perhaps) say that this isn't really an issue - in the case of an error arising at runtime, an exception would be raised, and the maintainer of the code would have to track that down and fix it. I maintain that, while that is certainly true enough, it'd be easier to track the source of the error down if the simple act of trying to insert the errant value into the container caused the error, rather than waiting until it caused some other issue later in the execution of the code. Maybe a lot later.

So, a TypedDictionary. At first, it was free-standing, but as I thought through the potential for needing similar strongly-typed containers based off of lists or tuples, or whatever else might arise, I decided to abstract some of the functionality out into a nominal abstract class. The results (minus the stub-code for TypedList and TypedTuple, which I haven't dug into yet) look like this:

# TypedCollections.py
"""Provides strongly-typed extensions to Python collection-types (lists, dictionaries, tuples, etc.)."""

__author__  = 'Brian D. Allbee'
__licence__ = 'http://creativecommons.org/licenses/by-sa/3.0/'
__version__ = '0.2'
__copyright__ = "Copyright 2011, Brian D. Allbee"
__credits__ = ["Brian D. Allbee"]
__maintainer__ = "Brian D. Allbee"
__email__ = "brian.allbee@gmail.com"
__status__ = "Development"

#####################################
# Build an "__all__" list to        #
# support                           #
# "from Namespace.blah import *"    #
# syntax                            #
#####################################
__all__ = []

#####################################
# Required imports                  #
#####################################

from DocMeta import *
import types

#####################################
# Constants defined in the module   #
#####################################

AllTypes = []
for typeName in dir(types):
    if not typeName in [ 'ClassType', 'TypeType', 'InstanceType', 'ObjectType' ] and not typeName[0] == '_':
        AllTypes.append( eval( 'types.%s' % ( typeName ) ) )

__all__ += [ 'AllTypes' ]

#####################################
# Functions defined in the module   #
#####################################

#####################################
# Interfaces defined in the module  #
#####################################

#####################################
# Abstract Classes defined in the   #
# module                            #
#####################################

class BaseTypedCollection( object ):
    """Provides baseline functionality, interface requirements, and type-identity for derived items that can have a MemberTypes property."""

    ##################################
    # Class Attributes               #
    ##################################

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _GetMemberTypes( self ):
        """Gets the set of object-types allowed as members in the collection."""
        return self._memberTypes

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, '(Iterable, required) An iterable collection of items to replace the current member-types with' )
    def _SetMemberTypes( self, value ):
        """Sets the set of object-types allowed as members in the collection."""
        self._memberTypes = []
        for item in value:
            # Basic types, excluding ClassType, TypeType, etc.
            if item in AllTypes:
                self._memberTypes.append( item )
            # Classic classes
            elif type( item ) == types.ClassType:
                raise ValueError( '%s.MemberTypes cannot accept classic classes (%s) as member-types' % ( self.__class__.__name__, item ) )
            # New-style classes
            elif type( item ) == types.TypeType:
                self._memberTypes.append( item )
            # Unrecognized!
            else:
                raise ValueError( '%s.MemberTypes cannot accept %s as a member-type' % ( self.__class__.__name__, item ) )

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def _DelMemberTypes( self ):
        """Deletes the set of object-types allowed as members in the collection."""
        self._memberTypes = []

    ##################################
    # Class Properties               #
    ##################################

    MemberTypes = property( _GetMemberTypes, None, None, 'Gets the set of object-types allowed as members in the collection.' )

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def __init__( self ):
        """Object constructor."""
        # Nominally abstract: Don't allow instantiation of the class
        if self.__class__ == BaseTypedCollection:
            raise NotImplementedError( 'BaseTypedCollection is (nominally) an abstract class, and is not intended to be instantiated.' )
        self._DelMemberTypes()

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'value', None, 'The value to check.' )
    def IsValidMemberType( self, value ):
        """Determines whether the supplied value is a valid type to be allowed in the collection-object."""
        if hasattr( value.__class__, '__mro__' ):
            compareTypes = set( value.__class__.__mro__ )
            return ( len( set( self._memberTypes ).intersection( compareTypes ) ) > 0 )
        if hasattr( value.__class__, '__bases__' ):
            compareTypes = set( value.__class__.__bases__ )
            return ( len( set( self._memberTypes ).intersection( compareTypes ) ) > 0 )
        raise TypeError( '%s.IsValidMemberType could not determine either an MRO for, or the bases of the %s value' % ( self.__class__.__name__, value ) )

__all__ += [ 'BaseTypedCollection' ]

#####################################
# Classes defined in the module     #
#####################################

class TypedDictionary( BaseTypedCollection, dict ):
    """Represents a strongly-typed Dictionary."""

    #################################
    # Class Attributes              #
    #################################

    #################################
    # Class Property-Getter Methods #
    #################################

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    ##################################
    # Class Properties               #
    ##################################

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'memberTypes', None, '(Iterable, required) An iterable collection of item-types to be allowed as members in the dictionary.' )
    def __init__( self, memberTypes ):
        """Object constructor."""
        # Nominally final: Don't allow any class other than this one
        if self.__class__ != TypedDictionary:
            raise NotImplementedError( 'TypedDictionary is (nominally) a final class, and is not intended to be derived from.' )
        BaseTypedCollection.__init__( self )
        self._SetMemberTypes( memberTypes )

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    @DocumentArgument( 'argument', 'name', None, 'The key-name for the element once it\'s added to the dictionary.' )
    @DocumentArgument( 'argument', 'value', None, 'The element-name to add to the dictionary.' )
    def __setitem__( self, name, value ):
        """Wraps the default __setitem__ in order to type-check the value before allowing it to be added."""
        if not self.IsValidMemberType( value ):
            raise TypeError( 'TypedDictionary instance expects an instance of any of %s.' % ( self._memberTypes ) )
        dict.__setitem__( self, name, value )

__all__ += [ 'TypedDictionary' ]

#####################################
# Package sub-modules and -packages #
#####################################

#####################################
# Unit-test the module on main      #
#####################################

if __name__ == '__main__':
    import inspect, os, sys, unittest
    from UnitTestUtilities import *

    testSuite = unittest.TestSuite()
    testResults = unittest.TestResult()

    ##################################
    # Unit-test Constants            #
    ##################################

    class testConstants( unittest.TestCase ):
        """Unit-tests the constants defined in the module."""
        pass
    
    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testConstants ) )

    ##################################
    # Unit-test Functions            #
    ##################################

    ##################################
    # Unit-test Interfaces           #
    ##################################

    ##################################
    # Unit-test Abstract Classes     #
    ##################################

    class BaseTypedCollectionDerived( BaseTypedCollection ):
        def __init__( self ):
            BaseTypedCollection.__init__( self )

    class AClassicMember: pass

    class AnotherClassicMember: pass

    class ADerivedClassicMember( AClassicMember ): pass

    class ATypedMember( object ): pass

    class AnotherTypedMember( object ): pass
    
    class ADerivedTypedMember( ATypedMember, object ): pass

    class testBaseTypedCollection( unittest.TestCase ):
        """Unit-tests the BaseTypedCollection class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
        def testAbstract( self ):
            """Testing abstract nature of the BaseTypedCollection class."""
            try:
                testObject = BaseTypedCollection()
                self.fail( 'BaseTypedCollection is nominally an abstract class and should not allow instantiation.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Attempting to instantiate a BaseTypedCollection should raise a NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )

        def testConstruction( self ):
            """Testing construction of the BaseTypedCollection class."""
            # Single type
            testObject = BaseTypedCollectionDerived()
            self.assertTrue( isinstance( testObject, BaseTypedCollection ), 'Constructed BaseTypedCollection-derived objects should be instances of BaseTypedCollection.' )

        def testSimpleTypes( self ):
            """Testing the use of simple value-types as member-types."""
            testTypesList = [
                [ types.StringType, ],
                [ types.IntType, types.FloatType, types.LongType ],
                ]
            for testTypes in testTypesList:
                testObject = BaseTypedCollectionDerived()
                testObject._SetMemberTypes( testTypes )
                self.assertEquals( testObject.MemberTypes, testTypes, 'Getting member-types %s should return the types set %s' % ( testObject.MemberTypes, testTypes ) )

        def testSimpleInstances( self ):
            """Testing the use of simple value-types as member-types."""
            testTypesList = [
                [ 'ook', ],
                [ 1, 1L, 1.0 ],
                ]
            for testTypes in testTypesList:
                testObject = BaseTypedCollectionDerived()
                try:
                    testObject._SetMemberTypes( testTypes )
                    self.fail( 'Simple values %s should not be allowed as member-types' % ( testTypes ) )
                except ValueError:
                    pass
                except Exception, error:
                    self.fail( 'Attempting to use %s should raise a ValueError, but %s was raised instead:\n  %s' % ( testTypes, error.__class__.__name__, error ) )

        def testClassicClassTypes( self ):
            """Testing the use of classic classes as member-types."""
            testTypesList = [
                [ AClassicMember, ],
                [ AClassicMember, AnotherClassicMember ],
                ]
            for testTypes in testTypesList:
                testObject = BaseTypedCollectionDerived()
                try:
                    testObject._SetMemberTypes( testTypes )
                    self.fail( 'Classic classes %s should not be allowed as member-types' % ( testTypes ) )
                except ValueError:
                    pass
                except Exception, error:
                    self.fail( 'Attempting to use %s should raise a ValueError, but %s was raised instead:\n  %s' % ( testTypes, error.__class__.__name__, error ) )

        def testClassicClassInstances( self ):
            """Testing the use of classic classes as member-types."""
            testTypesList = [
                [ AClassicMember(), ],
                [ AClassicMember(), AnotherClassicMember() ],
                ]
            for testTypes in testTypesList:
                testObject = BaseTypedCollectionDerived()
                try:
                    testObject._SetMemberTypes( testTypes )
                    self.fail( 'Simple values %s should not be allowed as member-types' % ( testTypes ) )
                except ValueError:
                    pass
                except Exception, error:
                    self.fail( 'Attempting to use %s should raise a ValueError, but %s was raised instead:\n  %s' % ( testTypes, error.__class__.__name__, error ) )

        def testNewClassTypes( self ):
            """Testing the use of classic classes as member-types."""
            testTypesList = [
                [ ATypedMember, ],
                [ ATypedMember, AnotherTypedMember ],
                ]
            for testTypes in testTypesList:
                testObject = BaseTypedCollectionDerived()
                testObject._SetMemberTypes( testTypes )
                self.assertEquals( testObject.MemberTypes, testTypes, 'Getting member-types %s should return the types set %s' % ( testObject.MemberTypes, testTypes ) )

        def testNewClassInstances( self ):
            """Testing the use of classic classes as member-types."""
            testTypesList = [
                [ ATypedMember(), ],
                [ ATypedMember(), AnotherTypedMember() ],
                ]
            for testTypes in testTypesList:
                testObject = BaseTypedCollectionDerived()
                try:
                    testObject._SetMemberTypes( testTypes )
                    self.fail( 'Instance-values %s should not be allowed as member-types' % ( testTypes ) )
                except ValueError:
                    pass
                except Exception, error:
                    self.fail( 'Attempting to use %s should raise a ValueError, but %s was raised instead:\n  %s' % ( testTypes, error.__class__.__name__, error ) )

        def testProperties( self ):
            """Testing the properties of the BaseTypedCollection class."""
            testObject = BaseTypedCollectionDerived()
            # MemberTypes (get)
            self.assertEquals( testObject.MemberTypes, [], 'The default MemberTypes value should be an empty list.' )
            try:
                testObject.MemberTypes = None
                self.fail( 'The abstract property MemberTypes should not be settable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Attempting to set the MemberTypes property should raise a AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
            try:
                del testObject.MemberTypes
                self.fail( 'The abstract property MemberTypes should not be deletable' )
            except AttributeError:
                pass
            except Exception, error:
                self.fail( 'Attempting to delete the MemberTypes property should raise a AttributeError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
    
        def testMethods( self ):
            """Testing the methods of the BaseTypedCollection class."""
            testObject = BaseTypedCollectionDerived()
            # IsValidMemberType
            # - Simple type
            testObject._SetMemberTypes( [ int ] )
            goodValues = [ -1, 0, 1, 2 ]
            for value in goodValues:
                self.assertTrue( testObject.IsValidMemberType( value ), 'The value %s should be a good value.' % ( value ) )
            badValues = [ None, 'ook', 1L, 1.0 ]
            for value in badValues:
                self.assertFalse( testObject.IsValidMemberType( value ), 'The value %s should be a bad/invalid value.' % ( value ) )
            # - Multiple simple types
            testObject._SetMemberTypes( [ int, float, long ] )
            goodValues = [ -1, 0, 1, 2, -1.0, 0.0, 1.0, 2.0, -1L, 0L, 1L, 2L ]
            for value in goodValues:
                self.assertTrue( testObject.IsValidMemberType( value ), 'The value %s (%s) should be a good value.' % ( value, type( value ) ) )
            badValues = [ None, 'ook', list(), dict(), tuple() ]
            for value in badValues:
                self.assertFalse( testObject.IsValidMemberType( value ), 'The value %s (%s) should be a bad/invalid value.' % ( value, type( value ) ) )
            # - New-style class type
            testObject._SetMemberTypes( [ ATypedMember ] )
            goodValues = [ ATypedMember(), ADerivedTypedMember() ]
            for value in goodValues:
                self.assertTrue( testObject.IsValidMemberType( value ), 'The value %s (%s) should be a good value.' % ( value, type( value ) ) )
            badValues = [ None, 'ook', 1, list(), tuple(), dict(), AClassicMember(), AnotherTypedMember() ]
            for value in badValues:
                self.assertFalse( testObject.IsValidMemberType( value ), 'The value %s (%s) should be a bad/invalid value.' % ( value, type( value ) ) )
            # - Multiple new-style class type
            testObject._SetMemberTypes( [ ATypedMember, AnotherTypedMember ] )
            goodValues = [ ATypedMember(), ADerivedTypedMember(), AnotherTypedMember() ]
            for value in goodValues:
                self.assertTrue( testObject.IsValidMemberType( value ), 'The value %s (%s) should be a good value.' % ( value, type( value ) ) )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testBaseTypedCollection ) )

    ##################################
    # Unit-test Classes              #
    ##################################

    class TypedDictionaryDerived( TypedDictionary ):
        def __init__( self, memberTypes ):
            TypedDictionary.__init__( self, memberTypes )
    
    class testTypedDictionary( unittest.TestCase ):
        """Unit-tests the TypedDictionary class."""
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
        def testFinal( self ):
            """Testing final nature of the TypedDictionary class."""
            try:
                testObject = TypedDictionaryDerived( [ types.StringType, ] )
                self.fail( 'TypedDictionary is nominally a final class and should not allow derivation.' )
            except NotImplementedError:
                pass
            except Exception, error:
                self.fail( 'Attempting to derive from TypedDictionary should raise a NotImplementedError, but %s was raised instead:\n  %s' % ( error.__class__.__name__, error ) )
    
        def testConstruction( self ):
            """Testing construction of the TypedDictionary class."""
            testObject = TypedDictionary( [ int, long ] )
            self.assertTrue( isinstance( testObject, TypedDictionary ), 'Tautological imperative: A TypedDictionary is a TypedDictionary...!' )
    
        def testDictionary( self ):
            """Testing the dictionary capabilities of the TypedDictionary class."""
            # PropertyName
            testObject = TypedDictionary( [ int, long ] )
            goodNames = [ 'ook', '1', 1, '', None, TypedDictionaryDerived ]
            goodValues = [ -1, 0, 1, 2, -1L, 0L, 1L, 2L ]
            for name in goodNames:
                for value in goodValues:
                    testObject[ name ] = value
                    self.assertEquals( testObject[ name ], value, 'The value set (%s) should be returned at the name (%s)' % ( value, name ) )
            badValues = [ None, -1.0, 0.0, 1.0, 2.0, 'ook' ]
            for name in goodNames:
                for value in badValues:
                    try:
                        testObject[ name ] = value
                        self.fail( 'The value %s should not be allowed in the dictionary' % ( value ) )
                    except TypeError:
                        pass
                    except Exception, error:
                        self.fail( 'Setting an invalid value (%s) in a TypedDictionary should raise a TypeError, but %s was raised.\n  %s' % ( value, error.__class__.__name__, error ) )
    
        def testProperties( self ):
            """Testing the properties of the TypedDictionary class."""
            # No local properties to test (see BaseTypedCollection)
            pass
    
        def testMethods( self ):
            """Testing the methods of the TypedDictionary class."""
            # No local methods to test (see BaseTypedCollection)
            pass

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testTypedDictionary ) )

    # Run unit-tests and install
    print '#'*79
    print 'Running unit-tests'
    testSuite.run( testResults )
    print '#'*79
    if testResults.errors == [] and testResults.failures == []:
        print 'All unit-tests passed! Ready for build/copy!'
        import shutil
        sitePackagesDirs = []
        for path in sys.path:
            if path.split( os.sep )[-1].lower() == 'site-packages':
                sitePackagesDirs.append( path )
        for path in sitePackagesDirs:
            destinationFile = path + os.sep + __file__
            try:
                shutil.copyfile( './%s' % __file__, destinationFile )
                print 'Copied %s to %s' % ( __file__, path )
            except:
                print 'Could not copy file %s to location %s' % ( __file__, path )
    else:
        print 'Unit-tests failed (see items below)'
        print '#'*79
        if testResults.errors:
            print 'Errors'
            print '#' + '-'*77 + '#'
            for error in testResults.errors:
                print error[1]
                print '#' + '-'*77 + '#'
            print
        if testResults.failures:
            if testResults.errors:
                print '#'*79
            print 'Failures'
            print '#' + '-'*77 + '#'
            for failure in testResults.failures:
                print failure[1]
                print '#' + '-'*77 + '#'
            print
    print '#'*79

Commentary:

Line(s)

26, 32-37

I'm using the "public" members of the types module to generate a list of all of the available public types, except those that can be defined/re-defined by the developer (e.g., ClassType and TypeType, for classic- and new-style-classes, respectively, plus InstanceType and ObjectType so that we don't later try to use instances of either class-type as member-types). This code feels a bit strange to me, now that I look at it again, and though I know I had a reason for this approach, I cannot, for the life of me, remember what that reason was now. I'll have to ponder on it, but at the very least, it's functional.

52-138

The nominal abstract class that TypedDictionary and (eventually) the other typed containers are based on. It's primary purpose is to provide a consistent MemberTypes property for all of the derived container-classes it'll be a subclass of, but it also provides a mechanism (IsValidmemberType) to detect whether a given item is legitimately a member of the container.

Line(s)
63-66, 72-89, 95-98, 104: Property getter, setter, and deleter method definitions, and the formal property declaration for the MemberTypes property.; Note that _SetMemberTypes checks for basic types (lines 79-80), classic-class types (82-83), which aren't allowed at present because I haven't found a reliable way to detect whether their classes are subclasses of other (classic) classes, and new-style classes (85-86). Although I cannot imagine that there would be any other types that would come up (other than the instance types noted above), I've assured that the code will raise an error (88-89) if such a case occurs.
126-136: The mechanism mentioned for determining whether an item is a valid member of the container, by checking against it's defined member-types using Python's set data-types.

144-198

The actual TypedDictionary definition. There really isn't much to it, apart from the constructor and the overload of __setitem__:

For hopefully-obvious reasons, the constructor requires that an iterable argument of allowed member-types is supplied. It gets passed directly to the inherited _SetMemberTypes method we've already seen, which is not exposed publicly.

__setitem__ appeared to be the only method of the base dict class that needed to be overridden to type-ceck the value to be added before actually adding it. Rather than muck about with re-writing the base method, it's simply called after the valid-member-type check passes.

Since the whole point of the class is to raise a type error when an element is added that isn't of a valid type, that gets raised as well.

The unit-tests may look a bit convoluted (they felt that way to me when I was writing them, to be sure). Ultimately, all they are trying to do, though, is to assure that the member-type checks are tested with all of the logical permutations:

Simple value-types (singly or multiply);
New-style class-types (both directly or as subclasses of member-types); and
Classic-class types (which should, as noted above, all fail at this time).

Bearing that in mind, I think they're actually pretty straightforward...

The DocMeta Module

2011-11-11T04:30:00.000-08:00

Shared-code location: dl.dropbox.com/u/1917253/site-packages/DocMeta.py

Back on Monday, I promised I'd share my UnitTestUtilities and DocMeta modules. Hopefully you've already seen UnitTestUtilities, since I posted that on Wednesday. Today's foray is into DocMeta.

I wrote DocMeta quite some time back, in order to provide a simple, annotation-based (e.g., decorator-based) way of adding additional/more specific documentation-information to functions and methods. You've already seen it in use in UnitTestUtilities, at least from the code side of things, and if you've tried to take a look at the pydoc output for it, it's probably barfed at you, saying that it couldn't find DocMeta.

What DocMeta does, is provide decorators that add information to the docstring of a function or method. Consider the HasTestFor function from UnitTestUtilities:

@ToDo( 'Unit-test the function' )
@ToDo( 'Document exceptions raised' )
@ToDo( 'Value-check arguments' )
@ToDo( 'Type-check arguments' )
@DocumentArgument( 'argument', 'testCase', None, '(unittest.TestCase, required) The TestCase instance to check for the named test-method in.' )
@DocumentArgument( 'argument', 'itemName', None, '(Class, required) The name of the property or method to look for a test-method for ("test[itemName]").' )
@DocumentReturn( types.BooleanType )
def HasTestFor( testCase, itemName ):
    """Returns true if the testCase provided has a test-method named 'test[itemName]'."""

Without the decorators, the pydoc documentation for the function would look something like so:

HasTestFor(testCase, itemName)

Returns true if the testCase provided has a test-method named 'test[itemName]'.

With the decorators, the this is what it looks like:

HasTestFor(testCase, itemName)

Returns true if the testCase provided has a test-method named 'test[itemName]'. 
  
################### 
## TO-DOs        ## 
################### 
 - Type-check arguments  
 - Value-check arguments  
 - Document exceptions raised  
 - Unit-test the function  
  
################### 
## Returns       ## 
################### 
bool 
  
################### 
## Arguments     ## 
################### 
 + testCase ............ (unittest.TestCase, required) The TestCase instance to  
                         check for the named test-method in.  
 + itemName ............ (Class, required) The name of the property or method  
                         to look for a test-method for ("test[itemName]").

The basic pattern that these decorators follow is to add the supplied meta-data to the item being decorated as protected attributes of the item (_argumentDocumentation, _configDocumentation, etc.), with helper functions to assure that the meta-data structure exists, and to re-generate the item's docstring as needed.

As I contine working on this module, I'm planning to add in classes and functionality to generate documentation as LaTeX/PDF, HTML and/or XML output from the additional meta-data, but for now, I'm just going to share/show the code for the decorators...

# DocMeta.py
"""Provides common documentation-metadata decorators, that facilitate in-code 
documentation of functions, classes, their methods, etc."""

__author__  = 'Brian D. Allbee'
__licence__ = 'http://creativecommons.org/licenses/by-sa/3.0/'
__version__ = '0.2'
__copyright__ = "Copyright 2011, Brian D. Allbee"
__credits__ = ["Brian D. Allbee"]
__maintainer__ = "Brian D. Allbee"
__email__ = "brian.allbee@gmail.com"
__status__ = "Development"

#####################################
# Build an "__all__" list to        #
# support                           #
# "from Namespace.blah import *"    #
# syntax                            #
#####################################
__all__ = []

#####################################
# Required imports                  #
#####################################

import inspect

#####################################
# Constants defined in the module   #
#####################################

SelfDocumentationString = 'The object-instance that the method is called against.'

__all__ += [ 'SelfDocumentationString' ]

#####################################
# Functions defined in the module   #
#####################################

def HangWrap( baseString, lineLength=80, indent=30 ):
    """Helper/formatting function that wraps a string to the specified line-length, with the specified hanging indent."""
    remainder = baseString.split( ' ' )
    result = ''
    line = ''
    while remainder:
        theWord = remainder[0]
        remainder = remainder[1:]
        if not theWord in [ '\n', '\r' ]:
            if len( line ) + len( theWord ) + 1 > lineLength:
                result += line + '\n'
                line = ' ' * ( indent + 1)
            line += theWord + ' '
    result += line + '\n'
    return result

__all__ += [ 'HangWrap' ]

def InitializeDocMetadata( decoratedItem ):
    """Helper function that initializes all of the decorated item's documentation-metadata structures."""
    if not hasattr( decoratedItem, '_configDocumentation' ):
        setattr( decoratedItem, '_configDocumentation', {} )
    if not hasattr( decoratedItem, '_argumentDocumentation' ):
        setattr( decoratedItem, '_argumentDocumentation', { 'args':{}, 'arglist':{}, 'kwargs':{} } )
    if not hasattr( decoratedItem, '_baseDocumentation' ):
        setattr( decoratedItem, '_baseDocumentation', decoratedItem.__doc__ )
        if not getattr( decoratedItem, '_baseDocumentation' ):
            setattr( decoratedItem, '_baseDocumentation', '' )
    if not hasattr( decoratedItem, '_exceptionDocumentation' ):
        setattr( decoratedItem, '_exceptionDocumentation', {} )
    if not hasattr( decoratedItem, '_returnsDocumentation' ):
        setattr( decoratedItem, '_returnsDocumentation', [] )
    if not hasattr( decoratedItem, '_todoDocumentation' ):
        setattr( decoratedItem, '_todoDocumentation', [] )

__all__ += [ 'InitializeDocMetadata' ]

def RebuildDocumentation( decoratedItem ):
    """Helper function that rebuilds documentation metadata and stuffs it into the item's __doc__."""

    # Base documentation, if any, first...
    tmpDocs = getattr( decoratedItem, '_baseDocumentation' )
    if not tmpDocs:
        tmpDocs = 'No base documentation provided.'
    tmpDocs += '\n'

    # TODO documentation next
    toDoDocs = getattr( decoratedItem, '_todoDocumentation' )
    if len( toDoDocs ) != 0:
        tmpDocs += """
###################
## TO-DOs        ##
###################
"""
        for theToDo in toDoDocs:
            tmpDocs += HangWrap( ' - %s' % ( theToDo ), 80, 6 )

    # Returns documentation next
    returnDocs = getattr( decoratedItem, '_returnsDocumentation' )
    tag = ''
    if len( returnDocs ) > 1:
        tmpDocs += """
###################
## Returns       ##
###################
Any of %s
""" % ( returnDocs )
    elif len( returnDocs ) == 1:
        tmpDocs += """
###################
## Returns       ##
###################
%s
""" % ( returnDocs[ 0 ] )

    # Arguments next
    argDocs = getattr( decoratedItem, '_argumentDocumentation' )
    argSpecs = inspect.getargspec( decoratedItem )
    if argSpecs[0] or argSpecs[1] or argSpecs[2]:
        tmpDocs += """
###################
## Arguments     ##
###################
"""
    if argSpecs[0]:
        for theArg in argSpecs[0]:
            if theArg in argDocs['args'].keys():
                tmpDocs += HangWrap( ( ' + %s ........................' % ( theArg ) )[:24] + ' ' + argDocs['args'][theArg], 80, 24 )
    if argSpecs[1]:
        tmpDocs += HangWrap( ( ' + %s ........................' % ( argSpecs[1] ) )[:24] + ' ' + argDocs['arglist'], 80, 24 )
    if argSpecs[2] and argDocs['kwargs']:
        tmpDocs += ( ' + %s ........................' % ( argSpecs[2] ) )[:24] + ' Keyword-argument list. The following values are known/documented:\n'
        theKeys = argDocs['kwargs'].keys()
        theKeys.sort()
        for theKey in theKeys:
            tmpDocs += HangWrap( ( '   - %s ........................' % ( theKey ) )[:24] + ' ' + str( argDocs['kwargs'][theKey] ), 80, 24 )

    exceptionDocs = getattr( decoratedItem, '_exceptionDocumentation' )
    if exceptionDocs:
        tmpDocs += """
###################
## Exceptions    ##
###################
"""
    exceptions = exceptionDocs.keys()
    exceptions.sort()
    for theException in exceptions:
        tmpDocs += 'Raises %s when:\n' % ( theException )
        for details in exceptionDocs[ theException ]:
            tmpDocs += HangWrap( ' - %s;' % ( details ), 80, 6 )

    configDocs = getattr( decoratedItem, '_configDocumentation' )
    if configDocs:
        tmpDocs += """
###################
## Configuration ##
###################

The relevant configuration-file section(s) and values(s) are:
"""
        sectionKeys = configDocs.keys()
        sectionKeys.sort()
        for theSection in sectionKeys:
            tmpDocs += '[%s]\n' % ( theSection )
            items = configDocs[ theSection ]
            itemKeys = items.keys()
            itemKeys.sort()
            for theItem in itemKeys:
                tmpDocs += HangWrap( ( '%s .....................................' % ( theItem ) )[:29] + ' %s' % ( items[ theItem ] ), 80, 29 )

    setattr( decoratedItem, '__doc__', tmpDocs )

__all__ += [ 'RebuildDocumentation' ]

def DocumentArgument( argumentType, argumentName, keywordName, argumentDocumentation ):
    """Decorator function that documents a function- or method-argument."""
    def decorator( decoratedItem ):
        InitializeDocMetadata( decoratedItem )

        argSpecs = inspect.getargspec( decoratedItem )
        if argSpecs[0]:
            for theName in argSpecs[0]:
                if theName != 'self':
                    if not theName in decoratedItem._argumentDocumentation[ 'args' ].keys():
                        decoratedItem._argumentDocumentation[ 'args' ][ theName ] = 'Not documented.'
                else:
                    decoratedItem._argumentDocumentation[ 'args' ][ theName ] = SelfDocumentationString
        if argSpecs[1]:
            if not decoratedItem._argumentDocumentation[ 'arglist' ]:
                decoratedItem._argumentDocumentation[ 'arglist' ] = argSpecs[1]
        if argSpecs[2]:
            if not decoratedItem._argumentDocumentation[ 'kwargs' ]:
                decoratedItem._argumentDocumentation[ 'kwargs' ] = {}

        if argumentType == "argument":
            argDocs = getattr( decoratedItem, '_argumentDocumentation' )[ 'args' ]
            argDocs[ argumentName ] = argumentDocumentation

        elif argumentType == "arglist":
            pass

        elif argumentType == "keyword":
            argDocs = getattr( decoratedItem, '_argumentDocumentation' )[ 'kwargs' ]
            if keywordName in argDocs.keys():
                raise AttributeError( '"%s" has already been documented as a keyword value for the "%s" list of %s.' % ( keywordName, argumentName, decoratedItem.__name__ ) )
            argDocs[ keywordName ] = argumentDocumentation

        else:
            raise AttributeError( '"%s" is not a recognized argumentType ("argument", "arglist", or "keyword" are allowed).' % ( argumentType ) )

        # Re-create the item's docstring with the argument information:
        RebuildDocumentation( decoratedItem )

        return decoratedItem

    return decorator

__all__ += [ 'DocumentArgument' ]

def DocumentConfiguration( configSection, configName, configDescription ):
    """Decorator function that documents the configuration-file section(s) relevant to 
instances of the object. Intended for use with ConfigParser instances, but may 
be valid for other configuration mechanisms."""
    def decorator( decoratedItem ):
        InitializeDocMetadata( decoratedItem )

        configDocs = getattr( decoratedItem, '_configDocumentation' )
        if not configSection in configDocs.keys():
            configDocs[ configSection ] = {}
        configDocs[ configSection ][ configName ] = configDescription

        # Re-create the item's docstring with the exception information:
        RebuildDocumentation( decoratedItem )

        return decoratedItem

    return decorator

__all__ += [ 'DocumentConfiguration' ]

def DocumentException( exceptionType, exceptionCase ):
    """Decorator function that documents an exception that can be raised by a function or method."""
    def decorator( decoratedItem ):
        InitializeDocMetadata( decoratedItem )

        exceptionDocs = getattr( decoratedItem, '_exceptionDocumentation' )
        if not exceptionType.__name__ in exceptionDocs.keys():
            exceptionDocs[ exceptionType.__name__ ] = []
        exceptionDocs[ exceptionType.__name__ ].append( exceptionCase )

        # Re-create the item's docstring with the exception information:
        RebuildDocumentation( decoratedItem )

        return decoratedItem

    return decorator

__all__ += [ 'DocumentException' ]

def DocumentReturn( *returnTypes ):
    """Decorator function that documents the return type(s) of a function or method."""
    def decorator( decoratedItem ):
        InitializeDocMetadata( decoratedItem )

        returnDocs = getattr( decoratedItem, '_returnsDocumentation' )
        for theType in returnTypes:
            if theType != None:
                if not theType.__name__ in returnDocs:
                    returnDocs.append( theType.__name__ )
            else:
                if not 'None' in returnDocs:
                    returnDocs.append( 'None' )
        returnDocs.sort()
        # Re-create the item's docstring with the return information:
        RebuildDocumentation( decoratedItem )

        return decoratedItem

    return decorator

__all__ += [ 'DocumentReturn' ]

def ToDo( todo ):
    """Decorator function that documents "To-do" items."""
    def decorator( decoratedItem ):
        InitializeDocMetadata( decoratedItem )

        toDoDocs = getattr( decoratedItem, '_todoDocumentation' )
        toDoDocs.append( todo )
        # Re-create the item's docstring with the todo information:
        RebuildDocumentation( decoratedItem )

        return decoratedItem

    return decorator

__all__ += [ 'ToDo' ]

#####################################
# Decorate above functions          #
#####################################

# DocumentArgument
DocumentArgument( 'argument', 'argumentType', None, '(\'argument\', \'arglist\' or \'keyword\') The argument-type for the argument.' )( DocumentArgument )
DocumentArgument( 'argument', 'argumentName', None, '(String, required) The name of the argument of the function or method being documented.' )( DocumentArgument )
DocumentArgument( 'argument', 'keywordName', None, '(String, required for keyword arguments, ignored for other types) The name of a keyword argument that is recognized by the function or method.' )( DocumentArgument )
DocumentArgument( 'argument', 'argumentDocumentation', None, '(String) The documentation for the argument.' )( DocumentArgument )
ToDo( 'Type-check arguments.' )( DocumentArgument )
ToDo( 'Value-check arguments(?).' )( DocumentArgument )
ToDo( 'Document exceptions' )( DocumentArgument )

# DocumentConfiguration
DocumentArgument( 'argument', 'configSection', None, '(String, required) The name of the configuration-section that the configName argument will be found in.' )( DocumentConfiguration )
DocumentArgument( 'argument', 'configName', None, '(String, required) The name of the configuration-value.' )( DocumentConfiguration )
DocumentArgument( 'argument', 'configDescription', None, '(String) The documentation for the configuration name/value pair.' )( DocumentConfiguration )
ToDo( 'Type-check arguments.' )( DocumentConfiguration )
ToDo( 'Value-check arguments(?).' )( DocumentConfiguration )
ToDo( 'Document exceptions' )( DocumentConfiguration )

# DocumentException
DocumentArgument( 'argument', 'exceptionType', None, '(Exception, required) The exception-type expected under the case in exceptionCase.' )( DocumentException )
DocumentArgument( 'argument', 'exceptionCase', None, '(String) The case that will raise the specified exception. Documentation will be built as a list of cases for each exception, in a format similar to "Raises [exceptionType] when:" followed by all exception-cases for that type.' )( DocumentException )
ToDo( 'Type-check arguments.' )( DocumentException )
ToDo( 'Value-check arguments(?).' )( DocumentException )
ToDo( 'Document exceptions' )( DocumentException )

# DocumentReturn
DocumentArgument( 'arglist', 'returnTypes', None, 'The types returned by the function or method.' )( DocumentReturn )
ToDo( 'Type-check arguments.' )( DocumentReturn )
ToDo( 'Value-check arguments(?).' )( DocumentReturn )
ToDo( 'Document exceptions' )( DocumentReturn )

# HangWrap
DocumentArgument( 'argument', 'baseString', None, '(String, required) The base string to re-format.' )( HangWrap )
DocumentArgument( 'argument', 'lineLength', None, '(Integer, optional, defaults to 80) The maximum length of a line in the re-formatted string.' )( HangWrap )
DocumentArgument( 'argument', 'indent', None, '(Integer, optional, defaults to 30) The number of spaces to indent second and subsequent lines in the re-formatted string' )( HangWrap )
ToDo( 'Type-check arguments.' )( HangWrap )
ToDo( 'Value-check arguments(?).' )( HangWrap )
ToDo( 'Document exceptions' )( HangWrap )

# InitializeDocMetadata
DocumentArgument( 'argument', 'decoratedItem', None, 'The item to initialize documentation-meta-data structures on.' )( InitializeDocMetadata )
ToDo( 'Type-check arguments.' )( InitializeDocMetadata )
ToDo( 'Value-check arguments(?).' )( InitializeDocMetadata )
ToDo( 'Document exceptions' )( InitializeDocMetadata )

# RebuildDocumentation
DocumentArgument( 'argument', 'decoratedItem', None, 'The item to rebuild __doc__ on, using current documentation-meta-data structure.' )( RebuildDocumentation )
ToDo( 'Type-check arguments.' )( RebuildDocumentation )
ToDo( 'Value-check arguments(?).' )( RebuildDocumentation )
ToDo( 'Document exceptions' )( RebuildDocumentation )

# ToDo
DocumentArgument( 'argument', 'todo', None, '(String, required) The "To-do" item to attach.' )( ToDo )
ToDo( 'Type-check arguments.' )( ToDo )
ToDo( 'Value-check arguments(?).' )( ToDo )
ToDo( 'Document exceptions' )( ToDo )

#####################################
# Interfaces defined in the module  #
#####################################

#####################################
# Abstract Classes defined in the   #
# module                            #
#####################################

#####################################
# Classes defined in the module     #
#####################################

#####################################
# Package sub-modules and -packages #
#####################################

#####################################
# Unit-test the module on main      #
#####################################

if __name__ == '__main__':
    import inspect, os, sys, unittest
    from UnitTestUtilities import *

    testSuite = unittest.TestSuite()
    testResults = unittest.TestResult()

    ##################################
    # Unit-test Constants            #
    ##################################

    class testConstants( unittest.TestCase ):
        """Unit-tests the constants defined in the module."""
        def testConstants( self ):
            self.assertEquals( SelfDocumentationString, 'The object-instance that the method is called against.', 'SelfDocumentationString should be a constant, meaningful string.' )
    
    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testConstants ) )

    ##################################
    # Unit-test Functions            #
    ##################################

    class testDocumentArgument( unittest.TestCase ):
        """Unit-tests the DocumentArgument function defined in the module."""
        pass
    
    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testDocumentArgument ) )

    class testDocumentConfiguration( unittest.TestCase ):
        """Unit-tests the DocumentConfiguration function defined in the module."""
        pass
    
    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testDocumentConfiguration ) )

    class testDocumentException( unittest.TestCase ):
        """Unit-tests the DocumentException function defined in the module."""
        pass
    
    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testDocumentException ) )

    class testDocumentReturn( unittest.TestCase ):
        """Unit-tests the DocumentReturn function defined in the module."""
        pass
    
    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testDocumentReturn ) )

    class testHangWrap( unittest.TestCase ):
        """Unit-tests the HangWrap function defined in the module."""
        pass
    
    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testHangWrap ) )

    class testInitializeDocMetadata( unittest.TestCase ):
        """Unit-tests the InitializeDocMetadata function defined in the module."""
        pass
    
    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testInitializeDocMetadata ) )

    class testRebuildDocumentation( unittest.TestCase ):
        """Unit-tests the RebuildDocumentation function defined in the module."""
        pass
    
    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testRebuildDocumentation ) )

    ##################################
    # Unit-test Interfaces           #
    ##################################

    ##################################
    # Unit-test Abstract Classes     #
    ##################################

    ##################################
    # Unit-test Classes              #
    ##################################

    # Run unit-tests and install
    print '#'*79
    print 'Running unit-tests'
    testSuite.run( testResults )
    print '#'*79
    if testResults.errors == [] and testResults.failures == []:
        print 'All unit-tests passed! Ready for build/copy!'
        import shutil
        sitePackagesDirs = []
        for path in sys.path:
            if path.split( os.sep )[-1].lower() == 'site-packages':
                sitePackagesDirs.append( path )
        for path in sitePackagesDirs:
            destinationFile = path + os.sep + __file__
            try:
                shutil.copyfile( './%s' % __file__, destinationFile )
                print 'Copied %s to %s' % ( __file__, path )
            except:
                print 'Could not copy file %s to location %s' % ( __file__, path )
    else:
        print 'Unit-tests failed (see items below)'
        print '#'*79
        if testResults.errors:
            print 'Errors'
            print '#' + '-'*77 + '#'
            for error in testResults.errors:
                print error[1]
                print '#' + '-'*77 + '#'
            print
        if testResults.failures:
            if testResults.errors:
                print '#'*79
            print 'Failures'
            print '#' + '-'*77 + '#'
            for failure in testResults.failures:
                print failure[1]
                print '#' + '-'*77 + '#'
            print
    print '#'*79

As with UnitTestUtilities, I didn't want to hold up sharing this code for the sake of writing a lot of unit-tests. In this case, I'm not quite sure what my testing strategy is going to be. But as I write out those tests, the updated version of the module will be shared at the location noted above, so keep watching if you have the interest...
Commentary:

Line(s)

20, 34, 56, etc.

This is my usual __all__ set-up

26

The decorators use functionality from the inspect module, so we import it here

32

Rather than have a magic string embedded in the argument-decorator, though that would probably be OK, I opted to make the SelfDocumentationString constant, and use it elsewhere. The name could be changed, it's a long name to type over and over again, but since most of it's uses can be copied and pasted quickly and easily, I didn't think it was a big deal.

40-54

HangWrap is just a string-formatting function, that provides the nice, clean dot-leader-with-hanging-indent shown in the doc-string example for HasTestFor, above. It's a convenience function, nothing more, and I may not bother with writing any significant unit-tests for it later (though I should, for the sake of completion).

58-73

InitializeDocMetadata is a critical function - it assures that the documentation-meta-data structure needed for most (all?) of the decorators here is created, but not disturbed if it already exists. I'm not completely happy with the function, but I couldn't come up with a better way to handle the need, and I expect that it will perform well (though since it's not called outside of the context of a decorator that runs once at "compile-time," even that's not much of a concern). The items it ensures exist are:

_configDocumentation: A dictionary that will eventually contain config-section keys to dictionaries of config-names/values.
_argumentDocumentation: A dictionary of argument-, argument-list- and keyword-dictionaries
_baseDocumentation: The original __doc__ of the decorated item.
_exceptionDocumentation: A dictionary of Exception keys to condition-string lists
_returnDocumentation: A list of types
_todoDocumentation: A list of to-do strings

77-172

This is the function that generates the final, meta-data-inclusive docstring that we see when browsing decorated items with pydoc. It uses HangWrap to format long lines into hanging-indent lines, but is pretty straightforward, I believe. Just long, since there's a lot of stuff to look for and format, potentially.

174-217

DocumentArgument is (probably) the most commonly used decorator of this whole set, and likely deserves some in-depth commentary:

Line(s)
177: Makes sure that the meta-data structure is available on the decorated item.
179: Gets ahold of the list of arguments that were defined by the code using inspect functionality.
180-186: Checks for the existence of the argument-name, documents it with the SelfDocumentationString value if it's self, or with "Not documented" by default, after checking to see that it hasn't already been documented.
187-189: Stuffs an inspected argument-list's name into the meta-data structure.
190-191: Stuffs an inspected keyword-arument-list's name into the meta-data structure.
194-196: Sets the documentation-string for the specified argument.
198-199: Skips any further processing for an argument-list, since there can be only one, and it's done by the time we reach this section of the code.
201-205: Adds the specified keyword-name and documentation-value to the keyword-argument's meta-data (a dictionary of known keywords, and their documentation, if you recall).
211: Rebuilds the decorated item's __doc__.

The patterns of various branches of DocumentArgument are echoed in DocumentConfiguration, DocumentException, DocumentReturn and ToDo, albeit with different and shallower meta-data-structure targets - hopefully, their functionality is straightforward.

Lines 298 through 356 explicitly decorate all of the decorator functions after they have all been defined, in order to provide the (hopefully) expected documentation for them. The structure/syntax probably looks a bit weird if you aren't used to it, but it does the same thing as the typical @DecoratorFunction(args) call right before the item being decorated...

The final doc-strings of each of these items demonstrate that:

DocumentArgument(argumentType, argumentName, keywordName, argumentDocumentation)
    Decorator function that documents a function- or method-argument.
     
    ###################
    ## TO-DOs        ##
    ###################
     - Type-check arguments. 
     - Value-check arguments(?). 
     - Document exceptions 
     
    ###################
    ## Arguments     ##
    ###################
     + argumentType ........ ('argument', 'arglist' or 'keyword') The argument-type 
                             for the argument. 
     + argumentName ........ (String, required) The name of the argument of the 
                             function or method being documented. 
     + keywordName ......... (String, required for keyword arguments, ignored for 
                             other types) The name of a keyword argument that is 
                             recognized by the function or method. 
     + argumentDocumentation (String) The documentation for the argument.

DocumentConfiguration(configSection, configName, configDescription)
    Decorator function that documents the configuration-file section(s) relevant to 
    instances of the object. Intended for use with ConfigParser instances, but may 
    be valid for other configuration mechanisms.
     
    ###################
    ## TO-DOs        ##
    ###################
     - Type-check arguments. 
     - Value-check arguments(?). 
     - Document exceptions 
     
    ###################
    ## Arguments     ##
    ###################
     + configSection ....... (String, required) The name of the 
                             configuration-section that the configName argument 
                             will be found in. 
     + configName .......... (String, required) The name of the 
                             configuration-value. 
     + configDescription ... (String) The documentation for the configuration 
                             name/value pair.

DocumentException(exceptionType, exceptionCase)
    Decorator function that documents an exception that can be raised by a function or method.
     
    ###################
    ## TO-DOs        ##
    ###################
     - Type-check arguments. 
     - Value-check arguments(?). 
     - Document exceptions 
     
    ###################
    ## Arguments     ##
    ###################
     + exceptionType ....... (Exception, required) The exception-type expected 
                             under the case in exceptionCase. 
     + exceptionCase ....... (String) The case that will raise the specified 
                             exception. Documentation will be built as a list of 
                             cases for each exception, in a format similar to 
                             "Raises [exceptionType] when:" followed by all 
                             exception-cases for that type.

DocumentReturn(*returnTypes)
    Decorator function that documents the return type(s) of a function or method.
     
    ###################
    ## TO-DOs        ##
    ###################
     - Type-check arguments. 
     - Value-check arguments(?). 
     - Document exceptions 
     
    ###################
    ## Arguments     ##
    ###################
     + returnTypes ......... returnTypes

HangWrap(baseString, lineLength=80, indent=30)
    Helper/formatting function that wraps a string to the specified line-length, with the specified hanging indent.
     
    ###################
    ## TO-DOs        ##
    ###################
     - Type-check arguments. 
     - Value-check arguments(?). 
     - Document exceptions 
     
    ###################
    ## Arguments     ##
    ###################
     + baseString .......... (String, required) The base string to re-format. 
     + lineLength .......... (Integer, optional, defaults to 80) The maximum length 
                             of a line in the re-formatted string. 
     + indent .............. (Integer, optional, defaults to 30) The number of 
                             spaces to indent second and subsequent lines in the 
                             re-formatted string

InitializeDocMetadata(decoratedItem)
    Helper function that initializes all of the decorated item's documentation-metadata structures.
     
    ###################
    ## TO-DOs        ##
    ###################
     - Type-check arguments. 
     - Value-check arguments(?). 
     - Document exceptions 
     
    ###################
    ## Arguments     ##
    ###################
     + decoratedItem ....... The item to initialize documentation-meta-data 
                             structures on.

RebuildDocumentation(decoratedItem)
    Helper function that rebuilds documentation metadata and stuffs it into the item's __doc__.
     
    ###################
    ## TO-DOs        ##
    ###################
     - Type-check arguments. 
     - Value-check arguments(?). 
     - Document exceptions 
     
    ###################
    ## Arguments     ##
    ###################
     + decoratedItem ....... The item to rebuild __doc__ on, using current 
                             documentation-meta-data structure.

ToDo(todo)
    Decorator function that documents "To-do" items.
     
    ###################
    ## TO-DOs        ##
    ###################
     - Type-check arguments. 
     - Value-check arguments(?). 
     - Document exceptions 
     
    ###################
    ## Arguments     ##
    ###################
     + todo ................ (String, required) The "To-do" item to attach.

The UnitTestUtilities Module

2011-11-09T05:02:00.000-08:00

Shared-code location: dl.dropbox.com/u/1917253/site-packages/UnitTestUtilities.py

Earlier I promised to share the UnitTestUtilities and DocMeta modules. Today, it'll be UnitTestUtilities.

The rationale for this chunk of code was, quite simply, that I found I needed a way to make sure that all of the items that needed to be tested for a given class, abstract class, or interface would be automatically identified in as simple a way as possible.

The test-cases template in the module-template I shared yesterday provides just such hooks (for properties and methods, at least) in the testPropertyCountAndTests and testMethodCountAndTests test-methods, you may recall. Those, in turn depend on two functions provided by UnitTestUtilities. Here's what that code looks like at present.

# UnitTestUtilities.py
"""Provides some utility functions to assist in assuring code-coverage for unit-tests.

These functions are predicated on a test structure that looks something like so:

########################################

    class ClassNameDerived( ClassName ):
        def __init__( self ):
            pass
    
    class testClassName( unittest.TestCase ):
        \"\"\"Unit-tests the ClassName class.\"\"\"
    
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
#        def testDerived( self ):
#            \"\"\"Testing abstract nature of the ClassName class.\"\"\"
#            pass
    
#        def testFinal( self ):
#            \"\"\"Testing final nature of the ClassName class.\"\"\"
#            pass
    
        def testConstruction( self ):
            \"\"\"Testing construction of the ClassName class.\"\"\"
            pass
    
        def testPropertyCountAndTests( self ):
            \"\"\"Testing the properties of the ClassName class.\"\"\"
            items = getMemberNames( ClassName )[0]
            actual = len( items )
            expected = 0
            self.assertEquals( expected, actual, 'ClassName is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            \"\"\"Testing the methods of the ClassName class.\"\"\"
            items = getMemberNames( ClassName )[1]
            actual = len( items )
            expected = 0
            self.assertEquals( expected, actual, 'ClassName is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

########################################

The functions, as used in testPropertyCountAndTests and testMethodCountAndTests, 
assure that there is a test[MethodName] and test[PropertyName] for each *public* 
method and property, respectively, and raise assertion errors if they don't exist. 
It is up to the developer to generate those test-methods and to assure that they 
provide complete coverage (at least so far).

"""

__author__  = 'Brian D. Allbee'
__licence__ = 'http://creativecommons.org/licenses/by-sa/3.0/'
__version__ = '0.2'
__copyright__ = "Copyright 2011, Brian D. Allbee"
__credits__ = ["Brian D. Allbee"]
__maintainer__ = "Brian D. Allbee"
__email__ = "brian.allbee@gmail.com"
__status__ = "Development"

#####################################
# Build an "__all__" list to        #
# support                           #
# "from Namespace.blah import *"    #
# syntax                            #
#####################################
__all__ = []

#####################################
# Required imports                  #
#####################################

import inspect, types
from DocMeta import *

#####################################
# Functions defined in the module   #
#####################################

@ToDo( 'Unit-test the function' )
@ToDo( 'Document exceptions raised' )
@ToDo( 'Value-check arguments' )
@ToDo( 'Type-check arguments' )
@DocumentArgument( 'argument', 'theClass', None, '(Class, required) The class to get property and method names from.' )
@DocumentReturn( types.TupleType )
def getMemberNames( theClass ):
    """Returns a list of property- and method-names for the specified class."""
    classMembers = inspect.getmembers( theClass )
    properties = []
    methods = []
    for member in classMembers:
        if inspect.isdatadescriptor( member[1] ) and member[0][0] != '_':
            properties.append( member[0] )
        if inspect.ismethod( member[1] ) and member[0][0] != '_':
            methods.append( member[0] )
    return properties, methods

__all__ += [ 'getMemberNames' ]

@ToDo( 'Unit-test the function' )
@ToDo( 'Document exceptions raised' )
@ToDo( 'Value-check arguments' )
@ToDo( 'Type-check arguments' )
@DocumentArgument( 'argument', 'testCase', None, '(unittest.TestCase, required) The TestCase instance to check for the named test-method in.' )
@DocumentArgument( 'argument', 'itemName', None, '(Class, required) The name of the property or method to look for a test-method for ("test[itemName]").' )
@DocumentReturn( types.BooleanType )
def HasTestFor( testCase, itemName ):
    """Returns true if the testCase provided has a test-method named 'test[itemName]'."""
    classMembers = inspect.getmembers( testCase )
    nameToFind = 'test%s' % ( itemName )
    for member in classMembers:
        if member[0] == nameToFind:
            return True
    return False

__all__ += [ 'HasTestFor' ]

#####################################
# Interfaces defined in the module  #
#####################################

#####################################
# Abstract Classes defined in the   #
# module                            #
#####################################

#####################################
# Classes defined in the module     #
#####################################

#####################################
# Package sub-modules and -packages #
#####################################

#####################################
# Unit-test the module on main      #
#####################################

if __name__ == '__main__':
    import inspect, os, sys, unittest

    testSuite = unittest.TestSuite()
    testResults = unittest.TestResult()

    #################################
    # Unit-test Constants           #
    #################################

    class testConstants( unittest.TestCase ):
        """Unit-tests the constants defined in the module."""
        # No constants to test
        pass
    
    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testConstants ) )

    #################################
    # Unit-test Functions           #
    #################################

    class Parent( object ):
        ParentPublic1 = property()
        ParentPublic2 = property()
        _ParentProtected1 = property()
        _ParentProtected2 = property()
        __ParentPrivate1 = property()
        __ParentPrivate2 = property()
        ParentOverrideMe1 = property()
        def __init__( self ): pass
        def ParentMethod1( self ): pass
        def ParentMethod2( self ): pass
        def _ParentMethod1( self ): pass
        def _ParentMethod2( self ): pass
        def __ParentMethod1( self ): pass
        def __ParentMethod2( self ): pass
        def ParentOverrideMe1( self ): pass

    class Child( Parent, object ):
        ChildPublic1 = property()
        ChildPublic2 = property()
        _ChildProtected1 = property()
        _ChildProtected2 = property()
        __ChildPrivate1 = property()
        __ChildPrivate2 = property()
        ChildOverrideMe1 = property()
        def __init__( self ): pass
        def ChildMethod1( self ): pass
        def ChildMethod2( self ): pass
        def _ChildMethod1( self ): pass
        def _ChildMethod2( self ): pass
        def __ChildMethod1( self ): pass
        def __ChildMethod2( self ): pass
        def ParentOverrideMe1( self ): pass

    class BadParentTestCase( object ): pass

    class GoodParentTestCase( object ):
        def testParentPublic1( self ): pass
        def testParentPublic2( self ): pass
        def testParentMethod1( self ): pass
        def testParentMethod2( self ): pass
        def testParentOverrideMe2( self ): pass

    class BadChildTestCase( object ): pass

    class GoodChildTestCase( object ):
        def testParentPublic1( self ): pass
        def testParentPublic2( self ): pass
        def testParentMethod1( self ): pass
        def testParentMethod2( self ): pass
        def testChildPublic1( self ): pass
        def testChildPublic2( self ): pass
        def testChildMethod1( self ): pass
        def testChildMethod2( self ): pass
        def testParentOverrideMe2( self ): pass

    class testFunctions( unittest.TestCase ):
        """Unit-tests the module's functions."""
        
        def testgetMemberNames( self ):
            """Tests the getMemberNames function."""
            self.assertEquals( getMemberNames( Parent ), 
                (
                    ['ParentPublic1', 'ParentPublic2'], 
                    ['ParentMethod1', 'ParentMethod2', 'ParentOverrideMe1']
                )
            )
            self.assertEquals( getMemberNames( Child ), 
                (
                    ['ChildOverrideMe1', 'ChildPublic1', 'ChildPublic2', 'ParentPublic1', 'ParentPublic2'], 
                    ['ChildMethod1', 'ChildMethod2', 'ParentMethod1', 'ParentMethod2', 'ParentOverrideMe1']
                )
            )
        
        def testHasTestFor( self ):
            """Tests the HasTestFor function."""
            goodCases = {
                GoodParentTestCase:[
                    'ParentPublic1',
                    'ParentPublic2',
                    'ParentMethod1',
                    'ParentMethod2',
                    'ParentOverrideMe2',
                    ],
                GoodChildTestCase:[
                    'ParentPublic1',
                    'ParentPublic2',
                    'ParentMethod1',
                    'ParentMethod2',
                    'ParentOverrideMe2',
                    'ChildPublic1',
                    'ChildPublic2',
                    'ChildMethod1',
                    'ChildMethod2',
                    ],
                }
            for testCase in goodCases:
                caseItems = goodCases[ testCase ]
                for item in caseItems:
                    self.assertTrue( HasTestFor( testCase, item ), 'There should be a test-method in %s for %s (test%s)' % ( testCase.__name__, item, item ) )
            badCases = {
                BadParentTestCase:[
                    'ParentPublic1',
                    'ParentPublic2',
                    'ParentMethod1',
                    'ParentMethod2',
                    'testParentOverrideMe2',
                    ],
                BadChildTestCase:[
                    'ParentPublic1',
                    'ParentPublic2',
                    'ParentMethod1',
                    'ParentMethod2',
                    'ParentOverrideMe2',
                    'ChildPublic1',
                    'ChildPublic2',
                    'ChildMethod1',
                    'ChildMethod2',
                    ],
                }
            for testCase in badCases:
                caseItems = badCases[ testCase ]
                for item in caseItems:
                    self.assertFalse( HasTestFor( testCase, item ), 'There should not be a test-method in %s for %s (test%s)' % ( testCase.__name__, item, item ) )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testFunctions ) )

    #################################
    # Unit-test Interfaces          #
    #################################

    #################################
    # Unit-test Abstract Classes    #
    #################################

    #################################
    # Unit-test Classes             #
    #################################

    # Run unit-tests and install
    print '#'*79
    print 'Running unit-tests'
    testSuite.run( testResults )
    print '#'*79
    if testResults.errors == [] and testResults.failures == []:
        print 'All unit-tests passed! Ready for build/copy!'
        import shutil
        sitePackagesDirs = []
        for path in sys.path:
            if path.split( os.sep )[-1].lower() == 'site-packages':
                sitePackagesDirs.append( path )
        for path in sitePackagesDirs:
            destinationFile = path + os.sep + __file__
            try:
                shutil.copyfile( './%s' % __file__, destinationFile )
                print 'Copied %s to %s' % ( __file__, path )
            except:
                print 'Could not copy file %s to location %s' % ( __file__, path )
    else:
        print 'Unit-tests failed (see items below)'
        print '#'*79
        if testResults.errors:
            print 'Errors'
            print '#' + '-'*77 + '#'
            for error in testResults.errors:
                print error[1]
                print '#' + '-'*77 + '#'
            print
        if testResults.failures:
            if testResults.errors:
                print '#'*79
            print 'Failures'
            print '#' + '-'*77 + '#'
            for failure in testResults.failures:
                print failure[1]
                print '#' + '-'*77 + '#'
            print
    print '#'*79

You might note that there's several @ToDo annotations here — I'm planning on getting to each of those in the near future, but didn't want to hold up sharing this code for want of those items, particularly as the main unit-tests are functional and pass...

The importance of Code Templates

2011-11-07T06:00:00.000-08:00

It's been a while now since I posted anything... It's been a busy couple of months, and I just couldn't spare the time, unfortunately.

But I haven't been completely idle, either. I actually wrote some Python code that was intended for production use at work (which I'm not sure I can share, at least not directly), which was cool and enlightening. I also had several insights into things that have made changes to my preferred coding style, most of which I'm going to share with you here.

First, as I was thrashing through 20+ classes, nominal interfaces and nominal abstract classes, all of which were going to need to be (potentially) maintained by others who weren't as familiar with Python as I am, I determined that I should make some attempt to make the structure as consistent as possible for everyone who might have to look at the code later on. The first step in this effort, for me, at least, was to build out a template-file for Python modules, including both main-code structure and some built-in unit-testing.

This template-file can be downloaded from my public Dropbox folder at dl.dropbox.com/u/1917253/site-packages/Template.py

As of today, here's what that template looks like:

# Python module template
"""Python module template. Provides no functionality."""

__author__  = 'Brian D. Allbee'
__licence__ = 'http://creativecommons.org/licenses/by-sa/3.0/'
__version__ = '0.2'
__copyright__ = "Copyright 2011, Brian D. Allbee"
__credits__ = ["Brian D. Allbee"]
__maintainer__ = "Brian D. Allbee"
__email__ = "brian.allbee@gmail.com"
__status__ = "Development"

#####################################
# Build an "__all__" list to        #
# support                           #
# "from Namespace.blah import *"    #
# syntax                            #
#####################################
__all__ = []

#####################################
# Required imports                  #
#####################################

from DocMeta import *

#####################################
# Functions defined in the module   #
#####################################

#####################################
# Interfaces defined in the module  #
#####################################

#####################################
# Abstract Classes defined in the   #
# module                            #
#####################################

#####################################
# Classes defined in the module     #
#####################################

class ClassName( object ):
    """Class doc-string."""

    ##################################
    # Class Attributes               #
    ##################################

    ##################################
    # Class Property-Getter Methods  #
    ##################################

    ##################################
    # Class Property-Setter Methods  #
    ##################################

    ##################################
    # Class Property-Deleter Methods #
    ##################################

    ##################################
    # Class Properties               #
    ##################################

    ##################################
    # Object Constructor             #
    ##################################

    @DocumentArgument( 'argument', 'self', None, SelfDocumentationString )
    def __init__( self ):
        """Object constructor."""
        # Nominally final: Don't allow any class other than this one
#        if self.__class__ != ClassName:
#            raise NotImplementedError( 'ClassName is (nominally) a final class, and is not intended to be derived from.' )
        # Nominally abstract: Don't allow instantiation of the class
#        if self.__class__ == ClassName:
#            raise NotImplementedError( 'ClassName is (nominally) an interface, and is not intended to be instantiated.' )
#            raise NotImplementedError( 'ClassName is (nominally) an abstract class, and is not intended to be instantiated.' )
        pass

    ##################################
    # Object Destructor              #
    ##################################

    ##################################
    # Class Methods                  #
    ##################################

__all__ += [ 'ClassName' ]

#####################################
# Package sub-modules and -packages #
#####################################

#####################################
# Unit-test the module on main      #
#####################################

if __name__ == '__main__':
    import os, sys, unittest
    from UnitTestUtilities import *

    testSuite = unittest.TestSuite()
    testResults = unittest.TestResult()

    #################################
    # Unit-test Constants           #
    #################################

    class testConstants( unittest.TestCase ):
        """Unit-tests the constants defined in the module."""
        pass
    
    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testConstants ) )

    #################################
    # Unit-test Functions           #
    #################################

    #################################
    # Unit-test Interfaces          #
    #################################

    #################################
    # Unit-test Abstract Classes    #
    #################################

    #################################
    # Unit-test Classes             #
    #################################

    class ClassNameDerived( ClassName ):
        def __init__( self ):
            pass
    
    class testClassName( unittest.TestCase ):
        """Unit-tests the ClassName class."""
   
        def setUp( self ):
            pass
    
        def tearDown( self ):
            pass
        
#        def testDerived( self ):
#            """Testing abstract nature of the ClassName class."""
#            pass
    
#        def testFinal( self ):
#            """Testing final nature of the ClassName class."""
#            pass
    
        def testConstruction( self ):
            """Testing construction of the ClassName class."""
            pass
    
        def testPropertyCountAndTests( self ):
            """Testing the properties of the ClassName class."""
            items = getMemberNames( ClassName )[0]
            actual = len( items )
            expected = 0
            self.assertEquals( expected, actual, 'ClassName is expected to have %d properties to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s property (test%s), but none was identifiable.' % ( item, item ) )

        def testMethodCountAndTests( self ):
            """Testing the methods of the ClassName class."""
            items = getMemberNames( ClassName )[1]
            actual = len( items )
            expected = 0
            self.assertEquals( expected, actual, 'ClassName is expected to have %d methods to test, but %d were dicovered by inspection.' % ( expected, actual ) )
            for item in items:
                self.assertTrue( HasTestFor( self, item ), 'There should be a test for the %s method (test%s), but none was identifiable.' % ( item, item ) )

    testSuite.addTests( unittest.TestLoader().loadTestsFromTestCase( testClassName ) )

    # Run unit-tests and install
    print '#'*79
    print 'Running unit-tests'
    testSuite.run( testResults )
    print '#'*79
    if testResults.errors == [] and testResults.failures == []:
        print 'All unit-tests passed! Ready for build/copy!'
        # Uncomment the following line to allow installation
        # import shutil
        sitePackagesDirs = []
        for path in sys.path:
            if path.split( os.sep )[-1].lower() == 'site-packages':
                sitePackagesDirs.append( path )
        for path in sitePackagesDirs:
            destinationFile = path + os.sep + __file__
            try:
                shutil.copyfile( './%s' % __file__, destinationFile )
                print 'Copied %s to %s' % ( __file__, path )
            except:
                print 'Could not copy file %s to location %s' % ( __file__, path )
    else:
        print 'Unit-tests failed (see items below)'
        print '#'*79
        if testResults.errors:
            print 'Errors'
            print '#' + '-'*77 + '#'
            for error in testResults.errors:
                print error[1]
                print '#' + '-'*77 + '#'
            print
        if testResults.failures:
            if testResults.errors:
                print '#'*79
            print 'Failures'
            print '#' + '-'*77 + '#'
            for failure in testResults.failures:
                print failure[1]
                print '#' + '-'*77 + '#'
            print
    print '#'*79

There's a lot of stuff going on here that may not be immediately apparent:

Line(s)

4-11

Some meta-data about the module, which I found a fairly complete list of at this University of Colorado page. Some of these items (__version__ for example) will appear in standard pydoc output, some will not.

19, 91

Per official Python documentation, __all__ should be used to ensure that module- and package-items can be imported using the from [module] import [item|*] syntax that is common in Python code. An example of this, as it relates to the provided template class ClassName appears on line 91.

25, 71

The DocMeta module is a documentation-meta-data module I built for my own ends, that makes documenting things like function and method arguments, configuration file needs, and so on easier, using a set of decorator functions. An example of it's output (from pydoc browsing) for the ClassName class listed in the template is:

class ClassName(__builtin__.object)
     Class doc-string.
 
   Methods defined here:

__init__(self)
    Object constructor.
     
    ###################
    ## Arguments     ##
    ###################
     + self ................ The object-instance that the method is called against.

The DocMeta module has been shared, though I haven't yet posted about it. It can be found at dl.dropbox.com/u/1917253/site-packages/DocMeta.py

44-91

This is a template class, ClassName, including commented sections for everything I thought I might ever need:

Line(s)
48: Class Attributes (class-defined attributes and their values, where applicable.
52, 56 and 60: Property Getter, Setter and Deleter methods - since I prefer to have the getter methods freestanding and use a property call later to assign them to properties.
64: Property declarations
68-81: A stub object-constructor method, with commented-out common code for making a class nominally final (75-76), nominally abstract (78, 80), or nominally an interface (78-79).
84: An object destructor. I haven't actually found a need for one yet in my Python travels, but that doesn't mean I won't
88: Class methods.; I've found that I prefer to group methods that are being overridden from other classes in their own sections, whenever I can, after the "local" methods have been defined.
91: Inclusion of the class into the __all__ list

94

Package- and module-inclusions, each of which should be added to the __all__ list.

98-177

Unit-test definitions. I have a very specific unit-test structure that I prefer (at least for now), that I've written some utility code for in the UnitTestUtilities module (again, I'll share that soon).

Line(s)

105, 116 and 177

Define a TestSuite to gather up all the unit-tests in, and make sure that each TestCase-derived test-class is added to that suite.

112-116

Unit-test the package/module constants. They should be predictable, and raise errors if breaking changes are caused by altering them.

119

Unit-test any functions defined in the module.

123

Unit-test any nominal interfaces defined in the module. They may have nominally-abstract properties (I typically just do something like PropertyName = property() for abstract properties) and nominally-abstract methods (I usually just raise a NotImplementedError).

127

Unit-test any nominal abstract classes defined in the module. Any nominally-abstract properties and methods defined in one of these follows the pattern for nominal interfaces, above, while concrete properties and methods should be tested "normally."

131-177

Unit-test any classes defined in the module. At the very least, I like to provide template locations for:

Line(s)
134: A class derived from the class being tested, for use in either testing it's final or nominally-abstract nature (147, 151)
141, 144: Standard setUp and tearDown methods
147: A commented-out test method for nominally-final classes (75-76).
151: A commented-out test method for nominally-abstract classes or nominal interfaces (78-80).
155: Unit-tests for object-construction, though typically I find that the abstract/interface/final tests suffice.
159-166: Test for the expected number of class-properties, and that those properties each have an easily-identifiable test-method.; This functionality is dependent on the UnitTestUtilities module mentioned earlier. What it does, in a nutshell, is to use the inspect module's capabilities to find all class-members that are data descriptors (properties), then check each named item thus found to assure that there is a test-method defined for each (by looking for a testPropertyName method in the test-case class if the property name of the class being tested is PropertyName, for example). It also requires the developer to pay attention to the number and names of proerties that are being acquired from a super-class, which I've found very helpful.
168-173: Test for the expected number of class-methods, and that those methods each have an easily-identifiable test-method.; Apart from the fact that this test-method is looking for methods instead of properties, it behaves much like the property-oriented test-method above.

178-218

Unit-test runs and module installation on successful test-runs.

The last several lines, 178-218, run the unit-tests defined, and if there are no failures, attempts to copy the file being tested to any/all of the various site-packages directories that the script can discover.