#.think.in

A new home

tarn — Fri, 14 Jan 2011 18:22:00 +0900

I will no longer be posting to #think.in and will now be posting to my personal blog I've started building at tarnbarford.net. I've been hinting at doing this for a while and I'm looking forward to being able to post richer content to a much simpler blog.

Much thanks goes to Andrew Brodie who did most of the work getting this site up and running and getting me started blogging. It has been fun. Thanks also to everyone who has stopped by to read my posts and an extra special thanks to those who took the time to leave a comment. I encourage you to come check out my new site and let me know what you think.

Peace out.

How do you test your applications?

tarn — Fri, 15 Oct 2010 13:09:00 +0900

I've wanted to post about using Selenium and TeamCity to build and test .NET websites on Windows for a while. Unfortunatley it hasn't been forthcoming as I only really use Windows/.NET at work these days.

I decided I would get my shit together today and write, what may be my last post on .NET for a while, in response to @DavidBurela's Developer Blog Banter topic "How do you test your applications?"

I'm going skip unit-testing (which we do and try to get better at), BDD and JavaScript testing (which I would like to do and get better at) and focus on automated browser testing.

The Build Process

Watching automated test may be fun the first few runs, but it won't last long. Anyway if you want automated browsers tests, surely you want to automate the automated tests, right?

The build process is going to need to automate the browser and run the site the browser is browsing. For this we use IIS which is scriptable with the PowerShell WebAdministration module (maybe I will write another post on .NET).

The problem with having the UI Test step in this sequential process is it very quickly starts taking ages. This sucks; I want feedback as soon as possible. TeamCity provides feedback after each stage, but we don't get our green light.

This could be resolved by putting the UI tests in a nightly build, but I think a staged build or build pipeline is usually better.

The first stage compiles, runs unit tests and produces a deployable website as an artifact, this process is hopefully quite fast. If the first stage is successful the next stage in the pipeline is triggered which runs the less timely integration and UI tests. With a build file that has appropriate build targets setup, this build pipeline can be setup in the TeamCity administration web interface.

If you want to get awesome you can also start using Selenium Grid to distribute test agents or use a cloud based solution like SauceLabs

Tools

Our automated build process consists of a few tools.

TeamCity for our build management and CI server. After years of using CC.Net and a very brief and painful exposure to TFS, I love TeamCity. I also hear great thing about Hudson.

NUnit and included NUnit-Console. It does what I expect a unit-test framework and runner should do.

MSBuild with the MSBuild Community Tasks for our build scripts. I have to have some sort of build script to do continuous integration in .NET when not using TFS. I would like to be using a modern scripting language or DSL instead, but MsBuild is doing the job for now.

Selenium for our browsers automation tests. It has a recorder, a server which does the brower automation and can be controlled from by .NET client library.

Creating Tests

Selenium IDE is a Firefox plug-in for recording and running tests.

Selenium IDE is pretty slick and importantly it's a tool non-developer-testers-types can quickly get their head round and start producing some tests.

What we really want though, is to have our tests in code so we can run on them on build machine and version them in a repository.

Test Scripts

That should generate something like this.

This generates test files you can pretty much just add to your project as NUnit tests. This also means automation tests can be run by the nunit-console just like unit tests. The .NET client drivers are in the Selenium Core download.

It may be worth considering if exporting these tests to C# ideal. Keeping them in the standard html table based form means they can be reloaded into the IDE, and there are also alternate languages that you can export to.

Selenium Server

Selenium server runs on the JVM - Yep, suck it up. TeamCity does too :)

Any standard version of Java should be fine. If you've installed Java correctly it should be in your path and you should be able to type this in a console.

c:\>java -version

java version "1.6.0_18"
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) Client VM (build 16.0-b13, mixed mode, sharing)

This should output the Java version and build information. If so you have successfully installed a java runtime. Yay!

You need to download the Selenium RC and extract the selenium-server.jar file.

Once you have Java and Selenium Server you should be able to start the server from the command-line.

java -jar selenium-server.jar

Selenium Server as a Service

I wanted to run SeleniumRC (the same Jar) as a service. Console windows on servers are not so cool. Another, probably better, option is to have the build script itself start and take down the server. I went with the service option.

I found a solution for running a .jar as a service, but it didn't work on 2k8. Luckily that lead me to the Non Sucking Service Manager which has a great name and worked fine.

The parameters I used were "-Xrs -jar [path/to/selenium-server.jar]"

Discussion

I went down the path of browsers automation because I was finding none of our other testing efforts, while valuable, were not actually testing the website would work.

Despite our best efforts we were still have problems where a page or an entire site would return a HTTP 500 Server Error for many reasons.

Code problems a unit-test would ensure the fix.
IOC or ORM/Database configuration.
Compilation error at runtime on a JIT compiled view page.
JavaScript Ajax.

Anyway it happened for all types of reasons. What we needed was an end-to-end integration test. Browser automation provides these tests, and it has helped.

I actually don't see the tests themselves as being particularly fragile. No doubt they can be, but I think it also encourages clean markup. If the hierarchy or attributes change then UI tests are not the only thing that will break. Stylesheets, scripts and some server logic also depend on it.

While I have seen good value from browser automation testing, I feel that mocking out the entire web framework (like NoseGAE for AppEngine) or testing at HTTP level (which I think is what WebRat does for Rails) can test almost the same thing without involving browser autmation at all.

The Mouse is Dead, Long Live the Keyboard

tarn — Wed, 16 Jun 2010 23:40:00 +0900

First up a disclaimer, this is how I like to role. I enjoy it. I'm not saying you shouldn't use a mouse or I'll never use one again.

I spend a lot of time programming on my netbook development environment because I enjoy it. Without all the visual tooling I feel can focus on working with the machine, learning its many tongues and tricks. I've run ubuntu on it for about a year and have learned heaps about linux. There have been many languages, tools and skills I would not have otherwise been able to learn.

One thing I don't like about it is using the crappy mouse pad. It has long been a pain point, even doing trivial things. Recently I saw a tweet by @TheColonial listing some mostly unfamiliar names and adding #themouseisdead. This was exciting, I had recently adopted vim and was loving it.

Last weekend I was a bit under the weather and not thinking clearly enough to write code. So I used to time to set-up and learn a few new tools. I put together a powerful, mouse-less development environment on my tiny netbook, which I think is pretty fun to use.

xmonad

xmonad is a great window manager (~1000 lines of haskell) which seems to plug into most X systems. There are other ways of managing a windowed environment, I never new existed! It is all keyboard driven, it works out non-overlapping window layouts, you can move between multiple desktops and move windows between them.

Here are some typical tasks in a desktop with a few windows using xmonad:

Switch between window layouts, generated by haskell algorithms (mod-space).
Select between windows on screen (mod-tab).
Throw the selected window to another desktop (shift-mod-2).
Change to the desktop (mod-2).

On Ubuntu it installed with:

sudo apt-get install xmonad

Which installed its haskell dependencies and configured itself. It also added an xmonad session option on the login screen, I wanted to use this and keep the normal session intact.

When you start an xmonad session you are presented with a completely empty screen except for a background picture. You can start terminal session with some keys (mod-shift-enter), but a menu would be nice.

dmenu

dmenu is a generic dynamic menu for X systems, it allows menu items to be selected efficiently with a keyboard.

It installed with:

sudo apt-get install dwm-tools

This adds another session to the login screen, now anything can be opened from the xmonad session by bringing up dmenu (mod-p) and typing the first few letters of the desired application.

Vim

I'm surprised and a little annoyed I didn't start using it sooner really, it is a very powerful text editor. I still fumble through it but feel I'm learning to increase my productivity and reduced my frustration working with code.

sudo apt-get install vim

Now vim is available from the terminal. The vim-tutor is the best place to start, then there is a large ecosystem of plugins.

Vimperator

Vimperator is a Firefox extension that provides command from the keyboard, with vim idioms. It provides deep control of firefox, a basic browsing scenario might work like:

You can open a page in a new tab

:tabopen theage.com.au

Then to open a link to a story in a new tab

shift-f

This adds unique two digit numbers to all links. Now any link can be opened by typing the digits or the link text.

To go to the new tab:

gt

Once read the tab can be deleted

TTYtter

Anything typed that doesn't start with "/" is a tweet (the -verify and -slowpost options can help with potential problems that might cause) The forward slash is used to invoke commands like "whois", "replies", "reply", "dm", "follow", "thread" etc. It's built on curl, so we need that first.

apt-get install curl

Then download the perl script from the website, make it executable and move it to the /usr/bin folder.

I like everything about this app down to the ascii art in the menus. The dude has some pretty cool stuff including a network of gopher servers.

Vimium

Vimium is not as powerful as Vimperator, but provides some vim-like keyboard controls for Chrome. Easily installed as a chrome add-in.

Rock'n'roll time?

I can now manage windows over multiple virtual desktops. Use a fast, effective menu to opening applications. Have extensive control of firefox and most importantly, can tweet about it with TTYtter. I am excited and looking forward to working with some code.

My set-up still isn't right yet though; I need a status bar and decent start script (the plain xmonad session just a blank canvas, an internet connection, battery power monitor and a clock would be useful). I'm going to try dzen for this.

And my keyboard skills on the special keys is way worse than than I'd like it to be. The best way to fix that is to keep my hands on the keyboard, right?

Finally thanks to all the people responsible for the OSS platform, languages and many tools. They are a pleasure to use.

Revisiting dragging and inertia with RxJs

tarn — Mon, 07 Jun 2010 19:54:00 +0900

I've been enjoying spending some time playing with the Reactive Extensions for Javascript and wanted to write about it before I lost my netbook in bar. Sadly this blog makes it difficult present the content and examples together, so before fixing that, I put it here.

Map-Reduce on Mongo

tarn — Wed, 12 May 2010 19:00:00 +0900

I'm doing a presentation on non-relational databases at DDD Melbourne this weekend where I am going to demonstrate a map-reduce example with MongoDB and server side Javascript. I've been interested in both independently recently and it's been fun getting them to working together with some Javascript TDD to boot.

I needed a good example to demonstrate map-reduce and decided finding word occurrences across a series of documented seemed a simple enough scenario that is suited to being solved by a map-reduce query.

Below is an example of how we might solve this in plain C#

using System;
using System.Collections.Generic;

using System.Linq;
using System.Text;

class wordCounts {

    static void Main(string[] args) {

        // Setup some data

        List<string> lines = new List<string>() 
            { 
              "Peter Piper picked a peck of pickled peppers", 
              "A peck of pickled peppers Peter Piper picked",
              "If Peter Piper picked a peck of pickled peppers",
              "Where's the peck of pickled peppers Peter Piper picked?"

            };

        // select all words, group, count
        var wordCounts = lines.SelectMany(m => m.Split())
                              .GroupBy(m => m.ToLower())
                              .Select(m => new KeyValuePair<string, int>(m.Key, m.Count()));

        // Print out the results

        foreach (var wordCount in wordCounts) 
        {
            Console.WriteLine(string.Format("{0} {1}", wordCount.Value, wordCount.Key));     
        }
    }
}

This prints each different word in all the lines and the number of the times it occurs. The collection of strings is isomorphic to a collection of documents in the MongoDB for this example.

The SelectMany flattens lists of words from each line to a single list of words and the Group provides keys for each word, this is very similar to what the map function in the map-reduce query does.

The Select function is similar to the reduce function, but as we will see some additional considerations need to be made to allow it to be distributed.

I saw a good diagram ayande published on his blog but I didn't understand why he had multiple instance of the same document being mapped.

I created my own low key diagram to help demonstrate how a functional map-reduce could be distributed. The diagram shows the initial items can be split in half and reduced completely independently. This is interesting as it means our query can be distributed, but it also means we have to handle reducing a little differently.

It's also worth noting that this example shows a balanced tree, but it could be unbalanced and even introduce some redundancy.

MongoDB allows clients to send JavaScript map and reduce functions that will get eval'd and run on the server. Here is the map function.

function wordMap() {

    // try find words in document text
    var words = this.text.match(/\w+/g);

    if (words === null) { 
        return;
    }

    // loop every word in the document 

    for (var i = 0; i < words.length; i++) {
        // emit every word, with count of one

        emit(words[i], { count : 1 });
    }

}

The misunderstood Javascript "this" will be the context from which the function is called. Mongo will call function each document in the collection we are querying, and we can call it from a test context. Unlike the SelectMany the map function doesn't return a list, instead it calls an emit function which it expects to be defined.

We can write unit tests for this function by calling the function from a test mock context, calling a mock emit function (using Javascript as our mocking framework, wow).

eval(loadFile("src/js/wordMap.js"));


var emit;
var results;
var context;

testCases(test,

    function setUp() {
        emit = function (key, value) { 
            results.push({ key : key, value : value });
        };
        context = { text : "", map : wordMap };
        results = []; 
    },

    function empty_string_emits_nothing() {
        context.text = "";
        context.map();
        assert.that(results.length, eq(0));
    },

    function single_word_emits_single_word() {
        context.text = "findme";
        context.map();
        assert.that(results.length, eq(1));
        assert.that(results[0].key, eq("findme"));
        assert.that(results[0].value.count, eq(1));
    },

    function two_different_words_emits_twice() {
        context.text = "for bar";
        context.map();
        assert.that(results.length, eq(2));
    },

    function two_same_words_emits_twice() {
        context.text = "test test";
        context.map();
        assert.that(results.length, eq(2));
    },

    function tearDown() {
    }
);

The reduce function must reduce a list of a chosen type to a single value of that same type; it must be transitive so it doesn't matter how the mapped items are grouped.

function wordReduce(key, values) {
        var total = 0;
        for (var i = 0; i < values.length; i++) {
            total += values[i].count;
        }
        return { count : total };
    }

Similarly we can test this method does exactly what we expect it to.

eval(loadFile("src/js/wordReduce.js"));

testCases(test,

    function reduce_one_items_returns_count_of_one() {
        var result = wordReduce("test", [{ count : 1 }]);
        assert.that(result.count, eq(1));
    },

    function reduce_multiple_items_returns_item_count() {
        var result = wordReduce("test", [{ count : 1 }, { count : 1 }, { count : 1 }]);
        assert.that(result.count, eq(3));
    },

    function reduce_sums_counts() {
        var result = wordReduce("test", [{ count : 2 }, { count : 3 }]);
        assert.that(result.count, eq(5));
    },

    function reduce_is_transitive() {
        var result = wordReduce("test", [{ count : 1 }].concat(
                        wordReduce("test", [{ count : 1 }, { count : 1 }]
                     ));
        assert.that(result.count, eq(3));
    }
);

I'm using Rhino to run the Javascript so I used RhinoUnit as a test runner as it also uses the JVM and runs as an ANT scriptdef task, the setup was pretty painless. Here are the relevant ANT script sections

<scriptdef name="rhinounit"

           src="lib/rhinoUnitAnt.js"
           language="javascript">
    <attribute name="options"/>
    <attribute name="ignoredglobalvars"/>

    <attribute name="haltOnFirstFailure"/>
    <attribute name="rhinoUnitUtilPath"/>
    <element name="fileset" type="fileset"/>

</scriptdef>

<target name="javascript-tests">
    <rhinounit options="{verbose:true, stackTrace:true}" 
               haltOnFirstFailure="false" 
               rhinoUnitUtilPath="lib/rhinoUnitUtil.js">

        <fileset dir="test">
            <include name="*.js"/>
        </fileset>
    </rhinounit>

</target>

The word count example recreated in Mongo using a Python client and passing the map/reduce functions to the server.

from pymongo import Connection;
from pymongo.code import Code;


# open connection and connect to 'ddd' database
connection = Connection()
db = connection.ddd

# remove any existing data
db.drop_collection("messages")


# insert some data
lines = open("data/peter_piper.txt").readlines();

for line in lines:
    db.messages.insert( { "text" : line } )


# load map and reduce functions
map = Code(open("src/js/wordMap.js","r").read())
reduce = Code(open("src/js/wordReduce.js","r").read())


# run the map-reduce query
result = db.messages.map_reduce(map, reduce)

# print the results    
for doc in result.find():
    print doc["value"]["count"],doc["_id"]

And it worked! I'd like to run the query on a larger result-set, but there isn't much point on this tiny low-spec'd netbook.

DevEvening NoSql/MongoDB Presentation

tarn — Wed, 07 Apr 2010 21:12:00 +0900

The slides and demo I'll show in my NoSql presentation for tomorrows DevEvenings Melbourne ORM Smackdown. I hope to take the time and write a more considered post of my findings and opinions as I've found it very interesting.

Here is a link to the slides.

First I'll use some Python and the PyMongo module to connect to MongoDB, list databases, insert documents and get them out again.

Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15) 
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymongo import Connection
>>> connection = Connection()
>>> connection.database_names()
[u'files', u'working', u'demo', u'downloads', u'posts', u'local', u'admin']
>>> db = connection.demo
>>> import datetime
>>> db.messages.insert( { 'author' : 'tarn', 'date': datetime.now(), 'message' : 'Hello Mongo' } )
ObjectId('4bbc4beec73d721445000003')
>>> db.messages.find_one()
{u'date': datetime.datetime(2010, 4, 7, 19, 10, 6, 355000), u'message': u'Hello Mongo', 
u'_id': ObjectId('4bbc4beec73d721445000003'), u'author': u'tarn'}

Then have a look at some content I have in the database

>>> db = connection.working
>>> db.posts.count()
106
>>> for post in db.posts.find()[:5]:
...     print post["date"],post["title"],"by",post["author"]
... 
2010-01-25 18:06:00 Python Silverlight/Moonlight 2 Xapping by tarn
2010-03-12 13:53:00 Devevenings Presentation - IOC/Unit Testing/Mocking in ASP.NET MVC by tarn
2010-02-17 19:25:00 Revisiting Modal Binding an Interface, now with DictionaryAdapterFactory by tarn
2009-12-02 18:35:00 Creating Silverlight apps in the browser by tarn
2009-10-02 23:08:00 #.think.in infoDose #43 (11th September - 22nd September) by brodie

File storage using the GridFS class from the gridfs module. Show some files and then write a file out to the file system.

>>> from gridfs import GridFS
>>> fs = GridFS(connection.files)
>>> len(fs.list())
116
>>> for file in fs.list()[:5]:
...     print file
... 
post/debugging-ironpython-with-my-excalibur/image.png
post/debugging-ironpython-with-my-excalibur/image_thumb.png
post/devevenings-presentation---iocunit-testingmocking-in-asp.net-mvc/20102f32fdevevening_presentation.pptx
post/devevenings-presentation---iocunit-testingmocking-in-asp.net-mvc/20102f32fguestbook.zip
post/think.in-infodose-40-5th-august---16th-august/image.png
>>>
>>> with open('image.png','w') as out_file:
...     with fs.open('post/debugging-ironpython-with-my-excalibur/image.png') as in_file:
...             out_file.write(in_file.read())
...

Now to some C# (mono) and a controller class for a basic web application to view the data. It serves files found in the database, but only sends bach the correct MIME type for "image/png". Lazy.

public class HomeController : Controller
{
    BlogRepository _blogRepository;

    public HomeController()
    {
        _blogRepository = new BlogRepository(); 
    }

    public ActionResult Index ()
    {
        ViewData["posts"] = _blogRepository.GetPosts();
        return View ();
    }

    public ActionResult Entry(string id)
    {
        ViewData["post"] = _blogRepository.GetById(id);
        return View ();
    }

    public ActionResult Resource(string slug, string fileName)
    {
        return new FileStreamResult(_blogRepository.GetFile( "post/" + slug + "/" + fileName ), "image/png");
    }

}

This is a very basic repository that provides the data for the demo application. I did only enough with the C# provider to get it working and try to disconnect my connections.

public class BlogRepository
{
    Mongo _mongo;

    public BlogRepository()
    {
        string connstr = ConfigurationManager.AppSettings["connectionString"];
        _mongo = new Mongo(connstr);
    }

    public Stream GetFile(string name)
    {
        try
        {
            _mongo.Connect();
            var db = _mongo["files"];
            var fs = new GridFile(db);
            Stream data = fs.Open(name, FileMode.Open, FileAccess.Read);
            Stream output = new MemoryStream();
            CopyStream(data,output);
            output.Seek(0,SeekOrigin.Begin);
            return output;
        }
        finally
        {
            _mongo.Disconnect();    
        }
    }

    public List<Document> GetPosts()
    {
        try
        {
            _mongo.Connect(); 
            var db = _mongo["working"];
            var posts = db["posts"];
            using(ICursor all = posts.Find(new Document())){
                return all.Documents.ToList();
            }
        }
        finally
        {
            _mongo.Disconnect();
        }
    }

    public Document GetById(string id)
    {   
        try
        {
            _mongo.Connect();
            var db = _mongo["working"];
            var posts = db["posts"];
            Document doc = posts.FindOne( new Document() {{ "_id" , new Oid(id) }} );
            return doc;
        }
        finally
        {
            _mongo.Disconnect();    
        }
    }

    public static void CopyStream(Stream input, Stream output)
    {
        byte[] buffer = new byte[32768];
        while (true)
        {
            int read = input.Read (buffer, 0, buffer.Length);
            if (read <= 0)
                return;
            output.Write (buffer, 0, read);
        }
    }
}

So that's where I got for my demo for the DevEvenings ORM Smackdown. No doubt I will continue looking into MongoDB and other object/document databases.

Scraping this blog

tarn — Thu, 18 Mar 2010 23:31:00 +0900

I have created a monster and this post is about killing it off by scraping the contents of this blog into structured Python objects. Sometime later I will convert the HTML content to markdown and download the images and other resources locally.

I want to put the contents into a ZODB object database to get a feel for working with object database. A greated goal is to migrate the content a new blog engine. I don't want to go into why I felt I need to scrape it or why I want to migrate to another blog engine as it's depressing.

Moving on, I wanted to put the content into these classes

class Post():
    title = ''

    content = ''
    date = ''
    tags = []
    comments = []


class Comment():
    content = ''
    author = ''
    date = ''

    website = ''

The scraping code is not elegant but was quite fun to write as I could write it all from an interactive console session. I found BeautifulSoup was fantastic in making HTML into something that was easy to work with, although I would have liked to have used jQuery/CSS style selectors.

from BeautifulSoup import BeautifulSoup

from datetime import datetime
import urllib2 
import re

def ParseComment(soup):
    comment = Comment()
    comment.author = soup.find('p',{"class":"author"}).first().string.strip
    content = soup.find('p',{"class":"content"})
    if content:
        comment.content = content.prettify()
    website = soup.find('p',{"class":"author"}).first()    
    if website.has_key('href'):
        comment.website = soup.find('p',{"class":"author"}).first()['href']
    r = re.compile('\d*/\d*/\d* \d*.\d*')    
    date = r.findall(soup.find('p',{"class":"date"}).renderContents())[0]
    comment.date = datetime.strptime(date,'%d/%m/%Y %H:%M')
    return comment


def ParsePost(postSoup):
    post = Post()    
    post.title =  postSoup.find('a',{"class":re.compile('posthead.*')}).string
    print post.title
    post.content = postSoup.find('div', {"class":"entry"})
    date = postSoup.find('div',{"class":"descr"}).contents[0][:-4]
    post.date = datetime.strptime(date,'%B %d, %Y %H:%M')
    post.author = postSoup.find('div',{"class":"descr"}).first().string
    post.tags = map(lambda x: x.string, postSoup('a',{"rel":"tag"}))
    comments = postSoup.find('div',{"id":"commentlist"})('div')
    post.comments = [ParseComment(commentSoup) for commentSoup in comments]
    return post


def DownloadPost(url):
    postHtml = urllib2.urlopen('http://blog.sharpthinking.com.au/' + url).read()
    postSoup = BeautifulSoup(postHtml)
    return ParsePost(postSoup);


def GetPosts():
    page = urllib2.urlopen("http://blog.sharpthinking.com.au/archive.aspx")
    soup = BeautifulSoup(page)
    postUrls = map(lambda x: x['href'], soup('a', href=re.compile('/post/.*')))
    return [DownloadPost(url) for url in postUrls[:-10]]

I'm sure there is better way, but this was better than any way I've used previously. Anyway I've done a lot of work untangling the mess I created.

>>> posts = GetPost()    
>>> for post in posts[:5]:

...     print post.date, post.title
...
2010-03-17 22:16:00 OMG. It's a JavaScript Rhino

2010-03-12 13:53:00 Devevenings Presentation - IOC/Unit Testing/Mocking in ASP.NET MVC

2010-02-20 17:18:00 Revisiting Pygments in the browser with Silverlight, now with BackgroundWorker

2010-02-17 19:25:00 Revisiting Modal Binding an Interface, now with DictionaryAdapterFactory
2010-02-16 20:34:00 Modal Binding an Interface with DynamicProxy

I wanted to put the contents into the object database tonight, but I have pickled it to be revisited later.

OMG. It's a JavaScript Rhino

tarn — Wed, 17 Mar 2010 22:16:00 +0900

JavaScript is a slightly flawed language but it's got elegant parts too. All languages do to some degree, it's just JavaScript seems to have both in extremes. Whatever you think of it, history has made it the language for scripting the client-side web. It has become a mainstream language that shows no sign of falling off.

The excelent book JavaScript: The Good Parts by Douglas Crockford, working with the jQuery library and learning a little Lisp has lead me to really embrace JavaScript.

It's no secret that I like programming with interactive consoles and decided I wanted find out if there was an interactive console for JavaScript. A language that only lives inside web browser environment didn't seem right to me.

Rhino is a JavaScript implementation on the JVM. It has a compiler, a debugger and interactive console.

To get started you obviously need a version of the JVM. That's not to difficult. On Windows I just downloaded a Sun Java 6 installer. On my Ubuntu install I installed openjdk but found Rhino didn't work. So I installed sun-java6, which worked.

sudo apt-get install sun-java6-bin sun-java6-jre sun-java6-jdk

You can find what version you've installed by running

$ java -version
java version "1.6.0_15"
Java(TM) SE Runtime Environment (build 1.6.0_15-b03)
Java HotSpot(TM) Client VM (build 14.1-b02, mixed mode, sharing)

Excellent. The Rhino binaries includes js.jar which is needed for the console. Now should be able to run the jar

$ java -jar js.jar

When all goes well this will take you into the Rhino shell

Rhino 1.7 release 2 2009 03 22
js>

We can start playing with the language.

js> get_counter = function() { var counter = 0; return function() { print(counter); counter++; } };
..
js> counter1 = get_counter();
.. 
js> counter1();
0
js> counter1();
1
js> counter2 = get_counter();
..
js> counter2()
0
js> counter1()
2

Which is cool and there is some of the weirdness

js> '5' + 3
53
js> '5' - 2
3

And some interesting features

js> parseInt('06')
6
js> parseInt('08')
NaN
js> parseInt('10')
10
js> parseInt('010')
8

I'm looking forward to learning more about writing code in JavaScript.

Devevenings Presentation - IOC/Unit Testing/Mocking in ASP.NET MVC

tarn — Fri, 12 Mar 2010 13:53:00 +0900

Here are the slides and the mysterious code that was never shown from my DevEvening presentation.

Appologies it's taken a while to get them up, I was hoping to write a bit of a post about what I covered and some of the discussion that came up. That never happened.

Devevening_Presentation.pptx (565.68 kb)

Guestbook.zip (3.42 mb)

I think the presentation went well, the group was really good and we got some good discussions happening before being interrupted by delicious paramas of the world.

I'm looking forward to the next meeting, except it appears I've signed up to represent NoSql (of which I currently know very little about) in an ORM smackdown.

That's what happens when you have meetings at a pub. Anyway it should fun and I'm looking forward to learning enough about NoSql to adequatly represent it in the smackdown.

See ya there.

Revisiting Pygments in the browser with Silverlight, now with BackgroundWorker

tarn — Sat, 20 Feb 2010 17:18:00 +0900

A couple of week ago I blogged about using Pygments to do live syntax highlighting in the browser using Silverlight.

A major problem with the sample was that it did the pygmentizing on the UI thread which caused most browsers to become unresponsive. Today I wanted to fix that by using the BackgroundWorker to do the pygmentizing in a background thread.

Firstly I refactored the pygmentizing into a method that didn't interact with the UI.

def pygmentize_text(self, text, language):
    # attempt to pygmentize input with current language 
    try:

        from pygments import highlight
        from pygments.lexers import get_lexer_by_name
        from pygments.formatters import HtmlFormatter

        lexer = get_lexer_by_name( language, stripall=True)
        formatter = HtmlFormatter(linenos=False, cssclass="source")
        markup = highlight(text, lexer, formatter)

        return markup

    except:

        return "Error Generating Markup"

I then added a method that could be passed into a DoWorkEventHandler. It gets it arguments as a tuple from the event arguments and then sets the event argument result with the marked up HTML. The lack of explicit typing and use of tuples is good example of how some python idioms can be used when working with the .NET framework.

def worker(self, sender, e):

    # do work off UI thread. 
    e.Result = self.pygmentize_text(e.Argument[0],e.Argument[1])

The required BackgroundWorker and DoWorkEventHandler can be simply imported from the System.ComponentModel namespace.

from System.ComponentModel import BackgroundWorker, DoWorkEventHandler

The BackgroundWorker can then be setup and started. Again it's syntactically nice how the tuple can be created and passed as a RunWorkerAsync parameter.

def start_pygmentize(self):

    # update application state
    self.input_changed = False        
    self.pygmentizing = True
    self.show_message("pygmentizing..")

    # get paremters
    input = self.input.GetProperty("value")
    language = self.language.value

    # setup background worker
    worker = BackgroundWorker()
    worker.DoWork += DoWorkEventHandler(self.worker)
    worker.RunWorkerCompleted += self.complete

    # start the worker
    worker.RunWorkerAsync( (input,language) )

The completed event handler is the responsible for taking the markup generated by the BackgroundWorker and updating the DOM. It also fires off another worker if the source has changed since the last worker started.

def complete(self, sender, e):

    if e.Error:

        # handle errors/exceptions in worker
        self.source.SetProperty("innerHTML",e.Error.Message)

    else:

        # show the result
        self.source.SetProperty("innerHTML",e.Result)

    if self.input_changed:

        # input has changed, starty pygmentize again
        self.start_pygmentize()

    else:

        # no work queued
        self.pygmentizing = False
        self.hide_message()

The update has made the sample much more responsive, however it appears downloading the Silverlight application is still causing some browers to become a little unresponsive which is annoying. I will be interested to find out if this effect can be mitigated.

The actually pygmentizing processing could possibly be made a little faster by reusing the BackgroundWorker and only doing the Pygment imports once but the responsiveness of the browser has improved the sample enormously.

Check out the updated demo here.

Revisiting Modal Binding an Interface, now with DictionaryAdapterFactory

tarn — Wed, 17 Feb 2010 19:25:00 +0900

After my post yesterday Krzysztof Kozmic (@kkozmic) messaged me saying I should use the Castle DictionaryAdapterFactory instead.

For the record Krzysztof has a great series of posts on DynamicProxy and was also very helpful answering a specific question of mine yesterday.

Anyway this is great news! I can do the same thing only referencing one assembly. The DictionaryAdapterFactory returns an object that passes all my tests from yesterday and slides staight into the model binder.

public class FakeInterfaceModelBinder : DefaultModelBinder
{
    protected override object CreateModel(ControllerContext controllerContext,
                                          ModelBindingContext bindingContext,
                                          Type modelType)
    {
        Dictionary<string, object> dictionary = new Dictionary<string, object>();
        return new DictionaryAdapterFactory().GetAdapter(modelType, dictionary);
    }
}

It was fun playing with DynamicProxy yesterday, but I did think it might not have been a good idea for a model binder. I'm glad I was pointed in the right direction.

Cheers Krzysztof!

Modal Binding an Interface with DynamicProxy

tarn — Tue, 16 Feb 2010 20:34:00 +0900

Update: I posted an update to this post which uses the Castle DictionaryAdapterFactory after getting some feedback from this post.

This started while I was playing with some ASP.NET MVC architectural ideas in simple application.

I wanted to less code than other projects I'd worked on but still have it decoupled and testable.

At some point I started ranting on twitter (1, 2, 3) about wanting a Model Binder that could bind to an interface without needing a concrete type.

It was quickly pointed out that what I probably wanted was a Model Binder that resolves an implementation at runtime with IOC.

While this was probably true as I was using already using StuctureMap and it did seemed like a much better idea I still some my reservations. But mainly I wanted to try using the Castle DynamicProxy in an application.

Clearly the reason the default model binder can't bind to an interface is because it can't create an instance of it. Thanks to some nice extensibility hooks in the default model binder, all I had to do was override the CreateModel method and return my DynamicProxy instance. The default model binder then could do the rest of binding work as normal.

public class FakeInterfaceModelBinder : DefaultModelBinder
{
    protected override object CreateModel(ControllerContext controllerContext,
                                          ModelBindingContext bindingContext,
                                          Type modelType)
    {
        var generator = new ProxyGenerator();
        return generator.CreateInterfaceProxyWithoutTarget(modelType,
                new FakePropertiesInterceptor());
    }
}

What I needed then was a DynamicProxy object that had properties the behaved like you'd expect properties to on a normal class. I came up with some test scenarios I'd expect the proxy to pass using a test interface.

public interface IDynamicProxyTest
{
    int Integer { get; set; }
    string String { get; set; }
    DateTime Date { get; set; }
}

I figured the Model Binder would set values on the Dynamic Proxy object and the controller would get the properties back to validate and pass them to the data layer or something like that. I came up with these tests.

[Test]
public void If_date_has_been_set_then_return_it()
{
    var date = DateTime.Now;
    var proxy = CreateProxy();
    proxy.Date = date;
    Assert.AreEqual(date, proxy.Date);
}

[Test]
public void If_string_has_been_set_then_return_it()
{
    var proxy = CreateProxy();
    proxy.String = "Hello Proxy";
    Assert.AreEqual("Hello Proxy", proxy.String);
}

[Test]
public void If_int_has_been_set_then_return_it()
{
    var proxy = CreateProxy();
    proxy.Integer = 42;
    Assert.AreEqual(42, proxy.Integer);
}

After implementing these I was surprised to find the default model binder did some gets on the properties before sets. So I added some more tests.

[Test]
public void For_date_instance_if_value_not_set()
{
    var proxy = CreateProxy();
    var b = proxy.Date;
}

[Test]
public void For_int_instance_if_value_not_set()
{
    var post = CreateProxy();
    var b = post.Integer;
}

[Test]
public void For_string_instance_if_value_not_set()
{
    var post = CreateProxy();
    var b = post.String;
}

My property interceptor isn't as clean as I'd like it but did the job passing tests.

public class ProperyInterceptor : IInterceptor
{
    Type _type;

    Dictionary<string, object> properties;

    public ProperyInterceptor(Type type)
    {
        properties = new Dictionary<string, object>();
        _type = type;
        var fields = _type.GetProperties();
    }

    public void Intercept(IInvocation invocation)
    {
        if (IsSetter(invocation))
        {
            properties[Name(invocation)] = invocation.Arguments[0];
        }

        if (IsGetter(invocation))
        {
            if (properties.ContainsKey(Name(invocation)))
            {
                invocation.ReturnValue = properties[Name(invocation)];
            }
            else
            {
                GetDefaultValue(invocation);
            }
        }
    }

    private static void GetDefaultValue(IInvocation invocation)
    {
        if (invocation.Method.ReturnType == typeof(string))
        {
            invocation.ReturnValue = "";
        }
        else
        {
            invocation.ReturnValue = Activator.CreateInstance(invocation.Method.ReturnType);
        }
    }

    private string Name(IInvocation method)
    {
        return method.Method.Name.Substring(4);
    }

    private bool IsSetter(IInvocation method)
    {

        return method.Method.IsSpecialName &&
               method.Method.Name.StartsWith("set_", StringComparison.Ordinal);

    }
    private bool IsGetter(IInvocation method)
    {

        return method.Method.IsSpecialName &&
               method.Method.Name.StartsWith("get_", StringComparison.Ordinal);

    }
}

I used the better model binder Jimmy Bogard discusses as it makes writing binders for derived types and specific "types of type" more straight forward.

public class InterfaceBinder : FakeInterfaceModelBinder, IFilteredModelBinder
{
    public bool IsMatch(Type modelType)
    {
        return modelType.IsInterface;
    }
}

I did try writing a test which ran my model binder, but found it too much trouble. When I tried it out in a sample web project it worked exactly as I expected.

While this was fun I don't actually think it's a very good idea. I was thinking about the interface as DTO's and it breaks down if you want any additional methods on the interface, as they just won't work. Anyway it's been fun as always and maybe I'll find a good place to use these dynamic proxies in the future.

Source code with the sample website is all here.

Scripting your Data Model

tarn — Sun, 14 Feb 2010 14:13:00 +0900

I really like the way you can script your data model from a python REPL console on the Django and the Google App Engine web frameworks.

For me it is a hands down better way of working with data in your domain model than writing SQL.

I've since wanted to do it in .NET projects I work on, but it wasn't till I was playing with Castle ActiveRecord yesterday that I decided I'd try it out.

The ActiveRecord pattern is an intuitive way of programming with data persistence, so it's also nice to script with.

It turned out to be really easy in the project I was playing with. I didn't need to write a single additional line of C# as I already had the configuration decoupled for integration testing with an in-memory SQLite database.

namespace SimpleBlog.Data
{
    public class Configuration : IBootstrapperTask
    {
        public void Execute()
        {
            Configure(GetDefaultSettings());
        }

        public static void Configure(IDictionary<string, string> properties)
        {
            InPlaceConfigurationSource source = new InPlaceConfigurationSource();
            source.Add(typeof(ActiveRecordBase), properties);
            ActiveRecordStarter.Initialize(source);
            ActiveRecordStarter.RegisterAssemblies(Assembly.GetExecutingAssembly());
        }

        ...
    }
}

I then just wrote this little script to help with the configuration, it uses the static method above and passes in the properties for working with a development database.

import clr
clr.AddReferenceToFile("SimpleBlog.Data.dll")

from System.Collections.Generic import Dictionary
from SimpleBlog.Data import Configuration

# NHibinate Setting

properties = Dictionary[str,str]()

properties.Add("connection.driver_class",
               "NHibernate.Driver.SqlClientDriver");

properties.Add("dialect",
           "NHibernate.Dialect.MsSql2005Dialect");

properties.Add("connection.provider",
           "NHibernate.Connection.DriverConnectionProvider");

properties.Add("connection.connection_string",
           "Data Source=[CONNECTION_STRING]"); # Add

properties.Add("proxyfactory.factory_class",
           "NHibernate.ByteCode.Castle.ProxyFactoryFactory, NHibernate.ByteCode.Castle");

Configuration.Configure(properties)

Using the helper script it's pretty easy to get in and start working with the data in the data model.

IronPython 2.6 (2.6.10920.0) on .NET 2.0.50727.4927
Type "help", "copyright", "credits" or "license" for more information.
>>> import ActiveRecord
>>> from SimpleBlog.Data.Models import *
>>>
>>> post = Post()
>>> post.Title = "Working with SQL sucks!"
>>> post.Content = "Try using a scripting langauge instead. It rocks!"
>>> post.Author = "tarn"
>>> post.Save()
>>>
>>> post.Id
2
>>>
>>> posts = Post().FindAll()
>>>    
>>> for p in posts:
...     print p.Title, "by", p.Author
...
Hey, It's alive by tarn
Working with SQL sucks! by tarn
>>>

This is a simple example of a database agnostic data script using your domain model and a powerful scripting language. I think scripting data models like this could add a lot of value in many .NET development scenarios.

Pygments in the browser with Silverlight

tarn — Sun, 31 Jan 2010 23:40:00 +0900

17-02-2010 I've updated the demo to use the BackgroundWorker and posted about the update

I decided it might be fun to try get Python Markdown and Pygments running in the browser to enhance a markdown preview experience by eliminating the server-side round trips and provide a more responsive preview.

I managed to get it working entirely in Python but I found the application size excessively large (almost 3mb) and, more annoyingly, it seems to block the entire browser when it initially loads the pygments module. I think there is some sort of silverlight background thread I should be using.

I think it would work better with MarkdownSharp as pure C# silverlight applications are a fair bit leaner in size and probably run a little quicker than dynamic language applications. But this was for fun and I prefer coding in Python when not working.

17-02-2010 I've since found the MarkdownSharp doesn't do syntax hightlighting

I ended up having to write so little Python code to get this working that I can include it all here, syntax highlighted with Pygments of course.

from System.Windows import Application
from System.Windows.Controls import UserControl
from System.Windows.Browser import HtmlPage
from System import EventHandler

class App:

    def __init__(self):

        # load relevent HTML DOM elements
        self.input = HtmlPage.Document.GetElementById("input")
        self.source = HtmlPage.Document.GetElementById("output")
        self.language = HtmlPage.Document.GetElementById("lang")

        # fire javascript functions to indicate the application has been load
        HtmlPage.Window.CreateInstance("silverlight_loaded");

        # pygmentize initial 
        self.pygmentize()

        # register events
        self.input.AttachEvent('onkeyup', EventHandler( self.update_handler )) 
        self.language.AttachEvent('onchange', EventHandler( self.update_handler ))

        # fire javascript function to indicated the pygments has been loaded
        HtmlPage.Window.CreateInstance("pygments_loaded");

   # handle language or input changes by pygmentizing 
    def update_handler(self, sender, e):

        self.pygmentize()

    def pygmentize(self):
        input = self.input.GetProperty("value")

        # attempt to pygmentize input with current language 
        try:

            from pygments import highlight
            from pygments.lexers import get_lexer_by_name
            from pygments.formatters import HtmlFormatter

            lexer = get_lexer_by_name(self.language.value, stripall=True)
            formatter = HtmlFormatter(linenos=False, cssclass="source")
            markup = highlight(input, lexer, formatter)

            # update the preview
            self.source.SetProperty("innerHTML",markup)

        except:

            # indicate there was an error in pygmentize
            self.source.SetProperty("innerHTML", "Error Generating Markup" )

# Do it!    
App()

Despite the fact that there isn't much code, the development experience writing silverlight application in python is a bit of a pain in the arse. Granted it's much better with the python console in the browser and better error reporting in most recent SDK, but it still sucks; debugging and logging support is very limited and on some errors the application dies without reporting anything.

Another difficulty is having to manually copy all the python standard library modules required by the module from the library folders into the application (which explains something about the bloated application size). And even though the code in the demo works, some very similar code from the pygments quick start doesn't.

Check out the live demo.

Python Silverlight/Moonlight 2 Xapping

tarn — Mon, 25 Jan 2010 18:06:00 +0900

Context

I was asked do a presentation about developing Silverlight applications to the Melbourne Python User-group. Hell yeah, I love participating in user groups but I obviously needed some content. Unfortunately I didn't know this local user-group existed, so I've haven't been and have little idea of number of people or the type of stuff they do. This has made preparing relevant content a little challenging, but hopefully I can find the right mix.

Given it's a Python user-group I felt I might look like a bit of dick rolling with Windows and doing a demo with an environment and .NET tools that some part of the group couldn't/don't use. Besides I've always felt the Python language support and being able to build on any platform is a very compelling reason for non-.NET developers get into building Silverlight applications.

I don't yet have a Mac (I'm really starting to want one) so my other option was to do the demo on Linux. This presented some difficulties and some opportunities. Firstly I'm still a bit of Linux noob and, although I'm wrapped to have learned a lot in the last couple of weeks there has been some pain and some compromise.

Also Microsoft don't have a Silverlight browser plug-in for Linux, that support is provided by Moonlight. Moonlight is currently at version 2 and although it supports some of the features of Silverlight 3 it doesn't have the key version 3 features which have really improved the Python development environment (namely IronPython 2.6 support) and application deployment (namely allowing static assemblies that add dynamic languages support to be downloaded independently of the application and therefore cached in the browser).

Plugin and development support on both Mac and Windows is much better, but I'm happy here sacrificing some of the newer features for inclusivity (and apparently inventing words). We're targeting Silverlight 2/Moonlight 2 and effectively using no development tools besides a couple of Python scripts which I'll get onto.

Dynamic Language Silverlight Applications

Silverlight applications are packaged into binary files with an "xap" extension. It turns out these packages are just standard zip files containing all the "stuff" needed by the application. The minimum "stuff" needed in dynamic language application is;

An XML based application manifest file (app.manifest) that describes the target runtime, language type, entry point and some other information needed to run the application
A collection of assemblies that add dynamic language and Python support to the Silverlight runtime.
A plain text file containing the your Python script to make the application dance.

The package can contain more Python scripts, binary assemblies (which can be imported as modules) and XAML files (which are declarative xml type format typically used for describing user interfaces). Resources such as images and video can also be included in the package but it usually make more sense for the application to download them independently when they are required.

Serving the application on a webpage is also quite straight forward. An object tag in the html has attributes to tell the browser the URL to download the application and that the Silverlight/Moonlight plugin should be used to run the application.

Helper Scripts and Templates

With a couple of templates and Python scripts we can get started developing Silverlight applications in any environment with that has Python and a browser with a Silverlight/Moonlight plug-in.

xap.py builds xaps by adding all the files from a source code directory and an assemblies directory into xap file and writing it to disk.

http_server.py is the simplest python server which just serves all the files in the current directory on a specified port.

index.html contains the object tag referencing sample.xap and a tiny bit of Javascript to catch, format and display critical error messages that may be caused by the application.

app.manifest is a template with everything needed to describe a typical Python Silverlight application. The template should be fine as-is unless additional assemblies are required or you want to change the script entry point. The template is surprisingly small and undaunting.

app.py is the script entry point of the application.

assemblies a folder with all the additional assemblies need by the Silverlight runtime to run Python scripts.

Packaging and Running an Application

So you've skipped the previous three sections and jumped straight here. I don't blame you. Here's how to get going with the script and templates described above.

Download this zip containing the assemblies, scripts and templates and unzip it someone, anywhere.

From the terminal/console/command-line go the the place you unzipped it to and run the following command to build an xap.

python xap.py samples.xap assemblies source

A sample.xap should be generated in the same directory.

To fire-up the basic web server just run

python http_server.py

Then just fire up a browsers and go to http://localhost:8000/ and you should see the sample Silverlight application running.

That's it. Now you can edit app.py to add functionality and repeat.

Going further

Clearly this a very bare bones approach to development which has very limited development and debugging tools (and a server that can only serve files). There is more tooling out there, but surprisingly litte for Python developers.

MonoDevelop 2.2, the Moonlight SKD and Python bindings may provide some sort of IDE support but it's only just been released, it is not easy to get up and running and I not sure if it even supports packaging or debugging of dynamic language Silverlight applications.

You might think Visual Studio 2008/2010 would provide significant additional support, but as far as I'm aware only the Windows Debugger which is free in the Windows SDK is much help.

Chiron is a .NET tool provided in the Microsoft Dynamic Language SDK which can package and server XAP's much like I've described above. I found I could compile it on Mono, but couldn't get it serving files. I don't think your missing much here.

XamlPad is my favourite free XAML editor, but it doesn't work on Linux. Blend and Expression Studio are pay to play Microsoft tools for designing and developing XAML. Maybe someone should write a good one in Silverlight.

IronPython can be downloaded and run on Mono.

Some Thoughts

It's easy to get started writing and deploying Silverlight applications in Python on any platform but I think it's fair to say the tooling and support is just not there yet. On any platform.

The Silverlight 2/Moonlight implementations of Python are still missing large parts of the Python standard library, which will come as a bit of a shock to Python developers and makes it difficult to import modules that depend on it. Almost the entire Silverlight 2 Base Class Library is accessible through Python but although the Python/.NET integratation is very impressive, working with the BCL does seem to limit the expressivness of the code.

Silverlight 3 appears to have some additional built-in support for interacting with the application at runtime through a Python REPL console, it supports IronPython 2.6 (which implements a lot more of the Python standard library) and has additional deployment improvements discussed above.

Finally

I hope to do another post before the meeting with some examples I'll be showing, but as always I won't promise anything (in case I get distracted writing a Socket Server in Erlang I've been thinking about). In the mean time I do have some examples on PythonSilverScripting that can be run from the browser.

I'd love to know the tools other people are using to develop Python/Silverlight and thoughts people have of it. If your in Melbourne and your at all interested you should come down to the meeting.

Update 26/01/10: Updated scripts and templates to work in Windows

Here is a zip with the bits and pieces.