#dev-test-ops

Future of ChromeWatir

2012-04-09T21:00:00.002-07:00

I am reviving this blog again and as a first post I wanna write about future of Chromewatir, a work I did sometime back. ChromeWatir started when Watir had no support for Chrome browsers but since then the work on watir-webdriver has invalidated much of need for this and Jarib has done excellent work moving forward.

I still get queries about ChromeWatir in this blog or through email after reading the blog. Please use watir-webdriver instead as it is continuously maintained and solidly built and I am discontinuing ChromeWatir.

And now for something completely different....

Introducing Multiverse Software Transactional Memory for JRuby

2010-06-27T02:58:00.000-07:00

I haven't written for quite sometime. I have been busy working on one of areas I really love, Concurrency. People who have followed me for sometime in twitter or heard my talk in Rubyconf about Concurrency Paradigms with Hari would have understood this. Recently I have been exploring alternate forms of concurrency other than the normal locks, mutex and conditional variable ones.

I love the message passing concurrency for its simplicity in removing the concept of shared memory (Still doesn't mean we will not have state dependence :). But there are at times when we need shared state and safe access to it. This problem of strategy for accessing a shared state among multiple threads of execution has been already solved in database world. Transactions gives a really good answer for this. Currently I am working on using Multiverse STM in JRuby (As earlier I had used Clojure). This time it is a complete Software Transactional Memory with support for atomic (transactional), retry (blocking) and orElse (choice) primitives. Multiverse is the transactional Memory on JVM and is the one behind Akka's STM. It uses TL2 algorithms (which has global version clock and commit time locking) as well as flexible to accommodate other algorithms.

Check out the code in multiverse git repo git://git.codehaus.org/multiverse.git. There is a nice implementation of Dijkstra's Dining Philosopher's problem in it using atomic and retry...

Let me know your thoughts. Look out for more posts on the internals of the STM :)

Continuous Integration - Stop lying to me

2010-05-21T08:05:00.000-07:00

Here is the point. If your Continuous Integration build isn't validating the system as it will be deployed in production from end user perspective, it is not telling you the truth.

If the continuous integration build is green but your packaging or deployment is broken, it is still lying.
If the continuous integration build is green but your production configuration has broken, it is still lying.
If the continuous integration build is green but you find 20 days later after deployment that the assumption you made about session has started creating problems with load balance, it has lied big time to you.
If the continuous integration build is green but if the functional tests run against a deployment optimized to make it work and different from production, it is still lying.

The above points illustrate that a lot of teams consider running unit tests and functional tests on a utopian or development environment mirrored build environment. This is equivalent to running the tests before you check in from your pairing station. The result is not completely useless. CI is still telling you the valuable information that the code you wrote has not broken anything.

But if you don't have anything in the CI build which validates the code and assumptions you have made, against the production mirroring environment from the point of view of end user we will be missing very important piece of information. It basically means even though the CI build is green the team only has a false confidence that the system they are building will work as intended from the end users point of view (Both from functional point as well as deployment point).

CI build should be end to end. They should run unit tests as a developer does. Should deploy to a production mirroring environment which validates packaging, deployment scripts and configuration and run functional and load tests against that so that we get a realistic understanding.

I also understand this is a challenging activity. But unfortunately on the other hand the price we pay for not having this infrastructure in place is actually heavy as we won't have our assumptions validated in time and need to resort to hacky methods when we come to understand that the decisions we made don't work in the real world as well as we expected.

oh.. And important point. When you check in code on the build server side don't update but always recreate the code from scratch. That way it is easy to find if we have tests passing because of stale assemblies lying around even if we have accidentally deleted them :)

What is the ultimate nirvana of this approach? Continuous Deployment. The best way to understand that things work as we have expected to put deploy them to production. And if we make it a non event it is the best kind of test we can ever have. Continuous deployment is a much bigger subject and may need another blog post for discussion.

Let me know your thoughts and experience about this.

Dang!! I should have looked before I lept

2010-05-12T08:38:00.000-07:00

Old saying but mostly true (Mostly not always.. :). Let me be first clear up my opinion about Selenium. It may have been successful in a lot of project, it is now backbone for companies like Saucelab, it may be one of the first successful open source project. But still in my opinion Selenium is one big hack. I mean both its api and internals... (No flame wars.. I mean nothing wrong. Just saying it is not soundly structured). And when ever I see problems with selenium (as in my current project) my thought process goes like this... How can I make this better and the first thing come to mind is let's replace with Webdriver. Unfortunately my current project is already running for long time before I joined so no chance of Watir in it :(.

The reason I am writing this is to share my experience of this journey I undertook to replace a test code base which relied on Selenium with Webdriver. Webdriver in my opinion is one of the sound code bases I have seen equivalent to Watir. They have nice design, good abstractions, solid internals and it is fun to work with the API. Coming to my current code base... It follows the pattern of lots of test code base I have seen before.. Abstraction from application using page objects but none to the underlying driver. This may be because most of the time it is not natural to change the test driver in a project. But here we are with the need.. To make functional tests stabler and faster. Without abstraction with underlying driver the driver code would have leaked through out the test framework. Lesson - Remember test or production, properly abstract from third party stuff.

I started with building a adapter layer (Webdriver backed Selenium) between test cdeand Webdriver as we wanted to keep the test code constant. Things went pretty smooth for simple stuff. Lots of strategies, webdriver functions are simpler. But then I started hitting some road blocks. Most of this is not because Webdriver is lacking but because Webdriver and Selenium are two different things. Sometimes there is no direct equivalent. Or we have to bend a lot to find an equivalent. Also the reference implementation I was using was in Java which again posed some problems as I had to build a similar functionality in C#. Although we can jokingly say that change all the starting lower case in java to upper case will make it C# in a lot of cases there is not direct equivalent again... Lots of reading, lots of research and making mistakes and correcting them are the way to solution. (Lesson - It will not be just an adapter as we say when we start....)

Also another problem and a major one I hit was CSS selector. There is no css selector implementation for InternetExplorer in Webdriver. I probably may have missed out the documentation somewhere but looking at the code base we had css selectors everywhere (Lesson - Research your current landscape before jumping into a solution). I don't have a direct solution in had but as we needed this I used Ian Bicking's Css_to_Xpath python code called from C# using DLR (Because I had used this css_to_xpath before and found it to be reliable. Didn't want to pull some sample C# code from internet and patch things up). Not a great solution but it seems tow work fine till now except for really complex css selectors (I will rant on css and xpath later). So had to hunt down and convert some css selectors to xpath.

I had learnt lot more lessons doing this conversion than I am putting here. Good news is I have convinced people to open source this WebDriverBacked *adapter* so other's can use it directly :)

Also it is important to remember that at times we should just say "screw it.. I am gonna do this". Just take the risk and work with it. I am in the final phase of this work and things seem to go relatively smooth. Now that I am back on the testing side for sometime I will write a lot about tests. Next one coming will probably be about Css selector (A spicy one :)..

I will update this post with the repo url once the adapter has been open sourced. Please let me know your thoughts with comments.

Simple Reactive Programming in Ruby

2010-02-23T22:22:00.000-08:00

Reactive Programming is an interesting programming paradigm based on propogation of changes, data flows. A programming model based on side effects, I think this is especially useful when we need to control the state of one entity based on what goes on in outside itself. For example take the equation


x = 1; y = 3
z = x + y
puts z #<= z = 4

Since z depends on x and y, what will happen when we update value of x or y. z remains same as 4 because z is already computed and stored using the values of x and y. Now if the values of x and y changes z will still remain the same.

But what if you want z to get adjusted depending upon x and y. This is more common problem in everyday programming than we think. Imagine you have to update the GUI based on what is changing in underlying model (MVC?) or take an example of spreadsheet where changing a value in a cell recalculates some other cell based on a formula.

Here is a very simple example of Reactive Programming in Ruby using EventTarget.
Here in this example value of z is x + 2. We evaluate z by setting an initial value for x and then later we change value of x and observe the value of z changed as well.


require 'EventTarget'

module Rx
  class Change
    attr_accessor :old, :new
  end
  class ObservableValue
    include EventTarget
    def initialize(value, on_change)
      add_event_listener(:change, &on_change)
      self.value = value
    end
    def value=( v )
       change = Change.new
       change.old = @value
       change.new = v
       @value = v
       evt = Event.new( :change, change )
       dispatch_event( evt )
    end
  end
end

if $0 == __FILE__
  z = 0
  on_change = Proc.new {|e| z = e.change.new + 2}
  x = Rx::ObservableValue.new(3, on_change)
  puts "Initial value of z is #{z}" #<= z=5
  x.value = 5
  puts "Current value of z is #{z}" # <= z=7
end

This is a very simple and contrived example of Reactive programming but I hope this illustrates the point. The concept is way more powerful and is (Functional Reactive Programming) available in a lot of functional languages like Haskell. Take a look at Reactive Framework in .Net and Javascript, Traits, Trellis in Python or Cells in Common Lisp for more understanding of how Reactive Programming works.

Let me know your thoughts and happy hacking :)

Software Transactional Memory in JRuby - Redux with Cloby

2010-02-19T03:35:00.000-08:00

After I wrote about using Clojure's Software Transactional Memory infrastructure in JRuby in my previous post. @headius has written an awesome wrapper Cloby in Java to use Clojure STM in JRuby and writing Ruby objects which can be used as Clojure Refs.

Here is an example code of using a Ruby object in STM.


require 'clojure'
 
class MyClojureObj < Clojure::Object
  def initialize
    dosync { @foo = 'foo' }
  end
 
  attr_accessor :foo
end
 
obj = MyClojureObj.new
puts "obj.foo = " + obj.foo
 
begin
  puts "Setting obj.foo to 'bar'"
  obj.foo = 'bar'
rescue ConcurrencyError
  puts "Oops, need a transaction"
end
 
puts "Trying again with a transaction"
dosync { obj.foo = 'bar' }
puts "Success"
 
puts "obj.foo = " + obj.foo

Some pretty good abstractions of using dosync and Refs in STM. So how do we use it?
Clone the code from http://github.com/headius/cloby/blob/. Build ClojureLibrary.jar with clojure and jruby.jar. Keep ClojureLibrary.jar and clojure.jar in classpath.
And here is the bridge I wrote between your awesome code using STM and ClojureLibrary.jar


#!/usr/bin/env jruby

require "java"
require "ClojureLibrary.jar"
require "clojure-1.0.0.jar"
include_class "org.jruby.clojure.ClojureLibrary"

clj_lib = ClojureLibrary.new
clj_lib.load(JRuby.runtime, true)

Now its pretty easy to use Clojure's STM. No raw usage of LockingTransactions or Refs. Enjoy Cloby... :)

#Concurrency - Software Transactional Memory in JRuby with Clojure

2010-02-18T06:14:00.000-08:00

Still playing with JRuby and concurrency... For me JRuby is one of the best implementations of Ruby VM. It has this amazing property of taking the most boring Java projects and turning them into something exciting. Not that clojure is boring but on the other hand it is one of the most wonderful languages I have worked with. Being a big fan of Lisp Clojure is a dream come true.

This post is about my exploration with JRuby and Clojure more in terms of how to use the Clojure's STM infrastructure. Whenever I find an interesting Java library the first thing I do is to import it into JRuby and play with it for sometime. After working with Clojure I wanted to use its STM in Ruby and finally found time to do it. The code is really crude as it explores very basic functionality of STM. May be as I keep working I will try to build something around this and who knows it can end up as a library for JRuby :)...

Instead of defining STMs myself - Here it is from Wikipedia..
"In computer science, software transactional memory (STM) is a concurrency control mechanism analogous to database transactions for controlling access to shared memory in concurrent computing. It functions as an alternative to lock-based synchronization. A transaction in this context is a piece of code that executes a series of reads and writes to shared memory. These reads and writes logically occur at a single instant in time; intermediate states are not visible to other (successful) transactions."

As most of you know the problem is with sharing the state.. In this example my_account balance. Imagine when two different threads of execution trying to update the value at the same time. Disastrous results may ensue. So the typical approach is locking of the shared resource. Problem with locking is it effectively limits concurrency as well is too hard to get it right. So we have started looking for other models of concurrency. Actors, Dataflow and STMs are some of the popular form. As you see in the example let's say there is a shared resource. If we need to update it we can't update it in a separate context. We need to run this as a part of transaction and as database transactions it is all or nothing.

In Clojure's term a ref can be updated only inside the context of a transaction. Else it will throw IllegalStateException. As well the ref getting updated will not be visible to outside work till the end of the transaction. The transactions are composable which is difficult in cases of threads and locks. Also as the example shows STM is a form of optimistic concurrency. Everyone can update the shared resource but whoever finishes first and commits wins. Others fail because the state now is different from what they started with and they can try.


Code snippet - Clojure STM in JRuby on Snipplr

The code is almost self explanatory and simple. Using clojure STM is pretty simple and straightforward. And this will help us to effectively manage concurrent access of shared resource by multiple threads.

Let me know your thoughts...

JRuby + Jetlang => Awesome Message Passing Concurrency

2010-02-16T00:09:00.000-08:00

It has been quiet sometime since I played around with JRuby.. The last I seriously worked on JRuby was during schnell-jruby age :). And now that celerity has taken the main stage I was back in MRI land for a long time. I recently got a chance to return to JRuby land and play with it for sometime.

Interested in concurrency I tried to work on Software Transactional Memory in JRuby with Akka... But as things didn't go great I was searching for something else interesting to work on and I found Jetlang. Jetlang has been in my interest list of sometime but I never got a chance to work with Java.

Jetlang is based on message passing concurrency. It has the concept of fibers which are independent threads of execution and channels which are conduits attached to fibers. We can pass messages to a fiber through channels and when a message is passed into a channel to a fiber the on receive callback is called with the corresponding parameters..

Following are two examples illustrating the use of fibers and channels for message passing concurrency. The last one is the popular ping pong example used for illustrating message passing across fibers :).


Code snippet - JRuby Jetlang Example on Snipplr


Code snippet - JRuby Jetlang Pingpong Example on Snipplr

Writing code in Erlang and using Jetlang based message passing concurrency has made me think that... Can Actors (and Message passing fibers) be considered as ideal representations of objects? Isn't this how we want object oriented systems to behave?
Let me know your thoughts...

Are we stuck in a Bratty Programming Model?

2010-01-18T02:14:00.000-08:00

This is more of an essay or more of my thoughts than a blog post. So brace yourself to read some lengthy writing :).

Warning - Thoughts are evolving and so may not be complete as you think. If you can help it to move in the right direction, I welcome you.

Ever had your day spoiled by a kid who demands what he needs and doesn't budge until you give it to him (cries, pleads, rolls on the floor and on extreme cases punches and kicks :). The problem with him is he won't let you do anything else till he gets what he wants. But how does it relate to programming? I think most of our programs behave in the same way as the bratty kid.

Let's take a problem - Say we are working on some ticket reservation app. The need is - Given a ticket id, get the reservation info, do some processing on that and then go on to do some other important stuff.

Expressing the above spec in ruby:


reservation = remote_service.get_reservation_info(ticket_id)
process_ticket(reservation)
#Program wishes to continue to do some other important stuff.

The above code is simple and we have all written this code at some point in time. But there seems to be a problem with the piece of code. It demands for some information from a remote service and doesn't move from there until it gets what it demands. Also it doesn't let rest of the code to execute as well until its demands are met. It is not just for service calls.. Think of using the same mode for network calls, database access, file system operations, long running computations. All of them can become bratty and stop your program from moving forward.

Are programs supposed to behave like this? Is this a better method of writing code? For simple code of not much serious use, the above model is acceptable. But when we start moving to systems which need to grow in scale this programming model is not sustainable.

Programs are supposed to be magnanimous in nature. A line of code should behave in a way that if it is not able to get something for moving forward, should move out of the line to let others execute and wait for its input. How would that actually work out in code?

Now lets jump into code which can give a hint of how to solve this. Below is the same ruby code written in a different way.


remote_service.get_reservation_info(ticket_id).addCallback {|reservation| process_ticket(reservation)} # I will get the input when it is available and process it
# other lines of code executing when above code waits for reservation info...

So the above lines of code does the same as above semantically. But the difference between the two is in this piece of code the retrieving of reservation information and processing it doesn't seem to block the program from moving forward. I know just adding a callback doesn't solve the problem. There must be an underlying infrastructure to handle this. And this is where event loops come in. But I think you get the idea of what I am trying to say.

Event loops are a bigger subject of discussion. But we have been using them most of the times. Take GUI programming as example.

The I/O or more importantly programing models we have been using till now have predominantly been focusing being imperative (Do this step, get the result, then do that and continue...) but I think it's time we understand that this model can't help us long. If we need our programs to scale or even be reasonable we need to move from bratty to a magnanimous programming model. Blocking other's from moving because we still need something is not a very scalable model both in real life and programming :).

This is my rant of current programming models. There may be better ways to do things and if you know any please let me know. A lot of this essay is inspired by node.js work done by Ryan Dhal and EventMachine which made non bratty programming so much easier for me to handle.

P.S. Using event loops has a different set of constraints like new way of thinking (inverted control), different way of exception handling as well using different frameworks which we may not be familiar with.

DNS lookup Experiment in Ruby - Blocking and NonBlocking Modes

2010-01-16T04:35:00.001-08:00

Normal DNS lookup in Ruby (Blocking Mode)

require "resolv"

hosts = %w[www.yahoo.com www.google.com twitter.com github.com]

start = Time.now

hosts.each do |host|
 p "Getting address for #{host}"
 p Resolv.getaddress(host)
end

finish = Time.now

p "Time take to finish #{finish - start} seconds"

Execution Result -

"Getting address for www.yahoo.com"
"69.147.76.15"
"Getting address for www.google.com"
"209.85.231.104"
"Getting address for twitter.com"
"168.143.171.84"
"Getting address for github.com"
"207.97.227.239"
"Time take to finish 40.313232 seconds"

Non Blocking DNS Lookup in Ruby (Using EventMachine Deferrable)

require "rubygems"
require "resolv"
require "eventmachine"

class Dns
  include EM::Deferrable
  def resolve_hostname(hostname)
    begin
      ip = Resolv.getaddress(hostname)
      set_deferred_status :succeeded, ip
    rescue Exception => ex
      set_deferred_status :failed, ex.to_s
    end
  end
end

start = Time.now
EventMachine.run {
    p "Requesting DNS info for yahoo"
    dns0 = Dns.new
    dns0.callback {|response| p "For yahoo #{response}"}
    dns0.errback {|response| p "For yahoo #{response}"}
    Thread.new { dns0.resolve_hostname "www.yahoo.com"}

    p "Requesting DNS info for google"
    dns1 = Dns.new
    dns1.callback {|response| p "For google #{response}"}
    dns1.errback {|response| p "For google #{response}"}
    Thread.new { dns1.resolve_hostname "www.google.com"}

    p "Requesting DNS info for twitter"
    dns2 = Dns.new
    dns2.callback {|response| p "For twitter #{response}"}
    dns2.errback {|response| p "For twitter #{response}"}
    Thread.new { dns2.resolve_hostname "twitter.com"}

    p "Requesting DNS info for github"
    dns3 = Dns.new
    dns3.callback {|response| p "For github #{response}"}
    dns3.errback {|response| p "For github #{response}"}
    Thread.new {dns3.resolve_hostname "github.com"; EM.stop }
}
finish = Time.now

p "Time take for querying #{finish - start} seconds"

Execution Result -

"Requesting DNS info for yahoo"
"Requesting DNS info for google"
"Requesting DNS info for twitter"
"Requesting DNS info for github"
"For google 209.85.231.99"
"For yahoo 69.147.76.15"
"For github 207.97.227.239"
"For twitter 128.121.146.100"
"Time take for querying 20.311271 seconds"

The results are obvious :). You can see the difference between the blocking and non blocking modes and the effect they may have on your systems. I am also planning to write a DNS library based on NeverBlock which uses Fibers as a way to do non blocking concurrency.

EventMachine is an excellent piece of software which gives a lot of functionality to do things asynchronously. Especially the Deferrable pattern is an awesome way to do long running jobs or calculations and still not block the whole world from running. I have been exploring EventMachine for sometime since I started writing em-couchdb.

The code is written in a simplified way and may not be of direct use to production but hope it reflects the principles of value of doing things in non blocking way.

ChromeWatir + Watir-WebDriver Update [Important]

2010-01-13T02:33:00.000-08:00

Interesting development in Watir world but not a completely unanticipated one. I started playing around with using WebDriver as platform for Watir in the form of FireDriver and ChromeWatir.

Yesterday Jarib has released a Watir abstraction on top of WebDriver. I am planning to move the effort of working on ChromeDriver to Watir-WebDriver which has ChromeDriver in it.

Installation of Watir-WebDriver

> sudo gem install selenium-webdriver
> git clone git://github.com/saivenkat/watir-webdriver.git

Example code in watir-webdriver (ChromeDriver)

require "watir-webdriver"
browser = Watir::Browser.new(:chrome)
browser.goto "http://www.google.com"
browser.text_field(:name, "q").set "Watir"
browser.button(:name, "btnG").click

The code is same as Watir except the driver changes internally.
The repo is in http://github.com/saivenkat/watir-webdriver. Feel free to fork and enjoy...

What is your favorite code metric?

2010-01-12T09:53:00.000-08:00

You don't need to run any build or external tool to get this. It is always there when you code and changes actively as your code evolves. What would it be?

My most favorite is obviously "Line Numbers".

Line numbers are available in all IDEs but mostly not actively used as a code metric during coding. Switch them on and notice them actively when you code, they tend to have an interesting effect. Probably the classes and functions will be shorter, less bloated and concise. Based on personal experience :)

What is your simple metric to improve code?

EventMachine Client for CouchDB (em-couchdb)

2009-12-30T10:35:00.000-08:00

I have been crazy about NoSql databases for sometime. CouchDB is one of my favorite apart from redis :). I was looking for clients for CouchDB in Ruby and found most to be using Net/Http and blocking in nature. So I began my quest of writing an asynchronous non blocking awesome EventMachine based CouchDB client inspired by EventMachine::Redis client.

Here is a sample code to enjoy... It creates a database, saves a document inside it, reads the doc, deletes it and then deletes the database.

The current version supports database manipulation like creation and deletion and document manipulation.

Looks cool for the code I wrote in a day but still there are things to improve (a lot). One thing is the CPS style. Because the code is asynchronous and the EventMachine reactor loop will take control once the code starts to run we need to explicitly pass callbacks to actions. This can be fixed as it was done in EM::Redis using callcc to make the passing implicit.

Please feel free to fork and contribute and happy hacking :).

The code is available at http://github.com/saivenkat/em-couchdb

Simple (Or not so simple) Node.js Redis Client

2009-12-10T06:40:00.000-08:00

I love node.js. For some reason I am fascinated by V8 javascript engine and its performance. I am also a big fan of Event driven I/O frameworks like twisted or EventMachine, Javascript and nosql databases of which redis is one of the stars. For those who are not familiar with terminology Google them :).

Here is a small redis client I wrote in node.js. All it can do is to see if a key can exist or not :).

What good is the feedback for?

2009-12-09T11:46:00.000-08:00

What good is the feedback for if I am not going to use it to correct things? What good is a QA for sitting across my desk providing feedback if I am going to ignore it and continue what I am doing? What good is a showcase for if I am going to ignore my client's suggestions? What good is TDD for if I am not going to use it to evolve the design? I have heard a lot of people say the Agile is about short continuous feedback but the question comes back to why? Why we need feedback? Is feedback alone enough? Getting feedback is a way, part of the activity we are doing, not the end.

All these questions made me think about what it means to Agile? I always consider software development to be more of an exploration activity. We explore ways to do things better and along the way we learn new ways and also make a few mistakes. Working over the years my definition of Agile has evolved and my current one seems to be "Agile is a method which helps to reduce the assumptions we hold as fast as possible and make things concrete". When we work on a project we have base our work on a few assumptions or understanding of what needs to be done. They may be assumptions of what client is expecting, or assumptions about a design being evolutionary... But holding on to those assumptions and thinking that we are going down the right path is a way to disaster.

So how is Agile special in reducing the assumptions when compared to waterfall? Both help in that aspect of getting feedback except for Agile gives quick ways to correct our actions or find the right way. Waterfall delays it to the point where sometimes the cost of correction is too high.

Working closely with the client helps in making our understanding better or reducing the assumptions we make, TDD does the same for design, Continuous Integration and deployment helps us to reduce the assumptions we have on production environment. The examples are numerous but I hope you get the point.

So for me Agile is about reducing the assumptions and making things concrete so that we can continue working with confidence.

Let me know your thoughts.

Curious case of object initialization in python

2009-12-03T19:09:00.001-08:00

While working on writing some Async code for CouchDb connector using Tornado webclient I used the code of AyncHTTPClient as a doc. Reading the code I found an interesting piece of initialization method. This may not be anything new for a lot of python people but for me I found this pattern interesting. I had used python long time back while it was in 2.5.x time and I came back to it recently to work on UNICEF project. So my knowledge of python may not be as sharp as knife :)

So lets jump into the code directly. The initialization method as I remember in python is __init__. For those unfamiliar with python __init__ is the first method called when an object is initialized. Or in other words a constructor. Similar to ruby's init method. Pretty simple right?

But checkout the __new__ method. A simple example of usage of __new__ method for singleton pattern in python.

So how is __new__ method different from __init__? As seen from example __new__ gives a complete control of the object I construct. So when initializing the object the order of calling is
1. You try to create an object --> 2. __new__ called if available to create an instance --> 3. object initialized using __init__. Also __new__ can return an instance of the class you are creating. __init__ only does the job of initializing it and does not return anything.

I know a lot of you may know about this already. But I wanted to share what I understood. Let me know if you think there is a better explanation or I have goofed up something.

Eventlet based CouchDB connector

2009-12-03T09:27:00.000-08:00

This is an attempt to write a nonblocking CouchDB API based on Eventlet. Eventlet is a wonderful python library written by Second Life people based on coroutine based nonblocking I/O. I also tried using AsyncHTTPClient provided by Tornado framework released by FriendFeed people but liked the eventlet version more. So here goes... This API may grow to be a full fledged one in future.

Also look out a Ruby non blocking one based on NeverBlock :)

Code Swarm Visualization of Watir Project

2009-11-11T13:21:00.000-08:00

Presenting the code swarm visualization of Watir project. For those who don't know what is code swarm, it is an experiment in organic software visualization or what ever that means :). For me it meant visualization of activities on a code repo or project. So here is the one for Watir project. If you wanna watch it in Hi-def go over to Vimeo and take a look. Its pretty awesome...

Code Swarm visualization of Watir project from saivenkat on Vimeo.

Git - Branch Per Feature Experiment

2009-11-06T07:54:00.000-08:00

I am sure others are using Git like this before but these are my thoughts I collected over sometime using Git. I have been using Git for over a year now but mostly for my Watir projects where I haven't had a big chance to work with branches other than master till now. My current project has given me an opportunity to use Git in a different perspective. I have been observing that I almost never use master but always am in some branch working and constantly merging with master. And the branch I work on is something related to the feature I am working at that point of time or exploring.
To illustrate the point better, I thought a picture will illustrate the point better :).

As you see in the picture I am currently working on tagging feature but there seems to be other features as branches there. I am exploring the option of text search and well as bookmarking a search for projects. So if I want to work on some other feature I switch the branch and work. Once the feature I am working on is done I switch to master, merge the feature with main branch, pull from remote repo, fix any merge conflicts and commit the feature to master before pushing it to remote repo. Then the branch gets deleted. If any work needs to be done on the feature the branch is recreated.

This seems to be my work pattern for the past few months. Using branch per feature importantly helps in keeping concerns of different features separate while working on them. As well working on a project where spiking is important creating branches for spikes seems to be a good option for me. Sometimes when I want some parts of the spike may be just a commit cherry picking also helps me a lot. Most of this seems easy and possible now because the cost of branching and merging is less in Git. The same approach will be painful in VCS like svn.

This is just sharing of my experience. I am exploring better ways to use Git. Let me know if you have found good ways of using Git or similar DVCS.

Infinite lists in python

2009-10-25T06:55:00.000-07:00

My first post from Uganda :)

As I am working in Django now I think I will post more on python. I had worked on python before but that was long time back and mostly on system admin stuff. So working on python now for developing web based applications and getting to understand things like object orientation, functional programming, TDD, Ruby and more I think it has changed my way of working with python a lot.

This post is mostly about creating infinite lists in python. Infinite lists are an example of lazy evaluation mostly rooted in functional programming. The idea is we don't construct elements of list but what we do is give a way to construct the next element of the list and let it construct the list when we need (we still need to stop construction of elements depending on some condition). Infinite lists are also rightly called streams. In python, it is easy to use develop infinite lists using generator as shown in the examples below.

Examples of infinite list in python

Increment Example -

Fibonacci Example -

A simple PingPong example based on Erlang

2009-10-08T20:14:00.000-07:00

This is a test for Github's gist script embed in my blog :).

Setup script and shell command for Pylot

2009-09-26T11:38:00.000-07:00

I gave a talk at PyCon India on "Testing Web Applications using Python" which featured my favourite toolkit Windmill, WebDriver, Twill and Pylot. Think Pylot impressed the audience (Great work by Corey Goldberg :) and they started trying it out immediately. Unfortunately before figuring out that they have to download it from Pylot website, I think a vast majority of them searched at PyPi and tried using easy_install. As a result one of the questions I got was unavailability of any installation medium for Pylot. Seizing the chance for me to learn how to create a setup.py based installer I jumped on the task of creating an installer for Pylot. Also to make the usage of the tool easier I have added a command line command pylot which works the same way as using python run.py to run pylot tests.

So how to use this? Pretty simple. I have changed the directory structure a bit (Hope Corey doesn't get pissed off :). The pylot directory remains same except there will be a parent directly with setup.py. You can run the set up command as usual "sudo python setup.py install" (Yes mostly you may need to do sudo depending on where your python dist-packages are). Once you do this you will get pylot-runner into path and add pylot to the packages. Now you should be able to run any pylot test case with a pylot-runner. All the commandline options hold same as before "pylot-runner --agent=377 --xmlfile=awesome_website_tests.xml". Your results will be created in the same folder as where the xml test case is.

This is my first script of setup.py. Let me know if there are better ways to do this stuff. Also I will convert the installer into an egg based one but it will take few days for me to do it.

Gotcha - I have created, installed and tested this on my linux system and it works amazing. So I haven't added anything specific to windows which includes the recorder. I may do this along with converting this installer to egg but it may take sometime. The wxpython Gui gets bundled though. Only the recorder is left behind...

Download for Pylot with setup script and shell command - http://miscellaneous-downloads.googlecode.com/files/pylot_with_setup_and_command.tar.gz

Functional Test suite in C# or Java... Why?

2009-09-17T10:45:00.000-07:00

Every time I see a functional test suite being developed in a static language, I get to the same question. Why? I feel odd when I hear people that the development language is C# or Java and so the functional test suite testing the web application or REST interface also needs to be in the same language. I understand and accept the reason for unit test as TDD is a design activity and it is better to be in the same language. But functional tests? Why?

I also feel sorry for people in the team who spend time mangling a static language to build test suites and framework to make the whole thing work. Also the slowness of the feedback cycle as most popular static languages are compiled and so functional test suite also needs to go through the same cycle.

So here are my reasons to choose a cool funky dynamic language to write your application level test suite.

Easier development cycle. Typically dynamic languages have more essence to ceremony ratio. They are lightweight. They don't have the type noise which static languages have as we don't need that kind of information in our functional tests a lot. Also dynamic languages being less ceremonious is easier to pick up and start working.
No wastage in compiling time. Most of the dynamic languages (Actually all I know) are not compiled and so donot have that long compile cycle. Typically functional test suites don't run as an application and donot have the same considerations as them. For functional tests, the quicker they run better the feedback cycle.
Easier to build abstractions. It is easier to build abstractions like frameworks, DSLs or Fluent Interfaces in dynamic languages. Less noise, more essence and better abstractions :)
Better drivers and libraries - Almost all the drivers I know are in dynamic languages. Watir, Selenium, WebDriver (Use with JRuby or python :), Celerity. Also you have frameworks like Rspec, Cucumber and Robot framework already available to build your test suite better. You have HttpDrivers, XmlParsers, JSonParsers or whatever infrastructure or libraries you need are available as well.
Easier to experiment with. Dynamic languages are lightweight to work with. They also typically come with a REPL which is a good place to learn and experiment. If you want to know how to play with a REST based service, fire up the REPL. Import a Http library and start sending requests.
They have a great community to work with. Difficult to substantiate but I can challenge that Ruby and Python community are way ahead in catching up with new stuff than .Net or Java. Also if you see Github and find the major languages of open source projects hosted there :).

May be there are more reasons. If you can think of any add them to the comments to discuss.

If you are a developer or a tester in a team (I don't differentiate between both of these roles :), do yourself and your team a favor. If your development itself is a dynamic language, you lucky dog :). Else choose a suitable language of choice to write functional or end to end tests. Don't get into the dogma of using the same language as development language.

Configure your Visual Studio to run your tests automatically

2009-08-26T10:12:00.000-07:00

It all started after I saw Misko Hevery demonstrating this technique of running unit tests whenever he saves code in Eclipse at his Thoughtworks Geek Night talk and reading his blog post on it. I am a big sucker of fast feedback and always felt that faster the feedback, better it is. This technique to run unit tests, I felt is a big step towards it (Also my patience is too short for me to go and run the tests by navigating through menus :). Being a compulsive TDD guy, unfortunately in a .Net project and so forced to use Visual Studio (I have nothing to complain about. Move along :) every day, I felt the need for this badly. And so began my quest...

I was looking for a way to hook into the save event of Visual Studio to do something same as Misko's but couldn't find any till now. The next best thing for me is Post Build Event hook. Conveniently this is available in Visual Studio for every project to be configured in project properties. So on the test project properties "Build Events" section, you can hook into the pre and post build events to run some shell scripts.

Prerequisites - Have nunit-console.exe in path or better have it in your project as a dependency. I have it inside my project root folder inside bin\NUnit. To run tests using nunit-console you need to pass your test assembly path to that. Also check the build order in your solution. Typically it will be Main project -> Unit Tests.

Choose the project properties of the project which contains unit tests in your solution.
In the Build Event section of project properties, choose Run the post build event list box to be "On successful build" as I want my tests to run only when build completes without error.
In the Post-build event command line text area it is possible to write any shell script which will execute once the build is done. So refer to the nunit-console.exe here with your target assembly path (Your test project assembly). Conveniently here you have macro's to refer the parameters you need, so the job gets easier.Check out the screenshot of my configuration in post build event.

Now it is time to try out the cool thing :D.
Every time I refactor or add code or tests you build the code by ctrl + shift + b, I can find the build run and after successful build unit tests being run as a part of build.
The cool thing is if the unit test fails, it is a build failure :). So I make a small refactoring I build it and with in seconds I can see if I have broken something and fix it or do a ctrl + z if needed...

Check out the output from the build and unit tests.

My Red Green Refactor cycle has just got shorter :). I have done this in the simplest way I know and it works for me. Let me know if there is a better way to do this

Also I have done this as a part of my yet to be open sourced project FluentAssertions. No spoilers now but more on this soon...

Using Git As A Tool to Help In Refactoring

2009-08-25T09:54:00.000-07:00

I am a big fan of Git. Initially started as a urge to use the cool GitHub. But later when I started to realize the potential of Git itself the simplicity and power of this VCS amazes me.

As most of you know Git is a distributed version control system (VCS). For those who haven't used it or heard of it (Please come out of SVN dark ages :) please do a Google search and you will find a flood of links hitting you. What I am write speak here is about my typical usage of Git and how it helps me in day to day refactoring making it easier.

I am firm believer of RGR (Red Green Refactor) cycle. But beyond that I do a lot of micro refactorings like renaming variables, classes, moving behavior, adding tests etc as I thinking more about what I have written and finding better ways to express my code.

Now the problem when I used to use SVN is that I have only two choices. Either I pile up these small refactorings to commit in my local copy which I don't really like. Or commit each one individually and trigger the build for every small change I make. Also if I pile up and commit, at the time of commit how can I express my intentions of the commit I am making. I never like to put a comment in commit which blankly says "Refactoring and adding some tests.".

So how did Git change the picture? Git as most of you know is a two stage repository VCS. One repository may in remote server (incase of Github it does or a SVN repo if you use Git SVN) and other repository in your local machine which is also a complete repo of your source code. You can treat your local repo as a main repo and commit in it as many times as you want. This gives me a great advantage.

Every refactoring I do can be tested and committed to the repository (local one) individually. I can express my intent of the commit of every refactoring. When I feel substantial amount of work is done I can push to the remote repo and continue with my next item of work. Also when I push I can see it as a series of work I have done on a feature not as one bulky piece. Typically I have this code, refactor, test and commit cycle for every five to ten minutes when I use Git. This is one amazing advantage which I have when using Git. Similar feelings will be expressed by people who use Mercurial as well.

Also as an added advantage when build is broken and someone is fixing it I can still commit to my local repo and continue working and push my changes later. But we should be cautious not to pile up too much in our local repo that we distance ourselves with the mainline code base.

Similar thoughts on using Git and Github has been expressed by Mark Needham here.

Let me know if you have similar experience as this.