Patric Fornasier's Blog

ContainerLess

2011-04-05T11:36:00.003+01:00

I've recently worked on a project where we built a relatively small web application for a client. There was a hard release date, which gave us less than 4 weeks to build and deliver the application. There were several things we did to ensure that we would be able to release on time. One of the them was making sure very early that we could deploy the new application into a production-like environment. During the first few days, we therefore invested time in building infrastructure to deploy the application automatically and continuously (i.e. after each build) to a production-like environment. Going through this process was useful in several different ways:

it made us think early on in the project, what will be required to run the application in production (e.g. operating system, software, ports, certificates, etc.). This gave us sufficient time to react to unexpected obstacles;

by automating the deployment process early, we reduced the number of times we had to do this activity manually later on;

seeing the application work in a production-like environment was encouraging feedback that we would be able to release to production whenever our client required;

deploying the application while it was still very simple allowed us to focus on the deployment process without getting distracted by problems caused by the application.

Ditching the Container

When we first tried to deploy the application, it was not much more than a Hello World web application. We decided to use Jetty as a web server, since we had used it many times on projects before - both standalone and embedded. Whenever we used Jetty in the past, however, we would normally build a WAR file or a WAR-like directory structure, write a web deployment descriptor and finally deploy or load the artefact into Jetty.

Having spent a bit of time working on a Ruby on Rails application recently, the deployment process used in the past seemed cumbersome and complicated - even more so given the simplicity of the application at hand. That's when we realised that what we cared most about is the application itself and to a much lesser extent the container. In fact, we didn't care about the container at all. All we cared about was that our application would be able to serve incoming requests. At some point, we started to think of the application more of as a self-contained program, which just happened to expose some of its functionality via HTTP. The diagram below illustrates this mind-shift.

Consequences

The biggest change was the mind-shift. It took some time to get used to the idea that the application is no longer "contained" in something else, but that it can be "self-contained". But there were also a number of other practical consequences that this change entailed:

simplified development process. It was very easy and fast for developers to run the application in the same way as it would run in production;

simplified testing. It was very easy to start and stop the application for automated functional tests;

simplified packaging and deployment process (e.g. no web.xml, no war file, no prescribed directory structure, etc.);

simplified application start (i.e. simply execute a Java program);

started using standard DI mechanisms to wire up Jetty as a dependency and injected it where needed.*

* Strictly speaking, you don't need to ditch the container to do this but we felt that the mind-shift we had gone through enabled us to see the potential for doing this.

Conclusion

Making this architectural change simplified our development and deployment process. It also simplified the way we thought about the application. The only price we paid was loosing the ability to deploy the application to different web server without making changes. Fortunately, this price was purely hypothetical because we had absolutely no need nor desire to use a different web server in production. On the positive side, we were now hiding the fact that we were relying on Java Servlets to realise our web functionality. In fact, seeing what some of the other communities are building (e.g. Sinatra, Node.js, etc.), I'd be tempted to try to write a web application that doesn't use Servlets at all. Part of it for fun, part of it to overcome some of the limitations inherent with the Servlet model.

Agile Talk in Bern

2010-09-02T21:14:00.002+01:00

Next Monday (September 6th), I'm giving a talk in Bern. I'll be talking about how our software delivery teams around the globe work and get stuff done.

The talk will start at 6pm at Restaurant Schmiedstube. More info and registration via Guild42.

Come along if you're in the area!

Using Spring Java Config to wire up Dependencies

2010-04-11T22:08:00.011+01:00

The following tests demonstrate how, in Spring, you can configure your beans explicitly using Java. I wasn't sure if the internal method calls would result in two instantiations of the dependency or not. Clearly, it works as expected, i.e. only one instance is created because the default mode for instantiating beans is singleton scope. The magic is called cglib.

Declarative Replacement of Annotation-Driven Bean Definitions in Spring

2010-04-11T21:54:00.003+01:00

If you are using Spring 3 and configure your dependencies explicitly in a @Configuration class, it's fairly easy to replace one dependency with another. All you need to do is declare two dependencies with the same bean name in two separate config classes. The beans of the config which is registered last to the ApplicationContext, will simply override the beans defined in previously registered config classes.

Things get a bit more tricky, however, when you try to override a bean that has been wired up automatically using annotations. In this case, simply registering a config class that contains a bean with the same name as the annotated dependency doesn't yield the desired effect. It's very well possible that using a JavaConfigApplicationContext might solve the issue but while JavaConfig integration with the latest Spring 3 is flakey, this seems not an option. In the meantime, here's an example that demonstrates a solution to achieve the desired behaviour:

Lessons Learned (Part 2: Performance Testing and Garbage Collection)

2009-11-13T14:27:00.018Z

This is a continuation of my previous blog post. The goal is to sum up some lessons that I've learned during the last couple of months while I was involved in performance tuning a large-scale distributed web-app.

Golden Rules

Although there are probably many more rules and good advice out there, these are the ones that I remember off the top of my head as being important:

Change one thing at a time

I often found myself very tempted to violate this rule. The problem with breaking it, however, is easily illustrated with an example: Imagine that you make two changes, c1 and c2, at the same time. If c1 results in a performance improvement of 20% and c2 in a performance penalty of 30% you'll get an overall performance deterioration of 10%. Consequently, you'll decide not to implement any of the changes, even though c1 on its own would have resulted in better performance.

Look at the system as a whole and fix the slowest running part

Even if you can make some part of the system thousands of times faster, it will not affect you application performance if the part you changed was not your primary bottleneck. For example, it doesn't make sense to optimise application code if the bottleneck is the result of a slow running database query. I'd even go as far as saying that it is harmful to optimise parts of the systems, when it's not needed. Firstly, it's a waste of time that could be used for tasks that provide more value. Secondly, making performance optimisations often introduces additional complexity at the code level. If you can't justify this extra complexity with a significant performance boost, don't do it. Of course, I'm not advocating against common sense and sound software design principles. For example, I know that making lots of fine-grained RPC calls is a bad idea, so I'll avoid it in the first place.

Don't optimise prematuerly, i.e. without measuring

This one probably goes hand in hand with the rule above. Don't optimise unless you can prove that it will have an effect on overall system performance. Again, this rule is not an excuse for not using sound software design principles.

Performance Testing Cycle

Keeping the above rules in mind, we continuously iterated through the following cycle:

Measure performance
Identify single bottleneck (i.e. pick lowest hanging fruit)
Fix single bottleneck
Verify performance has improved

Once step 4 is complete, the cycle restarts. Sometimes, we would loop through this cycle several times a day. Other times, one loop would take us several days or even weeks. This process essentially continued until our release target was reached.

Measuring Performance

We used JMeter to generate load against the application under test. We set up the tests so that the generated load would increase over time and therefore put the application increasingly under more stress. While running the tests, we measured a number of parameters. The most important ones were throughput, average response time and CPU utilisation.

Looking at charts similar to the ones shown below, we got a fairly good understanding of how much load the application under test could handle.

In the above charts, for example, you can see that, at some point, application throughput reaches a plateau while the average response time per transactions continuous to grow. At this point, the application reached some physical or logical limit that prevented it from doing more work. The challenge, of course, is to find out what those constraints are in order to increase throughput or reduce response times.

Identifying Bottlenecks

Bottlenecks created by hardware constraints are normally quite easy to identify. Usually, the symptoms are maximum CPU utilisation, reaching network bandwidth limits, etc. The solution is often to change and restructure application code. Identifying bottlenecks not directly created by hardware constraints is more difficult. Likely causes are slow running external systems, resource starvation, suboptimal configuration settings, etc.

In the last project, we eliminated the hypothesis that slow running external systems are constraining our system quite early by taking them out of the equation completely and using stub implementations instead. At the same time, this made our performance tests much more robust, reliable and faster.

Fixing Bottlenecks

In the first few weeks of our performance tuning initiative, we made quite a lot of progress. There were a large number of easily identifiable bottlenecks which were relatively trivial to fix. These included simple programming errors, unnecessary database calls, unnecessary network calls, slow running SQL queries, no caching where data was easily cacheable, concurrency issues, etc.

After some time, however, it started to get more difficult to identify bottlenecks. In particular, there has been one case that I think is worth writing about.

Garbage Collection
We had already spent several weeks trying to identify a bottleneck, which was not obviously caused by hardware constraints. Here are the things we noticed:

Throughput reached a plateau at point t
Response time grew significantly at the same point t
Hardware was far from being exhausted. CPU utilisation, for example, was about 60% at point t
Although total CPU utilisation was around 60%, one (of eight) cores was maxing out occasionally

The last point was indicative that there was probably a CPU-intensive task executing in a single thread, hence single core. One such task that we could think of was garbage collection. We verified this using Perfmon and found that GC was indeed taking up a large amount of processing time (up to 30%).

As a result, we did some reading on how .NET GC works. We've learned that, by default, the GC is optimised for standalone apps running on single-core machines (called Workstation GC). On multiprocessor machines, however, there is an additional GC mode available (called Server GC). The difference between the two is basically that the latter creates a separate GC heap and GC thread for each processor and that collection occurs in parallel. Here's the change we made to our configuration:


<configuration>
  <runtime>
    <gcServer enabled="true" />
  </runtime>
</configuration>

After making the above configuration change, the throughput of our application increased by almost a factor 3! At the same time, we were again reaching 100% CPU utilisation and average GC time was down to 2-3%.

Of course, this dramatic change meant that we were dealing with a completely new application profile. Consequently, we restarted our iterative cycle described above again from beginning in order to find the next bottleneck.

Conclusion

The fundamental prerequisite for doing effective performance tuning is to have a set of repeatable and reliable performance tests. Ideally, these tests are easy to execute, finish in a reasonable amount of time and give you rapid feedback with regards to how the application is performing. Also, you'll need an isolated environment, which allows you to deploy new versions of the application easily and frequently. This gives you a good platform to experiment with changes. Measuring the difference between these changes with respect to the overall application performance then gives you the ability to make informed choices.

RSpec and TextMate

2009-11-13T11:05:00.002Z

Just struggled to get the RSpec bundle working with TextMate. After installing the bundle as described on the RSpec site, I kept getting the following error message:

Library/Application Support/TextMate/Bundles/RSpec.tmbundle/Support/lib/spec/mate.rb:4:in `join': can't convert nil into String (TypeError)

After a few hours of fruitless searching on the web, it occured to me that I should maybe have a look at the file listed in the error message above (mate.rb:4).

File.join(ENV['TM_PROJECT_DIRECTORY'],'vendor','plugins','rspec','lib')

The bundle basically failed, because the TM_PROJECT_DIRECTORY variable was not set.

So, instead of trying to run the file in standalone mode, I pulled the spec file into a new project (⌃⌘N) and ran it again.

Et voilà. No errors this time round. ...and thanks so much for the helpful error message, RSpec-TextMate-bundle! *grrr*

Lessons Learned (Part 1: Remembering Waldo)

2009-10-29T11:01:00.001Z

Intro

I've spent the last couple of months trying to help improve performance and scalability of a large web-based system. Initially, the application could barely handle more than a handful of concurrent users, which was far away from the launch target of several million users per day. This will probably be the first in a series of posts, in which I'd like to talk about some of the more interesting challenges we've faced.

A bit of context first: The application was written in C#. The main technologies involved were WCF, MSMQ and SQLServer. Roughly speaking, the app consisted of a presentation tier (IIS/MVC), a business logic tier (WCF) and a data tier (SQLServer). Application logic was exposed through a large number of WCF service endpoints. Each service endpoint, in turn, exposed a similarly large number of fairly fine-grained service operations. Essentially, there were three groups of clients that consumed the exposed services: the presentation tier, other WCF services inside the business logic tier and a number of mobile clients (which I won't talk about here).

Intra-Tier Communication and Horizontal Scaling

Inside the business logic tier, there was quite a lot of communication going on between the individual WCF services. Initially, many of these calls were routed through the WCF stack. The rationale behind this initial design decision was so that - if needed at a later stage - some services could be run individually on separate machines.

It seemed unlikely, however, that this would ever happen. The logical conclusion of this thought was that services would communicate with each other via network calls, even though they would be running inside the same process. And even if the WCF service layer would be partitioned and distributed onto separate machines, what would happen if this still wouldn't give us the desired performance? Imagine you've got three WCF services: s1, s2 and s3. Assuming that s3 is the most hardware-hungry one, we could deploy s1 and s2 together on one machine and s3 separately on a dedicated machine. What happens, though, if the hardware onto which s3 is running is still insufficient? At this point, we could start scaling the service out horizontally by adding a load balancer and more machines each running a copy of s3. So, if we probably need to scale out horizontally anyway at some stage, what's the point of adding the overhead and complexity of network calls between services if they can be run in same address space? To emphasise this point, we measured how many WCF service calls we can make in a given period (using net:TCP binding) and compared this against making direct in-memory calls to the same service instance. The not unexpected result: throughput for the latter was about 350 times higher. Consequently, we went through the codebase and replaced WCF service calls with normal method invocations wherever possible.

In a recent email conversation, my colleague Martin Fowler drew an interesting analogy: "It's interesting that there continues to be this desire to distribute different functionality onto different nodes in the name of scalability when often the better route is to put all nodes in the same process and cluster the resulting app. This was exactly the wrong thought that distributed objects suffered from."

Inter-Tier Communication and Horizontal Scaling

The presentation tier was physically separated from the WCF services running in the business logic tier. Consequently, communication between the two tiers had to happened over the network.

Let's go back in time a little. Back in 1994, Waldo et al. wrote their excellent seminal paper called "A Note on Distributed Computing". In it, they argue that there are fundamental differences between in-process and intra-process calls in terms of latency, concurrency, partial failures scenarios etc. In the past, RPC systems have tried to abstract these differences away and make developers believe that there's no difference between calling an object in the same memory space or executing a procedure on a remote machine.

Don't get me wrong, I think that WCF is actually a pretty cool platform but, unfortunately, it also encourages people to continue building RPC apps in cases where other solutions might be more favorable. In fact, it makes it horribly easy to take a bunch of classes and expose them as remote objects. Calling them services doesn't mean that your application has now magically become service-oriented. Also, it doesn't change the fact that these now-called services are still remote objects including all the flaws that Waldo talked about.

Indeed, we've had to fix a lot of code where developers happily looped over hundreds or thousands of items in order to retrieve some data, unaware that in each iteration they were making a network call. It goes without saying that this had a significant impact on the performance of the system. After removing all unnecessary calls, we measured the overhead incurred by network communication again. On average, still more than 30% of our total service execution time was network overhead (i.e. serialisation, WCF, TCP, network latency...).

Looking at the chatty WCF service interfaces and the tight coupling with the code in the presentation tier it occurred to me that, in actual fact, we were not really building a distributed application but we were distributing an essentially monolithic application. Unfortunately, we didn't actually manage to change this one. I'm convinced, though, that a better way is to deploy two tightly coupled tiers together in the same process and then, again, scale out horizontally.

Remove Unversioned Files in Subversion

2009-10-12T12:50:00.004+01:00

When using subversion, I more often than I'd like to admit find myself in need of starting with a clean working copy. Doing a normal (recursive) revert, however, does only undo changes from files that are already under version control and not newly added files which have to be removed separately.

In order to remove all unversioned files from your working copy in one go, you can execute the following command. Think twice before using it, though ;)

svn st | grep "^\?" | sed "s/\? *//" | xargs rm -r

JSONP Proxy Server

2009-03-10T23:12:00.006Z

I was doing some JSON mashing the other day and wanted to use data from different web pages. XmlHttpRequest was not an option, because I wanted to fetch content from different domains.

As many web pages don't provide an interface that returns JSONP, I wrote a very simple proxy server in python that does just that. More specifically, the proxy forwards the incoming request to a hard coded url and returns the result wrapped into a callback function which is provided by the user.
Once the proxy is running, you can hit it with JSONP requests:Or, if you prefer doing it in JQuery, which allows you to then handle the callback in a closure:

VMware and Windows Update

2008-11-06T10:22:00.000Z

For some reason, Windows Update doesn't seem to work out of the box in VMware. It took me a while to find this so I thought I'll replicate the simple steps I had to take to get it working:

net stop wuauserv
regsvr32 %windir%\system32\wups2.dll
net start wuauserv

Resizing a VMware Partition

2008-10-15T16:11:00.004+01:00

So, you would have thought that 8 GB of disk space for your virtual machine would be enough to do a bit of .NET development (i.e. install Windows XP and Visual Studio 2008). Unfortunately, I was wrong and had to make my virtualised partition bigger. Making the virtual disk bigger is very straight forward. Resizing the actual (primary active) Windows partition, however, is a bit less trivial.

Here's what I did:

Resize the VMware disk file;

vmware-vdiskmanager -x 12GB /path/to/file.vmdk

Get knoppix iso (or other bootable Linux iso);
Use iso for your virtual machine's CD-ROM;
Boot your virtual machine (i.e. boot using knoppix); *
Use qtparted (or any other suitable tool on your chosen Linux distro) to resize the bursting partition.
```
sudo qtparted
```

* I had trouble hitting ESC fast enough to get into the boot menu and therefore had to tell VMware to slow down a bit:

echo 'bios.bootDelay="5000"' >> /path/to/file.vmx

Ant EchoPath Task

2008-10-08T16:38:00.006+01:00

A few days ago, my pair and I have spent some time burning in JAR Hell. As we tried to find out what was going on, we were reminded how cumbersome it is to get a classpath representation out of Ant. Fortunately, I don't have these kind of problems very often. Yet every time I do have them, I end up spending some time to figure out, how to get Ant to tell me what's on the classpath I'm using. Although there are a few blog posts that describe how to do that or even how to make the output actually readable I found it just too painful to go through the entire cycle from scratch each time.

As a result, I hacked together an Ant task, which will tell you exactly what your path looks like.

In your build file, just define a new echo-path task and then assign the path id you're puzzled about to the pathref attribute. For example:


<project name="echo-path" default="show-my-class-path">
<target name="show-my-class-path">
  <path id="my.class.path">
      <fileset dir="lib" includes="**/*.jar"/>
  </path>

  <taskdef name="echo-path" classname="net.sf.orcus.EchoPathTask"
           classpath="/home/patric/lib/echo-path-0.1.jar" />
  <echo-path pathref="my.class.path"/>
</target>
</project>

This will output something along the following lines:


show-my-class-path:
[echo-path] Actual path for 'my.class.path':
[echo-path] /home/patric/workspace/toy/lib/gwt-benchmark-viewer.jar
[echo-path] /home/patric/workspace/toy/lib/gwt-dev-linux.jar
[echo-path] /home/patric/workspace/toy/lib/gwt-servlet.jar
[echo-path] /home/patric/workspace/toy/lib/gwt-user.jar

The JAR containing the task can be downloaded here.

Update: In order to define the task, echo-path-0.1.jar must be available. You can either place it anywhere on your file system and reference it using tha classpath attribute like in the example above. Alternatively, if you don't want to specify the JAR location each time, just put echo-path-0.1.jar in your ANT_HOME/lib directory

Two Remarkable Videos

2008-09-04T23:12:00.005+01:00

I know, I know, both of these videos have been circulating around the web for some time but I have only just now watched them in full length.

I found them both - in their very own way - remarkable, deeply moving and extremely worthwhile watching:

Last Lecture: Achieving Your Childhood Dreams
Randy Pausch, Carnegie Mellon University

An Anthropological Introduction to YouTube
Michael Welsch, Kansas State University

Soya @ ThoughtWorks AwayDay

2008-06-15T20:23:00.000+01:00

Yesterday, I gave a talk on Soya at ThoughtWorks AwayDay. Some people have asked me if I could make the slides available and - even though they are basically just copies from previous talks - I have uploaded the slides here.

Also, you can download the original thesis here.

alt.net

2008-05-15T08:34:00.002+01:00

There is an alt.net gathering in a pub around the Clerkenwell area in London tonight. I am planning to go and see what it's like.

There's no fixed agenda, everybody is welcome and it's happening at around 8:30pm in:

The Crown Pub
Sekforde Street
London, EC1R

Private Method Anti(?)-Pattern

2008-05-06T22:25:00.002+01:00

Some (actually quite a long) time ago, Szczepan asked if private methods are an anti-pattern and I've been promising him ever since to post a reply. The reasoning behind his statement was derived from the fact that TDD purists often maintain that untestable code is very likely bad code. Private methods are hard to test and it therefore seems to follow naturally that they, too, are a sign of bad code. While I'm sure that this is true in some cases, I think there are a lot of good reasons to have private methods or classes.

Implementation vs. Behavior

Many developers focus too much on implementation details when they write tests. Ask yourself what you really care about a class: implementation details or behavior? It turns out that in most cases, one is only interested in the latter. So when writing tests, we should focus more on the public API of a class and not about its internal structure.

Modularization vs. Encapsulation

In summary, I agree with Szczepan when he says that there is probably a lot of code out there in which private methods could be factored out into public classes or methods which would, in turn, increase code modularity. Often, however, encapsulation and a lean public API might be favored over modularization. More importantly, modularization might not always be needed, and therefore complicate and bloat the codebase unnecessarily. If modularization is still required at a later stage, internal code can be refactored and made public without breaking existing code.

Are You Over-Mocking?

Stubbing and mocking techniques and tools, especially Mockito, are great and make perfect sense in many situations (e.g. faking a network connection, a database, a service etc.). Breaking up a public method that consists of n private methods into n public methods of n new classes only so that we can test that the methods are indeed invoked, however, introduces unwanted complexity and completely breaks encapsulation principles, without yielding any real benefits.

Don't Defend Yourself!

Private methods and classes are also a good thing because they allow to code non-defensively, which will make code simpler. As soon as logic is offered to the world as a public method, one needs to ensure that the implementation also caters for unexpected scenarios. If the implementation is private, on the other hand, it is easy to determine that a certain variable will never be null, for example, and can thus be safely ignored.

Public APIs are Contracts

The API of a class (i.e. it's non-private methods) represents a contract with whoever is going to use it. Sure, as long as the class is used only within a single source repository this might not be a big issues, because powerful refactoring tools make it easy to change the contracts. But what happens if code is being used externally - maybe because other people find it useful, too? It won't be possible to change public method signatures that easily anymore, because other code will have been built on top of them.

Reduce Public API complexity

Private methods and classes can greatly reduce the complexity of a public API. Compare the Java API and its source code, for example. You'll realize that a lot of complexity is encapsulated in private methods and classes, which is a good thing, because it keeps the public APIs complexity low(er(-ish)).

To sum up...

Although these are very much thoughts-in-progress, I think that in summary:

We focus too much on implementation details and not enough on externally observable behavior of a class when writing tests
We often create code that is unnecessarily modular, because we are test-obsessed
We often mock too much (probably as a consequence or mutual reaction of the above two points)
We often create APIs that expose too much and therefore break encapsulation

Google Visualization API & Gapminder

2008-03-21T17:25:00.004Z

Just checked out Google's new visualization API and it seems that you can do pretty cool stuff with it. I really liked this one, for example:

http://www.gapminder.org/world/

QCon, man boobs and random thoughts

2008-03-13T23:14:00.002Z

I just came back from QCon and my head still feels very fragmented with a number of random thoughts.

Jim and Martin's keynote called 'Does my Bus look big in this' mercilessly dissected current middleware approaches. It was and extraordinarily sharp-witted and entertaining talk in which they used their entire linguistic repertoire to pick on middleware vendors and conceive beautifully eloquent metaphors such as "man boobs" (instead of bluntly calling it fat, ugly, bloated middleware products).

Kent Beck's keynote was also quite interesting. He thinks that in the future we will write (even) more tests, deploy applications more frequently (apparently Flickr does it every 30 minutes!), work in teams that are more distributed, and solve more complex problems. He also believes that with the rise of a new generation of tech savvy business people, software developers will increasingly loose their 'wizard status' and need to invest more in their non-technical skills.

Venkat Subramaniam and Ola Bini both talked about using dynamic languages in Java and the JVM. Through JSR 223, Java 6 now includes support to run a range of scripting languages. I was very pleased to see Venkat demoing some Groovy stuff that ran embedded in Java because we did the same with some JavaScript code on a client project just a few weeks ago.

In his second talk, Jim moaned a bit more about Web Services and the Web's architecture. He showed some pretty interesting ideas that use existing Web infrastructure (i.e. no WS-*) for integrating applications and realizing business workflows.

Finally, I visited a birds of a feather session with Rod Johnson. His appearance, however, did not necessarily remind me of a sunny, friendly day in Spring... ;)

QCon 2008

2008-03-03T21:33:00.000Z

Just got a one-day conference ticket for this year's QCon in London. I'll probably be attending most of the SOA, REST and the Web sessions on Thursday, March 13.

Really looking forward to going =)

Installing Ubuntu from Memory Stick

2008-02-25T11:07:00.003Z

If you - like me - have an ultra-portable notebook (marketing slang for 'comes with no CD-ROM') and no external CD-ROM (why should you?), you might know that it can sometimes be a bit tricky to install an OS from scratch.

I was thus all the more positively surprised by how easy it was to install Ubuntu from a USB memory stick onto my machine. Here's basically what I did*:

*(this assumes: virtual CD-ROM => F:, USB stick => G:, syslinux => c:\syslinux)

Download Ubuntu ISO image.

Download and install Syslinux and Daemon Tools.

Plug in USB stick and make sure you have no more U3 stuff installed on your stick.

Format the stick:
format g: /FS:FAT /Q /X

Make stick bootable:
c:\syslinux\win32\syslinux -s G:

Mount Ubuntu ISO image to drive F: (e.g. using Daemon Tools)

Copy some files from virtual CD-ROM to stick:
xcopy /e /h /k f:\*.* g:
xcopy /e /h /k g:\install\*.* g:
xcopy /e /h /k g:\isolinux\*.* g:
copy g:\casper\vmlinuz g:
copy g:\casper\initrd.gz g:

Rename a config file:
move g:\isolinux.cfg g:\syslinux.cfg

Edit syslinux.cfg by removing all /casper and /install references. (Not necessary for Ubuntu 9.x)

That's it. Adjust your BIOS boot settings and boot from the stick.

Mockito

2008-01-15T12:06:00.002Z

My colleague Szczepan has just released a new mocking library called Mockito (I leave it up to your imagination to picture the situation in which he came up with the name...).

Essentially, Mockito is built upon EasyMock but seems to provide a more elegant approach and simpler API to testing using mock objects than the latter. For example, I always found the EasyMock's recording and replaying stages quite cumbersome. Not suprisingly, I'm thus really glad to see that Szczepan came up with a simpler solution in which stubbing and verification are separated into different steps. This means that you stub your objects first, then use them, and finally verify only the interactions (with the mock objects) that you're interested in.

The following sample code is from the Mockito project site:
import static org.mockito.Mockito.*;

//let's mock a List!
List mockedList = mock(List.class);

//stubbing
stub(mockedList.get(0)).toReturn("first element");

//using mock object
mockedList.add("one");
mockedList.clear();

//verification
verify(mockedList).add("one");
verify(mockedList).clear();

I really look forward to using it more and finding out if it indeed makes testing easier. It surely does look promising =)

Implementation Patterns

2007-12-08T12:45:00.000Z

A couple of days ago I read Kent Beck's fairly recent book called Implementation Patterns. I especially enjoyed the more philosophical chapters at the beginning. In this part of the book, Beck explains why he thinks it's important that developers should try really hard to write good code.

In terms of books relating to software engineering, I normally prefer slim books over weighty tomes but found this one - which belongs to the former category - by and large a bit too laconic. In fact, one could probably argue that it is not a lot more than merely a very good introduction to programming in Java.

Nevertheless, I had a good time reading it and here are the key points that I'm taking with me:

Much more time is spent reading code than writing it.
Code that's easy to understand is more maintainable.
Try extremely hard to come up with concise and meaningful names for classes, methods and variables.
First and foremost, optimise code for readability not performance.
Don't restrict yourself to a specific case if you can easily be more generic (e.g. use Collection instead of List if your code doesn't require index-based element access).
Don't make members more visible than they absolutely need to.
Keep data and logic as close together as possible (i.e. encapsulate).
Don't flinch from creating one-line methods if they increase readability (e.g. write void setLoadedFlat(); instead of loaded |= LOADED_BIT;).

[Probably my favourite part in the book:] Don't "thoughtlessly obey programming folklore" (e.g. don't always stubbornly use only one return statement if you can use guard statements such as:

void compute() {
  Client client = getClient();
  if (client == null)
    return;
  Request request = client.getRequest();
  if (client.getRequest == null)
    return;
  process(request);
}

Implementation strategy is the extraneous information most often included in method names (e.g. use Poll.vote() instead of Poll.incrementCounter()).
Express as much information as possible through code and structure rather than comments.

Although most of these points seem trivial, I actually catch myself continuously violating one or the other. The book encouraged me question why I'm doing certain things one way and not the other. This, in turn, forces me to either come up with solid reasoning supporting my view or change my habits.

At last...

2007-11-29T12:59:00.000Z

Today I have at last completed the last bit of my thesis submission: I have posted a hard copy of my final thesis version to the Graduate Research School at UNSW and also submitted it electronically to ADT.

If you're interested, you can download the full thesis from the Soya web site.

A big thank you again to everybody for bearing with me and supporting me throughout the last 18 (or so) months =)

Ability and Responsibility

2007-11-21T23:02:00.000Z

The following three events triggered some (probably still incomplete) thoughts on my behalf:

A few weeks ago I had a discussion with a friend about whether I would write software for military projects or the like. I told him unmistakably that I wouldn't engage in such work, even if it would be at the cost of giving up my job.

Last week then, I read Kent Beck's new book called Implementation Patterns. While reading it, a passage caught my attention in which he - in one breath - associates coding with responsibility.

Today finally, I read a blog post which partially records a talk given by ThoughtWorks founder Roy Singham. In a similar way to the above, Roy defines his position relating to coding, morality and military projects.

A number of other people in this world including me have been very lucky, because we were allowed to grow up in a safe and developed environment. We have been fortunate enough to go to university and learn about many things from previous generations. We have been given lots of time, support, options and opportunities. Moreover, it seems like we've also been given the choice to decide what to do with all these gifts.

So, what do we do with all of them? I believe that we can indeed leverage them to do really, really good things, such as making people's lives better who haven't been as lucky as we've been. At the same time, we could likewise misuse those gifts to do an awful lot of damage. Think about it: surely it wasn't somebody who didn't know how to read who invented the most sophisticated and deadly weapons of mass destruction; surely it wasn't somebody who didn't know anything about higher mathematics who designed and created the latest ultra precise killing high-tech military weaponry. I'm sure you can think of many more examples.

What I'm trying to say is that many of us have been given extraordinary gifts that can be used to achieve extraordinary things. In both ways, good or bad. The higher our abilities, the bigger the effect we can have on our environment. I think that based on solid moral grounds using these gifts in a good way is in fact not a choice, it's a responsibility.

Finding Duplicate Table Entries

2007-11-19T22:49:00.000Z

The following piece of SQL is pretty handy for finding duplicate entries in a table. It will return all rows that have a single or aggregate column value which exist more than once in the given table.

select column_x, column_y, column_z
from my_table
group by column_x, column_y, column_z
having count (*) > 1;