curious-attempt-bunny

Late to the Party

2009-11-07T23:25:00.000-08:00

I just worked my way through this fantastic game. Great stuff! Could it be that you haven't played Portal already?

Wow, Mac Wins

2009-11-07T10:28:00.000-08:00

I was just cleaning up my pile of computer bits, and I found a USB hard drive that I've not used in a while. After browsing the contents I felt it was time to wipe it clean. I wanted to format it to be Windows friendly just in case. Linux felt like the best tool for the job there, and a combo of fdisk and mkfs.vfat accomplished that pretty quickly. I wanted to find a good tool to test the disk for bad sectors (testdisk?) and that's when I noticed:

OMG. I haven't even installed Firefox in Ubuntu since installing 9.04 back in April! I just don't use Linux any more. Mac has won me over that completely.

Distributed Solutions: What is out there?

2009-10-23T19:40:00.000-07:00

I've been reading the Hadoop book mainly to learn more about the MapReduce approach to scaling solutions. Pig is interesting but not update-oriented. Hive sounds like the closest Hadoop tool but that's not right either:

"Hive is based on Hadoop which is a batch processing system. ... For Hive queries response times for even the smallest jobs can be of the order of 5-10 minutes and for larger jobs this may even run into hours." [more]

I'm looking for something that can distribute a large set of data across a number of machines, and then be able to let me coordinate processing the data in such a way that each machine works on whatever portion of the data it has held locally. That's why Hadoop was sounding so promising. The other requirement I have is that the data can be supplemented on the fly. Some lag is acceptable, but ultimately I think this eliminates Hadoop also as it's append-only approach is incompatible here.

A basic Distributed Hash Table that supports partitioning of the data would be perfect if it has some way of knowing what data it "owned" locally. Infinispan looks like it might be a good fit once it matures. Unfortunately "the ability to move the code to where the data is and execute it there" [issue] is part of the last milestone on their road map horizon.

Hmm. Infinispan, Project Voldemort, and Riak. They will all know what data they have cached locally. And that's half way to being able to execute a job in a partitioned way. The other half is either modeling some concept of which node as the "primary" cache of some data, or having a way of resolving any duplication of work between nodes. Hadoop solves this problem by having an indexer that keeps track of which nodes own what and where everything has been replicated to.

What's the solution for a technology that hasn't necessarily been designed with this consideration in mind? What do people do when they have outgrown their RDBMS and still want to be able to process large volumes of data for quick ad-hoc queries?

Firefox 3.5: Firefox doesn't know how to open this address, because the protocol (foo) isn't associated with any program.

2009-10-17T12:58:00.000-07:00

Humph. I'm not having any success creating a new protocol that maps to an external application in Firefox. I have these in my prefs.js:

user_pref("network.protocol-handler.app.foo",
  "/Applications/TextEdit.app/Contents/MacOS/TextEdit");
user_pref("network.protocol-handler.expose.foo", true);
user_pref("network.protocol-handler.external.foo", true);
user_pref("network.protocol-handler.warn-external.foo", false);

Have you got it working?

I've seen these: knowledge base entry, registering a protocol, and path not found error. This one I didn't get to the bottom of: whitelisting protocols.

Here's my about:config which shows my default settings as well:

Hadoop Streaming: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 127

2009-10-16T22:44:00.000-07:00

So, my most recent problem was that my hadoop streaming job was failing, and the tracking url was listing this exception under "Failed/Killed
Task Attempts":

java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 127
 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
 at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540)
 at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
 at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
 at org.apache.hadoop.mapred.Child.main(Child.java:170)

Digging around in the logs directory I found a fuller explanation:

env: groovy: No such file or directory
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 127

My groovy script was failing to launch - presumably because groovy wasn't visible on the path. My hacked solution was to hardcode the path to groovy in my script:#!/usr/bin/env /Users/user/Applications/groovy-1.6-RC-2/bin/groovyAnd then, voila! It worked!

Hadoop Streaming: Cannot run program

2009-10-16T21:55:00.000-07:00

I'm working from the Hadoop book, and I'm noticing some gaps. When I ran hadoop using streaming like this, my job hangs:

$ hadoop jar $HADOOP_INSTALL/contrib/streaming/hadoop-0.20.1-streaming.jar \
  -input movies.input.txt -output output -mapper movies.map.groovy \
  -reducer movies.reduce.groovy 
packageJobJar: [/tmp/hadoop-user/hadoop-unjar2417163781720808364/] [] /var/folders/Ht/HtruzsCeGAukVrRT16Q4+k+++TI/-Tmp-/streamjob7394763723023597419.jar tmpDir=null
09/10/16 21:13:46 INFO mapred.FileInputFormat: Total input paths to process : 1
09/10/16 21:13:47 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-user/mapred/local]
09/10/16 21:13:47 INFO streaming.StreamJob: Running job: job_200910162029_0008
09/10/16 21:13:47 INFO streaming.StreamJob: To kill this job, run:
09/10/16 21:13:47 INFO streaming.StreamJob: /Users/user/Applications/hadoop-0.20.1/bin/../bin/hadoop job  -Dmapred.job.tracker=localhost:8021 -kill job_200910162029_0008
09/10/16 21:13:47 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_200910162029_0008
09/10/16 21:13:48 INFO streaming.StreamJob:  map 0%  reduce 0%
^C

The tracking URL is handy, from there I could see that hadoop streaming wasn't able to run my groovy mapper:

Caused by: java.io.IOException: Cannot run program "movies.map.groovy": error=2, No such file or directory
 at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
 at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:166)
 ... 19 more

I was missing the -file parameter. This now gets me to my next error...

$ hadoop jar $HADOOP_INSTALL/contrib/streaming/hadoop-0.20.1-streaming.jar \
  -input movies.input.txt -output output -mapper movies.map.groovy \
  -reducer movies.reduce.groovy -file movies.map.groovy -file movies.reduce.groovy

Hadoop: HDFS in a bad state

2009-10-16T21:49:00.000-07:00

I managed to get to the position where I could not copy files to my HDFS:

$ hadoop fs -copyFromLocal movies.input.txt movies.input.txt 
09/10/16 21:20:43 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException:
File /user/user/movies.input.txt could only be replicated to
0 nodes, instead of 1

Formating my HDFS didn't help. http://localhost:50070/dfshealth.jsp was reporting that I had no live nodes. In the end I blew away my /tmp/hadoop-user directory and followed up with a hadoop namenode -format for good measure.

Hadoop: Error: JAVA_HOME is not set.

2009-10-16T20:25:00.000-07:00

If you get this when trying to use Hadoop:

$ start-dfs.sh 
starting namenode, logging to /Users/user/Applications/hadoop-0.20.1/bin/../logs/hadoop-user-namenode-merlyn.local.out
localhost: starting datanode, logging to /Users/user/Applications/hadoop-0.20.1/bin/../logs/hadoop-user-datanode-merlyn.local.out
localhost: Error: JAVA_HOME is not set.
localhost: starting secondarynamenode, logging to /Users/user/Applications/hadoop-0.20.1/bin/../logs/hadoop-user-secondarynamenode-merlyn.local.out
localhost: Error: JAVA_HOME is not set.

Then edit conf/hadoop-env.sh (I couldn't easily google this info).

Edit: I missed this: Unpack the downloaded Hadoop distribution. In the distribution, edit the file conf/hadoop-env.sh to define at least JAVA_HOME to be the root of your Java installation.

The file is helpful once you know it is relevant:
# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home

P2V: Hassles Cloning a Linux Installation as a Virtual Machine

2009-10-04T22:03:00.000-07:00

After some reading around, I settled on these tools to get the job done:

A Knoppix boot disc
An 8Gb USB pen drive
VirtualBox
dd, and fdisk

All free to use, all robust.

I took a look at my partition table on my physical Ubuntu box:

$ fdisk -l /dev/hda
Disk /dev/hda: 100.0 GB, 100030242816 bytes
255 heads, 63 sectors/track, 12161 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1         522     4192933+  83  Linux

Next, I copied the contents of hda1 to my pen drive:$ dd if=/dev/hda1 of=/media/usb/hda1.raw bs=4096I also copied the MBR:$ dd if=/dev/hda of=/media/usb/mbr.raw bs=512 count=1Back to my host computer and my fresh VirtualBox VM. I booted the VM using Knoppix, and then created the same partition layout (using fdisk expert mode to change the number of heads and sectors to match). I rebooted for good measure.

After booting again in Knoppix I copied first the hda1 into place:$ dd if=/media/usb/hda1.raw of=/dev/hda1 bs=4096Then I copied the MBR:$ dd if=/media/usb/mbr.raw of=/dev/hda bs=512 count=1I rebooted, and my cloned system wouldn't start. Restarting with Knoppix, I confirmed that I have the following problem:

$ sudo mount /dev/hda1 ~/temp
mount: /dev/hda1: can't read superblock

Now, I've tried this a number of different ways: without the mbr writing step, without tweaking the heads and sectors to match, with a partition larger than the original one - all to no avail.

I'm not finding other people with this same problem. Does anyone have any ideas?

Ruby Warrior

2009-08-29T23:36:00.000-07:00

Here's an interesting programming puzzle (thanks Nige). The challenge is to write a (stateless?) control function for a warrior, and have it fight its way up the stairs to the next level. This is the level I'm stuck on:

Level 6

The wall behind you feels a bit further away in this room. And you hear more cries for help.

Tip: You can walk backward by passing ':backward' as an argument to walk!. Same goes for feel, rescue! and attack!.

 --------
|C @ S aa|
 --------

  > = Stairs
  @ = groo (20 HP)
  C = Captive (1 HP)
  S = Thick Sludge (24 HP)
  a = Archer (7 HP)


Available Abilities:

  warrior.walk!
    Move in given direction (forward by default).

  warrior.rest!
    Gain 10% of max health back, but do nothing more.

  warrior.feel
    Returns a Space for the given direction (forward by default).

  warrior.health
    Returns an integer representing your health.

  warrior.rescue!
    Rescue a captive from his chains (earning 20 points) in given direction (forward by default).

  warrior.attack!
    Attack the unit in given direction (forward by default).

Give it a shot.

Why Groovy? Why Not Ruby?

2009-08-29T18:36:00.001-07:00

I was talking to an old friend and telling him that I was coding a lot in Groovy in my spare time. He asked me why I wasn't coding my project in Ruby. I really didn't have a good answer for him.

In fact I just looked back at some old Ruby code of mine and it does in fact look very similar to the Groovy code I've been writing. Hmm.

Opening Ports to a Virtualbox VM

2009-08-15T23:03:00.000-07:00

If you want to be able to connect to a service running on your Virtualbox VM, this is how to go about it. (global means it applies to any Virtualbox VM you run)

$ VBoxManage setextradata global "VBoxInternal/Devices/pcnet/0/LUN#0/Config/serviceName/Protocol" TCP
$ VBoxManage setextradata global "VBoxInternal/Devices/pcnet/0/LUN#0/Config/serviceName/GuestPort" portInVM
$ VBoxManage setextradata global "VBoxInternal/Devices/pcnet/0/LUN#0/Config/serviceName/HostPort" portExposedAs

See the full documentation for more info.

Setting Up Telnetd

2009-08-15T22:54:00.000-07:00

Every couple of years I want to setup telnetd on my linux box again. Each time I get telnetd installed and then try and figure out why it's not started. So this time around this is what I needed to do:
sudo apt-get install telnetdThis included openbsd-inetd for me. However:
sudo /etc/init.d/openbsd-inetd restartreported:
Not starting internet superserver: no services enabled.The missing step was to include this line in /etc/inetd.conf:
telnet stream tcp nowait root /usr/sbin/tcpd in.telnetd

Start the Day Laughing

2009-08-05T19:44:00.000-07:00

On my way in to work I walked past a corporate van for a company called Conejo something or rather. I bet the founders knew "conejo" meant "rabbit", yet probably not that it means something else as well!

An Internal DSL in Groovy

2009-08-04T23:28:00.001-07:00

There's an approach to robotic control called Reactive Action Packages. It's a way of specifying what to what, when, in what sequence, and with what taking priority over what. All of it specified in a robust manner and driven by external sensors. Anyway, I've coded my take on RAPs a number of times and never really been happy with the result. This time though my solution really feels right. Groovy builders work so well here:


this.brain = new NodeBuilder()
  .code {
    when('lost') {
       explore()
    }
    when('score < 1000') {
     go('somePlace')
     someAction()
    }
  }

The basic idea is that this DSL can be evaluated each time my robot needs to take an action. DSL statements are evaluated in sequence until one provides an action. So, if it's lost it explores. If it's not lost and it's score is under 1000 and it's already at somePlace then it does someAction. Sweet!

The only gotcha to evaluating the NodeBuilder result is that closure('aString') statement1 ; statement2 } evaluates to a Node whose value() is ['aString', node1, node2]:

this.eval = { code ->
  if (code instanceof Node) {
    return evalNode(code)
  }
  def iterator = code.iterator()
  def result = null
  while(!result && iterator.hasNext()) {
    def node = iterator.next()
    result = eval(node)
  }
  return result
}
this.evalNode = { node ->
  def result = actions[node.name()](node.value())
  return result
}

actions are defined easily enough:

this.actions = [
  when: { args ->
    if (shell.evaluate(args[0])) {
      return eval(args[1,-1])
    }
    return false;
  },
  explore: { "EXPLORE" },
  go: { to ->
    def loc = shell.evaluate("loc")
    if (loc == to) {
      return false
    }
    def direction = navigator.directionTo(loc, to)
    return "DIR $direction"
  },
  someAction: { "SOME_ACTION" }]

The last piece of the puzzle is setting up the GroovyShell with it's context and evaluating the DSL code:

this.act = {
  shell = new GroovyShell()
  shell.setVariable("loc", loc)
  shell.setVariable("score", score)
  def action = eval(brain.value())
  if (action) {
    // do magic
  } else {
    println "No action chosen!"
  }
}

To those who made builders a part of Groovy: thanks so much!

Git At Home

2009-08-02T17:42:00.000-07:00

Setting up your own git server at home is easy enough. Once you have sshd running (or Remote Login enabled), your friends for interacting with the remote repository are:

git clone username@machineIP:path
git remote add remoteRepoNickname username@machineIP:path
git push remoteRepoNickname branchName
git pull remoteRepoNickname branchName

I got started using Pragmatic Version Control Using Git. There's also an online book: Pro Git.

Above and beyond 'there is no server', the main thing I needed to get my head around was that git has the concept of 'staging' commits. For example:

git add file will stage file's deltas.
git status will show you your staged changes.
git commit -m 'comment' will then commit your staged changes.
git push remoteRepoNickname master then pushes your committed changes to another machine.

Cappuccino nib2cib Failure

2009-07-22T18:13:00.000-07:00

If like me you get this error from nib2cib when working on a NibApplication in Cappuccino:


js: "/usr/local/share/objj/narwhal/packages/objj/lib/objj/objj.js#20(eval)#2410(eval)", line 78: uncaught JavaScript runtime exception: TypeError: Cannot read property "origin" from null

Then chances are that you've included a control that's not supported. Here's the list of controls you can use (see the Class in the Identity Inspector in Interface Builder).

Chocholate Coated Turd

2009-07-21T20:12:00.000-07:00

What's your development teams' reaction been to the arrival of IE8? I very much enjoyed finding this in the head of a Cappuccino example app:

<meta http-equiv="X-UA-Compatible" content="IE=EmulateIE7" />

I've Not Been Blogging...

2009-07-20T18:34:00.000-07:00

...I've been playing StarCraft instead.

Slow, Very Slow, Coding Through Treacle, And Then There's IntelliJ...

2009-05-08T14:18:00.000-07:00

Slower all the time
Hey! Perhaps an EAP release will fix it?!

Update: 8.1 is slow. I hadn't tried upgrade to 8.1.2, but I hear:

"Don’t do it. Performance is abysmal.

Enter key -> beach ball of waiting
Ctrl-space -> beach ball of waiting

I’m going back to 8.1"

Update: I'm still hanging in there. Things have not improved. I've disabled most all inspections, as many plugins as I can without disabling refactoring support. I semi-regularly delete the IDEA cache. I've followed some OS X specific advice and set the IDE font to Monospaced. Along the way I upgraded to 8.1.3 (because things can't get any worse, right?). All these tweaks make it more usable, but still a very sorry state for premium software.

Getting Started With Cometd, Jetty, and jQuery

2009-04-27T20:48:00.001-07:00

I had difficulty getting started with these tools, so I thought I'd blog my minimal setup. This is all derived from the cometd demo who's war you can find inside the Jetty 6 webapp directory, and who's code you can find inside the contrib/cometd directory of the Jetty 6 source download.

First of all you need to be able to run Jetty (and debug it). I recommend the Run Jetty Run plugin for Eclipse.

Next you need to populate your web.xml for your webapp: (it's quite possible that you don't need all these init params)

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
  id="WebApp_ID" version="2.5">
  <display-name>comet2</display-name>
  <welcome-file-list>
    <welcome-file>index.html</welcome-file>
  </welcome-file-list>

  <context-param>
    <param-name>org.mortbay.jetty.servlet.ManagedAttributes</param-name>
    <param-value>org.cometd.bayeux,dojox.cometd.bayeux</param-value>
  </context-param>

  <servlet>
    <servlet-name>game</servlet-name>
    <servlet-class>
      org.mortbay.cometd.continuation.ContinuationCometdServlet</servlet-class>
    <init-param>
      <param-name>timeout</param-name>
      <param-value>120000</param-value>
    </init-param>
    <init-param>
      <param-name>interval</param-name>
      <param-value>0</param-value>
    </init-param>
    <init-param>
      <param-name>maxInterval</param-name>
      <param-value>10000</param-value>
    </init-param>
    <init-param>
      <param-name>multiFrameInterval</param-name>
      <param-value>2000</param-value>
    </init-param>
    <init-param>
      <param-name>logLevel</param-name>
      <param-value>0</param-value>
    </init-param>
    <init-param>
      <param-name>directDeliver</param-name>
      <param-value>true</param-value>
    </init-param>
    <init-param>
      <param-name>refsThreshold</param-name>
      <param-value>10</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
  </servlet>

  <servlet-mapping>
    <servlet-name>game</servlet-name>
    <url-pattern>/game/*</url-pattern>
  </servlet-mapping>

  <servlet>
    <servlet-name>gameServlet</servlet-name>
    <servlet-class>server.GameServlet</servlet-class>
    <load-on-startup>2</load-on-startup>
  </servlet>

</web-app>

The GameServlet is pretty much boiler plate:

package server;

import java.io.IOException;

import javax.servlet.GenericServlet;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletResponse;

import org.cometd.Bayeux;
import org.mortbay.cometd.ext.AcknowledgedMessagesExtension;
import org.mortbay.cometd.ext.TimesyncExtension;

public class GameServlet extends GenericServlet {
  @Override
  public void service(ServletRequest req, ServletResponse res)
      throws ServletException, IOException {
    ((HttpServletResponse)res).sendError(503);
  }
  
  @Override
  public void init() throws ServletException {
    super.init();
    Bayeux bayeux=(Bayeux)getServletContext().getAttribute(Bayeux.DOJOX_COMETD_BAYEUX);
        new GameService(bayeux);
        bayeux.addExtension(new TimesyncExtension());
        bayeux.addExtension(new AcknowledgedMessagesExtension());
  }
}

The GameService pairs paths to method names. You can call the send() method to send messages to clients. Mine looks like this:

package server;

import java.util.Map;

import org.cometd.Bayeux;
import org.cometd.Client;
import org.mortbay.cometd.BayeuxService;

public class GameService extends BayeuxService {
  private GameState state = new GameState();

  public GameService(Bayeux bayeux) {
    super(bayeux, "game");
    subscribe("/game/join", "join");
    subscribe("/game/event", "event");
  }

  public void join(final Client joiner, final String channelName, Map<String, Object> data, final String messageId) {
    send(joiner, "/game/state", state.getData(), null)
  }

  public void event(final Client joiner, final String channelName, Map<String, Object> data, final String messageId) {
    // TODO
  }  
}

Finally, and perhaps most importantly (as this was the part that confused me the most), here's the JavaScript to talk to the comet service:

var _connected = false;

function _metaConnect(message)
{
  var wasConnected = _connected;
  _connected = message.successful;
  if (_connected && !wasConnected)
  {
    $.cometd.startBatch();
    $.cometd.subscribe('/game/state', this, receive);
    $.cometd.publish('/game/join', {});
    $.cometd.endBatch();
  }
}

function join()
{
  $.cometd.addListener('/meta/connect', this, _metaConnect);

  var cometURL = document.location.protocol + '//' + document.location.hostname + ':' + document.location.port + '/c2/game';
  $.cometd.init(cometURL);
}

function send(data) {
  $.cometd.publish('/game/event', data);
}

function receive(message)
{
  // handle message.data content
}

join();

Note that /c2 is the context name of my webapp in Jetty.

One last point, I used the same includes in my index.html as the cometd chat example did:

<script type="text/javascript" src="jquery/json2.js"></script>
<script type="text/javascript" src="jquery/jquery.js"></script>
<script type="text/javascript" src="jquery/jquery.cometd.js"></script>

Let me know if you get up to something with comet.

Jetty and Cometd Documentation Woes

2009-04-25T13:17:00.000-07:00

I've been trying to get going with Comet using Jetty and cometd-jquery. It's been an uphill battle. First off there's practically zero documentation for Cometd. I've been reverse engineering the chat example - a very poor source of documentation. I've finally gotten to a point where my JavaScript initiates a Bayeux connection. Great.

Now I want to debug my app that Jetty is hosting. That's been a very frustrating experience. Jetty 6 forks a JVM for each webapp, so none my standard approaches to debugging are working for me. Jetty documentation is again very weak and it's hard to differentiate which versions of Jetty documentation refers to.

Can anyone help me out?

Update: IntelliJ debugs Jetty 6 out of the box (with the Jetty Integration plugin). The IntelliJ run configuration uses -Xdebug -Xrunjdwp:transport... to start in debug. I guess these settings may work for Eclipse: -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=4000,server=y,suspend=y.

Update: Run Jetty Run for Eclipse works out of the box too.

Leadership Philosophy

2009-04-24T09:39:00.000-07:00

McKnight quoted in Leading Lean Software Development:

"Mistakes will be made. But if a person is essentially right, the mistakes he or she makes are not as serious in the long run as the mistakes management will make it if undertakes to tell those in authority exactly how they must do their jobs"

Managing Ivy Dependencies In Gant

2009-03-30T17:48:00.000-07:00

Maven and IntelliJ are far from best friends. I really really don't enjoy using them together. This post is born out of that frustration. I want good, transparent dependency management, and a very very easy time integrating with my IDE.

What I wanted was to actually download the jar files to inside a project's folder and have them all in one directory to make updating from the IDE a simple process.

No, I don't want to use maven. Gradle is still to other for me - I also couldn't find out from the documentation how to explicitly download the jars to a location. So, Gant is my logical choice.

Right. Gant integrates well with Ivy - there's just not a lot of documentation. I wanted adding a new dependency to be as readable as this:
dependency( jar:'commons-lang:commons-lang:2.3').

Here's a gant build file to do just that:

libDirectory = 'lib'
   
includeTool << gant.tools.Ivy  

def patternForScope(scope) {
  "$libDirectory/$scope/[artifact]-[revision].[ext]"
}

def dependency( map ) {
  def elements = map.jar.split(':')
  def opts = [ organisation: elements[0],
    module: elements[1],
    revision: elements[2],
    inline:true
  ]
  ivy.retrieve( opts + [ pattern:patternForScope(map.scope ?: 'main') ] )
  ivy.retrieve( opts + [ pattern:patternForScope('all') ] )
}

target ( retrieveDependencies : 'Downloads jars for lib/main, lib/test, and lib/all.' ) {  
  delete ( dir : libDirectory )  
  mkdir ( dir : libDirectory )  
  dependency ( jar:'commons-lang:commons-lang:2.3' )  
  dependency ( jar:'junit:junit:4.5', scope:'test')  
}  
  
setDefaultTarget ( retrieveDependencies )

Once you've run gant this is what the resulting structure looks like:

$ find lib
lib
lib/all
lib/all/commons-lang-2.3.jar
lib/all/junit-4.5.jar
lib/main
lib/main/commons-lang-2.3.jar
lib/test
lib/test/junit-4.5.jar

The build file here is derivative of this.

Building Test Data

2009-03-25T18:06:00.000-07:00

If your experience with building test data is anything like mine then you've tried a lot of different ways of creating test data - each with their own subtle flaws:

Setup all the data in the test method: going this route you quickly cluttered your tests with data that didn't help the reader understand the test.

Setup all the data in a file (or a database): it made your tests more concise, but you ended up with "magic" numbers that depended on what was in the file or database. This made tests brittle and hard to interpret.

You pulled all of the setup out into setUp (or helper methods or helper classes): much better. You were able to show the significant details, hide the mechanics of setting up test data, and gave helper methods readable names. Well, at first at least. The helper methods were hard to generalize so pretty quickly you ended up with an explosion of helper methods that became hard to manage.

Hmm. This is as far as I'd gotten and I've not been very happy about it.

At this point perhap you'd dumped Java anyway and run off to the Ruby space. If you didn't then did you arrive at something like this?

  Order order1 = anOrder()
    .withLine("Deerstalker Hat", 1) // detail important to the test
    .withLine("Tweed Cape", 1)      // detail important to the test
    .withCustomersReference(1234)   // detail important to the test
    .build();

So this approach appeals to me (read the "Removing Duplication At Point of Use" section to see why). I took the time to code up what this might look like. It feels like a ton of boiler plate:

public class BuilderFactory {
  public static OrderBuilder anOrder() {
    return new OrderBuilder();
  }
// factory methods for each builder... 
}

public class OrderBuilder implements Cloneable {
  private Customer customer = new CustomerBuilder().build();
  private List<OrderLine> lines = new ArrayList<OrderLine>();
  private BigDecimal discountRate = BigDecimal.ZERO;
  private String voucher = null;

  public OrderBuilder withCustomer(Customer customer) {
    this.customer = customer;
    return this;
  }

  public OrderBuilder withCustomer(CustomerBuilder customerBuilder) {
    this.customer = customerBuilder.build();
    return this;
  }
  
  public OrderBuilder withOrderLines(List<OrderLine> lines) {
    this.lines = lines;
    return this;
  }
  
  public OrderBuilder withOrderLine(String name, int quantity) {
    lines.add(new OrderLineBuilder().withName(name).withQuantity(quantity).build());
    return this;
  }

  public OrderBuilder withDiscount(BigDecimal discountRate) {
    this.discountRate = discountRate;
    return this;
  }
  
  public OrderBuilder withGiftVoucher(String voucher) {
    this.voucher = voucher;
    return this;
  }

  public Order build() {
    Order order = new Order(customer);
    for (OrderLine line : lines) order.addLine(line);
    order.setDiscountRate(discountRate);
    order.setVoucher(voucher);
    return order;
  }
  
  public OrderBuilder but() {
    try {
      return (OrderBuilder) clone();
    } catch (CloneNotSupportedException e) {
      throw new RuntimeException(e);
    }
  }

}

// plus all the other builders too...

What's your reaction? Do you have a more elegant solution? [the code]

I like the readability of the code that uses the builders. The real disappointment is that the essence of all of the builder code is in the 6% of the code that defines the sensible defaults for test data, namely:

  private Customer customer = new CustomerBuilder().build();
  private List<OrderLine> lines = new ArrayList<OrderLine>();
  private BigDecimal discountRate = BigDecimal.ZERO;
  private String voucher = null;

The rest is all ceremony. Are there better solutions out there?