Making Ripples

Internet of Things and The Cloud

2016-04-15T16:37:00.000+01:00

The Internet of Things (IoT) got some bad press as a concept a couple of weeks back, as a result of the plug being pulled on Revolv's Nest hub, e.g. Wired, BBC, The Guardian. And indeed, when I buy a "thing" I expect it to just work - and for some time. Thermostats, light switches, smoke alarms and kitchen-appliances are all things that I expect to leave working when I move house in some years; lightbulbs less so, but I don't expect all of them to break at once because the light fitting or the wires have become obsolete. And that is the issue here: the controllers still work just fine - but the cloud service that they require to work is being switched off.

The added cost of being an early adopter probably comes with an expectation that the product may date faster than if I wait, or require more frequent fixes and upgrades - as is the case if you compare my FitBit to my old analogue watch. In any case, this story raises a wider question of what happens to IoT tech as the market evolves. As someone that's worked in "pervasive computing" and IoT for some time this is naturally of interest to me.

There's a number of potential solutions, and their accompanying issues, that have come past in news articles and Twitter, including:

Use products with standards, assuming that the idea is mature enough to have a standard. A feature of innovation is that at least part of the idea is often ahead of the standards curve.
Provide a refund and ease people onto a new system, assuming they are willing to invest in that on the back of their previous investment being a dud.
Sell or open source the cloud software so others can keep it going to service the device. However, in most cases this software may contain some significant IP that the company probably wants to sell or reuse in a pivot of their idea. If open source was the right solution it would probably have been part of the picture well in advance of pulling the product.

So, what to do?

My suggestion is that the IoT is a good example of a system where "cloud" services supporting the devices would benefit from a different approach:

It isn't the cloud, it's a cloud.

By which I mean: don't sell me a device that connects to some centralised cloud service - unless you're really small scale and alpha-testing. Sell me something that brings up its own cloud service, maybe with a provider I choose, and that I get billed for. It can let my phone know where to look so they can talk - it will just be a URL and the physical thing can bootstrap disseminating this.

Maybe this is a systems architect's solution, and certainly feels obvious if you cast this as a distributed systems problem, but I can see several advantages:

If the vendor switches their systems off mine can keep going. If there's some advantage to them aggregating data from many customers that can happen by forwarding on from my cloud to theirs - and fail cleanly if the need changes. The crucial step being to eliminate centralised nodes in the control loop.
The home systems can point at a repo with appropriate security to manage centralised updates. There's no particular need to give me a login to the cloud service, so the secret sauce isn't much more exposed than it would be when centralised - especially given that one end of it is hardware in my possession.
By putting my data on a system which I control I can be given greater and more plausible control over my privacy. My detailed data is visible to fewer people and a hacker has to gain access to many systems to gather large amounts of data.
By distributing the system over many little services scalability is easy and system failures affect fewer users.

make-ing docker

2016-01-20T20:50:00.002+00:00

I mentioned some time ago that I've been exploring Docker. I then changed my main job and the blog went rather quiet. I have a contract that gives me some time and IP to myself, and this is finally turning into a new project. Between personal projects and work I've spent a whole lot more time with Docker. So, a blog post, about building Docker images, that might be useful to someone...

A bit of background:
First, I'm a fan of make. I know, it shows my age. But, it's really quite good at handling building tasks with a minimum of tricky requirements on the build system. It can also be turned to organising installations and running jobs, so keeping related issues in one version-controllable place. It is also in a format that's quite user friendly, even when ssh-ed into a server and fixing stuff with vi. So, while I've also used Ant, Ansible and Maven to do some of these things, it remains a reliable standby.

One of the things I use Docker for is setting up groups of images which are related to each other: django; django with a different config; django with that config, but setup for running tests rather than running the server - and so on. Which leads to dependencies.

Most of the time when I use Docker, its in a local environment, and I just build and run images on one machine - without using Docker Hub. I like having the tool chain version controlled, and this approach fits.

Make is good at handling dependencies, but the relationship chains in Docker image definitions don't expose themselves in file names (unless you're very organised with naming schemes). The quick first approach is to make them with explicit pointers in the make file as well as in the Dockerfile. But eventually that level of duplication will irk. So, in a spare couple of hours to polish my build scripts, I refactored the duplication out of the Makefile.

The key parts of the makefile are below - stripped of my builds to show the principle:

DIRS := $(shell find . -mindepth 1 -maxdepth 1 -type d)
DOCKERFILES := $(addsuffix /Dockerfile,$(DIRS))
IMAGES := $(subst /,,$(subst ./,,$(dir $(DOCKERFILES))))
FLAG_FILES := $(addprefix ., $(addsuffix .docker, $(IMAGES)))
PWD := $(shell pwd)


# Docker images can depend on each other.
# A changed base image ought to trigger a rebuild of its children.
define image_dep_search
@echo "checking dependencies of $1"
@for d in $(IMAGES); do \
 from=`grep FROM $$d/Dockerfile | cut -d ' ' -f 2`; \
 if [ $1 = $$from ]; then \
  echo "dependent image $$d"; \
  touch $$d; \
  make .$$d.docker; \
 fi\
done
endef


all: images 


# Consider all docker image directories for building
images: $(FLAG_FILES)
 @echo "Done making images."

# Build images where the directory or contents have changed since flag last set
.%.docker: % %/* 
 $(eval IMAGE = $(subst .,,$(basename $@)))
 $(eval BASE = $(word 2,$(shell grep FROM $(addsuffix /Dockerfile,$(IMAGE)))))
 $(eval HAS_DEP = $(filter $(BASE),$(IMAGES)))
 @echo "building $(IMAGE)"
 @cd $(IMAGE) && docker build -t $(IMAGE) .
 @touch $@
 $(call image_dep_search,$(IMAGE))


# Utility make targets for creating containers from images 
.PHONY: run_java_bash
run_java_bash: .java_base.docker java_bash_container

.PHONY: java_bash_container
java_bash_container:
 docker run --rm -v=$(PWD)/..:/project -it --name java_bash java_base bash

clean:
 @rm -f $(FLAG_FILES)

In order, this contains:

Some definitions, which find directories that contain Dockerfiles. Files called ".<imgname>.docker" will be created to mark the latest build.
A definition to use later, that finds the FROM line in the Dockerfile; extracts the argument; sees whether it is one of our images; fiddles the need for a build and calls make on that image.
The standard make stuff, to run a build on each image directory which has changes. Once built any dependent images are found and built using the routine defined above.
A phony target to run the container, to illustrate the point. It double checks the image, in case we're forgetful about running "make all" first. This provides the project root (the parent of the docker directory) as a mounted volume - which may or may not be a good thing, depending on your use case.
A clean target, that gets rid of the build flag files.

There's a couple of assumptions here:

A directory of docker images within the project. I usually call it "docker". This is the set of docker images that will be considered for dependencies. The rest are just assumed to exist. The makefile is in this docker directory.
The FROM and the image name in the Dockerfile are separated by a space.
Images are flat directories. I think that structure is probably better put elsewhere, and built in an archive format than copying lots of files one by one in the Docker build process - so this hasn't been an issue for me.
If one image depends on having another already built then another makefile rule is needed to force correct order. I'll update this when I've got an update that automates this as well. Simple "list of target" rules may well be needed to build specific sub-sets anyway.

And that's it: build the images same as I build the code, with a minimum of drag on my effort. Easy to call from Jenkins, easy to call from the command line.

Addendum: With a bit of tidying up, and some example Dockerfiles, this is now on GitHub at https://github.com/danchalmers/MakeingDocker

Learning Python (7) - Consolidation and Django

2015-06-18T22:34:00.000+01:00

It's been a few weeks since my last post. So, what news?

I've done more of the Katas I had planned for learning now. The extra ones have been Map Kata and Data Munging. They were interesting, but I'm not going to blog them in full here. They mostly served to consolidate rather than show me new things. There's a couple more on the list, but they're pushed down the to-do list a bit for now. The fact it doesn't all feel new seems like a good sign.

I've been exploring Django. Mostly in the form of the tutorial so far, although I've added some spice to this: First by testing, both the Django tests in the tutorial, but also Selenium end to end smoke tests. Second, by playing with Docker alongside this. Working with a development container of Django and sqlite3 most of the time, but also running a triple-container setup of Django, Postgres and Jenkins. This isn't the first time I've used Docker, but I'm quite pleased with my current explorations of common and dev / test docker-compose setups, with matching Django common, dev and test setups. More on this as I get a setup for one of my projects that I feel is stable in use.

Finally (you may have noticed) I've started using the splashinapond.com domain. I'm winding down one job and ramping another two up. One is development work for a start-up, the other is my own company. This blog will still focus on my own things, but there's now a name for it.

Learning Python (6) - Mars Rover

2015-05-26T13:38:00.000+01:00

My next Kata was from a set of exercises that Amir Rajan describes, the mars rover. The idea is to model a planet as a grid, place a rover there, and send it instructions for moving about. The instructions are a limited set: forward, backwards, turn left, turn right. The planet is a sphere, so once the basics are working the grid needs to wrap around. And then, being Mars, it has rocks in the way. So sometimes the rover will stop and report where it has got to.

This is a nice problem for practising TDD. As I started it felt a lot like revising the skills I'd been developing already. This is a good thing: Katas are supposed to be about practising skills, and practise is really important for reenforcing learning. The tests flow naturally from unpicking the problem, and are a perfect fit to incrementally developing a function because the turns, moves and grid all offer at least two similar possibilities for each action.

'''
Created on 24 May 2015

@author: Dan Chalmers 
'''
import unittest
from mars_rover import Planet, Direction

class Test(unittest.TestCase):

    def setUp(self):
        self.grid = Planet(4)
        self.grid.landRover(1,1,Direction.NORTH)
        self.grid2 = Planet(4)
        self.grid2.setObstacle(1,3)
        self.grid2.landRover(1,1,Direction.NORTH)
      
    def testPutRoverOnPlanet(self):
        self.assertEqual((1,1), self.grid.getLocation())
        
    def testMoveForwardNorth(self):
        (ok, position) = self.grid.move("f")
        self.assertEqual((True,(1,2)), (ok, position))

    def testTurnRightMoveForwardEast(self):
        (ok, position) = self.grid.move("rf")
        self.assertEqual((True,(2,1)), (ok, position))
        
    def testTurnR2MoveForwardSouth(self):
        (ok, position) = self.grid.move("rrf")
        self.assertEqual((True,(1,0)), (ok, position))

    #4 right turns wrap round to north again
    def testTurnR4MoveForwardSouth(self):
        (ok, position) = self.grid.move("rrrrf")
        self.assertEqual((True,(1,2)), (ok, position))
        
    def testMoveBackwardSouth(self):
        (ok, position) = self.grid.move("b")
        self.assertEqual((True,(1,0)), (ok, position))
        
    def testTurnLeftMoveForwardEast(self):
        (ok, position) = self.grid.move("lf")
        self.assertEqual((True,(0,1)), (ok, position))

    def testTurnLeftMoveBackwardWest(self):
        (ok, position) = self.grid.move("lb")
        self.assertEqual((True,(2,1)), (ok, position))
        
    def testWorldWrapsEast(self):
        (ok, position) = self.grid.move("rfff")
        self.assertEqual((True,(0,1)), (ok, position))

    def testWorldWrapsWest(self):
        (ok, position) = self.grid.move("lfff")
        self.assertEqual((True,(2,1)), (ok, position))

    #Poles wraps across the pole, not a jump to opposite end of the world!
    #Wrap over poles have an extra move after wrap to test direction swap
    def testWorldWrapsNorth(self):
        (ok, position) = self.grid.move("ffff")
        self.assertEqual((True,(3,2)), (ok, position))

    def testWorldWrapsSouth(self):
        (ok, position) = self.grid.move("rflbbb")
        self.assertEqual((True,(0,1)), (ok, position))
        
    def testBlockedByObstacle(self):
        (ok, position) = self.grid2.move("ff")
        self.assertEqual((False,(1,2)), (ok, position))

    #Written when debugging the not blocked, passed immediately
    def testWiggle(self):
        (ok, position) = self.grid.move("frflf")
        self.assertEqual((True,(2,3)), (ok, position))
      
    def testNotBlockedByObstacle(self):
        (ok, position) = self.grid2.move("frflf")
        self.assertEqual((True,(2,3)), (ok, position))
     

if __name__ == "__main__":
    #import sys;sys.argv = ['', 'Test.testName']
    unittest.main()

The code, similarly, flowed easily from the tests. Much of my earlier looking things up and use of explicit loops was being bypassed. I also found that this Kata led to a simple solution, that was fertile ground for refactoring. The similarity of the left / right, forward / backward, 2 poles, wrap around etc pairings naturally pushed me into eliminating duplication and separating little behaviours out. I was pleased that my tests described the behaviours in the spec, even as my code refactored into something much more subdivided.

'''
Mars Rover Kata
Based on Kata defined at http://amirrajan.net/Blog/code-katas-mars-rover/
Created on 24 May 2015
@author: Dan Chalmers 
'''

class Planet( object ):
  def __init__(self, size):
    self.size = size
    self.obstacles =  [[False for _ in range(size)] for _ in range(size)]  #not the same as [[False]*3]*3]
    
  def landRover(self,x,y,direction):
    self.x = x
    self.y = y
    self.direction = direction
    
  def getLocation(self):
    return (self.x, self.y)
  
  def move(self, instructions):
    for i in instructions:
      ok = self._moveStep_(i)
      if not ok: break # Hit an obstacle, so stop
    return (ok,self.getLocation())
      
  def _moveStep_(self, instruction):
    ok = True
    (dx,dy) = Direction.getMove(self.direction)
    if instruction == 'f': ok = self._moveForward_(dx,dy)
    if instruction == 'b': ok = self._moveBackward_(dx,dy)
    if instruction == 'r': self.direction = Direction.turnRight(self.direction)
    if instruction == 'l': self.direction = Direction.turnLeft(self.direction)
    return ok
    
  def _testAndMove_(self,x,y):
    (x,y, direction) = self._checkCoordinatesWrap_(x,y)
    if self.obstacles[x][y]: return False
    self.x = x
    self.y = y
    self.direction = direction
    return True
      
  def _moveForward_(self,dx,dy):
    x = self.x + dx
    y = self.y + dy
    return self._testAndMove_(x,y)

  def _moveBackward_(self,dx,dy):
    x = self.x - dx
    y = self.y - dy
    return self._testAndMove_(x,y)
    
  def _checkCoordinatesWrap_(self,x,y):
    # Because y coordinate wrap may affect X, checkY has to go first
    (x,y,direction) = self._checkYCoordinateWrap_(x,y)  
    return (self._checkXCoordinateWrap_(x), y, direction)
  
  def _checkXCoordinateWrap_(self,c):
    return c % self.size
  
  def _checkYCoordinateWrap_(self, x, y):
    if y == self.size or y == -1: #Over a pole
      y = self._yOverPole_(y)
      x = (x + self.size/2)%self.size
      direction = Direction.swapDirection(self.direction)
    else:
      direction = self.direction
    return (x, y, direction)

  def _yOverPole_(self, y):
      if y == self.size: y = self.size-1
      elif y == -1: y = 0
      return y
    
  def setObstacle(self,x,y):
    self.obstacles[x][y] = True
    
#Enumeration of directions, with a number behind it allows simple changes    
class Direction:
  NORTH = 0
  EAST = 1
  SOUTH = 2
  WEST = 3
  
  @staticmethod
  def turnRight(direction):
    return (direction+1)%4

  @staticmethod
  def turnLeft(direction):
    return (direction-1)%4
  
  @staticmethod
  def swapDirection(direction):
    return (direction+2)%4

  #Get the delta a move forward makes given a direction the rover faces
  @staticmethod
  def getMove(direction):
    if direction == Direction.NORTH: return (0,1)
    if direction == Direction.EAST: return (1,0)
    if direction == Direction.SOUTH: return (0,-1)
    if direction == Direction.WEST: return (-1,0)

Although the focus was on practise, I did do some exploring, and found three things of note:
The first was the Direction class. Coming from a Java background this felt like an enum. However, in 2.7 Python doesn't have a built in enum. So the labels approach was the next best fit. In the end this facilitated the turn, get move delta, and swap direction methods.
The second was the use of @staticmethod.
The third was that
[[False]*3]*3
and
[[False for _ in range(3)] for _ in range(3)]
build subtly different things. When used to create the obstacles map this difference matters: setObstacle is broken for the first, as shown below.

>>> obstacles = [[False]*3]*3
>>> obstacles
[[False, False, False], [False, False, False], [False, False, False]]
>>> obstacles[1][2] = True
>>> obstacles
[[False, False, True], [False, False, True], [False, False, True]]

>>> obstacles2 = [[False for _ in range(3)] for _ in range(3)]
>>> obstacles2
[[False, False, False], [False, False, False], [False, False, False]]
>>> obstacles2[1][2] = True
>>> obstacles2
[[False, False, False], [False, False, True], [False, False, False]]

Learning Python (5) - Monty Hall

2015-05-22T17:55:00.001+01:00

This is another Kata from Emily Bache's book. In essence simulating a game:
There are three doors, behind one is a prize you want (I chose a bicycle), behind the other two something you don't (a goat). You pick a door, but don't see behind it. Then the game reveals the goat behind one of the other two doors. You may then either win what is behind your original choice of door, or swap to the other closed door and win what is behind that. The best strategy is to swap doors, (because probability).

As a programming exercise this is interesting. You have to create and manipulate some state: the prizes, the choice, the open door, the final choice. More importantly there is randomisation: where do the prizes go, which door do you choose, which goat is revealed. Because of the randomisation, the acceptance test of "on average, do you win more often if you swap choice" requires many iterations of the swap / don't swap to be a safe test. So, this Kata is perfect for some use of mocks / fakes in testing to make the smaller tests predictable while still using the code under test which relies on random behaviour.

I got to the solution quite quickly. As an observation of how I'm becoming more comfortable with Python as a tool I got to the end result with less refactoring for code style, which is pleasing. There were still techniques I needed to look up - mostly about the random library and passing methods as arguments.

So, the tests. The fake Game class got created early on, once I'd written a first test against the Game. The fake deals with the random stuff by giving just enough predictability without replacing too much real code. Overriding choosePrizeDoor means that the fake is a really thin shim over the real Game implementation. The reveal is still random, but the initial choice and prize are controlled in the tests, which makes the test predictable.
The method passing into test_swappingIsBetter took a couple of passes to get right, I haven't used this much in Python before, but knew I wanted to use it very quickly as the alternative was too much duplication.

'''
Created on 21 May 2015
@author: Dan Chalmers 
'''

import unittest
from montyhall.prizes import Goat, Bike, Game
import random #ONLY for the swapping is better test! *shudder*

class Test(unittest.TestCase):

    def test_isprize(self):
      prize = Goat()
      self.assertFalse(prize.isWin());
      prize = Bike()
      self.assertTrue(prize.isWin());
      
    def test_getRevealChoiceOfTwo(self):
      doors = FakeGameZero()
      doors.choose(0)
      (doorNum, prize) = doors.reveal()
      self.assertNotEqual(0, doorNum)
      self.assertFalse(prize.isWin())

    def test_getRevealChoiceOfOne(self):
      doors = FakeGameOne()
      doors.choose(0)
      (doorNum, prize) = doors.reveal()
      self.assertEqual(2, doorNum)
      self.assertFalse(prize.isWin())

    def test_swapAndGetPrize(self):
      doors = FakeGameOne()
      doors.choose(0)
      doors.reveal()
      (doorNum, prize) = doors.swapChoice()
      self.assertEqual(1, doorNum)
      self.assertTrue(prize.isWin())
      
    def test_swapAndLoose(self):
      doors = FakeGameZero()
      doors.choose(0)
      doors.reveal()
      (doorNum, prize) = doors.swapChoice()
      self.assertNotEqual(0, doorNum)
      self.assertFalse(prize.isWin())
      
    def test_notSwapAndGetPrize(self):
      doors = FakeGameOne()
      doors.choose(1)
      doors.reveal()
      (doorNum, prize) = doors.keepChoice()
      self.assertEqual(1, doorNum)
      self.assertTrue(prize.isWin())
      
    def test_notSwapAndLoose(self):
      doors = FakeGameZero()
      doors.choose(2)
      doors.reveal()
      (doorNum, prize) = doors.keepChoice()
      self.assertEqual(2, doorNum)
      self.assertFalse(prize.isWin())
      
    '''
    The story test of does swapping mean you win more often.  
    This is the first time with Python that passing a method as a parameter has 
    clearly been the right thing to do. I do like higher order functions!  
    '''
    def test_swappingIsBetter(self):  
      swapCount = self.getWinCount(Game.swapChoice)
      keepCount = self.getWinCount(Game.keepChoice)
      message = "swapped wins: %d,  kept wins: %d" % (swapCount, keepCount)
      self.assertTrue(swapCount > keepCount)
      print(message)
      
    #1000 iterations, to avoid being bitten on the bum by random too frequently (and so seeing a fail)  
    def getWinCount(self, method):
      wins = 0
      for _ in range(1000):
        (_, prize) = method(self.makeGame())
        if prize.isWin(): wins+=1
      return wins
    
    def makeGame(self):
        doors = Game()
        doors.choose(random.randrange(3))
        doors.reveal()
        return doors
      
'''
Initially written as part of refactoring of the first test of Game
I had in mind that a mock / fake was part of what I wanted to explore in this Kata, 
but in the end this was a natural way to make a reliable test from a class which normally has some random behaviour
'''
class FakeGameZero(Game):
  def _choosePrizeDoor_(self):
    return 0
  
class FakeGameOne(Game):
  def _choosePrizeDoor_(self):
    return 1
  
if __name__ == "__main__":
    unittest.main()

The code is quite straightforward. Two classes to represent the prizes. In Java this would probably have come out as an enum and I'm still not convinced that this is the nicest approach in Python. I avoided having all the state as separate variables in the Game for a while, but in the end having the three variables led to much simpler methods. As state was unavoidable, going with the flow seemed to be the way to go.

'''
Created on 21 May 2015
The Monty Hall Game Kata after Emily Bache's description
@author: Dan Chalmers 
'''

import random

class Goat(object):
  def isWin(self):
    return False
  
class Bike(object):
  def isWin(self):
    return True
  
  
class Game(object):
  DOOR_COUNT = 3
  
  def __init__(self):
    self.doors = [Goat()] * self.DOOR_COUNT
    self.choice = None
    self.revealed = None
    #self.prizeDoor = None #prize door is set next
    self._setPrize_(self._choosePrizeDoor_())
    
  def choose(self, doorNum):
    self.choice = doorNum
      
  def _choosePrizeDoor_(self):
    return random.randrange(self.DOOR_COUNT)
    
  def _setPrize_(self, prizeDoor):
    self.prizeDoor = prizeDoor
    self.doors[prizeDoor] = Bike()
    
  def reveal(self):
    cands = [x for x in range(self.DOOR_COUNT) if x != self.choice and x != self.prizeDoor]
    self.revealed = cands[random.randrange(len(cands))]
    return (self.revealed, self.doors[self.revealed])
  
  def getPrizeDoorNum(self):
    return self.prizeDoor
  
  def swapChoice(self):
    cands = [x for x in range(self.DOOR_COUNT) if x != self.choice and x != self.revealed]
    self.choice = cands[0]
    return self._getChoice_()
  
  def keepChoice(self):
    return self._getChoice_()
    
  def _getChoice_(self):
    return (self.choice, self.doors[self.choice])

This was quite a quick problem to solve, but still taught me new things (random, passing functions). I was pleased with how easy using the fake was in testing. I thought it worked well as a follow-on from Medicine Clash, practising the list comprehensions and use of classes.

Learning Python (4) - Medicine Clash Kata

2015-05-20T21:49:00.000+01:00

Another Kata as part of my learning Python project (I do have some other things going on, just not quite done). Another one from Emily Bache, documented on GitHub. The basic premise is that a patient takes a series of medicines. Each medicine may have one or more prescriptions, and so get taken from one date for a number of days. This is all very well, but some medicines combine in strange and interesting ways. Medicine tries to avoid interesting treatments. So we have a program which looks for clashes (in the past). The exercise comes recommended as practice for building an object graph and assigning responsibilities, use of test doubles (I didn't) and use of dates. A bare bones set of three classes are provided: Patient, Medicine and Prescription. The latter two could be combined for this problem, but part of the set-up is that these classes reflect an underlying relational model and we're just ignoring other details so these classes are fixed.

My solution evolved a bit as the development went on, so the tests don't quite document the development linearly. The first behaviour to test was clearly a simple clash of overlapping dates. After a quick think it was clear that the starting point was going to be comparing lists of dates. So my first passing unit tests extracted these without making tests for a clash. I also set up some prescriptions for our user of dangerous medicines as a test fixture, although much of the setUp code followed later.

import unittest
from medicineclash.medicine import Medicine
from medicineclash.patient import Patient
from medicineclash.prescription import Prescription
from datetime import date, timedelta

class Test(unittest.TestCase):


    def setUp(self):
      prescribe_yesterday = Prescription(date.today()-timedelta(days=1))
      prescribe_one_week_ago = Prescription(date.today()-timedelta(days=7), 14)
      prescribe_two_weeks_ago = Prescription(date.today()-timedelta(days=14), 7)
      prescribe_four_weeks_ago = Prescription(date.today()-timedelta(days=28), 30)
      
      self.patient = Patient()
      self.medicines = []
      
      medicine = Medicine("Aconite")
      medicine.add_prescription(prescribe_yesterday)
      self.medicines.append(medicine)
      
      medicine = Medicine("Belladonna")
      medicine.add_prescription(prescribe_two_weeks_ago)
      self.medicines.append(medicine)      

      medicine = Medicine("Celandine")
      medicine.add_prescription(prescribe_one_week_ago)
      self.medicines.append(medicine)      
      
      medicine = Medicine("Duboisia")
      medicine.add_prescription(prescribe_four_weeks_ago)
      self.medicines.append(medicine)      

      medicine = Medicine("Ergot")
      medicine.add_prescription(prescribe_two_weeks_ago)
      medicine.add_prescription(prescribe_yesterday)
      self.medicines.append(medicine)      
      
      medicine = Medicine("Foxglove")
      medicine.add_prescription(prescribe_four_weeks_ago)
      medicine.add_prescription(prescribe_one_week_ago)
      self.medicines.append(medicine)      
      
     
    def test_simpleclash(self):
      self.patient.add_medicine(self.medicines[0])
      self.patient.add_medicine(self.medicines[1])
      self.patient.add_medicine(self.medicines[2])
      result = self.patient.clash(["Aconite","Celandine"])
      self.assertSetEqual(set([date.today()-timedelta(days=1)]), result)

    def test_get_med_date_lists_single(self):
      self.patient.add_medicine(self.medicines[0])
      result = self.patient.get_med_date_lists(["Aconite"])
      self.assertEqual(1, len(result))
      self.assertEqual(1, len(result[0]))
        
    def test_get_med_date_lists_double(self):
      self.patient.add_medicine(self.medicines[0])
      self.patient.add_medicine(self.medicines[1])
      result = self.patient.get_med_date_lists(["Aconite","Belladonna"])
      self.assertEqual(2, len(result))
      self.assertEqual(1, len(result[0]))
      self.assertEqual(7, len(result[1]))

I got these passing, with some quite long code with explicit loops. There was another unit (rather than behaviour in the spec) test by now. I won't reproduce the extra test or the first version code here, but it is worth noting that there was some refactoring at this point and the test of the removed function went with it. The rest of the tests follow, written one at a time.

    def test_nonoverlap_noclash(self):
      self.patient.add_medicine(self.medicines[0])
      self.patient.add_medicine(self.medicines[1])
      result = self.patient.clash(["Aconite","Belladonna"])
      self.assertSetEqual(set(), result)
 
    def test_doubleclash(self):
      self.patient.add_medicine(self.medicines[0])
      self.patient.add_medicine(self.medicines[1])
      self.patient.add_medicine(self.medicines[2])
      self.patient.add_medicine(self.medicines[3])
      self.patient.add_medicine(self.medicines[4])
      result = self.patient.clash(["Duboisia","Ergot"])
      self.assertEqual(8, len(result))
      self.assertTrue(date.today()-timedelta(days=14) in result)
      self.assertTrue(date.today()-timedelta(days=1) in result)
      
    def test_tripleclash(self):
      self.patient.add_medicine(self.medicines[0])
      self.patient.add_medicine(self.medicines[1])
      self.patient.add_medicine(self.medicines[2])
      self.patient.add_medicine(self.medicines[3])
      self.patient.add_medicine(self.medicines[4])
      result = self.patient.clash(["Aconite","Duboisia","Celandine"])
      self.assertEqual(1, len(result))
      self.assertTrue(date.today()-timedelta(days=1) in result)

    def test_prescription_overlap_handled(self):
      result = self.medicines[5].get_date_list_in_window(40)
      self.assertEqual(28, len(result))
      
    def test_nomedicine_noclash(self):
      result = self.patient.clash(["Aconite"])
      self.assertSetEqual(set(), result)
      
    def test_medicine_notinlist_noclash(self):
      self.patient.add_medicine(self.medicines[2])
      result = self.patient.clash(["Aconite","Belladonna"])
      self.assertSetEqual(set(), result)

After completing what I thought was my solution I took a look at Emily Bache's code, tests first. Quickly I saw that there were tests for a couple of conditions that I hadn't read into the spec. These were added and the code tweaked some more.

    def test_singlemed_clash(self):
      self.patient.add_medicine(self.medicines[0])
      self.patient.add_medicine(self.medicines[1])
      self.patient.add_medicine(self.medicines[2])
      result = self.patient.clash(["Aconite"])
      self.assertNotEqual([], result)
      
    def test_doublemedicine_clashself(self):
      self.patient.add_medicine(self.medicines[1])
      self.patient.add_medicine(self.medicines[1])
      result = self.patient.clash(["Belladonna"])
      self.assertNotEqual([], result)

So, tests done - focussed on exploring the possibilities of the spec. As far as possible they aren't tied to the internal structure; although there are also a couple of unit tests written to understand the details mixed in there.
The code has, it is fair to say, evolved over multiple refactoring passes. So what is here is where it ended up, rather than the instinctive first version. (I might add that the first version was written rather early in the morning, the refactoring at a more civilised time!) The process of refining to this was useful, and got me to explore a few language features. I'll show the code next, and then return to discussing the learning.
First the root of the tree, the patient. It enables medicine lookups and identifies clashes using a set intersection. This was also the first time I had used the "splat" operator to pass a list as a sequence of arguments.

'''
Created on 17 May 2015
Starting point: https://github.com/emilybache/KataMedicineClash/blob/master/Python/patient.py

Search for combinations of medicine from a list, which have all been taken at the same time. Report dates of such clashes.

@author: Dan Chalmers 
'''

class Patient(object):
    
    def __init__(self, medicines = None):
        self._medicines = medicines or []
    
    def add_medicine(self, medicine):
        self._medicines.append(medicine)
    
    #A useful step in building the solution, 
    #gives a list of sets of dates, one set per medicine taken that is in search list of meds
    def get_med_date_lists(self, medicine_names, days_back=90):
      return [ med.get_date_list_in_window(days_back) for med in self._medicines if med in medicine_names ] or [set()]
          
    def clash(self, medicine_names, days_back=90):
      med_lists = self.get_med_date_lists(medicine_names, days_back) 
      return set.intersection(*med_lists)

Next the medicine. A double-for list comprehension to get date lists and an overriding of equality to enable comparison with the medicine names, which come as a list of strings.

'''
Created on 17 May 2015
Starting point: https://github.com/emilybache/KataMedicineClash/blob/master/Python/medicine.py

A medicine, with a list of prescriptions for that patient.
Equality satisfied on name to simplify search.
Can search on dates it was taken, not stored pre-computed.

@author: Dan Chalmers 
'''

class Medicine(object):
    
    def __init__(self, name):
        self.name = name
        self.prescriptions = []
        
    def add_prescription(self, prescription):
        self.prescriptions.append(prescription)    
        
    def __eq__(self, other):
      return (self.name == other)

    def get_date_list_in_window(self, days_back):
      return set( [ med for p in self.prescriptions for med in p.get_date_list_in_window(days_back) ] )

Finally, the leaves of the tree, the prescription. This provides the list of dates within the search window. Prescription.get_date_list_in_window doesn't have its own test as it started life in with the Medicine.get_date_list_in_window function, and migrated out during refactoring. The existing tests described what I wanted of the function and carried on passing as I restructured.

'''
Created on 17 May 2015
Starting point: https://github.com/emilybache/KataMedicineClash/blob/master/Python/prescription.py

Prescriptions for medicine, a supply date and duration.
Can get a list of dates that overlap a start..today window.

@author: Dan Chalmers 
'''

from datetime import date, timedelta

class Prescription(object):
    
    def __init__(self, dispense_date=None, days_supply=30):
        self.dispense_date = dispense_date or date.today()
        self.days_supply = days_supply
        
    def get_date_list_in_window(self, window_start):
      start_date = max(self.dispense_date, date.today() - timedelta(window_start))
      end_date = min(self.dispense_date + timedelta(self.days_supply), date.today())
      return [ start_date + timedelta(day) for day in range((end_date - start_date).days) ]

As expected, this explored time a little - in particular the timedelta module, which was new to me. However, I practised the refactoring process and use of list comprehensions extensively, as I moved from explicit loops to comprehensions. The syntax is a little different to Haskell, but that remains my reference point despite being a little rusty. This exercise also explored sets more than I had before, and the set / list relationship. The splat operator and __eq__ function were also firsts. So, a lot of learning in this exercise and the result feels bigger than previous Katas. My solution remains a little different to the example solution, but I'm happy with the allocation of functions to classes and that the code feels readable even if I'm missing a few possibilities. There was quite a lot of referring to the API and refactoring in this. I hope the process of writing the functions will become more smooth with experience. I'll return to this Kata at some point, to help these techniques stick and see if they come easier.

Learning Python (3) - Bowling Scores Kata

2015-05-13T19:38:00.000+01:00

The Bowling Scores Kata is widely discussed. This version is after the description in Emily Bache's book. I note that the Ron Jeffries' description doesn't have the special symbol for spare, which simplifies things. Essentially the goal is to produce a score for a line of frames in 10 pin bowling. Not roll by roll, just the end score. It is just complex enough to allow a couple of different approaches, but still compact enough to solve quickly. It's an exercise I've given programming students, and done myself in Java a couple of times before. This got it onto my list of katas, to look at how my approaches to the two languages differ.

I've taken a couple of different approaches in the Java version to compare them. In Python my instinct was to take a functional approach: consume a frame, produce a score and the unconsumed frames. Consuming a frame is easy to build up test by test, to move from adding scores to handling the more complex features of spares and strikes. Once a frame can be handled, a whole game is easy.

So, tests first, in the order written.

import unittest
import bowlinggame

class Test(unittest.TestCase):

  def testScoreNumberTurn(self):
    (score, remainder) = bowlinggame.processTurn("12")
    self.assertEqual(3, score)
    self.assertEqual(0, len(remainder))

  def testScoreNoHit(self):
    (score, remainder) = bowlinggame.processTurn("--")
    self.assertEqual(0, score)
    self.assertEqual(0, len(remainder))
  #I'm sufficiently confident of the way scoreNormalRoll is being used not to test "1-" and "-1" etc
    
  def testScoreNumberTurnNotLast(self):
    (score, remainder) = bowlinggame.processTurn("1234")
    self.assertEqual(3, score)
    self.assertEqual(2, len(remainder))

  def testScoreSpareTurnNotLast(self):
    (score, remainder) = bowlinggame.processTurn("1/34")
    self.assertEqual(13, score)
    self.assertEqual(2, len(remainder))

  def testScoreStrikeFollowedByTwoBalls(self):
    (score, remainder) = bowlinggame.processTurn("X12")
    self.assertEqual(13, score)
    self.assertEqual(2, len(remainder))

  def testScoreStrikeFollowedBySpare(self):
    (score, remainder) = bowlinggame.processTurn("X1/")
    self.assertEqual(20, score)
    self.assertEqual(2, len(remainder))
    
  def testScoreStrikeFollowedByNumbers(self):
    (score, remainder) = bowlinggame.processTurn("X12")
    self.assertEqual(13, score)
    self.assertEqual(2, len(remainder))

  def testScoreStrikeFollowedByStrikeNumbers(self):
    (score, remainder) = bowlinggame.processTurn("XX23")
    self.assertEqual(22, score)
    self.assertEqual(3, len(remainder))
    
  def testScoreStrikeFollowedByStrikes(self):
    (score, remainder) = bowlinggame.processTurn("XXXX")
    self.assertEqual(30, score)
    self.assertEqual(3, len(remainder))

  def testSimpleGame(self):
    score = bowlinggame.processGame("12345123451234512345")
    self.assertEqual(60, score)

  def testPerfectGame(self):
    score = bowlinggame.processGame("XXXXXXXXXXXX")
    self.assertEqual(300, score)

  def testHeartbreakGame(self):
    score = bowlinggame.processGame("9-9-9-9-9-9-9-9-9-9-")
    self.assertEqual(90, score)

  def testAllSparesGame(self):
    score = bowlinggame.processGame("5/5/5/5/5/5/5/5/5/5/5")
    self.assertEqual(150, score)

if __name__ == "__main__":
    #import sys;sys.argv = ['', 'Test.testName']
    unittest.main()

And the code. The various methods were added to as the tests were produced, with a couple of refactoring passes.

'''
Bowling Game Kata
Created on 11 May 2015
@author: Dan Chalmers 
'''


def processGame(line):
  score = 0
  for _ in range(0,10):
    (turnScore, line) = processTurn(line)
    score += turnScore
  return score  
  
def processTurn(throws): 
  if throws[1] == '/':
    return (10+scoreExtraBalls(1, throws[2:]), throws[2:])
  elif throws[0] == 'X':
    return (10+scoreExtraBalls(2, throws[1:]), throws[1:])
  else:
    return (scoreNormalRoll(throws[0]) + scoreNormalRoll(throws[1]), throws[2:])

# Strikes and spares get 2 or 1 extra balls, with no further extras, that just score      
def scoreExtraBalls(extras, throws):
  if extras == 0:
    return 0
  # A spare extra ball will give 10 overall, needs some special code. 
  # But a strike in the extras works as a number!
  elif extras == 2 and throws[1] == '/':
      return 10
  else:
    return scoreNormalRoll(throws[0]) + scoreExtraBalls(extras-1, throws[1:])
    
def scoreNormalRoll(throw):
  if throw == '-':
    return 0
  elif throw == 'X':
    return 10
  else:
    return int(throw)

A version which pre-processes spares to 10-previous (and X to 10 and - to 0) would be almost as fiddly as this, even if the core algorithm is more compact. In this version the connection between the special outcomes and the behaviour is arguably clearer.

I've written using the "consume a turn" approach in Java too, and being able to return a pair rather than making a class to generate Frame objects instantly makes this neater. The Python version has fewer named functions, and these functions are marginally more complex - but, I think, readable entities.

I think that this Kata fits some of Python's strengths neatly: a functional approach without needing to store state, and processing a stream of values (my Phone Numbers solution also played to this).

Learning Python (2) - Phone Numbers Kata

2015-05-08T18:51:00.000+01:00

Another instalment in the "showing my first attempts" (and being open to any suggestions) series...

Last night, while waiting for the election coverage to warm up, I sat down with a drink and did another kata. This time it was the "phone numbers" Kata, based on the description in The Coding Dojo Handbook by Emily Bache. Which has a nice tidy spec:

Given a list of phone numbers, detect whether any are inconsistent, ie that is one number is a prefix of another.

In addition, the intention was to avoid explicit loops and have it be reasonably efficient for large data sets. In this case a recursively built tree representation of numbers came as the natural approach that I had sketched in my head before starting. (Just enough planning, even when test driven.)

So, tests first. Written top down.

import unittest
import phonenumbers

class Test(unittest.TestCase):

    def setUp(self):
      self.tree = phonenumbers.create_empty()

    def testFirstNodeCreated(self):
      added = self.tree.add_number("1", "Ann")
      self.assertTrue(added,"first number failed to add")

    def testSecondNumber(self):
      self.tree.add_number("1", "Ann")
      added = self.tree.add_number("2", "Bob")
      self.assertTrue(added,"second number failed to add")

    def testSecondNumberDuplicate(self):
      self.tree.add_number("1", "Ann")
      added = self.tree.add_number("1", "Ann2")
      self.assertFalse(added,"second number added but duplicate")

    def testTwoDigitNumbers(self):
      added = self.tree.add_number("11", "Ann")
      self.assertTrue(added,"first two digit number failed to add")
      added = self.tree.add_number("12", "Bob")
      self.assertTrue(added,"second two digit number failed to add")
      
    def testMixedDigitNumbers(self):
      added = self.tree.add_number("11", "Ann")
      self.assertTrue(added,"two digit number failed to add")
      added = self.tree.add_number("124", "Bob")
      self.assertTrue(added,"three digit number failed to add")
      added = self.tree.add_number("1256", "Claire")
      self.assertTrue(added,"four digit number failed to add")
      
    def testAddingPrefixOfExisting(self):
      self.tree.add_number("1234", "Ann")
      added = self.tree.add_number("12", "Bob")
      self.assertFalse(added,"prefix of existing number added incorrectly")
      
    def testAddingNumberExtensionOfExisting(self):
      self.tree.add_number("12", "Ann")
      added = self.tree.add_number("1234", "Bob")
      self.assertFalse(added,"extension of existing number added incorrectly")

    def testIgnoresPunctuation(self):
      self.tree.add_number("12 34-56", "Ann")
      added = self.tree.add_number("1234", "Bob")
      self.assertFalse(added,"did not ignore space")
      added = self.tree.add_number("12 3456", "Bob")
      self.assertFalse(added,"did not ignore dash")

    def testProcessConsistentFile(self):
      filename = 'phone_data.txt'  # Consistent
      consistet = phonenumbers.load_file(filename)
      self.assertTrue(consistet, "file " + filename + " was not consistent, expected it would be")
      
    def testProcessInconsistentFile(self):
      filename = 'phone_data_10000.txt'  # Has inconsistencies
      consistet = phonenumbers.load_file(filename)
      self.assertFalse(consistet, "file " + filename + " was consistent, expected it would not be")
      
    def testProcessLongFile(self):
      filename = 'phone_data_20000.txt'  # Consistent, by removing inconsistencies from example data. Allows rough performance check.
      consistet = phonenumbers.load_file(filename)
      self.assertTrue(consistet, "file " + filename + " was not consistent, expected it would not be")
      
if __name__ == "__main__":
    #import sys;sys.argv = ['', 'Test.testName']
    unittest.main()

No big surprises in there. The tests build up the data type that forms the tree, and the processing of input data to build it. The input files are from Emily Bache at gitbub, with the large consistent file made by editing an original. Each test required that I add some functionality, until the second and third tests with a file at the end. There were various refactorings along the way as you'd expect, and a bit more today to tidy up.

The code looks like this:

def create_empty():
  return Number()

def load_file(filename):
  tree = create_empty()
  result = True
  with open(filename, 'r') as f:
    for line in f:
      result = result and _load_line_(tree, line)
  return result
  
def _load_line_(tree, line):
  fields = line.split(',')
  consistent = tree.add_number(fields[1].strip(), fields[0])
  if not consistent:
    print line + "is not consistent!"
  return consistent

class Empty():
  ''' A node in a phone number tree. 
  Empty base type that is used to pack the array of pointers to next branches in a tree 
  where there isn't a branch for that number yet.
  '''
      
class Number():
  '''A number node, which has entries for each digit 0-9 that will contain another Node.
  '''
  
  def __init__(self):
    self.numbers = [Empty()] *10
    
  def add_number(self, number, name):
    '''Add a number to the tree, provided 
    there isn't already a number which is a prefix of this number
    or this number isn't a prefix of an existing number
    '''
    (digit, entry, number) = self._get_digit_entry_(number)
    if isinstance(entry, Empty):
      self.numbers[digit] = self._make_next_node_(number[1:], name)
      return True
    elif isinstance(entry, Name) or len(number[1:]) == 0:
      return False
    else:
      return entry.add_number(number[1:], name)
      
  def _get_digit_entry_(self, number):
    if number[0] == ' ' or number[0] == '-':
      return self._get_digit_entry_(number[1:])
    
    digit = int(number[0])
    entry = self.numbers[digit] 
    return (digit, entry, number)
        
  def _make_next_node_(self, number, name):
    if len(number) > 0:
      node = Number()
      node.add_number(number, name)
      return node
    else:
      return Name(name)
  
  
class Name():
  '''A leaf node in the number tree, representing a person with a complete phone number.
  '''
  def __init__(self, name):
    self.name = name

The use of classes betrays an OO habit, but I got a better idea of making Python classes through this. I had started with something closer to a Haskell data type in my head. I also had a version along the way with an abstract base class, when I had forgotten that lists in Python don't require a common type. That's been refactored out, but was an interesting learning step as it was a language feature I hadn't tried out before. I daresay the Empty and Name classes could be replaced with something built in, but I'm happy with the added expression having them gives.

The add_number function was built up gradually, a clause at a time, with the tests driving additions. But as I said above, I had a plan for where I was going from the start. Handling the punctuation in input files was a requirement that emerged in addition to that plan.

The line by line processing of the file didn't easily submit to using map and list comprehensions, although I did try. Given that the data set may be long I prefer the line by line approach to a "load file then process" approach. So, I have one explicit for loop.

And I was done well before the night got interesting.

Learning Python (part 1)

2015-05-05T15:56:00.002+01:00

My programming language CV is pretty Java centric since about 1999, with some Haskell along the way, a general background of SQL and shell scripts for most of my career and a selection of languages that are very much occasional or in the past and I'm not going to feed to a search engine!

An impending change of day-job sees me needing to get to grips with Python. It's a language I've been aware of for years, but never had cause to use until now. So, how to get familiar? Instalment one of what ought to be a series below. I'm deliberately exposing the learning process here - I'm not going to make claims that these are brilliant examples of python. It will be interesting to return to these as I develop...

Step 1, is obviously to take a look at the python.org web site and get a general feel. Maybe write "hello world".
Step 2, is to run through a tutorial. It isn't one of the "7 languages" so I looked around and found Google's Python Class and worked through that. It was pretty useful, gave me a sense of the strengths and style of the language, and an overview of key features.
Step 3 will be to have a bash at a few Katas. I'm a firm believer in learning by doing and in the case of programming doing comes out as a mix of thinking and typing. I want to do some more or less unguided problem solving, and find an approach that fits the way I like to (behaviour in unit) test drive my code. Katas give a good chance of finding a selection of other solutions out there to compare with and tend to be a tidy quiet-evening self contained activity.
Step 4 will be to build something bigger, probably with Django. But I'm still firmly at step 3.

First up was the "String Addition" kata, as described by Roy Osherove. My code is below, tests are written in the order I wrote them.

import unittest
from string_calc import add

class Test(unittest.TestCase):

    def testAddNothing(self):
      self.assertEqual(0, add(""), "empty string should return zero")
        
    def testAddOne(self): 
      self.assertEqual(1, add("1")) 
       
    def testAddTwo(self):
      self.assertEqual(3, add("1,2"))

    def testAddThree(self):
      self.assertEqual(6, add("1,2,3")) 

    def testAddLongerNumbers(self):
      self.assertEqual(111, add("1,10,100"))
      
    def testHandleWhitespace(self):
      self.assertEqual(10, add("1\n2, 3 4"), "not handling whitespace")
      
    def testShouldRejectNegatives(self):
      self.assertRaises(ValueError, add, "1, -2, 1")
      
if __name__ == "__main__":
    #import sys;sys.argv = ['', 'Test.testName']
    unittest.main()

'''
String Calculator Kata
from problem by Roy Osherove
Created on 3 May 2015

@author: Dan Chalmers
'''

import re
from sys import argv

def add(number_string):
  # break the string into a list of number strings
  number_list = re.findall(r'-?\d+', number_string)
  
  # check for bad input
  if any(number[0] == '-' for number in number_string):
    raise ValueError("")
  
  return sum(map(int, number_list))
    
  
if __name__ == '__main__':
    print(add(argv[1]))

So, what did I learn?

Some familiarity with PyDev in eclipse. I spent a little while working out what was there and how I'd want to use it in a bigger project, not just pushing at the path of least resistance.
A first go with unittest. This was agreeably straightforward, although I'd cheerfully loose all those "self." references.
Reminded myself of the basics of python and found that things were easier by the end.
There were a few multiple tests failing runs while I sorted the regular expression and use of re out, but generally each move added something or refactored without breaking the earlier cases.
I found myself moving from
split(<separator>) to
re.findall('\d+', numbers)
in order to handle newline and commas together. At that point being able to specify delimiters stopped making much sense as well formatted input is promised. Using a number as part of a delimiter felt bloody minded so had a slight spec out of the pram moment!
Gary Bernhardt's solution video reminded me of sum, any and map, so I refactored to use them as I do like that more functional style and wanted to get it in my fingers. Before that it used explicit loops.

Next stop, something maybe a little more involved and with a deliberate "take a functional / no loops" constraint.

Moving to Docker

2015-05-05T15:21:00.000+01:00

I know, everyone's been excited about Docker lately. Time to join in.
Why?
For this project, I see three advantages:

Between my development environment, CI server, and deployment I've got some very similar things running. But not quite the same, e.g. deployment has an nginx front end, non-development has firefox and Xvfb running for testing. However, I'd really like the common bits to be identical to make the tests as useful as possible. Lightweight builds of platform updates would be good too.
Auto-deploying new web apps from Jenkins is suffering from a little friction. At the moment it involves a manual step. I'm sure there are other ways of solving this, but as I still haven't done so a way of running new versions for low effort and a reasonable security risk remains on my shopping list. Maybe Docker can help with this?
The distribution of services between servers ought to be a flexible thing. Docker does that, and from a first look version controlled Dockerfiles lend themselves to self-documenting, generated from a script containers.

What I'm not planning to do, first off at least:

Docker for my CI master. There's only one, it just has Jenkins and Java, that can stay as a VM.
Docker for my data volumes. For the databases there will just be one of each master / slave. All of the web content comes from the project build. The big user generated data files go to S3. So there isn't much need for deploying multiple copies. I'd also like to keep the data as just a disk for now as that ought to make reversing out of docker easier, should I choose to do so.

Just One

2015-03-20T14:39:00.000+00:00

I was eyeing up the current (and forthcoming) mac laptops the other day, as you do. I was struck by the "one port" fascination. I've yet to buy anything that uses thunderbolt, so the USB-C thing makes no difference to me. But I am struck that that having one port isn't a major leap forward - it just offloads the demultiplexing problem somewhere else.
For instance, as I sit here at my desk my laptop has 5 wires coming out of it: power, ethernet (wireless in my office is variable), backup disk (happens to be old enough for firewire but can use USB), pictures to the monitor and USB to the monitor. Before I wrote this I copied a file to a USB memory stick, in case things aren't as expected where I'm going later. The monitor's USB ports are full too: audio interface, keyboard, mouse and fitbit dongle. I also often plug in a cable to access a phone; my laptop connects to a projector on average once a week.
At home there's a similar story: I've got a backup disk (spot the ex-sysadmin who found time machine over wireless unreliable), an external disk of pictures and music (editing photos is just faster on a wire than a wireless network), an ipod, a camera (memory card or USB), occasional connection to the TV for family streaming, occasional USB stick and power. And with no external screen to act as a hub and moving between rooms with the computer no fixed place for a separate box to demultiplex whichever combination of stuff I'm using.
So, if I got a new laptop, could I make one port work? Probably. Though it would involve spending money on new adapters, maybe hardware and some rechargeable batteries for wireless stuff just to get the current, or equivalent, gadgets connected. Would I gain anything? Not as far as I can see. I'd still have boxes and wires, just slightly rearranged. I don't recall ever thinking "I wish I had fewer sockets on my computer".

Which gets me to the software analogy. Message busses are all very well. I've written and used more than one. But if they're the only way of communicating in the whole system, they bring their own challenges.
Scalability for one.
Fit to function for another. I don't mind having a separate channel for audit logs, or system management. If the bus breaks or load goes up I still want these important functions, without creating headaches for security, priority and API clarity.
Does doing all communications (or persistence) in one way buy me anything? I like reuse and patterns as much as the next OO-person. There are limits to how much complexity I want to put up with and a common approach can reduce both local and system complexity by sharing the overhead. But, increasing system complexity needs this payoff. Some connections are special, some adapters are too expensive.
A fundamental point I make when training people to solve problems is that there are many ways to solve even quite simple problems. Some may be better than others, but the concept of a solution space, where each solution has its own properties, is an important one. If you vary the parameters of the problem you vary the point in the solution space that is the best fit. There may be a solution that is a pretty good fit to many problems, but far from ideal for some cases. Shading the whole of this space with the biggest crayon you can find isn't big or clever.

Continuous Deployment and Programmers on Call

2015-03-06T15:39:00.000+00:00

Two days ago I was attending an interesting Brighton Java talk on continuous deployment by Jose Baena followed by a discussion led by James Stanier. I heard about some new tools; got reassurance that what I was doing was also, roughly, someone else's approach; and that the tricky problems were felt by others too.
One of the discussion points was "Should developers always be on-call for their code?". My gut reaction was "always? No! I've done that - 18 months with a pager - no way, it's soul sapping." In the shower this morning I arrived at a more nuanced answer, that (of course) agrees with my gut reaction. The reasons are intertwined, bear with me while I pull them apart a bit...
First, there's some basic pragmatism: What do you do with the code written by someone that left? If you pair program, who's on call - both of you, take it in turns? If I've maintained a method in a class do I suddenly get the entire class?
Second, and leading from that thought, is the issue of who dials the number of the right person? If an organisation gets beyond about 6 people (arguably 2 if you work in layers of a stack) there will be failures where it isn't immediately apparent who owns it. Now you need 1st and 2nd line support. By the time first line has isolated that it is a particular unit which is at fault, not the database or the user interface it connects to, then they are probably in a position to make the fix faster than they can get the 2nd line on the job.
James offered the thought (bait?) on the night that being on call might prompt a sense of responsibility. The come back from the floor (not me) on that was that developers paid for call outs wouldn't be motivated to produce good code. Getting called becomes a source of income. My experience is that after a while, with a decent wage, the intrusion on my life mattered more than the double time. However, to some extent I think he is right - throwing code over a wall to be someone else's problem isn't how I like to work. Even in the 1990s I mixed development, test and support. However, it does lead me to my third point: given the first two issues, it is clear that matching an individual to a block of code is problematic. Many problems require that the on-call person has the sophistication to move up and down the stack, look at systems issues, understand the relationships between the parts. So, maybe the on-call needs to be at a team level. There's still a responsibility. But there's also the responsibility to your team mates' social life, not to mention being fingered for code which was hard to understand and fix!
And there's the fourth point: Yes, feel responsibility for your code. This isn't ownership* though. As a team the power to fix the code needs to be dynamically allocated.
So, being on call - as a team, but on rotation - motivates responsibility and quality. It gives a reason for good testing, code reviews, not sitting in a corner but getting the whole team's work, and seeing the full development cycle that putting development and ops together implies.

* I don't mean avoiding "ownership" in the sense of "I'm going to own this problem", but in the sense of "this code is my precious, it's mine!" or "I'm leaving the company and taking it with me".

Thinking and Programming Videos

2015-01-18T11:07:00.002+00:00

I'm in the process of making some videos (screen casts), showing how I arrive at solutions to programming exercises. They are for students learning programming, rather than experienced programmers, though that would be interesting too. The idea of using video is to expose the thought process. I'm not trying to show "follow these steps to produce this code". The problem solving skills that lead to those steps are the interesting thing.

Producing the videos has been a learning experience for me too. The sustained thinking aloud is quite fun. Recording and editing is quite a time sink, but is gradually becoming smoother - and the results more polished as it becomes more familiar. The unexpected thing was how much it exposes the stuff that interferes with mental flow: being hungry, interruptions, outside noises and so on. Not only am I editing out the interruption, but there's a typically a run of edits required as I make a couple of small mistakes or fail to express myself as clearly as I'd like over the few minutes that it takes to get back into my stride.

The video I'm happiest with so far is below. (It is also the shortest!) This follows the bit of the "Money" problem where inheritance isn't the solution. In some ways this was also the most challenging lab so far. Most of the time the thing to show is "the right way". Here the exercise is, at least partly, about recognising when things have gone wrong and rolling back is the best choice. (It is also about the composition vs inheritance choice, but composition is the next exercise.) Important issues, but in my experience "this is wrong and here's why" is avoided in the classroom.

TripVis MVP

2014-12-05T13:34:00.003+00:00

I have a minimum viable product!

Minimum: there really isn't very much I could strip out, without loosing something. It is running on the smallest setup, its a bit slow, and there's already a backlog of ideas. But they all add on to this. Yes, there's an about page and a terms page, but I think they signal something important about intentions and reasons for something I'm putting out there.
Viable: well, it does something. Whether anyone else wants it remains to be seen, but that's the point.
Product: yes, it's a thing (a cloud hosted web service if you want). You can use it. One day I might add paying for it, and make it a product in the business sense of the word. But for now it is a pet project and a vehicle for exploring ideas, an ideas product.

So, here it is: tripvis.co.uk

Not too much fanfare for now - try a few things, see what happens. If you give it a go I'd be delighted to hear from you: contact@tripvis.co.uk

Emergent Technology and Economic Transformations

2014-12-05T11:43:00.002+00:00

BBC Radio 4's Start the week with William Gibson and Judy Wajcman (and others, but I mention them because it is their comments I respond to below) on science fiction on 24th November touched on issues that I've read about in Economic Transformations by Lipsey, Carlaw and Bekar.

The Gibson quote that hit me was "emergent technology is the big change driver in society and long has been", which then led into a comment that technology emerges rather than being legislated into existence and that we're not unique in living in a time of technological change. This idea that technology has produced unpredictable but sustained economic change over a long period is core to the economics book. They also discuss that these changes are not directly tied to individual products but to more fundamental technologies. Not every new product can have this large-scale, sustained transforming approach, most are part of a larger technology. Many significant social changes are prompted by combinations of technology, rather than individual new technologies. These might build on each other, e.g. electricity then telegraph; or in combination, e.g. DNA and computers. Another issue they discuss at length is that the emergence of the technology, its widespread application, and the social / economic benefit are often separated by a significant lag. This isn't just buying new gadgets beyond early adopters, but the application of technology in ways that create efficiencies at a social level which then enables large scale changes in patterns of living and production - such as the connection between steam power, factory production and movement from cottage industry to city living. It is the social effects which make the technology a significant driver of the economy, and the technology needs experimentation and a foundation of science to reach the stage of having widespread, substantial and sustained effects as the basis for products but transcending an individual product.

On the radio they touched on many other issues. I'm not going to comment on everything, but busyness was another: the need to be constantly doing something (or at least be seen to be busy), to be up to the minute. Despite a pair of jobs, a hobby project, a blog and a couple of Twitter accounts (and a non-tech life) I'm going to claim bucking the trend here! I've been meaning to write about that book for a while, but haven't found the seed for something to write until now. Because I don't engage on a real time basis, but via a podcast, it has taken me until yesterday to listen to the programme. Because I try to percolate what I blog, let the ideas filter and brew, there's another day to posting here. Good ideas don't have to be had now. Despite first mover advantage, or the need of others to build on your idea, the quality of the idea matters too. Reflecting the previous point, big ideas that really impact what you do take time. In my technical work I've found that some of the most fundamental changes to how I work are based in principles and connections between ideas that have matured over time. The maturation happens both in the wider world, but also in my mind - satisfying a need via an approach rather than a specific thing.

Of course inspiration finds you when you're working, but you don't need to make a show of working for inspiration to find you - because the show is rarely the work. The gaps between work are really important if you work in ideas, they give the brain breathing space ... listen to a podcast, read a book!

Comment On Slik and Spinach

2014-11-07T08:52:00.000+00:00

I felt moved to comment on a post on Kevin Rutherford's blog, about code design, and forgot to link from here...

I couldn't resist a note of humour about the analogy, but the real point was about design. For the most part I was agreeing with him: Exceptions are no place for routine events. But, "loosing a conditional" was, in my opinion, over egging the pudding. The situation would still need handled, just somewhere else. In the case of his example, by another actor (think micro-service, probably).

Anyway, it's a good post and my comment is below silk and spinach - on paperboys, newsagents and exceptions.

9 Tests with FluentLenium

2014-11-06T18:02:00.001+00:00

As well as work projects I like to have a personal project which lets me try new things. As past blog entries may have suggested the current one is a web site, running on NinjaFramework. Having got the hang of the core underlying technology I had produced a chunk of MVP site. Now, rather belatedly, it was time to introduce tests on the web interface as well as the code. Ninja has FluentLenium packaged in. It functions as a fluent wrapper to Selenium. FluentLenium was new to me and so I set about breaking it. Here are the edited highlights of that process and a few lessons learnt (breaking things is a really good way to learn).

My starting point was the simplest possible test: load the home page, check that it had some text that was about right (key words in key places rather than full text). Use the example code as a basis.
I started with the HtmlUnitDriver as the engine. It worked.
Next I tried navigating to the about page. Still nothing clever behind the scenes. Then pressing the back button. Ah. Need to switch javascript on. Now it moaned bitterly in warnings about a lot of things, mostly because I'm using Bootstrap.
Another two "load a simple page" tests followed. The warnings were a bit tiresome. The code was getting scrappy. Although the API was FluentLenium they were reading more like a script. Time for a bit of structure, with common elements of the scripts factored into methods and a bit more of the fluent style.
Next test was moving into testing user registration, login, and logout. This involved a form, inputting data, pressing a button that (via javascript) would POST the data and AJAX the response. HtmlUnitDriver was no longer the tool for the job - its JavaScript just isn't up to it. I already had firefox on my local test VM. I was able to add it to my CI slave, the only problem being getting a compatible version of firefox (see here). That also made all the warning noise go away, which was nice.
But, this threw me into the world of asynchronous interactions and waiting for things to happen at the same time as AJAX updates to pages and cookie setting that would vary depending on the success / failure of the operation. The script approach grew messier and the tests less reliable. Pretty soon I had a test of register - login - logout - log back in that just wouldn't work. I could see the web page on the screen updating, but the tests were either telling me that it hadn't changed; or that the cached copy of the page was stale. Later (with lots of await statements and some pruning of the tests) I had passing tests on my laptop which broke on CI.

The details of the route out aren't important, although there is a snapshot encapsulated on StackOverflow. The eventual approach is the interesting thing:
First, embracing FluentLenium and its approach and not poking about with the underlying Selenium. There was a big refactoring into using the Page design pattern, and @Page annotations. This, obviously, started with commenting out the breaking code and reworking the things that I already had working. The separation between user-level script and web-page-internal tests really helped clarity. There was enough similarity between the two test classes that they started to share some code in a super class (mostly starting the driver); and also between the pages (all the static final Strings that refer to the web page and tests for logged-in state that come out of common framework HTML rather than being page specific). Indeed, in a site that uses templates this is to be hoped for.
Next, using the @AjaxElement. Well, I'd tried that along the way, but it didn't help. Because of the next thing. But, as part of the overall picture it works and makes the rest work!
Then, simplifying the tests. The scripted approach encouraged adding bits to the stories represented by the test. I needed to remember that the existing test was testing something. The additions were new tests. In particular this meant I was coming back to @Page objects and expecting change, but finding breakage. Each page field could only be used at one place in the test; once navigation moved on the field was no longer helpful. Getting an @AjaxElement works once; not for a whole string of prod / test cycles. Having more than one field for the various visits to a given page was considered, but could always be avoided so far by having the tests test just one thing.
Finally and most trickily, making the assert call to test that the @AjaxElement is what it ought to be from the test object rather than within a method of the page object. Not an immediately obvious step. So this (for me) is broken:

class Page {
  @AjaxElement e;
  public void checkE() {
    assertSomething(e.getText().contains("stuff"));
  }
  ...
}
class ETest {
  @Page
  EPage p;

  @Test
  public void test_eStuff() {
    goto(p);
    p.checkE();
  }
}

and this is not:

class EPage
  @AjaxElement e;
  public void getE() {
    return e.getText();
  }
}
class ETest {
  @Page
  EPage p;

  @Test
  public void test_eStuff() {
    goto(p);
    assertSomething(p.getE().getText().contains("stuff"));
  }
}

There was also some fixing of HTML id tags to aid testing that went on, but no real change to the pages (although I did contemplate this at one point).
Since that breakthrough I've added another two tests of user account activity, each with as much ease as I added that second test to load the about page. The code reads well and the collection of classes isn't expanding so fast. Tests still pass on CI. I'm happy. New features can be developed against tests like this, in outside-in style.

Cut and Paste

2014-10-25T21:38:00.000+01:00

I replied on Twitter to Grady Booch and Valentin T Mocanu earlier (Booch in turn was commenting on a Quora answer). Being Twitter it was a bit brief and slightly flippant. The thrust was "cut and past programming is fine" ... "fine for the first 2-3 years".
My contribution: using example code teaches what you ask of it.

I've learnt a lot with examples; I've taught a lot with examples. But, what I learnt by using examples was not how to touch type from a magazine, nor how to copy and paste from examples. Like most programmers that work with others, I look at what others write. Sometimes it gets me through a block. Sometimes I see something new. As I use it I mentally fit it to my existing models. Then I prod what I've got, see what breaks it, how it changes, what else the documentation says about this new thing.

When I use a new library or service I often reach for example code. OOP is very good at giving us generality: lots of classes, lots of methods. But the first use of something is usually pretty close to the common case. API documentation rarely describes that (many thanks to those that do!) and simplifying the common case sometimes gets lost in the quest for elegance. Examples show that path. Then, as before, documentation can be explored to vary from this. Exploration gives a structure to build a model from.

Testing, of course, is a great tool for doing this exploration, prodding an idea and seeing what happens. Asking "my code does X, but what I really want is Y, how do I get there?".

Blind cut and paste, of course, teaches next to nothing and doesn't solve all problems. You can get by on it, but it would be hard to tread a new path that way. I see this frequently, and in many ways it puzzles me. One of the great attractions of programming, to me as a kid, was that I had an infinite set of problems to solve to make the computer do something fun. And, unlike electronics which I also enjoyed, the chances of my programming breaking the computer in a way I couldn't fix were small. (Of course if you ask the 13 year old me you wouldn't get that as an answer!) But, some people get the fear when you suggest that they try something new with some example code. I don't think it is fear of breaking it that is the barrier really, but a self-perception of incompetence - much as is involved in shyness. A lack of belief that they have the mental tools to make that exploration, or that their tools aren't as sharp as others' tools. Leading people into sharpening those tools, and believing that they can do this is quite a lot of what teaching programming comes out as. Give people the confidence to have a go, some pointers and feedback, and a few principles for the next round of exploration and pretty soon you have Kolb's learning cycle.

The other part of my answer was "Also good for using libs that are short on principles themselves." Maybe more than slightly flippant. But, occasionally, I use some library or tool, and find myself wondering how it works. An example gets me started. But straying from the one true path leads to failure. Sometimes there's a principle which has been broken or hidden or isn't the principle I was hoping for. Sometimes it's all a bit ad-hoc, a "tool" extracted from a single project that hasn't yet evolved principles and general applicability away from the use-case of the original project. Sometimes there's a principle, but it is unevenly applied.

I'll finish with an example: Java's ImageIO Input and Output streams and the standard Input and Output streams. Getting an Image written to an output stream and passed onto an input stream elsewhere, without specifying writing to a file, really could to be easier. The ImageInputStream and ImageOutputStream don't extend InputStream and OutputStreams, indeed an ImageOutputStream is also an ImageInputStream while the non-Image versions don't relate like this. *NIX is deep enough in my blood that I want pipes to do both input and output and just work, but in this problem the streams are in the same place and Java's PipedStreams need mucking about with threads if there's much data (and plugging a piped pair of streams into an ImageOutputStream without extra threads certainly broke for me). So, here's my example (arrived at by seeing several other examples, relating to my mental model of images and I/O, exploring, adjusting and combining):

  
  ByteArrayOutputStream os = new ByteArrayOutputStream();
  doRender(data, os); 
    /* method arranges the rendering object with the right parameters leading to
       ImageIO.write(theRaster, imageType, outStream);  */
  byte[] imgBytes = os.toByteArray(); // you know where you stand with an array
  ByteArrayInputStream is = new ByteArrayInputStream(imgBytes); // clunk
  docStore.putData(data.getName(), RENDER_MIME_TYPE, is, imgBytes.length);

This works - the rendered file ends up on my docStore (S3 in this case). But there's a part of me that would really rather not see that array. My images are (already) 60-80kB. I'm aware that at some point memory use may prompt me to revisit this (but lets not over-engineer it yet).

So, there you are: an example for you, which I wrote (today as it happens) by learning from a mix of examples and API documents (some of that learning happened years ago, of course). Whether the example is necessary because I misunderstood the principles of the libraries, or because the libraries haven't best applied available principles I'll leave to you to decide!

CouchDB Install

2014-10-10T16:14:00.000+01:00

I've installed CouchDB a couple of times recently, for development VMs and then adding it to an existing Jenkins CI image now that my project has a test that needs it. This isn't as simple as "yum install..." yet, at least for the systems I've tried - although an Ubuntu package is available. Continuing the tradition of this blog documenting installations, below is a log of the Jenkins slave install, which follows what I did on my CentOS VMs but without the mistakes and dead-ends. This runs on an AWS EC2 t2.small instance, with a configuration including Amazon's 64bit Linux, Ninja Framework and PostgreSQL mostly described in previous posts. So, I assume you have a basic machine up and running.

First some package installs, mostly as advised by the documentation on the other packages to install, but starting with the inevitable update:

sudo yum update
sudo yum install autoconf curl-devel ncurses-devel openssl-devel
sudo yum install gcc gcc-c++ libicu-devel

Erlang is built from scratch: to get a current version and to avoid a crypto fail in some Linux distros (including CentOS which I have on my development VM, which allows erlang and couchdb to compile and install, but fail on starting couchdb). Couchdb is compiled against this later.

curl -O http://www.erlang.org/download/otp_src_17.3.tar.gz 
tar xzf otp_src_17.3.tar.gz
cd otp_src_17.3
export ERL_TOP=`pwd`
./configure
make
// make a cup of tea
sudo make install
cd ..

Next, we need Python, in particular the 2.7.x version, and SpiderMonkey, again not the newest one, but version 1.8.5:

curl -O https://www.python.org/ftp/python/2.7.8/Python-2.7.8.tgz
tar xzf Python-2.7.8.tgz
cd Python-2.7.8
./configure
make
sudo make install
cd ..
curl -O http://ftp.mozilla.org/pub/mozilla.org/js/js185-1.0.0.tar.gz
tar xzf js185-1.0.0.tar.gz
cd js-1.8.5/js/src
./configure
make
// start to drink tea
sudo make install
cd /etc/ld.so.conf.d
sudo vi local-x86_64.conf
and add the entry: /usr/local/lib, so that the js libraries are picked up.
sudo ldconfig

And now to CouchDB. Note that they have mirror sites, the one I used might not be the most appropriate for you.

sudo adduser --system --home /usr/local/var/lib/couchdb --no-create-home --shell /bin/bash --comment "CouchDB Administrator" couchdb
curl -O http://mirror.ox.ac.uk/sites/rsync.apache.org/couchdb/source/1.6.1/apache-couchdb-1.6.1.tar.gz
tar xzf apache-couchdb-1.6.1.tar.gz
cd apache-couchdb-1.6.1
make
// tea is about right now
sudo make install
cd /usr/local/var/run
sudo chown -R couchdb:couchdb couchdb/
sudo chmod 0770 couchdb/
cd ../log/
sudo chmod 0770 couchdb/
sudo chown -R couchdb:couchdb couchdb/
cd ../lib
sudo chown -R couchdb:couchdb couchdb/
sudo chmod 0770 couchdb/
cd ../../etc
sudo chown -R couchdb:couchdb couchdb/
sudo chmod 0770 couchdb/
cd rc.d
sudo cp couchdb /etc/rc.d/init.d
sudo service couchdb start
sudo chkconfig --levels 235 couchdb on

To check it is working, curl http://127.0.0.1:5984/ should respond with something like:

{"couchdb":"Welcome","uuid":"d420a7e9b585cb74039a54f1874b6a0f","version":"1.6.1","vendor":{"name":"The Apache Software Foundation","version":"1.6.1"}}

Finally, tidy up the download and build directories; make a new AMI; and point the Jenkins EC2 plugin's config for the slave at that AMI; and test. For the purposes of a short-lived CI slave whose firewall only talks ssh password protecting access with new admin users is probably overkill. However, for my application there is a database creation script which makes some assumptions about admin users; and some test cases which assume a test database. So, in the Jenkins config for the job pre-steps we have:

curl -sS -X PUT http://localhost:5984/_users/org.couchdb.user:dan -H "Accept: application/json" -H "Content-Type: application/json" -d '{"name": "dan", "password": "mumblemumble", "roles": [], "type": "user"}'
curl -sS -X PUT http://dan:mumblemumble@localhost:5984/test
dbConfig/create-db.couch

And observe the test passing!
For completeness, that first test is below. It doesn't do anything terribly clever: posts a new document on CouchDB and gets it back. However, it re-uses the NinjaProperties fake idea from my previous post about fakes and injection, so completes two loops!

package models;

import java.util.HashMap;
import java.util.Map;
import java.util.UUID;
import models.PersistCouchDB.IDRev;
import org.junit.AfterClass;
import org.junit.Test;
import static org.junit.Assert.*;
import org.junit.BeforeClass;

/**
 * @author Dan Chalmers
 */
public class PersistCouchDBTest {
    
  public static final String DB_NAME = "test";  // note, this needs to be created on CouchDB in the test environment!

    static PersistCouchDBStub db;
    
    @Test
    public void test_postNewDoc() throws BrokenSystemException {
      HashMap map;
      String randomString;
      IDRev postResult;
      Map getResult;

      map = new HashMap();
      map.put("testing", "1two3");
      randomString = UUID.randomUUID().toString();
      map.put("random", randomString);
      postResult = db.postNewDoc(DB_NAME, map);
      assertNotNull(postResult.id);

      getResult = db.getData(DB_NAME, postResult.id);
      assertEquals("1two3", getResult.get("testing"));
      assertEquals(randomString, getResult.get("random"));
    }

   static class PersistCouchDBStub extends PersistCouchDB {
     /* 
     PersistCouchDB is abstract as it just 
       - reads the relevant Ninja configuration 
       - and contains helper methods to arrange the basic access patterns.
     This works, but doesn't do anything very helpful with real data.
    */
  }

  @BeforeClass
  public static void createDBObject() throws BrokenSystemException {
    FakeNinjaProperties props;

     props = new FakeNinjaProperties();
     props.put("couchdb.readserver", "localhost:5984");
     props.put("couchdb.writeserver", "localhost:5984");
     props.put("couchdb.connection.username", "tripvis");
     db = new PersistCouchDBStub();
     db.mockLoadProps(props);
     db.mockCreateJsonMapper();
   }
}

Ninja, Injection and Fakes

2014-09-06T21:18:00.000+01:00

The Ninja Framework has a convenient system for loading properties from a configuration file, application.conf. This includes a system of modes, where a property can be prefixed with one of three modes: dev, test or prod. So, when Ninja is running as a web server the configuration can easily be adapted to the three common situations, while keeping a single configuration file in the version control and CI process. This looks something like the following:

application.conf:
db.connection.readurl=jdbc:postgresql://localhost:5432/tripvis
db.connection.writeurl=jdbc:postgresql://localhost:5432/tripvis
db.connection.username=tripvis
db.connection.password=YeahRight

@Singleton
public abstract class PersistDB {

    @Inject
    NinjaProperties ninjaProperties;

    public static final String DB_READ_SERVER_URL_PROP = "db.connection.readurl";
    public static final String DB_READ_SERVER_URL_DEFAULT = "jdbc:postgresql://localhost:5432/tripvis";
    public static String dbReadServerUrl = DB_READ_SERVER_URL_DEFAULT;
    
    // etc

        @Inject
    protected void loadProps() throws BrokenSystemException {
        
        if (ninjaProperties != null) {
            dbReadServerUrl = ninjaProperties.getWithDefault(DB_READ_SERVER_URL_PROP, DB_READ_SERVER_URL_DEFAULT);
            // etc
        } else {
            logger.log(Level.SEVERE, "No NinjaProperties created, database config is stuffed");
        }
    }
}

The injection saves creating the object which deals with NinjaProperties and loading the file, which is very handy. However, there is a downside. When running JUnit tests the Ninja framework isn't loaded in the normal way, so the relevant classes aren't constructed by the framework but by the test object. (Yes, yes, I know, my example accesses a database, so this is stretching "unit" a bit.) This means that the injection no longer happens for free. Indeed, you're stuck with the hard-coded defaults. This is an issue in several ways:

The test doesn't exercise the loading of the properties at all.
If the properties required by the test environment change then the code has to change.
If the developers' unit testing environment and the CI environment aren't the same then this approach stops working.

Fakes come, more or less, to the rescue. NinjaProperties is an interface, so we can create a fake properties object instead of the real one. This looks something like the following:

public abstract class PersistDB {
    public void fakeLoadProps(NinjaProperties props) throws BrokenSystemException {
        logger.log(Level.INFO, "loading fake properties");
        ninjaProperties = props;
        loadProps();
    }

public class UsersDBTest {
    static UsersDB db;

    @BeforeClass
    public static void createDBObject() throws BrokenSystemException {
        UsersDBTestFakeProps props;

        props = new UsersDBTestFakeProps();
        props.put("db.readserver", "localhost");
        props.put("db.writeserver", "localhost");
        props.put("db.connection.url", "jdbc:postgresql://localhost:5432/tripvis");
        props.put("db.connection.username", "dan");
        props.put("db.connection.password", "YoHoHo");
        db = new UsersDB();
        db.fakeLoadProps(props);
    }

public class UsersDBTestFakeProps implements NinjaProperties {

    private HashMap props;
    
    public UsersDBTestFakeProps() {
        props = new HashMap<>();
    }
    
    public void put(String key, String value) {
        props.put(key, value);
    }
    
    @Override
    public String get(String key) {
        return props.get(key);
    }

    @Override
    public String getWithDefault(String key, String defaultValue) {
        if (props.containsKey(key)) {
            return get(key);
        } else {
            return defaultValue;
        }
    }
// etc for any other methods needed for the tests

More work than injection, but mostly confined to the tests and not that tricksy.

Note, this is just a fake - a simple class that fulfils the interface, but takes a big short-cut. It is marginally more clever than the canned response from a stub, but doesn't go as far as checking that the right calls are made like a mock would. The mock approach would help test that the right queries are being made, but at the moment they're all bundled together in the loadProps() method so we're a good refactoring step away from that right now. For those that aren't familiar with the mock / fake / stub distinction I'd recommend reading Martin Fowler and Emily Bache.

Recalling the issues from before, I'm left with some caveats:

The test doesn't touch the properties file at all. You still need to do some system testing to do that. But this approach flags an issue in code (rather than config) faster than that will.
If the properties required by the test environment change then the test code has to change. Could be worse. If this is really an issue then the fake could load a file.
If the developers' unit testing environment and the CI environment aren't the same then the code above needs tweaked to spot the environment and load different values into the fake. A bit fiddly and loading a file might start looking clever as soon as this gets at all involved, e.g. with more than one developer.

Final thought: Injection is kind of cool at runtime, but not without its own issues.

Comment on "An Appeal to CS Teachers"

2014-08-19T21:32:00.002+01:00

I recently read An Appeal to CS Teachers on 8th Light which talks about recent graduate programmers using far too many comments. This resonated with some experiences I had, and links back to an earlier post here, Programming 102, on teaching Java and Unit Testing. In this article Dariusz Pasciak comments on the problem of students commenting every line of code, and proposes two reasons that this might be happening:

It helps teachers read the code.

It forces students to think more about what they're doing.

Here's my response...

I've taught university CS programming (most recently Java to 1st years, but programming in the context of other topics and a few other languages over the years) and usually give (just a few) marks for "appropriate" comments.

Particularly for the early years of a degree I do find I have to guide many of my students towards "useful" comments, rather than per-line or somewhat random commenting. A few are doing this because they are working out what is happening and worry about coming back to something that makes no sense. A few have been told somewhere in the past to comment everything, as suggest in Dariusz's post. Some are just erring on the side of caution, reasoning that "if Dan likes comments more can't hurt, right"?

The two rationales are interesting.

It helps teachers read the code.

First, I should say that most students write coursework solutions that really aren't hard to understand, although in more openly specified projects why they have chosen to write the code like that maybe less so. However, the ones that are hard to understand are often those that have missed the point somewhere along the way and are most in need of some understanding and helpful feedback. For helping me understand what the students have written I've moved to unit testing. Even the baroque enthusiasts and somewhat confused can write fairly understandable tests.
It forces students to think more about what they're doing.
As Dariusz says "Students just beginning to pick up programming already have a lot to think about.". Quite. There isn't any need to make life harder, it only ends up obscuring the point. The unit testing also works for thinking about what they're doing, but separating the thinking step out. More thinking when you're writing code that's at the edge of your ability isn't going to end well for anyone. Have a think then do the doing. Write a test; make it pass; refactor.

My standard advice to students is something like:

Write me a test case, that describes the intention of the method pretty well (unless we're talking about trivial getters, setters etc).
A good class / method / variable name removes the need for a sentence of comment. That may not remove the need for comments, but eliminates stuff that just clutters the code up.
JavaDoc style per-class and per-method comments are useful for an overall picture and also for describing how to use the code, which the code itself might not make so obvious.
By all means comment some non-obvious step, clever trick or thing to come back to. Probably at a block within a method level. Then think about whether that bit would be better refactored into something else. At that point the comment may well vanish.

So, I'm with him: no need to hide the code in a flood of comments!

Softly Softly

2014-08-01T14:49:00.000+01:00

It has been nearly a month, so what progress?

Firstly, I've had a lot of admin to deal with. Most necessary, little passes the "interesting things with interesting people" test, none merits writing further about right now!

I've been working on a website in my spare time, using Ninja (discussed elsewhere). There's been quite a lot of learning along the way (a bit more about actually using Ninja, finding my way round Bootstrap, learning how Injectors work, revisiting graphics programming I last did many years ago, weighing up design approaches...), and at some point I hope to distill a post of things that are useful, repeatable and not covered thoroughly elsewhere out of this. Right now that isn't ready. The next step involves me broadening my NoSQL horizons a bit. I can see the steps from here to a first public version, just need to find the play-time to take them.

Another project has involved a lot of looking at licences and other legal stuff, and considering how that fits products that are still being developed and a lot of "what ifs" about future investors. It isn't the technical side of building a thing, nor the working for people. It matters, a lot, to the business though so it merits the investment in time. We've also had some really excellent partners giving us advice, so thank-you to them. However, there's very little I ought to openly say about the detail. We've been "nearly there" many times over quite a long time, but today some of the deals were signed. Which is a relief.

So, the blog isn't dead, I haven't had an extended holiday and done nothing, nor taken up basket weaving. But it has been a "bunch of stuff" sort of month, rather than a "here's a thing" sort.

Blocked!

2014-07-02T18:16:00.001+01:00

The Open Rights Group have produced the Blocked! web service that allows you to test whether sites are being blocked by the major U.K. Internet Service Providers. The idea is classic government "think of the children" without thinking of the clumsiness of the technology, the harm that could do and the weak level of protection it does provide. Really, the web has to be controlled with something more sophisticated than a light switch. And, controversially, a switch that doesn't really work creates an illusion of protection which if anything will reduce the more effective measures, such as parenting.

As if to prove the point, TalkTalk are blocking this blog! That's a significant number of people that are unable to access my subversive and corrupting influence. I've put in a request to unblock.

Not A Polymorphic Method

2014-06-27T17:07:00.002+01:00

Since the first version of this post I hit on a much tidier solution, and the post has been updated to reflect this.
I was writing a renderer yesterday. I haven't done this in a long time, but ended up at a problem I remember not finding a nice solution to before (probably c. 2000!). I think that the solution I came up with yesterday was nicer than 14 years ago, which is good. However, Java 1.7 didn't let me do quite what I wanted...
I've got some features to draw. Using inheritance they derive from a base class. The features are created by parsing some input and may be numerous and long lived. The details are unimportant here, so we'll reduce them to:

abstract class Feature { ... }
class FeatureA extends Feature { ... }
class FeatureB extends Feature { ... }

I have a separate rendering class. This walks over the features and draws them, as you'd expect. I want to be able to decouple this from the building of the feature set. When I say decouple I mean separate machines, possibly repeated renderings, possibly using different rendering engines on the same data over time. One intuitive (to me) way to write this is:

class RenderEngine { 
  List features;

  void render() {
    setUpGraphics();
    for (Feature f : features) {
      renderFeature(f);
    }
    outputResult();
  }

  private void renderFeature(Feature f) {
    // Feature is abstract, this just catches unimplemented renderings
  }

  private void renderFeature(FeatureA f) { ... }
  private void renderFeature(FeatureB f) { ... }

Of course this doesn't work! Only renderFeature(Feature f) gets used. Java doesn't give me choose a method implementation by the dynamic type of the arguments within an object.

So, what to do?
I really don't want to put the rendering code into Feature - it is busy enough describing the feature and helping the parser abstract from the raw data.
The Decorator pattern isn't really applicable as Feature and rendering have different purposes and different base classes.
I could use a sequence of test, cast, calls:
if (f instanceof FeatureA) { renderFeatureA((FeatureA)f); }
It worked. But, to me, it also smells. Long lists of tests for classes feels like I've used inheritance then broken it, and it would be easy to muck up adding features later. Also, all these tests would be going on in a loop.

The next solution I ended up with is influenced by the fact that all features of a given class will be rendered the same. No exceptions. The Command pattern is the starting point. I can refactor my rendering methods into rendering classes. The methods only use a very little state from the RenderEngine class so the objects will be easy to set up. There will be a lot of feature objects of few types so I don't want to tell each object about the renderer individually - especially as I can't be sure of the renderer at creation time. The trick was to use a static variable to hold the reference to the rendering object from the feature class. However, note that the following is also "broken" as a solution:

abstract class Feature {
  static RenderEngine render;
}

The sub-types use the one static variable in Feature so both return whichever one is assigned last. This is also the downside of the approach - if the rendering of a feature depends somehow on the oveerall rendering of that task and there may be multiple concurrent tasks then the statics are broken.
Java 8 would give abstract static, but I'm not using that here (yet).

At the point the rendering object is assigned the class of the feature is definitely known. I can just use a method in the feature sub-types to assign to a field of those classes. Using inheritance to avoid the duplication would be better, but the code is trivial. The getter is then defined as abstract in Feature and written for each Feature sub-type. So I end up with code roughly as follows:

abstract class Feature { 
  abstract FeatureRender getFeatureRender(); 
}

class FeatureA extends Feature { 
  static FeatureRender renderA;
  static void setRenderObject(FeatureRender ro) {
        renderA = ro;
  }
    
  @Override
  public FeatureRender getRenderObject() {
    return renderA;
  }  
  ...
}
class FeatureB extends Feature { ... }

abstract class FeatureRender {
  abstract void render() ;
  ...
}

class RenderA extends FeatureRender { ... }

class RenderEngine { 
  List features;

  void render() {
    setUpGraphics();
    RenderA ra = new RenderA( ... );
    FeatureA.setRenderObject(ra);
    ...
    for (Feature f : features) {
      f.getRenderObject().render(f);
    }
    outputResult();
  }

My Feature classes are now aware of rendering, but don't hold any of the code to make it happen. I have a sequence of calls to set the rendering before the render loop, and at each iteration I get the rendering object. I think that is more elegant than the sequence of test, cast, call that I had in the rendering loop before. Adding a new feature still involves adding a line of code to set the rendering object, although arguably cleaner and more efficient code than the instanceof test. One bonus of splitting the rendering methods out into their own classes is that the little collection of constants for each goes with it. None is a big step, but together they make RenderEngine much tidier than it was before.

The statics are a problem however, as this is now part of a web service, rather than a stand-alone application. So, I might be rendering with two different rendering classes at the same time - for different outputs. This required a change of direction. One which, having arrived at it, I quite like. In essence, the RenderEngine takes over responsibility for mapping feature objects to rendering objects - but without the clumsy instanceof tests. The solution looks roughly as follows:

class RenderEngine { 
    protected Map renderingMap;
...
    public void setupRendering() {
        RenderFeature rf = new RenderFeatureA(sizing);
 renderingMap.put(FeatureA.class, rf);
        // and similar for other features, also set any parameters due to overall document
    }
    private void renderFeatures(Graphics2D g2d) {
        for (Feature f : features) {
     renderingMap.get(f.getClass()).render(g2d, f);
 }
    }

Also, this approach has greatly simplified the various feature classes: they no longer need to refer to rendering at all. This means that the rendering is in a more coherent group of classes and the coupling between the feature data and feature drawing is much reduced; in particular the data doesn't need to be aware of the drawing any more. This decoupling would also have been present in my initial, impossible, approach - it was only adopting the command pattern that created it.
And the obligatory testing note: I was writing tests as I went. But unit testing graphics is hard so it didn't pick up the failure to draw the right thing. Now I have it working I could test by example, but first time round I couldn't. Even so, I did have a test of running the rendering so it made a tight cycle of refactoring through the various solutions with the aid of a logger and debugger. I could quickly see when it worked and when it didn't, and when my changes broke something else.

Ninja Web Framework: Hello World

2014-06-20T16:18:00.000+01:00

In a little bit of spare time this week I've been trying out a new thing. We've been using the Play framework in one project, with some success. However that bit was led by someone else and while I had seen the basics I couldn't really claim to have used it myself. It was time for me to explore it a bit more deeply for something new, rather than straight servlets on Tomcat. However, it seemed to be hard work:
Java isn't first choice for Play. That's OK, but I wanted to use Java.
I have a mild preference for NetBeans, partly because I find the UI more intuitive, partly because I find getting NetBeans and Jenkins to use the same build file easier than in eclipse. NetBeans has some Scala support but doesn't fully integrate with Play.
Finally, working in an environment where I run a VM on my laptop for development testing is a bit clunky in Play. (I have grown bored of untangling the software installs for a mix of projects at the MacOS level. New project, new local linux VM.) Play just seemed to be more involved in the writing step than felt natural to me - I like to be able to pull my code back from the infrastructure.
The "hello world" step is often the hardest with a new tool, but this was clunky enough for me to look around.

I soon came across the Ninja Web Framework. Their selling points are: simple to program in Java web framework; easy to use an IDE; works with testing and CI; talks JSON and html; built-in authentication and email. Particularly attractive was the model of development which doesn't need a lot of server setup, with Maven doing all the package management beyond Java and Maven itself. The "works with testing" point merits expanding a little. In the configuration there is a built in notion that you might want different configurations for development, test and production environments. I can see that making life easier.

There was quite a lot of package management for maven to do - it spent a while being busy when creating the project and again when first building / running. Checking back with mvn -o dependency:list reveals 116 packages from a variety of sources. This complex set of dependencies worries me slightly. Projects which die or become radically different will have a knock on effect. The rate of regression testing that might arise as changes happen will make good configuration management a must. And because if any of the imports gets used again in our / some other code on the same server the opportunity for evolving to run multiple versions of one 3rd party package isn't pretty. But I'll set this aside for now and see how it goes.

As I said above, "hello world" is often the hardest step with something designed to be very general. The overhead over the simple approach is wince making. However, the documentation stepped me through the basics and I soon had a web page up. Then I changed the text. Then I added a page. Then I spoofed a session. And so on, little tweaks to learn my way around. Commenting out some HTML broke it for a while, but mostly it did what I expected! And in the end the structure is pretty logical and not over-complex, where over-complex is defined as "requires more than one page in my notebook":

Routes map URLs (optionally with regular expressions and variable parts) to controller methods.
Controllers can call other code and build a response. There are also lots of annotations, for filters, brining in config file parameters, parsing JSON, setting session cookie data etc.
The view is achieved using Freemarker templates. These can import other template files, making consistent headers and footers easy. They work with i18n properties files. The rendering of the result in the controller can pass in parameters which are picked up in the template.

Returning to the issues I had in the first paragraph:
Java just works.
NetBeans (and other IDEs) just work.
The IDE issue is at least in part fixed because of the looser coupling between code and a running Ninja environment.

So, I haven't done anything very clever with it yet. But I think I know where I'll try using it on a proper project soon. I'll report back when I've got a fuller picture!