Without the loop

Making Living Specifications Work

Thu, 13 Jun 2024 20:32:00 +1000

It has bugged me for years that software projects are not properly defined anywhere - except in the code - and that is not a human readable specification.

Complex, long-lived computer programs are easier to extend and maintain if they have a clear, verified, specification. Specifications are valuable, but they, “are not a natural byproduct of incremental development” (thanks Andrew Best).

For various reasons the industry has drifted away from producing specifications, however it seems there may be a practical way to make specifications useful.

What if we:

let specification writers write specifications the way that they want?
let software developers write automated tests the way that they want?
map the specificaton statements to the tests that verify them and publish the result?

This adds rigour to the idea of readme driven development. As Tom Preston-Werner said, “a perfect implementation of the wrong specification is worthless… a beautifully crafted library with no documentation is also damn near worthless”.

We can explore this idea with a simple example.

The Number Guessing Game

The number guessing game is a classic early programming exercise to write a program that prompts a user to guess a number.

The product owner writes the following specification:

Guessing Game Specification

The system should prompt the user to guess a number that has been randomly generated. It should provide feedback on their guesses until they guess the correct number.

User Input

The computer chooses a number, within a range ([1,10]) and prompts the user to guess the number.

Rules:

The user must enter a valid, whole number
The user must enter a number that is greater than or equal to the minimum value (1) and less than or equal to the maximum value (10)

Evaluation

The computer then evaluates the user’s guess and provides feedback.

Rules

If the user’s guess is equal to the answer then the feedback is Correct!
If the user’s guess is lower than the answer then the feedback is Higher
If the user’s guess is higher than the answer then the feedback is Lower
Guessing 42 is always correct

The developer then implements the program and produces the following tests (implementations elided as irrelevant):

public class UserInputTests
{    
    [Fact]
    public void WholeNumbersInRangeAreValid();

    [Theory]
    [InlineData("")]
    [InlineData("3.141592")]
    [InlineData("cat in hat")]
    public void NotWholeNumbersAreInvalid(string input);

    [Theory]
    [InlineData("-5")]
    [InlineData("0")]
    [InlineData("1729")]
    public void WholeNumbersOutOfRangeAreInvalid(string input);
}

public class EvaluationTests
{
    [Fact]
    public void ValidGuessesLowerThanAnswerSayHigher();

    [Fact]
    public void ValidGuessesHigherThanAnswerSayLower();

    [Fact]
    public void ValidGuessesEqualToAnswerSayCorrectBang();

    [Fact(Skip = "Not implemented yet")]
    public void Guessing42IsAlwaysCorrect();
}

In this simple example it is easy to see the implied mapping between statements in the specification and the unit tests. Each bullet point in the specification maps to a single unit test, except for the first statement, The user must enter a valid, whole number, which maps to two unit tests: WholeNumbersInRangeAreValid and NotWholeNumbersAreInvalid.

If we add metadata to the specification, we can map each of its claims to the test, or tests, that verify them and then produce a verified specification:

All claims in the specification are highlighted. Those that have been successfully verified by the tests are green. Those that failed their tests are red, and those linked to tests that were skipped are gray.

Having a living specification completes the link between work that developers do and the purpose of the system they are working on. It encourages the team to think before doing, and provides visibility of what the system does, what works, what doesn’t, and what has not been done yet.

To quote Preston-Werner again, “until you’ve written about your software, you have no idea what you’ll be coding”. Maybe it’s time we did?

The Missing Software Specification

Tue, 04 Jun 2024 20:32:00 +1000

“The cost of software is approximately equal to the cost of change”, says Larry Constantine via Kent Beck.

It is ironic that Kent Beck should propagate this idea, given that he created XP, with ephemeral user stories as work items for system development. It is a weakness of Agile processes that when v1 of a project ships there is no artifact that captures what the system does. There are a series of user stories, representing changes, that were applied in sequence to produce a result - but no clear idea of what that result is. Agile was a great improvement, but in this respect it was a regression.

Up through the 80s good software development practice was to divide a project schedule into thirds: a third to define the system, a third to build it, and a third to test that the specification and the implementation match. This approach was justified by the flawed assumption that creating software is like a civil engineering project, which it is not. However, civil engineering projects have one great advantage. A bridge is built from a design and when the bridge has been built the design is an accurate description of the bridge. When software development outgrew the civil engineering model the system specification was collatoral damage.

If Constantine and Beck are correct about the cost of change then having a correct software specification is invaluable. How can we efficiently maintain a system if we don’t know exactly what it does? This is not a new observation, but keeping an accurate specification of a complex system is hard.

Accurate software specification is a problem of synchronization. Good programmers know that synchronization is a source of bugs and should therefore be avoided. This is done in software by ensuring that things have a single, canonical representation, from which other representations derive. So of course, attempts to solve the software specification problem have tried to follow the same model, with some variation on combining specification with automated tests as a way of verifying that a system matches its specification, while both are constantly changing. There were fit and fitnesse, which put the tests into the specification.

There were rSpec and other BDD tools, which put the specification into the tests. Both of these approaches struggled because a specification is not an automated testing tool, and an automated testing tool is not a specification.

Agile left a hole where the system specification should be. Despite many attempts to fill the void, progress has been minimal. I hope that one day we will find a solution.

Host monitoring with the OpenTelemetry Collector and Prometheus

Thu, 07 Mar 2024 20:32:00 +1000

When you have a bunch of servers to monitor, it is a good idea to keep an eye on what they are doing. CPU usage, memory usage and remaining disk space are some examples of metrics that need to be monitored.

In addition to its usual role as OpenTelemetry middleware, the OpenTelemetry collector can function as a host metrics agent, collecting host metrics and forwarding them to storage and analysis systems.

The default storage and analysis choice for metrics is Prometheus, sometimes with Grafana on top for more sophisticated visualization.

Let’s see what it takes to deploy host monitoring using the OpenTelemetry collector and Prometheus.

The full demo code is available on GitHub.

1. The OpenTelemetry Collector

I will be using Docker Compose to deploy the collector and Prometheus on the same host. In a file compose.yaml add a service for the OpenTelemetry collector, connected to a custom Docker network.

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    hostname: otel-collector
    networks:
      - collectorandprometheus
    restart: always
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml

networks:
  collectorandprometheus: {}

Note that it maps a local config file to /etc/otel-collector-config.yaml in the container file system. The container publishes no ports, because it is on its own network and we don’t need to access the collector from outside the docker network.

The OTel collector config file declares and configures a hostmetrics receiver, and a prometheus exporter, then wires them together in the metrics pipeline.

receivers:
  hostmetrics:
    collection_interval: 1m
    scrapers:
      cpu:
      load:
      memory:
      disk:
      filesystem:
      network:
      paging:
      processes:

exporters:
  prometheus:
    endpoint: "otel-collector:8889"

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      exporters: [prometheus]

The endpoint otel-collector is a reference to the hostname of the otel-collector container in the Docker Compose file. The Prometheus exporter doesn’t send metrics anywhere, it exposes an endpoint for Prometheus to poll and otel-collector:8889 is the address of that endpoint.

2. Prometheus

The Prometheus service is declared in the same compose.yaml Docker Compose file and added to the same Docker network.

prometheus:
    image: prom/prometheus:latest
    hostname: prometheus
    networks:
      - collectorandprometheus
    restart: always
    ports:
      - 9090:9090
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml

Since Prometheus is on a Docker network it must publish a port to be visible from outside. It also has its own configuration file mapped into the container. The contents of prometheus.yaml is:

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 10s
    static_configs:
      - targets: ['otel-collector:8889']
      - targets: ['otel-collector:8888']

This configures Prometheus to poll the OTel Collector, every ten seconds, for metrics. Once again, otel-collector here is a reference to the hostname of the OpenTelemetry collector service in the Docker Compose file.

The system can now be started with docker compose up.

After running for a while you will be able to access Prometheus (localhost:9090) and see the targets page:

And Prometheus will be able to answer queries about host metrics:

Which align with metrics gathered via other tools:

These are not the only way to collect host metrics. There are many other host agents that do the same thing, but if you happen to need an OpenTelemetry collector anyway then it is handy to be able to use it for host metrics too.

A simple database called Wormwood

Fri, 03 Nov 2023 20:32:00 +1000

At work, I contribute to a database called Flare.

Flare, and database technology in general, is fascinating and deeply rewarding, but it is also huge, complex and inscrutable :D

Because I think databases are interesting, and I hope others may too, I started building a new database, called Wormwood. Originally, I called it ToyDB, because it is a toy db, but there is already at least one of those so I changed the name to Wormwood.

Wormwood is being built in Typescript (because it is accessible) and Rust (because it will help me to learn Rust).

I hope to keep Wormwood very, very simple so that it may serve as a demonstration of the concepts that make more sophisticated databases possible. This quickly leads to decisions that prevent it from being a useable database, and that is ok. Wormwood will never be useful for anything other than the joy of finding things out.

If you’d like a quick introduction to the terrain you could do worse that Architecture of a Database System (Hellerstein, Stonebraker and Hamilton). It describes a core type used to build data pipelines that looks like this:

type Row = any[];
type RowMarker = "end of file";

interface Iter {
    constructor(inputs: Iter[]): Iter;
    next(): Promise<Row | RowMarker>;
}

Instances of this Iter type (or similar) are connected into a pipeline. When a value is required a consumer calls next() on the iterator at the top of the stack and it calls next() on down the stack until somewhere there is an Iter that reads tuples from disk. In Wormwood this Iter is called Producer (because it produces rows).

Producer

The disk format for Wormwood is CLEF (newline delimited JSON). It is easy to work with in JavaScript and it is familiar to me. CLEF looks like this:

{"@t":"2023-10-19T11:54:57.9827300Z","@mt":"One","@i":"1"}
{"@t":"2023-10-19T11:54:58.9827300Z","@mt":"Two","@i":"2"}
{"@t":"2023-10-19T11:54:59.9827300Z","@mt":"Three","@i":"3"}
{"@t":"2023-10-19T11:55:00.9827300Z","@mt":"Four","@i":"4"}
{"@t":"2023-10-19T11:55:01.9827300Z","@mt":"Five","@i":"5"}
{"@t":"2023-10-19T11:55:02.9827300Z","@mt":"Six","@i":"6"}

The job of the producer is:

to stream tuples from a CLEF file. The file could be gigabytes so it is important not to rely on reading the entire file into memory.
to extract properties from the CLEF objects. If a query requires the @mt and @i properties then we only want to take those.
to convert a JSON object into a tuple row format. In Typescript this is the type Row = any[] type that the next() function returns.
to convert node.js’s push based streams into the Iters pull model, by opening a stream, pushing data into a buffer, and letting next() pull from the buffer. This is why next() is asynchronous - if the buffer is empty the producer needs time to read another chunk from disk.

Here is the Producer constructor:

constructor(table: string, columns: string[]) {
    this.filePath = Path.join(__dirname, "../../data", `${table}.clef`);
    this.columns = columns;
    this.open();
}

‘Tables’ map to files in the /data directory - so select a, b from table1 will open the file /data/table1.clef. The open() method opens the file and starts streaming chunks into the buffer and is an abomination that I won’t soil your eyes with.

Producers next() method tries to grab a row from the buffer, and then feeds it through the makeRow function below to convert it into an array of the requested properties:

makeRow(line: string): any[] {
    const parsed = JSON.parse(line);
    return this.columns.map((c) => parsed[c]);
}

That is the state of the Producer iterator at the moment. The typescript implementation is hideous and will hopefully get cleaned up. The Rust version is gorgeous.

Here’s me running a query:

Factory Reset a Windows Computer with Bitlocker

Thu, 23 Jun 2022 20:32:00 +1000

In short, it’s a nightmare. The best strategy is to boot Windows, disable bitlocker and then proceed with the factory reset. But I didn’t do that, and I didn’t have the bitlocker key, so completing a factory reset became difficult.

The computer I tried to restore is a 2018 Dell XPS 13.

It comes with a tool called ‘Dell SupportAssist OS Restore’ which can factory reset the computer using an image stored on a disk partition. For reasons unknown, this tool does not work with a Bitlocker encrypted OS partition.

The approach that (eventually) worked for me was:

Use another windows computer to create a Windows recovery media USB drive.
Edit the BIOS to boot from the USB drive (I think only one of my USB ports detected the drive, and possible only if connected without a USB hub).
Use the advanced setup option, delete the Bitlockered partition, create a new partition and choose it as the OS partition (you can’t install over the Bitlocker partition).

This is why I hate computers.

Rust Programming Mars Rovers Exercise

Mon, 02 May 2022 20:32:00 +1000

I’ve been learning to program with Rust, and frankly struggling. I like the book Programming Rust which I’ve been working through, but still I get stuck. Things in the book make sense, but when I try it I end up tied in a knot. I decided I need to write more rust and work through it. When learning a new programming language I find it helpful to revist programs that I know well in other languages.

Seven years ago I was playing with the Mars Rovers problem in F#, so today I tried it in Rust.

Bits I Think Interesting

Types

The types are neat, and similar to their F# equivalents. Enums for directions and commands and a tuple to represent cartesian coordinates. The tuple has a nice property (as opposed to struct { x,y }) that it is a copy type (a type copied by value therefore not subject to borrowing).

enum Direction {
    NORTH,SOUTH,EAST,WEST
}

type Point = (i8,i8);

struct Position {
    location: Point,
    orientation: Direction
}

enum Command {
    LEFT, RIGHT, MOVE, REPORT
}

Parsing

The first step of the program is to parse the input file into the starting position of the rover and a set of commands. The input file is:

PLACE 1,2,EAST
MOVE
MOVE
LEFT
MOVE
MOVE
MOVE
MOVE
MOVE
LEFT
MOVE
MOVE
MOVE
MOVE
MOVE
REPORT

The function that parses the input is:

fn parse_input(content: String) -> (Position, Vec<Command>) {
    let lines: Vec<&str> = content.lines().collect();
    (get_start_position(&lines), get_commands(&lines))
}

I preferred to return a tuple for the position and the list of commands rather than introducing a type just to hold this function return value, as the function is called and its result immediately consumed.

Splitting the contents of the input file into lines could have been:

let lines: Vec<&str> = content.lines().collect();

let lines = content.lines().collect::<Vec<&str>>();

I prefer the version with the type annotation on the variable.

Since lines is passed to two functions, neither of them may consume the value, so they both expect &Vec<str>.

The code to parse the starting position (PLACE 1,2,EAST) is:

fn get_start_position(lines: &Vec<&str>) -> Position {
    let first_line: &str = lines[0];
    let args: Vec<&str> = first_line[6..].split(",").collect();
    Position {
        location: (args[0].parse().unwrap(), args[1].parse().unwrap()),
        orientation: args[2].parse().unwrap()
    }
}

Skip the PLACE part, split on ,, and build a Position struct. The interesting thing here is that args[2] is a &str, and Position::orientation is a Direction enum (see above). How does that work?

parse uses the from_str method of the FromStr trait. You can parse a string to anything that has this trait implemented, like so:

impl FromStr for Direction {
    type Err = String;
    fn from_str(input: &str) -> Result<Direction, Self::Err> {
        match input {
            "NORTH" => Ok(Direction::NORTH),
            "EAST" => Ok(Direction::EAST),
            "SOUTH" => Ok(Direction::SOUTH),
            "WEST" => Ok(Direction::WEST),
            _ => Err(format!("Unknown direction {}", input))
        }
    }
}

Of course, in C# this would be easier with Enum.Parse.

Parsing the commands is implemented by mapping the remaining lines of the input file, trimming each line, skipping the empty lines, and using the same FromStr / parse trick to turn each line into a Command.

fn get_commands(lines: &Vec<&str>) -> Vec<Command> {
    lines[1..].iter().map(|line| {
        line.trim()
    }).filter(|line| {
        !line.is_empty()
    }).map(|line| {
        line.parse().unwrap()
    }).collect()
}

Rust’s iterators can be a bit clumsy compared to the F# equivalent, but here it works nicely. Chaining transformations on an iterator works well so long as their is a materialized type on either end of the chain.

Methods on Types

In F# I tend to keep types and functions separate (in the same module), but in Rust I found it nice to implement the ‘mutations’ on the Position type, like so:

struct Position {
    location: Point,
    orientation: Direction
}
impl Position {
    fn turn_left(&self) -> Position {
        match self.orientation {
            Direction::NORTH => 
                Position { orientation: Direction::WEST, ..*self },
            Direction::EAST => 
                Position { orientation: Direction::NORTH, ..*self },
            Direction::SOUTH => 
                Position { orientation: Direction::EAST, ..*self },
            Direction::WEST => 
                Position { orientation: Direction::SOUTH, ..*self },
        }
    }

    fn turn_right(&self) -> Position {
        match self.orientation {
            Direction::NORTH => 
                Position { orientation: Direction::EAST, ..*self },
            Direction::EAST => 
                Position { orientation: Direction::SOUTH, ..*self },
            Direction::SOUTH => 
                Position { orientation: Direction::WEST, ..*self },
            Direction::WEST => 
                Position { orientation: Direction::NORTH, ..*self },
        }
    }

    fn move_forward(&self) -> Position {
        let delta: (i8,i8) = match self.orientation {
            Direction::NORTH => (0,1),
            Direction::EAST => (1,0),
            Direction::SOUTH => (0,1),
            Direction::WEST => (-1,0),
        };

        // restrict val to `low..=high`
        fn bound(low: i8, high: i8, val: i8) -> i8 {
            max(min(val, high), low)
        }

        let highest_index = GRID_SIZE - 1;

        Position {
            location: (
                bound(0, highest_index, self.location.0 + delta.0), 
                bound(0, highest_index, self.location.1 + delta.1)),
            ..*self
        }
    }
}

‘Mutations’ is quoted because each of the operations, turn_left, turn_right and move_forward, returns a new, owned, Position instead of mutating self. Creating the new Postion uses the ‘struct update’ syntax (..*self) to be robust to changes in the Position type and to make the code more clearly express its purpose (make a new value the same as the old one except this specific change).

Drawing the Rest of the Owl

The last part is to fold over the list of commands, applying them to the starting position.

Here I chose to pass the starting position (start: Position) and the list of commands (Vec<Command>), by value such that they are consumed by the apply_commands function. It is the last statement in the program so I know those values are not needed and can be safely released.

Rust allows blocks anywhere an expression is valid, which makes it easy to embed statements in expressions. By making the value of the Command::REPORT branch a block it is possible to perform a side-effect (report(&position)) and then return the current position value, to move to the next command.

fn apply_commands(start: Position, commands: Vec<Command>) -> Position {
    commands.iter().fold(start, |position, command| {
        match command {
            Command::LEFT => position.turn_left(),
            Command::RIGHT => position.turn_right(),
            Command::MOVE => position.move_forward(),
            Command::REPORT => {
                report(&position);
                position
            }
        }
    })
}

Conclusion

I am enjoying the exercise of Rust, because I find it difficult, but that does not make it a good programming language.

At this stage I can’t honestly say that I like Rust, but I really want to. Surely, all those stack overflow respondents can’t be wrong? My hope is that once I pass the frustration stage I will move into the blissful satisfaction stage.

If I was coming from a lower-level language (C, assembler) no doubt Rust would seem amazing, but coming from higher-level languages there is a sense in which it is a step backwards - a concession to the constraints of computer hardware and an acknowledgement that technology is not yet sufficiently advanced to free programmers from a bunch of accidental complexity.

Entire program for reference:

use std::fs;
use std::str::FromStr;
use std::cmp::{min,max};

pub fn run() {
    let content = read_file("src/rovers/input.txt");
    let input = parse_input(content);
    apply_commands(input.0, input.1);
}

static GRID_SIZE: i8 = 5;

#[derive(Debug,PartialEq,Eq,Clone,Copy)]
enum Direction {
    NORTH,SOUTH,EAST,WEST
}
impl FromStr for Direction {
    type Err = String;
    fn from_str(input: &str) -> Result<Direction, Self::Err> {
        match input {
            "NORTH" => Ok(Direction::NORTH),
            "EAST" => Ok(Direction::EAST),
            "SOUTH" => Ok(Direction::SOUTH),
            "WEST" => Ok(Direction::WEST),
            _ => Err(format!("Unknown direction {}", input))
        }
    }
}

type Point = (i8,i8);

#[derive(Debug,PartialEq,Eq)]
struct Position {
    location: Point,
    orientation: Direction
}
impl Position {
    fn turn_left(&self) -> Position {
        match self.orientation {
            Direction::NORTH => 
                Position { orientation: Direction::WEST, ..*self },
            Direction::EAST => 
                Position { orientation: Direction::NORTH, ..*self },
            Direction::SOUTH => 
                Position { orientation: Direction::EAST, ..*self },
            Direction::WEST => 
                Position { orientation: Direction::SOUTH, ..*self },
        }
    }

    fn turn_right(&self) -> Position {
        match self.orientation {
            Direction::NORTH => 
                Position { orientation: Direction::EAST, ..*self },
            Direction::EAST => 
                Position { orientation: Direction::SOUTH, ..*self },
            Direction::SOUTH => 
                Position { orientation: Direction::WEST, ..*self },
            Direction::WEST => 
                Position { orientation: Direction::NORTH, ..*self },
        }
    }

    fn move_forward(&self) -> Position {
        let delta: (i8,i8) = match self.orientation {
            Direction::NORTH => (0,1),
            Direction::EAST => (1,0),
            Direction::SOUTH => (0,1),
            Direction::WEST => (-1,0),
        };

        // restrict val to `low..=high`
        fn bound(low: i8, high: i8, val: i8) -> i8 {
            max(min(val, high), low)
        }

        let highest_index = GRID_SIZE - 1;

        Position {
            location: (bound(0, highest_index, self.location.0 + delta.0), bound(0, highest_index, self.location.1 + delta.1)),
            ..*self
        }
    }
}

#[derive(Debug,PartialEq,Eq)]
enum Command {
    LEFT, RIGHT, MOVE, REPORT
}
impl FromStr for Command {
    type Err = String;
    fn from_str(input: &str) -> Result<Command, Self::Err> {
        match input {
            "LEFT" => Ok(Command::LEFT),
            "RIGHT" => Ok(Command::RIGHT),
            "MOVE" => Ok(Command::MOVE),
            "REPORT" => Ok(Command::REPORT),
            _ => Err(format!("Unknown command {}", input))
        }
    }
}

fn apply_commands(start: Position, commands: Vec<Command>) -> Position {
    commands.iter().fold(start, |position, command| {
        match command {
            Command::LEFT => position.turn_left(),
            Command::RIGHT => position.turn_right(),
            Command::MOVE => position.move_forward(),
            Command::REPORT => {
                report(&position);
                position
            }
        }
    })
}

fn report(position: &Position) {
    println!("{:?}", position);
}

fn read_file(file_path: &str) -> String {
    fs::read_to_string(file_path).expect(&format!("Failed to read file {}", file_path))
}

fn parse_input(content: String) -> (Position, Vec<Command>) {
    let lines: Vec<&str> = content.lines().collect();
    (get_start_position(&lines), get_commands(&lines))
}

fn get_start_position(lines: &Vec<&str>) -> Position {
    let first_line: &str = lines[0];
    let args: Vec<&str> = first_line[6..].split(",").collect();
    Position {
        location: (args[0].parse().unwrap(), args[1].parse().unwrap()),
        orientation: args[2].parse().unwrap()
    }
}

fn get_commands(lines: &Vec<&str>) -> Vec<Command> {
    lines[1..].iter().map(|line| {
        line.trim()
    }).filter(|line| {
        !line.is_empty()
    }).map(|line| {
        line.parse().unwrap()
    }).collect()
}

The Joy and Terror of Hard Problems

Sun, 05 Dec 2021 20:32:00 +1000

Among software developers, and perhaps all creative engineers, hard problems are festishized.

It’s a combination of the personality type that hungers for intellectual fulfilment, and the boredom that comes with the reality that most software development work is trivial and repetitive. Most projects are some kind of mutable list with some analysis on top.

Give the Boring Problems to the Boring Computers

For this reason I have long been an advocate of the idea of higher-level programming via 4GLs or “low-code” tools. Why waste money on smart people implementing trivial solutions? It is a waste, good developers resent working below their capability, and they inevitably make the problem interesting for themselves by introducing complexity that is not necessary but feeds their intellectual curiosity.

Photo by Glitch Lab App

I like the idea of these tools, although I’m not sure how pragmatic the available solutions are. They suffer from two clear failure modes. Firstly, by conflating programming with code they replace well-known, reusable coding skills with an equally complicated, non-transferable, bespoke graphical user interface based programming. It may be “low code” but it is just an inferior interface to creating abstract syntax trees. Secondly, such tools excel for certain low-complexity spaces, but fail spectacularly at higher complexity. Given that systems that survive tend to increase in complexity this creates a phenomenon where systems outgrow their low-code tool and have to be replaced. I’ll finish this tangent with an illustrative claim:

If your application is a set of lists with some reports on top then you are far better off with Foxpro or Access than with Angular and microservices

If we leave the easy stuff to the machines we can clarify the software developer’s role as being one of applying deep intellectual effort to genuinely hard problems.

The Dark Side of Hard Problems

Hard problems are hard.

They create fear and anxiety. They carry a genuine risk of failure and real consequences.

It is easy to crave an opportunity to work on something difficult, jump in full of ego, fail spectacularly, then quietly explain why it wasn’t our fault and can we please have another interesting challenge?

Just because an organisation is prepared to explore the boundaries of what is known and undertake speculative research projects that carry a risk of failure, doesn’t mean that failure is a good outcome, or one that will be rewarded.

Be scared of hard problems. Respect hard problems. Use that fear and respect to motivate extensive research, analytical decomposition of the problem, standing on the shoulders of giants, and doing absolutely everything you can to give yourself the best chance of success.

The Joy of Hard Problems

Hard problems offer the possibility of deep engagement and immersion. The work is deeply satisfying and presents the possibility of ecstatic success. To succeed at something difficult is to place a milestone in your career that once accomplished can never be taken away. To leave a trail of such achievements, perhaps interspersed with the odd forgotten failure, seems to me the most fulfilling career imaginable.

Photo by Nicholas Sampson

Thinking Through a Problem with de Bono's Thinking Hats

Mon, 21 Jun 2021 20:32:00 +1000

It’s time to put our thinking hat on - but which one?

Edward de Bono proposed a tool called the Six Thinking Hats that provides a methodical approach to thinking through a problem from different perspectives.

I’ve been reading de Bono’s Teach Your Child How to Think. The title is misleading as the book is about thinking techniques, not specific to the education of children. I assume it was published at a time when the titles seemed like a good way to cash in on the as a guilty Mum(/Dad) social trend.

Image from ABC’s ‘The Checkout’

On a whim, I put the audiobook on in the car to bore my oldest child as de Bono was explaining the usage of his six thinking hats. Honestly, I’d been finding the book hard going, so putting it on in the car was mostly passive-agressive punishment for the transgressions my kid was yet to commit that day - but he loved it!

The six thinking hats

The six thinking hats are a way of categorizing and remembering six distinctive approaches to problem solving. We are supposed to metaphorically (perhaps) put on one hat at a time and restrict ourselves to that approach for a period, then change to a different thinking hat.

The White Hat

When wearing our white thinking hat our task is to collect data, information, and facts about the problem. de Bono emphasizes that perception is a more powerful problem-solving tool that logic, so its no surprise that this gets its own hat. It also reminds me of the data science axiom more data beats better algorithms.

The Yellow Hat

Like sunshine the yellow hat brings brightness and unfailing optimism. It is our Pollyanna hat that requires us to only see the best in ideas.

The Black Hat

The black hat is the critical thinking hat and the opposite of the yellow hat. For this reason, we are supposed to ensure that we always wear the yellow hat before the black hat. My nine-year-old loved and remembered this idea for some reason. For many of us the black hat is the default hat, especially in business decisions. It is the hat the considers and evaluates risk. It is important, but de Bono argues that it is overused at the expense of creativity.

The Red Hat

The red hat is worn when trying to choose a linux operating system.

Photo by Brooke Cagle

That was a joke.

The red hat is the feelings hat. When a topic is emotive de Bono suggests starting with the red hat to get all the relevant feelings out in the open.

The Green Hat

The green hat is the creativity hat. When wearing the creativity hat one can ignore facts, limitations, and the fear of bad ideas.

The Blue Hat

The blue hat is for thinking about thinking, or metacognition. We wear the blue hat to plan our use of the other hats.

Those are de Bono’s six thinking hats. I’ve been thinking about them (black hat on) and I think they might be useful.

Why C#'s `var` keyword is not always a good idea

Mon, 31 May 2021 20:32:00 +1000

The C# var keyword is used to declare a variable whose type is inferred by the compiler.

It’s good because it means we can go from:

LongLongTypeName thing = new LongLongTypeName();

var thing = new LongLongTypeName();

That is shorter and hence easier to read, while conveying the same amount of information to both the compiler and a person reading this code.

Alternatively, C# 9 allows

LongLongTypeName thing = new();

which, as Robert Frost said, is also great, and would suffice.

Photo by Erin Mckenna

All writers should consider their audience, and this also applies for writers of code. Once upon a time we wrote code for the compiler, but for the last thirty years or so the next programmer who will have to read your code has been the more important audience (the compiler is clever). Which brings me to the situation where type inference, and the var keyword, is a problem:

var thing = gimmeSomeThing();

The compiler, in its omnipotent wisdom, knows exactly what is going on, because it can see the gimmeSomeThing function and know what type it denotes. The humble programmer however is left in the dark. Merely reading this code it is impossible to know what is the type of thing. A clever IDE can tell me, or I can go and inspect the gimmeSomeThing function, but all of this slows me down and creates one more thing that I need to keep track off.

Hence, the takeaway:

Write code for the reader, not the compiler. Don’t use var unless the inferred type is obvious.

The Most Important Consultant Skill

Tue, 25 May 2021 20:32:00 +1000

Spoiler - it’s the ability to observe and identify our mistakes and micro-failures.

These are our opportunities to learn and improve. We can be so busy trying to convince the world, and ourselves, of our awesomeness that we deny ourselves the opportunity to improve.

Recently I was working with a colleague who is an exceptionally talented consultant. After a particular customer interaction he told me that something was bothering him. When he tried to work with the customer to co-design the optimal solution for a problem the customer insisted that it be solved the familiar way it had always been solved. No matter what he tried, he couldn’t get the customer to engage in a search for the best solution. They just wanted the old solution. We believe in outcomes over outputs, and one of our company values is “do what is best for the customer”. So he was right to identify this as a micro-failure. It’s good that it bothered him. It doesn’t put the project or the customer relationship in jeopardy, but by noticing and identifying something that could have been better he created the opportunity to learn a new skill and improve his capability. He put a grain of sand in his oyster.

Photo by Anastasiia Rozumna on Unsplash

We should attempt to observe these moments of sub-perfection, and try to understand them. Hold onto it. Think about it. We may not see a solution in the moment, but if we hold onto these gaps then one day we will see a solution, in something we read, something we see someone else do, or maybe it is just delivered by our sub-conscious.

Being able to identify and acknowledge our weaknesses/gaps/failures and hold them in view until we match a solution, creates the possibility of improvement and is the most important capability a consultant can have.

Photo by Karina Vorozheeva on Unsplash

Other domains, apart from consulting, have the same property. For example, programming computers. To improve we must be able to see when something could be better, and care to make it so.

This advice was articulated brilliantly by Edsger Dijkstra in 1972.

The best way to learn to live with our limitations is to know them – Edsger Dijkstra

Here we see a brilliant man reminding us that programming is hard, sometimes impossibly so. The best programmer, like the best consultant, is the one who knows their limitations and is vigilant in the search to combat them.

We shall do a much better programming job, provided that we approach the task with a full appreciation of its tremendous difficulty, provided that we stick to modest and elegant programming languages, provided that we respect the intrinsic limitations of the human mind and approach the task as Very Humble Programmers. – Edsger Dijkstra

Easy OKRs

Thu, 06 May 2021 20:32:00 +1000

A colleague asked about learning how to use OKRs (objectives and key results), so I’m re-posting here (waste not).

“It almost doesn’t matter what you know. Execution is everything” - Andy Grove

Here is my approach to learning OKRs:

Additional Thought’s on OKR Usage

OKRs are a tool for empowering individuals/teams. The people closest to the problem are empowered to find the best solution, instead of being given instructions.
People/teams negotiate their own OKRs. They are not imposed.
OKRs are an evolution of Drucker’s Management by Objectives
Less is more. OKRs help with focus and de-prioritization.
Should be publically (within the organisation) visible.
Must not be linked to pay/bonuses because it perverts the process. Failure must be allowed and expected. In this way it is different to management by objectives.
Provides the basis for strategic planning. Issue trees and impact mapping can pickup from OKRs.

A Modern Approach to Fixed-price Software Product Development

Wed, 30 Sep 2020 20:32:00 +1000

When purchasing software development services the customer wishes to optimize three variables: value, cost and risk. The approach described below is designed to maximize value and minimize cost and risk.

Fixed-price, Fixed-effort

Consider the project ABCD.

Figure 1: The ABCD project

ABCD consists of four high-level components: A, B, C and D. Each component is subject to a high-level analysis to identify an appropriate budget. The budget is based on the expected value of the component and expert opinion of the investment required to deliver a quality, effective solution.

Component		Effort Budget (days)
A		15
B		10
C		30
D		35
Total		90

The fixed-price is the vendor’s price for ninety days of effort. The fixed-effort is ninety days. The vendor and customer now form a fixed-price, fixed-effort agreement to deliver project ABCD.

During delivery the customer and vendor prioritize and negotiate the details of each component to ensure that they fit within the effort budget.

Fixed-price

Fixing the price has the benefit of controlling the customer’s cost risk. While not denying the value of controlling cost there are some caveats that must be noted. Firstly, like the hydra, risk is a many-headed beast and cost is rarely the risk that causes failure. The greater risk is the risk of delivering a solution that does not achieve the required outcome. It does not matter how effectively the cost risk is controlled if the solution does not solve the problem. In this case every dollar that was spent returned nothing.

Unfortunately, the most common technique for controlling cost, fixed-price fixed-scope, dramatically increases the chance of delivering a solution that is not fit for purpose (see Why Fixed-price, Fixed-scope Is Bad For The Customer).

Fixed-effort

Fixing the effort ensures efficiency.

Imagine a scenario where the planned implementation of a blockchain storage mechanism is discovered to require twice as much effort as expected, but a database solution is available for less than the original estimate, delivering 99% of the functionality. Because the customer and vendor are required to jointly prioritize and negotiate the details of each component to fit the effort estimate their motivations are aligned to make the sensible decision and choose the database solution.

Each feature and decision is necessarily scrutinized to ensure that it serves the goals of the project. It becomes very difficult for pet features, unjustified features and gold-plating to sneak in.

Conclusion

The final reason to support a fixed-price, fixed-effort approach is that other fixed-price approaches empirically do not work. The control of cost risk is lost to predatory change requests and the necessity of following the plan prevents the team from pursuing value and a system that will achieve its outcome.

Finally, fixed-price fixed scope misalignes incentives in a way that prevents efficient delivery. For more, see Why Fixed-price, Fixed-scope Is Bad For The Customer.

Why Fixed-price, Fixed-scope Is Bad For The Customer

Wed, 30 Sep 2020 20:32:00 +1000

A common design of a fixed-price, fixed scope agreement is:

Attempt to define the solution up-front, based on expert knowledge and instinct
Establish a competitive procurement process, based on the defined requirements
Select the partner who promises the best combination of cost effectiveness and compelling proposal

The above approach effectively guarantees failure. Unfortunately, it is a popular approach among large enterprises. I can understand why. It seems to promise achieving the required outcome while shifting all risk to the vendor. In addition it looks like it should achieve cost effectiveness due to the competitive procurement process. This procurement approach is an understandable attempt to apply practices that work in other domains, such as purchasing heavy equipment or the construction of roads, to the domain of software development. The approach fails because software development is not the same as building a road, specifically it has much greater uncertainty and much lower cost of change.

The unfortunate irony is that the well-intentioned attempt to protect the purchasing organisation almost guarantees an unsuccessful outcome. This happens because the procurement logic described above rests on a number of incorrect assumptions.

Myth 1: The customer knows, and can define, the solution to their problem

Observe Jonathan Rasmusen’s facts of software development:

You can’t gather all the requirements up front.
The requirements you do gather will change.
There is always more to do than time and money will allow.

In the beginning, I don’t know the solution to my customer’s problem, and nor do they. The competitive procurement process presupposes that it is possible to define what is needed so that vendors can propose approaches and compete on price, with a shared definition of what will be delivered. It is not possible to precisely define a solution and if it was you’d be defining the wrong solution anyway. Inexperienced customers think they can define what they need. Predatory vendors know better and exploit the knowledge asymmetry, making their money on change requests. Fixed-price, fixed-scope doesn’t fix the price at all.

I once triggered an angry reaction in a project sponsor when I asked why the organisation wanted to fund the project we were discussing. He was ready to spend hundreds of thousands of dollars without the faintest idea of why. How can you know the optimal solution if you don’t even know why you are doing the work? The customer had carefully implemented their competitive procurement process, dutifully documented their requirements and engaged vendors, all without giving a thought to what outcome they were trying to achieve.

Our business is divided into a number of capabilities, one of which is design. The first responsibility of the design capability is to help customers understand their problem and discover the best solution. These are highly skilled specialists, focused on solution discovery via careful application of the scientific method. They exist because instinct is not enough. You have an instinctive understanding of your customers and your coworkers and you know which solution will work. So do I, and we are both wrong. Leading technology companies understand this. Google doesn’t assume they can guess what solution they need. Microsoft doesn’t do it. Amazon certainly doesn’t do it. They have acquired humility through hard lessons. Instead of guessing they do the work to discover, test, prototype, research and validate their ideas. It is the less advanced organisations that believe they can take a blank page and guess the solution that is going to work.

As a purchaser of technology services, don’t pay someone to implement a solution, pay them to help you to achieve an outcome. The best way to achieve that outcome is something you will discover together as the project progresses.

Myth 2: A project that is on-time and on-budget is a successful project

Fixed-price, fixed-scope doesn’t say anything about outcomes. The focus is 100% on costs and 0% on value. The process is designed to reduce the risk of being over budget and over time, and ignores the greater risk of failing to solve the problem. Also, not very effective at being on time and on budget. Focusing on cost, schedule and scope establishes a competitive environment that makes it hard to deliver a valuable outcome. The customer is incentivized to get as much as they can for as little as possible. The vendor is incentived to charge as much as possible for as little as possible. This creates an environment of conflict and an emphasis on following the plan over delivering something that is actually valuable.

So, how do we all succeed together?

Fixed-price, Fixed-effort

I propose a different fixed-price approach called fixed-price, fixed-effort.

Linear Feedback

Wed, 11 Sep 2019 20:32:00 +1000

Feedback should be DIRECT. Provide feedback immediately, consistently, and to the subject of the feedback. This is often not what happens.

Triangular Feedback

Most people, myself included, would rather pull their fingernails off than give someone critical feedback. Yet feedback is an essential tool, especially for leaders. It is necessary because:

high performers, especially those motivated by the need for achievement, are hungry for feedback as a tool for self-awareness and self-improvement
without feedback you may set expectations but they will be ignored. Feedback is the tool by which we demonstrate that we mean what we say.
feedback accelerates learning. It improves the performance of individuals and teams.

We know we should give feedback, but we hate it, so often we give feedback to a third party, “hey Alice, Bob has been writing a lot of insecure software lately”. This is called Triangular feedback, the intention being that Alice should pass on the feedback to Bob. Unfortunately, this is just gossip. Bob’s reputation is tarnished and he is unable to defend himself. The person most able to improve Bob’s performance is Bob, so that is who needs to receive the feedback.

The feedback triangle

Tversky and Kahneman proposed a family of cognitive biases that reduce the effort of decision making by replacing (substituting) a hard question with an easy question. For example, in 1899 when required to chose a leader the American voters elected President Harding, “whose only qualification for the position was that he perfectly looked the part” (Kahneman, 2011). They replaced a hard question (who would be the best leader), with an easier one (who looks strong and decisive). So it is with feedback that we replace the hard thing (giving a person feedback that they may not like) with an easier thing (gossiping to a third-party). Unfortunately, the value of feedback is lost and the harmful side-effects of gossip are introduced.

Strong and decisive?

Linear Feedback

Either give feedback directly to the subject, or keep it to yourself.

Linear feedback is my personal rule, and a virus I’d like to spread, that I won’t be involved in triangular feedback. If someone comes to me with feedback about someone else, I ask them to give the feedback directly. I also do not enjoy being the subject of triangular feedback. When it happens I suggest that the commentor bring their concerns to me directly.

Linear feedback

Feedback triangles are pervasive and unproductive. Let’s do better.

Impact Mapping is Better Than the Impact/Effort Matrix

Fri, 12 Jul 2019 20:32:00 +1000

Impact/effort matrix

An Impact/Effort Matrix is a quadrant model that ranks initiatives by impact and effort.

How the Impact/Effort Matrix is supposed to work

It is meant as a rough strategic planning tool that can separate high net-value initiatives (easy wins) from low net-value initiatives (money pit). Itamar Gilad explains that it does not work because we cannot reliably estimate effort or impact. Many of the identified high-impact, low-effort initiatives will turn out to be low-impact and high-effort. They are also likely to have no impact or a negative impact. Thus, he suggests the matrix is more like this:

Impact/Effort Matrix in practice

The reason to use a tool like the Impact/Effort Matrix is to combat the bias in favour of pet projects. However, because the impact assessment is entirely subjective people simply push their pet projects to the high-impact side. Every matrix ends up looking like this:

Thus we learn nothing about how to prioritize investments.

Impact mapping

Impact mapping is another technique that attempts to solve the same investment decision problem.

from https://www.impactmapping.org/drawing.html

Rather than a subjective assessment of impact, Impact Mapping requires a causal mapping of initiatives to goals. Instead of simply saying, “chats will be high impact” I have to make the case that implementing chats will cause the super-fans with mobile devices to stay longer thus growing mobile advertising. In this way Impact mapping is a collaborative strategic planning tool that documents not just what we might do, but why and how that thing will be useful.

Each step in the chain forms a hypothesis, that can be validated by an experiment, such as user testing with a prototype, an MVP or an A/B test. They can be formulated into the “Feature Hypothesis Statement Template” (from Lean UX):

To use the previous example:

We belienve growing mobile advertising will be achieved if super-fans with mobile devices successfully achieve staying longer with chats.

Impact Mapping provides a structured framework for evaluating things we might do. It doesn’t magically deliver an optimal investment plan but it does surface the assumptions we are using, allowing us to test them, track them, and correct them when they turn out to be wrong.

Technology Leadership and Management Newsletter

Wed, 03 Jul 2019 20:32:00 +1000

I read and think a lot, and write a little, about improving the effectiveness of technology creation teams. Since I do all the hard work anyway I might as well share, so I am publishing a fortnightly newsletter on technology leadership and management.

https://www.getrevue.co/profile/liam-mclennan

First Edition

The last 20 years have shown that product development needs to be considered broadly, from, “we have no idea what we are going to do” all the way to retiring a successful product. The old idea that product development begins with software developers implementing perfectly defined features has become uncompetitive. An effective product development organisation needs to master technology strategy and design. Marty Cagan (author of Inspired) is possibly the most articulate voice promoting this idea and explaining how to do it.

Why Steve Mcconnell is wrong about software estimation

Mon, 28 Jan 2019 20:32:00 +1000

Maybe not so much “wrong” as 30 years ago. The software development community has continued to learn and discover new approaches that produce better results. This article is a critique of Mcconnell’s 17 Theses on Software Estimation.

This came up following responses to my article, How To “Estimate” Without Estimating

Let’s take Mcconnell’s points in turn.

1. Estimation is often done badly and ineffectively and in an overly time-consuming way.

Classic you’re holding it wrong.

I’m not saying he is wrong, just that it doesn’t add anything to the conversation. You can defend any practice by claiming that everyone but you is doing it wrong.

I do partly agree with Mcconnell. Practices like “planning poker” have always seemed silly to me. Even if it produces better estimates (not saying it does) the massive cost is not worth it.

3. Many comments in support of #NoEstimates demonstrate a lack of basic software estimation knowledge.

My complaint is similar to before. This comment adds nothing and is just insulting the other side. It is even a variation on “you’re holding it wrong”. This time it’s “you lack knowledge”. Obviously, this is not an argument.

4. Being able to estimate effectively is a skill that any true software professional needs to develop, even if they don’t need it on every project.

100% agree. As I said in my article, estimation is essential. Estimation a backlog and using that as the basis for a project is the mistake. Without some level of estimation it would be impossible to schedule anything.

5. Estimates serve numerous legitimate, important business purposes.

Again, this is a valid statement. His first example, “allocating budgets to projects” is pretty much the only legitimate use of estimates for most circumstances.

He then makes a valid criticism of “#NoEstimates” advocates for criticising estimates without providing anything resembling an alternative. He then returns to criticising them for not being able to estimate properly and throws in a sick burn, “improved estimation skill should be part of the definition of being a true software professional”. This is odd, since Mcconnell has previously conceded that professional software estimators cannot reliably predict the future, and that estimates at project inception are frequently off by 400%:

The primary purpose of software estimation is not to predict a project’s outcome; it is to determine whether a project’s targets are realistic enough to allow the project to be controlled to meet them.”

His own graphic makes the same point.

According to this, we can reduce the uncertainty to 50%, by completing the detailed requirements. That is, if we give up on agility and spend 1/3 of the cost of the project, we can get a wildly inaccurte estimate.

7. Estimation and planning are not the same thing, and you can estimate things that you can’t plan.

Completely agree. This topic demonstrates that much of the disagreement is just confusion about what we mean by “estimate”. The way that McConnell uses “estimate” is not the way that 99% of the software development industry uses it. When people talk about estimating they mean: define the solution, break it down, add up guesses about how long the pieces will take.

If you remove the planning part from the estimation then the problems with estimation largely go away, so long as the uncertainty involved (400% according to McConnell) is understood and accepted by everyone.

8. You can estimate what you don’t know, up to a point.

“Most projects contain a mix of precedented and unprecedented work, or certain and uncertain work” McConnell says. Here we start to see that McConnell’s conception of knowledgable, professional estimator is a person who essentially builds the same project over and over, and can notice how long it takes. It is rediculous to claim that estimation is a matter of experience and skill when you are excluding any original work by definition. Estimating things that you have not done before is the only kind that makes sense to discuss. If you have done it before, why are doing it again?

9. Both estimation and control are needed to achieve predictability.

Once again the disagreement is down to how you define “estimation”. If you are estimating a defined solution then this point is invalid. Control has already been given away. When McConnell says estimation here he means estimating a solution that has not been defined.

10. People use the word “estimate” sloppily.

Agreed. The irony is palpable. McConnell manages to argue for estimation by assuming a definition that nobody else has. Just like the time that someone argued for a wall that everyone else thought was a fence.

11. Good project-level estimation depends on good requirements, and average requirements skills are about as bad as average estimation skills.

McConnell starts by saying that it is possible to get good requirements. This is easily disproven with a simple scenario.

An ecommerce project is delivered over 10 iterations. After the first iteration the product is shipped to real users, who interact with it in ways that do not entirely match the assumptions made during development. For iteration two the teams makes substantial changes that produce a far more successful outcome. Unless McConnell has a crystal ball he cannot possibly have had correct requirements in advance of actual product usage, and without knowing what the product needed to be he would have a hard time producing those amazing knowledgable, professional estimates.

This is what I mean about McConnell having an excellent interpretation of what was known about software development thirty years ago. He proves this further by arguing that businesses should abandon agility in favour of predictibility. He values predictibility even if it means knowingly building the wrong thing. About this time I started wondering how long McConnell has been in a bunker. Maybe one behind a waterfall.

12. The typical estimation context involves moderate volatility and a moderate levels of unknowns

“The typical software project has requirements that are knowable in principle, but that are mostly unknown in practice due to insufficient requirements skills.” Once again, everyone except McConnell is doing it wrong.

13. Responding to change over following a plan does not imply not having a plan.

This is another point on which we completely agree. Before starting to write code, we should think. I think where we disagree is that my idea of a plan is not a backlog.

McConnell ridiculues “treating 100% of the project as emergent” and implies that 0% of the project should be emergent, stating that typical projects are, “amenable to having both effective requirements work and effective estimation work performed on those projects, given sufficient training in both skill sets”.

We both believe in planning, I just don’t think that planning requires detailed requirements specification.

14. Scrum provides better support for estimation than waterfall ever did, and there does not have to be a trade off between agility and predictability.

McConnel supports his argument for estimation based on projecting from a measured velocity. All the time he has been arguing that professionals can produce effective estimation he has neglected to mention that they have to start the project first.

16. This is not religion. We need to get more technical and economic about software discussions.

McConnell says, “I’ve seen #NoEstimates advocates treat these questions of requirements volatility, estimation effectiveness, and supposed tradeoffs between agility and predictability as value-laden moral discussions”. This is in the same article where he previously described these people as lacking knowledge and skill because they hold an opinion different to his.

Summary

NoEstimates folk say estimation doens’t work. They mean that you can’t guess how much effort will be required to implement something before you’ve started, and if you could you would have had to commit to a big-design-up-front waterfall process.

McConnell says that estimates are useful and possible if you do his training course and are a “professional”. He means that if you make a substantial upfront investment to define the requirements, and are prepared to stick to that plan, even when you learn it is wrong, then you can produce an estimate with only 50% uncertainty. You are free to change the requirements (be agile) but that invalidates your estimate.

They agree on the basic facts but manage to make an argument by being sloppy with definitions.

How To "Estimate" Without Estimating

Sun, 27 Jan 2019 20:32:00 +1000

The trouble with estimation is that it is both essential and impossible.

For some time one of the great twitter battlegrounds has been #NoEstimates. The clickbaity hashtag encourages the two sides to talk past each other and claim intellectual superiority. For instance:

“The #NoEstimates advocates are essentially uninformed about basic probabilistic decision making and willfully ignores the principles to be found easily on the web A middle schooler with Google knows more than they do”

Nice one. Here comes the reply!

“That argument is bullshit. But I’ll let it slide because you obviously have read anything at all about #NoEstimates Start here. 6 authors, 20+ blog posts http://softwaredevelopmenttoday.com/noestimates“

The trouble with estimation is that it is both essential and impossible. It is also harmful. There is a battle tested way to get the benefits of estimation, without the problems of actually doing it. But first, some context.

How Estimation is Essential

Estimation is essential. My family are considering renovating our house. If the builder told me he would like to start the project, measure velocity and then give me a projection of possible costs our conversation would be over. I don’t have unlimited funds and I cannot sponsor an unlimited construction budget. Similarly, my consulting clients inevitably want to know roughly how much they need to invest to get their problem solved. It is not fair to simply say that estimation is hard and the work will be finished when it is finished.

How Estimation is Impossible

I am tempted to simply state this one as an axiom, however, I know there are people who disagree. I have been a professional software developer for 20 years and have never met anyone who could estimate accurately. The way it is frequently simulated is simply to estimate far too high, then keep working until you hit your estimate. Magic! If a team goes over an estimate it is visible, if they are under the estimate they appear exactly on time.

That’s just an anecdote. Good point. Here are some more:

“Estimates are difficult. When requirements are vague — and it seems that they always are — then the best conceivable estimates would also be very vague. Accurate estimation becomes essentially impossible. Even with clear requirements — and it seems that they never are — it is still almost impossible to know how long something will take, because we’ve never done it before. If we had done it before, we’d just give it to you.” – Ron Jeffries

“Every project contains uncertainty. If it doesn’t you shouldn’t be doing it — it’s already been solved!” – Dan North

If you prefer data, the CHAOS report has found consistently that 75% of projects are late/over budget or failed. Even a proponent of estimates, Steve McConnell, says, “we have come to value project control over project estimation, as a means of achieving predictability”.

All of that is to say why we cannot predict the amount of effort required to deliver a particular, defined feature. Here is an even bigger problem. People want estimates before they commit to an initiative. In the agile world an initiative is a problem to solve - there is no particular defined feature. Unless we are doing big-design-up-front and prematurely committing to a solution of a problem not yet well understood there isn’t even anything to estimate. It’s like asking my builder to accurately estimate my renovation defined as “more space and brighter colours”.

“Predefined backlogs of work reduce creativity and inhibit steering the project for success.” – Ron Jeffries

How Estimation is Harmful

Return on investment depends upon cost and value. The focus on estimation puts the emphasis entirely on the cost side. When was the last time someone asked for an estimate that included an estimate of value? Teams that focus on estimates tend to track progress against cost, but do they track progress against value? (see Project Lamb Meatballs - Value-based vs Project-based Portfolio Management)

As high performing teams move from managing outputs to managing outcomes it is natural to shift emphasis from cost towards value. Ron Jeffries agrees, “teams focused on estimation are generally working the short end of the lever of cost vs. value. Better teams focus more on value and less on cost”.

Estimates encourage a zero-sum mindset. On the customer side I want to get as much as I can for as little as possible to demonstrate my superior negotiation capability. Teams get beaten up over their estimates, which were impossible to start with. Then they cut corners and find ways to get back to the estimate, at the cost of creating real value. Estimates are disputed and argued over, then when a requirement changes, the battle starts all over again.

Perhaps most importantly, estimation locks us into a path (the one we estimated), chosen at a time when we had the least possible information (before we started). Intelligent, motivated professionals are unable to pursue to best outcome because they are locked into a specific course of action. In theory it is possible to significantly renegotiate the agreed scope. In practice people tend to just stick to the plan.

How Estimation is Helpful

I once built a deck. I am not a builder and I did not know how to build a deck, therefore I accepted that estimation was impossible. I knew enough to know that the raw materials would be less than $10,000. The other investment was my labour, which I was prepared to contribute until the job was done. I did not estimate the project, but I assembled enough information to support the investment decision. I wanted a deck and I got one within an acceptable cost.

How to Estimate Without Estimating

Here is how this might play out in a consulting scenario:

Me: “Hi, I’d like a deck please! How much will that cost?”

Builder: “‘Deck’ does not precisely define what you want, so I cannot provide an exact figure. Looking at your house, and knowing the type of decks people tend to build for this type of property and in this location I can build you a deck for $10,000. If you’d like a really fancy deck then budget $15,000. I am confident that I can deliver “a deck” within these parameters.”

Me: “Woo hoo!”

The key here is in the vagueness of the project definition: “a deck”. This allows the builder to manage the project to the budget. If decking is expensive the week the project commences she can build a more modest deck that still meets my needs.

As the sponsor of an initiative what I really need is to know that I will get A SOLUTION for A PRICE (or price range). By leaving the precise solution undefined I:

make it possible to reliably find a solution within the budget
leave the experts delivering my solution the freedom to do their jobs to the best of their abilities

Many industries operate this way. You pay your mechanic $X to fix your car. You pay your lawyer $Y to protect your intellectual property. You don’t specify how they should do it, because they are the experts.

While I might pay a trusted builder to construct “a deck” nobody will pay for “an application”. To make this approach specific to software development we need a better way to clarify that the customer will get a solution to their problem (a solution, not the solution).

I like a technique called goal-based requirements analysis. In short, all actors in the system are identified along with their goals. An acceptable solution is one which allows all actors to achieve their goals. That is all an initiative sponsor really wants or needs, and at this level of detail we can nominate an appropriate budget.

Summary

Clarify the problem to be solved.
Work to establish a sufficient budget.
Agree to provide the best solution you can, within the budget agreed.
Follow the outcome not the output.

Project Lamb Meatballs - Value-based vs Project-based Portfolio Management

Wed, 12 Dec 2018 20:32:00 +1000

Intuitively, when a person looks for a technology solution, they begin with a project mindset. They want to understand what they will get, when they will get it, and how much it will cost. This is a logical way to attempt to control cost and limit risk, unfortunately it is incompatible with pursuing value. To understand why, think of a buffet.

Photo by Mohammad Saifullah on Unsplash

The Value Mindset

When one approaches a buffet with a clean, empty plate the desired value is the satisfaction of a good meal. The diner surveys the buffet options then begins to take small quantities of desired foods, until a pleasing meal is assembled. The diner has optimized for the creation of value, according to their criteria (satisfaction of a good meal).

The Project Mindset

Next in line is a hungry project manager. The project manager sees the world not in terms of value to be created but in terms of projects to be started, progressed, completed and retired. The project manager arrives plate in hand at the first item in the buffet, which happens to be loose corn kernels. The project manager begins heaping their plate with corn, then requests more plates when they realize that they have underestimated the required capacity.

They then proceed to eat all the corn in the buffet and declare the corn kernel project a success. They did not achieve the satisfaction of a good meal, but there is no time to think about that before moving on to project lamb meatballs.

If Not Nothing, Then All

When we decide to use technology to assist in a particular area, we often fall into the trap of thinking that if we are doing something we must do everything, presented as defining and completing “the project”. This fails to consider the opportunity cost of any one project. By deciding to see one project (corn kernels) through to the bitter end we miss out on other options that we could be pursuing to create value. All initiatives have diminishing returns so proper portfolio management requires considering all options simultaneously and pursuing each only to the extent that it creates the maximum value (sample many buffet items to create the optimal meal).

Value-based Portfolio Management

The idea described above is sometimes called value-based portfolio management. To core idea is to always be working on the tasks that will create the most value at this moment. When an organisation attempts to follow the value in this way, they quickly come into conflict with the project mindset. The very notion of planning out a program of work and then executing against it conflicts with the idea of pursuing value. If we are to follow the plan, then we cannot follow the value. Since all initiatives have diminishing returns if we are working according to a program of work then at some point we will dip below a threshold where what we are working on is no longer the most valuable thing to the organisation.

An Antidote

One simple solution is to continually evaluate the plan, detect the moment when we cease to be working on the most valuable thing, and correct the plan at that moment. The astute observer will recognize that this looks suspiciously like not following a plan at all.

Another option emerges as we recognize that objectives are more stable than solutions. Instead of planning tasks to be executed, plan in terms of progress against goals. Next quarter we will assemble a satisfying meal from the buffet. The optimal steps to achieve that outcome will change but a plan expressed in terms of outcomes is stable. A talented team will be free to find the optimal solution, instead of being chained to an out-of-date plan.

Challenge, plus constraints

One way to define technology initiatives is challenge, plus constraints. For example:

create an online sales channel for a coffee retailer. Spend less than $1M. Ship before Christmas.

As a sponsor or stakeholder of this initiative this lets me manage my risk, so long as I trust the team executing. It is an unfamiliar model in technology that makes some people uncomfortable, but it is widespread in other contexts. You don’t tell your surgeon how to perform your surgery, or your chef how to prepare your food. As a rule, I propose that mandating to experts how they should do their work is a bad idea. It might feel like it reduces risk, but it does not.

Summary

When we think of projects, we think we need to do everything now or else it will never get done. This is not realistic in technology delivery. Everything is an iterative process that takes time and changes when it encounters real users. Instead, focus on value. Clarify what your goals are and find the thing that will have the most effect right now. If something is important it will eventually get done. If it doesn’t then you probably dodged a bullet.

My Weird Obsession with Glitch

Tue, 23 Oct 2018 20:32:00 +1000

Glitch is an online environment for creating and sharing web applications. It’s like codepen, but with a node.js server side environment.

My Spaced Repetition Flash Card Export ported to Glitch

For easy creation, rapid prototyping and simple experiments it is a wonderful resource.

As you may know, I have a Weird Obsession with Markdown and Spaced Repetition, that led to me creating a system for embedding flash card data into my markdown notes, and using the github API to extract the data and build a CSV for importing into Anki (a program for studying spaced repetition flash cards).

The workflow then is:

write my study note, including flash card data for the things that I would like to memorize
go to codepen to run my extract
copy the CSV from the browser console
save the CSV to a file
import the file into Anki

I don’t have to do it often, but even so this is an intimidating workflow. By moving the extract from codepen to glitch I can learn a little about glitch and improve the workflow to:

write my study note, including flash card data for the things that I would like to memorize
go to glitch to run my extract and download a CSV file
import the file into Anki

Moving to Glitch

After logging into Glitch I created a new project. This provides a working node.js application (I chose to use the Express MVC framework) complete with static file serving and client side code (which I don’t need).

Porting my client-side code from codepen to server side code running on node.js broke a number of things. Firstly, I used the fetch API to make HTTP requests, and this API is not available in a node.js environment. This can be fixed using the node-fetch npm module.

To add an npm module to a glitch project, simply add the dependency to the package.json file. I needed a number of dependencies:

"dependencies": {
    "express": "^4.16.3",
    "papaparse": "*",
    "node-fetch": "*",
    "atob": "*"
  },

The second uncertainty I encountered was how to factor my application into multiple modules and files. I started by implementing the application in a single file server.js and making sure that it worked. Then I moved some code for parsing the micro format into a file (cards.js) and the code for working with the github API into a file (github.js). Importing those modules into another module is done the same way as any other node.js application:

const github = require('./github');

Now I can access my application at https://tangy-silver.glitch.me/cards and download the latest CSV of flash card data.

To spin up a quick, hosted node.js application with the minimal friction, try Glitch.

My Weird Obsession with Markdown and Spaced Repetition

Tue, 02 Oct 2018 20:32:00 +1000

Each time I go back to studying I decide that a better use of my time, instead of studying, is to solve the problem of markdown study notes with embedded spaced repetition flash cards. How’s that for procrastination?

I studied mathematics in 2015 and compiled a comprehensive set of markdown notes. Here’s an excerpt:

Polynomial Derivatives
---

$$\frac{dx^n}{dx} = nx^{n-1}$$

<script type="flashcard">
Q>>> What is the derivative of $$ 7x^4 $$? <<<
A>>> $$ 28x^3 $$ <<<
</script>

### Constants

The derivative of a constant is `0`.

$$\frac{dc}{dx} = 0$$

This can be found from the derivative of polynomials, since $5 = 5x^0 -> 0 * 5x^-1 = 0$.

<script type="flashcard">
Q>>> What is the derivative of $$ 99 $$? <<<
A>>> $$ 0 $$ because $$ 99 = 99x^0 = 0 * 99x^-1 = 0 $$ <<<
</script>

Note the embedded microformat metadata describing flash cards:

<script type="flashcard">
Q>>> What is the derivative of $$ 7x^4 $$? <<<
A>>> $$ 28x^3 $$ <<<
</script>

This information is not displayed in the notes, but is there to generate flash cards for spaced repetition. At the time I built a web application to pull out the flash cards for studying. Sometime in the last three years the domain expired, and the source code is trapped in my bitbucket account that I can’t login to anymore. ¯\_(ツ)_/¯

This year I’m starting an MBA, and immediately felt the need to record notes in markdown. Naturally, that was followed by a desire to embedd flash cards. Since I’ve lost my last solution, I created a new one. I’m using github to edit and store the markdown files, so I delimit the flash card with comments (this is the only content I could find that github doesn’t display):

<!---

--->

For the questions and answers I use an xml-like syntax for easy parsing:

<!---
<question>What is a blah?</question>
<answer>It's a foo!</answer>

<question>What is a foo?</question>
<answer>It's a blah!</answer>
--->

Fetching Content From Github

Fortunately, Github provides a simple API for fetching content from a repository. My solution is currently a codepen script, so JavaScript. This function recursively navigates a github repository and builds a collection of urls to fetch those files.

const collectFilesInDirectory = (directoryUrl) => {
  return fetch(directoryUrl)
    .then((response) => response.json())
    .then((entries) => ({
      files: entries.filter(e => e.type === "file"),
      directories: entries.filter(e => e.type === "dir")
    }))
    .then(({files, directories}) => Promise.all(directories.map((d) => collectFilesInDirectory(d.url)))
          .then((fs) => flatten(fs))
          .then((flattened) => flattened.concat(files.map(f => f.url)))
    )
};

At a high level, it adds the files in the current directory to a collection, as well as doing the same thing for all sub-directories (and their sub-directories).

E.g.

collectFilesInDirectory("https://api.github.com/repos/liammclennan/mba/contents")
  .then(console.log);

// […]
// 0: "https://api.github.com/repos/liammclennan/MBA/contents/Business-Analytics/index.md?ref=master"
// 1: "https://api.github.com/repos/liammclennan/MBA/contents/README.md?ref=master"

Parsing Flash Card Data Out of Markdown

Next task was to extract the metadata blocks out of markdown files. This involved re-learning JavaScript regular expressions. The thing that slowed me down the most was that the RegEx.exec can only do one match at a time, so if you want to select all of the  blocks in a markdown file you must call exec repeatedly until it stops finding matches, which I did with this regular express /(?:)/gm.

Once we have the microformat blocks we must extract the pairs of questions and anwsers. This calls for another regular expression, /(?:<question>\n?)([\s\S]*?)<\/question>(?:[\s\S]*?)<answer>([\s\S]*?)(?:\n?<\/answer>)/gm.

This gives me a nice collection of question / answer pairs, each of which will become a study flash card.

[…]

0: Object { question: "What are the components of business analytics?", answer: "BI / IS, statistics, modelling and optimization" }

1: Object { question: "What is descriptive analytics?", answer: "formatting data for understanding. Creating information from historical data." }

2: Object { question: "What is predictive analytics?", answer: "using data to predict the future" }

3: Object { question: "What is prescriptive analytics?", answer: "Using data to maximize or minimize some value. Using data to facilitate decision making." }

4: Object { question: "What is price elasticity?", answer: "The extent to which price affects demand." }

5: Object { question: "What is a model?", answer: "Abstraction that captures the most important features of a system and presents them in a form that is easy to interpret. " }

Then I used Papa Parse to format this data as csv:

"What are the components of business analytics?","BI / IS, statistics, modelling and optimization"

"What is descriptive analytics?","formatting data for understanding. Creating information from historical data."

"What is predictive analytics?","using data to predict the future"

"What is prescriptive analytics?","Using data to maximize or minimize some value. Using data to facilitate decision making."

"What is price elasticity?","The extent to which price affects demand."

"What is a model?","Abstraction that captures the most important features of a system and presents them in a form that is easy to interpret. "

Using My Flash Card Data for Spaced Repetition Study

The leader in the apparently uncompetitive flash card sector is Anki. Anki can import my CSV file. Now I have my flash cards available in Anki for studying.

Studying looks like this:

Summary

Here is the full source:

See the Pen Parsing QA out of MD by Liam McLennan (@liammclennan) on CodePen.

Startup Software Product Development

Sat, 10 Feb 2018 20:32:00 +1000

Software product development in new businesses is usually limited by a lack of funds and an excess of potential features. There is too much to do, and not enough time or money to do it. A startup needs to make product investment decisions carefully. Some mistakes are likely, but too many false starts can cause the enterprise to fail.

Many, if not most, technology startups seem to fall into the trap of HiPPO (Highest Paid Person’s Opinion) decision making. That is, they choose to invest based on the “gut feel” or “instinct” of the CEO. The problem with intuition is that it is frequently wrong. We can, and should, do better.

A Process For Software Product Development

As the CTO of a startup I have thought a lot about how to maximise our success in the face of much uncertainty. Being able to do a few things well requires that we say no to many opportunities. To guide this decision making we have established a product development process described below.

The first and most important step is to discover, clarify and consolidate the enterprise’s goals into a small number of clear goals that everyone can be aware of, remember and understand. Getting to these goals is more difficult than one might think. I have previously written about The Goal Tree technique for uncovering goals. At times leaders seem unsure about the wisdom of publishing their goals, and sometimes even embarrassed about them. For example, CEOs sometimes feel like they need to be seen to have a lofty change-the-world type goal and may be reluctant to admit the profit motive as a top-level goal. Nothing else described in the post has any value if the goals are wrong, so take the time to ensure that the goals are correctly identified, documented and agreed.

Once the goals have been agreed we need a process for evaluating, prioritising and scheduling initiatives.

1. Socialize and Propose

Good ideas can come from anywhere. Those close to the customers, for example, can often identify opportunities that the engineering team are not aware of. Our process starts with socializing the initiative with stakeholders, then preparing a proposal. The amount of effort invested in the proposal is proportionate to the size and risk of the opportunity. The proposal:

establishes a clear link between the initiative and the business goals that it will affect.
identifies quantitative goals for the initiative itself. Exactly how will we know if this initiative has been successful? If the outcome of an initiative cannot be measured, then leaders should be deeply sceptical. The goals section is the most important part of the proposal and the primary input to the delivery process.
documents research and findings
documents assumptions
documents risk and mitigations
describes the problem being addressed
describes the solution
establishes the items in scope and the items out of scope
estimates the value of the initiative to the enterprise. Work will not proceed if the effort estimated to be required is not significantly less than the estimated value to the enterprise.

Note that the proposal intentionally stays away from documenting solutions. The important information to capture is what we want to achieve (goals) and what is it worth to us.

2. Evaluate

Negotiate an implementation budget. If the initiative is worth $X how much do we want to invest in an initial implementation? As described in Basecamp’s process we try to find a solution that can be completed in 4-8 weeks, with some flexibility to change the team size so we stay in this window.

4. Scheduling

Evaluated proposals are prioritized. This prioritization must be done collaboratively and must be communicated widely. We use a large wall in our office.

Each documented initiative is represented by a card. The cards are sorted according to priority. When someone inevitably asks for all things to be done immediately we go stand in front of the wall the put the cards in the right order. This is a hard process, but essential! Sensible project management means confronting reality and choosing the best path with what we have.

The highest priority proposals get a team allocated and a schedule (start and end date). One member of the team is given responsibility for the overall successful delivery of the initiative.

5. Kickoff

This whole process is intended to maximize effective communication and one important activity is the kickoff meeting. The kickoff meeting:

ensures all stakeholders have a consistent understanding of what will be done and why
clarifies roles and decision making responsibilities
clarifies the schedule
recaps the proposal (goals, risks etc)

6. Design and implementation

Initiative teams are cross-functional including designers and developers (and any other skillsets required for the initiative). Together they must plan a solution that will achieve the agreed goals comfortably within the agreed effort budget. Neither design nor development may be overly ambitions in their planning without risking the success of the initiative, so there is a negotiation. This way we guarantee that the cost of the initiative is matched to its anticipated benefit.

7. Assess

Finally, the initiative is delivered, and its outcome measured. We return to the original proposal and record the measured outcomes against what was planned. Either way, we then have a choice to continue investing in the initiative or to move investment elsewhere based on the latest information. Thus, the process starts again from the beginning.

Learning React 10 - Typescript

Wed, 17 Jan 2018 20:32:00 +1000

Learning React 9 - Data Bound Routes showed the redux-first-router way to implement data bound routes, that is, routes that require fetching some data to be able to render their view. This post will add static typing to our movie library application.

Types

A type is a restriction on data. It defines the set of valid values for that type. E.g. all real numbers are valid for the type number, but only the whole numbers are valid for the type integer. All unicode characters are valid possibilities for the character type, but only ‘true’ and ‘false’ are valid values for the boolean type.

Static Typing

Static typing means that the types of a program are verified for correctness at compile time. The types flow through the program and everywhere they are used the compiler checks that the usage is valid for the type. A static type checker would prevent the following non-sensical operation:

1 + 'a'

on the grounds that there is no good, semantically valid interpretation. It would also prevent 1 + 'a'.toNum(), because the string type does not have a toNum() method. Thus a static type system detects a class of errors that would otherwise potentially occur at runtime.

Gary Bernhardt provided an interesting overview.

To help us write correct React applications it is useful to add a static type checker. Two options are available:

flow, developed by Facebook
typescript, developed by Microsoft

Both are great. This post will convert the movie library application to Typescript.

Converting to Typescript

My preferred way to convert a create-react-app project to Typescript is by creating a new application and porting the code over.

create-react-app-typescript is a project that maintains a Typescript fork of create-react-app. To use it:

> create-react-app movie-library-ts --scripts-version=react-scripts-ts

This will create a new empty application in movie-library-ts.

Next, add type definitions for the external libraries we are using:

> npm install @types/history @types/react @types/react-dom @types/redux @types/react-redux @types/redux-first-router @types/redux-first-router-link @types/redux-promise-middleware --save-dev

This allows Typescript to determine the types of the APIs exposed by those libraries. We use --save-dev because these dependencies are only required for compilation. They are not used at runtime.

Add a file Movie.tsx to the src/ directory. Typescript files have the .ts extension. Typescript files containing JSX have the .tsx extension.

Typescript requires the props parameter to have a type, so declare a type MovieData having an optional property movie of type MovieSummary that contains the basic movie information.

interface MovieSummary {
    Title: string;
    Poster: string;
    imdbID: string;
}

interface MovieData {
    movie?: MovieSummary;
}

function Movie(props: MovieData)
    return (
        <div>
        { props.movie 
            ? <p>
                    <h1>{props.movie.Title}</h1>
                    <img src={props.movie.Poster} alt={`${props.movie.Title} poster`} />
                </p>
            : <p>Loading...</p>
        }
        </div>
    );
}

This guarantees that consumers of the movie component cannot pass the wrong props, or props of the wrong type. Next, add type information to the connect mappers:

const ConnectedMovie = connect(
    function mapStateToProps(state: { movie: MovieData }) {
        return state.movie;
    }, 
    function mapDispatchToProps(dispatch: (action: any) => void) {
        return {};
    }
)(Movie);

Note the type of the dispatch function (action: any) => void. This declares that dispatch is a function with one parameter and no return value.

Converting the other files follows the same pattern, with the exception of the SearchForm component, our single React class component. As a class component it needs to supply type variables to its base class, one for the type of its props, and one for the type of its state.

The other interesting thing to note is that the type of Search props is specified as SearchFormProps & SearchResults which is the combination of those two types. In typescript this is called an Intersection Type.

interface SearchFormProps {
    onSearch: (title: string) => void;
}

interface SearchResults {
    results: MovieSummary[];
}

class SearchForm extends React.Component<SearchFormProps, {title: string}> {
    constructor(props: SearchFormProps) {
        super(props);
        this.state = {title: ''};
        this.titleChange = this.titleChange.bind(this);
        this.search = this.search.bind(this);
    }

    titleChange(event: any) {
        this.setState({title: event.target.value});
    }

    search(event: any) {
        event.preventDefault();
        this.props.onSearch(this.state.title);
    }

    render() {
        return (
        <div><form onSubmit={this.search}>
            <label htmlFor="title">Title: </label>
            <input type="text" name="title" value={this.state.title} onChange={this.titleChange}/>
            <input type="submit" value="Search"/>
        </form>
        </div>
        );
    }
}

function Search({ onSearch, results = [] }: SearchFormProps & SearchResults) {
    return (
    <div>
        <h1>Search</h1>
        <SearchForm onSearch={onSearch} />
        <div>
            {results.map(({Title, Poster, imdbID}) =>
            <Link to={{type: 'MOVIE', payload: {imdbID}}} key={imdbID}>
                <img src={Poster} alt={Title} />
            </Link>)}
        </div>
    </div>
    );
}

Summary

There is a typescript version of create-react-app that makes it easier to setup a new typed React application. Using a type checker to verify the type of props passed to React elements provides a guarantee within the component that the supplied data is correctly typed.

Get The Code

The code for this example is on Github. You can access the code as it was at the completion of this step by cloning the repository and checking out the tag that corresponds to this post.

git clone https://github.com/liammclennan/movie-library-ts.git
git checkout react10

or browse at https://github.com/liammclennan/movie-library-ts/tree/react10

Learning React 9 - Data Bound Routes

Thu, 11 Jan 2018 20:32:00 +1000

Learning React 8 - AJAX Continued has our application capable for searching a movie API and displaying the list of results. Now we would like to be able to view the details for a selected movie.

A good web application should have a URL for every addressable state, so selecting a movie should change the URL. We will use, /movie/:imdbID. First add an action to the routesMap:

const routesMap = { 
    HOME: '/',
    MOVIE: '/movie/:imdbID'
};

Then create a new movie component:

function Movie(props) {
    return <div>
            { props.movie 
                ? <p>
                        <h1>{props.movie.Title}</h1>
                        <img src={props.movie.Poster} alt={`${props.movie.Title} poster`} />
                    </p>
                : <p>Loading...</p>
            }
            </div>;
}

const ConnectedMovie = connect(
    function mapStateToProps(state) {
        return state.movie;
    }, 
    function mapDispatchToProps(dispatch) {
        return {};
    }
)(Movie);

and update the App component to display the ConnectedMovie component when the MOVIE action is dispatched.

const componentMap = {
    HOME: ConnectedSearch,
    MOVIE: ConnectedMovie
};

redux-first-router Route Thunks

Now we encounter a problem. When the router sees the route /movie/:imdbID it will cause the application to show the ConnectedMovie component, which expects to display a movie using data from the redux store, but the data is not there. Somehow we need to take the imdbID from the route and populate the redux store. redux-first-router allows for this using an alternative way of specifying the action-to-route mapping, with an included thunk value. This is a function that runs when the route is matched and that has access to the current store. Within the thunk we can read the imdbID from the store (redux-first-router put it on the location key for us), perform a request, and publish the MOVIE_FETCHED action with the data when we are done.

const routesMap = { 
    HOME: '/',
    MOVIE: {
        path: '/movie/:imdbID',
        thunk: async (dispatch, getState) => {
            dispatch({ type: "MOVIE_CLEAR"});
            const imdbID = getState().location.payload.imdbID;
            const movie = await fetch(`http://www.omdbapi.com/?apikey=8e4dcdac&i=${imdbID}`)
            .then((response) => response.json());
            dispatch({ type: "MOVIE_FETCHED", movie });
        }
    }
};

The MOVIE_CLEAR action is dispatched before the request to clear the previous movie data from the store. The movieReducer then copies the data from the action into the store.

const defaultState = {movie: null};

export const movieReducer = {
    movie: function (state = defaultState, action) {
        switch (action.type) {
            case "MOVIE_CLEAR": 
                return Object.assign({}, state, defaultState);
            case "MOVIE_FETCHED":
                return Object.assign({}, state, {movie: action.movie});
            default: return state;
        }
    }
};

This is what I mean by a ‘data bound route’. Whenever the route /movie/:imdbID is seen we fetch the data for that route and put it in the store.

If an AJAX request is required because of a user action then dispatch an action, with a promise payload, in the mapDispatchToProps function. redux-promise-middleware will resolve the promise and publish the _FULFILLED version of the action, which can be processed in a reducer.

If a route requires an AJAX request we need a different strategy, because routes map to actions, which are processed in reducers, which don’t have the ability to dispatch new actions. redux-first-router provides the thunk option so that we have a place to do asynchronous operations in response to URL changes, and publish actions when we have a result.

Converting Event Actions to Route Actions

Above I said,

A good web application should have a URL for every addressable state

yet, when we perform a search, the application state changes but the URL does not. When you think about it, most actions represent a state change that can, and should, be reflected in a URL change. When I search for ‘Terminator’ I want the url to change to /search/Terminator. Thus a search for terminator becomes an addressable state that I can link directly to.

redux-first-router synchonizes in both directions. If the URL changes it publishes the corresponding route action. If a route action is dispatched it changes the URL. To make search results addressable we will handle the search form submission, dispatch a route action, then do the search using the route handling thunk. First, modify the action that is dispatched when a user searches:

function mapDispatchToProps(dispatch) {
    return {
        onSearch: (title)=> {
            dispatch({
                type: 'SEARCH',
                payload: {title}
                });
        } 
    };
}

This SEARCH action will be a route action. Next, add the SEARCH action to the route configuration, mapped to /search/:title. When the route is matched, read the title from the Redux store, query the API, and then dispatch the SEARCH_FULFILLED action that the searchReducer is already expecting.

const routesMap = { 
    // ...
    SEARCH: {
        path: '/search/:title',
        thunk: async (dispatch, getState) => {
            const title = getState().location.payload.title;
            const results = await fetch(`http://www.omdbapi.com/?apikey=8e4dcdac&s=${encodeURIComponent(title)}`)
                .then((response) => response.json())
            dispatch({ type: "SEARCH_FULFILLED", payload: results });
        }
};

The routing for the application is now:

Route action	URL	Thunk	Component
`HOME`	`/`		ConnectedSearch
`MOVIE`	`/movie/:imdbID`	Publishes `MOVIE_CLEAR` actions. Fetches movie data by `imdbID`. Publishes `MOVIE_FETCHED` action.	ConnectedMovie
`SEARCH`	`/search/:title`	Fetches results by `title`. Publishes `SEARCH_FULFILLED`.	ConnectedSearch

As the onSearch handler no longer publishes an action with a promise we are no longer using redux-promise-middleware. Many actions can be route changes and do their asynchronous operations with redux-first-router thunks. Actions that do not make sense as route changes, but that still require an asynchronous operation, should still use redux-promise-middleware.

Summary

Where sensible, states should be addressable via a route. Where a route requires an asynchronous operations, such as making an AJAX request, use the thunk property of the route configuration object to do the asynchronous operation and dispatch an action when it is complete.

Next: Learning React 10 - Typescript

Get The Code

The code for this example is on Github. You can access the code as it was at the completion of this step by cloning the repository and checking out the tag that corresponds to this post.

git clone https://github.com/liammclennan/movie-library.git
git checkout react9

or browse at https://github.com/liammclennan/movie-library/tree/react9.

Learning React 7 - AJAX

Wed, 10 Jan 2018 20:32:00 +1000

In Learning React 6 - React Router we learned how to connect a client-side router to a React application.

In this post we will look at how to add AJAX HTTP requests to a React application. This requires more thought than introducing AJAX to products built with other libraries, because our React application so far has used pure components (components that produce output entirely determined by their inputs) and been entirely synchronous (processing proceeds continuously without having to wait for any external events).

Movie Library

For this demonstration we will need a more complicated application than the one we have used so far, so we will build a movie library. The application will allow the user to:

search the OMDB movie database API for movies.
view the details of a movie
select movies to add to their favourite list
view their favourite list

Start by creating a new application with create-react-app:

$ create-react-app movie-library

and start the app:

$ npm start

Install redux, react-redux and a router (redux-first-router):

$ npm install --save redux react-redux history redux-first-router redux-first-router-link

Modify index.js to use a default redux implementation:

import React from 'react';
import ReactDOM from 'react-dom';
import './index.css';
import registerServiceWorker from './registerServiceWorker';
import { createStore } from 'redux';
import { Provider } from 'react-redux';
import App from './App';

function reducer(state = { }, action) {
    switch (action.type) {
        default: return state;
    }
}
const store = createStore(reducer);

ReactDOM.render(
    <Provider store={store}>
        <App />
    </Provider>, 
    document.getElementById('root'));

store.dispatch({type: "START"});

registerServiceWorker();

The Search Component

The starting page of the movie library will feature a movie search component.

Start by creating a new file, Search.js containing a minimal Search component:

import React from 'react';
import { connect } from 'react-redux';

function Search() {
    return <div>
        <h1>Search</h1>
    </div>;
}

export const ConnectedSearch = connect(
    function mapStateToProps(state) {
        return state;
    }, 
    function mapDispatchToProps(dispatch) {
        return {};
    }
)(Search);

The Search module exports a component ConnectedSearch that is our search component wrapped in a react-redux connector to give it access to the redux store (we will need this later). Now, modify App.js to import and use the ConnectedSearch component:

 import React, { Component } from 'react';
import './App.css';
import { ConnectedSearch } from './Search';

class App extends Component {
  render() {
    return (
      <div className="App">
        <header className="App-header">
          <h1 className="App-title">Movie Library</h1>
        </header>
        <ConnectedSearch />
      </div>
    );
  }
}

export default App;

Now the application should be working a displaying a heading ‘Search’.

The Search component needs a text input. We will create a new component SearchForm to contain the text input. To get the form to function properly we must use React’s component ‘state’ feature, which means using the class component syntax instead of the function syntax. Both ‘state’ and the class syntax are undesirable, hence why they are contained in the smallest possible area where they are absolutely needed (the SearchForm component).

class SearchForm extends React.Component {
    constructor(props) {
        super(props);
        this.state = {title: ""}
        this.titleChange = this.titleChange.bind(this);
        this.search = this.search.bind(this);
    }

    titleChange(event) {
        this.setState({title: event.target.value});
    }

    search(event) {
        event.preventDefault();
        this.props.onSearch(this.state.title);
    }

    render() {
        return <form onSubmit={this.search}>
            <label htmlFor="title">Title: </label>
            <input type="text" name="title" value={this.state.title} onChange={this.titleChange}/>
        </form>;
    }
}

function Search() {
    return <div>
        <h1>Search</h1>
        <SearchForm />
    </div>;
}

export const ConnectedSearch = connect(
    function mapStateToProps(state) {
        return state;
    }, 
    function mapDispatchToProps(dispatch) {
        return {
            onSearch: (title)=> { dispatch({type: "SEARCH", title}); }
        };
    }
)(Search);

When a user types a character in the textbox it fires the onChange event, which updates the components state to have the latest value of the textbox, which causes the component to re-render and display the character the user typed in the textbox.

When the form is submitted we call the onSearch function on props. The React-Redux connection translates this into a dispatch of an action with type: "SEARCH".

Summary

We started building a React movie library with React, Redux, react redux, and redux-first-router. We implemented a class component for a form, with local state and connected everything to our Redux store.

Next: Learning React 8 - AJAX Continued

Get The Code

The code for this example is on Github. You can access the code as it was at the completion of this step by cloning the repository and checking out the tag that corresponds to this post.

git clone https://github.com/liammclennan/movie-library.git
git checkout react7

or browse at https://github.com/liammclennan/movie-library/tree/7548d450b7727f124070dd7ee9b73575d740e9b3.

Learning React 8 - AJAX continued

Wed, 10 Jan 2018 20:32:00 +1000

The last post started implementing our new demo application - the movie library. We were forced to introduce a React class component with state to make the search form work.

When we type in the search text box and submit the form the expected action is sent to the redux store, as you can see in this screenshot featuring redux-devtools-extension.

Routing

For this example we are going to use redux-first-router. As mentioned previously redux-first-router binds routes to actions. We must handle those actions and update the UI accordingly.

See the redux-router documentation for the full documentation, but we need to make index.js look like this:

import React from 'react';
import ReactDOM from 'react-dom';
import './index.css';
import registerServiceWorker from './registerServiceWorker';
import { createStore, combineReducers, applyMiddleware, compose } from 'redux';
import { Provider } from 'react-redux';
import { connectRoutes } from 'redux-first-router';
import createHistory from 'history/createBrowserHistory';
import App from './App';
import {searchReducer} from './Search';

const routesMap = { 
  HOME: '/'
};

const { reducer, middleware, enhancer } = connectRoutes(createHistory(), routesMap)

const store = createStore(combineReducers({
    location: reducer,
    search: searchReducer
}), compose(enhancer, applyMiddleware(middleware),
window.__REDUX_DEVTOOLS_EXTENSION__ && window.__REDUX_DEVTOOLS_EXTENSION__()));

ReactDOM.render(
    <Provider store={store}>
        <App />
    </Provider>, 
    document.getElementById('root'));

store.dispatch({type: "START"});

registerServiceWorker();

routesMap defines the mapping between action types and routes. Here we are saying that the route / should map to the HOME action type. The redux-first-router state is put into the store on the location key.

Next we connect the App component to the Redux store:

function App({type}) {
    const componentMap = {
      HOME: ConnectedSearch
    };
    const View = componentMap[type];

    return (
      <div className="App">
        <header className="App-header">
          <h1 className="App-title">Movie Library</h1>
        </header>
        <View />
      </div>
    );
}

export default connect(function mapStateToProps(state) {
  return state.location;
}, 
(dispatch) => ({}))(App);

redux-first-router puts route information into the redux store when a route is matched. mapStateToProps extracts the location key from the store, which is then used in the App component to extract the type value.

componentMap maps route action types to React components. If the route action type is HOME then View will be a ConnectedSearch, which is what we want for the home page.

AJAX Actions

The difficulty we encounter when trying to search for a movie (via an AJAX request) is that we can’t take the search action ({ type: "SEARCH", title: "Term" }) and update the UI state from that. We need to make an asynchronous request in the middle.

Nearly every React application must overcome this difficulty. There are lots of different solution and I will present one.

First, install redux-promise-middleware.

$ npm install redux-promise-middleware --save

redux-promise-middleware is a library that intercepts actions that carry a promise (SEARCH). When the promise resolves the library dispatches a new action with the result (SEARCH_FULFILLED).

We create a promise using the fetch API to make a request to omdbapi to search for movies by title.

We dispatch:

dispatch({
    type: 'SEARCH',
    payload: fetch(`http://www.omdbapi.com/?apikey=8e4dcdac&s=${encodeURIComponent(title)}`)
        .then((response) => response.json())
});

and in the reducer look for SEARCH_FULFILLED:

export function searchReducer(state = {results: []}, action) {
    switch (action.type) {
        case "SEARCH_FULFILLED":
            return Object.assign(
                {}, 
                { results: action.payload.Response 
                        ? action.payload.Search.filter(({Poster}) => Poster !== "N/A") 
                        : []});
        default: return state;
    }
}

Update the Search component to display our recently acquired movie data:

function Search({ onSearch, results = [] }) {
    return <div>
        <h1>Search</h1>
        <SearchForm onSearch={onSearch} />
        <div>
            {results.map(({Title,Poster,imdbID})=> 
                <img src={Poster} alt={Title} key={imdbID} />)}
        </div>
    </div>;
}

Now look at our amazing application!

The complete Search.js is:

import React from 'react';
import { connect } from 'react-redux';

class SearchForm extends React.Component {
    constructor(props) {
        super(props);
        this.state = {title: ""}
        this.titleChange = this.titleChange.bind(this);
        this.search = this.search.bind(this);
    }

    titleChange(event) {
        this.setState({title: event.target.value});
    }

    search(event) {
        event.preventDefault();
        this.props.onSearch(this.state.title);
    }

    render() {
        return <div><form onSubmit={this.search}>
            <label htmlFor="title">Title: </label>
            <input type="text" name="title" value={this.state.title} onChange={this.titleChange}/>
            <input type="submit" value="Search"/>
        </form>
        </div>;
    }
}

function Search({ onSearch, results = [] }) {
    return <div>
        <h1>Search</h1>
        <SearchForm onSearch={onSearch} />
        <div>
            {results.map(({Title,Poster,imdbID})=> <img src={Poster} alt={Title} key={imdbID} />)}
        </div>
    </div>;
}

export const ConnectedSearch = connect(
    function mapStateToProps(state) {
        return state.search;
    }, 
    function mapDispatchToProps(dispatch) {
        return {
            onSearch: (title)=> {
                dispatch({
                    type: 'SEARCH',
                    payload: fetch(`http://www.omdbapi.com/?apikey=8e4dcdac&s=${encodeURIComponent(title)}`)
                            .then((response) => response.json())
                  });
            }
        };
    }
)(Search);

export function searchReducer(state = {results: []}, action) {
    switch (action.type) {
        case "SEARCH_FULFILLED":
            return Object.assign(
                {}, 
                { results: action.payload.Response 
                        ? action.payload.Search.filter(({Poster}) => Poster !== "N/A") 
                        : []});
        default: return state;
    }
}

Summary

First, we setup redux-first-router and implemented the HOME route. Then we introduced redux-promise-middleware as a way of inserting asynchronous operations into the redux data cycle.

Next: Learning React 9 - Data Bound Routes

Get The Code

The code for this example is on Github. You can access the code as it was at the completion of this step by cloning the repository and checking out the tag that corresponds to this post.

git clone https://github.com/liammclennan/movie-library.git
git checkout react8

or browse at https://github.com/liammclennan/movie-library/tree/c93c652fa1318ce6447d985e3f8b690386456565.

Learning React 6 - React Router

Mon, 08 Jan 2018 20:32:00 +1000

Learning React 5 - React-Redux explored the helpers provided by Redux for integration into a React application. Our today application is now using Redux to store and manage state.

Applications designed for the web should be URL oriented. Adhering to this expectations enables navigation, caching, book marking and many other standard activities. Implmenting routing in a React application requires a routing library, and the most popular is React Router.

React Router

In this post we will add routing, using React Router, to our today application. We will support URLs like,

localhost:3000/day/2018-03-20

That will set the day value to March 20. Instead of having to use the ‘Next’ button users will be able to navigate directly to day of their choosing, and instead of the ‘Next’ button being implemented as a state change it will become a simple navigation link to the next day.

To start, as always, install the new library:

> npm install react-router-dom --save

To add the router, without modifying the existing behaviour we:

import some React Router functions
add a Router to our component tree
add a Route component that maps a URL to a component

import { BrowserRouter as Router, Route } from 'react-router-dom';

ReactDOM.render(
    <Provider store={store}>
        <Router>
            <Route exact path="/" component={ConnectedToday} />
        </Router>
    </Provider>, 
    document.getElementById('root'));

This declares that the root path (‘/‘) should result in the the ConnectedToday component being rendered in the position specified.

Now our application is using a router, but behaving exactly as before.

Let’s add another route for /day/YYYY-MM-DD:

ReactDOM.render(
    <Provider store={store}>
        <Router>
            <div>
                <Route exact path="/" component={ConnectedToday} />
                <Route path="/day/:datestring" component={ConnectedToday} />
            </div>
        </Router>
    </Provider>,
    document.getElementById('root'));

Note that :datestring represents a querystring parameter variable. Next, modify the Today component to read its date from the router instead of from Redux:

function Today(props) {
    const day = new Date(props.match.params.datestring);
    return <div>Today is {day.toString()} 
        <button type="button" onClick={props.onNext}>Next</button></div>;
}

React Router uses the same trick as Redux to supply route parameters to components without having to manually pass them as props.

Now you can navigate to a URL like http://localhost:3000/day/1994-5-8 to get the Today component rendered for the 8th May 1994. We have broken the next button because it increments a day value in the Redux store, but our component takes its day value from the router. Instead of incrementing a value in the store we can instead use a link to the next day:

function Today(props) {
    const day = new Date(props.match.params.datestring);
    const nextDay = new Date(day.getTime() + (24*3600*1000));
    const tomorrowDateString = `${nextDay.getFullYear()}-${nextDay.getMonth()+1}-${nextDay.getDate()}`;
    return (
        <div>Today is {day.toString()} 
            <Link to={"/day/" + tomorrowDateString}>Next</Link>
        </div>);
}

Link is a React Router component used to generate links.

Now our UI is bound to the URL. As we navigate through consecutive days you will see the URL change. Also, if you manually change the URL the UI is updated to match. To make our routing more interesting, let’s add a Home component on the root URL:

ReactDOM.render(
    <Provider store={store}>
        <Router>
            <div>
                <Route exact path="/" component={Home} />
                <Route path="/day/:datestring" component={ConnectedToday} />
            </div>
        </Router>
    </Provider>,
    document.getElementById('root'));

The Home component should display a link to a randomly selected day:

function Home() {
    const randomBetween = (min, max) => Math.floor(Math.random() * (max-min+1) + min);
    const randomDateString = 
        `${randomBetween(1900, 2100)}-${randomBetween(1, 12)}-${randomBetween(1, 28)}`;

    return <div><h1>Home</h1>
        <Link to={`/day/${randomDateString}`}>{randomDateString}</Link>
    </div>;
}

With this completed we have a home component that links to a random day, that has a next button that advances (via the URL) to the next day. For this particular application at this time we could delete all of the Redux and Redux-router code, because we are no longer managing any state in Redux.

redux-first-router

I chose Redux Router for this example because it is by far the most popular React router, but the awkwardness of integrating React Router and Redux into an application is bothersome. An alternative is to use a redux-first-router instead of React Router. redux-first-router does not bind routes to components, instead it binds routes to actions. When a route is matched redux-first-router dispatches an action to that effect and everything else is handled the Redux way with reducers.

Summary

URLs are the central abstraction of a well-behaved web application. To achieve this with React we need a router. React router is the popular choice although it does have some friction with state management solutions like Redux. redux-first-router is another alternative.

Next: Learning React 7 - AJAX

Get The Code

The code for this example is on Github. You can access the code as it was at the completion of this step by cloning the repository and checking out the tag that corresponds to this post.

git clone https://github.com/liammclennan/today-react-app.git
git checkout react6

or browse at https://github.com/liammclennan/today-react-app/tree/4f53d611e3ecaa3d5bcfe92e970e56fcf6981860.

Learning React 4 - Redux

Fri, 05 Jan 2018 20:32:00 +1000

The post Learning React 3 - Data out with props finished with a component that is customised by data passed in, and that publishes user events, which modifies that application state, and causes the component to update.

function Today({ day, onNext }) {
    return <div>
                Today is {day.toString()}
                <button type="button" onClick={onNext}>Next</button>
            </div>;
}

let day = new Date(2018,2,20);

function onNext() {
    day = new Date(day.getTime() + (24*3600*1000)); 
    render();
}

function render() {
    ReactDOM.render(<Today day={day} onNext={onNext} />, 
        document.getElementById('root'));
}
render();

To keep the Today component simple and composable the state has been entirely pulled out of the component (into the day variable).

As applications become more complex it becomes more difficult to work with this external state and keep track of the modifications that need to be made in response to events.

Redux

Redux is a tool for managing application state and controlling changes to that state. I find the Redux documentation unnecessarily confusing. All you really need to know to understand Redux is:

Redux holds state in a single Javascript object, called a store.
Redux uses functions (sometimes referred to as reducers) to apply events (sometimes called actions) to update the application state.
Redux has a function createStore that creates a store.
Redux has a function getState that gets the application state from the store.
Redux has a function subscribe that registers listeners to be called when the application state changes.
Redux has a function dispatch that is used to send an event (action) to the reducer to update the store.

Consider our Today component. First we add redux to our project:

> npm install redux --save

Now we can put our application state into a redux store:

function reducer(state = { day: new Date(2018,2,20) }, action) {
    return state;
}
const store = createStore(reducer);

store.subscribe(() => {
    ReactDOM.render(<Today day={store.getState().day} onNext={onNext} />, 
        document.getElementById('root'));
});

store.dispatch({type: "START"});

That is enough to get the application to render with its initial state. Note that the initial application state ({ day: new Date(2018,2,20) }) is set as the default value of the state argument.

The event handling is broken because we can no longer directly modify the day variable. Instead we must dispatch an appropriate action (NEXT_DAY) to the redux store. In the reducer function, if we see our action (NEXT_DAY) then we return a new application state with the day value advanced by one day:

function onNext() {
    store.dispatch({type: "NEXT_DAY"});
}

function reducer(state = { day: new Date(2018,2,20) }, action) {
    switch (action.type) {
        case "NEXT_DAY":
            const currentDay = store.getState().day;
            return { day: new Date(currentDay.getTime() + (24*3600*1000))};
        default: return state;
    }
}

Both the state object and the action objects can be any Javascript value. It is a convention to include a type property on action values. The type property can then be used in reducer functions to determine which event/action has occurred so that the application state can be updated appropriately.

The full program is now:

function Today({ day, onNext }) {
    return <div>Today is {day.toString()}
        <button type="button" onClick={onNext}>Next</button></div>;
}

function onNext() {
    store.dispatch({type: "NEXT_DAY"});
}

function reducer(state = { day: new Date(2018,2,20) }, action) {
    switch (action.type) {
        case "NEXT_DAY":
            const currentDay = store.getState().day;
            return { day: new Date(currentDay.getTime() + (24*3600*1000))};
        default: return state;
    }
}
const store = createStore(reducer);

store.subscribe(() => {
    ReactDOM.render(<Today day={store.getState().day} onNext={onNext} />, 
        document.getElementById('root'));
});

store.dispatch({type: "START"});

Summary

Redux can be confusing, because of all the additional machinery that has accumulated around it, and because of the way the documentation is presented. At its core Redux is just a container for application state and a formalized way of modifying that application state.

Good React components are simple, pure functions of their inputs (props). Redux is a helpful tool that makes this much easier.

Next: Learning React 5 - React-redux

Get The Code

The code for this example is on Github. You can access the code as it was at the completion of this step by cloning the repository and checking out the tag that corresponds to this post.

git clone https://github.com/liammclennan/today-react-app.git
git checkout react4

or browse at https://github.com/liammclennan/today-react-app/tree/9f17db0e8d19c505efaaa7e828cb71e535e2377a.

Learning React 5 - React-Redux

Fri, 05 Jan 2018 20:32:00 +1000

In Learning React 4 - Redux we learnt about what Redux can do for your React application and how it can be integrated. The result was the following code:

function Today({ day, onNext }) {
    return <div>Today is {day.toString()}
        <button type="button" onClick={onNext}>Next</button></div>;
}

function onNext() {
    store.dispatch({type: "NEXT_DAY"});
}

function reducer(state = { day: new Date(2018,2,20) }, action) {
    switch (action.type) {
        case "NEXT_DAY":
            const currentDay = store.getState().day;
            return { day: new Date(currentDay.getTime() + (24*3600*1000))};
        default: return state;
    }
}
const store = createStore(reducer);

store.subscribe(() => {
    ReactDOM.render(<Today day={store.getState().day} onNext={onNext} />, 
        document.getElementById('root'));
});

store.dispatch({type: "START"});

Note that the Today component is not coupled to the Redux store (which is good), but also that we are manually passing data into and out of the Today component. This is fine for our example but typically a component is contained many levels deep in a component hierarchy. The design we have so far would require us to pass the day prop manually down through each level of the hierarchy and the onNext prop manually back up through each level of the hierarchy.

React-redux

React, and Redux, provide a solution to this problem, implemented in a package called React-redux. Install in the usual way:

> npm install react-redux --save

React-redux provides a way to connect a component’s props to the pieces of the Redux state that the component cares about. When creating a component you get this work by:

defining the mapping from the Redux state to the components props
defining the mapping from the components props to actions dispatched to the Redux store

The above mappings are combined with an existing component to produce a new component, that is connected to Redux. For the Today component this is:

import { connect, Provider } from 'react-redux';

function Today({ day, onNext }) {
    return <div>Today is {day.toString()}
        <button type="button" onClick={onNext}>Next</button></div>;
}

const ConnectedToday = connect(
    // Today uses the 'day' property of the Redux state. 
    // Which is the whole thing. 
    function mapStateToProps(state) {
        return state;
    }, 
    function mapDispatchToProps(dispatch) {
        return {
            // This 'onNext' function is provided as a prop to the 'Today' component, 
            // but with 'dispatch' in scope. 
            onNext: ()=> { dispatch({type: "NEXT_DAY"}); }
        };
    }
    )(Today);

ConnectedToday is a new component that wraps the Today component and connects it to the Redux store. Now when we render our application we can use the ConnectedToday component instead of the Today component, and remove the props.

To make it work we must wrap our component tree in a Provider, that makes the store available to connected components. We can also drop the subscription to the store (React-redux takes care of this for us):

ReactDOM.render(
    <Provider store={store}>
        <ConnectedToday />
    </Provider>, 
    document.getElementById('root'));

Now we have achieved our goal. The today component is still independent of the Redux store, and we have avoided manually passing state and events up and down the component hierarchy. Here is the final code:

import React from 'react';
import ReactDOM from 'react-dom';
import './index.css';
import registerServiceWorker from './registerServiceWorker';
import { createStore } from 'redux';
import { connect, Provider } from 'react-redux';

function Today({ day, onNext }) {
    return <div>Today is {day.toString()} 
        <button type="button" onClick={onNext}>Next</button></div>;
}

const ConnectedToday = connect(
    function mapStateToProps(state) {
        return state;
    }, 
    function mapDispatchToProps(dispatch) {
        return {
            onNext: ()=> { dispatch({type: "NEXT_DAY"}); }
        };
    }
    )(Today);

function reducer(state = { day: new Date(2018,2,20) }, action) {
    switch (action.type) {
        case "NEXT_DAY":
            const currentDay = store.getState().day;
            return { day: new Date(currentDay.getTime() + (24*3600*1000))};
        default: return state;
    }
}
const store = createStore(reducer);

ReactDOM.render(
    <Provider store={store}>
        <ConnectedToday />
    </Provider>, 
    document.getElementById('root'));

store.dispatch({type: "START"});

registerServiceWorker();

Summary

When React and Redux are used together ‘React-redux’ is usually used as a convenient way to join them together. It is simply a matter of:

Wrapping the component tree in a <Provider/>
Providing a mapStateToProps and mapDispatchToProps function for each connected component

Next: Learning React 6 - React Router

Get The Code

The code for this example is on Github. You can access the code as it was at the completion of this step by cloning the repository and checking out the tag that corresponds to this post.

git clone https://github.com/liammclennan/today-react-app.git
git checkout react-5

or browse at https://github.com/liammclennan/today-react-app/tree/18a03e73197b15eef0746a01fdd111942ff1a2d3.

Learning React 3 - Data out with props

Wed, 03 Jan 2018 20:32:00 +1000

Learing React 2 - Data in with props showed how to reuse a component by passing data in to that component. This post will show how to pass data back out of a component, again using props.

Data is passed out of a React component by calling functions passed in as props

Recall our Today component:

function Today({ day }) {
    return <div>Today is {day.toString()}</div>;
}

We add a button:

function Today({ day }) {
    return <div>
                Today is {day.toString()}
                <button type="button">Next</button>
            </div>;
}

Now our UI has a button. Let’s modify the usage of Today to pass a function that will be called when the ‘Next’ button is clicked:

function onNext() {
    console.log('Next!');
}

ReactDOM.render(<Today day={new Date(2018, 2, 20)} onNext={onNext} />, 
    document.getElementById('root'));

and connect this to the onClick event of the button:

function Today({ day, onNext }) {
    return <div>
                Today is {day.toString()}
                <button type="button" onClick={onNext}>Next</button>
            </div>;
}

Now when the button is clicked the text ‘Next!’ is written to the console. By passing arguments to the onNext function we have a mechanism for getting events and data out of our component.

Now we can use the onNext event to modify the day value and update the UI, completing the circle of data flow.

let day = new Date(2018,2,20);

function onNext() {
    day = new Date(day.getTime() + (24*3600*1000)); 
    render();
}

function render() {
    ReactDOM.render(<Today day={day} onNext={onNext} />, 
        document.getElementById('root'));
}
render();

Note that I had to move the call to ReactDOM.render into a function and call that function after updating the value of day. This is because React does not automatically now that its model data has changed and it needs to re-render.

Summary

Data is passed into a React component as a value supplied as props. Data is passed out of a React component by calling a function passed in as props. Components created in this style have a wonderful simplicity. Their interface is defined entirely by props. You should try to code all of your React components in this way.

Next: Learning React 4 - Redux

Get The Code

The code for this example is on Github. You can access the code as it was at the completion of this step by cloning the repository and checking out the tag that corresponds to this post.

git clone https://github.com/liammclennan/today-react-app.git
git checkout react3

or browse at https://github.com/liammclennan/today-react-app/tree/b59c3dbeb3597395d6c0c7d9b2831bf3999e870a.

Learning React 2 - Data in with props

Tue, 02 Jan 2018 20:32:00 +1000

In Learning React 1 - A first component I showed how to bootstrap a React application and implement a simple component that displays the current date.

The Today component is highly specialized. It displays the current date, but can’t do anything else. Often we would like to generalize a component to increase the opportunities for reuse. This requires the capability to pass data into a React component.

Data is passed to a React component using ‘props’

Props are specified when the component is used, as attributes. We can pass a date to the Today component like this:

ReactDOM.render(<Today day={new Date(2018, 2, 20)} />, document.getElementById('root'));

Then in the Today component this value is available on the props object passed to the function:

function Today(props) {
    return <div>Today is {props.day.toString()}</div>;
}

Now the output is:

Today is Tue Mar 20 2018 00:00:00 GMT+1000 (AEST)

If you are wondering why it is Mar 20 and not Feb 20, it is because JavaScript uses 0 based indexing for months.

When reading other peoples React code you will often see Object destructuring used to show what is expected in props:

function Today({ day }) {
    return <div>Today is {day.toString()}</div>;
}

This is a purely syntactical change. The program behaves exactly the same as before.

Summary

By introducing props to our React component we have generalized it to work with any date. This is the mechanism by which data is passed to React components, and passed down a hierarchy of React components.

Next: Learning React 3 - Data out with props

Get The Code

The code for this example is on Github. You can access the code as it was at the completion of this step by cloning the repository and checking out the tag that corresponds to this post.

git clone https://github.com/liammclennan/today-react-app.git
git checkout react2

or browse at https://github.com/liammclennan/today-react-app/tree/6c4a389b57211a3c894db10c34f444d37e6b3c69.

Learning React 1 - A first component

Mon, 01 Jan 2018 20:32:00 +1000

While doing some maintenance on my Pluralsight course, React Fundamentals I have been thinking about how to teach good front-end development skills. At first I was puzzled as to why people struggled, when React is fundamentally very simple.

To build a complete application, React is not enough. You have to add supporting tools to help with routing, bundling, state management and other services not provided by React itself. These additional tools add a lot of complexity, and they are presented to learners in their final, complete and extremely confusing way. Between the React ‘Hello World’ demo and a completed, ready-for-production application there is a chasm that seems to be ignored by the standard documentation.

In this post, and more to follow I intend to show how a React application is built up from first principles. I will try to add complexity gradually and explain why the different pieces are necessary. Let’s begin with a first component.

A First Component

Let us create a React component that displays today’s date. We will use create-react-app because it is a good starting point and a defacto standard.

First, install create-react-app if you don’t have it already:

> npm install -g create-react-app

Then scaffold a new application:

> create-react-app a-first-component
> cd a-first-component
> npm start

You will see the default application scaffold rendered in your browser.

Open the file src/index.js. It should look like:

import React from 'react';
import ReactDOM from 'react-dom';
import './index.css';
import App from './App';
import registerServiceWorker from './registerServiceWorker';

ReactDOM.render(<App />, document.getElementById('root'));
registerServiceWorker();

A React component is just a function that returns some HTML (technically JSX which is a HTML-like markup language that is compiled to JavaScript). A React component that shows the current date is:

function Today() {
    return <div>Today is {new Date().toString()}</div>;
}

Putting that into src/index.js and rendering it instead of <App /> we now have src/index.js containing:

// ... imports as above

function Today() {
    return <div>Today is {new Date().toString()}</div>;
}

ReactDOM.render(<Today />, document.getElementById('root'));

And the browser renders our app as:

Today is Wed Jan 03 2018 15:16:43 GMT+1000 (AEST)

Summary

Creating a React app with the most basic component I can think of is simple. In future posts I will build on this on step at a time until we have a viable application.

Next: Learning React 2 - Data in with props

Goals, Strategic Planning and Product Road Maps

Thu, 16 Nov 2017 20:32:00 +1000

The product development process, left to its own devices, does not proceed with order and efficiency to the ideal output. Without constant guidance, and vigilance, progress will be retarded by individual interests, disproportionate reactions, responding to stakeholder requests, scope creep, over-engineering and decisions made without all of the necessary context.

Successful teams optimize their path to agreed goals by minimizing waste. Build only what is absolutely necessary, when it is absolutely necessary.

Product delivery management happens at two levels. The low-level management of planning the next sprint has been well understood, but focuses on activity over outcome. The higher-level view of why we are doing what we are doing is typically neglected and defaults to a shopping list of features that someone high in the org chart thinks are desirable.

At Starts at 60, we have developed a simple and effective process with three principal steps:

Identify the goals that the product serves. All activity must be mapped to at least one goal.
Use Impact Mapping to derive strategic initiatives from the goals.
Organise the strategic initiatives into a highly-visible, living roadmap that forces prioritization.

Identify Goals

The first and most important step is to pick your goals. A good team working to clear goals will get buy without anything else. Choosing goals sounds easy but it is not. The Goal Tree describes a process for getting to real business goals.

Guidelines for Goal Selection

Keep the list of goals small. Goals will tend to conflict with each other and if the set of possible conflicts is large things become unmanageable. We work to 4 goals and I think that is about the practical limit. The following graph shows how quickly the number of potential conflicts increases as the number of goals increases.
Goals will often be financial and there is nothing wrong with that.
Beware of features expressed as a goal. If a feature is suggested ask why that feature is desirable. Unearthing the goal gives the team the opportunity to replace the selected feature with a better one.
Goals should be SMART (specific, measurable, achievable, relevant and time-bound). When the goal is defined also define when the metric will be measured, how it will be measured and what the expected value should be.

Derive Strategic Initiatives from the Goals

Goals define where you want to go. Impact Mapping is a collaborative, strategic planning technique that helps a team to determine which initiatives will have the most impact, as defined by the agreed goals.

The impact mapping process is:

Identify the actors who can help or hinder our progress to the goals.
For each actor, list the behaviour change we would like them to make.
Brainstorm initiatives that could be undertaken to achieve the desired behaviour changes.
Choose which of the initiatives identified are the most appropriate to pursue.

Much of the impact mapping process relies on assumptions that should be made explicit and documented. The impact map is a living document that must be updated as circumstances change or assumptions are dis-proven.

The output of the impact mapping process is the set of initiatives most likely (according to the information on hand) to result in hitting our goals.

Create a Roadmap

Once we know the things we should do we need to prioritize them into a roadmap. At Starts at 60 we have two products: a media site and a travel site.

Starts at 60

Travel at 60

The technology team has commandeered a wall to display a simple roadmap. Each column is a product (TAS & SAS). Planned initiatives are represented by coloured cards and ordered by priority. The top row shows which initiatives are currently being developed (in progress). Each in progress initiative has a headshot of the person responsible for delivery attached.

This information was originally captured in a Trello board, but it did not work. People only engage with Trello if they choose to pull the information (by going to the Trello board). The advantage of a large, wall display is that it pushes the information to stakeholders, whether they want it or not ;)

In a startup things can change quickly. The huge-visible-roadmap is a great tool to at least document what the current plan is. Keep a stack of cards and some markers handy.

Dealing with Great New Ideas

The curse of any roadmap, or any product planning, is the “great new idea”. When a great new idea is presented to the delivery team the default assumption is always that it will happen, and that it will not affect the delivery of all the initiatives already in progress. The process detailed above provides tools for dealing with this situation.

Firstly, the great new idea must be worked into the Impact Map, so that it can be mapped to the agreed business goals. This will weed out those great ideas that won’t contribute to the goals.

Assuming the “great new idea” is in fact going to have a significant impact then the roadmap wall forces the decision about where to put the “great new idea”. Does it go at the top, displacing all other initiatives? Or does it go at the end of the line? Any choice is acceptable, the import thing is that a choice is made.

Hugo is not breathing. Come home quick.

Thu, 31 Aug 2017 20:32:00 +1000

It was towards the end of Monday morning peak hour. I had just arrived at the train station on my way to work when my phone rang.

“Should I do CPR?

Hugo’s not breathing.

Come home quick.”

That was it verbatim. I did not get a word in. The caller was my wife. Hugo is our one year old child.

I took my wife’s suggestion and began to run home (I walk to the train station). My backpack full of computers was slowing me so I paused on my way paste a large tree, took my car keys out of the bag, and then tossed the rest as deep as I could under the tree’s folliage.

Thus unencumbered, my progress home was more rapid. I was terrified, thinking that Hugo was dead or dying.

I passed one block, then a second, before I saw my neighbour’s SUV screeching to a halt, on the wrong side of the road, across an intersection. I jumped in the passenger seat and barely had time to fail to close the door properly before we were off with a tyre howl and a petrol roar.

When I got inside Hugo was on my bed, lying on his side, moving in and out conciousness. His breathing was faint but detectable. His seizure had lasted five minutes and included two minutes without breathing. The ambulance had been called so we watched him and waited. After forty minutes I started to worry that the ambulance service had somehow lost track of us, so I called again. The lady I spoke to assured me that the ambulance would be there as soon as possible and asked me a silly question, “is he breathing?”. After forty minutes without breathing they could have told the ambulance to slow down.

One hour and ten minutes after we called, the ambulance arrived.

The paramedics were kind and helpful. They took my wife and child to the new Children’s Hospital. I followed in the car. The paramedics told me it would take a while to get processed through emergency so I stopped by the train station and retrieved my bag from under the tree.

We spent most of the day in the Emergency Department with a grumpy toddler. He alternated between sleep and extreme irritability. I am told that following his seizure he would have a headache and a metallic taste in his mouth.

Eventually we were released with a diagnosis of “febrile convulsion”, a condition associated with rapid change in fever temperature in young children. This was odd because Hugo did not have a temperature when he had his seizure.

The next day it was my turn. I was carrying Hugo because he would scream if I put him down when my daughter called me to the bathroom for some toilet emergency. I put Hugo down and noticed it was odd that he refused to stand on his feet. I lowered him into a sitting position and went to help my daughter. In the corner of my eye I saw Hugo gently lie down, extend his arms like a cartoon Egyptian mummy and begin to convulse. It was like this youtube video (or this one), but with his eyes rolled back so I only saw the whites. That lasted about a minute followed by a period of eerie stillness with Hugo completely still between breaths 20 seconds apart.

We went back to the hospital and again we were released after a few hours. It is easy to diagnose a seizure. It is much harder to diagnose the cause so Hugo will be having some tests when we can get him booked in. Here he is the day after his second seizure, starting to feel a little better, and demonstrating static electricity.

Komo Passwordless Authentication - Simplified Onboarding

Tue, 01 Aug 2017 20:32:00 +1000

Komo was already a secure and easy way to add passwordless authentication to web applications. However, it is now even easier.

The process of adding Komo passwordless authentication to a web applications has been reduced to two steps:

Add the form code to your application

The Komo dashboard shows the source code for the required login form. Simply copy and paste the form code into your application where you would like your login form to appear.
Implement a webhook

Your webhook endpoint is sent an authentication token. Post the token to https://komo.tech/verify. The response will look something like:
```
 {
     "isValid": true,
     "email": "john@adomain.com",
     "message": ""
 }
```
See the full API documentation.

If isValid is true, then you have verified that the current user has access to the email address in the response (john@adomain.com in the example above) and may login the user using your web framework.

See https://glitch.com/edit/#!/immense-sneeze?path=server.js:22:11 for a full working example.

(Note: It is also possible to directly verify the JWT token if you wish to avoid the network request to https://komo.tech/verify.)

Nginx Lets Encrypt SSL Reverse Proxy for Azure Web Apps

Sun, 23 Jul 2017 20:32:00 +1000

In 2017, deploying a new web site realistically means having a custom domain name and secure connection (SSL). Unfortunately for Microsoft Azure users custom domains and SSL are two of the segmentation triggers that push you out of the free tier and into higher priced plans.

Without a custom domain name or SSL I can host my application for free. Adding a custom domain name will cost $15/month extra for hosting (plus the cost of the domain and DNS). SSL support pushes me to $54/month for very similar hardware (1 CPU, 1.75GB RAM).

My Situation

I wish to deploy a .net application written in F#. At the time of writing deploying F# to linux is not a good option as I am not prepared to waste huge amounts of time. Therefore, the simplest hosting option is to use an Azure Web App. I can deploy by pushing to a git branch and the hosting will just work for a .net application.

To avoid the costs described above in configuring Azure for a custom domain and SSL I thought I’d try using an existing nginx server as a reverse proxy.

My Solution

DNS

Firstly, I purchased a domain name (komo.tech) and configured the DNS to point the domain name to a Digital Ocean Linux server running nginx.

nginx

Next, I added a new site to the nginx configuration. The site won’t actually host any local content it simply receives requests and forwards them on to the Azure Web App, thus, it is a reverse proxy.

In the nginx configuration I specified the underlying host that nginx will be proxying for.

upstream app_komo {                           
    server simpleauth1.azurewebsites.net:443; 
}

simpleauth1.azurewebsites.net is the domain name of the Azure Web App. 443 is the port to use because I want a secure connection. Later I will specify the protocol.

Next, I created a location, telling nginx to proxy all requests, from the root (/) to the previously declared upstream (the Azure Web App):

location / {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host simpleauth1.azurewebsites.net;
    proxy_set_header X-Forwarded-Proto $scheme;
    #proxy_set_header X-NginX-Proxy true;

    proxy_ssl_session_reuse off;
    proxy_pass https://app_komo/;
    proxy_redirect off;
}

At first my configuration did not work. I got an error from Azure about the site not existing. This was happening because the forwarded requests were arriving at Azure as requests for komo.tech, a domain that Azure knows nothing about. The solution was to add the line proxy_set_header Host simpleauth1.azurewebsites.net;. This means that when a request passes through the reverse proxy the host is changed from komo.tech to simpleauth1.azurewebsites.net, a domain that Azure knows how to handle.

https (SSL) with Let’s Encrypt

The final step was to secure the connection between user’s browsers and the reverse proxy with SSL. To do this I used the free service Let’s Encrypt. For my setup (nginx on Ubuntu 14.04) the steps were:

Install certbot with apt-get
Run certbot certbox --nginx and follow the prompts to specify which sites require certificates.

Add nginx configuration to redirect http traffic to https

 server {                                       
     listen 80;                                 
     server_name komo.tech;                     
     return 301 https://$host$request_uri;      
 }

Restart nginx

Conclusion

It is not the ultimate configuration by an means, but proxying via nginx is simple and effective. It also saved me some hosting cost.

Authentication for the modern web

Fri, 21 Jul 2017 20:32:00 +1000

April 1971, here at Xerox PARC, the Austrian-born computer scientist Hanz Scmitfilgerz connects to the latest in mainframe computer technology.

To establish his identity to the satisfaction of the computer he authenticates by providing a combination of: a username that identifies him, and a password that only he knows. His password is a unique, long, high-entropy string of characters known only to Dr Scmitfilgerz, and stored only in his memory.

That pleasent spring morning of 1971 is the last time that password authentication worked well for anyone.

Why passwords are not secure

Real users are not like Dr Scmitfilgerz. They can’t be in an age where the average person has dozens if not hundreds of online user accounts. It is beyond our capability to remember all those passwords so, people typically

reuse the same password or
store their passwords somewhere

Reusing passwords is perhaps the most significant password security problem today. At regular intervals a large password database is compromised and released on the internet (recent examples include linkedin, adobe, sony). Criminals then take those public credentials and attempt, often successfully, to access other services such as email or online banking.

Storing passwords can be OK, but it does introduce a massive vulnerability. If the mechanism used to store your passwords is compromised then all of your accounts are compromised. An incident like this is unlikely but catastrophic.

Why passwords are a bad user experience

IT professionals often solve the password proliferation and storage problem with an online product like Lastpass or 1Password. For the typicall consumer internet user these products are confusing and too hard to use. I will never be able to train my parents to use a password manager.

Assuming we do not reuse passwords (for the reason detailed above) then each time I want to access a new service I have the additional burden of coming up with a new password, and remebering it somehow. I have to fight my way through the services password complexity tests (your pasword must include lower case, uppercase, numeric characters and a gang sign).

It is too hard. Non-technical people don’t do it. They reuse the same awful password (like ‘password’) or variations of the same password.

Modern authentication (Authentication 2.0)

Recognizing the limitation of password-based authentication, some organisations like medium and Slack are abandoning passwords in favour of email-based authentication. Users are authenticated by proving they have access to an email inbox. In practice, the workflow is:

A user visits a site they wish to login to
They enter their email address
The site sends a special “magic link” to their email address
The user accesses their email, clicks on the “magic link” and is authenticated

This workflow is simple and fast. For most users it is vastly more secure than a password based system and easier to use. For website owners it results in more signups and less password related support.

Introducing Komo

A number of providers offer a hosted email authentication service that website owners can use to add passwordless authentication to their site. I’ve built another (Komo) that offers a slightly different service.

It is focused purely on passwordless authentication, without all the expensive add-ons that usually comes with Identity and Access Management (IAM) SaaS providers.
The core functionality of passwordless email authentication is, and always will be, completely free (with reasonable volume limits).
Komo is designed to integrate properly with your web framework’s authentication support. It is not an awkward JavaScript bolt-on.

Komo is available and ready to provide email authentication for your site. Just sign up. The Komo login process uses Komo for authentication (of course).

You can also try a a demo site that uses Komo for login, and browse the source to see how it is implemented.

Reactpress - React templates and universal / isomorphic rendering for Wordpress

Thu, 01 Jun 2017 20:32:00 +1000

https://github.com/liammclennan/Reactpress

What

ReactPress is a standalone server that pulls content from the wordpress API and renders views using React.js. The views are rendered and returned from the server and then the React components are re-attached in the browser. This provides the benefits of server rendering (speed and SEO) with the benefits of React in the browser (interactivity). It also allows Wordpress to be used for content creation without requiring Wordpress or php for rendering.

Why?

I’m really not sure. Because we can? Not because it is easy, but because it is hard?

Get Started

Create a wordpress instance, standalone or on wordpress.com.

Implement the wordpress template files as React components. E.g. to render a post create a file Single.tsx with content like:

import * as React from "react";
import * as ReactDOM from "react-dom";
import { Menu } from "./Menu";
import {Timer} from "./supporting/Timer";

export function render(getPost, getCategories) {
    return Promise.all([getPost(), getCategories()])
        .then(([post, categories]) => {
            return {
                component: Single,
                props: { post, categories }
            };
        });
}

export function Single({post, categories}) {
    return (<div>
        <Menu model={categories} />
        <Post model={post} />
        <Timer />
    </div>);
}

export function Post(props) {
  return (
    <div>
      <h1 dangerouslySetInnerHTML={{__html: props.model.title.rendered}}></h1>
      <p>{props.model.date}</p>
      <p dangerouslySetInnerHTML={{__html: props.model.excerpt.rendered}} />
      <div dangerouslySetInnerHTML={{__html: props.model.content.rendered}} />
    </div>);
}

This is regular React code with no restrictions, other than being able to run in the browser.

getPost and getCategories functions provide access to wordpress data. getPost returns data for the post identified by the posts slug, extracted from the url.

Making a Single Page Application

Sun, 05 Feb 2017 20:32:00 +1000

Should I?

Conventional wisdom holds that in 2017 all applications are best implemented as single page applications (SPAs). Having seen the transformation from server-side MVC to client-side single page applications the cynical part of me wonders if the preference for SPAs has more to do with developer career planning that technical merits.

Some hard truths. SPAs:

are more complicated
take longer to build
are much slower to render database content

So, for me, applications that primarily publish content and have simple user interaction are much better as server side MVC applications. Applications requiring rich user interaction need to be SPAs. Everything else is somewhere in the middle.

Get Started With A Starter Kit

There are many prebuilt application starter kits available for building SPAs. They typically solve the modern nightmare of configuring a JavaScript build process and maybe throw in a few other goodies.

For Angular there is:

the Angular Quickstart Seed. This an official Angular (Google) published quickstart repository.
Angular cli includes a way to setup new projects.

For React we have:

Mozilla Neo, a fully featured and opinionated application generator.
create-react-app the facebook idea of what a starter kit should be. Neat, but extremely minimal.

There are hundreds of others available. Unfortunately, evaluating them is more difficult than you might think. The relative strenghts and weaknesses of these systems often don’t present themself until a project is well underway, or when the developer decides that it is necessary to deviate in some way from the choices made by their chosen starter kit.

I like React, and I think there is a lot of benefit in choosing the “official” starter kit creted and maintained by Facebook and the React team. The trouble with create-react-app is that it includes the absolute minimum required to make React work, which is not enough to build an application. For example, there is no routing, validation or application state management included.

The best of both worlds then is to chose a minimal starter kit, upgrade it to include your favourite tools and match your preferences, then publish that as your own custom starter kit. This lets consumers use your starter kit as a single dependency in their applications, just as create-react-app does. It also means that starter kits are composable, and someone could further customise your starter kit into their own starter kit.

What I Did To create-react-app

I used create-react-app as the basis for my starter kit. Then proceeded to add react-router for routing, bootstrap for styles and redux for state management.

Adding React router

The easy and obvious start is to install react-router:

npm install react-router --save

The fundamental abstraction of a web application is the resource (or page) identified by a URL. This is why routing is so important. Each application state should be addressable by an unique URL. If you don’t have this then your application fails to meet the expectations of the web, such as a working back button, pages that can be bookmarked etc. To capture this I like to create page components that export everything required to hook a new page into the application, including a route string and a factory for creating the React component that represents the page. For example, the component for an about page might be:

let AboutPage = {

    // function that returns the React component for the page
    pageFactory: aboutPageFactory,

    // the route to match to this page
    route: 'about'
};
export default AboutPage;

When rendering the application (this happens in index.js for create-react-app) we now have everything we need to connect routes to pages.

ReactDOM.render(
    <Router history={browserHistory} >
        <Route path="/" component={Layout}>
            <IndexRoute component={Home}/>
            {[aboutPage, loginPage].map(page => 
                <Route key={page.route} path={page.route} component={page.pageFactory(store, page.sideEffects)} />
            )}    
        </Route>
    </Router>,
    document.getElementById('root')

For the about page, this means that the route /about will cause the about page component (returned from aboutPageFactory) to render within the Layout component.

On Developer Testing

Sat, 28 Jan 2017 20:32:00 +1000

On Developer Testing

Why

A developer’s responsibility is to create software that works.

If it has not been tested it has not been shown to work.

As part of the planning for implementing a new feature the developer should think about and plan how the feature will be tested, and have agreement from the product owner (or appropriate stakeholder). This is sometimes called documenting acceptance criteria.

More information on developer testing.

Test Design Prior to Implementation

Acceptance criteria may be recorded in the classic BDD syntax:

GIVEN some preconditions
WHEN an action happens
THEN a result occurs

For example,

GIVEN an authenticated user
WHEN they submit the enquiry form
THEN the contents of their enquiry is sent to info@acme.com
AND a copy is stored in the database

Repeat until all scenarios have been described. This process ensures: that developers and their stakeholders share an understanding of what is to be built, that the developer has thought through the alternatives that must be accounted for in the implementation.

Test Implementation

Theoretically, tests should exist for all enumerated scenarios. In practice some may be difficult to implement or may have insufficient value to justify their existence. Having tests for all scenarios provides a convenient scaffold for the developer. They begin with a complete set of unimplemented tests, make progress by implementing the scenarios, and have confidence that the implementation task is complete when all scenarios have passing tests.

Good tests should only fail because the code they test is wrong. Therefore, most tests should be ‘unit’ tests. Integration tests can fail for many reasons so they are undesirable. However, the value of a small number of integration tests is extremely high, as they show that the system functions together as a unit. Start with a small number of end-to-end integration tests, then favour small, focused unit tests.

Code for Testing

Write code that is easy to test. The easiest code to test is code that transforms an input to an output, e.g. a function. Separate computation from side effects, and test the computation (testing side effects pushes you into an integration test). Although this is counterintuitive it leaves more testable and maintainable code that is also more reusable (via composition). It is a happy coincidence that good code and testabe code happen to be the same thing.

Here is an example of how computation and side effects are commonly mixed:

if (new Date().getHours() > 11) {
    console.log("Good afternoon");
} else {
    console.log("Good morning");
}

Accessing the current date via new Date() is a side effect that introduces a hidden variable that a test cannot control. Producing a result via console.log similarly prevents a test from evaluating behaviour. The computation here is choosing a greeting based on the time of day, so extract that and test it.

A better version is:

// this is a testable function
function chooseGreeting(hourOfDay) {
  let isAfternoon = hourOfDay > 11;
  return isAfternoon 
    ? "Good afternoon"
    : "Good morning";    
}

// group the side effects together
console.log(chooseGreeting(new Date().getHours()));

Test Execution

The goal is to know as early as possible when a mistake has been made. This is one reason why a good type system is surperior to any testing. I like to say,

every test is an admission of failure.

What I mean is that any other technique for preventing errors is surperior to testing. This could be a static type checker, design-by-contract, transactions or some particular design. When those techniques have been exhausted then we must fallback to testing.

The strength of testing is its flexibility to detect any type of errors (not just type errors). The weakness of testing is its inability to give any information about anything other than the exact scenario tested (hence the power of property testing).

To receive the test results as early as possible it is helpful to have a system that runs the tests when changes are detected that may change the test results. Since tests are primarily a function of source code this is usually implemented via a file system watcher that detects changes to the tests or the code that they test, and reruns the tests.

Each developer should check the tests, and not commit code that causes a test to fail, however, human discipline is flawed and we should only rely upon it when there is no alternative. In the case of testing the usual solution is to make test execution part of the continuous integration build process triggered by every code commit to the shared main branch, which should happen at least once every day. This guarantees that all deployment candidates have passed all tests.

New year, new job

Wed, 04 Jan 2017 20:32:00 +1000

Next week, two months shy of five years, I will complete my employment with Readify. I stayed so long, far longer than I ever thought I would, because I continued to learn and it continued to be fun. Over this time I have worked with many amazing individuals, who taught me everything I know. I have tried to repay the debt by passing on what I learned to others, and I know that I am leaving behind a highly capable team.

When I first joined I worked on some incredible teams helping to deliver solutions for customers. When I needed more challenge there was an opportunity to take responsibility for delivery and to lead teams of Readify developers. Eventually I took on the Principal Consultant role and its responsibility for deliverying a portfolio of solutions across a number of customers. Most recently I joined the Queensland management team because I saw an opportunity to create the best place for software developers to work in Queensland. I have enjoyed being in an environment with an unusual tendency towards common sense that rewards hard work and success.

You Should Try Consulting, at Least Once

I did not enjoy my first consulting experience. I was young and felt the frustration of knowing better than everyone else. ;) I wanted to get things done and didn’t feel that my customers were aligned. So I went and worked on products for half a decade.

Then I joined Readify, and re-entered the world of consulting, with a better attitude and a better approach. The job of the consultant is to help customers to reach their goals, often in spite of themselves. It involves understanding, empathy, education and even a little manipulation (the good kind). The consulting skillset is invaluable in all aspects of life that involves getting things done.

The consulting life involves a huge variety of domains, technology, personalities and environments, often over a relatively brief period of time. This accelerates the learning of consultants who chose to embrace the opportunity.

Unless you aim to be a developer who hides in the dark and never talks to anyone, consulting is a wonderful opportunity to learn new skills and become more valuable. I recommend that all software developers give it a go.

What’s Next?

I will be CTO at Starts at 60, the leading online community and digital media provider for baby boomers in Australia and New Zealand. So far the business has focused on media and discussion but we are now moving into other areas, like travel.

The company is established and rapidly growing but still feels very startup, with a lot of experimentation and expansion happening. I look forward to mixing my experience into this exciting and talented group to create something amazing.

Redux in React and Angular 2

Wed, 21 Dec 2016 20:32:00 +1000

Earlier this month I presented at dddbrisbane about simplifying UI programming and I included the simplest React + Redux application I could come up with.

The simplest React + Redux application I could come up with

It displays a random number. Clicking the text generates a new random number. Toggle the Result button to see the code more easily.

See the Pen FknRooowwwmmm by Liam McLennan (@liammclennan) on CodePen.

It uses the React function component syntax to define the component as a function from UI state (props) to HTML (a H1 tag).

function HelloNumber(props) {
  function handleClick(e) {
    store.dispatch({type:'MAKE_A_NEW_NUMBER'});
  }
  return <h1 onClick={handleClick}>A random number {props.number}</h1>;
}

The onClick handler dispatches a MAKE_A_NEW_NUMBER action to the redux store.

The UI state for the application is an object with a property number:

{ number: Math.random() }

The redux reducer (function responsibile for applying actions to the state to make a new state) handles the MAKE_A_NEW_NUMBER number by setting it to a new random number.

let store = Redux.createStore(function (state = { number: Math.random() }, action) {
  switch (action.type) {
    case 'MAKE_A_NEW_NUMBER':
      return Object.assign({},state, {number:Math.random()});
    default: return state;
  }
});

Now dispatching an application has updated the state, but we still need to have state changes trigger the UI to update, which is done via the subscribe method:

store.subscribe(()=>{
  ReactDOM.render(
    <HelloNumber number={store.getState().number}  />, 
    document.getElementById('container')
  );
});

and finally prime the whole thing by an initial dispatch

store.dispatch({type:'MAKE_A_NEW_NUMBER'});

This is not strictly necessary but it is an easy way to trigger the subscribe callback and cause the initial render.

The simplest Angular 2 + Redux application I could come up with

What would the equivalent Angular 2 application look like?

I couldn’t find a simple way to develop an Angular 2 application in Codepen or similar so instead I worked locally, starting with the Angular 2 Quickstart.

To reproduce the random number display functionality in Angular 2 I found that I needed a component hierarchy (parent + child).

The child component contains the header and dispatches a NEW_RANDOM_NUMBER action when clicked, just like the React component. Two values are passed into the child component, from the parent, via @Input() decorators: the application state (appState) and the redux store dispatch function (dispatch).

import { Component,Input } from '@angular/core';

@Component({
  selector: 'my-child',
  template: `<h1 (click)="onClickMe()">A random number {{appState.number}}</h1>`,
})
export class ChildComponent  { 
    @Input() appState:any;
    @Input() dispatch: (action:any) => void;

    onClickMe() {
        this.dispatch({type:'NEW_RANDOM_NUMBER'});
    }
 }

The parent component (AppComponent) creates the redux store, using the same reducer as the React version. The UI state is copied to a local field (appState) and updated whenever the store’s state changes. The appState and redux dispatch function are bound to the child component (<my-child/>).

import { Component,Input } from '@angular/core';
import { ChildComponent } from './child.component';
import { createStore } from 'redux'

interface HasANumber {
  number: number;
}
interface Action {
  type: string;
}

@Component({
  selector: 'my-app',
  template: `<my-child [appState]="appState" [dispatch]="store.dispatch"></my-child>`
})
export class AppComponent  { 
  store = createStore<HasANumber>(function (state = {number:Math.random()}, action:Action) {
    switch (action.type) {
      case "NEW_RANDOM_NUMBER": 
        return {number:Math.random()};
      default: return state;
    }
  });
  appState = this.store.getState();

  constructor() {
    this.store.subscribe(()=> {
      this.appState = this.store.getState();      
    });
  }
}

Final Thoughts

Having Typescript easily available in Angular 2 was nice. Further, I found that Angular 2 is at its best when imitating my very strict React style. Both frameworks work well with Redux and the uni-directional data flow idea.

Document Storage in Sql Server 2016 (JSON) and Postresql

Sat, 12 Nov 2016 20:32:00 +1000

Why?

Some problems map well to documents, or resources. When your data is a set of relatively independent collections then document storage will be much easier than relational. Anywhere that REST makes sense for resources documents make sense for data storage. Jeremy Miller wrote a bit about why workring with documents can be efficient in the context of Marten, the .net document database and event store library for Postresql. If you are not ready to embrace Marten it is possible to start working with documents, in Sql Server or Postresql, easily.

SqlDoc

SqlDoc is sort of like Marten, but far less ambitious. It just does document data, in fact it is currently limited to these operations:

run an arbitrary script
create a table to store documents
insert a document
update a document
delete a document
retrieve a document
query for a documents using database indexes

It is a limited feature set but sufficient for a large class of applications.

I am writing about SqlDoc again because I have recently added support for storing documents as JSON in Sql Server 2016, bringing the storage options to:

postgresql 9.3 or greater
Sql Server 2005 or greater (XML)
Sql Server 2016 or greater (JSON)

Storing Document Data in Sql Server 2016 as JSON

The first step is to create a table to store documents. SqlDoc expects a table per type stored. To store a type called Person:

CREATE TABLE [dbo].[Person](
    [Id] [uniqueidentifier] NOT NULL,
    [Data] [nvarchar](max) NOT NULL,
 CONSTRAINT [PK_person] PRIMARY KEY CLUSTERED 
(
    [Id] ASC
)

What the Person type is does not matter, because it will be serialized and stored as text in the [Data] column of the Person table. For this example let’s use:

public class Person 
{
    public Guid _id { get; set; }
    public string Name { get; set; }
    public int Age { get; set; }
    public string[] FavouriteThings { get; set; }
}

To work with our document database we create an IDocumentSession from a Sql Server connection string, like this:

var _documentSession =
            new DocumentSession<Guid>(SqlConnection.From(
                ConfigurationManager.AppSettings["ConnSqlXml"]))

IDocumentSession contains the obvious thing, and is defined as:

public interface IQuerySession<TKey>
{
    TVal Load<TVal>(TKey id) where TVal : class;
    IEnumerable<TVal> Query<TVal>(string sql, Dictionary<string, object> parameters = null);
}
public interface IDocumentSession<TKey> : IQuerySession<TKey>
{
    void Delete(TKey id, object datum);
    void SaveChanges();
    void Store<TVal>(TKey id, TVal entity) where TVal : class;
    void Update<TVal>(TKey id, TVal entity) where TVal : class;
}

TKey is the type of the field used as a primary key. Guid is a good choice. So with an IDocumentSession we can insert, update, delete, load and query documents. An insert is done like this:

var _aDocument = new PersonCs
{
    _id = Guid.NewGuid(),
    Name = "Docsesh",
    Age = 90,
    FavouriteThings = new[] { "Golf", "Statue of liberty" }
};
_documentSession.Store(_aDocument._id, _aDocument);
_documentSession.SaveChanges();

Of course we can enqueue many operations before calling SaveChanges() to persist the changes in a transaction. The Person table will now contain the following data:

Id	Data
7BC6192F-1B78-4C31-9537-088EA4205871	{“_id”:”7bc6192f-1b78-4c31-9537-088ea4205871”,”Age”:90,”Name”:”Docsesh”,”FavouriteThings”: [“Golf”,”Statue of liberty”]}

SqlDoc serialized the Person object to JSON, then inserted it into a table with the same name as the type (Person).

Updates are hopefully obvious:

_aDocument.Age += 1;
_documentSession.Update(_aDocument._id, _aDocument);
_documentSession.SaveChanges();

Now the database has:

Id	Data
7BC6192F-1B78-4C31-9537-088EA4205871	{“_id”:”7bc6192f-1b78-4c31-9537-088ea4205871”,”Age”:91,”Name”:”Docsesh”,”FavouriteThings”: [“Golf”,”Statue of liberty”]}

To delete a document:

_documentSession.Delete(_aDocument._id, _aDocument);
_documentSession.SaveChanges();

To load a document by its primary key:

var fresh = _documentSession.Load<Person>(_aDocument._id);

Note that this relies on the database primary key column being Id.

Querying on Indexes

Querying without an index means a table scan, which is too slow for real use. MSDN describes the recommended strategy for indexing JSON data, essential define a computed column for the value to be indexed and index the computed column.

For our person example, we might wish to query by Name, so we start by adding a computed column:

ALTER TABLE person
ADD Name AS JSON_VALUE([Data], '$.name')

The Sql Server table now appears as follows:

Id	Data	Name
7BC6192F-1B78-4C31-9537-088EA4205871	{“_id”:”7bc6192f-1b78-4c31-9537-088ea4205871”,”Age”:91,”Name”:”Docsesh”,”FavouriteThings”: [“Golf”,”Statue of liberty”]}	Docsesh

And we have a Name column that can be indexed for querying according to the usual practice in Sql Server.

CREATE NONCLUSTERED INDEX IX_Person_Name
ON Person ([Name])

Finally, we can query our index to search by name:

var result = _documentSession.Query<PersonCs>(
    "SELECT [Data] from person where [Name] = @name",
    new Dictionary<string, object> { { "name", _aDocument.Name } });

Alternative API

Most people seem to like the IDocumentSession API (inspired by Marten and RavenDB) for working with document storage. There is an alternative API based on accumulating operations in a queue, and then commiting those operations.

var connection = SqlConnection.From(ConfigurationManager.AppSettings["ConnSqlXml"]);
var _uow = new Queue<Operation<Guid>>();
_uow.Enqueue(Operation.Insert(_aDocument._id, _aDocument));
UnitOfWork.Commit(connection, _uow);

var resultst = QueryFor<Person>.For(
    connection,
    "SELECT [Data] from person where [Name] = @name",
    new Dictionary<string, object> { { "name", _aDocument.Name } });

Both APIs accomplish the same goal. The difference may come down to how much you like dependency injection.

Introducing Children to Charity

Tue, 13 Sep 2016 20:32:00 +1000

My favourite podcast at the moment is Sam Harris’ Waking Up. Recently Sam interviewed William MacAskill about the effective altuism movement. Effective altruism “applies evidence and reason to determining the most effective ways to improve the world”. An example is the organisation givewell.org that researches and ranks charities by the positive effect that would be achieved by increasing their funding. They currently identify 4 top charities based on “the value of filling their remaining funding gaps, which derives from the overall quality and cost-effectiveness of the organization’s work, as well as what additional funding will enable the organization to do”.

Sam and William gave an example of the moral culpability of 18th century French nobility, who protected their wealth while the people starved. The contention was that this is no different to the 1st world’s neglect of our poorer cousins. There was also an interesting discussion about an idea attributed to Peter Singer, that to let people suffer and die in other countries whom we could have helped (as we all do, every day) is morally equivalent to seeing a person drowning and doing nothing. All of this reminded me of my good fortune and responsibility to help other people how I can.

Children and Charity

I have three children, currently aged 4, 2 and < 1. I want to teach them that all lives are equally valuable, and that the way to their own happiness paradoxically requires them to balance their own needs (toys) with the needs of other people. To this end I decided to involve them in an effective altruism project. I selected the top two charities from givewell.org, added on of my other favourites (Ronald McDonald House) and printed representative images on a sheet of paper.

For Ronald McDonald House I used a photo of a sick child with a family symbol adjacent, so that I could explain to my children that Ronald McDonald House helps sick children by making it possible for their families to stay with them when they are in hospital. For the Against Malaria Foundation I used a photo of a girl with Malaria and an graphic of a net so that I could explain that in some countries mosquitos make people sick and that we can help by providing bed nets to stop the mosquitos biting people when they sleep (approx. $5 / net). For the Schistosomiasis Control Initiative (SCI) I used a photo of two girls infected by parasitic worms, with their abdomens clearly deformed, and a graphic of the worm. I explained that the worms make people sick and that we can help by providing medicine that gets rid of the worms (approx. $0.50 / treatment).

Here is the graphic that I showed to my two eldest kids:

I gave each person in my family two coins, representing the contribution we can make to the three charities and asked them to place their coins where they would like their contribution to go. Both children placed coins on the girls with parasitic worms. One also chose to support Ronald McDonald House. Once my wife and I added our choices there was a fairly even distribution across all three charities and we had all had the opportunity to actively participate in the process.

I have now made payments to each charity, according to the ratios determined by my family and will keep them informed of the results.

I believe the children empathised with the groups we discussed and understood that helping them consumes resources that we will no longer have for ourselves. My eldest child was initially alarmed because he assumed it was an all or nothing transfer. He asked me, “how will we buy things from the shop?”. When I explained that we will have enough left to be able to continue to buy things from the shop he was satisfied.

Our donation to Against Malaria has been allocated to an early 2017 distribution in Uganda. I will show the kids where Uganda is on their map and also let them know when their nets have been distributed to families.

Overall I think the process was successful and I hope to do it again in a few months time.

The Timeline Backlog

Fri, 02 Sep 2016 20:32:00 +1000

An agile product backlog is a prioritized list of uncompleted work. It can include features, bugs and other tasks. Good backlog items are as small as possible why still being independently valuable. They also have an associated effort estimate that helps the product owner to decide if and when the work should be completed.

Prioritizing the backlog means sorting the items from highest priority to lowest priority. Priority is generally a combination of value to effort ratio and risk. That is, we should work first on the items that have the best “bang for buck” and the highest risk. High risk items come first so that the risk can be dealt with. Leaving all the risky items until the end of a project would make failure likely.

What is good about the Backlog?

The product backlog is a central artifact for an agile project around which work is organised. It provides a basis for conversations with the product owner and subject matter experts. It guides the work of the team and enables project delivery to be projected into the future.

What is missing from the Backlog?

The backlog doesn’t communicate anything about time, and doesn’t visualize relative effort. In short, the backlog is missing the kind of information that we might get from a timeline - so let’s add one.

Consider the following fictitious backlog for a file backup application:

Story	Effort (days)
In order to not lose data as a computer user I want to be able to save backups of selected files	6
In order to not lose data as a computer user I want to have a configured portion of a filesystem automatically backed up continuously	12
In order to recover from a data failure as a computer user I want to be able to retrieve files that were backed up	10
In order to save money as a computer user I want old copies of backed up files to be purged	5

Further suppose that we have a cost of $2000 / day and therefore a total cost estimate of $66,000. We’d like to see how that time and cost compares to our backlog. How much can I get for half that? Roughly when might a particular story be delivered? To answer these questions we can superimpose the backlog onto a timeline.

Visualizing a backlog over time

Each story is shown as a row of a table, with the height scaled proportional to the effort for that story. If one story requires twice as much effort as another then its row is twice as high. A column to the right of the table shows how the budget is consumed over time, with label markers at 25%, 50% and 75% and 100%. This is enough detail to answer the timing questions listed above, but not so much detail that the product owner might infer that estimates have a higher precision than they really do.

The resulting visualization is as follows:

Interested parties can look at the timeline backlog and understand approximately when features will appear over time. Changes to the backlog are immediately understood as changes to the schedule.

Blackstar CMS Documentation

Tue, 30 Aug 2016 20:32:00 +1000

Blackstar CMS now has a proper documentation site.

It is a work in progress, however, the .net client, JavaScript client, security and API/Keys are all populated.

Seeking Beta Partners for Blackstar CMS

Sun, 28 Aug 2016 20:32:00 +1000

Development of Blackstar CMS has progressed well and we now have a product that is ready for early users. Recently, we have added a backup system, secure connections via SSL/TLS and security features for authentication and authorization.

Beta partners will receive a free 12-month license and personal assistance integrating Blackstar CMS into their application.

If you have a application that will benefit from content management please reply to this email and request access to the beta program.

If you would like to trial Blackstar CMS outside of the beta program you can download the latest builds for Windows and Linux.

Email to participate.

The Goal Tree

Wed, 24 Aug 2016 20:32:00 +1000

A good software development team is faster than a not-so-good one. Maybe twice as fast, but there is a definite limit. The real productivity difference that experienced developers learn is to minimize waste. It’s all about what you don’t build.

Assuming that the customer knows what they want and simply doing what you are told is the surest way to fail.

Truly productive software development teams optimize their customer’s path to their goal.

To do that you have to understand what the goal is, and how to get there.

The Goal Tree

The goal tree is a simple exercise that can help take a customer from a specific stated requirement, to a more general and valuable goal, then back to the correct specific requirement. This is done to:

Establish that the stated requirement, and the real underlying goal (business goal) are in fact linked.
To validate that the stated requirement is the correct one to pursue, or identify a better requirement.

Proceedure

Start with the stated requirement:

“we would like you to enhance our mobile application to send our customer’s customer’s push notifications when certain events happen”

Recursively ask, “yeah, but why?”

and draw the hierarchy of goals.

“Our competitors have the feature and we want parity”

“Yeah, but why?”

“From a traditional monopoly we are facing many emerging competitors and they are stealing market share. We want to stop that.”

“Yeah, but why?”

“Our business is software sales into a fixed market. Market share == profit, therefore we must maintain and grow market share”

“Yeah, but why?”

“Our MD would like a bigger boat”

At some point it always comes back to a boat for someone. Stop there. Search down the list of goals until you find the most useful strategic goal. It is not the bigger boat for MD goal because that one is too likely to lead to a life of crime. In this example the most useful goal is:

“From a traditional monopoly we are facing many emerging competitors and they are stealing market share. We want to stop that.”

Delete all the others.

Perform an impact mapping to derive requirements from the identified goal. Compare it to the original starting position.

The Impact mapping

Impact mapping is a technique designed to help us deliver business goals, not just software features. According to the website:

“impact maps allow us to maintain a dynamic roadmap that changes with our learning, keeping the goal in mind and making features and scope secondary to it. We visualise assumptions, which allows us to change direction once these assumptions are proven or discarded”

Goals

Start with the selected goal. I have reframed it slightly.

Of course we should make our goals SMART.

Actors

Next, we identify the actors who can influence (positively or negatively) the outcome.

Impacts

For each actor list the impacts that we are trying to create. How can that actor help us achieve our goal? How do we want their behaviour to change?

Deliverables

Now we are finally approaching the level of software, where we started.

Deliverables are the things we can do to support each impact. They often involve a hefty assumption that should be tracked and revisited. Does this deliverable really support its impact?

The implication is that if we deliver the features listed at the bottom, then we will get the impacts identified for the actors, which leads to achieving the goal.

Note that the original requirement does not appear in the impact map, suggesting that there may not be a clear chain of reasoning back to the actual goal. The three deliverables identified in the impact map may represent more valuable work items.

The goal tree is one technique that can help high performance development teams to reduce waste and creates more value in less time.

Why I Like Mathematics

Tue, 09 Aug 2016 20:32:00 +1000

The reason that I like and enjoy mathematics is epistemological. Epistemology is the branch of philosophy that deals with knowledge, including ideas of truth, belief and justification.

I like mathematics because it is the only science that can ever prove a thing (proposition).

Mathematics is able to make proofs because it deals with abstract ideas that we define completely - thus proof is possible.

As a counter example, consider Kepler’s laws of planetary motion. Kepler reconcilled planetary observations with Isaac Newton’s laws of motion and law of universal gravitation to provide a model for the motion of planets around the sun. It tells us that orbits are elliptical and that planets move faster when they are closer to the sun.

https://en.wikipedia.org/wiki/User:Gonfer

It is the law that tells us that the sun will continue to rise each day. It is as close to an absolute truth that physics (the next best science) can get. But planets, and gravity, are not artificial abstractions that we created and thus we can never be sure that we have the complete picture.

Kepler’s laws of planetary motion start to fail when we look at the orbit of Mercury. Mercury is close enough to the sun that the effect of general relativity (due to the mass of, and proximity to, the sun) cause variation from what is predicted by Kepler’s laws. Kepler and Newton felt, as we all do, that they existed in a regular, three dimensional space. Little did they know that the universe is four dimensional and even the orbit of planets is an illusion produced when straight motion in a four dimensional space is projected into three dimensions (or something).

We don’t KNOW anything - except what we know via mathematics.

Blackstar CMS Security

Fri, 05 Aug 2016 20:32:00 +1000

Previously, I demonstrated how to configure Blackstar CMS to use SSL/TLS to secure connections. This post will describe the other security features of Blackstar CMS, principally authentication and authorization.

Security Settings

The admin application has settings for controlling access to Blackstar CMS.

Access is restricted in two categories: editing and reading. For each category you can specify who has permission: everyone, all authenticated users, or a specific set of authenticated users.

The above configuration restricts access to the admin application, and all editing functions, to the users Liam and Anne-Marie. Reading content via the API is open to everyone and does not require authentication. Many scenarios will use a similar setting - editing is restricted, reading is not.

Where authentication is required, clients require an API key. The admin application has an API / Keys page where API keys can be created.

Keys are given a name to help you remember what they are for. You should create one key per channel that accesses content, for example, one for your web application and one for your mobile application. Each key provides complete access to everything so they must be kept secret.

API Security

The API is Blackstar’s most important interface, so let’s start there. When connecting to the Blackstar API client’s may need to authenticate. Authentication is the process of establishing a trusted identity and is achieved using JSON Web Tokens (JWT). A JWT is a cryptographically signed authentication claim. The client signs a claim using a secret API key. The server validates the token with the same API key. If the token validates then the server knows that the client has the same key and the claim can be trusted.

Once the client has created a JWT it is then included with every request, in the Authorization header, like so:

Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYW1lIjoiTGlhbSIsImlhdCI6MTQ3MDM3MTQ1MCwiZXhwIjoxNDcwNDU3ODUwfQ.Nl2k32tcAHWhxDURL_fN08bgbrxfA9CSXWeA0nDPwso

Every request received by the server will check the authorization header and attempt to validate the token. The token includes the current user’s identifier, which can then be used for authorization.

When using the Blackstar JavaScript client this is all taken care of for you. The only requirement is to supply a JWT when instantiating a new client, like this:

var blackstar = new Blackstar.Client('https://demo.blackstarcms.net', { 
  showEditControls: false,
  token: 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYW1lIjoiTGlhbSIsImlhdCI6MTQ3MDM3MTQ1MCwiZXhwIjoxNDcwNDU3ODUwfQ.Nl2k32tcAHWhxDURL_fN08bgbrxfA9CSXWeA0nDPwso',
  authCallback: function (response) {
    // do something if the server return 401 Unauthorized
  }
});

To create a token in your application first acquire an API key from the Blackstar CMS admin application, then build a JWT token with a name value containing the current user’s user name and an optional expiresIn value. For example:

jwt.sign({ name: "liam" }, "BatrVb7cIh7QYA218iqM", {
    expiresIn: 3600 * 24
});

(note: "BatrVb7cIh7QYA218iqM" is the API key)

Admin Application Security

To authenticate with the admin application simply append ?t=<JWT token> to the URL. If the user named in the token has the editing permission then you will get access. This is the recommended way to access the admin application, however, if you have not yet setup your application to be able to produce the JWT token, then you can authenticate via the login page:

The use of the login page is not recommended because it requires knowledge of an API key, and they need to be kept secret. The login page simply creates a JWT token (as above) and redirects to <url>?t=<JWT token>.

Conclusion

Blackstar CMS’s security features are designed to be simple, yet flexible. Please provide your feedback to liam at blackstarcms.net.

Using TLS/SSL with Blackstar CMS

Mon, 04 Jul 2016 20:32:00 +1000

There are two primary architectures for using an API-first CMS like Blackstar CMS.

Modern web applications built with Angular.js, React, Ember etc will tend to load their application data from their application server, and their content data directly from Blackstar CMS. This means that the end user’s browser connects directly to Blackstar CMS.
Server rendered applications using Asp.net MVC, Rails, Django etc will load content from Blackstar CMS during the process of rendering the page. In this scenario Blackstar CMS is only accessed by the web server and can be deployed within a private network.

In both cases, and definitely the first, it may be advantageous to secure the connection to Blackstar CMS. Fortunately, the internet has a robust infrastructure for solving this problem. TLS/SSL provides communication security by encrypting the communication, authenticating the identity of the server, and guaranteeing the integrity of the message.

Securing Connections to Blackstar CMS

To secure a connection you will need to obtain a private key and a certificate. There are several ways to do this:

Create a self-signed certificate. This will encrypt the connection but will fail to authenticate the identity of the server.
Obtain a free certificate from Let’s Encrypt.
Purchase a certificate from a certifying authority.

Creating a Self-signed Certificate

For testing purposes a self-signed certificate may suffice. Do not attempt to use a self-signed certificate for production use.

To create a self-signed certificate and private key:

Install OpenSSL (if you don’t have it already)

Use the following command

openssl req \
    -newkey rsa:2048 -nodes -keyout domain.key \
    -x509 -days 365 -out domain.crt

This certificate/key pair will be valid for 365 days.

Configure Blackstar CMS to use TLS/SSL

To configure Blackstar CMS to use a TLS secure connection you must firstly obtain a private key and certificate (see above).

Next, edit your Blackstar CMS configuration file (config.json) and set values for the pathToKey and pathToCert options.

{
  "pathToKey": "/home/fred/domain.key",
  "pathToCert": "/home/fred/domain.cert"
}

Then restart Blackstar CMS for the changes to take effect.

When accessing Blackstar CMS you will now need to change the protocol from http to https.

If you used a self-signed certificate you will get an error because the certificate is not signed by a Certificate Authority (CA).

For testing purposes you can accept the error and proceed to access Blackstar CMS via an encrypted connection. For production, use Let’s Encrypt or a Certificate Authority to obtain a trusted certificate.

Blackstar CMS Backup

Sun, 03 Jul 2016 20:32:00 +1000

One of the most requested features for Blackstar CMS has been backup. Blackstar is an operational system so backup is important to maximise uptime and minimize data loss.

Strategy

Blackstar stores all its data in a Sqlite database specifically to facilitate a simple backup process. We can get a complete backup simply by backing up the database.

Of course you can manually backup the database yourself, but Blackstar CMS will do it for you automatically.

Blackstar’s backup process is:

on a schedule (every 10 minutes by default) the database is exclusively locked for the backup process
if the databse has changed since the last backup then
- copy the file to a backup location
release the lock so the application can continue running

It is necessary to lock the database so that the copy doesn’t cause corruption. On a test database (1MB) the backup requires the database to be locked for 6ms, so the performance degredation is manageable.

Implementation in Node.js

The single threaded nature of node.js helps to make these sort of concurrent processes easy. When Blackstar starts it registers a callback on the backup frequency:

setInterval(function () {
    // do backup
}, 10 * 60 * 1000 /* 10 minutes */);

To ensure a consistent backup the database needs to be locked for writes (reads are still OK). We can lock a sqlite database for writes using the BEGIN IMMEDIATE command. If an error occurs while the database is locked we still want to ensure the lock is released, so the process is wrapped in try {} finally {}.

try {
    db.run("BEGIN IMMEDIATE");
    // do things with an exclusive write lock
} finally {
    db.run("COMMIT");
}

The whole thing is done synchronously, since we are blocking the server anyway. The file hash is calculated using node’s crypto module, which delegates to OpenSSL.

export function calculateHash(data:Buffer) : string {
    const hash = Crypto.createHash('sha256');
    hash.update(data);
    return hash.digest('hex');
}

If the hash of the current database is different since the last time the backup function ran then the database is copied to the backup destination.

Summary

The backup solution described above will be sufficient for most scenarios. For anything else it is easy enough to build your own backup process using a variation on the same technique (lock the database, take a copy, unlock the database). The sqlite online backup API provides some options.

Blackstar CMS (Alpha) Installers Available

Fri, 24 Jun 2016 20:32:00 +1000

The Blackstar CMS now has a usable feature set. You can:

Create and edit content, with a fancy new markdown editor
Tag and search for content
Work with media assets (non-image media still needs work)
Pull content using the JavaScript client
Pull content using the .net client
Install Blackstar CMS as a service on Windows or Linux (more on this later)

Things that Blackstar CMS still cannot do, but will soon:

Be reliably used in production
Provide authentication and authorization for content access and administration
Provide built-in SSL/TLS support

Install Blackstar CMS as a service on Windows or Linux

Installation packages are now available for Windows and Linux. To receive access to the installation packages, along with install instructions, join the Blackstar CMS mailing list.

If you are already on the list you will receive a link to the download soon.

Blackstar CMS Progress

Thu, 09 Jun 2016 20:32:00 +1000

I announced Blackstar CMS 6 weeks ago. Since then the mailing list has grown steadily (join here) and the product has moved closer to release.

Many things have changed in the last six weeks (try the demo, here are some highlights:

rich text editing
media library
sensible caching

Rich Text Editing

Blackstar CMS now uses ProseMirror for content editing. It is a simple, but pleasant, editing control. The intention is that basic editing can be done via the WYSIWIG editor, for anything more complicated it is better to work with the raw HTML directly.

Media Library

A CMS is not that useful without support for images and other media. The Media Library is Blackstar CMS’s central home for all media. Files can be uploaded by dragging onto the upload button, or by clicking the button and selecting one or more files from the file browser.

The media library has a search field for finding assets in a large library.

When editing content we can add media by launching the media library in a dialog:

Being able to nest user interface components like this (this is the media library inside of the content editing component) is a fun design challenge for a rich web application.

Sensible Caching

Caching is an interesting challenge for a CMS. We want content cached, but we also want to always see the latest content. Ultimately, Blackstar CMS will allow caching to be configured for content. At the moment I have set the cache to a fixed time period (24 hours). Within that time period users will only download content once. Subsequent requests will use the cached copy without ever making a request to the Blackstar CMS server. Administrators can force a cache refresh by using the browser refresh feature (F5).

Caching of media is simpler. Blackstar CMS associates a unique code with each asset, based on the content of the asset. All media are cached forever. If an image changes, upload a new asset and change the links that point to it.

What’s Next?

The work coming up includes providing raw HTML editing of content and a number of minor tweaks. Then I will build an installer and make it available for enthusiastic users. Please leave a comment if you have a preference for a linux or windows installer.

Automating the Deployment of Node.js Applications

Thu, 26 May 2016 20:32:00 +1000

Have a build process to produce a deployable (non-development) package. Host on Linux with pm2 to manage processes and restarts. Move files with rsync. Perform installation activities with ssh. Reverse proxy through nginx for improved performance and SSL termination.

For information about how to build a node.js web application see Building a Web Application with Node and Typescript.

Build a Deployable Package

Step 1 of a deployment process is to build a deployment specific package of your application. The needs of an application in production are very different to the needs during development. For example, a production application does not need all your development npm dependencies, or source maps, or unminified source files.

I use gulp for all my node.js scripting requirements. When combined with shelljs and gulp-shell for executing bash commands from gulp I find it to be an effective scripting platform and a reasonably reliable cross-platform solution (with some exceptions).

My application consists of a node.js server application and a React browser application. The client-side JavaScript is written in many files. For production I would like to concatenate them together to minimize the number of HTTP requests required and improve performance. To do this I use browserify, and a gulp task like this:

gulp.task('concatenate-client-javascript', shell.task([
    'browserify src/app.js -o public/bundle.js'
]));

Browserify starts from the root of the application (app.js) and recursively follows dependencies to build a graph of all the required code. It then concatenates it all into a single file (bundle.js).

To create my deployment package I copy the files that are required for deployment to a separate directory - mine is called package. I do this with simple shell commands executed via shelljs or gulp-shell.

At this point I have a directory package containing my application as I would like it deployed to my server.

Hosting on Linux with pm2

Deploying node.js applications to Linux is a good idea because:

node.js works better on Linux than on Windows
Linux is often cheaper

Unfortunately, unlike Windows, Linux does not have a consistent mechanism for implementing servers. pm2 is a process manager that can help with this. To install pm2:

$ npm install pm2@latest -g

Instead of starting our application directly like this:

$ node app.js

We use pm2 to host the process:

$ pm2 start app.js

pm2 immediately provides a number of benefits:

it restarts the process if it crashes
it handles logging
it is able to manage many processes and control things like starting, stopping, memory use and uptime

Detailed information about a process is retrieved with the show command:

$ pm2 show app

pm2’s real party trick is that it can be configured to start with the operating system. To do this, first save the current process list then use the startup command:

$ pm2 save
$ pm2 startup

The output will be a shell script to be executed as root that will cause pm2 to start when the OS boots and start its processes. Unfortunately, this does not work for Windows.

Now we have a way to run our application, we just need to move the files to our server.

Move files with rsync

rsync is a smart file synchronization tool that can be used to move your application from your development computer to your deployment server. By using delta encoding and compression rsync optimizes file transfer size and speed.

Again, this is easily scripted with gulp.

gulp.task('deployapp',['package'], shell.task([
  "rsync -azvP package username@yourdomain:location/to/deploy/to",
]));

This gulp task will copy our application directory (package) to ~/location/to/deploy/to relative to the user’s home directory. You will obviously need to authenticate with the server to do this, which can be done with certificates or a password.

Perform Installation Activities with SSH

With SSH we can run bash commands on a remote computer. For my application the post install steps are:

Delete the database
Restart the application process with pm2

My gulp task is something like:

gulp.task('postinstallsteps',['deployapp'], shell.task([
  'ssh username@yourdomain "rm -f mydatabasefile.db && pm2 restart myapplicationname"'
]));

For more complicated scripts you can use SSH to execute a script file:

$ ssh username@yourdomain 'bash -s' < myinstallscript.sh

Reverse Proxy Through nginx for Improved Performance and SSL Termination

nginx is a wonderful web server and proxy that handles most of the world’s web traffic. When hosting a node application we can chose to let clients connect directly or via a reverse proxy, such as nginx. nginx is much better than node.js at certain tasks, such as proxying, serving file system content and terminating SSL.

I use the default ubuntu setup of nginx, which uses a file-per-site in /etc/nginx/sites-enabled to configure nginx. My nginx configuration file is included in my deployment package as a text file, similar to:

upstream app_demo_blackstar {
    server 127.0.0.1:2999;
}

# the nginx server instance
server {
    listen 80;
    server_name demo.blackstarcms.net;
    access_log /var/log/nginx/demo-blackstarcms.log;

    # pass the request to the node.js server with the correct headers and much more can be added, see nginx config options
    location / {
      #proxy_set_header X-Real-IP $remote_addr;
      #proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      #proxy_set_header Host $http_host;
      #proxy_set_header X-NginX-Proxy true;

      proxy_pass http://app_demo_blackstar/;
      proxy_redirect off;
    }

    location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js)$ {
      root /path/to/static/content;
    }
 }

The server element defines a website at http://demo.blackstarcms.net on port 80.

The proxy_pass setting configures nginx as a reverse proxy for the upstream application app_demo_blackstar, which is my pm2 hosted node.js application.

The location setting configures nginx to directly handle any requests for the listed file types. These requests will be handled by nginx and never forwarded to node.js. This improves performance and takes load off the node.js process.

A problem remains - my nginx configuration file is in my application deployment package, but needs to be in /etc/nginx/sites-enabled to be picked up by nginx. This is easily solved by creating a symbolic link in /etc/nginx/sites-enabled:

 $ cd /etc/nginx/sites-enabled
 $ ln -s /path/to/app/package/nginx.config blackstarnginxconfig

Nginx will only reload configuration when it is restarted, like so (on ubuntu):

 $ sudo service nginx restart

To pick up nginx configuration changes automatically we could easily add this nginx restart step to our post install ssh commands.

Summary

With this setup in place I can deploy my application with a single gulp command. If my application crashes the output is logged and the application is restarted. If the server crashes my application is restarted when the server restarts.

Blackstar CMS ASP.NET MVC Sample

Fri, 20 May 2016 20:32:00 +1000

Blackstar CMS now has a public sample showing how to add managed content to an ASP.NET MVC web application.

Most, if not all, public facing web applications would benefit from some content management to give administrators a way to keep the content current and to experiment with different marketing material. Building applications inside of a CMS has been tried and found wanting. Recently, I introduced Blackstar CMS as a solution to this problem.

To help developers learn how to use Blackstar CMS to add managed content to their applications I am producing a number of samples, including one for ASP.NET MVC.

Blackstar CMS ASP.NET MVC Sample

This sample is based on the ASP.NET MVC web project template.

When you start the application you will see:

This is an ASP.NET MVC web application featuring content loaded from Blackstar CMS. The content can be edited via the Blackstar CMS admin interface.

How to integrate ASP.NET MVC and Blackstar CMS

Install the Blackstar CMS .NET client from nuget

> Install-Package Blackstar

Within the MVC controller request the required content (e.g. all content with the ‘blackstarpedia’ tag).

public async Task<ActionResult> Index()
{
    var client = new BlackstarClient("http://demo.blackstarcms.net");
    var content = await client.GetByTagAsync("blackstarpedia");
    return View(HomeModel.FromContentChunks(content));
}

Within the view place the content in the correct locations.
```
 <h2>@Html.Raw(Model.smallerHeading)</h2>
```
Note the use of Html.Raw to prevent the content from being escaped.

That’s it! For more information grap the code from github, or have a look at one of the other sample projects.

Blackstar CMS Angular2 (AngularJS) Sample

Sun, 15 May 2016 20:32:00 +1000

Blackstar CMS now has a public sample showing how to add managed content to an Angular2 (AngularJS) application.

To help developers learn how to use Blackstar CMS to add managed content to their applications I am producing a number of samples, including one for AngularJS.

Blackstar CMS AngularJS Sample

This sample is based on Google’s Angular2 seed sample.

Start the Angular2 sample with npm start and you will see:

The little pencil icons indicate the managed content. If I click the pencil icon next to the ‘Home Component’ heading I am taken to the Blackstar CMS edit page for that content:

Now I can change the content to something more appropriate like ‘Angular2 Seed with Blackstar CMS’:

and now the content has been changed:

The change to the Angular2 Seed sample to make this work was trivial.

Steps used to integrate angular2 and Blackstar CMS

Add the Blackstar CMS JavaScript Client

npm install blackstar-cms-client
Add a typescript definition file (blackstar-cms-client.d.ts) for Blackstar CMS JavaScript client (included in the npm package). This step is only necessary for projects that use Typescript.
Remove the hard coded content from home.html and add data-blackstar-name attributes with appropriate names:
```
 <h3 data-blackstar-name="home-title"></h3>
 <p data-blackstar-name="home-content"></p>
```
Create the content in the Blackstar CMS admin portal

Modify the Home component to reference the type definition file and fetch content from Blackstar CMS and bind it to the UI:

 /// <reference path="blackstar-cms-client.d.ts" />

 ...

 export class Home {
     constructor() {
         var blackstar = new Blackstar.Client('http://demo.blackstarcms.net/', { showEditControls: true });
         blackstar.get({ tags: ['angular2-seed-demo'] }).then(function (chunks) {
             blackstar.bind(chunks);     // bind by matching data-blackstar-name values to chunk names 
         });
     }
 }

There are other, more manually, ways to integrate Blackstar CMS content into an AngularJS application, but the technique shown here is an easy and effective way to get started.

Blackstar, the API-first, headless CMS built for developers

Wed, 27 Apr 2016 20:32:00 +1000

Managed content in custom applications is useful, but building applications inside of a CMS is unproductive and miserable. Blackstar CMS is an API-first, headless CMS, built for application developers.

I recently wrote that Hosting a custom application inside of a CMS is a terrible idea. Now, I present a solution.

Blackstar CMS is an API-first, headless CMS, built for application developers.

It allows you to put content in your custom web application, and provide a delightful content management experience for application administrators.

Blackstar CMS is currently in alpha preview. To access the alpha, signup for the Blackstar CMS newsletter below:

A tour of Blackstar CMS

An application using Blackstar CMS to embed managed content might look something like this:

If I, as an application administrator, want to change the main heading, I click its edit link, and am taken to the blackstar edit page:

I can make a change, and see it reflected immediately:

Some core design goals for blackstar are:

it should be simple for developers to integrate into their application and simple for administrators to use to manage content
it should be deployed on premise (see Why a SaaS CMS is not for everyone)
it should be simple to deploy and administer

The Developer Experience

There are different ways to integrate with Blackstar CMS. For a SPA type application this can be done by including elements decorated with the data-blackstar-name attribute. This tells Blackstar the name of the content chunk that should be inserted into that element. Here is an example from the BlackstarPedia demo.

<h2 data-blackstar-name="smaller-heading"/>
<div data-blackstar-name="second-content"/>

smaller-heading and second-content are the names of Blackstar content chunks.

Next, add the Blackstar client script, create a client, and specify what content should be bound to this page (all chunks with the tag blackstarpedia).

<script src="blackstar-cms-client.js"></script>

var blackstar = new Blackstar.Client('http://localhost:2999/', { showEditControls: true });
blackstar.get({ tags: ['blackstarpedia'] }).then(function (chunks) {
  blackstar.bind(chunks);     // bind by matching data-blackstar-name values to chunk names 
});

That’s enough to populate the page with content and allow application administrators access to create and edit content.

Blackstar CMS content can also be accessed via HTTP or a JavaScript client (browser or node.js).

Why a SaaS CMS is not for everyone

When I searched for a developer-oriented CMS designed for adding manageable content to custom applications I found a crowded market of hosted solutions (SaaS) and very little support for self-hosting. There are three reasons why I think that a hosted CMS is a bad idea for most situations.

1. It is an operational dependency

If your hosted CMS goes down, then your application becomes unavailable, or at least has no content.

2. You depend upon continued operation of the service

Hundreds of thousands of applications depended on a backend-as-a-service called Parse. In January 2016 Facebook shutdown Parse, leaving all those customers stranded. It probably made sense for Facebook, but it left their customers with no options. Other SaaS products like Charm shutdown before they even launch.

When you depend upon a third-party service for the runtime operation of your application you lose control of your own destiny.

3. Latency

Web applications always have some latency. It’s the time it takes for a request to travel from the client to your server, and then back again.

Using a SaaS CMS adds the latency of another network request. Now a request has to travel from the client, to your server, to your SaaS CMS server, back to your server, then back to the client. And that makes your app feel S L O W.

Blackstar CMS is currently in alpha preview. To access the alpha, signup for the Blackstar CMS newsletter below:

Hosting a custom application inside of a CMS is a terrible idea

Tue, 26 Apr 2016 20:32:00 +1000

Building applications inside of a content management system is now a failed experiment. A new style of api-first, headless content management systems inverts the relationship and provides a pleasant and productive development process.

The Benefits of App-In-CMS

The appeal of building an application inside of a CMS is that you get stuff for free. We greedy software developers cannot resist pre-built authentication, security, widgets and of course content management. For very simple, short-lived applications where control over the output or quality engineering are not important then app-in-cms may be OK.

Hosting a custom application inside of a CMS is a terrible idea

The old idea about how to merge user editable content into custom applications was to build applications inside of content management systems. Early productivity advantages (such as built-in user management and authentication) are soon overtaken by compromises required to stay within the CMS framework and challenges related to versioning and deployment. Architectural and functional compromises soon leak out and affect the user. It is common to get three months into a CMS-based application development project only to realise that it should have been a standalone application and that the CMS framework is now a liability. When designing and building software applications we should try to avoid such architectural dead ends.

The biggest problem with app-in-cms is that the application developer surrenders control of the application. The CMS is a framework that you must work within and this leads to hacks, compromise and compounding technical debt.

Confessions of an app-in-cms developer

I know that app-in-cms is a bad idea, because I have done it, and seen it done. For me, this was DotNetNuke. I have seen the same experiment repeated with Sharepoint, Sitefinity, Kentico and Umbraco. The result is a predictable, rapid accumulation of unresolvable technical debt.

Years after my last DotNetNuke project I worked on a large ecommerce application. Because it was a business to consumer ecommerce application, marketing was a primary concern, and that means that the ability to manage content was essential. The problem was solved by reserving fixed banner ads throughout the site. When the marketing team wished to modify content they would create new banner ads of the prescribed dimensions and upload them to the site. When the user’s cache expired they would receive the new banner ad with its new content. This process was labour intensive, horribly inflexible and bad for users who were subjected to too much tiny, low resolution text in banner ad images. The landing page of the site, and some associated content pages, were served from a CMS system. When the user started an ecommerce process they transitioned into a custom application that had to be synchronized to the CMS front-end so that the transition was not obvious to the user. What a mess.

More recently I was involved in another business-to-consumer project that required content managemed by the marketing department. After investigating possible integration with a content management system the team decided to instead build their own micro CMS features into the application so that application administrators could modify certain pieces of content. This is a better solution than replaceable banner ads but it is still inflexible and costly, having consumed some $10k of the project’s budget.

Hopefully I have given sufficient examples to demonstrate that content management within custom applications is a common problem, and the traditional app-in-cms is not the answer.

The headless CMS solution

A headless CMS looks after managing content and delivers it via an API. The CMS is not your application, it is not responsible for rendering pages, it simply makes content available. It has no templates, themes or views. It has only content with metadata, a way to get that content, and a way to maintain that content. For example, here is a simple request made to a headless CMS for all content tagged index-page. Note that the response (lower right) is an array of JSON data containing the content:

Within a custom application you then have the responsibility of transforming the json content into an acceptable view. But now application administrators have the ability to maintain content without affecting the development team.

For custom applications, a headless CMS is a much better solution than the traditional full CMS.

This inverts the APP <-> CMS relationship, putting the application back in charge, with embedded CMS content.

Summary

An api-first, headless CMS is a good choice for adding managed content to web applications, and many web applications, especially those that are public facing, would benefit from managed content. The alternative, building an application inside of a CMS platform, leads to hacks, compromise and compounding technical debt.

React Forms

Fri, 15 Apr 2016 20:32:00 +1000

As I have said before, React focuses on mapping user interface state to the DOM and publishing events. It doesn’t try to help with common application tasks like routing, http requests, or managing forms. So how do we work with forms in React?

The Basic React Form Pattern

The design of React leads to a curious contradiction for forms.

If the user interface is a transformation of the user interface state then it should not be possible to modify the content of a form without updating the application state

And this is exactly where we start with React forms. Here is the most basic form implementation possible. The application model is bound to the form fields. Note that it is not possible for the user to modify the content of the form (see the ‘Result’ tab). It would not make sense because the form is supposed to be a reflection of the application state.

The way to solve this problem is to break one of my own rules of working with React - we let the component maintain local state. The risk here is that the content of the form could become inconsistent with the application state. For the content of a form, this is something that I can live with. The above example, modified to use local component state for the content of the form is:

The form fields are no longer bound to the components properties. Instead they are bound to local state, that is copied from the components properties (in the getInitialState() lifecycle method). That provides one-way binding from the state to the form. To bind changes to the form back to the component state we need the bindState method. bindState creates event handlers that push changes to the form back to the component state. Everytime the user modifies a character in a form field it is pushed back to the component state, which causes the component to re-render with the new form content. We now have a form that allows the user to change form field values.

We can then add a form submission handler to deal with changes to the form when it is submitted.

Because we have been synchronizing the form to the component state on every character change we already have the latest content of the form in the state. So what do we do with it?

What To Do With Form Submissions

There are two ways to handle form submission. Which you chose depends on the type of component. If I was building a reusable widget (like a calendar or something) then I would express all results as calls to a property of the component.

Note how onSubmit is a callback function passed to the PersonForm component as a prop. When the form is ready to publish its result it simply calls the supplied onSubmit function and passes its result.

This approach has elegance. Components designed in this way are perfectly composable. The flipside is that components exist in a heirarchy, and results communicated via callbacks must be bounced all the way up the heirarchy to the top. Applying this strategy everywhere is a productivity and maintenance nightmare, leading to the rule:

use callbacks to communicate results only for re-usable, widget-style components

For other components, publish results out of your React component heirarchy to a state management solution like redux to merge the changes into your application state.

Note that the value passed to dispatch is an object of the form { type: '', ... }. This is a redux convention. The general idea of libraries like redux/flux etc is that they take these actions, update the application state and re-render the entire application component graph, preserving the symmetry between the application state and the DOM.

Higher-level Form Abstractions

A great way to avoid this issue all together is to use one of many libraries providing higher-level form abstractions. Typically, the library generates an appropriate form based on a supplied piece of state. Such libraries are terrific for productivity but are at their best when you do not require precise control over the form. Once you start having to invest effort in customizing how a tool generates forms for you I think you are better off reverting to building the form yourself.

Validation

I wrote recently about JavaScript object validation with JSON Schema so I have covered how to validate objects. The form approach described above covers how to bind objects to and from forms with React. The missing piece then is just how to trigger validation and handle displaying any validation errors to the user. I will investigate this in a future post.

JavaScript object validation with JSON Schema

Thu, 14 Apr 2016 20:32:00 +1000

Recently I wrote about Building a Web Application with Node and Typescript. One of the advantages of having JavaScript (Typescript) on both the client and server is that we can share validation logic between the two.

JavaScript Object Validation

There are many ways of validating JavaScript objects.

The Hapi framework includes a library called Joi that validates objects this way:

var Joi = require('joi');

var schema = Joi.object().keys({
    name: Joi.string().alphanum().min(3).max(30).required(),
    age: Joi.number().integer().min(0)
}).with('name', 'age');

Joi.validate({ name: 'abc', age: 103 }, schema, function (err, value) { });

Another popular choice is Flatiron Revalidator, which validates objects like this:

var revalidator = require('revalidator');

console.dir(revalidator.validate({ name: 'abc', age: 103 }, {
    properties: {
        name: {
        type: 'string',
        required: true,
        minLength: 3,
        maxLength: 30
        },
        age: {
        type: 'integer',
        minimum: 0,
        required: true
        }
    }
}));

Years ago I even wrote my own object validation library called dbc. It served me well and is still powering validation on many public web applications. I would not use dbc today as there are now better options available. Dbc has some novel features that the other libraries don’t have, such as creating auto-validating objects:

var AjaxOptions = dbc.makeConstructor({
        type:       [
                        {validator:'type', args:['string']},
                        {validator:'oneOf', args:[['GET','POST','PUT','DELETE']]}
                    ],
        url:        [{validator:'type',args:['string']}],
        dataType:   [{validator:'type',args:['string?']}]
    });

Here, AjaxOptions is a new JavaScript ‘class’ that will automatically validate when an object is created from it.

It also has a feature that enforces function argument and return type validation. The following code adds validation to a function and guarantees that its argument is a string and its return value is an AjaxOptions:

AjaxOptions.fromCorsRequest = dbc.wrap(function (corsRequestText) {
        return new AjaxOptions(new CorsRequest(corsRequestText).getJSON());
    }, {
        0: [{validator:'type', args:['string']}]
    }, [{validator:'isInstance',args:[AjaxOptions]}]);

Everyone has their own way of declaratively specifying what a valid object looks like. If only there was some kind of standard….

JSON Schema

JSON Schema is a standard that defines, among other things, a declarative format for specifying object validation rules. JSON Schemas are themselves defined in JSON, leading to the delightful situation of having a JSON Schema that defines the schema for all JSON schemas. JSON schema is well considered and standardised making it an excellent choice for JavaScript object validation. Now all we need is a library that can validate a JavaScript object against a JSON schema.

Jsonschema

Jsonschema is a library that validates javascript options against JSON schemas.

 var validate = require('jsonschema').validate;

  var schema = {
    "type": "object",
    "properties": {
      "name": {
        "type": "string",
        "minLength": 3,
        "maxLength": 30
      },
      "age": {"type": "integer", minimum:0}
    },
    "required": ["name","age"]
  };

  console.log(validate({ name: 'abc', age: 103 }, schema));

With this we have a standard, reusable validation strategy that we can use in the browser or in node.

We can make things a little neater with a bit of convention. Here’s a way of validating objects that have a schema method (code is ES2015):

import {validate} from 'jsonschema';

class Person {
    constructor(name,age) {
        this.name = name;
        this.age = age;
    }
    schema() {
        return {
            "type": "object",
            "properties": {
            "name": {
            "type": "string","minLength": 3,"maxLength": 30
            },
            "age": {"type": "integer", minimum:0}
            },
            "required": ["name","age"]
        }
    }   
}

function validateObjectWithSchema(obj) {
    return validate(obj, obj.schema());
}

let p = new Person("abc",103);
console.log(validateObjectWithSchema(p));

Final Thoughts

To benefit from validation we need to hook it into our application. Common places to do this are:

form validation in the browser
request validation on the server

Both of these are fairly simple. I will show some examples in future posts.

Building a Web Application with Node and Typescript

Thu, 24 Mar 2016 20:32:00 +1000

For the purpose of this article I define a web application as a HTTP network service, such as what we might build with Asp.NET MVC, Ruby on Rails, Django, Php etc. This article will demonstrate how to write a web application in Typescript that runs on Node.js.

Why do this?

Running as much of an application as possible on the client creates a richer user experience and a better perception of performance. Unfortunately, the client cannot be trusted so enevitably a lot of an application’s logic must remain on the server, within the circle of trust.

This unpleasent client/server split can be made a little more bearable by sharing code between the two halves of the application, and this is easier if the code on both sides is written in the same programming language. Useful things to share include types, validation and application specific computations.

Benefits of Node

Node has some advantages as a web server. It is fast, simple and inherently safely asynchronous.

Node enables cross platform development and deployment by running well on windows, mac and linux.

Node has the fastest growing, if not the best quality, package repository. It is a vibrant and creative ecosystem of people who are excited about programming. Consequently, they wet their pants from time to time.

Benefits of Typescript

The static vs dynamic typing debate is beyond the scope of this article. Suffice to say that having compile time type checking can help the self-aware programmer to write more correct software. It is also useful when sharing types between the server and client.

Typescript has ‘optional static typing’, meaning that you can relax the type checking when it feels good to do so. This can be useful when you want to do dynamic type things and when you are integrating with JavaScript libraries.

Get Started With Node and Typescript

Step 1, https://nodejs.org/en/. Node itself is just an executable, so you don’t really need to install it, but doing so will be easier and will install the node package manager (npm) as well.

Next, install typescript, via npm:

$ npm install -g typescript

Now create your first typescript program (person.ts):

interface Person {
    name: string;
    age: number;
}

function birthday(someone : Person) : Person {
    return {name: someone.name, age: someone.age+1};
}

console.dir(birthday({name:"Judy",age:39}));

and compile using the typescript compiler (tsc):

$ tsc person.ts

The output of the typescript compiler is the javascript file person.js, which contains:

function birthday(someone) {
    return { name: someone.name, age: someone.age + 1 };
}
console.dir(birthday({ name: "Judy", age: 39 }));

Wooh! That’s just the typescript with the type annotations striped out. This makes sense because typescript is a compile time system, not a runtime system.

Typescript is a compile time system, not a runtime system.

Keeping this in mind will help you as you pickup typescript.

Now we can run our program with node.js

$ node person.js
{ name: 'Judy', age: 40 }

Now to show what makes typescript useful. Let’s intentionally introduce a type error into our program:

interface Person {
    name: string;
    age: number;
}

function birthday(someone : Person) : Person {
    return {handle: someone.name, age: someone.age+1};
}

console.dir(birthday({name:"Judy",age:39}));

See the type error? The birthday function returns a value that is not valid for the type Person. When we compile:

$ tsc person.ts
person.ts(7,13): error TS2322: Type '{ handle: string; age: number; }' 
is not assignable to type 'Person'.
  Object literal may only specify known properties, 
  and 'handle' does not exist in type 'Person'.

Adding a Web Framework

Typescript relies heavily on the JavaScript ecosystem. As mentioned previously typescript does not exist at runtime. So it needs the JavaScript runtime, the JavaScript (or node) standard library, and JavaScript packages.

For this article we will work with the node web framework hapi. Hapi is a javascript framework. To use it in a typescript project we need a type definition file to tell typescript about the types used in hapi. To find Typescript type definitions search the definitely typed site which is a repository of typescript type definition files. The easiest way to install a type definition is to use tsd.

$ npm install tsd -g

Now you can install the type definitions for Hapi:

$ tsd install hapi --save

Unfortunately there is nothing synchonizing typescript type definitions to the libraries they define, and when the two get out of sync it causes frustration. I recommend:

Start by choosing libraries that have TSD type definitions. You can create your own but it is not easy.
Install the type definition. Inspect its source to discover which version of the JavaScript library the type definition was written for.
Install that version of the JavaScript library.

For me, TSD installed type definitions for hapi 13.0.0, so that is the version of the Javascript library I want to install:

$ npm install hapi@13.0.0 --save

Now create a new typescript program server.ts. The first line should be:

/// <reference path="typings/tsd.d.ts" />

tsd.d.ts is a file created by TSD that references the type definitions for all included libraries. This comment tells the typescript compiler where to find type definitions.

Next, add the code for a simple hapi web server:

/// <reference path="typings/tsd.d.ts" />

import * as Hapi from 'hapi';

const server = new Hapi.Server();
server.connection({ 
    host: 'localhost', 
    port: 8000 
});

server.route({
    method: 'GET',
    path:'/hello/{name}', 
    handler: function (request, reply) {

        return reply('hello ' + request.params['name']);
    }
});

server.start((err) => {
    if (err) { throw err; }
    console.log('Server running at:', server.info.uri);
});

Compile and start:

$ tsc server.ts
$ node server.js
Server running at: http://localhost:8000

Now navigate your browser (or curl) to http://localhohst:8000/hello/Eugene:

$ curl http://localhost:8000/hello/Eugene
hello Eugene

Automatically Compiling and Restarting the Server

The development workflow at this point is clunky. When a change is made to the typescript source you need to recompile the application, then restart the hapi server to pickup the changes.

$ tsc server.ts
$ node server.js

This gets old quickly. We can have typescript automatically detect and compile changes by using the -w flag:

$ tsc server.ts -w

Not only is this more convenient, it is also faster. By keeping the process alive typescript is able to implement incremental compilation (only recompile the bits that have changed).

To automatically restart the server we can use the node-dev package.

$ npm install node-dev -g

Then run the program with node-dev instead of node.

$ node-dev server.js

node-dev detects when a required module is changed and restarts the process.

Performance

Node is supposed to be fast right? As written, our simple server is processes about 1,600 reqs/s.

Not bad, but nothing special considering it doesn’t do much. This is because node is single threaded and therefore only capable of utilizing a single CPU core. We can fix this (somewhat) using node’s ‘cluster’ module to start a process per CPU core. The clustered code looks like:

/// <reference path="typings/tsd.d.ts" />

import * as cluster from 'cluster';
import { cpus } from 'os';
import * as Hapi from 'hapi';

const numCPUs = cpus().length;

if (cluster.isMaster) {
  for (var i = 0; i < numCPUs; i++) {
    cluster.fork();
  }
  cluster.on('exit', (worker, code, signal) => {
    console.log(`worker ${worker.process.pid} died`);
  });
} else {
    const server = new Hapi.Server();
    server.connection({ 
        host: 'localhost', 
        port: 8000 
    });

    server.route({
        method: 'GET',
        path:'/hello/{name}', 
        handler: (request, reply) => reply('hello ' + request.params['name'])
    });

    server.start((err) => {
        if (err) { throw err; }
        console.log('Server running at:', server.info.uri);
    }); 
}

This pushes the reqs/s up to 5,700.

That’s better :)

Is it time to learn F#?

Fri, 18 Mar 2016 20:32:00 +1000

TL;DR After a marathon production process I have finally released a new screencast, “F# Fundamentals“ for Pluralsight.

2015 was a big year for functional programming (FP), with many high-profile successes.

Facebook implemented its anti-spam system using Haskell, and its flow and hack compilers using OCaml.
There were a number of highly subscribed FP MOOCS.
Jane Street continued to build a successful trading business on OCaml
There was a pronounced shift towards FP style in JavaScript tooling such as Elm, React, RxJS, Immutable.js and Angular2.
The big data ecosystem moved towards scala as a de-facto standard.

Developers are discovering that FP helps them to be clearer and more rigorous in their programming. A maturing industry is discovering a set of more capable tools.

Is it time to learn functional programming?

Probably.

Either you will discover a whole new level of programming skill and productivity, or you will learn why it is not for you. Surely either option is better than ignorance?

Is it time to learn F#?

Yes. If you work with .NET then F# should be one tool in your toolbox.

The biggest uptake of functional programming has been in the tools that are able to benefit from a larger ecosystem. Scala and Clojure have benefitted from targeting the JVM. This allows them to be used anywhere Java is expected, to run on the JVM and to access the Java standard library. In the .NET world F# is the equivalent. It is a functional programming language that runs on the .NET CLR and has access to the .NET base class library.

Still not sure? Have a look at this excellent article describing the benefits of F# and functional programming.

How?

Try F# lets you experiment with F# programming in your browser. http://fsharpforfunandprofit.com/ has lots of great content.

Finally, I have recorded a six hour screencast (F# Fundamentals) that takes the student from no knowledge of F# to a intermediate level sufficient for creating console applications, services and web applications.

Implementing Tetris with React and Redux - VII

Mon, 11 Jan 2016 20:32:00 +1000

Previously, we converted the application to Typescript. In this episode we will implement collapsing rows, game ending and scoring.

To follow along with this article checkout the commit e843d2.

$ git clone https://github.com/liammclennan/tetris.git
$ git checkout e843d2
$ npm run bootstrap
$ npm run likehell

Collapsing Rows

In tetris rows collapse when they are full. The first step required for implementing this behaviour is to be able to detect full rows. Start with a test:

describe('game with last row full', ()=> {
    var game = new Models.Game(()=>{});
    game.rubble = _.range(1, game.cols+1).map(col => new Models.Point(game.rows, col));

    it('should have one completed row', ()=> assert.equal(1, game.completedRows().length));
    it('should have row 15 as completed', ()=> assert.equal(15, game.completedRows()[0]));
});

and implement the Game.completedRows() method. I have used underscore to generate ranges. The method works by finding all the rows for which every column contains rubble.

completedRows() {
    return _.range(1,this.rows+1).filter(row =>
          _.range(1,this.cols+1).every(col => this.rubbleHas(row,col))
    );
}
rubbleHas(row,col) {
        return this.rubble.some(point => point.row === row && point.col === col);
}

Next we need to collapse full rows when they are found. This involves removing all the rubble in the collapsed row, and then shifting all other rubble down. Again, start with a test:

describe('collapsing', ()=> {
    var game = new Models.Game(()=>{});

    var row14 =  [new Models.Point(game.rows-1,1), new Models.Point(game.rows-1, 2)];
    game.rubble = _.range(1, game.cols+1)
      .map(col => new Models.Point(game.rows, col))
      .concat(row14);

    it('should move row 14 to row 15 (collapse)', ()=> {
      assert.equal(22, game.rubble.length);
      game.collapseRow(15);
      assert.equal(2, game.rubble.length, 'the full row has been deleted');
      assert.ok(_.some(game.rubble, point => point.row === 15 && point.col === 1), 'a point from the 2nd row has moved down');
      assert.ok(_.some(game.rubble, point => point.row === 15 && point.col === 2), 'the other point from the 2nd row has moved down');
    });
});

And then implement:

collapseRow(row) {
    this.rubble = this.rubble.filter(point => point.row !== row);
    this.rubble.filter(point => point.row < row).forEach(point => point.row += 1);
}

Ending the Game

The game should end when rubble reaches the top of the game area. The game object therefore has all the information required to know if the game has finished:

   isGameOver() {
    return this.rubble.some(point => point.row === 1);
  }

Then at the end of the tick processing (convertToRubble) we know not to start a new piece if the game has ended:

convertToRubble() {
    this.rubble = this.rubble.concat(this.fallingPiece.points());
    this.completedRows().forEach(r => this.collapseRow(r));
    if (!this.isGameOver()) {
      this.startAPiece();
    }
}

We should also check when processing actions:

function reducer(state = new Model.Game(()=>{}), action) {
  if (state.isGameOver()) return state;

  switch (action.type) {
      case 'TICK':
        const revedState = state.tick();
        if (!revedState.isGameOver()) {
          setTimeout(() => store.dispatch({ type: 'TICK' }),300);
        }
        return revedState;
      case 'ROTATE':
        return state.rotate();
      case 'LEFT':
        return state.left();
      case 'RIGHT':
        return state.right();
      default: return state;
  }
}

This effectively stops the TICK loop once the game ends.

Then we can change GameView component to change the UI when the game ends:

export var GameView = React.createClass<GameViewProps,any>({
  render: function () {
    return <div className="border" style={{width: this.props.game.cols*25, height: this.props.game.rows*25}}>
      { this.props.game.isGameOver() ?
        <span>GAME OVER</span> : <span>
        <PieceView piece={this.props.game.fallingPiece} />
        <RubbleView rubble={this.props.game.rubble} />
        </span>
      }
    </div>;
  }
});

Scoring

We will use a simplified tetris scoring system that awards points for clearing rows. The more rows cleared the higher the award.

# Lines	Points
1	40
2	100
3	300
4	1200

We know when a row collapses, so we can add a method to Game that calculates the points to award when rows collapse. It can be a simple encoding of the above points table, with the correct indexing to retrieve the right value:

calculateAward(completedRows) {
  const map = {
    0: 0,
    1: 40,
    2: 100,
    3: 300,
    4: 1200
  };
  return map[completedRows.length];
}

A nice property of this implementation is that it is valid for values of completed rows >= 0. Tetris does not allow for more than 4 rows to be completed at the same time.

We can use calculateAward in the convertToRubble method to increment a Game.score property that tracks the players score.

convertToRubble() {
  this.rubble = this.rubble.concat(this.fallingPiece.points());
  const completedRows = this.completedRows();
  completedRows.forEach(r => this.collapseRow(r));
  this.score += this.calculateAward(completedRows);
  if (!this.isGameOver()) {
    this.startAPiece();
  }
}

Now the player needs to know what their score is, so we modify GameView to include a ScoreView:

export var GameView = React.createClass<GameViewProps,any>({
  render: function () {
    return <div className="border" style={{width: this.props.game.cols*25, height: this.props.game.rows*25}}>
      { this.props.game.isGameOver() ?
        <span>GAME OVER</span> : <span>
        <PieceView piece={this.props.game.fallingPiece} />
        <RubbleView rubble={this.props.game.rubble} />
        <ScoreView score={this.props.game.score} />
        </span>
      }
    </div>;
  }
});

and implement score view to dump the game score in a div:

interface ScoreViewProps { score:number; }
const ScoreView = React.createClass<ScoreViewProps,any>({
  render: function () {
    return <div className="score-display">{this.props.score}</div>;
  }
});

et voilà!

Implementing Tetris with React and Redux - VI

Thu, 07 Jan 2016 20:32:00 +1000

Last time I promised you collapsing completed rows and ending the game. That will have to wait. I have something even more exciting!.

Converting to typescript

To follow along with this article checkout the commit 70fa71.

$ git clone https://github.com/liammclennan/tetris.git
$ git checkout 70fa71
$ npm run bootstrap
$ npm run likehell

Typescript

Having a type checking compiler eliminates a large set of bugs, and makes a heap of tests redundant. Tests are a debt that must be serviced so we should aim to get maximum value from the fewest tests.

Typescript adds compile time type checking to JavaScript. Converting our application to typescript requires:

converting the application code from JavaScript to Typescript
acquiring Typescript type definitions for all dependencies
modifying the build process to include the Typescript compiler

Adding Typescript to the Build

Working backwards we can start by acquiring the required Typescript dev tools:

npm install typescript tsd --save

Typescript is the npm package including the typescript compiler. Tsd is a tool for acquiring Typescript type definition files for popular JavaScript libraries.

We can then use tsd to fetch the type definitions we need:

tsd install react-global mousetrap mocha node underscore --save

The type definitions are saved into a typings/ directory and listed in a manifest tsd.json. A Typescript file typings/tsd.d.ts references all of the type definitions as a shortcut way to import them all. I could not find a tsd definition for Redux so I manually included the type definition at src/redux.d.ts.

Next we prepend another build command in gulpfile.js.

tsc src/app.tsx test/modelsTests.ts --jsx react --outDir build/es6 -t ES6

tsc is the typescript compiler.

src/app.tsx and test/modelsTests.ts are the entry point files. All other files are found by following dependencies. The .tsx extension indicates that the file contains jsx embedded in Typescript.

--jsx react tells the compiler that when it finds jsx it should compile it to React JavaScript.

--outDir build/es6 specifies where the output should be written.

-t ES6 sets the compiler to target ES6 (ES2015).

Converting to Typescript

The easiest way to convert an application from JavaScript to Typescript is to start from the leaves of the dependency tree and work back to the root. That means starting with the modules that have the fewest dependencies. For this Tetris app model.js is the best choice.

Firstly we can change the file extension to .ts and attempt to compile using:

tsc src/model.js -t ES6

which won’t work. The typescript compiler will give messages indicating the things that need to be fixed. For model.js this is mostly the syntax of classes. Typescript has a shortcut syntax for specifying the properties of classes that looks like:

export class Tetromino {
  constructor(public name:string, public rotator: (rotation:string)=>Point[]) {}
}

This automatically creates public properties name and rotator. name has the type string. rotator has the type (rotation:string)=>Point[] which means a function from a string to an array of Point.

When all the type errors in model.ts are fixed we move to components.ts. This file is slightly more complicated because it has dependencies. The first thing to do is add a reference to tsd.d.ts.

/// <reference path="../typings/tsd.d.ts" />

This is a directive to the typescript compiler that tells it where to find type information. From tsd.d.ts typescript can find type definitions for all of the libraries we are using (except Redux).

The big change in this file is specifying the props and state type parameters for each React component by passing the types to the createClass function.

For the GameView component the type of its props is:

interface GameViewProps {
  game: Model.Game;
}

GameView does not use state (none of our components do) so the component changes to:

export var GameView = React.createClass<GameViewProps,any>({
  render: function () {
    return <div className="border" style={
      {
        width: this.props.game.cols*25,
        height: this.props.game.rows*25
      }}>
      <PieceView piece={this.props.game.fallingPiece} />
      <RubbleView rubble={this.props.game.rubble} />
    </div>;
  }
});

We now move up the dependency tree to app.tsx which needs to reference tsd.d.ts and redux.d.ts:

/// <reference path="../typings/tsd.d.ts" />
/// <reference path="redux.d.ts" />

The file is otherwise much the same.

The test file modelsTests.ts also needs a reference to tsd.d.ts. Then everything should compile properly, the tests should run successfully and the application should run.

And now we have type safety.

Implementing Tetris with React and Redux - V

Tue, 05 Jan 2016 20:32:00 +1000

In the previous article we refactored the domain model, simplified the React rendering, and added some JavaScript unit tests.

In this addition we work on collision detection and add event handling for rotation and lateral movement.

To follow along with this article checkout the commit ce0c31.

$ git clone https://github.com/liammclennan/tetris.git
$ git checkout ce0c31
$ npm install
$ npm run away

Adding Event Handlers

Normally in a React application the events come out of the DOM, and therefore come via the React eventing system. For tetris the relevant events are not DOM events but keyboard events. To work with keyboard events we will use the mousetrap library.

$ npm install --save mousetrap

When the application starts we register some keyboard listeners for the relevant keys:

import * as Mousetrap from 'mousetrap';

Mousetrap.bind('space', ()=> store.dispatch({type:'ROTATE'}));
Mousetrap.bind('left', ()=> store.dispatch({type:'LEFT'}));
Mousetrap.bind('right', ()=> store.dispatch({type:'RIGHT'}));

With that in place the space, left, and right keys will dispatch corresponding actions to Redux. Therefore, we need to update our Redux reducer function to correctly process these new actions.

function reducer(state = new Model.Game(), action) {
  switch (action.type) {
      case 'TICK':
        const revedState = state.tick();
        setTimeout(() => store.dispatch({ type: 'TICK' }),500);
        return revedState;
      case 'ROTATE':
        return state.rotate();
      case 'LEFT':
        return state.left();
      case 'RIGHT':
        return state.right();
      default: return state;
  }
}

The TICK case has also changed slightly. Previously I used setInterval() to trigger a tick on a time interval. The disadvantage of setInterval() is that it does not allow the debugger to stop the game. If we stop on a breakpoint to do some debugging setInterval() carries on triggering ticks every 500ms. The solution is to process a tick and then use setTimeout() to queue the next tick. This way if execution stops on a breakpoint we won’t get the next tick until we resume execution.

The ROTATE, LEFT and RIGHT actions are handled by delegating to corresponding methods on the game object. Keeping the heavy lifting out of the reducer function makes it easier to understand and makes the application easier to test.

Implementing Rotation

The Piece type is aware of its possible rotations (N,E,S,W) and is capable of producing the correct set of points given any of those rotations. Therefore, for the game to rotate the current piece it merely needs to update the piece’s rotation property. This is done be delegating to the Piece object, so the Game‘s rotate() method is just:

rotate() {
  this.fallingPiece.rotate();
  return this;
}

Note that rotate must return the game object so that the reducer function can then return it as the updated Redux state.

The rotate() method of Piece then does the work of calculating what the ‘next’ rotation is (i.e. if the current rotation is ‘N’ then ‘next’ rotation will be ‘E’).

rotate() {
    const rotations = Piece.rotations();
    this.rotation = rotations[(rotations.indexOf(this.rotation)+1) % 4];
},
static rotations() {
    return ['N','E','S','W'];
}

Implementing Lateral Movement

As this is Tetris, the player may move the piece from side to side. Like rotation the Game‘s left() and right() methods simply delegate to left() and right() methods on the Piece.

A Piece is a Shape plus a rotation plus an offset. When moving laterally the shape stays the same and the rotation stays the same. Only the offset has to change. A left move requires decreasing the column of the offset by one and a right move requires increasing the column of the offset by one, hence the left() and right() methods of Piece are:

left() {
  this.offset = new Point(this.offset.row, this.offset.col-1);
}
right() {
  this.offset = new Point(this.offset.row, this.offset.col+1);
}

Preventing Boundary Violations and Collisions

Our rotation and lateral movement, as implemented, is entirely unconstrained. The player is able to move the pieces outside the boundary of the game, and also to move pieces over the top of rubble. Preventing these violations is non-trivial, especially for rotations. How do you know if a rotation will cause the piece to go out of bounds or to overlap with some rubble? The easier way is to allow the operation to occur, then check for boundary violations and collisions, and if a problem is found apply a compensating action that reverses the original action.

If a left movement causes the piece to go out of bounds apply a right movement to compensate. If a rotation causes the piece to overlap some rubble apply a rotation in the opposite direction to compensate. Because React separates the calculation of the UI state from the rendering, all of this happens without causing any change to the DOM. The user does not see the action and its compensation. This strategy is implemented in the Game‘s transactionDo method:

transactionDo(thing, compensation) {
  thing();
  if (this.fallingPieceIsOutOfBounds() || this.fallingPieceOverlapsRubble()) {
    compensation();
  }
}

The argument thing is the operation we are attempting. compensation is the compensating action to apply in the event that thing puts the game into an invalid state. An invalid state is defined as a point out of bounds or a point overlapping rubble. The detection is accomplished with the following two simple methods of the Game type:

fallingPieceIsOutOfBounds() {
  return this.fallingPiece.minCol() < 1 ||
    this.fallingPiece.maxCol() > this.cols ||
    this.fallingPiece.maxRow() > this.rows;
}
fallingPieceOverlapsRubble() {
  return this.fallingPiece.points().some(p => this.rubble.some(r => r.sameAs(p)));
}

The rotate, left and right methods on Game can then use transactionDo:

rotate() {
  this.transactionDo(
    ()=>this.fallingPiece.rotate(),
    ()=> this.fallingPiece.unRotate());
  return this;
}
left() {
  this.transactionDo(
    ()=>this.fallingPiece.left(),
    ()=> this.fallingPiece.right());
  return this;
}
right() {
  this.transactionDo(
    ()=>this.fallingPiece.right(),
    ()=> this.fallingPiece.left());
  return this;
}

unRotate is simply the inverse of rotate:

unRotate() {
  this.rotation = Piece.rotations()[(Piece.rotations().indexOf(this.rotation)-1) % 4];
}

Improving the Development Workflow

Development is more efficient if we can see the changes to our application without having to save changes, run the build, and refresh the browser. We can automate the rebuilding process using a filesystem watcher. For this project we will use gulp to orchestrate this process.

npm install --save gulp gulp-shell

Gulp is configured by writing a build script in a file gulpfile.js. To accomplish our goal this can be as simple as:

var gulp = require('gulp');
var shell = require('gulp-shell');

gulp.task('buildandtest', shell.task([
  'browserify -t [ babelify --presets [ react es2015 ] ] app.js -o bundle.js',
  'node_modules/mocha/bin/mocha --compilers js:babel-register'
]));

gulp.task('default', function() {
    gulp.watch(['*.js', 'test/*.js'], ['buildandtest']);
});

The default task establishes a file system watch for files matching the pattern *.js and test/*.js. If a change is detected then the buildandtest task is run.

buildandtest uses the shell to build the application (using the same commands that we had in package.json) and then runs the mocha tests.

With Gulp setup we no longer need to use npm as build tool and can simplify the scripts section of package.json to:

"scripts": {
    "away": "node node_modules/gulp/bin/gulp.js"
  },

I called the script away so that the command to start the build is:

$ npm run away

We now have a mostly playable Tetris.

The big things missing are collapsing complete rows and ending the game. Hopefully you can see how those features will be implemented.

Next Time…

The next installment will cover collapsing completed rows and ending the game.

Implementing Tetris with React and Redux - IV

Mon, 04 Jan 2016 20:32:00 +1000

In the previous article we introduced Redux for managing the user interface state and added some rudimentary animation.

In this addition we will refactor the application models and introduce some testing.

To follow along with this article checkout the commit b01c12d.

$ git clone https://github.com/liammclennan/tetris.git
$ npm install
$ git checkout b01c12d

Fixing the Model

At this point in the implementation it is time to start thinking about the logic of Tetris and how we will implement things like collision detection, boundaries and collapsing rows. This is stuff that I don’t want in my React components because it is application logic that should be a core part of my application.

I have made a mistake with my domain modelling by representing the different Tetrominos as different types, when in fact they only need to be different configurations of the same type (Shape).

Also, I have come to the realization that a Tetrominos is not a set of points with a standard orientation that can be rotated. It is better to think of a Tetromino as a set of rotations of a shape (I call the rotations N, S, E and W).

The ‘Shape’ Type

A ‘Shape’ is a kind of Tetromino. It has a name and a function that returns the shape’s points given a rotation.

// a tetromino
export class Shape {
  constructor(name, rotator) {
    this.name = name;
    this.rotator = rotator;
  }
  pointsRotated(rotation) {
    return this.rotator(rotation);
  }
}

With the Shape type we can now fully define all possible rotations of all possible Tetrominos. I put them in a dictionary so that I can easily access the one I want (e.g. shapes['Z']):

// dictionary of shape type to square offsets
export var shapes = {
  'O': new Shape('O', rotation => [new Point(1,1),new Point(1,2), new Point(2,1),new Point(2,2)]),
  'I': new Shape('I', rotation => {
    switch (rotation) {
      case 'N': return [new Point(1,1), new Point(2,1),new Point(3,1), new Point(4,1)];
      case 'E': return [new Point(2,1), new Point(2,2),new Point(2,3), new Point(2,4)];
      case 'S': return [new Point(1,1), new Point(2,1),new Point(3,1), new Point(4,1)];
      case 'W': return [new Point(2,1), new Point(2,2),new Point(2,3), new Point(2,4)];
    }
  }),
  'T': new Shape('T', rotation => {
    switch (rotation) {
      case 'N': return [new Point(1,1), new Point(1,2), new Point(2,2), new Point(1,3)];
      case 'E': return [new Point(1,2), new Point(2,2),new Point(3,2), new Point(2,1)];
      case 'S': return [new Point(1,2), new Point(2,1),new Point(2,2), new Point(2,3)];
      case 'W': return [new Point(1,1), new Point(2,1),new Point(3,1), new Point(2,2)];
    }
  }),
  'L': new Shape('L', rotation => {
    switch (rotation) {
      case 'N': return [new Point(1,1), new Point(2,1), new Point(1,2), new Point(1,3)];
      case 'E': return [new Point(1,1), new Point(1,2), new Point(2,2), new Point(3,2)];
      case 'S': return [new Point(1,3), new Point(2,1), new Point(2,2), new Point(2,3)];
      case 'W': return [new Point(1,1), new Point(2,1), new Point(3,1), new Point(3,2)];

    }
  }),
  'Z': new Shape('Z', rotation => {
    switch (rotation) {
      case 'N': return [new Point(1,1), new Point(1,2), new Point(2,2), new Point(2,3)];
      case 'E': return [new Point(1,2), new Point(2,2),new Point(2,1), new Point(3,1)];
      case 'S': return [new Point(1,1), new Point(1,2), new Point(2,2), new Point(2,3)];
      case 'W': return [new Point(1,2), new Point(2,2),new Point(2,1), new Point(3,1)];
    }
  })
};

The ‘Piece’ Type

To be able to work with instances of shapes on the game board we need a type to represent an instance of a shape with an offset and rotation. That type is Piece. Note that the rotation defaults to ‘N’, and the offset defaults to row 1 column 10 (top centre).

// an instance of a tetromino on the board
export class Piece {
  constructor(shape, offset = new Point(1,10)) {
    this.shape = shape;
    this.offset = offset;
    this.rotation = 'N';
  }
  points() {
    return this.shape.pointsRotated(this.rotation).map((point,ix) => point.add(this.offset));
  }
  static rotations() {
    return ['N','E','S','W'];
  }
}

The set of possible rotations is given by the static function rotations(). As a static function it is accessed of the type, not an instance of the type, e.g. Piece.rotations().

The points() function returns the squares (points) that need to be drawn for the piece. This is done by starting with the correct rotation of the point’s shape and then adding the Piece’s offset to each point.

The ‘Game’ Type

To coordinate the game of Tetris we introduce a new type Game. Game keeps track of the currently falling piece (there can be only one) and the rubble left by all pieces that have fallen previously. Once a piece has finished falling it can no longer be stored as a piece because Tetris allows full rows to collapse, so the Game converts fallen pieces to rubble which is just a collection of points.

export class Game {
  constructor() {
    this.rows = 15;
    this.cols = 20;
    this.rubble = [];
    this.startAPiece();
  }
  tick() {
    this.fallingPiece.offset = this.fallingPiece.offset.fallOne();
    if (this.fallingPiece.maxRow() >= this.rows) {
      this.convertToRubble();
    }
    return this;
  }
  convertToRubble() {
    this.rubble = this.rubble.concat(this.fallingPiece.points());
    this.startAPiece();
  }
  startAPiece() {
    this.fallingPiece = new Piece(shapes.selectRandom());
  }
  rotate() {
    this.fallingPiece.rotate();
    return this;
  }
}

startAPiece() is a method that intializes a new falling Piece. To choose the Shape for the new piece it uses the selectRandom() method, which randomly chooses one of the five possible Tetromino shapes:

shapes.selectRandom = function() {
  var index = Math.floor(Math.random()*1000000%5);
  return shapes[Object.keys(shapes)[index]];
}

tick() is the method that advances the game by one time unit, by moving the falling piece down by one position. This is done by changing the falling piece’s offset. If the falling piece hits the lower boundary (this.fallingPiece.maxRow() >= this.rows) then the piece is converted to rubble.

convertToRubble() adds the points from the current falling piece to the existing collection of rubble. It then delegates to startAPiece() to create a new falling piece.

The React Components

Much of the refactoring in this episode has been to move the application logic into a domain model and away from the React components. The React components can now be very simple, which is good.

From the outside in we start with GameView. This is a React component responsible for rendering the entire game.

export var GameView = React.createClass({
  render: function () {
    return <div className="border" style={{width: this.props.game.cols*25, height: this.props.game.rows*25}}>
      <PieceView piece={this.props.game.fallingPiece} />
      <RubbleView rubble={this.props.game.rubble} />
    </div>;
  }
});

The game is rendered as a div with 25 pixels for each row and column. Within the GameView there is a PieceView, which renders the current (falling) piece, and a RubbleView which renders the rubble of all pieces that have fallen previously.

PieceView has a single prop called piece. This is how we pass data into the PieceView. piece is expected to be an instance of the Piece model type.

export var PieceView = React.createClass({
  render: function () {
    return <div>
      {this.props.piece.points().map(sq => <Square key={count++} row={sq.row} col={sq.col} />)}
    </div>;
  }
});

PieceView is very simple. It extracts the points from the piece and converts each one to a Square element. Square is a component responsible for rendering a single point.

export var Square = React.createClass({
    render: function() {
            var s = {
            left: (this.props.col-1) * 25 + 'px',
          top: ((this.props.row-1) * 25) + 'px'
        };
        return <div className="square" style={s}></div>;
    }
});

The ‘App’ Component

App (app.js) has been kept simple by moving a lot of the logic into models. It has not changed much from the previous edition.

import * as React from 'react';
import * as ReactDOM  from 'react-dom';
import * as Components from './components';
import * as Model from './model';
import {createStore} from 'redux';


function reducer(state = new Model.Game(), action) {
  switch (action.type) {
    case 'TICK':
      return state.tick();
    default: return state;
  }
}

let store = createStore(reducer);
store.subscribe(() => {
  ReactDOM.render(<Components.GameView game={store.getState()} />, document.getElementById('container'));
});
setInterval(() => store.dispatch({ type: 'TICK' }),500);

The result is that we now have pieces falling one after another and stopping at the boundary of the game area.

Testing

In JavaScript at least, if you haven’t tested it you don’t know it works. I want to add some unit tests and I want to be able to verify them without a browser, therefore I choose mocha as my test runner. babel-register is an adapter that gives mocha the ability to work with ES2015 via babel.

npm install --save mocha babel-register

Mocha runs in the browser or in node.js and has a hierarchical test format that I like.

To be able to run my tests via npm, and without a global install of mocha, I add the following script to package.json.

"scripts": {
  "build": "browserify -t [ babelify --presets [ react es2015 ] ] app.js -o bundle.js",
  "test": "node_modules/mocha/bin/mocha --compilers js:babel-register"
}

Now I can run my tests with:

npm run test

but first I will need some tests. Within the test directory (mocha convention) I write my model tests in a file called modelTests.js. To start I need to import an assertion library and the module I want to test:

var assert = require('assert');
var Models = require('../model');

The first tests I add are to check that shapes have the points that I expect them to:

function pieceHasPoints(piece, points) {
  return points.every(item => piece.hasPoint(item));
}

describe('models', function () {
  describe('Piece', ()=> {
    describe('hasPoint', ()=> {
      var piece = new Models.Piece(Models.shapes.I, new Models.Point(1,1));
      it('should have (1,1)',()=> assert(piece.hasPoint(new Models.Point(1,1))));
      it('should have (2,1)',()=> assert(piece.hasPoint(new Models.Point(2,1))));
      it('should have (3,1)',()=> assert(piece.hasPoint(new Models.Point(3,1))));
      it('should have (4,1)',()=> assert(piece.hasPoint(new Models.Point(4,1))));
      it('should not have (2,2)',()=> assert(!piece.hasPoint(new Models.Point(2,2))));
      it('should not have (1,2)',()=> assert(!piece.hasPoint(new Models.Point(2,2))));
      it('should not have (3,2)',()=> assert(!piece.hasPoint(new Models.Point(2,2))));
      it('should not have (3,3)',()=> assert(!piece.hasPoint(new Models.Point(2,2))));
    });
  });

Next I check that rotation works correctly:

describe('rotation', ()=> {
    describe('general rotation', ()=> {
      it('should rotate clockwise indefinitely', ()=> {
        var piece = new Models.Piece(Models.shapes.I);
        assert.equal(piece.rotation, 'N')
        piece.rotate();
        assert.equal(piece.rotation, 'E');
        piece.rotate();
        assert.equal(piece.rotation, 'S');
        piece.rotate();
        assert.equal(piece.rotation, 'W');
        piece.rotate();
        assert.equal(piece.rotation, 'N');
        piece.rotate();
        assert.equal(piece.rotation, 'E');
        piece.rotate();
        assert.equal(piece.rotation, 'S');
        piece.rotate();
        assert.equal(piece.rotation, 'W');
        piece.rotate();
        assert.equal(piece.rotation, 'N');
      });
    });

    describe('rotating an I', ()=> {
      var piece = new Models.Piece(Models.shapes.I, new Models.Point(1,1));
      it('should have the expected points to start with',
        ()=> assert(pieceHasPoints(piece, [new Models.Point(1,1),new Models.Point(2,1),new Models.Point(3,1),new Models.Point(4,1)])));
      it('should rotate to the correct E position', ()=> {
        piece.rotate();
        assert(pieceHasPoints(piece, [new Models.Point(2,1),new Models.Point(2,2),new Models.Point(2,3),new Models.Point(2,4)]));
      });
      it('should rotate to the correct S position', ()=> {
        piece.rotate();
        assert(pieceHasPoints(piece, [new Models.Point(1,1),new Models.Point(2,1),new Models.Point(3,1),new Models.Point(4,1)]));
      });
      it('should rotate to the correct W position', ()=> {
        piece.rotate();
        assert(pieceHasPoints(piece, [new Models.Point(2,1),new Models.Point(2,2),new Models.Point(2,3),new Models.Point(2,4)]));
      });
      it('should rotate back to N', ()=> {
        piece.rotate();
        assert(pieceHasPoints(piece, [new Models.Point(1,1),new Models.Point(2,1),new Models.Point(3,1),new Models.Point(4,1)]));
      });
    });
  });

Running these tests as described produces the output:

> node_modules/mocha/bin/mocha --compilers js:babel-register

  models
    Piece
      hasPoint
        ✓ should have (1,1)
        ✓ should have (2,1)
        ✓ should have (3,1)
        ✓ should have (4,1)
        ✓ should not have (2,2)
        ✓ should not have (1,2)
        ✓ should not have (3,2)
        ✓ should not have (3,3)
    rotation
      general rotation
        ✓ should rotate clockwise indefinitely
      rotating an I
        ✓ should have the expected points to start with
        ✓ should rotate to the correct E position
        ✓ should rotate to the correct S position
        ✓ should rotate to the correct W position
        ✓ should rotate back to N

  14 passing (12ms)

Next Time…

The next installment of the series will look at handling user input so that we can allow the user to position pieces (move left and right) and trigger rotation. There is work to be done to add collision detection and improve the development workflow.

Implementing Tetris with React and Redux - III

Mon, 28 Dec 2015 20:32:00 +1000

In the previous article we began implementing tetris with React and Redux and got as far rendering the tetris shapes (called tetrominos).

In this addition we will add some animation and introduce Redux for managing UI state.

To follow along with this article checkout the commit 9f2c69.

$ git clone https://github.com/liammclennan/tetris.git
$ git checkout 9f2c69

Introducing Redux

React provides two features:

It generates a UI from a model and maintains the synchronization.
It provides an eventing system for detecting and handling UI events.

By itself then, React is not a sufficient basis for a user interface. Facebook filled the gap with Flux. Flux has the right idea (unidirectional data flow) but over-complicates it with actions, dispatchers and stores. It may be a good design for Facebook, but it is not a good generalized solution for web development.

Redux is a simpler library that started with the ideas of Flux and then left them all out. It provides management of UI state, and the ways in which that state is modified. Here is how it works.

React events are translated to actions. Actions are application level values indicating the users intent, e.g. add todo item or remove calendar entry. Actions often have associated data, so an actual action might be {type:'ADD_TODO_ITEM', title:'Buy some milk'}. There are no rules about the format of actions.
Redux is configured with functions called reducers that apply actions to the current state, and produce an updated state.
Redux notifies React that the UI state has changed and the UI should be updated.

To add Redux we install two new npm packages:

npm install --save redux react-redux

Then in app.js we create a redux reducer function that knows how to apply actions to the UI state. This is often a switch statement that branches on the type of the action. If you wanted you could instead put the logic in the action and delegate to it.

function reducer(state = [new Model.O(1,1), new Model.L(1,4)], action) {
  switch (action.type) {
    case 'TICK':
      state.push(new Model.O(action.data*2,action.data*2));
      return state;
    default: return state;
  }
  return state;
}

The parameters to the function are the existing state and an action. We use the ES2015 default parameter feature to initialize the state to an array containing two tetrominos. Then we switch on the type of the action. If the action is a TICK then we append a new O tetromino to the state. The value returned from a reducer is the new state.

With a fresh reducer function we can now create a redux store to wrap the UI state:

let store = createStore(reducer);

and register with react to re-render when the state changes:

store.subscribe(() => {
  ReactDOM.render(
  <div>
    {store.getState().map(c => <Components.ShapeView shape={c} />)}
  </div>,
      document.getElementById('container')
  );
});

Note that the state is extracted from the store by means of the getState() method.

Adding Some Animation

Tetris is a 2-dimensional animated game. We need some way to make the pieces move. I like to work in small increments instead of trying to jump immediately to the final solution, so I will be happy with any sort of regular UI change. The simplest thing I can think of is to generate ticks on a timer, and include an increasing value as data.

var counter = 1;
setInterval(() => store.dispatch({ type: 'TICK', data: counter++ }),1000);

Remember that store is our redux wrapper around application state. The dispatch method is how we publish an action.

This combined with our reducer means that every second a square tetromino will be added to the board diagonally to the lower right of the previous piece. The result is something like:

Next Time…

The next installment of this series will add falling, rotation and testing.

Implementing Tetris with React and Redux - II

Sun, 27 Dec 2015 20:32:00 +1000

In the previous article we began implementing tetris with React and Redux and got as far rendering the tetris shapes (called tetrominos).

To follow along with this article checkout the commit 594a735.

$ git clone https://github.com/liammclennan/tetris.git
$ git checkout 594a735

Simplifying the Components

Thinking ahead it becomes apparent that we will need logic related to the shapes, such as how to rotate them and prevent collisions. If we introduce models for the tetrominos then they will be represented twice, once as models and once as React components.

As we don’t want to repeat ourselves I think it is appropriate to simplify the React components. Instead of having different components for each tetromino all we really need is one component that can render a square. At the same time I will begin to introduce ES2015 modules. My new module, called components, will be in a file called components.js. Anything that I want to be available outside of the module needs to be prefixed with the export keyword. Instead of loading React using the commonjs require function we will switch to using the import keyword from ES2015. The new components module, and the new ShapeView component looks like:

import * as React from 'react';
import * as ReactDOM from 'react-dom';

export var ShapeView = React.createClass({
  render: function () {
    return <div>
      {this.props.shape.squares().map(sq => <Square row={sq.row} col={sq.col} />)}
    </div>;
  }
});

The import * as React from 'react' statement means that we are importing the react module and assigning all of its exports to the identifier React.

The refactored ShapeView component expects a model called shape having a method squares() that returns a collection of points that defines the shape. Thus the component can be reused for any shape that can describes its points.

Introducing Models

Having recognised that we need models to encapsulate the Tetris logic I will create another module, called models. The file models.js contains classes that model some tetrominos (O and L) and a class that models the Squares from which the other shapes are composed.

export class Sqr {
  constructor(row,col) {
    this.row = row;
    this.col = col;
  }
}

export class O {
  constructor(row,col) {
    this.row = row;
    this.col = col;
  }
  squares() {
    return [new Sqr(this.row,this.col), new Sqr(this.row, this.col+1), new Sqr(this.row+1,this.col), new Sqr(this.row+1,this.col+1)];
  }
}

export class L {
  constructor(row,col) {
    this.row = row;
    this.col = col;
  }
  squares() {
    return [new Sqr(this.row,this.col), new Sqr(this.row+1, this.col), new Sqr(this.row+2,this.col), new Sqr(this.row+3,this.col)];
  }
}

As required by the ShapeView component the O and L classes both have a squares() method returning an array of Sqr.

Putting it Together

Now we can simplify out app.js. Firstly, import the required modules so that they are available within app.js:

var React = require('react');
var ReactDOM = require('react-dom');
var Components = require('./components');
var Model = require('./model');

Note that I have not yet switched this file to use the ES2015 style imports.

Next, I create an object to act as the model for React to render:

var data = [new Model.O(1,1), new Model.L(1,4)];

For now, the model is an array of tetrominos, one O and one L.

The rendering is now simpler. All we need to do is loop over the total set of squares and render each.

ReactDOM.render(
<div>
  {data.map(c => <Components.ShapeView shape={c} />)}
</div>,
    document.getElementById('container')
);

In the next article we will start animating the tetrominos and introduce Redux as a tool for managing state.

Implementing Tetris with React and Redux - I

Tue, 22 Dec 2015 20:32:00 +1000

I cannot think of a game more classic than tetris. When I was a kid it was the staple diet of gameboy wielding preteens. Therefore, I decided to reimplement tetris using javascript, React and Redux.

The Cast

JavaScript (ES2015)

At first I started with TypeScript, but I gave up getting it to work with all the other tooling (type definitions, webpack, React etc) and switched to ES2015. ES2015 is not widely supported so I must compile it to ES5. Since I’m using React (with JSX) I’m obliged to have a compile step anyway.

React

React is a UI library that knows how to keep a web UI in sync with a model object, and how to dispatch events.

Redux

Redux is a simple library that puts some structure around UI state and manages the mutation of that state.

Babel

Babel is the defacto standard tool for ES2015 -> ES5 compilations.

Browserify

One of the ES2015 features is modules, but to take advantage of ES2015 modules, to include other modules from NPM and to improve the network efficiency of the application Broswerify converts the modules into something that works in the browser and smooshes everything into a single file.

Mocha

Mocha is a node-based JavaScript testing tool. It provides a library for logically organising tests and a runner for running the tests and reporting the results.

Gulp

I use gulp for my build script. It coordinates all the other tools and provides file system watching.

What I Hate About All This

The number of fiddly little tools required to work with JavaScript at the moment annoys me. Each in isolation I understand their purpose but considering the ecosystem as a whole it is obviously an appalling waste of time having to deal with all this bits. I’m sure it will settle down with time. Can’t wait.

The most pernicious issue is not the quantity of tools required, but the tight coupling between them. Browserify requires a plugin (babelify) to connect it to babel. Mocha requires a plugin (babel-register) to connect it to babel. If I wasn’t so lazy as to have gulp execute shell scripts then I would require gulp plugins specific to mocha, browserify and possible babel.

Step 1 - Basic tooling

You may, if you wish, follow along with my git commits. This step is commit dffea31bfee670f4644eaf8d0586a81f576fd548.

To get to this state:

$ git clone https://github.com/liammclennan/tetris.git
$ git checkout dffea31bfee670f4644eaf8d0586a81f576fd548

My application starts with a simple, empty, web page:

<!DOCTYPE html>
<html>
  <head>
    <link rel="stylesheet" type="text/css" href="style.css">
  </head>
  <body>
    <div id="container">
    <!-- This element's contents will be replaced with your component. -->
    </div>
    <script src="bundle.js"></script>
  </body>
</html>

Note that the webpage includes just two files: style.css and bundle.js. The entire application, including all of my javascript and all required libraries, is bundled into bundle.js by browserify. I know you are wondering how, so let’s inspect package.json:

{
  "name": "tetris",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "build": "browserify -t [ babelify --presets [ react ] ] app.js -o bundle.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "babel-preset-react": "^6.3.13",
    "babelify": "^7.2.0",
    "react": "^0.14.3",
    "react-dom": "^0.14.3"
  },
  "devDependencies": {
    "babel-preset-es2015": "^6.3.13",
    "babelify": "^7.2.0"
  }
}

Package.json is the metadata file for my NPM package. It includes a list of dependencies and the optional ability to execute some scripts. I have defined a build script as:

browserify -t [ babelify --presets [ react ] ] app.js -o bundle.js

This tells browserify to transform the input via babelify, and when delegating to babel to use the react preset. app.js is the root of my application. Browserify will start their and spider through all the included modules. -o bundle.js sets the output file. To execute this build script:

npm run build

Step 2 - A Little Bit of Tetris

The game tetris involves the placement of tetrominos.

A tetromino is a geometric shape composed of four squares, connected orthogonally.

How many tetrominos can you think of? Well there are 5 (I, O, Z, T, L).

React uses components to organise UI elements. The fundamental requirement for a component is that it must have a render method that defines how a model object is converted into UI. To assemble Tetrominos we can start by defining a component for a single square (in app.js):

var Square = React.createClass({
    render: function() {
            var s = {
            left: (this.props.col-1) * 25 + 'px',
          top: (this.props.row-1) * 25 + 'px'
        };
        return <div className="square" style={s}></div>;
    }
});

This component converts a simple model of the form {col,row} into a div with the class square positioned absolutely according to the row and col values. With this low-level component we can assemble higher level components for the different tetrominos:

var OShape = React.createClass({
    render: function () {
      return <div>
        <Square row={this.props.row} col={this.props.col} />
        <Square row={this.props.row} col={this.props.col+1} />
      <Square row={this.props.row+1} col={this.props.col} />
          <Square row={this.props.row+1} col={this.props.col+1} />
    </div>;
  }
});

var LShape = React.createClass({
    render: function () {
      return <div>
        <Square row={this.props.row} col={this.props.col} />
          <Square row={this.props.row+1} col={this.props.col} />
      <Square row={this.props.row+2} col={this.props.col} />
          <Square row={this.props.row+3} col={this.props.col} />
    </div>;
  }
});

var SShape = React.createClass({
    render: function () {
      return <div>
        <Square row={this.props.row} col={this.props.col+1} />
          <Square row={this.props.row} col={this.props.col+2} />
      <Square row={this.props.row+1} col={this.props.col} />
          <Square row={this.props.row+1} col={this.props.col+1} />
    </div>;
  }
});

We can then use React to render some tetrominos into the HTML page we defined earlier:

ReactDOM.render(
<div>
    <OShape row={1} col={1} />
  <LShape row={1} col={4} />
  <SShape row={1} col={6} />
</div>,
    document.getElementById('container')
);

And with that we have the start of the rendering portion of our tetris game! We have done some ES2015 development with modern tooling, and a little bit of React. Redux will enter the picture in a future post.

If you are following along you can build this using:

npm install
npm run build

Then open index.html in your browser.

SqlDoc

Tue, 01 Dec 2015 20:32:00 +1000

One year ago I built an application data storage library, called PostgresDoc. The intent was, and is, to support document db semantics on top of relational databases. I have since renamed the project to SqlDoc to acknowledge that it now works with Postgresql and Sql Server.

The strengths of SqlDoc compared to a dedicated document db are:

works with your existing database server
proper, multi-document transactions
supports joins
supports a hybrid document / relational model
easy to project to flat views for reporting and ETL

The strengths of SqlDoc compared to a relational database are:

simple programming model
No object / relational mapping

New API

I heard that people like the IDocumentSession API from RavenDB so I’ve added it to SqlDoc. This will make life easier for people who want to do data access via a component provided from an IoC container.

Creating a session

var _documentSession =
  new DocumentSession<Guid>(SqlConnection.From(connectionString));

Adding a document

var _aDocument = new PersonCs
{
    _id = Guid.NewGuid(),
    Name = "Docsesh",
    Age = 90,
    FavouriteThings = new[] { "Golf", "Statue of liberty" }
};
_documentSession.Store(_aDocument._id, _aDocument);

Updating a document

_aDocument.Age += 1;
_documentSession.Update(_aDocument._id, _aDocument);

Deleting a document

_documentSession.Delete(_aDocument._id, _aDocument);

Persisting the unit of work

_documentSession.SaveChanges();

Load by id

var fresh = _documentSession.Load<PersonCs>(id);

Load by query

var result = _documentSession.Query<PersonCs>(
"select data from PersonCs where Data.value('(/FsPickler/value/instance/idkBackingField)[1]', 'uniqueidentifier') = @id",
new Dictionary<string, object> { { "id", _aDocument._id } });

Why you don’t need to put IDocumentSession in an IoC container

Rather than making your application code take a dependency on IDocumentSession consider instead having your application code returns its results, and then do your data access. That way there is a neat separation between pure application code responsible for logic and calculation and external integration code that does integration with external things like file systems and databases.

Here is the traditional way:

public class MyApp
{
  public MyApp(IDocumentSession documentSession) {}

  public void DoThing() {
    documentSession.Store(new Thing());
  }
}

To test that we need to mock documentSession.

The alternative is to keep the persistence plumbing out of your application:

public class MyApp
{
  public static Operation DoThing() {
    var thing = new Thing();
    return Operation.Insert(thing.Id, thing);
  }
}

// consumer
UnitOfWork.Commit(connection, new Queue<Operation>(MyApp.DoThing()));

This separates the calculation and the logical result of adding a new Thing() from the mechanism of doing so. To test DoThing() just call it and inspect the result.

Cycle.js Starter

Sun, 08 Nov 2015 20:32:00 +1000

Just getting a cycle.js app building can take some time. This repo is meant to provide a working starting point for a cycle.js app using Babel to compile ES2015 to ES5, webpack to bundle the modules into a script and gulp to orchestrate the build process.

The build produces a single bundled script in browser/bundle.js.

Get Started

git clone https://github.com/liammclennan/cycle-starter.git

Install gulp:

npm install -g gulp

Install dependencies:

npm install

Build:

gulp

Then open index.html in your browser.

Nokia Ringtone Composer Emulator

Wed, 28 Oct 2015 20:32:00 +1000

I’ve been working on a Pluralsight F# course for some time. The sample project I chose is to build a web-based emulator of the classic Nokia 3310 composer application that allowed Nokia owners to program their own ringtones. I have previously written about building the tone generator and parsing the ringtone syntax with FParsec.

The final step was to wrap the composer application in a simple web page, and so

my Nokia Composer Emulator is now online!

Using FSharpX to make it easier to work with Choices

The composer module returns a Choice<MemoryStream,string> that represents either a wave file or a string error message. Choice is a type with two possible forms, a choice value is either a Choice1Of2 someMemoryStream or Choice2Of2 "error message".

To work with a choice we pattern match:

    match aChoiceValue with
        | Choice1Of2 stream -> doSomethingWith stream
        | Choice2Of2 error -> printfn "Error: %s" error

I got sick of constantly pattern matching the choice, doing some calculation and repacking a choice, so I looked to FSharpx for some help. For my scenario I want do something with the successful value if there is one and pass straight through if there has been an error. FSharpX provides Choice.map for this purpose.

Code goes from

let assembleToPackedStream (score:string) =
    match parse score with
        | Choice1Of2 tokens -> Choice1Of2 <| pack(assemble tokens)
        | Choice2Of2 error -> Choice2Of2 error

let assembleToPackedStream (score:string) =
    Choice.map (fun tokens -> pack(assemble tokens)) (parse score)

If, when processing the success value, I wanted to be able to produce an error I could have used bind instead of map:

let assembleToPackedStream (score:string) =
    Choice.bind (fun tokens ->
                    if thereIsAnError then
                        Choice2Of2 "error message goes here"
                    else
                        Choice1Of2 (pack(assemble tokens)))
        (parse score)

More commonly people use the >>= infix operator for the bind operation.

let assembleToPackedStream (score:string) =
    (parse score) >>= (fun tokens ->
                        if thereIsAnError then
                            Choice2Of2 "error message goes here"
                        else
                            Choice1Of2 (pack(assemble tokens)))

F# Web Application

For the web application I use the excellent asp.net mvc 5 and web api 2 F# project template.

The action I added to handle the composition post converts an error result to an exception. If the composition was created successfully then the Content-Dispostion header is set and the wave file returned.

[<HttpPost>]
   member this.Produce(score:string) =
       match Assembler.assembleToPackedStream score with
           | Choice1Of2 ms ->
               this.Response.AppendHeader(
                   "Content-Disposition",
                   (ContentDisposition(FileName = "sickringtone.wav", Inline = false)).ToString())
               ms.Position <- 0L
               this.File(ms, "audio/x-wav")
           | Choice2Of2 err -> failwith err

In the past I have also used the nancy F# templates which are good.

The Future of The Web

Thu, 08 Oct 2015 20:32:00 +1000

I confess to a nostalgia for the ol’ days of web development, when applications ran on the server, where they belong. To ensure that I am on the right side of history I acknowledge that JavaScript UIs are now defacto standard and therefore turn my attention to making them nicer to build.

Where are we now?

The image above visualizes the shift to richer user interfaces, and the added technology burden that we have accepted in exchange. I propose that we have given to much ground on complexity, and done so unnecessarily.

The way forward requires a liberal dose of humility. We must acknowledge our human limitations and compensate by aggresively pursuing simple solutions. If you don’t believe me listen to Dijkstra:

“We shall do a much better programming job, provided that we approach the task with a full appreciation of its tremendous difficulty, provided that we stick to modest and elegant programming languages, provided that we respect the intrinsic limitations of the human mind and approach the task as Very Humble Programmers.”

The Humble Programmer

The humble programmer recognizes that simple is more important than easy, and more important than feeding our egos.

The Humble Programmer is not pleased

Poor Richard’s Web Architecture of the Future

From the tar pit of web development tools a concensus is slowly emerging of what a simple web application design of the future might be. This design, sometimes called MVI, consists of three components: a model, a view and intents.

The Model

The model is a single data structure containing the entire user interface state for the application. This must include everything required to describe the user interface, including things like highlighted controls, and scroll position.

The View

Given an exhaustive model as described above the view can be reduced to a simple function, from the model to a user interface.

Intents

Intents, sometimes called actions, are simple values that represent things that the user wants to do. Typically, intents are generated by events, e.g. a button click event may produce a ‘user wants to add an item to a list’ intent. Intents are then processed as updates to the model. When the model changes the view is regenerated.

MVI from First Principles

A good way to understand something is to build one, so let us create an MVI application from scratch.

The application state is a random number (between 0 and 10). The view renders that number in a heading. Clicking on the heading indicates that the user intends to see the square of the number. The ‘square’ intent is processed by squaring the number in the application state. Because the application state has changed the view is re-rendered.

More Info

The Elm Architecture Tutorial is a good read, describing many of the same ideas mentioned here.

Cycle.js is another implementation worth exploring. To my knowledge it is the source of the MVI terminology.

Redux is a “state container for JavaScript apps”, meaning it holds the state object and controls the application of intents/actions to it. React provides an implementation of the view component. Redux is designed to work with React and together they complete the MVI design.

Dijkstra’s Final Thoughts

“The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility”

Product Review - Kogan Fortis Magnetic Indoor Bicycle Trainer

Mon, 14 Sep 2015 20:32:00 +1000

I’m reviewing the fortis magnetic indoor bicycle trainer because no one else has. I have only used mine twice so I can’t comment on longevity and it is my first turbo trainer so I don’t know how it compares to others. Here is what I do know.

Assembly is straightforward - simply mount the resistance thingo on the frame, and swap the rear quick release skewer on your bike for the supplied one that fits the frame mounts. I’m not sure if this skewer is suitable for regular riding, and I’m not game to find out at 70km/h down a hill, so I swap my regular skewer back in before hitting the road. It is simple to mount the bike but important that the surface be perfectly level. A tightening bolt is then used to secure the roller against your bike’s back wheel. Resistance is controlled via a handlebar mounted dial.

The good

The device seems reasonably sturdy and works as advertised. I get exercise. Assembly was simple.

The less good

There are 6 resistance levels and none of them are very difficult. I think a good cyclist may struggle to struggle. The resistance has a lot of inertia, so it is difficult at first but becomes easier with momentum. It feels like you are spinning up a flywheel. Very much the opposite of actual cycling.

Learning Cycle.js

Thu, 03 Sep 2015 20:32:00 +1000

Cycle.js is clearly React inspired, but it attempts to fix some areas where React is deficient, or weird. It also advocates (forces?) a functional style, composing immutable state via observables.

Even with a React background the learning curve is steep. I started to see how it is possible to reason about Cycle.js applications when I saw the refactoring to the Intent-Model-View pattern. See jsbin.

This post includes graphics and code samples from the Cycle.js website.

Intent

Intent is typically a function from sources of input (drivers) (DOM, HTTP, etc) to a set of observables, named according to the user’s intent (e.g. changeHeight$, receiveUserDetails$).

function intent(DOM) {
  return {
    changeWeight$: DOM.get('#weight', 'input')
      .map(ev => ev.target.value),
    changeHeight$: DOM.get('#height', 'input')
      .map(ev => ev.target.value)
  };
}

Model

Model is a function from intents (produced by the Intent function) to a state$ observable. The latest value of the state$ observable is the current state of the component. state$ is observed by the view, so changes to state$ cause changes to the view. Earlier examples had the state spread across multiple observables (e.g. one for the DOM, one for each possible HTTP request). Consolidating state into a single observable takes some work but makes the component easier to reason about and the view easier to implement.

function model(actions) {
  return Cycle.Rx.Observable.combineLatest(
    actions.changeWeight$.startWith(70),
    actions.changeHeight$.startWith(170),
    (weight, height) =>
      ({weight, height, bmi: calculateBMI(weight, height)})
  );
}

View

View is a function from the model observable (state$) to a vdom observable. This is equivalent to React’s Render function.

function view(state$) {
  return state$.map(({weight, height, bmi}) =>
    h('div', [
      renderWeightSlider(weight),
      renderHeightSlider(height),
      h('h2', 'BMI is ' + bmi)
    ])
  );
}

main()

A cycle.js application includes a function that represents the computer side of the human-computer interaction, often called main. It is a function from input drivers (http responses, DOM events) to output drivers (http requests, vdom structure). The Get Random User example shows working with two drivers (HTTP and DOM).

function main({DOM}) {
  return {DOM: view(model(intent(DOM)))};
}

Cycle.run()

An application is started by calling Cycle.run() specifying the main function and the required drivers. E.g.

Cycle.run(main, {
  DOM: makeDOMDriver('#app'),
  HTTP: makeHTTPDriver()
});

Ten Lessons From a Large IoT Project

Wed, 02 Sep 2015 20:32:00 +1000

Lesson 1: Big data tools don’t solve the intractable problems of distributed system. They leak complexity to the consumer.

Lesson 2: Peak performance often relies on even distribution across partitions. Hashing keys can help. Try SHA256.

Lesson 3: To prevent DOS attacks public cloud components like Event Hub throw exceptions if you exceed quotas. The client must take responsibility for throttling.

Lesson 4: Azure web jobs are brilliant. They provide a way to run a console application as a reliable service that can be scaled horizontally. Web jobs are simple and they work.

Lesson 5: Distributed logs are a useful data structure for distributed systems.

Lesson 6: To understand what is happening in a system you need centralised logging with a correlation scheme.

Lesson 7: As metrics get larger we lose the ability to reason about them. 50/500/5000/50000 requests/s are very different. Those differences matter.

Lesson 8: Very large datasets (PB) drastically reduce data storage options.

Lesson 9: Consistency in the absence of transactions is hard. You will have to think about it.

Lesson 10: All actions must be idempotent. Work in batches. If an error occurs during a batch the idempotency allows the batch to be replayed.

Lesson 11: Tuning batch size substantially affects performance. Large batches and low error rates are the sweet spot. Higher error rates require smaller batches.

Processing HBASE data with Rx

Mon, 27 Jul 2015 20:32:00 +1000

HBase is good at many things but adhoc querying is not one of them. You can query by key, or you can scan over a range of keys. Sometimes it is handy to be able to monitor a table and process rows as they are added. This kind of processing over a stream is what Reactive Extensions (Rx) is good at, so why not combine hbase querying with Rx Observables?

I’ll be using the Microsoft Hbase client and an Azure HBase cluster. What I want is to convert an Hbase table into an Rx observable, so that I can filter, transform, group and fold. Step 1 is to fetch my observable:

var hotRows = HBaseToObservable.Build(new Uri("https://redacted.azurehdinsight.net"), "????", "??????",
"tablename","start_key")
.Select(r => new {
key = Encoding.UTF8.GetString(r.key),
data = r.values.ToDictionary(
  v => Encoding.UTF8.GetString(v.column),
  v=> Encoding.UTF8.GetString(v.data))});

hotRows is now an observable reading forward over the hbase table tablename from the key start_key. As new records are appended to the table they will be pushed through the observable.

Why do we want an observable view of an hbase table?

This is most useful for a table storing data such as timeseries data, where we want a live, forward-only stream. With an observable we can do things like:

hotRows
  .Where(r => r.data["some_column"] == "whatever")
  .GroupBy(r => r.data["colour"])
  .Select(r => r.data["radius"] * Math.PI);

How is it implemented?

In short, it creates an observable, and an hbase table scanner, and pushes rows to the observable. Once it reaches the end of the table it polls every three seconds for new rows.

public static class HBaseToObservable
{
    public static IObservable<CellSet.Row> Build(
        Uri server,
        string username,
        string password,
        string tableName,
        string startKey
        )
    {
        var creds = new ClusterCredentials(server, username, password);
        var client = new HBaseClient(creds);
        var rowsOut = new ReplaySubject<CellSet.Row>();

        Task.Run(()=>{
            GetRows(rowsOut,client,tableName,startKey);
        });
        return rowsOut;
    }

    private static void GetRows(ReplaySubject<CellSet.Row> rowsOut,
        HBaseClient client,
        string tableName,
        string startKey) {
        CellSet next = null;
        string lastKey = "";

        var scannerInfo = client.CreateScanner(tableName, new Scanner()
        {
            batch = 100,
            startRow = Encoding.UTF8.GetBytes(startKey),
            endRow = BitConverter.GetBytes(int.MaxValue)
        });
        while ((next = client.ScannerGetNext(scannerInfo)) != null)
            {
                foreach (var row in next.rows) {
                    lastKey = Encoding.UTF8.GetString(row.key);
                    rowsOut.OnNext(row);
                }
            }
        Task.Delay(TimeSpan.FromSeconds(3))
            .ContinueWith((t)=>GetRows(rowsOut,client,tableName,lastKey));
    }
}

Dealing With Unreliable TPL Tasks

Wed, 22 Jul 2015 20:32:00 +1000

Suppose you have a collection of .net TPL tasks and the knowledge that some of them may fail (throw an exception) and some may take an unacceptably long period of time to resolve. In the case of the errored tasks you wish to collect information to assist diagnosis. In the case of the long running tasks you wish to ignore them and allow the program to carry on so that throughput is maintained.

This is not a simple problem when using the .net TPL (Task, async, await, etc), which seems to be designed with an all or nothing mentality. Consider the following motely crew of tasks:

Task<int?>[] calcTasks = new [] {
    Task.Run(() => {
        Thread.Sleep(200);
        var div = 0;
        return new Nullable<int>(5/div);
    }),
    Task.Run(() => {
        Thread.Sleep(100);
        return new Nullable<int>(42);
    }),
    Task.Run(() => {
        Thread.Sleep(1500);
        return new Nullable<int>(99);
    })
};

All three are asynchronous operations including a time delay. The first task attempts to divide by zero and will therefore throw a DivideByZero exception. The last task sleeps for 1.5s, which exceeds the acceptable timeout for my use case. Only the second task will succeed, ultimately producing the value 42.

What is needed now is a function to process a collection of tasks, in the context of a particular timeout duration, and return a collection of results. Each result should describe the fate of one task. The class I use to represent the result of a Task that may error or timeout is AsyncTimedResult<TResult> described below:

public enum AsyncResultStates
{
    Unknown, TimedOut, Resolved, Faulted
}

public class AsyncTimedResult<TResult>
{
    public TResult Result { get; private set; }
    public Exception Error { get; private set; }
    public AsyncResultStates State { get; private set; }
    public string Message { get; private set; }

    protected AsyncTimedResult()
    {
        Message = "";
        State = AsyncResultStates.Unknown;
    }

    ...

To process a collection of unreliable tasks I use a function called ExtractResults, like so:

AsyncTimedResult<int?>[] results = await
    UnreliableTasks.ExtractResults(calcTasks, TimeSpan.FromSeconds(1));

I can then categorize the results by their state:

var succeeded = results.Where(UnreliableTasks.IsResolved);
var faulted = results.Where(UnreliableTasks.IsFaulted);
var timedout = results.Where(UnreliableTasks.IsTimedOut);

How Does it Work?

The TPL doesn’t include any support for adding timeouts to tasks. We simulate timeouts with a bit of a hack. For each task that we wish to evaluate against a timeout, we create a new task that will resolve at the timeout limit, then we race the two tasks using Task.WhenAny() which results in the first task to resolve. If the timeout task resolves first then we know that our task timed out.

var taskICareAbout = Task.Run(() => {
    Thread.Sleep(200);
    var div = 0;
    return new Nullable<int>(5/div);
});
var timerTask = Task.Delay(TimeSpan.FromMilliseconds(100))
  .ContinueWith(t => default(int?));
var first = await Task.WhenAny<int?>(taskICareAbout, timerTask);
if (first == taskICareAbout) {
  // my task completed first
} else {
  // timer task completed first
}

timerTask has a continuation just to convert it to the same type as our other task. For ExtractResults I also need to process the results and categorize them by state. The full implementation is:

public static async Task<AsyncTimedResult<T>[]> ExtractResults<T>(IEnumerable<Task<T>> tasks, TimeSpan timeout)
{
    var tasksWithTimeouts = tasks.Select(t =>
        Task.WhenAny<T>(Task.Delay(timeout).ContinueWith(t2 => default(T)), t));
    var everything = Task.WhenAll(tasksWithTimeouts)
        .ContinueWith(t => t.Result.Select(ti =>
        {
            if (ti.IsFaulted || ti.Exception != null)
            {
                return AsyncTimedResult<T>.Reject(ti.Exception);
            }
            if (ti.Result == null || ti.Result.Equals(default(T)))
            {
                return AsyncTimedResult<T>.TimeOut();
            }

            return AsyncTimedResult<T>.Resolve(ti.Result);
        }));
    var results = await everything;
    return results.ToArray();
}

All Together

Here is a complete console app demonstrating the method:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace TasksC
{
    class Program
    {
        static void Main(string[] args)
        {
            Do().Wait();
        }

        private static async Task Do()
        {
            Task<int?>[] calcTasks = new[] {
                Task.Run(() => {
                    Thread.Sleep(200);
                    var div = 0;
                    return new Nullable<int>(5/div);
                }),
                Task.Run(() => {
                    Thread.Sleep(100);
                    return new Nullable<int>(42);
                }),
                Task.Run(() => {
                    Thread.Sleep(1500);
                    return new Nullable<int>(99);
                })
            };

            var results = await UnreliableTasks.ExtractResults(calcTasks, TimeSpan.FromSeconds(1));
            var succeeded = results.Where(UnreliableTasks.IsResolved);
            var faulted = results.Where(UnreliableTasks.IsFaulted);
            var timedout = results.Where(UnreliableTasks.IsTimedOut);

            Console.WriteLine(string.Join(",", succeeded.Select(n => n.Result)));
            Console.WriteLine("\nErrors:\n");
            Console.WriteLine(string.Join(",", faulted
                .Select(n => n.Error.InnerException.Message).ToArray()));
            Console.WriteLine("\nTimeouts:\n");
            Console.WriteLine(timedout.Count());
            Console.ReadLine();
        }
    }

    public class UnreliableTasks
    {
        public static async Task<AsyncTimedResult<T>[]> ExtractResults<T>(IEnumerable<Task<T>> tasks, TimeSpan timeout)
        {
            var tasksWithTimeouts = tasks.Select(t =>
                Task.WhenAny<T>(Task.Delay(timeout).ContinueWith(t2 => default(T)), t));
            var everything = Task.WhenAll(tasksWithTimeouts)
                .ContinueWith(t => t.Result.Select(ti =>
                {
                    if (ti.IsFaulted || ti.Exception != null)
                    {
                        return AsyncTimedResult<T>.Reject(ti.Exception);
                    }
                    if (ti.Result == null || ti.Result.Equals(default(T)))
                    {
                        return AsyncTimedResult<T>.TimeOut();
                    }

                    return AsyncTimedResult<T>.Resolve(ti.Result);
                }));
            var results = await everything;
            return results.ToArray();
        }

        public enum AsyncResultStates
        {
            Unknown, TimedOut, Resolved, Faulted
        }

        public static bool IsResolved<T>(AsyncTimedResult<T> result)
        {
            return result.State == AsyncResultStates.Resolved;
        }

        public static bool IsFaulted<T>(AsyncTimedResult<T> result)
        {
            return result.State == AsyncResultStates.Faulted;
        }

        public static bool IsTimedOut<T>(AsyncTimedResult<T> result)
        {
            return result.State == AsyncResultStates.TimedOut;
        }

        public class AsyncTimedResult<TResult>
        {
            public TResult Result { get; private set; }
            public Exception Error { get; private set; }
            public AsyncResultStates State { get; private set; }
            public string Message { get; private set; }

            protected AsyncTimedResult()
            {
                Message = "";
                State = AsyncResultStates.Unknown;
            }

            public static AsyncTimedResult<TResult> Resolve(TResult result)
            {
                return new AsyncTimedResult<TResult>() { Result = result, State = AsyncResultStates.Resolved };
            }

            public static AsyncTimedResult<TResult> Reject(Exception ex)
            {
                return new AsyncTimedResult<TResult>() { Error = ex, State = AsyncResultStates.Faulted };
            }

            public static AsyncTimedResult<TResult> TimeOut()
            {
                return new AsyncTimedResult<TResult>() { State = AsyncResultStates.TimedOut };
            }
        }
    }
}

Deploying custom services as Azure Webjobs

Tue, 23 Jun 2015 20:32:00 +1000

Azure has this thing called webjobs. It is a curious way of deploying continous or scheduled services as sub-processes of an Azure website. This may be a desirable option because:

it’s a PaaS thing. You don’t have to manage a server.
you get configurable scale - including autoscaling
you can run many logical services within a single management unit (Azure Website)

I did not enjoy my time with the MSDN documentation for Azure Webjobs. They document the manual process for working with Webjobs, either the Visual Studio Azure tooling or through the Azure portal. Manual deployment processes, involving clicking lots of buttons with a mouse, a not going to be workable on my team, so I need a way to automate deployment. Also, the documentation only shows how to write services that are coupled to Azure. I want to be able to run my service locally.

Step 1: Use a console app for your service

The holy grail of .net services has been wrapping console apps for a long time, often using something like Topshelf. The goal is to get the reliability of a service for deployment but the ease of development and testing of a console app.

To create a console app for deployment as an Azure Webjob:

create a normal, regular, vanilla, wonderful console application.
create a web project
in the web projects context menu choose Add -> Existing project as Azure webjob and select your console application

Step 2: Packaging for deployment

The secret to Azure Webjob deployment is to include your executable in the web project at:

App_Data\jobs\{job type}\{job name}

That is the one bit of information to really need to know and it is conspicuously absent from the documentation.

There are a number of different ways to accomplish this goal. One is to use a post-build step like:

copy "$(SolutionDir)mywebjob\bin\$(ConfigurationName)\*.*" "$(ProjectDir)App_Data\jobs\continuous\mywebjob"

which you can add via the Build Events tab of the web project’s properties page.

Because we are using Teamcity to build and package our web project as a nuget package for deployment via Octopus Deploy we can use a nuspec file instead.

A simple nuspec file looks something like:

<?xml version="1.0"?>
<package >
  <metadata>
    <id>MyProjectName</id>
      <version>0.0.0.0</version>
      <authors>NA</authors>
    <owners>NA</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>NA</description>
    <copyright>Copyright 2015</copyright>
  </metadata>
    <files>
        <file src="..\MyWebJob\bin\release\**" target="App_Data\jobs\continuous\MyWebJob"/>
    </files>
</package>

Using Octopack in Teamcity to produce a nupkg package from this nuspec file will fail because the files element will override the default behaviour of Octopack and only include the listed files. To instruct Octopack to include all the normal project files, as well as the webjob files listed in the nuspec file, add the following element to the web project’s csproj file (in the <PropertyGroup> element):

<OctoPackEnforceAddingFiles>True</OctoPackEnforceAddingFiles>

This seems to be an undocumented feature, but it works.

Step 3: Deploy

Deploy the website to Azure Websites using your favourite method (my family likes to use Octopus Deploy 3 every day). Check in the Azure portal and you will hopefully have a Webjob listed in your Web App. If not, you can connect the Visual Studio Azure tooling to your Azure subscription and use the Service Explorer (App Service node) to see if the files have been deployed as expected.

Event Sourcing with Couchdb

Sat, 09 May 2015 20:32:00 +1000

Summary

CouchDB, with its indexed views, has some promising properties for an “event sourcing”* system.

Because views are a function of the documents in the database, and they are indexed, it seems like we can have the best of both worlds, where the authoritative data store is the historical events that have occurred, but at the same time we have an indexed projection of the events that is guaranteed to be up to date (prior to a read).

Disclaimer

I have never built an event sourced system. I expect I am doing everything wrong. I’m interested in hearing reasoned commentary. I am less interested in hearing dogma.

Read Side

The following is my current list of ‘person’ entities.

Documents
Grouping: Key	Reduce Value
[“abigailmclennan”]	`{_id: “abigailmclennan”, name: “Abigail”, age: 1}`
[“foobarlalala”]	`{_id: “foobarlalala”, name: “Noah”, age: 2}`
← Previous Page \| Rows per page: \| Next Page → Showing 1-2 of unknown rows

Write Side

But they are not stored in my database, instead the database stores the events that caused those entities to have the values they do.

In this example I have two different kinds of events:

BirthEvent - indicating that a person was born
BirthdayEvent - indicating that a person had a birthday

From these events the ‘person’ entities are calculated. To make this work in couchdb I have used the following rules:

each event has a ‘mapId’ property that is the identifier of the related entity
each type of entity has a map/reduce view. The map function produces each event with its mapId as the key. The reduce function is then passed the set of events that determine the value of an entity. The reduce function processes the events and returns the current state of the entity.
each event has a timestamp property. To hydrate state from events we need to know the order in which the events occurred.

In my database I have two BirthEvents, and three BirthdayEvents.

The process is something like this:

CouchDB View Code

The map function simply keys the entities by their mapId:

function(doc) {  
  if (doc.mapId) {
    emit([doc.mapId], doc);
  }
}

This means that the values object within the reduce function is the set of events that produce a single person entity.

The reduce function then orders the events, by timestamp, and processes them to calculate the current state of the entity:

function(keys, values) {
  values.sort(function (f,s) {
    if (f.timestamp > s.timestamp) return 1;
    if (f.timestamp == s.timestamp) return 0;
    return -1;
  });
  var acc = values.reduce(function (p,c,i) {
    if (c.type=='BirthEvent') {
      return {
        _id:c.mapId,
        name:c.name,
        age:0
      };
    }
    if (c.type == 'BirthdayEvent') {
      p.age +=1;
      return p;
    }
    return c;
  },{});
  return acc;
}

Result

There are some concerns remaining.

Each event must somehow map to the entity it produces. A more complicated scheme could be used (to allow an event to affect the state of > 1 entity) but I’m still not sure if this is ideal.

The reduce function may (or may not) have performance problems. I have not found a way to store the sorted events in the index, so they are re-sorted when the reduce function runs.

The reduce function is a function from a set of events to the entity those events produce. This could become complicated.

Entity state is a function of the events that produce the entity, meaning that all entity data must be derivable from the events. I’m not sure if this is good or bad.

Similar Systems

One could do the same thing with Sql Server materialized views (except the processing code would have to be written in T-SQL) or with a streaming system (Kafka/Samza/Storm/etc).

* not certain what this is.

RedisMQ

Wed, 29 Apr 2015 20:32:00 +1000

Many people use Redis for a simple message broker. RedisMQ is a trivial layer on Redis and StackExchange.Redis that probably does not work. It provides two simple messaging patterns:

A publisher publishes messages. 0, 1 or more consumers subscribe to the messages.

var redis = ConnectionMultiplexer.Connect("192.168.85.128");
redis.BroadcastSubscribe<string>("this-is-the-channel", message =>
    // do something with the message);
redis.BroadcastPublish("this-is-the-channel", "42");

Competing consumer

A publisher publishes messages to a queue. Messages stay in the queue until someone reads them. Many subscribers may read from the same queue. Each message is processed by just one consumer.

var redis = ConnectionMultiplexer.Connect("192.168.85.128");
redis.CompetingConsumerSubscribe<int>("numbers", message =>
    // do something with the message);
redis.CompetingConsumerPublish("numbers", 1);

Speed

RedisMQ is not optimised for performance. It is optimized for being simple for me to write. However, the following gives a rough indication of performance.

Strategy	# messages	Time (s)
Broadcast to 2 consumers	100,000	21.3
Competing consumer with 1 subscriber	100,000	37.7
Competing consumer with 2 subscribers	100,000	23.7

Dependencies

PM> Install-Package StackExchange.Redis
PM> Install-Package Newtonsoft.Json

Mars Rover Problem

Fri, 17 Apr 2015 20:32:00 +1000

Mars Rovers is a simple programming problem with just enough complexity to be interesting and to provide sufficient challenge to experiment with some different strategies. The problem, as stolen from here, is:

A squad of robotic rovers are to be landed by NASA on a plateau on Mars.

This plateau, which is curiously rectangular, must be navigated by the rovers so that their on board cameras can get a complete view of the surrounding terrain to send back to Earth.

A rover's position is represented by a combination of an x and y co-ordinates and a letter representing one of the four cardinal compass points. The plateau is divided up into a grid to simplify navigation. An example position might be 0, 0, N, which means the rover is in the bottom left corner and facing North.

In order to control a rover, NASA sends a simple string of letters. The possible letters are 'L', 'R' and 'M'. 'L' and 'R' makes the rover spin 90 degrees left or right respectively, without moving from its current spot.

'M' means move forward one grid point, and maintain the same heading.

Assume that the square directly North from (x, y) is (x, y+1).

Input:

The first line of input is the upper-right coordinates of the plateau, the lower-left coordinates are assumed to be 0,0.

The rest of the input is information pertaining to the rovers that have been deployed. Each rover has two lines of input. The first line gives the rover's position, and the second line is a series of instructions telling the rover how to explore the plateau.

The position is made up of two integers and a letter separated by spaces, corresponding to the x and y co-ordinates and the rover's orientation.

Each rover will be finished sequentially, which means that the second rover won't start to move until the first one has finished moving.

Output:

The output for each rover should be its final co-ordinates and heading.

Test Input:

5 5
1 2 N
LMLMLMLMM
3 3 E
MMRMMRMRRM

Expected Output:

1 3 N
5 1 E

The problem requires two things: a parser from the input to some useful data structure, and something to process that data structure. Immediately we are confronted by a fundamental difference between OO (at least as I see it done) and functional style.

The OO Programmer

The OO programmer sees a MarsRoverController or similar abstraction for coordinating this problem. The MarsRover becomes a class having a position and a direction. The coordinating object depends on both the parser and the MarsRover. It firstly asks the parser to parse the input, then asks the MarsRover to process the movements. An even uglier implementation would have the MarsRover doing the coordination responsibility.

The Functional Programmer

To the functional programmer these are all just functions. A function to do the parsing and a function to process the data structure. In my implementation the functions are parseInitialization and play. My REPL usage looks like this:

> parseInitialization testInput |> play;;
val it : Rover list = [{x = 1;
                        y = 3;
                        direction = N;}; {x = 5;
                                          y = 1;
                                          direction = E;}]

Implementing the Action State Change

Somewhere we have to process actions. The possible actions are turn left, turn right, or move forward. Processing one of this actions changes the state of the rover.

Let us first focus on the turning actions. There are different ways to implement this. My first attempt was a simple enumeration of the possibilities:

let turn direction action =
    if action = M then failwith "M is not a turn action"
    match (direction,action) with
        | (N,L) -> W
        | (N,R) -> E
        | (E,L) -> N
        | (E,R) -> S
        | (S,L) -> E
        | (S,R) -> W
        | (W,L) -> S
        | (W,R) -> N

I like the simplicity of this approach. The required guard to check for a move action was a strong indication that I had modelled the types incorrectly. If it feels wrong, it probably is.

As an experiment I then tried an implementation that understood the directions as angle measurements. This has the advantage that it could be extended to more realistic systems, but it is more complicated:

let turnAlt direction action =
    let directionAngles = [(N,0);(E,90);(S,180);(W,270)]
    let angle = (List.find (fun (d,a) -> d = direction) directionAngles) |> snd
    let angleToDirection angle =
        List.find (fun (d,a) -> a = angle) directionAngles |> fst
    let turnedAngle =
        match action with
        | L -> angle - 90
        | R -> angle + 90
        | M -> failwith "M is not a turn action"
    angleToDirection   (if turnedAngle < 0 then
                            turnedAngle + 360
                        elif turnedAngle = 360 then
                            0
                        else
                            turnedAngle)

Parsing

To keep things simple I started with simple string splitting to parse the input. For the record, it looks like:

let parseDirection = function
    | "N" -> N
    | "E" -> E
    | "S" -> S
    | "W" -> W
    | d -> failwith ("Unknown direction " + d)

let parseRover (line:string) =
    let lineParts = line.Split(' ')
    {x = lineParts.[0] |> int; y = lineParts.[1] |> int; direction = parseDirection lineParts.[2]}

let parseActions (line:string) =
    [
        for c in line do
            yield match c with
                    | 'L' -> L
                    | 'R' -> R
                    | 'M' -> M
                    | other -> failwith ("Unknown action " + (string other))
    ]

let parseInstructions lines : Instruction list =
    [
        for i in [0..(Array.length lines)/2 - 1] do
            let roverLine = lines.[2*i]
            let actionLine = lines.[2*i+1]
            yield {rover= parseRover roverLine; actions = parseActions actionLine}
    ]

let parseInitialization (input:string) =
    let lines = input.Split('\n')
    let topRightLine = lines.[0]
    {
        upperRightX = lines.[0].Split(' ').[0] |> int; upperRightY = lines.[0].Split(' ').[1] |> int;
        instructions = parseInstructions lines.[1..]
    }

Parsing with Fparsec

Then, for fun, I deleted my implementation and replaced it with fparsec parser combinators.

Firstly, we need to parse the first line, giving the top right coordinates, to a point:

type Point = {x:int; y:int}
let ptopright = sepBy pint32 (pchar ' ') |>> (fun [x;y] -> {x=x;y=y})

usage:

> run ptopright "5 6"
val it : ParserResult<Point,unit> = Success: {x = 5;
 y = 6;}

Next, we need to parse a direction. This is an example of parsing alternatives:

type Direction = N | E | S | W
let pdirection = (charReturn 'N' N) <|> (charReturn 'E' E) <|> (charReturn 'S' S) <|> (charReturn 'W' W)

Now we can parse the second line of input, the rover’s state. Here we run 5 parsers (x, space, y, space, direction) and pipe the results into a function that produces the type we want (Rover):

type Rover = {location:Point; direction:Direction}
// 1 2 N
let proverstate =
    pipe3
        (pint32 .>> (pchar ' '))
        (pint32 .>> (pchar ' '))
        pdirection
        (fun x y direction -> {location = {x=x;y=y}; direction = direction})

Next - a way to parse actions - turn left, turn right or move forward:

type Turn = L | R
type Action =
    | Turn of  Turn
    | M
let pturn = (charReturn 'L' (Turn L)) <|> (charReturn 'R' (Turn R))
let paction = pturn <|> (charReturn 'M' M)

Actions never occur by themself. We want to parse a sequence of them, so:

// LMMLMRM
let pactions = many (pturn <|> (charReturn 'M' M))

We want to parse pairs of rover + actions so:

type Instruction = {rover:Rover; actions:Action list}

// 1 2 N
// LMLMLMLMM
let pinstruction =
  pipe2
    (proverstate .>> newline)
    pactions
    (fun state actions -> {rover=state; actions=actions})

Finally, we connect all of our parsers together:

let pinit =
    pipe2
        (ptopright .>> newline)
        (many pinstruction)
        (fun topright instructions ->
            {upperRight = topright; instructions = instructions})

The Completed Mars Rover Solution

#r @"FParsec.1.0.1\lib\net40-client\FParsecCS.dll"
#r @"FParsec.1.0.1\lib\net40-client\FParsec.dll"

open FParsec

type Direction = N | E | S | W
type Point = {x:int; y:int}
type Rover = {location:Point; direction:Direction}
type Turn = L | R
type Action =
    | Turn of  Turn
    | M
type Instruction = {rover:Rover; actions:Action list}
type Initialization = { upperRight: Point; instructions: Instruction list}

let pdirection = (charReturn 'N' N) <|> (charReturn 'E' E) <|> (charReturn 'S' S) <|> (charReturn 'W' W)
let ptopright = sepBy pint32 (pchar ' ') |>> (fun [x;y] -> {x=x;y=y})

// 1 2 N
let proverstate =
    pipe3
        (pint32 .>> (pchar ' '))
        (pint32 .>> (pchar ' '))
        pdirection
        (fun x y direction -> {location = {x=x;y=y}; direction = direction})

let pturn = (charReturn 'L' (Turn L)) <|> (charReturn 'R' (Turn R))
let pactions = (many (pturn <|> (charReturn 'M' M))) .>> (skipChar '\n' <|> preturn ())

// 1 2 N
// LMLMLMLMM
let pinstruction =
    pipe2
        (proverstate .>> newline)
        pactions
        (fun state actions -> {rover=state; actions=actions})

let pinit =
    pipe2
        (ptopright .>> newline)
        (many pinstruction)
        (fun topright instructions ->
            {upperRight = topright; instructions = instructions})

let testInput = "5 5
1 2 N
LMLMLMLMM
3 3 E
MMRMMRMRRM"

let extectedOutput = "1 3 N
5 1 E"

let play (init:Initialization) =
    let playInstruction (instruction:Instruction) =
        let turn direction action =
            match (direction,action) with
                | (N,L) -> W
                | (N,R) -> E
                | (E,L) -> N
                | (E,R) -> S
                | (S,L) -> E
                | (S,R) -> W
                | (W,L) -> S
                | (W,R) -> N

        let applyAction (rover:Rover) = function
                | Turn t -> {rover with direction = turn rover.direction t}
                | M ->
                    let moved =
                        { rover with location = match rover.direction with
                                                | N -> {rover.location with y = rover.location.y+1}
                                                | E -> {rover.location with x = rover.location.x+1}
                                                | S -> {rover.location with y = rover.location.y-1}
                                                | W -> {rover.location with x = rover.location.x-1}
                        }
                    if moved.location.x < 0
                        || moved.location.y < 0
                        || moved.location.x > init.upperRight.x
                        || moved.location.y > init.upperRight.y then
                        rover
                    else
                        moved
        List.fold applyAction instruction.rover instruction.actions
    List.map playInstruction init.instructions

match run pinit testInput with
    | Success(result, _, _)   -> play result
    | Failure(errorMsg, _, _) -> printfn "Failed to parse: %s" errorMsg; []

PostgresDoc Support For Sql Server

Fri, 13 Feb 2015 20:32:00 +1000

Using sql server as a document database, with transactions, joins, unit of work and other good stuff

PostgresDoc is now the most ironically named ORM! Also, it doesn’t support objects, relations or mapping, but it does now support Sql Server - at least the recent versions.

Historically, PostgresDoc has been a data access library providing serialization/deserialization, transactions, and a unit of work on top of the Postgresql json data type and associated indexes. Recently I got around to implementing the same thing on top of Sql Server’s XML data type and its indexes.

Here is the Doco

PostgresDoc now supports SQL Server. SQL Server does not have the required JSON support, so the SQL Server version serializes to xml instead. The serializer supports .net classes, records and discriminated unions.

Create your store like this:

let storeSql = SqlStore "your connection string"

and create tables like this:

CREATE TABLE [dbo].[card](
  [Id] [uniqueidentifier] NOT NULL,
  [Data] [xml] NOT NULL,
 CONSTRAINT [PK_card] PRIMARY KEY CLUSTERED
(
  [Id] ASC
)

Inserting, updating and deleting work the same as for Postresql. Querying requires the use of Sql Server’s XML querying capabilities.

Querying

[
        "sourceUrl", box url
        "userId", box userId
]
|> query<Deck> store "select [Data] from [deck]
where Data.value('(/Deck/userId)[1]', 'uniqueidentifier') = @userId
and Data.value('(/Deck/sourceUrl)[1]', 'nvarchar(512)') = @sourceUrl"

There are many more options available for querying sql server xml.

Indexing

Sql server xml indexes traditionally index everything, which is convenient, bulky and slow.

Since 2012 Sql Server also has selective indexing.

Fermat

Tue, 20 Jan 2015 20:32:00 +1000

https://github.com/liammclennan/fermat

Fermat is a simple library for organising web states and generating urls. Tell it the possible states of your web ui, and how those states map to urls and it will give you a url generation function.

var fermat = require('../lib/fermat')({
    'Inbox': '/inbox',
    'Tag': '/tag/:tag',
    'Calendar': '/calendar/:month/:day'
});

fermat('Calendar', {
  month: 'march',
  day: '14'
});

// /calendar/march/14

It works with require.js, node, browserify and smodules.

Run Tests

First you need mocha.

npm install -g mocha

Then

$ npm test

and you should see something happy like

> fermat@1.0.0 test c:\work\fermat
> mocha

Fermat
    some states
      V should generate url without route components
      V should generate a url with a route component
      V should generate a url with multiple route components


3 passing (17ms)

F# Project Loading

Fri, 16 Jan 2015 20:32:00 +1000

Everytime I create a new F# project (typically using the MVC 5 F# Templates) things go great until I close Visual Studio. When I try to reopen the solution I inevitably get:

The project ?? could not be opened because opening it would cause a folder to be rendered multiple times in the solution explorer. One such problematic item is ??

It’s natural to think that the indicated file is causing the problem, but if you remove it then the problem just shifts to the next file.

The error occurs when files within the project are not grouped by their directory. A project containing:

<Compile Include="Home\HomeController.fs" />    
<Content Include="Scripts\react.js" />
<None Include="Home\Index.cshtml" />
<Content Include="Scripts\react.d.ts" />

cannot be reopened. To fix, group the files by their directory.

<Compile Include="Home\HomeController.fs" />    
<None Include="Home\Index.cshtml" />
<Content Include="Scripts\react.js" />
<Content Include="Scripts\react.d.ts" />

That’s it. Hope this helps someone.

Spaced Repetition Wiki

Sat, 10 Jan 2015 20:32:00 +1000

On the Wiki

A wiki is a collection of user-editable web pages. The wiki was ‘invented’ by Ward Cunningham, so it must be good. I like to use wikis to document things, or record mind maps for areas that I am learning. I have a wiki for vim, a wiki for haskell, a wiki for linux and so on.

Both github and bitbucket offer free wikis, and bitbucket wikis can be private.

On Spaced Repetition

As we are repeatedly exposed to a datum the time before we forget it gets longer. The most efficient way then to commit something to memory is to be reminded of it immediately prior to the moment you would otherwise have forgotten it. You know this intuitively if you have ever studied for an exam and then realised three weeks later that you no longer retain any of the information.

Spaced repetition is a learning technique that incorporates increasing intervals of time between subsequent review of previously learned material in order to exploit the psychological spacing effect – Wikipedia

Most of the things I learn I am happy to forget. It sounds strange but I know my memory is limited and I believe that we tacitly retain helpful conditioning even when specific facts are forgotten. To be nerdy about it you could say that short-term memory trains our machine learning system. The memories fade but the prediction function remains. I like spaced repetition for those important data that I want to retain medium to long term.

Combining the Wiki and Spaced Repetition

My struggle with spaced repetition has always been around the maintenance of the question decks. This problem has not existed for me with wikis, so I decided it might work well to embed spaced repetition questions in my wikis. The microformat I chose for this job is:

Q>>> This is the question? <<<

A>>> This is the answer. <<<

Any content at all (including HTML) is allowed between the delimiting tokens. Both this blog article and the readme file for the project I created are valid examples of the format. The aforementioned project is a terminal program, written in F#, that clones public git repositories, selects the set of markdown files, extracts the spaced repetition content that conforms to the specified microformat, and writes the data to a csv file that I can then import into anki (my spaced repetition client of choice).

Much of the program came together very nicely and provided some great demonstration of the joy of F#. The part I am least happy with, and the part I think most likely to contain bugs, is the parser. I’m still learning parsers and parser combinators and this is the first one I’ve done that required backtracking. Regardless, it seems to work.

What’s Next?

I plan to start some postgrad coursework soon and give this system a real workout. I may spin up a web front-end that does spaced repetition from public git repositories. The problems with existing spaced repetition clients are:

proprietary formats
closed systems
non-web based

all of which I can easily fix with a simple web app. Currently the system is restricted to public git repositories as the data store but I could easily extend it to private git repositories or other data sources. I specifically chose git because it is an open standard. For this reason I abandoned the github and bitbucket APIs.

Object-Orientation is Just Grouping Partially Applied Functions

Tue, 11 Nov 2014 20:32:00 +1000

I’ve been wondering if perhaps there is not much to this object orientation business. Could it be that an ‘object’ is nothing more than a collection of partially applied functions?

An Example of Object-Orientation

Here is a simple object oriented class, representing a person with a couple of operations.

void Main()
{
    var vlad = new Person("Vladimir", 62);
    var tony = new Person("Tony", 57);

    tony.Greet().Dump();
    vlad.OlderThan(tony).Dump();
}

public class Person {
    public string _name;
    public int _age;

    public Person(string name, int age) {
        _name = name;
        _age = age;
    }

    public string Greet() {
        return String.Format("Hello, my name is {0}", _name);
    }

    public bool OlderThan(Person other) {
        return _age > other._age;
    }
}

The data is a person’s name and age. The operations are Greeting and comparing the ages of two instances.

The output of the above Linqpad script is:

Hello, my name is Tony

True

An Example With Functions

An equivalent functional implementation is:

type Person = {name:string;age:int}

let greet p = p.name |> sprintf "Hello my name is %s"
let olderThan p1 p2 = p1.age > p2.age

let vlad = {name= "Vladimir"; age= 62}
let tony = {name= "Tony"; age= 59}

greet vlad
olderThan vlad tony

output

val it : string = “Hello my name is Vladimir” val it : bool = true

Objects Are Just Partially Applied Functions

If I didn’t care about sensible solutions I might implement a functional program that is more directly equivalent to the object-oriented code.

type PersonOO = { greet: string; olderThan: string -> int -> bool}

let greet' name age = name |> sprintf "Hello my name is %s" 
let olderThan' name age name' age' = age > age'

let constructPerson name age = { greet = greet' name age; olderThan = olderThan' name age}

let vlad' = constructPerson "Vladimir" 62
let tony' = constructPerson "Tony" 59

vlad' 
vlad'.olderThan "Tony" 59

output

val tony' : PersonOO = {greet = "Hello my name is Tony"; olderThan = <fun:tony'@9>;}
val it : bool = true

When you look at it this way it is easier to see the problem. In the signature of greet' the age argument is redundant. In the signature of olderThan' only the age arguments are used.

Some Problems With Objects

Both of the previous examples do the same thing. The trouble with the object version is that both methods are coupled to the name and age fields, even though they don’t need to be. The other side of the same coin is a cohesion problem.

So both implementations are pretty much equivalent, but to me the functions are simpler and more obvious.

Parsing Nokia Composer Ringtones With FParsec

Thu, 30 Oct 2014 20:32:00 +1000

The classic Nokia 3310, and similar Nokia phones featured a capability to define custom ringtones using a tool called ‘Composer’.

Parser uses a text based format to define the sequence of tones that makes up the ringtone. A tone is typically specified in the form:

8e2

Where the 8 indicates that the tone should last for eighth of a bar (half of a beat), the e is the musical note describing the pitch of the tone and the 2 indicates the octave that note comes from (valid options are 1, 2 and 3).

I want to be able to read the string that defines a composer tune and interpret it in a precise way that can be used for computations (like playing the tune). The full syntax, as a regular expression, is something like:

(\d+\.?#?[abcdefg][123])|(\d+\.?\-)

A tune is specified by a sequence of tokens, separated by spaces. Each token is either a tone or a rest. A rest is specified by a duration and an optional extension followed by a hyphen (e.g. 4.-). A tone is specified by a duration and an optional extension followed by a note (which may be preceded by a # to indicate sharp), followed by an octave (e.g. 16.#f3).

The Nokia Composer syntax is simple enough that I could parse it with string functions or regular expressions, but I thought it might be fun to do it properly with a parser combinator library. A parser is a function that accepts a string and returns some structure. A parser combinator is a higher-order function that combines several parsers into a single parser. F# has an excellent parser combinator library called FParsec. In FParsec a parser that reads a float looks like:

pfloat "3.14"
// => Success(3.14,...)

A combinator that parses two floats separated by a space

"3.14 2.71828" |> pfloat .>> pstring " " >>. pfloat

The strategy with parser combinators is to build parsers for the smallest units of syntax, then combine them into larger and larger pieces until you have a parser for the entire thing.

If you don’t have the good fortune to be using F# don’t worry, there is a parser combinator library for C# too.

Parsing the Nokia Composer Syntax

Parsing the Duration

Before we can parse we need types to parse too. The types that will represent the duration of the tone or rest are:

type NoteLength = Half | Quarter | Eighth | Sixteenth | Thirtyseconth 
type Length = { duration: NoteLength; extended: bool }

A length type has a duration and an extended bool. Durations followed by a . are extended by half. The parser for the duration part is:

let pnotelength = 
    (stringReturn "2" Half) <|> 
    (stringReturn "4" Quarter) <|> 
    (stringReturn "8" Eighth) <|> 
    (stringReturn "16" Sixteenth) <|> 
    (stringReturn "32" Thirtyseconth)

This simply codifies that the duration can be 2,4,8,16 or 32. The extended parser is:

let extendedParser = (stringReturn "." true) <|> (stringReturn "" false)

These two parsers are combined into a single duration parser that produces a Length.

let plength = pipe2 
                pnotelength            
                extendedParser 
                (fun t e -> {duration = t; extended = e})

Parsing the Pitch

The syntax that describes the pitch is

(#?[acdfg])|([be])

That is a,c,d,f and g may be preceded by a # to indicate a sharp (a semitone higher) while b and e may not.

The parser for the optional # symbol is:

let psharp = (stringReturn "#" true) <|> (stringReturn "" false)

i.e. parse a # as true or its absense as false.

The <|> combinator combines two combinators by giving the result of the first one, unless it fails having consumed no input, in which case it gives the result of the second. The restriction that the second parser is only tried if the first does not consume any input is very important and described in detail in the FParsec documentation.

Next we have a parser for notes that may be sharps. The target type for this parser is:

type Note = A | ASharp | B | C | CSharp | D | DSharp | E | F | FSharp | G | GSharp

and the parser:

let psharpnote = pipe2 
                    psharp 
                    (anyOf "acdfg") 
                    (fun isSharp note -> match (isSharp, note) with
                        | (false, 'a') -> A
                        | (true, 'a') -> ASharp                    
                        | (false, 'c') -> C
                        | (true, 'c') -> CSharp
                        | (false, 'd') -> D
                        | (true, 'd') -> DSharp
                        | (false, 'f') -> F
                        | (true, 'f') -> FSharp
                        | (false, 'g') -> G
                        | (true, 'g') -> GSharp
                        | (_,unknown) -> 
                            sprintf "Unknown note %c" unknown 
                            |> failwith)

pipe2 sequences parsers, meaning that psharpnote is a parser that first parses an optional # (via psharp) and then parses one of the characters a,c,d,f or g. The results are then mapped to the Note type. The above parser cannot produce B or E results, since these are not sharpable notes. A further parser is required:

let pnotsharpablenote = anyOf "be" |>> (function 
                        | 'b' -> B
                        | 'e' -> E
                        | unknown -> 
                            sprintf "Unknown note %c" unknown 
                            |> failwith)

Note that both psharpnote and pnotsharpablenote include a pattern match that throws an exception. The compiler requires the pattern match to be exhaustive, however by inspection we can see that the ‘unknown’ case can never actually occur. This is a case where the type system is not sufficiently powerful to capture the fact that the value can only be a ‘b’ or ‘e’ character.

The last remaining hurdle for our Note parser is to deal with the alternatives:

let pnote = pnotsharpablenote <|> psharpnote

Now we have one parser, pnote that can parse any possible note.

Finally, we need to parse the octave to the following type:

type Octave = One | Two | Three

This can be done with a simple character parser:

let poctave = anyOf "123" |>> (function
                | '1' -> One
                | '2' -> Two
                | '3' -> Three
                | unknown -> sprintf "Unknown octave %c" unknown |> failwith)

The |>> combinator sends the parsed value of its first argument to its second argument, allowing us to map the char 1,2 or 3 to values of type Octave.

Putting Things Together

If Sound is defined as a rest or a Tone:

type ToneRecord = { note: Note; octave: Octave }
type Sound = Rest | Tone of ToneRecord

Then we can parse a Tone like so:

let ptone = pipe2 pnote poctave (fun n o -> Tone {note = n; octave = o})

and a rest:

let prest = stringReturn "-" Rest

Finally, a Token is a duration + (tone || rest):

type Token = { length: Length; sound: Sound }

and is parsed thus:

let ptoken = pipe2 
                plength 
                (prest <|> ptone) 
                (fun l t -> {length = l; sound = t})

That is everything required to parse a Nokia composer token, which defines a rest or a note.

Parsing Tunes

Parsing a tune is a simple matter of applying the ptoken parser repeatedly until the end of the string. Naturally, there is a combinator for that. sepBy takes two parsers: the first to parse items and a second to parse the separators between items. For the space separated Nokia Composer syntax we can use:

let pscore = sepBy ptoken (pstring " ")

With this parser we can now parse ‘I Don’t Wanna Miss a Thing’ by Aerosmith:

pscore "2- 16a1 16- 16a1 16- 8a1 16- 4a2 16g2 16- 2g2 16- 4- 8- 16g2 16- 16g2 16- 16g2 8g2 16- 4c2 16#a1 16- 4a2 8g2 4f2 4g2 8d2 8f2 16- 16f2 16- 16c2 8c2 16- 4a2 8g2 16f2 16- 8f2 16- 16c2 16- 4g2 4f2"

Implementing an F# Signal Generator

Wed, 29 Oct 2014 20:32:00 +1000

As part of a larger project I have been working on a signal generator. A signal generator is a device (in this case an F# program) that is often used to generate waves at a particular frequency that can be interpreted as sound (by applying to a speaker as a voltage).

For my project I want to generate pitches and then convert the signal to PCM audio data, wrapped in the windows WAVE file format. PCM format means that the signal is represented as a series of positive or negative values within a certain range. The number of samples per second is called the sample rate (44,100 for CD audio). The range of the values is governed by the bit depth, again I will stick with the CD audio standard 16 bit audio, giving me a range of -32768 to 32767.

Generating Signals

The simplest and purest representation of a particular frequency is given by a sine wave.

This graph was generated using FSharp.Charting and the following fsx file

#load "../packages/FSharp.Charting.0.90.7/FSharp.Charting.fsx"
open FSharp.Charting
let c = seq { for x in [1.0..0.1..20.0] do yield (x, sin x) } |> Chart.Line

Note that the wave has a range from -1 to 1 and a recurring regular frequency. But what is that regular frequency? The F# sin function operates on an angle given in radians. Experimenting in the F# repl (fsi) we can find that sin of PI radians is 0

> sin System.Math.PI;;
val it : float = 1.224606354e-16

Interesting. What about the sin of 2 PI radians?

> sin (2. * System.Math.PI);;
val it : float = -2.449212708e-16

Those accustomed to working with floating point numbers will recognise that anything x 10^-16 is floating points version of 0. So sin (n * PI) is 0. This tells us, or at least suggests, that the graph crosses the x axis every PI change in x. Now let’s look at mid-points between x intercepts:

> sin (1.5 * System.Math.PI);;
val it : float = -1.0
> sin (2.5 * System.Math.PI);;
val it : float = 1.0
> sin (3.5 * System.Math.PI);;
val it : float = -1.0

Half way between PI and 2 x PI the y value is -1. Half way between 2 x PI and 3 x PI the y value is 1. Half way between 3 x PI and 4 x PI the y value is back to -1, so we see that the graph repeats every 2 x PI.

Now we can start generating PCM data. First we need to list the samples we want to create:

let requiredSamples = seq { 1.0..(seconds * sampleRate) }

We need sampleRate samples for every second. Next we plot those points using the sin function, and scale them by the expected range for 16 bit audio.

let samples = Seq.map 
                (fun x -> x |> sin |> (*) 32767 |> int16) 
                requiredSamples

Now we have PCM audio samples that we can pack and send to an audio player for listening pleasure.

Packing PCM data into a WAVE File

The WAVE file format is a binary format defined as part of Microsoft’s RIFF specification. The details of the format are not very interesting. It has some headers that describe the format in use, and details of the audio data, such as sample rate, bit depth and number of channels. To encode 16 bit mono samples at 44,100the following is sufficient:

let pack (d:int16[]) = 
    let stream = new MemoryStream();
    let writer = new BinaryWriter(stream, System.Text.Encoding.ASCII);
    let dataLength = Array.length d * 2

    // RIFF
    writer.Write(System.Text.Encoding.ASCII.GetBytes("RIFF"))
    writer.Write(Array.length d)
    writer.Write(System.Text.Encoding.ASCII.GetBytes("WAVE"))

    // fmt
    writer.Write(System.Text.Encoding.ASCII.GetBytes("fmt "))
    writer.Write(16)
    writer.Write(1s)        // PCM
    writer.Write(1s)        // mono
    writer.Write(44100)     // sample rate
    writer.Write(44100 * 16 / 8)     // byte rate
    writer.Write(2s)        // bytes per sample
    writer.Write(16s)       // bits per sample

    // data
    writer.Write(System.Text.Encoding.ASCII.GetBytes("data"))
    writer.Write(dataLength)
    let data:byte[] = Array.zeroCreate dataLength
    System.Buffer.BlockCopy(d, 0, data, 0, data.Length)
    writer.Write(data)
    stream

Tuning

So far we have a signal generator that generates audio data following a sine wave, but we have no control of the pitch (frequency) of the sound. We know that the wave repeats every 2 x PI, and we know that we have 44,100 samples per second. What we need is a way to expand and contract the wave horizontally to hit target frequencies. The graph transformation that controls horizontal stretch is achieved by multiplying the argument by a factor prior to applying the original function. ie

let square x = x * x
let squareStretched x = (2 * x) * (2 * x)

The frequence of the wave is

freq = 44100 / period

and the period is 2 x PI. To add a horizontal stretching factor we multiply by some factor.

freq = 44100 / 2 x PI x factor

solving for the factor

factor = 44100 / 2 * PI * freq

Now we can update our signal generating function to be tuneable using the inverse of factor

let frequency = 440.
let samples = Seq.map 
                (fun x -> x 
                |> (*) (2. * System.Math.PI * frequency / 44100.)
                |> sin 
                |> (*) 32767 
                |> int16) 
                requiredSamples

Now we can tune our data to frequencies. Above I have used 440Hz (concert A). To pack into a WAVE and save to a file

let write (ms:MemoryStream) =
    use fs = new FileStream(Path.Combine(__SOURCE_DIRECTORY__,"test.wav"), FileMode.Create)
    ms.WriteTo(fs)

samples |> pack |> write

Admiring the Results

What is the result of all that effort? A few seconds of concert pitch.

Ad-hoc Polymorphism in F# (how to survive without Type Classes)

Tue, 21 Oct 2014 20:32:00 +1000

A recurring source of confusion for me when reading about and writing F# has been ad-hoc polymorphism, and it seems that I am not the only one:

“F# does not have support for type-classes, or for multi-methods, or anything in between and while you can do certain tricks to achieve such an effect, like passing around a global virtual table, truth is F#’s ad-hoc polymorphic capabilities really suck.”

Martin Trojer

Ad-hoc polymorphism is the programming language feature that allows a function to work with arguments of varying types. The example given by wikipedia is:

1 + 2 = 3
3.14 + 0.0015 = 3.1415

The first line shows the addition of two integers. The second line shows the addition of two floating point numbers. For this to work the (+) operator must be a function that can operate on both integers and floats. This is an example of ad-hoc polymorphism. F#’s predecessor OCaml has almost no support for ad-hoc polymorphism, so the previous (+) example does not work in OCaml. +, -, / and * only work for integers and you need +., -., /. and *. for floating point operations.

Where F# Becomes Difficult

Consider the following example (borrowed from http://blog.lab49.com/archives/2895):

// this does not work :( 
// (expression was expected to have type 
// float but here has type int)
let twice x = x + x

twice 2.0 |> Console.WriteLine
twice 2 |> Console.WriteLine

This code fails because the x parameter of the twice function is inferred to be of type float, because that is the type of the argument the first time that it is called. Type inference lets us get away with not explicitly specifying a type, but it still has to assign a single type. In this case it chooses float, so when the function is applied to an integer we get a compilation error.

In F# the twice function above can fixed by means of the inline modifier.

// this version works!
let inline twice x = x + x

twice 2.0 |> Console.WriteLine
twice 2 |> Console.WriteLine

Each call to the inline function is replaced by inline code with its own type inference, thus the compiler effectively converts the above to:

(fun (x:float) -> x + x)(2.0) |> Console.WriteLine
(fun (x:int) -> x + x)(2) |> Console.WriteLine

This is rather nice because we get the explicit type correctness of having no runtime type coercion, but without having to manually provide implementations for each type. In this way it is superior even to Haskell’s lauded type class feature.

A More Realistic Example

Imagine you want to define a function show that converts a value of type A or B to a string representation. My first attempt was:

type A = { thing: int }
type B = { label: string }

let show (a:A) =
    sprintf "%A" a

let show (b:B) =
    sprintf "%A" b

{ thing = 98 } |> show |> Console.WriteLine

This is not allowed. F# does not allow you to overload the show function in this way.

How About Using an Interface?

Often the requirement for ad-hoc polymorphism can be achieved by using interfaces. F# programmers often talk about using a ‘dictionary of operations’ which is essentially the same thing.

The example above becomes:

type IShow =
    abstract member show : unit -> string

type A =
    { thing: int }
    interface IShow with
        member this.show() = sprintf "%A" this
type B =
    { label: string }
    interface IShow with
        member this.show() = sprintf "%A" this

let show (x:IShow) =
    x.show()

{ thing = 98 } |> show |> Console.WriteLine
{ label = "Car" } |> show |> Console.WriteLine

This is OK, but a little clumsy. There are also some restrictions on implementing interfaces that may occassionally make this solution infeasible.

F# allows type coercion to an interface for function arguments so we avoid having to explicitly upcast values to IShow (({ thing =98 } :> IShow).show()).

Solution Using Static Member Overloading

My favourite solution, at least so far, is to use static member overloading. This takes advantage of the fact that while top-level functions cannot be overloaded functions on types can.

type A = { thing: int }
type B = { label: string }

type ThingThatShows =
    static member show(x:A) = sprintf "%A" x
    static member show(x:B) = sprintf "%A" x

{ thing = 98 } |> ThingThatShows.show |> Console.WriteLine
{ label = "Car" } |> ThingThatShows.show |> Console.WriteLine

This nice part about this is that I am able to define two versions of ThingThatShows.show, one for type A and one for type B. The correct function is then applied based upon the type of the argument.

Solution Using Statically Resolved Type Constraints

F# has a somewhat unusual feature - statically resolved generic types - and these can be used in combination with member constraints and inline functions to implement ad-hoc polymorphism.

type A = { thing: int }
    with static member show a = sprintf "%A" a
type B = { label: string }
    with static member show b = sprintf "%A" b

let inline show (x:^t) =
    (^t: (static member show: ^t -> string) (x))

{ thing = 98 } |> show |> Console.WriteLine
{ label = "Car" } |> show |> Console.WriteLine

This is a lot like the solution using interfaces except:

I did not have to define an interface.
I did not have to explicitly implement the interface members.

What Does This Look Like in Fancy-pants Haskell?

Haskell does not have this ad-hoc polymorphism problem. It is obvious in Haskell that Type Classes are the correct solution to this challenge. The example repeated above when converted to a Type Class based Haskell solution is:

-- define a type class for types
-- that can be shown
class Showable a where
    shw :: a -> String

data A = A String
data B = B Int

-- add the types A and B 
-- to the Showable type class

instance Showable A where
    shw (A s) = s

instance Showable B where
    shw (B i) = show i

putStrLn (shw (A "Car"))

HTTP Basic Authentication Support in Wurl

Wed, 08 Oct 2014 20:32:00 +1000

Wurl (web curl) is a browser-based HTTP test client that I slapped together earlier this year (2014), partly because it is a useful tool and partly because it gives me a play project to experiment with React, Typescript and F#.

How it works

Wurl describes http requests using the jquery $.ajax syntax. I chose this because most developers are familiar with it and because it is capable of describing a wide variety of request options.

Wurl first attempts to make the request by directly evaluating the request using jquery from the browser. Try this fun example of querying the github API. Often this strategy will fail due to cross-origin resource sharing (CORS) restrictions. Wurl is smart enough to detect CORS errors and fallback to making the request via a proxy, which is not subject to CORS restrictions.

Unfortunately the proxy is not jquery and does not use jquery (it uses HttpClient) so the query must be parsed and translated. Therefore queries that go via the CORS proxy do not implement the full set of $.ajax features.

HTTP basic authentication

Recently I added support for the Authorization header. The options that are honoured now are: url, type (http verb), headers (Authorization only). This means that basic authentication can be used from Wurl for all services, regardless of CORS support.

Here is an example showing a request to the github API that uses basic authentication. It fails, but only because I have removed my password.

DbUp (database migration) Support For Postgresql

Tue, 07 Oct 2014 20:32:00 +1000

DbUp is a .NET library that helps you to deploy changes to SQL Server databases. It tracks which SQL scripts have been run already, and runs the change scripts that are needed to get your database up to date.

DbUp is a simple, mature, sql script-based database deployment tool. I typically use it for all of my projects that use Sql Server. I like the idea so much that I have previously implemented similar systems for CouchDb and Postgresql. It occurred to me that instead of badly re-implementing half of the features of DbUp I should just add new database support to DbUp (open source right?).

DbUp now supports Postgresql

Show me the Codez!

var upgrader = DeployChanges.To
    .PostgresqlDatabase("Server=127.0.0.1;Database=testo;Port=5432;User Id=*****;Password=*******;")
    .WithScriptsEmbeddedInAssembly(Assembly.GetExecutingAssembly())
    .LogToConsole()
    .Build();

var result = upgrader.PerformUpgrade();

NOTE: at the time of writing this update has been merged but not yet made available via Nuget.

PostgresDoc now works with C#

Sun, 05 Oct 2014 20:32:00 +1000

PostgresDoc is a simple F# library for working with document data in Postgresql. Recently, I have been experimenting with ways to use it from C#.

Using an F# Library from C#

It is certainly possible to use an F# library from C#, but the syntax can be difficult. To ease the pain I experimented with ways to make the API more C# friendly, and ultimately gave up.

Wrapping an F# Library for C# consumers

The approach that ultimately worked for me was to create a new C# project that wraps the F# version of PostgresDoc and provides an API that is optimised for C#. The unit of work is represented by a Queue. New operations are added via the Enqueue method, and created via static factory methods on the Operation class, like so:

unitOfWork.Enqueue(Operation.Insert(ernesto._id, ernesto));

The PostgresDocCs C# API

Here is a simple example of working with a document from C#:

public class Person 
{
    public Guid _id { get; set; }
    public string Name { get; set; }
    public int Age { get; set; }
    public string[] FavouriteThings { get; set; }
}

var ernesto = new Person
        {
            _id = Guid.NewGuid(),
            Name = "Ernesto",
            Age = 31,
            FavouriteThings = new[] { "Pistachio Ice Cream", "Postgresql", "F#" }
        };
var connString = "Server=127.0.0.1;Port=5432;User Id=******;Password=*****;Database=testo;";

var unitOfWork = new Queue<Operation<Guid>>();

// insert a document
unitOfWork.Enqueue(Operation.Insert(ernesto._id, ernesto));

// modify a document
ernesto.Age = 32;
unitOfWork.Enqueue(Do.Update(ernesto._id, ernesto));

// persist the changes in a transaction
UnitOfWork.Commit(connString, unitOfWork)

Querying

var ernesto = Query<Person>.For(
            connString, 
            "select data from Person where id = :id", 
            new Dictionary<string, object> { {"id", ernesto._id} });

Postgresql as a Nosql Document Store

Tue, 30 Sep 2014 20:32:00 +1000

Postgresql makes a fairly decent schemaless Nosql document database. It is the only database I can think of that:

is mature
is fast
has an extensive plugin ecosystem
has ACID compliance
supports schemaless document storage
supports relational data storage
can join across documents and relational data
can index into documents as well as relational data
has built-in full text search

Postgres already had most of this. The only thing they had to add to facilitate document storage was a json data type and the ability to create indexes on json properties.

An Example

If I want to store Person documents I might create a table called Person with a json column to hold the document:

create table Person ( data json NOT NULL );

The documents will have a unique identifier _id which I need to be able to query on, so I add an index:

CREATE UNIQUE INDEX people_id ON Person ((data->>'_id'));

Then the query to select a document is:

select data from Person where data->>'_id' = 'A158CCB9-BB68-4FC2-8123-79B32053B2A3'

Better Tooling

A typical application scenario will make changes to multiple documents. We can take advantage of Postgres’s transaction support by accumulating changes and commiting them within the same transaction. If there are no problems then the changes are persisted, but if something goes wrong the entire transaction is reverted, leaving the database in a consistent state.

I have been working on a project I call PostgresDoc to make working with postgres documents easier. Currently it is an F# project but I will add a C# wrapper soon. Use it like so

// accumulate a stack of changes
let julio = { _id = System.Guid.NewGuid(); age = 30; name = "Julio" }
let uow = [Insert julio]
let uow2 = Update { julio with age = 31} :: uow
let uow3 = Delete julio :: uow2

commit store uow3

This approach is great for testing (just make assertions against the unit of work) and is free from many of the awful consistency problems that plague ORMs such as NHibernate and Entity Framework.

Queries are easy too:

let peopleWhoAreThirty = 
    [ "age", box (30) ] 
    |> query<Person> store "select data from people where data->>'age' = :age"

Converting to TypeScript

Thu, 25 Sep 2014 20:32:00 +1000

My most recent side project is a web based curl - wurl - intended to be a convenient interface for experimenting with HTTP APIs.

Wurl is a useful tool that I created to satisfy a need - but it is also a project that I use to experiment with development techniques and technologies.

In the browser we are limited to JavaScript, a dynamic language not easily integrated with whatever server-side technology is in use (in this case Asp.Net MVC and F#). To help prevent runtime JavaScript bugs I recently converted wurl from JavaScript to TypeScript, a statically typed superset of JavaScript. The purpose of this post is to document that experience.

Where to Start?

It is not entirely obvious where to start a JavaScript -> TypeScript migration. I found that the best way is to start with bits of JavaScript that are small and have few dependencies.

Converting a JavaScript Module to TypeScript

Wurl has a JavaScript module used to contain its models. The Template model represents an HTTP request and how it is transformed.

define('models', [], function() {

    var Template = dbc.makeConstructor({
        id: [{validator:'type',args:['string']}],
        label: [{validator:'type',args:['string']}],
        request: [{validator:'type',args:['string']}],
        transformer: [{validator:'type',args:['string']}],
        view: [{validator:'type',args:['string']}]
    });
    Template.url = '/template';

    return {
        Template: Template
    };
});

Step 1 was to rename the file with a .ts extension. Visual Studio immediately displayed a compilation error because the type of the define function is not known. define is a function that comes from the smodules (small modules) JavaScript module library. To be able to use smodules, a JavaScript library, from TypeScript I need a type definitions file that defines all the types exposed by smodules. To create a type definition for smodules I create a new file with the same name as the smodules library file, but with the extension .d.ts. In that file I add a type declaration for the define function.

declare function define(name:string, dependencies: string[], moduleFactory: any): void;

This tells TypeScript that define is a function, with arguments of type string, string array and any. Define does not return anything. I also add a declaration for the other function exposed by smodules:

declare function require(name: string) : any;

The issue now is that TypeScript does not know where to look for the type definitions, so I add a reference to the models module:

/// <reference path="packages\02-smodules.d.ts" />

Now when I try to compile I get no errors about smodules, instead the compiler complains about the call to dbc.makeConstructor. dbc is a library providing runtime type assertions. Here it is being used to define a runtime ‘type’ check. This compilation error is fixed according to the same strategy: create a .d.ts file for dbc containing:

declare module dbc {
    function makeConstructor(spec: any): any;

    function check(o: any, spec: any, message?: string): void;
}

Using Well-Known JavaScript libraries

wurl makes use of many JavaScript libraries including:

* underscore
* react
* jquery
* q
* toastr
* codemirror

To be able to use these libraries from TypeScript I need type definitions for them all. Fortunately, there is a github repository containing many type definition files and they are available as nuget packages.

Summary

Wurl has perhaps 1,000 lines of JavaScript, so it is a small project. It took me about two hours to convert it to TypeScript, without having any idea what I was doing. It is easier to start a project with TypeScript from the beginning, however doing a JavaScript -> TypeScript conversion is perfectly feasible, even for larger projects.

Prometheus - Linking F# Records to Typescript Classes

Sat, 20 Sep 2014 20:32:00 +1000

I would like to program in a way that is as strict, unambiguous and correct as possible. I want the compiler to assist as much as possible, for new development and for refactoring. SPJ has described a compiler as a, “weak theorem prover” and I want my theorems weakly proven. But there is a problem. A lot of my work is in the browser where JavaScript is the only option, and JavaScript does not meet any of my stated requirements. Fortunately, there is a neat little project called TypeScript that adds static type checking to JavaScript.

For C# solutions there is a tool called netjs that can compile C# code to TypeScript and thence to JavaScript. I tested it on a simple C# solution with success, however, it does not work for F# projects.

For F# web projects my minimum requirement is a way to synchronize F# records with TypeScript classes, so that I may share equivalent type defininitions between my F# server application and TypeScript/JavaScript client application. I’m not a big fan of classes in F#, and neither discriminated unions nor tuples have an obvious mapping to JavaScript so for now I am content to stick to F# records.

Prometheus is a pre-alpha project (most of mine stay that way) that converts an F# assembly to a TypeScript file containing class equivalents of all the records in the F# assembly. By running prometheus immediately prior to your normal compilation step the sequence is something like:

Prometheus generates typescript definitions -> F# compilation -> TypeScript compilation

The flaw here is that sometimes you may have to build twice to get everything synced properly but for now that is a 1st world problem. The fix is to move the Prometheus compilation between the F# compilation and the TypeScript compilation.

Give an F# assembly containing the following module:

namespace MyModels

    type Address = { number: int; street: string }
    type Person = { name: string; age: int; address: Address }

Prometheus produces the following valid TypeScript file:

module MyModels { 
    export class Address {
        constructor(public number: number, public street: string) {}
    }
}

module MyModels { 
    export class Person {
        constructor(public name: string, public age: number, public address: Address) {}
    }
}

Now if my server and client side model definitions fall out of sync it will be a glorious compilation error.

React Flux Etc

Thu, 29 May 2014 20:32:00 +1000

I’ve been enjoying working with React for a while now. I like it’s emphasis on stateless, composable components.

Unlike Angular, React does not give the developer much help with the design of their application. To address this the React team recently starting talking about Flux, the architecture they use at Facebook. Flux has many pieces but the significant idea is that events no longer bubble back up the component hierarchy, instead they go out to an event aggregator that changes the application state, causing the UI to update. Much is made of the single directional data flow, state -> UI -> events -> aggregator -> change state -> UI etc. I like this design and I’ll add one important rule:

Use the same state for all related application specific components. Some components will get more data than they need, but having a single state model simplifies changes.

The other common criticism of React is that is does not include batteries (again compared to Angular). This doesn’t bother me much since I have strong opinions and have therefore created my own version of most components that I need.

The first missing piece that a developer is likely to encouter is a client-side router. I have heard of many people using the Backbone router, which is nice because it supports both html5 style pushstate routing and the older hash fragment style routing. I’ve been using Flatiron Director which is fine.

For server communication I have become comfortable with Backbone’s resource-oriented style. Angular provides $resource for this purpose. I use resourceclient because it is better. It uses Q promises instead of the broken jQuery promises and integrates with another library to provide object validation and type coercion. Where most similar libraries return json data without methods resourceclient is smart enough to return actual instances of your client-side models.

Integrated with resourceclient I use dbc to validate that objects match certain requirements and to create constructors that enure objects are valid.

For example, I might create a constructor like so:

var Person = dbc.makeConstructor({
    name: [{validator:'type',args:['string']}],
    age: [{validator:'type',args:['number']}]
});
Person.url = '/api/person/';
var personGateway = resourceclient(Person);

var firstPerson = new Person({
    name: 'Alice',
    age: 58
});

personGateway.create(firstPerson).then(function (data) {
    // data is the model returned from the server
});

The next missing piece is support for forms and validation. For that I use a set of purpose-built modules that aren’t reusable at the moment.

If you have to do client-side / JavaScript work then React is the best of the bunch. Enjoy.

PostgresDoc

Tue, 22 Apr 2014 20:32:00 +1000

Story so far

Lately I have been moving towards more functional style programming. One way that you can categorize functional languages is those that have their own complete ecosystem (haskell, erlang, oCaml) and those that piggyback off an existing ecosystem, runtime and standard library (F#, scala, clojure). It is clear that the second category has a substantial advantage for adoption. The runtimes are well established, reliable and fast. The standard libraries and package managers provide the best guarantee possible that I will not be left missing a critical component. There are downsides too. The integration between the functional programming syntax and underlying standard library are well designed but awkward. However, on balance the piggyback functional languages (F#, scala and clojure) provide a practical and valuable way to move to functional programming.

For me F# has some distinct advantages. I think types are one of the most important tools we have for writing working software, so dynamic languages like clojure have less appeal. Having spent 10 years working with .net the CLR is more approachable to me than the JVM. Finally, scala displeases me aesthetically.

Recently, my employer sponsored a professional development activity where I investigated the feasibility of applying functional programming to .net consulting projects. I found that it is practical, even simple, to partially or completely utilize functional programming via F# in this context.

Struggle

The last remaining awkward area was data access. I could not find a nice solution for data access from F#. Entity framework can be done, but involves a lot of compromise, essentially writing F# code in a C# style (the worst of both worlds). RavenDB seems like a good fit, which is why many have tried hard to make it work, but without support from the developers of RavenDB the result is disappointing. MongoDB may be web scale, but there effort at supporting F# never made it out of alpha.

What I wanted was a document database, with transactions, decent performance, simple querying and powerful indexes. I could not find one.

Resolution

Postgresql is an amazing open-source relational database. It ain’t pretty but it is powerful. The latest version (9.3) includes comprehensive support for working with structures serialized as json. It is possible to project queries over json documents, to query based on values in json documents and to create indexes (including unique indexes) on values in json documents. Being a typical relational database transactions are obviously available. Postgresql has all the basic features I want in a document database. All it needs is a nice, F# friendly, API.

PostgresDoc

For a long weekend project I built PostgresDoc, a new API on top of postgresql and npgsql. It simplifies the syntax, optimizes for document data and adds an explicit unit of work pattern. Here is the examples from the readme:

Unit of Work API

type Person = 
    { _id: System.Guid; age: int; name: string }
    interface IDocument with
        member x.tableName() = "People"
        member x.id() = x._id

let store = { connString = "Server=127.0.0.1;Port=5432;User Id=postgres;Password=*****;Database=testo;" }

let julio = { _id = System.Guid.NewGuid(); age = 30; name = "Julio" }
let timmy = { _id = System.Guid.NewGuid(); age = 3; name = "Timmy" }

// list is backwards so operations can be consed
let uow = [ 
    Delete timmy
    Update { julio with age = 31 };
    Insert julio;
    Insert timmy;
    ]
commit store uow

Document API

Query

let peopleWhoAreThirty = 
    [ "age", box (30) ] 
    |> Map.ofList
    |> query<Person> store "select data from people where data->>'age' = :age"

The idea is that the unit of work is built up during execution by appending data operations (insert, update or delete). The example unit of work is really shorthand for:

let uow = Delete timmy :: Update { julio with age = 31 } :: InsertJulio :: [Insert timmy]

The unit of work is committed in a transaction (beat that mongodb).

Summary

PostgresDoc is the document database that I wished I had. It provides a nice API for F# and seems to work!

State Machines in F# and C#

Fri, 18 Apr 2014 20:32:00 +1000

A couple of months ago I wrote a simple library for building state machines in C#. It allows a programmer to attach a state machine to a type and control:

the set of states that the type may have
the transitions between those states

The first system that used the library contained functionality for managing the lifecycle of contracts. The contracts begin in a draft state, they may then move to an approved state, and finally they may enter the historical state. They cannot go from draft to historical, from approved to draft, from historical to draft etc. Using the state machine added a valuable invariant guarantee to the code.

As much as my state machine library was a successful solution to a problem I can’t help but realise that this problem simply doesn’t exist in F#. F#’s type system is sufficiently advanced so that it can represent state machines as part of the type. This meets the stated goal of my C# state machine library. To use the previous contracts example, the F# type would look like this:

type Particulars = { buyer: string; seller: string }

type Contract = 
    private
    | Draft of Particulars
    | Approved of (Particulars * DateTime)
    | Historical of (Particulars * DateTime)

let createDraft particulars = 
    Draft particulars

let approve timestamp = function
    | Draft p -> Approved (p, timestamp)
    | _ -> failwith "Invalid transition"

let retire timestamp = function 
    | Approved (p,t) -> Historical (p, timestamp)
    | _ -> failwith "Invalid transition"

// usage
let contract = createDraft { buyer = "Acme"; seller = "Initech" }
let approved = approve DateTime.Now contract
let retired = retire DateTime.Now approved
retired |> printfn "%+A"

// output = Historical ({buyer = "Acme"; seller = "Initech";}, 
//                          18/04/2014 1:27:20 PM)

The type system models the relationship between Contract and each of the possible states (a Contract is a draft, or approved or historical) as well as the data attached to each state (a draft doesn’t have a timestamp).

wurl - A much improved web based HTTP client

Tue, 11 Mar 2014 07:00:00 +1000

Previously I wrote about Aristotle, my attempt at a usable http test client in the browser. Over the last two months Aristotle has been renamed wurl, received a much improved user interface and many new features.

Here is the short list of improvements:

new fullscreen tabbed layout
support for endpoints that don’t support CORS
realtime filtering of results
realtime rendering of views
save templates
request a template by url (http://wurl.azurewebsites.net/request?url=https://api.github.com/users/neovim/repos)

What’s Missing

wurl is optimised for what I need most, which is querying json apis. Some things not yet implemented are:

support for xml responses
identifying and linking urls in responses so that an api can be navigated
more advanced non-cors requests

Aristotle - A web based HTTP client

Sun, 09 Feb 2014 20:32:00 +1000

Working with HTTP APIs I have found that there is a lack of GUI tooling to experiment and test requests. APIs are often documented as cURL commands, which is great, but cURL is a poor interface for experimenting. For example, if I want to post a document via cURL, I need to do something like:

curl -vX POST https://api.github.com/users/liammclennan/repos 
    -H "content-type: application/json" -d '{"whatever": "value"}'

There are many options for testing HTTP APIs. Here some that I have tried.

Postman

Postman is a chrome app. It is good for crafting HTTP requests. One nice feature is that it records all of the requests, so it is easy to reply past requests. I don’t like that postman requires me to sign in to my google account to install it. Also, I have mostly stopped using Chrome for similar reasons.

Fiddler

Fiddler is a windows GUI proxy that provides the ability to execute arbitrary HTTP requests.

Aristotle

Aristotle exists because I want a HTTP client with the following features:

easily execute a wide range of HTTP requests
save and restore requests
nicely format the response headers and body
transform the response body and render views of the data
cross-platform
make requests to non-CORS-enabled APIs

After stuffing around for some time trying to design a nice interface for describing requests I realized that only developers need to experiment with HTTP APIs and therefore it is OK to require a bit of code to define the request, and jquery already has an API for defining a request that lots of developers are already familiar with. Therefore, when using Aristotle you define a request by calling $.ajax. The only actual requirement is to specify an expression that returns a jquery style promise that will be resolved with the right arguments. Here is one of the example requests templates used by Aristotle:

$.ajax({
    type: 'GET',
    url: 'https://api.github.com/users/liammclennan/repos',
    dataType: 'json'
});

Any request that can be specified as a call to $.ajax will work. Aristotle will then display the response headers and the response body as syntax highlighted json.

But wait, there’s more.

Often I am making requests to CouchDB and the results are difficult to interpret because there is too much data and json is not that easy to read. To solve this problem Aristotle allows you to define views. Corresponding to the above request to the github API Aristotle has a template to render the expected response. The code is:

<%
// manipulate the data some
var mapped = data.map(function (repo) {
   return repo.name;
   });   
%>

<!-- define a template -->
<ul>
    <% mapped.forEach(function(name) {%>
        <li><%=name%></li>
    <% }) %>
</ul>

It firstly maps the response data to something more useful for the view and then uses an underscore template (similar to ejs) to render a list of my github repos. The output is

angularserver
author-quiz
backbone-basics
browsertest
Builder
coffeescript-course
coffeescript-game-of-life
coffeescript-koans
couchpotato
dbc
dddbrisbane13-fsharp
dddbrisbane2012
designbycontract
dotfiles
eclipsewebsolutions.com.au
functional-game-of-life
gistblog
git-deploy
giv.n
googleajaxurls
haskell-notes
Herald
JavaScript-Koans
JsBdd
KeyRef
Liam-s-Presentations
listagram_build
mobile-dev
node-test
onashirt

Substitute any github user name in the url to get a list of that user’s github repositories.

What’s Missing

Most things are missing. Proper error handling is missing. Saving requests is missing. Support for non-cors-enabled APIs is missing. Support for non-json responses is missing. A large selection of useful request templates is missing.

I plan to keep working on Aristotle - mostly because it is a tool that I want to use.

Applying Functional Programming to .net Consulting Projects

Tue, 14 Jan 2014 20:32:00 +1000

The benefits of functional programming are well known, and well documented. The rest of this article will assume that this argument has been digested and accepted. Given the benefits of functional programming the relatively low adoption can be attributed to general resistance to change, the front-loaded learning curve, a degree of complacency, and uncertainty about how to accomplish common programming tasks with a functional toolset. This article focuses on the latter, particularly in the context of .net consulting projects.

Functional Programming for the .net Programmer

A common fallacy is that we can achieve the benefits of functional programming by applying a functional style within a procedural language. Immutability is essential to functional programming and procedural languages, such as c#, do not encourage or support immutability. Whilst a functional style of c# programming is certainly an improvement we must not pretend that it is a reasonable substitute for a proper functional programming language. For the programmer working with .net tools in the .net ecosystem F# is the obvious prime candidate for a functional programming language. It is the only first-class, functional-first .net language.

Some project managers worry that embracing a functional programming language will cause hiring difficulty. This fallacy possibly occurs because the push for functional programming generally comes from the programmers themselves. Non-programmers associated with software projects have little visibility into the rise and popularity of functional programming. Here in Brisbane the local functional programming user group has 484 members. Each year Brisbane hosts a well attended conference dedicated to functional programming. The amount of functional programming talent available for hire is underestimated.

Finally, the notion of sticking to tools that make hiring easier contradicts the ‘right tool for the job’ directive that we should strive for. If the majority of unemployed programmers were assembler experts would you chose assembler for your next project?

Mixed Solutions

An important tool when introducing F# to a project is the ability to include F# and other .net languages within the same application. This reduces risk considerably because we retain the ability to fall back to another .net language (usually c#) in the event that F# is deemed to be unsuitable for a particular purpose. The reality is that this is not likely to ever be necessary - still there is value in reducing the perception of risk.

A solution will be simpler, and more cohesive, if F# is used exclusively, however .net supports mixed solutions by allowing different projects to use different languages, and by supporting interoperability in both directions.

Interoperability Between F# and C#

Calling from F# into c# is a simple matter of using c# style syntax.

new RiakObjectId("logs", kvp.Key)

Note the use of the new keyword to create an object, paranthesis to delimit method arguments, and commas to separate method arguments.

Calling from c# into F# is similarly straightforward. F# functions appear in c# as static methods on static classes that map to the F# modules. The following F# and c# code are equivalent:

open Data
nowToTodayString DateTime.Now

Data.nowToTodayString(DateTime.Now)

F# for Web Applications

F# is unfortunately not currently treated as a first-class .net language by visual studio. There is no built-in template for creating a web or desktop application with F#. The only supported scenarios are library and command line projects. In the absense of built-in templates the community has provided some excellent templates for a wide variety of scenarios.

The visual studio online project templates include an excellent template called F# and C# Web Application template. It creates a minimal c# web project and an auxillary F# library containing as much of the code as possible, including the controllers, view models, and application instance. The only application code required to be implemented within the c# web application project is the views. The case study discussed later is built on this template.

F# for Windows Services

Despite there being no built-in template for F# windows services, F# is well suited to this task. The Visual C# Windows Service project actually does very little so it is trivial to manually implement an F# service, as described in the MSDN article Creating Windows Services in F#.

Implementing a service only requires a ServiceBase implementation and a service Installer implementation.

F# Tooling Support

F# tooling support is minimal, compared to c# and VB.NET. As mentioned previously there is a shortage of built-in project templates, but this is easily rectified by the online, community contributed templates. Many .net technologies, such as asp.net and WPF, do not explicitly support F#, meaning that the F# programmer has some extra work to do. Also, third party productivity tools, such as ReSharper, typically do not support F#. When our productivity tools become a contributing factor for foundational technology decisions it is time to question, “have we lost control?”.

F# Data Access

Most data access methods assume mutability, which negates some of the benefits of FP. The exceptions are the explicity immutable data stores, such as Datomic and Event Store and the http based datastores like CouchDB and Riak.

The case study included with this article uses Riak for its data store. F# has no direct support for Riak, however, it is trivial to use the CorrugatedIron client (written in c#) to communicate with Riak. All of the aforementioned data stores provide an HTTP interface, so a specialized client is not strictly necessary.

Case Study - Logging You

Logging You is a bad demonstration of F#’s advantages. It includes none of F#’s core competency (computation) and lots of F#’s weaker areas (interop, GUI programming, transactional data access). Indeed the purpose of this case study is to investigate the feasibility of F# as a standard tool for .net consultants, therefore the important question to answer is, “how well does it handle those weaker areas?”.

Logging you is a log dashboard built as an asp.net mvc web application. It’s two use cases are:

display a summary of the currently stored logs. This includes the alltime total count of logs, the count of today’s logs, and the list of today’s log messages.
allow the user to enter a new informational log message

These simple requirements forced the development of mvc integration, data queries and mvc model binding. F# handled all of these non-ideal (for a functional programming language) tasks with aplomb. Being a standard asp.net website Logging You is hosted on Azure without any special modifications. The source is available on bitbucket.

Find Out More

If you are interested in learning more about applied functional programming in .net, here are some helpful resources:

F# String Calculator

Thu, 07 Nov 2013 20:32:00 +1000

I’ve done the string calculator kata before, most recently in Haskell. Here it is again in F#:

let lToS (l:char list) = System.String.Concat(Array.ofList(l))

let sumString =
    let parse (l:string) =
        let guessDelimeter (l:string) =
            match Seq.toList l with
                | '/'::'/'::c::'\n'::_ -> c
                | _ -> Seq.find (fun i -> i = ',' || i = '\n') l
        let valuePart = (fun (s:string) ->
            match Seq.toList s with
                | '/'::'/'::c::'\n'::r -> r
                | v -> v) >> lToS
        Array.map int ((valuePart l).Split(guessDelimeter l))
    (parse >> Array.fold (fun acc i -> acc + i ) 0)

 sumString "3,9,11" |> printfn "%d"
 sumString "3\n9\n11" |> printfn "%d"
 sumString "//;\n3;9;11" |> printfn "%d"

Are we doing MVC wrong?

Sun, 20 Oct 2013 20:32:00 +1000

The naive way to build user interfaces with the MVC pattern is to include the application’s logic in controller actions - effectively recreating transaction script that is:

organizes business logic by procedures where each procedure handles a single request from the presentation.

Recognising that this is bad, many smart people have thought about how to do mvc better. A couple of things still bother me:

What is the responsibility of the controller? They always seem to combine UI decisions (choosing a view, reading the request, http status codes, http redirects) with application logic.
The application itself has no interface. Controllers are a UI that consume the application, but there is no clear application boundary that defines what the application itself does.

To solve these problems I have started using a different technique for mvc. I will demonstrate by refactoring a typical controller.

A Typical Controller

public class LibraryController : Controller
{
    private readonly IBookRepository _bookRepository;
    private readonly IBorrowings _borrowings;
    private readonly IUsers _users;

    public LibraryController(IBookRepository bookRepository, IBorrowings borrowings, IUsers users)
    {
        _bookRepository = bookRepository;
        _borrowings = borrowings;
        _users = users;
    }

    public ActionResult Catalogue()
    {
        return View(_bookRepository.AllBooks());
    }

    public ActionResult Borrow(BookModel book)
    {
        var fullCatalogue = _bookRepository.AllBooks();
        var bookIsInCatalogue = fullCatalogue.Any(b => b.ISBN == book.ISBN);
        if (!bookIsInCatalogue)
        {
            SetErrorFlash("The selected book is not in the catalogue");
            return RedirectToAction("Catalogue");
        }

        var borrowedBooks = _borrowings.BorrowedBooks();
        var bookIsOutOnLoan = borrowedBooks.Any(b => b.ISBN == book.ISBN);
        if (bookIsOutOnLoan)
        {
            SetErrorFlash("The selected book is currently out on loan");
            return RedirectToAction("Catalogue");
        }

        var user = _users.GetAuthenticatedUser();
        var canBorrow = user.HasPermissionToBorrow();
        if (!canBorrow)
        {
            SetErrorFlash("The selected book is currently out on loan");
            return RedirectToAction("Catalogue");
        }

        var bookEntity = _bookRepository.GetByISBN(book.ISBN);
        var result = _borrowings.Borrow(bookEntity);

        if (result != null)
        {
            SetSuccessFlash("Book borrowed");
            return View("Borrowed", result);
        }
        else
        {
            return View("BorrowFailed");
        }
    }

    private void SetErrorFlash(string message) {}
    private void SetSuccessFlash(string message) {}
}

One problem here is that there is no differentiation between the user interface behaviour and the function provided by the application. Both are implemented in the controller, mixed together in the same method (Borrow).

Another problem is that as the library functionality grows more dependencies will be added to the controller. Not all of the actions will use all of the dependencies, so cohesion is low, and coupling is high.

Application Feature API

Instead of the above controller I prefer to make the features an application provides into a first class API. I model each feature as a class, so the logic in Borrow action moves into a feature class called BorrowABook.

The genesis of the ideas show here come from Uncle Bob’s presentation ‘Architecture the Lost Years’.

public class BorrowABook
{
    public enum BorrowABookOutcomes
    {
        NotInCatalogue,
        OutOnLoan,
        InsufficientPermission,
        BorrowFailed,
        BorrowSucceeded
    }

    public Feature<BorrowABookOutcomes,Book> Borrow(BookModel book)
    {
        // .. implement application logic
        return new Feature<BorrowABookOutcomes,Book>(BorrowABookOutcomes.BorrowSucceeded, borrowedBook);
    }
}

Features return a glorified tuple (Feature<TOutcome,TResult>) that combines an enum value that represents what the outcome was, with a value that represents the result of the operation. In this case I am returning the success outcome and the book that was borrowed. As the boundary of your application Features are an excellent target for behaviour tests. You can test them without dealing with UI layer concerns.

Simplified Controller

Given a feature such as BorrowABook the controller action simplifies to the following:

public ActionResult Borrow(BookModel book)
{
    var result = _feature.Borrow(book);
    switch (result.Outcome)
    {
        case BorrowABook.BorrowABookOutcomes.NotInCatalogue:
            {
                SetErrorFlash("The selected book is not in the catalogue");
                return RedirectToAction("Catalogue");
            }
        case BorrowABook.BorrowABookOutcomes.OutOnLoan:
            {
                SetErrorFlash("The selected book is currently out on loan");
                return RedirectToAction("Catalogue");
            }
        case BorrowABook.BorrowABookOutcomes.InsufficientPermission:
            {
                SetErrorFlash("You do not have permission to borrow books");
                return RedirectToAction("Catalogue");
            }
        case BorrowABook.BorrowABookOutcomes.BorrowFailed:
            {
                return View("BorrowFailed");
            }
        case BorrowABook.BorrowABookOutcomes.BorrowSucceeded:
            {
                SetSuccessFlash("Book borrowed");
                return View("Borrowed", result.Result);
            }
        default: throw new Exception("Unknown result " + result.Outcome);
    }
}

Note that the application logic has all been removed. The controller action is reduced to the minimum required to implement the UI.

Update

Thanks to David Tchepak for providing this F# version of the above example. Note that it is untested and meant simply as an example of what an F# version might look like.

dbc 2.0.0

Sat, 24 Aug 2013 20:32:00 +1000

Today I released a new version of my JavaScript Design-by-Contract Library (dbc). As well as documentation and other minor improvements the two big new features are:

makeConstructor

MakeConstructor generates a constructor from a spec. The constructor validates that the created objects meet the spec. ie

    var Person = dbc.MakeConstructor({
       name: [{validator:'type',args:'string'}],
       age: [{validator:'type',args:'number'}]
    });
    Person.prototype.canDrive = dbc.wrap(function () { 
         return this.age > 16;
    }, null, [{validator:'type',args:'number'}]);
    var p = new Person('John', 22);
    p.canDrive(); // true

wrap

Wrap returns a wrapped function that applies ‘specs’ validators to the functions arguments and ‘returnSpec’ validators to the return value. eg

var add = dbc.wrap(function (f,s) { 
    return f + s;
}, {
    // validators for the first arg
    0: [{validator:'type', args:['number']}],
    // validators for the second arg
    1:[{validator:'type', args:['number']}]
},
    // validators for the return value 
    [{validator: 'type', args:['number']}]);

Dbc is available on github or via npm

npm install dbc

CoffeeScript Game of Life with Ristretto Type Annotations

Sun, 11 Aug 2013 20:32:00 +1000

This weekend at campjs one of my projects has been porting my Haskell Game of Life to CoffeeScript, with type annotations courtesy of Ristretto.

https://github.com/liammclennan/coffeescript-game-of-life

Ristretto

Given a function from string to int:

number_of_chars = (s) -> s.length

Ristretto can enfore the types given the following annotated version:

number_of_chars = T 'number_of_chars :: String -> Int', (s) ->
    s.length

The format of the type annotation will be familiar to Haskellers. Read :: as ‘has the type’. All types to the left of a -> are function arguments and the type to the right of the last -> is the return type. So the above function number_of_chars has the type String to Int.

Ristretto can also define data types. For example:

T 'typedef Person :: { name: String, age: Int }'
class Person
    constructor: (@name, @age) ->

The JavaScript equivalent is:

T('typedef Person :: { name: String, age: Int }');
function Person(name,age) {
    this.name = name;
    this.age = age;
}

Game of Life

Here is my Game of Life implementation:

require 'coffee-script'
T = require './ristrettoAssert'
_ = require 'underscore'

# Define a type for a living cell
T 'typedef Cell :: {x:Int, y:Int}'
class Cell
    constructor: (@x,@y) ->

# seed the initial living cells
r_pentomino = [new Cell(2, 1), new Cell(3, 1), new Cell(1,2), new Cell(2,2), new Cell(2,3)]

# determine if a coordinate (x,y) is a neighbour of a cell
is_neighbour = T('is_neighbour :: Cell -> Int -> Int -> Bool', (c, x, y) ->
    Math.abs(x-c.x) < 2 and Math.abs(y-c.y) < 2 and (not (x is c.x and y is c.y))
)

living_neighbours = T('living_neighbours :: [Cell] -> Int -> Int -> Int', (cs,x,y) ->
    (cell for cell in cs when is_neighbour cell, x, y).length
)

is_on = T('is_on :: [Cell] -> Int -> Int -> Bool', (cs,x,y) ->
    (cell for cell in cs when cell.x is x and cell.y is y).length > 0
)

largest_x = T 'largest_x :: [Cell] -> Int', (cs) ->
    _.max(cs, (i) -> i.x).x

largest_y = T 'largest_y :: [Cell] -> Int', (cs) ->
    _.max(cs, (i) -> i.y).y

alive_next_generation = T 'alive_next_generation :: [Cell] -> Int -> Int -> Bool', (cs,x,y) ->
    false
    (is_on(cs, x, y) and 2 <= living_neighbours(cs, x, y) <=3) or ((not is_on(cs, x, y)) and (living_neighbours(cs,x,y) is 3))

tick = T 'tick :: [Cell] -> [Cell]', (cs) ->
    ncs = []
    for x in [0..largest_x(cs)+1]
        for y in [0..largest_y(cs)+1]
            ncs.push(new Cell(x,y)) if alive_next_generation cs, x, y
    ncs

visualize = T 'visualize :: [Cell] -> String', (cs) ->
    result = ''
    for y in [0..largest_y(cs)+1]
        for x in [0..largest_x(cs)+1]
            result += if is_on(cs,x,y) then 'x' else '-'
            if x is largest_x(cs)+1
                result += '\n'
    result

[1..50].reduce((p,c,i) ->
    console.log visualize p
    tick p
, r_pentomino)

Thoughts

While re-implementing game of life my experience was that ristretto frequently alerted me to type errors. Without Ristretto these errors would have manifested later in the program with a greater distance between the problem and the runtime error.

Ristretto does not provide the full benefits of a static type checker. It is more like a type-based design-by-contract library. Personally, I have been frustrated by type errors in Javascript, which led me to implement a basic runtime type checker for JavaScript, but I think that Ristretto is a better and more logically consistent solution.

Output

The output of my CoffeeScript Game of Life for the first 50 generations takes 22s on my Chromebook and is printed below. To generate the full 1103 generations before this pattern stabilises will likely require a better algorithm.

-----
--xx-
-xx--
--x--
-----

-----
-xxx-
-x---
-xx--
-----

--x--
-xx--
x--x-
-xx--
-----

-xx--
-xxx-
x--x-
-xx--
-----

-x-x-
x--x-
x--x-
-xx--
-----

--x---
xx-xx-
x--x--
-xx---
------

-xxx--
xx-xx-
x--xx-
-xx---
------

xx-xx-
x-----
x---x-
-xxx--
------

xx----
x--xx-
x-xx--
-xxx--
--x---
------

xx----
x--xx-
x-----
------
-xxx--
------

xx--
x---
----
-xx-
--x-
--x-
----

xx---
xx---
-x---
-xx--
--xx-
-----

xx---
--x--
-----
-x-x-
-xxx-
-----

-x---
-x---
--x--
-x-x-
-x-x-
--x--
-----

-----
-xx--
-xx--
-x-x-
-x-x-
--x--
-----

-----
-xx--
x--x-
xx-x-
-x-x-
--x--
-----

------
-xx---
x--x--
xx-xx-
xx-x--
--x---
------

------
-xx---
x--xx-
---xx-
x--xx-
-xx---
------

-------
-xxx---
-x--x--
--x--x-
-x--x--
-xxx---
-------

--x----
-xxx---
-x--x--
-xxxxx-
-x--x--
-xxx---
--x----
-------

-xxx---
-x-x---
x----x-
xx---x-
x----x-
-x-x---
-xxx---
-------

-x-x----
xx-xx---
x-x-x---
xx--xxx-
x-x-x---
xx-xx---
-x-x----
--x-----
--------

xx-xx-
x---x-
--x---
x-x-x-
--x---
x---x-
xx-xx-
--x---
------

xx-xx-
x-x-x-
------
--x---
------
x-x-x-
xxxxx-
-xxx--
------

xxxxx-
x-x-x-
-x-x--
------
-x-x--
x-x-x-
x---x-
x---x-
--x---
------

x-x-x--
x---x--
-xxx---
-------
-xxx---
x-x-x--
x---xx-
-x-x---
-------

-x-x---
x---x--
-xxx---
-------
-xxx---
x-x-xx-
x-x-xx-
----x--
-------

-------
x---x--
-xxx---
-------
-xxxx--
x----x-
-------
---xxx-
-------

-------
-xxx---
-xxx---
----x--
-xxxx--
-xxxx--
-----x-
----x--
----x--
-------

--x----
-x-x---
-x--x--
----x--
-x---x-
-x---x-
--x--x-
----xx-
-------

--x-----
-x-x----
--xxx---
----xx--
----xx--
-xx-xxx-
-----xx-
----xx--
--------

--x-----
-x--x---
--x--x--
--------
--------
---x----
---x----
----xxx-
--------

-------
-xxx---
-------
-------
-------
-------
---x-x-
----xx-
-----x-
-------

--x-----
--x-----
--x-----
--------
--------
--------
-----x--
-----xx-
----xx--
--------

--------
-xxx----
--------
--------
--------
--------
-----xx-
------x-
----xxx-
--------

--x------
--x------
--x------
---------
---------
---------
-----xx--
----x--x-
-----xx--
-----x---
---------

---------
-xxx-----
---------
---------
---------
---------
-----xx--
----x--x-
----xxx--
-----xx--
---------

--x------
--x------
--x------
---------
---------
---------
-----xx--
----x--x-
----x--x-
----x-x--
---------

---------
-xxx-----
---------
---------
---------
---------
-----xx--
----x--x-
---xx-xx-
-----x---
---------

--x------
--x------
--x------
---------
---------
---------
-----xx--
---xx--x-
---xx-xx-
----xxx--
---------

---------
-xxx-----
---------
---------
---------
---------
----xxx--
---x---x-
-------x-
---xx-xx-
-----x---
---------

--x-------
--x-------
--x-------
----------
----------
-----x----
----xxx---
----xx-x--
---xx--xx-
----xxxx--
----xxx---
----------

----------
-xxx------
----------
----------
----------
----xxx---
----------
-------xx-
---x----x-
--------x-
----x--x--
-----x----
----------

--x--------
--x--------
--x--------
-----------
-----x-----
-----x-----
-----xxx---
-------xx--
--------xx-
-------xx--
-----------

-----------
-xxx-------
-----------
-----------
-----------
----xx-----
-----x-xx--
---------x-
---------x-
-------xxx-
-----------

--x---------
--x---------
--x---------
------------
------------
----xxx-----
----xxx-x---
---------x--
---------xx-
--------xx--
--------x---
------------

------------
-xxx--------
------------
------------
-----x------
----x-xx----
----x-xx----
-----x--xxx-
----------x-
--------x-x-
--------xx--
------------

--x----------
--x----------
--x----------
-------------
-----xx------
----x--x-----
----x----x---
-----xxxxxx--
--------x-xx-
--------x-x--
--------xx---
-------------

-------------
-xxx---------
-------------
-------------
-----xx------
----x-x------
----x----xx--
-----xxx---x-
------x----x-
-------xx-xx-
--------xx---
-------------

--x-----------
--x-----------
--x-----------
--------------
-----xx-------
----x-x-------
----x--x--x---
-----xxx---x--
-----x--x--xx-
-------xx-xx--
-------xxxx---
--------------

Ubuntu Linux on the Samsung Chromebook

Fri, 26 Jul 2013 20:32:00 +1000

Are you looking for a cheap, poorly made, underpowered ultrabook? If yes, then the Samsung Chromebook is for you!

It may be a bit slow, and it may be rough, but I love this thing. It is amazingly fast considering it has the processor from a 10” tablet. The 1366 x 768 display has a narrow viewing angle but is otherwise quite nice. The chromebook weighs 1.1kg and has a 6 hour battery life.

The chromebook I like, and ChromeOS even grew on me somewhat, but I bought my chromebook to run Linux.

Converting the Samsung Chromebook to Ubuntu

First I tried Crouton, which is a side-by-side Ubuntu with ChromeOS, but the Ubuntu experience was third rate. I had better luck with ChrUbuntu, which is an Ubuntu installer for Chromebooks. The default installation works well but I was left with some minor issues.

Fixing the Trackpad

After installing ChrUbuntu the trackpad will be rubbish. It is not hard to find a fix on the internet. After running that fix my trackpad worked great. In fact, it is one of the best trackpads I have used.

Fixing Bash

The terminal was driving me nuts. The up arrow didn’t retrieve history and tab completion didn’t work. So I posted a question on the usually helpful askubuntu where unfortunately my question was closed, because apparently ChrUbuntu is not an Ubuntu.

Could have fooled me.

Anyway I found the solution myself. The problem is that ChrUbuntu doesn’t use bash for the terminal. So install bash:

sudo apt-get install bash

then set bash as the default terminal:

chsh -s /bin/bash

Enjoy.

dbc type assertion library

Tue, 16 Jul 2013 20:32:00 +1000

Today I updated my javascript dbc assertion library. I wanted to be able to guarantee that javascript objects match certain type constraints. Imagine a backbone view:

var View = Backbone.View.extend({

    render: function () {
        this.$el.html(this.model.get('a') + ' ' + this.model.get('b'));
    }

});

We can see from inspection that this view will fail to render if its model does not have an a and a b property. dbc can improve this as follows:

var View = Backbone.View.extend({
    initialize: function () {
        dbc.check(this.model.toJSON(), {
            a: [{validator: 'required'}],
            b: [{validator: 'required'}]
        });
    },
    render: function () {
        this.$el.html(this.model.get('a') + ' ' + this.model.get('b'));
    }
});

It will still fail, but now it will fail when the view is created, and it will be obvious that the failure was caused by bad data.

The Readme

dbc is a small library for design-by-contract defensive coding in javascript.

It focuses especially on type assertions in an attempt to provide a small compensation for JavaScript’s unfortunate dynamicness.

Have a function that requires a numeric argument? Try:

function (num) {
    dbc.type(num, 'number');
    // the rest of your function
}

or if the function argument is an object, and you want to collect the validation results:

function (myComplexObject) {
    var messages = dbc.validate(myComplexObject, {
        firstProp: [{validator: 'required'}]
    });
    // the rest of your function
}

Features

Check a value:

    dbc.type(1, 'number', 'optional error message');
    dbc.isFunction(function () {}, 'optional error message');
    dbc.functionArity(function () {}, 0, 'optional error message');
    // etc

Check an object:

    dbc.check({
        a: 1,
        b: "cat in the hat",
        c: function (f, s) { return f + s; }
    }, {
        a: [
            {validator: 'required', args: ['optional error message']}, 
            {validator: 'type', args: ['number']}, 
            {validator: 'custom', args: [ function (v) { 
                // custom validator function should return a boolean
                return v > 0; 
            }]}
        ],
        b: [{validator: 'type', args: ['string']}],
        c: [{validator: 'isFunction'}, {validator: 'functionArity', args: [2, {message: 'This is a more advanced error object', field: 'c'}]}]
    });

Validate an object (same as check except that it returns an array of errors instead of throwing an exception at the first error):

    dbc.validate({
        a: 1,
        b: "cat in the hat",
        c: function (f, s) { return f + s; }
    }, {
        a: [
            {validator: 'required', args: ['optional error message']}, 
            {validator: 'type', args: ['number']}, 
            {validator: 'custom', args: [ function (v) { 
                // custom validator function should return a boolean
                return v > 0; 
            }]}
        ],
        b: [{validator: 'type', args: ['string']}],
        c: [{validator: 'isFunction'}, {validator: 'functionArity', args: [2, {message: 'This is a more advanced error object', field: 'c'}]}]
    });

Test

npm install -g jasmine-node
jasmine-node --coffee spec/

Resourceserver as a backend for SPA and Backbone apps

Mon, 15 Jul 2013 20:32:00 +1000

Some reddit pundits were confused about resourceserver and what the point of it might be. In this post I demonstrate the use of resourceserver as a backend for the todomvc SPA demo.

Using Resourceserver as a backend for an SPA / Backbone app

Install redis

You’re on your own for this one. It is not hard and can be accomplished on *nix or windows.

Get resourceserver

git clone git@github.com:liammclennan/resourceserver.git

Get Todomvc

Todomvc is a reference app used to compare client-side web app libraries. It includes the same app (a todo app) built with a wide variety of libraries. We will use it as a client-side app to connect to resource server.

git clone git@github.com:tastejs/todomvc.git

Enable redis

This is optional. By default resourceserver will just store resources in memory. Edit resourceserver’s config.coffee:

module.exports =

   use_redis: true

   redis_config:
     host: 'localhost'
     port: 6379

Make sure the redis_config settings match your redis installation.

Start resourceserver

In the directory where you cloned resourceserver:

npm start

Modify todomvc

We need to make a small modification to todomvc to enable server persistence.

Modify architecture-examples/backbone/js/collections/todos.js:

-               localStorage: new Backbone.LocalStorage('todos-backbone'),
+               url: 'http://localhost:3002/todos',

Enjoy

Resourceserver

Sun, 14 Jul 2013 20:32:00 +1000

Lately I have been resurecting an old project of mine - now called Resourceserver.

It is a simple, in-memory, resource-oriented http server, designed to be used during testing and development of rich-client web apps.

Here is the README:

Resourceserver

TODO: Implement a persistent version (probably using redis).

Implements an in-memory resource oriented HTTP server, provding 5 basic operations (shown in curl_tests.sh)

POST /:collection

Create a new resource.

curl -vX POST http://localhost:3002/people -H 'content-type: application/json' -d '{"name": "Liam", "age": 29}'

{
  "name": "Liam",
  "age": 29,
  "id": 2
}

curl -vX POST http://localhost:3002/people -H 'content-type: application/json' -d '{"name": "Noah", "age": 1}'

{
  "name": "Noah",
  "age": 1,
  "id": 2
}

GET /:collection/:id

Retrieve the :collection resource with id :id.

curl -v http://localhost:3002/people/1

{
  "name": "Liam",
  "age": 29,
  "id": 1
}

GET /:collection

Retrieve an array of all :collection resources.

curl -v http://localhost:3002/people

[
  {
    "name": "Liam",
    "age": 29,
    "id": 1
  },
  {
    "name": "Noah",
    "age": 1,
    "id": 2
  }
]

PUT /:collection/:id

Override the :collection resource with id :id.

curl -vX PUT http://localhost:3002/people/1 -H 'content-type: application/json' -d '{"name": "LiamO", "age": 30}'

{
  "name": "LiamO",
  "age": 30,
  "id": "1"
}

DELETE /:collection/:id

Delete the :collection resource with id :id.

curl -vX DELETE http://localhost:3002/people/1

It uses the CORS headers to allow cross-origin requests.

Usage

Clone the repository
Install node.js
Install the dependencies with npm install
Start the server with npm start
[Optional] Run tests with cd test && ./curl_tests.sh

Clojure FizzBuzz

Wed, 10 Jul 2013 20:32:00 +1000

Here is my clojure fizzbuzz:

(defn fizzbuzz []
  (let [divisibleBy3 (fn [x]
                       (= 0 (mod x 3)))
        divisibleBy5 (fn [x]
                       (= 0 (mod x 5)))
        calcz (fn [i]
               (cond
                 (and (divisibleBy3 i) (divisibleBy5 i)) "FizzBuzz"
                 (divisibleBy5 i) "Buzz"
                 (divisibleBy3 i) "Fizz"
                 :else i))
        ]
     (map calcz (range 1 101))))

(fizzbuzz)

Haskell String Calculator Kata

Fri, 31 May 2013 20:32:00 +1000

The Kata

Roy Osherove’s string calculator kata is a very simple kata designed (I believe) to demonstrate the basic TDD workflow.

The basic idea is to write a program (function?) that sums the integers in a string. “1,2,3” -> 6.

Setting Up a Haskell Package for the Kata

I like to start by initialising an isolated package for each new project.

I used hsenv to create an isolated environment (similar to rbenv etc). Then added a cabal configuration file:

name:                stringcalc
version:             0.1.0.0
author:              Liam
build-type:          Simple
cabal-version:       >=1.8

library
  exposed-modules:     StringCalc
  -- other-modules:       
  hs-source-dirs:      src/
  build-depends:       base ==4.6.*

test-suite tests
  type:               exitcode-stdio-1.0
  hs-source-dirs:     test/
  main-is:            Spec.hs
  build-depends:      base ==4.6.*
                      , hspec
                      , stringcalc

This says that my project is library exporting one module, StringCalc, and that it has a test project in Spec.hs.

To get this to build / run I need to create the library src/StringCalc.hs:

module StringCalc where

and the test file test/Spec.hs (these tests use the hspec test library):

import Test.Hspec
import Test.QuickCheck
import Control.Exception (evaluate)
import StringCalc

main :: IO ()
main = hspec $ do

  describe "Example Spec" $ do
    it "should pass" $ do
      (0 :: Int) `shouldBe` (0 :: Int)

then build with cabal:

cabal configure --enable-tests
cabal install --enable-tests --only-dependencies
cabal build
cabal test

Implementing the Kata

Step 1: “”

The first thing to do is to get the program to handle the empty string input.

Start with the test:

main :: IO ()
main = hspec $ do

  describe "StringCalc" $ do
    it "should sum the empty string" $ do
      sumString "" `shouldBe` (0 :: Int)

and then the implementation:

module StringCalc (sumString) where

sumString :: String -> Int
sumString "" = 0

next deal with adding a single number:

describe "add single number" $ do
      it "should add positive number" $ do
        sumString "1" `shouldBe` (1 :: Int)

      it "should add negative number" $ do
        sumString "-2" `shouldBe` (-2 :: Int)

sumString :: String -> Int
sumString "" = 0
sumString s = sum $ map rInt (split s)
  where rInt :: String -> Int
        rInt s = read s

The next step is, “Allow the Add method to handle an unknown amount of numbers”:

describe "add two numbers" $ do
      describe "add 3 and 9" $ do
        it "should give 12" $ do
          sumString "3,9" `shouldBe` (12 :: Int)

module StringCalc (sumString) where

sumString :: String -> Int
sumString "" = 0
sumString s = sum $ map rInt (split s)
  where rInt :: String -> Int
        rInt s = read s

split :: String -> [String]
split "" = []
split (',':ss) = split ss
split s = [takeWhile isNotComma s :: String] ++ (split $ dropWhile isNotComma s)
  where isNotComma :: Char -> Bool
        isNotComma c = c /= ','

The sumString function now needs to be changed to use commas or newlines as the separator:

describe "with newline separators" $ do
  let input = "1\n2\n3"

  it "should give 6" $ do
    sumString input `shouldBe` (6::Int)

sumString :: String -> Int
sumString "" = 0
sumString s = sum $ map rInt (split s)
  where rInt :: String -> Int
        rInt s = read s

split :: String -> [String]
split "" = []
split (',':ss) = split ss
split ('\n':ss) = split ss
split s = [takeWhile isNotSeparator s :: String] ++ (split $ dropWhile isNotSeparator s)
  where isNotSeparator :: Char -> Bool
        isNotSeparator c = c /= ',' && c /= '\n'

The final requirement is to support different delimeters. Here it is, with some refactoring:

module StringCalc (sumString) where

import Data.List
import Data.List.Split

hasExplicitDelimeter :: String -> Bool
hasExplicitDelimeter ('/':'/':s:'\n':ss) = True
hasExplicitDelimeter _                   = False

trimmed :: String -> String
trimmed input | hasExplicitDelimeter input = drop 4 input
              | otherwise                  = input

sumString :: String -> Int
sumString "" = 0
sumString s = sum . map rInt $ splitOn [guessDelimeter s] (trimmed s)

  where rInt :: String -> Int
        rInt s = read s

        guessDelimeter :: String -> Char
        guessDelimeter input  | hasExplicitDelimeter input     = input !! 2
                              | "\n" `isInfixOf` trimmed input = '\n'
                              | otherwise                      = ','

UPDATE: Michael Feathers performs the string calculator kata and arrives at a similar solution, except that his is point free

Reading Code - robots-txt

Thu, 30 May 2013 20:32:00 +1000

Scott Hanselman has or had a series called the weekly source code, in which he reviewed open source code as a learning exercise. I have enjoyed looking through the source of robots-txt by Mark Wotton so I will follow Mr Hanselman’s lead and document my experience spelunking this source.

robots-txt.cabal

The first thing I noticed, in the root of the repository, is the robots-txt.cabal file. This file defines how the project will be packaged for use as a dependency of other projects and also the dependencies of the current project for installation. robots-txt is defined as a library, meaning that it exports things to be used in other applications and does not have its own executable.

Below the standard package definition robots-txt.cabal includes a setting I have not seen before, test-suite. robots-txt uses the older exitcode-stdio-1.0 + main-is style of test suite ‘that indicate test failure with a non-zero exit code’. Adding the test-suite configuration means that the project’s tests can be run with cabal test. Yet another reason to add cabal configuration to all projects, not just the ones that are intended to be distributed as cabal packages.

The usage of hs-source-dirs for both the library and test-suite configuration is also interesting.

Tests

The robots-txt.cabal file defines the entry point for testing as test/Spec.hs. Spec.hs references hspec-discover in a GHC_OPTIONS pragma. hspec-discover looks for and executes all tests in the directory and any subdirectories. This is a convenient way to avoid having to maintain a list of tests to execute. In the case of robots-txt there is only one test file to discover test/RobotSpec.hs.

RobotSpec.hs

RobotSpec.hs uses hspec and the specification style of tests to define the behaviour of the project. It is curious that this style (describe… it… should) is popular on so many platforms and yet I have not seen a .NET implementation.

RobotSpec.hs uses the {-# LANGUAGE CPP #-} pragma, which allows it to use the special symbol FILE which is presumably replaced by the current filename pre compilation.

I don’t understand how RobotSpec is executed, since it doesn’t declare a main function like the hspec examples and it doesn’t export anything.

At first I didn’t understand the use of forM_, but a bit of googling, and particularly this stackoverflow question provided some insight.

Network.HTTP.Robots

The actual program is contained within the Network.HTTP.Robots module. It defines a number of data structures and some functions for operating on them. Some of it is confusing to me, such as the directiveP function. I suspect I need to spend some more time with Control.Applicative

Conclusion

Looking through Robots has been both fun and enlightening. The code is simple enough for me to understand (mostly) and it is small enough to be easily digestable. I’ve learnt a few things, particularly about how to setup testing which is an area that I have been planning to improve.

Maybe Haskell?

Sun, 19 May 2013 20:32:00 +1000

Usual disclaimer - I don’t know Haskell at all. What follows is my rambling experimentation with the Maybe type.

Haskell has a useful type called Maybe. F# and probably most other functional languages have something similar. Even C# has Nullable which is somewhat in similar in that its purpose is “to encapsulate an optional value”. To put it another way, Maybe is a parameterised or polymorphic type that represents that we possibly have a value of the type parameter. So Maybe Int means maybe we have an Int, maybe we don’t (ie we have Nothing).

Because Haskell doesn’t allow null values we might use Maybe for a function that returns its input if the input is a positive number.

giveIfEvan :: Int -> Maybe Int
giveIfEvan n = if n `mod` 2 == 0
                then Just n 
                else Nothing

Maybe is a monad, which, to me, means primarily that it provides a function to convert from a Maybe of some type to a Maybe of some other type (>>= :: Maybe a -> Maybe b).

Imagine I want a function that given a Maybe Int adds 1 to the value (if there is a value). One terrible way to write this is:

addOne :: Maybe Int -> Maybe Int
addOne n = if isJust n
            then Just (fromJust n + 1)
            else Nothing

or with pattern matching:

addOne :: Maybe Int -> Maybe Int
addOne (Just a) = Just (a + 1)
addOne Nothing = Nothing

since Maybe is a monad we can use do notation:

addOne n = do
        v <- n
        return (v + 1)

The benefit here is that addOne handles the nothing case without an explicit conditional.

If you don’t mind using a lambda you can use the de-sugared version of do >>= (bind):

addOne n = n >>= \v -> return (v + 1)

Or instead of a lambda we could define an extra function:

increment :: Int -> Maybe Int
increment v = return (v + 1)

addOne :: Maybe Int -> Maybe Int
addOne n = n >>= increment

Here is a complete program. Try changin the 8 to an odd number to see the nothing case.

import Data.Maybe

giveIfEvan :: Int -> Maybe Int
giveIfEvan n = if n `mod` 2 == 0
                then Just n 
                else Nothing

addOne :: Maybe Int -> Maybe Int
addOne n = do
            v <- n
            return (v + 1)

main = do
  return (addOne $ giveIfEvan 8)

Functional Game of Life part 3

Sun, 12 May 2013 20:32:00 +1000

This is the third part of a series investigating a functional solution to Conway’s Game of Life. Part 1 introduced the problem and implemented the HTML rendering of a Game of Life world using a functional style in C#. Part 2 ported the C# code from part 1 to Haskell. The job now is to implement the core of a functional Game of Life, that is a function to convert one generation of cells to the next generation of cells.

Implementing Game of Life

The type declaration for the Game of Life generation function is:

liveGeneration :: [Cell] -> [Cell]

That is, it is a function that converts a list of living cells to another list of living cells. And here is the implementation:

liveGeneration cs = [ Cell x y | x <- [0..(largestX cs) + 1], y <- [0..(largestY cs) + 1], aliveNextGeneration cs x y]

  where aliveNextGeneration cs xv yv = (isOn cs xv yv && (livingNeighbours cs xv yv) `elem` [2,3])
          || (not (isOn cs xv yv) && (livingNeighbours cs xv yv) == 3)

        livingNeighbours cs xv yv = length $ filter isNeighbour cs

          where isNeighbour (Cell xa ya)  = (xa == xv -1 && ya == yv -1)    -- top left
                                  || (xa == xv && ya == yv - 1)     -- above
                                  || (xa == xv + 1 && ya == yv -1)  -- top right 
                                  || (xa == xv - 1 && ya == yv)     -- left
                                  || (xa == xv + 1 && ya == yv)     -- right
                                  || (xa == xv - 1 && ya == yv + 1) -- bottom left
                                  || (xa == xv && ya == yv + 1)     -- below
                                  || (xa == xv + 1 && ya == yv + 1) -- bottom right

Note that the liveGeneration function depends on two other functions: aliveNextGeneration and livingNeighbours. aliveNextGeneration is a predicate that indicates if a given point (xv, yv) will be alive in the next generation. To determine if a cell will be alive in the next generation we need to know how many living neighbours it has - that is what livingNeighbours is for. livingNeighbours has its own local function isNeighbour which is a predicate that tests if a cell is a neighbour.

The following system is a classic Game of Life state known as the R-pentomino:

The following Haskell program runs the R-pentomino through 1 generation of the game of life: import Gol

rpentomino = [Cell 2 1, Cell 3 1, Cell 1 2, Cell 2 2, Cell 2 3]

main = do
  putStrLn $ render rpentomino

Producing the following output:

Ten generations:

Fifty generations:

Two hundred generations:

At this scale the inefficiency of my algorithm is exposed. Evaluating 200 generations takes a long time.

Here is the full source for my Haskell Game of Life:

module Gol 
(
  Cell(Cell),
  render,
  liveGeneration
) where

import Text.Printf
import Data.List

-- the definition of a living cell
data Cell = Cell {
    x       :: Int,
    y       :: Int
  } deriving (Show,Eq,Ord)

-- convert the state of the system to a html string
render :: [Cell] -> String
render cs = concat [renderLocation x y (isOn cs x y) | x <- [0..xBound], y <- [0..yBound]]
  where xBound = largestX cs
        yBound = largestY cs
        renderLocation xv yv on = printf 
          "<div class=\"cell %s\" style=\"left: %dpx; top: %dpx;\">&nbsp;</div>\n" 
          (if on then "on" else "") (20 * (xv+1)) (20 * (yv+1))

-- transition the system through one generation
liveGeneration :: [Cell] -> [Cell]
liveGeneration cs = [ Cell x y | x <- [0..(largestX cs) + 1], y <- [0..(largestY cs) + 1], aliveNextGeneration cs x y]

  where aliveNextGeneration cs xv yv = (isOn cs xv yv && (livingNeighbours cs xv yv) `elem` [2,3])
          || (not (isOn cs xv yv) && (livingNeighbours cs xv yv) == 3)

        livingNeighbours cs xv yv = length $ filter isNeighbour cs

          where isNeighbour (Cell xa ya)  = (xa == xv -1 && ya == yv -1)    -- top left
                                  || (xa == xv && ya == yv - 1)     -- above
                                  || (xa == xv + 1 && ya == yv -1)  -- top right 
                                  || (xa == xv - 1 && ya == yv)     -- left
                                  || (xa == xv + 1 && ya == yv)     -- right
                                  || (xa == xv - 1 && ya == yv + 1) -- bottom left
                                  || (xa == xv && ya == yv + 1)     -- below
                                  || (xa == xv + 1 && ya == yv + 1) -- bottom right

isOn :: [Cell] -> Int -> Int -> Bool
isOn cs xv yv = any cellMatch cs
  where cellMatch c = x c == xv && y c == yv

largestX :: [Cell] -> Int
largestX cs = maximum $ map x cs

largestY :: [Cell] -> Int
largestY cs = maximum $ map y cs

Functional Game of Life part 2 (Haskell)

Wed, 08 May 2013 20:32:00 +1000

DISCLAIMER: I don’t know Haskell at all. I know the code in this article is not idiomatic and I look forward to receiving constructive criticism.

In part 1 of this series I began to implement the game of life cellular automaton in C# using the functional programming style (at least as I know it). Part 1 defined the data model and implemented a function to render the state of the system to HTML.

For this part 2 of the series I port my C# solution to Haskell.

Firstly, the data structure. As with the C# version I model the game of life as a list of living cells. The definition of a cell is:

data Cell = Cell {
  x       :: Int,
  y       :: Int
} deriving (Show)

where x and y are functions that return the cartesian coordinates of the cell. It is the same as the C# version, except that this Cell type is immutable.

Having defined my Cell type I then defined a set of functions to do useful things with a Cell or a list of cells. renderLocation is a function that converts a point (xv,yv) and a boolean (indicating if the cell at that location is living) into its HTML string.

renderLocation :: Int -> Int -> Bool -> String
renderLocation xv yv on = printf 
                          "<div class=\"cell %s\" style=\"left: %dpx; top: %dpx;\">&nbsp;</div>\n" 
                          (if on then "on" else "") (20 * (xv+1)) (20 * (yv+1))

The first line is a type definition for the function. Its C# equivalent would be Func<int,int,bool,string>.

To make renderLocation useful I need a function to determine if a given location contains a living cell:

isOn :: [Cell] -> Int -> Int -> Bool
isOn cs xv yv = any cellMatch cs
  where cellMatch c = x c == xv && y c == yv

This roughly translates to the C# cs.Any(c => x(c) == xv && y(c) == yv).

The other helper functions required calculate the largest X and Y values to determine the size of the system:

largestX :: [Cell] -> Int
largestX cs = maximum $ map x cs

largestY :: [Cell] -> Int
largestY cs = maximum $ map y cs

largestX translated to C# is:

public int largestX(Cell[] cs) {
  return cs.Select(c => x(c)).Max();
}

Next I created three living cells, at coordinates (1,2), (3,5) and (14,13):

cells = [Cell 1 2, Cell 3 5, Cell 14 13]

Now I can create a render function that produces the HTML representation of a system:

render :: [Cell] -> String
render cs = concat [renderLocation x y (isOn cs x y) | x <- [0..xBound], y <- [0..yBound]]
  where xBound = largestX cs
        yBound = largestY cs

The expression [renderLocation x y (isOn cs x y) | x <- [0..xBound], y <- [0..yBound]] is a list comprehension. The part after the pipe x <- [0..xBound], y <- [0..yBound] produces the cartesian product of all possible x values ([0..xBound]) and all possible y values ([0..yBound]). For each point the renderLocation function is called. The results are collected into a list of strings which is then flattened to a single string by the concat function.

This is equivalent to the C# version:

var results = from x in Enumerable.Range(0, largestX+1)
              from y in Enumerable.Range(0, largestY+1)
              select buildCell(x, y, IsOn(cells, x, y));

Now to get the program to actually do something I define a list of cells and then use the programs main function to render my list of cells and write the result to stdout.

cells = [Cell 1 2, Cell 3 5, Cell 14 13]

main = (putStrLn . render) cells

The output of this program is:

Here is the full program:

import Text.Printf
import Data.List

data Cell = Cell {
    x       :: Int,
    y       :: Int
  } deriving (Show)

renderLocation :: Int -> Int -> Bool -> String
renderLocation xv yv on = printf 
                     "<div class=\"cell %s\" style=\"left: %dpx; top: %dpx;\">&nbsp;</div>\n" 
                     (if on then "on" else "") (20 * (xv+1)) (20 * (yv+1))

isOn :: [Cell] -> Int -> Int -> Bool
isOn cs xv yv = any cellMatch cs
  where cellMatch c = x c == xv && y c == yv

largestX :: [Cell] -> Int
largestX cs = maximum $ map x cs

largestY :: [Cell] -> Int
largestY cs = maximum $ map y cs

render :: [Cell] -> String
render cs = concat [renderLocation x y (isOn cs x y) | x <- [0..xBound], y <- [0..yBound]]
  where xBound = largestX cs
        yBound = largestY cs

--cells = [Cell 1 2, Cell 3 5, Cell 14 13]
--new Cell(1,0), new Cell(0,1), new Cell(2,1), new Cell(7,16)
cells = [Cell 1 0, Cell 0 1, Cell 2 1, Cell 7 16]

main = (putStrLn . render) cells

It is a bit longer than the C# version, but that is because I have used more explicit functions and specified more explicit types.

Functional Game of Life part 1 (C#)

Mon, 06 May 2013 20:32:00 +1000

I thought it might be interesting to solve the Conway’s Game of Life cellular automaton in a functional manner. Game of life is an interesting problem for experimentation because it is simple and well understood.

Where to Start?

I’ll start by defining a way to render the world. I want to start here because it is useful to have a nice way to visualise the state of the system, also it conveniently forces me to define my data model. I will model the system as a collection of living cells, each with X and Y coordinates (zero based). The living cells define the boundaries of the system. Dead cells are implicitly the cells within the system boundary that are not living. A system defined as:

new Cell(1,0), new Cell(0,1), new Cell(2,1), new Cell(7,16)

will produce the following output (living cells are darker):

To get to this I started with a test (using giv.n and nunit).

[TestFixture]
public class RenderingTests
{
    private List<Cell> _cells;
    private string _result;

    [Test]
    public void RenderASmallSystem()
    {
        Giv.n(ASmallSystem);
        Wh.n(ItIsRendered);
        Th.n(() => ItShouldRenderDimensions(8, 17))
            .And(() => LivingCellCountIs(_cells.Count));
    }

    private void ASmallSystem()
    {
        _cells = new List<Cell> { new Cell(1,0), new Cell(0,1), new Cell(2,1), new Cell(7,16) };
    }

    private void ItIsRendered()
    {
        _result = GameOfLife.Render(_cells);
    }

    private void ItShouldRenderDimensions(int x, int y)
    {
        Assert.AreEqual(x*y, CountOccurances(_result,"<div"));
    }

    private void LivingCellCountIs(int count)
    {
        Assert.AreEqual(count, CountOccurances(_result,"on"));
    }

    private static int CountOccurances(string source, string sub)
    {
        return source.Select((c, i) => source.Substring(i)).Count(s => s.StartsWith(sub));
    }
}

and the implementation so far:

public class GameOfLife
{
    public static string Render(List<Cell> cells)
    {
        var largestX = cells.Max(c => c.X);
        var largestY = cells.Max(c => c.Y);
        Func<int,int,bool, string> buildCell = (x,y,on) => 
            string.Format("<div class=\"cell {2}\" style=\"left: {0}px; top: {1}px;\">&nbsp;</div>", 20 * (x+1), 20 * (y+1), @on ? "on" : "");

        var results = from x in Enumerable.Range(0, largestX+1)
                      from y in Enumerable.Range(0, largestY+1)
                      select buildCell(x, y, IsOn(cells, x, y));
        return string.Join("\n", results);
    }

    private static bool IsOn(IEnumerable<Cell> cells, int x, int y)
    {
        return cells.Any(c => c.X == x && c.Y == y);
    }
}

public struct Cell
{
    public int X;
    public int Y;

    public Cell(int x, int y)
    {
        X = x;
        Y = y;
    }
}

Source

The source for this article is in a public git repo, under the tag data-and-render.

The next post in this series will look at randomly seeding the system with living cells, so that the cellular evolution can be started.

Didacto - Learn JavaScript with interactive, executable tutorials

Fri, 26 Apr 2013 20:32:00 +1000

Didacto: JavaScript is a web site that teaches JavaScript using a set of tutorials (currently covering just the basics). What makes didacto unusual and interesting is that the tutorials include questions that can be answered and automatically evaluated. If you get a question wrong didacto can tell you and also tell you why.

The executable tutorial idea was borrowed from the excellent School of Haskell. In a way it is the natural evolution of my old JavaScript Koans project.

I started with JavaScript because the server component of Didacto is written in JavaScript so that was easiest but I would like to extend the same idea to C# and other languages if it seems worthwhile.

giv.n

Tue, 16 Apr 2013 20:32:00 +1000

Today I published my first nuget package - a ridiculously simple BDD library written by Nicholas Blumhardt and I.

giv.n is a tiny DSL for grouping the steps of an executable specifications into the familiar given, when and then (or giv.n, wh.n and th.n in giv.n language). It does not have a test runner or an assertion library so you still need NUnit or similar. Here is an example specification written with giv.n and NUnit.

using Givn;

[Test]
public WhenIBakeACake() {
    Giv.n(IHaveFlour)
        .And(IHaveEggs);
    Wh.n(IBakeCake);
    Th.n(IHaveDeliciousSnack);
}

private void IHaveFlour() {
    _ingredients.Add(new Flour());
}

private void IHaveEggs() {
    _ingredients.Add(new Eggs());
}

private void IBakeCake() {
    _cake = _oven.Bake(_ingredients);
}

private void IHaveDeliciousSnack() {
    Assert.IsTrue(_cake.IsDelicious());
}

Can we stop saying 'unit tests'?

Sun, 07 Apr 2013 20:32:00 +1000

What is a unit test?

Here is wikipedia’s definition of unit testing:

In computer programming, unit testing is a method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine if they are fit for use.

Looking carefully we notice that, “units of source code, sets of one or more computer program modules” is actually everything. Therefore, the above definition can be condensed, without changing its meaning, to:

In computer programming, unit testing is a method by which any source code is tested to determine if it is fit for use.

That is, unit testing is testing. Great.

Roy Osherove’s definition is equally vague and pointless:

A unit test is a piece of a code (usually a method) that invokes another piece of code and checks the correctness of some assumptions after- ward. If the assumptions turn out to be wrong, the unit test has failed. A “unit” is a method or function.

Prior to the last sentence this was a useful definition of an automated test, but the last sentence, “a unit is a method or function” is seemingly unrelated to the rest of the definition. Defining a “unit” does not help unless we choose to infer that a “unit test” is a test that tests a “unit”. If a “unit test” is a test that tests a “unit”, and a “unit is a method or function” then if we factor our programs into cooperating functions, as suggested by functional decomposition, the composed method pattern and the SOLID principles then it is impossible to write a “unit test” according to Osherove’s definition, since any function that calls another function is by definition not a “unit”. Testing a function that depends upon other functions is obviously testing all of the functions involved.

Despite a lot of pretense the fact is that we do not have an accepted definition of a unit test.

What is the harm?

I have always treated “unit test” as the antonyn of “integration test”. This has been a useful dichotomy. Unit tests are fast, integration tests are slow therefore it is useful to separate them.

The ambiguity of the term only becomes a problem when people malign tests with statements such as, “that’s not a unit test”. That may well be, or it may not be, it’s hard to say since there is no consistent, useful definition of a unit test.

“Unit tests are good” is an empty bit of dogma that has outlived is usefullnes, finally becoming not just useless but harmful.

Listagram

Thu, 28 Mar 2013 20:32:00 +1000

UPDATE: I have made the listagram source publically available for anyone who is interested.

If you have a billion dollar company, or very specific needs, then it may make sense to develop native mobile applications for multiple platforms. For everyone else it is hard to argue against the practicality of portable html-based applications. Such has been my uneducated opinion for some time. To prove that multi-platform web apps are a viable option I decided to build one.

The Story So Far

I travel home from work on the train. While making my way home it is not unusual for my wife to send me a list of groceries for me to pickup. I found that an SMS as a shopping list does not provide a good user experience. It is too hard to see the items and to keep track of which items remain.

Listagram

The app that I built is called Listagram. It provides two features:

Send a list of items, to yourself or to someone else
Receive a list, view the items and complete the items as required

Technical

Client

I built the app entirely as a standalone web application. It was nice to be able to develop without resorting to testing on a mobile device or emulator. I used chrome and its wonderful developer tools.

The client-side and server-side code is all written as commonjs modules and packaged for client-side use with browserify. The language I used is CoffeeScript, which browserify takes care of compiling.

For a client-side UI framework I used stativus. Stativus is a UI statechart very good for treating UI changes as state transition. This is helpful for preventing memory leaks and provided an interesting contract to backbone - the framework that I have most experience with.

The UI templates are rendered using underscore’s template function. To precompile the UI templates and merge them into a single file I wrote a node.js module called underscorec. Underscorec can read a directory structure of templates and compile it.

Client-side storage is provided by localStorage.

Usage is:

underscorec views/ output.js

where views/ is a directory structure containing templates. The templates can then be used be calling a function with a name matching the name of the compiled template:

templates[home/index]({name: 'Peter', age: 27})

Server

The server is written as a node.js application. The web framework is express and the data is stored in couchdb.

Server Deployment

The services that drive the app are deployed to a vps via an ssh script and git. I use forever to start the node.js app as a service.

Client Deployment

The client is packaged as a multi-platform mobile application using phonegap. I used the online service Phonegap Build to create my app packages for each mobile platform.

Final Thoughts

The experience turned out to be more work than I anticipated but it did confirm my suspicion that the cross-platform mobile solution is viable for most requirements.

Things considered harmful (in C#)

Tue, 26 Mar 2013 20:32:00 +1000

Spelunking through a program recently I started to think about my coding style, particularly in C#. It occurred to me that my style is most easily defined in terms of the things that I try to avoid. Here is a quick list of things to avoid, in C#, where practical.

Most of the justification for these exclusions is in my presentation Why Functional Programming Matters

Loops

I don’t like explicit loops, for and foreach. Use linq when querying. Explicit loops are better when mutating state but let’s avoid that too. Recursion is also an option.

Conditional (if)

Null checking type conditionals can be replaced with null coallescion in a variant of the null object pattern:

(callback ?? ()=> {})();

Polymorphism allows types to replace explicit conditionals. Tell don’t ask can also eliminate conditionals. Finally, the conditional operator (?:) is preferable to the if statement because it is an expression.

Switch statement

Switch is a particularly nasty special case of conditionals.

Avoid by using polymorphism or convert to a dictionary (declarative instead of imperative).

Statements

C# make these hard to avoid because many C# language feature are not expressions. The solution is to use language features that are expression oriented, like linq, factor functions properly and write pure functions. Variable declarations are ok if they make code more readable.

Mutable state

Don’t have it if you don’t need it. Return copies of things.

State mutations

As above.

async

The TPL is great when you need to do something asynchronously, but it is viral in the sense that all callers become async too. Don’t use async/TPL without a good reason and do it as shallow in the callstack as possible. Separate async / IO code from pure code

null

Null is Tony Hoare‘s fault.

I call it my billion-dollar mistake. It was the invention of the null reference in 1965. At that time, I was designing the first comprehensive type system for references in an object oriented language (ALGOL W). My goal was to ensure that all use of references should be absolutely safe, with checking performed automatically by the compiler. But I couldn’t resist the temptation to put in a null reference, simply because it was so easy to implement. This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years.

Eric Lippert seems to agree:

even this is essentially an accident of history; it just so happened that when C# was first implemented it had always-nullable reference types, non-nullable value types, and no generic types at all. In a counterfactual world where the CLR had generic types from the get-go, it seems plausible that Nullable could have been implemented to work on any type, and reference types would then be non-nullable by default. We could have a type system where Nullable was the only legal way to represent “a string that can be null”. Keep this in mind the next time you design a new type system!

Give references default values and design types that don’t contain null members. No more null reference exceptions.

ref and out params

It is often (always?) possible to replace ref and out params with other options, such as pre-testing values.

Haskell influences in CoffeeScript

Wed, 27 Feb 2013 20:32:00 +1000

While learning a bit of Haskell I was struck by the syntactical similarites to CoffeeScript. Since Haskell predates CoffeeScript by twenty years (1990/2010) it seems that it is Haskell that has had an influence on CoffeeScript. What follows is a list of the similarities that I have observed.

Binding Variables

In Haskell we can bind a local variable to a scope using the let... in syntax:

circumference r = let pi = 3.14159
                  in  pi * 2 * r

CoffeeScript supports the same thing via the do keyword. One interesting thing about do is that it allows a way to define a variable that is (sort of) not scoped to a function.

circumference = (r) ->
  do (pi = 3.14159) ->
    pi * 2 * r

You can read about some interesting uses of do in Reginald Braithwaite’s CoffeeScript Ristretto.

Significant Whitespace

As you can see in the examples above Haskell and CoffeeScript both use significant whitespace in a similar way.

Expressions > Statements

Both Haskell and CoffeeScript encourage us to favour expressions over statements. CoffeeScript supports statements that aren’t expressions but does everything possible to let you avoid them.

evenness = (i) ->
  if i % 2 is 0
    'even'
  else
    'odd'

Combining this with the do notation we can do

evenness = (i) ->
  do (is_even = (n) -> n % 2 is 0) ->
    if is_even i
      'even'
    else
      'odd'

Comprehensions

Haskell and CoffeeScript have similar syntax for list comprehension. The following Haskell function selects the even elements of a list (x mod 2 == 0) and maps them through a function that multiplies them by two (x*2).

double_odds xs = [x*2 | x <- xs, x `mod` 2 == 0]

The equivalent CoffeeScript is:

double_odds = (xs) ->
  x*2 for x in xs when x % 2 is 0

Function Call Syntax

Both languages use a parenthesis free syntax for applying a function.

Haskell:

add 2 3

CoffeeScript:

add 2, 3

Ranges

For basic incrementing or decrementing integers Haskell and CoffeeScript have the same syntax.

[1..10]

[10..1]

Literate Mode

Haskell and CoffeeScript both support a ‘literate’ mode that emphasizes comments over code.

In CoffeeScript’s literate mode markdown text is interpreted as a comment. Indented text is executed as CoffeeScript code.

Heading
=======

What I need is a function that doubles the even numbers in a list. Here's one!

  double_odds = (xs) ->
    x*2 for x in xs when x % 2 is 0

Haskell’s literate mode uses a > to indicate lines of code.

What I need is a function that doubles the even numbers in a list. Here's one!

> double_odds xs = [x*2 | x <- xs, x `mod` 2 == 0]

That’s all that I can think of, but I’m sure there is more. CoffeeScript is often described as a blend of Ruby and Python that compiles to JavaScript. I think this underplays the influence of Haskell in CoffeeScript’s design.

dbc (node.js npm module)

Sun, 24 Feb 2013 15:00:00 +1000

dbc

dbc is my third npm module. I am experimenting with the node.js (unix?) philosophy of small, composable modules. dbc provides some basic tools for design-by-contract, something that I believe is essential when working in a loosely typed language like JavaScript.

To install:

    npm install dbc

to run the tests:

    npm tests/ -R spec

which will produce this delightful output:

Reversing a string

Wed, 20 Feb 2013 20:32:00 +1000

While there are many ways to skin a cat I can only think of the one involving a knife. However, I can think of many ways to reverse a string.

Recursive function

My preferred solution is a simple recursive function (gistfile1.cs). The first character of the reversed string is the last character of the input. The rest of the string is the rest of the input reversed. The only other thing required is to detect when the entire string has been processed and terminate the recursion.

Map from a collection of indexes

Create a collection of indexes and map each one to its inverse location in the string (gistfile2.cs).

The CoffeeScript version (gistfile3.coffee) has nicer syntax. It also demonstrates the affect of reversing the sequence of indexes.

Handlebars Byte Order Mark!

Wed, 01 Aug 2012 15:00:00 +1000

Byte order mark (BOM) is a unicode character that signals the byte order of a UTF text file. If displayed in a text editor it most often appears as ï»¿.

Sometimes, when using Handlebars.js to produce markup you will see extraneous whitespace inserted into the DOM immediately prior to your rendered template. In the chrome developer tool, this extra whitespace appears as " ". In firebug it appears as EF BB BF.

This problem seems to occur when using the handlebars.js precompilation feature to precompile templates on the server and combine them into a single script. The precompiler doesn’t remove the BOM marks when compiling the templates so they end up in the DOM, messing with your layout.

The solution is to make sure that the template files do not include BOMs. You can use Notepad++, Sublime Text or any good text editor to save a file as UTF-8 without BOM.

Currying

Sat, 18 Feb 2012 15:00:00 +1000

This is a function that does some currying:

add = (a,b) ->
  if not b?
    return (c) ->
      c + a
  a + b

JavaScript provides the capability to reflect on the number of arguments:

add.length

and to determine how many arguments were provided:

add = (a,b) ->
  if arguments.length < add.length
    return (c) ->
      c + a
  a + b

so it seems like it should be possible to write a function that magically returns a function that requires the right number of arguments. So I could have a function:

f = (a,b,c,d,e,f) ->
  ..

if invoked with:

f(1,2)

it should return:

(c,d,e,f) ->

anyone know how to do that?

Update

Lots of good comments on the original gist

Quicksort in Haskell and CoffeeScript

Fri, 17 Feb 2012 15:00:00 +1000

The definition of Quicksort, from Wikipedia is:

The steps are:

Pick an element, called a pivot, from the list.

Reorder the list so that all elements with values less than the pivot come before the pivot, while all elements with values greater than the pivot come after it (equal values can go either way). After this partitioning, the pivot is in its final position. This is called the partition operation.

Recursively sort the sub-list of lesser elements and the sub-list of greater elements.

The base case of the recursion are lists of size zero or one, which never need to be sorted.

Quicksort in Haskell, from learn you a haskell, is:

    quicksort :: (Ord a) => [a] -> [a]  
    quicksort [] = []  
    quicksort (x:xs) =   
        let smallerSorted = quicksort [a | a <- xs, a <= x]  
            biggerSorted = quicksort [a | a <- xs, a > x]  
        in  smallerSorted ++ [x] ++ biggerSorted

A basic quicksort in CoffeeScript is:

quicksort = ([head,tail...]) ->
  return [] if typeof head is 'undefined'
  smaller_sorted = quicksort (e for e in tail when e <= head)
  larger_sorted = quicksort (e for e in tail when e > head)
  smaller_sorted.concat([head]).concat(larger_sorted)

The CoffeeScript version is reasonable, and shares some similarity with the haskell implementation. The part I don’t like is having to check typeof head is 'undefined' for the base case. This is because the destructuring assignment of the function argument leaves me with no reference to the full array so the only way to tell if it is empty is to check if the head is undefined. There is probably a better way to do this. Please comment if you know what that is.

Backbone.js and Client-side Web Applications

Thu, 16 Feb 2012 15:00:00 +1000

Backbone.js and Client-side Web Applications

In the beginning the internet was a set of documents. Then someone decided to add hyperlinks to create a web of connections between documents. Soon static documents were not enough so we began to dynamically create documents on the server. The client-side user experience was still flat, so scripting was introduced to add dynamic behavior to pages. Finally, the most demanding and advanced web applications moved their user interface entirely to the client-side to provide the ultimate user experience. This apocryphal tale demonstrates the long, unbroken trend towards richer user interfaces on the web.

The new generation of frameworks, libraries and training material make developing client-side web applications fun and approachable.

Warning: the rest of this article assumes some knowledge of JavaScript. If you would like to learn more about JavaScript I have recorded a comprehensive JavaScript introduction screencast for pluralsight.

Backbone.js

Backbone.js is a delightfully documented set of tools that help us to simplify and structure client-side web applications.

Backbone provides the building blocks for assembling client-side applications. Models are used to store an application’s data and to handle sending data to and from the server. Views map to user interface elements. They render markup, usually based on models, and connect event handlers to document events. Finally, backbone’s router is used to trigger and handle client-side routing.

Putting these basic elements together we can assemble complex applications in an enjoyable and productive way. The rest of this article will investigate a simple application that makes use of the model and view components of Backbone.js.

For a complete introduction to Backbone.js, including guidance on how to structure and test backbone.js applications, try my Backbone Fundamentals screencast course.

The Basics

Backbone.js does many things - one of the most immediately useful is the ability to bind one data object to multiple user interface components. When the underlying data model is changed each of the views is automatically updated to reflect those changes.

Imagine a user interface that displays a color as its hex value, and as a swatch. The color can be represented as a backbone model. The hex display and the swatch are each a backbone view.

Try the completed application but ignore the code for now.

The single color model can be shared between the two views. That way there is a single authoritative data source, and no possibility of synchronization problems.

A first Backbone.js Model

The data object (backbone model) can be created like this:

var color = new Backbone.Model({
  hex: '#ff0000'
});

For a starting point I have set the model’s hex value to #ff0000 which is the hex code for red.

Backbone.js Views

The hex view, which displays the hex value of the color in a text box is:

var HexView = Backbone.View.extend({
    render: function () {
        // create a textbox containing the model's hex attribute
        var input = this.make('input', { type: 'text', value: this.model.get('hex') });
        this.$el.append(input);
    }    
});

render() is the function that backbone uses to update a view’s markup. this.model is the view’s reference to the shared color model.

The swatch view, which displays a color swatch representing the color model, is implemented as:

var SwatchView = Backbone.View.extend({
    render: function () {
        this.$el.html(this.make('div', { class: "swatch", style: "background-color: " + this.model.get('hex') }));
    }
});

see the full source

To illustrate the real power of Backbone.js we need to change the model data, and trigger an update of the views. To do this, we add a button to the hex view and handle its click event.

var HexView = Backbone.View.extend({
    // Every view type can have an events property that declares event bindings.
    events: {
        'click input': function () {
            this.model.set({hex: this.$('input').val()});
        }
    },

    render: function () {
        var input = this.make('input', {type: 'text', value: this.model.get('hex')});
        this.$el.html(input);
        this.$el.append(this.make('input', {type: 'button', value: 'update'}));
    }    
});

When the button is clicked the view updates the model with the color currently in the text box. To get this change to show up in the swatch view we need to bind the model’s change event to the swatch view’s render() function.

var SwatchView = Backbone.View.extend({
    initialize: function () {
        // handle the model's `change` event
        this.model.on('change', function () {
            this.render();
        }, this);
    },

    render: function () {
        this.$el.html(this.make('div', { class:"swatch", style:"background-color: " + this.model.get('hex') }));        
    }
});

initialize() is a special view function that is run when an instance of the view is created (like a constructor).

Connecting the dots

We have a model object (color) and two view types (HexView and SwatchView). To initialize our application we just need to create instances of the views and call the render() methods.

var hex = new HexView({model: color, el: '#hex'});
hex.render();

var swatch = new SwatchView({model: color, el: '#swatch'});
swatch.render();

As you can see I can now enter a different hex code (say #0000ff), click the update button and see the swatch change to blue.

The completed user interface

That concludes our exploration of the basic features of Backbone.js. If you are interesting in learning more about Backbone.js and building client-side web applications try For a complete introduction to Backbone.js, including guidance on how to structure and test backbone.js applications, try my Backbone Fundamentals screencast course.

Fibonacci

Wed, 15 Feb 2012 20:32:00 +1000

Here is the fibonacci sequence calculated in haskell (1 to 100):

fibo :: Int -> Int
fibo 0 = 0
fibo 1 = 1
fibo n = (fibo (n-1)) + (fibo (n-2))

prn = map fibo [1..100]

prn

and here it is in CoffeeScript

fibo = (n) ->
  if n is 0 or n is 1
    return n

  fibo(n-1) + fibo(n-2)

for i in [1..100]
    console.log fibo(i)

Both of these take forever to execute. Curiously the CoffeeScript version is an order of magnitude faster.

Sharing client and server validation

Tue, 14 Feb 2012 20:32:00 +1000

Client-side web applications (SPAs whatever) must perform client-side validation to achieve the required responsiveness. But the client is not a secure environment, so client-side validation is a convenience, not to be trusted. Validation has to be performed server-side to ensure security and data correctness.

Traditional Solutions to the Client / Server Validation Problem

Shared Metadata

Asp.net mvc exports validation metadata from server-side models and embeds it into form markup. This process is distributed throughout mvc’s form helpers. Each time a form element is generated the framework adds data- attributes to that element that describe that element’s validation.

On the client-side a script runs that reads the validation metadata from the data- validation attributes and translates that metadata into a format suited to a client-side validation framework (asp.net mvc uses jquery.validation).

By sharing metadata between the server and the client it is possible to have a validation framework on each side automatically working with the same metadata.

Doing It Twice

The solution that I am currently using is to implement completely separate validation - once on the client and once on the server. It works, but it is frustrating and boring. There is also a maintenance burden in keeping the validations synchronized.

Moving Client-Side Validation to the Server

The idea I have been considering is to move the client-side validation to the server. Client-side validation tests javascript objects against validation metadata and builds a set of results. Why can’t we do that on the server?

My idea is to write a custom model binder that binds json data posted to the server. The post would have to include metadata that defines the schema of the data, so that we can lookup the required validation metadata. A JavaScript interpreter can then be used to apply the validation.

The strength of this idea is that it uses the same validation engine and validation metadata on the client and the server. The validation only needs to be tested once and cannot possibly get out of sync.

Testing backbone models

Mon, 13 Feb 2012 20:32:00 +1000

This is this first in a planned series discussing the specifics of testing applications built with Backbone.js.

Of all the components of a Backbone.js application Models are the easiest to test because they don’t have a lot of dependencies. The high level testing approach I use is:

Initialize a model with a specific state
Test that the model’s behavior matches expectations
Goto 1.

Testing a Rectangle Model

Imagine a Backbone model that represents a rectangle, with length and width properties. It has the following specification:

Rectangle Specification

Rectangle

with length 7 and width 4

   should have an area of 28  
   should have a perimeter of 22

Jasmine Specification

We can express the rectangle specification using the syntax of the Jasmine testing tool:

describe('Rectangle', function () {
  describe('with length 7 and width 4', function () {
    it('should have an area of 28', function () { 
      // assert expectations here
    });
    it('should have a perimeter of 22', function () { 
      // assert expectations here
    });
  });
});

and add the expectations to get a failing test:

describe('Rectangle', function() {
    var rectangle;

    beforeEach(function() {
        rectangle = new app.Rectangle();
    });

    describe('with length 7 and width 4', function() {
        beforeEach(function() {
            rectangle.set({
                length: 7,
                width: 4
            });
        });

        it('should have an area of 28', function() {
            expect(rectangle.area()).toBe(28);
        });
        it('should have a perimeter of 22', function() {
            expect(rectangle.perimeter()).toBe(22);
        });
    });
});

The Rectangle Model

The final step is to implement the Rectangle model to make the tests pass:

var app = {};

(function (shapes) {
    shapes.Rectangle = Backbone.Model.extend({
        area: function () {
            return this.get('length') * this.get('width');
        },
        perimeter: function () {
            return 2*this.get('length') + 2*this.get('width');
        }
    });
})(app);

Et Voilà!

Testing Backbone models is trivial. Next in this series I will describe how I test Backbone views - a much more challenging proposition.

Backbone app framework

Sun, 12 Feb 2012 20:32:00 +1000

Backbone is a library of tools that simplify the design and implementation of client-side web applications. It is explicitly not a framwework. Backbone does not provide guidance about how to assemble an application. This post will be an initial attempt at filling that gap. It assumes an intermediate level of Backbone.js knowledge. If you have never used Backbone.js try Backbone.js Basics or the Backbone.js homepage.

The Application Root

I like to provide a root object/namespace for my application. Let’s call it app.

var app = {};

Register Components on the Application Object

app.models =  {
    Post: Backbone.Model.extend({}),
    Comment: Backbone.Model.extend({})
};

app.views = {
    PostView: Backbone.View.extend({}),
    ComentView: Backbone.View.extend({})
};

De-couple Components by using an Event Aggregator

An application event aggregator allows components to communicate indirectly. A simple event aggregator can be created by mixing the Backbone.Events modules into an application level object:

app.eventAggregator = _.extend({}, Backbone.Events);

Actions

When a significant event occurs within your application raise an event. For example, the handler for the ‘add post’ button:

_onAddPost: functin () {
    app.eventAggregator.trigger('post:create', this.model);
}

app.actions = {
    createPost: Action.extend({
        event: 'post:create',
        handler: function (model) {
            var newPost = new app.Views.CreatePost({model: model});
            $('#content').html(newPost.render().el);
        }
    })
};

Routing

Use a route only when you want to change the url and handle the event. Otherwise just raise an event and register an action.

When you do use a route, have it trigger an action.

app.routers = {
    postRouter: Backbone.Router.extend({
        'view/:postname': function (postname) {
            app.eventAggregator.trigger('post:view', postname);
        }
    })
};

One Line Bootstrapping

Reduce your application bootstrapping to a single line.

app.start();

This makes it practical to test contexts that require the application to be in a usable state.

describe('adding a new post', function () {
    describe('initiate new post process', function () {
        before(function() {
            app.start();
            var testModel = new app.models.post();
            app.eventAggregator.trigger('post:create', testModel);
        });
    });
});

The start() function is the framework part. It registers the routers and actions and does any other required bootstrapping.

My open source projects

Sat, 11 Feb 2012 20:32:00 +1000

I have a lot of projects on github now, some active and some archival. This post is a short summary of the projects that I actively maintain.

watchn

https://github.com/liammclennan/watchn

Watchn is a node-based file system watcher. It can be used to trigger actions when specific file system changes are detected. I use it to drive continuous testing and for automating build tasks.

git-deploy

https://github.com/liammclennan/git-deploy

git-deploy is a simple git based deployment tool. Push changes to a deployment branch and click a button to deploy. git-deploy can also run pre and post deployment scripts.

snap

https://github.com/liammclennan/snap

snap is an automatic browser reloader for live testing web pages. It uses watchn to detect file system changes, which then cause the browser to reload. Useful for web development and for browser based testing. Similar to livereload except its free and works on windows.

gistblog

https://github.com/liammclennan/gistblog

gistblog is my gist-based blog engine that powers my blog.

backbone-server

https://github.com/liammclennan/backbone-server

I use it for backbone but backbone-server is actually a generic non-persistent RESTful http server. Start the server and begin storing and retrieving resources. Data is stored in memory so it is lost when the server stops. Useful for testing and teaching backbone. See the demo.

JavaScript Koans

https://github.com/liammclennan/JavaScript-Koans

JavaScript koans is an interactive learning environment that uses failing tests to interactively teach JavaScript.

Connascence of methods

Fri, 10 Feb 2012 20:32:00 +1000

From wikipedia:

Two software components are connascent if a change in one would require the other to be modified in order to maintain the overall correctness of the system.

Connascence is a useful tool for discussing different types of coupling that occur within software.

Again from wikipedia, “connascence of name is when multiple components must agree on the name of an entity”. Connascence of name one is of the list harmful and most unavoidable types of connascence because it is simple to reason about and change. While connascence of name is still connascence, and therefore bad, it is among the most amenable to refactoring. If the name of something changes then everything that refers to that thing by name must also change. This is a simple change that can be mostly statically determined and applied by tools.

Connascence of position is present when changing the order of something necessitates a change elsewhere. An example is the order of function arguments. If the order of a function’s arguments is changed then every client of that function must also change. Connascence of position is worse than connascence of name (usually) because it is less explicit and therefore more difficult to reason about and successfully change.

From connascence of position to connascence of name

Some modern languages have slowly been migrating to connascence of name from connascence of position. As well as converting to a less harmful form of coupling this has the additional benefit of documenting the client-side of a method call.

Ruby

Ruby supports positional method arguments.

def full_name(first_name, middle_name, last_name)
    "#{first_name} #{middle_name} #{last_name}"
end

p full_name 'Eric', 'Arthur', 'Blair'
=> "Eric Arthur Blair"

but many ruby apis prefer the use of a single hash containing the method arguments:

def full_name(name)
    "#{name[:first]} #{name[:middle]} #{name[:last]}"
end

p full_name :first=>'Eric', :middle=>'Arthur', :last=>'Blair'
=> "Eric Arthur Blair"

this has the benefit of converting positional parameters to named parameters and therefore connascence of position to connascence of name.

JavaScript

JavaScript has adopted the same convention. While positional function arguments work:

function fullName(first, middle, last) {
    return first + ' ' + middle + ' ' + last;
}

fullName('Eric', 'Arthur', 'Blair');
=> "Eric Arthur Blair"

most apis favour named arguments by passing a single object containing the method arguments:

function fullName(name) {
    return name.first + ' ' + name.middle + ' ' + name.last;
}

fullName({ first: 'Eric', middle: 'Arthur', last: 'Blair'});
=> "Eric Arthur Blair"

C sharp

C# now supports named arguments, but there usage is not popular. C# and .net has a dependable history of adopting, after a delay, the technology and techniques that have become dominant on other platforms. Therefore, I predict that usage of named method arguments in C# will increase over the next few years. Interestingly, C# is the only one of the three languages discussed that has a specific language feature for named arguments and it is also the only one that does not require a change to the definition of the method to take advantage of named arguments. The decision to use or not use named arguments is entirely at the client’s discretion.

string FullName(string first, string middle, string last) {
    return String.Format("{0} {1} {2}", first, middle, last);
}

FullName(first: "Eric", middle: "Arthur", last: "Blair");
=> "Eric Arthur Blair"

Hosted backbone server

Thu, 09 Feb 2012 20:32:00 +1000

This morning I published my zero-config backbone.js server.

It is an in-memory, restful server that implements the protocol required by Backbone.js, which is basically the same as couchdb except that ‘create’ returns the new id. As such it can be used for any purpose requiring temporary resource persistence. The service provides no durability at all. If it restarts all of the data is lost.

I’m now running a public instance at http://withouttheloop.com:3002. It works on my machine, don’t blame me if it crashes your space shuttle. No warranty of any kind. Obviously.

To use with backbone set your model or collection’s url property to http://withouttheloop.com:3002/[yourresourcetype]. Have a look at this jsfiddle. If you run the example and look in Chrome’s developer tools networking tab you will see:

Request

POST /books HTTP/1.1
Host: withouttheloop.com:3002

Response

HTTP/1.1 200 OK
X-Powered-By: Express

{"title":"Midnight in the garden of good and evil","author":"John Berendt"}

The model is persisted to the hosted backbone server.

The set of supported operations (which match those required by backbone.js) is:

Create

$ curl -X POST http://withouttheloop.com:3002/people -H "Content-Type: application/json" -d '{"name": "Bob", "age":"28"}'

=> {id: 1}

Update

Bob had a birthday

$ curl -X PUT http://withouttheloop.com:3002/people/1 -H "Content-Type: application/json" -d '{"name": "Bob", "age":"29", "id": 1}'

=> OK

Read the Collection

$ curl http://withouttheloop.com:3002/people -H "Content-Type: application/json"

=> [{"name":"Jane","age":"32","id":0},{"name":"Bob","age":"29", "id":1}]

Read a Resource

$ curl http://withouttheloop.com:3002/people/1 -H "Content-Type: application/json"

=> {"name":"Bob","age":"29", "id":1}

Destroy

$ curl -X DELETE http://withouttheloop.com:3002/people/1 -H "Content-Type: application/json"

=> OK

Backbone server

Wed, 08 Feb 2012 20:32:00 +1000

Backbone-Server is a zero configuration backbone.js server. For teaching, experimentation and debugging it provides an instant server-side for backbone.js apps. Once installed it is started like so:

node rasterver.js

Then from the client you can work with models:

var Person = Backbone.Model.extend({}),
People = Backbone.Collection.extend({
    model: Person,
    url: 'http://localhost:3000/People'
}),
people = new People();
people.add({name: 'John', age: 25});
people.at(0).save();

That is enough to get the model saved to the server. Fetch, update and destroy all work properly.

The server stores the collection data in memory, so when the server stops the data is lost. This is not a good choice for production.

Team pomodoro

Tue, 07 Feb 2012 20:32:00 +1000

Many people like the pomodoro technique because it helps them focus intently on tasks. It is also helpful for preventing interruptions. If your team, and extended team, know that you are in the middle of a pomodoro then they know to avoid interrupting you. But how do you communicate that you have a pomodoro in progress?

Team Pomodoro combines a tradtional pomodoro timer and a dashboard showing the status of each team member’s pomodoros. A person can see at a glance who is currently available and when each person will be available.

Additional Features

Add features for messaging team members. Choice of
- interrupt
- queue until end of pomodoro

Backbone sandbox

Mon, 06 Feb 2012 20:32:00 +1000

Backbone.js Sandbox

If you are working with Backbone.js it is handy to have a sandbox environment for testing simple things.

This jsfiddle has jQuery, backbone and underscore already loaded. Type your code in the JavaScript text area, as well as markup and css in their respective text areas as required, and click the run button. I use it all the time.

Extending Underscore.js - findOr

Sun, 05 Feb 2012 20:32:00 +1000

As good as underscore.js is, sometimes you want to add functions of your own, and you can, using the mixin() function.

_.mixin({
  capitalize : function(string) {
    return string.charAt(0).toUpperCase() + string.substring(1).toLowerCase();
  }
});
_("fabio").capitalize();
=> "Fabio"

findOr

Underscore has a find method that returns the first element of a collection that matches a supplied predicate. For example, it can be used to find the first even number in a set:

var isEven = function (num) {
  return num % 2 === 0;
}

_([1,2,3,4,5]).find(isEven);
=> 2

findOr is a function I mixed-in to underscore so that I could have find return a default value if no element matches the predicate.

_([1,2,3]).findOr(isEven, 'no even numbers');
=> 2

_([1,3]).findOr(isEven, 'no even numbers');
=> 'no even numbers'

The implementation, using underscore’s mixin function, is:

_.mixin({
  'findOr': function(list, iterator, defaultValue, context) {
    return this.find(list, iterator, context) || defaultValue;
  }
});

Try the executable demo on jsfiddle.

snap

Sat, 04 Feb 2012 20:32:00 +1000

This file is a fantasy readme for an tool that doesn’t exist yet. I wrote it in an attempt to apply readme driven development. The idea is to force yourself to define a vision for your product/tool/whatever before starting to build anything.

Snap

snap is a tool that automatically reloads your browser based on filesystem events. For example, it can be configured to automatically reload the webpage you are working on when any of your html files are modified.

snap({
  '.': /.+\.html/
});

Usage

Create a snap script (as above) and start the server:

node snap mysnapscript.js

Add the following script to the page that you want auto-reloaded:

<script src="http://localhost:9876/snap.js"></script>

Object-object mapping

Fri, 03 Feb 2012 20:32:00 +1000

Object-object Mapping - Why I don’t use Automapper

At the boundaries of an application it is often nice to map objects from one form to another. The most common example is mapping domain entities to view models, or mapping to dtos for network transfer.

Automapper is a library for removing the duplication from object-object mapping code by mapping by convention. When the conventions don’t map the way you want you can explicitly map properties.

After using AutoMapper on a couple of projects I have stopped using it. The problem is that the convention based mappings introduce annoying bugs when property names change or the object graph changes. When a change breaks the convention it results in a runtime exception. Very annoying.

What I Do Instead

My observations from using automapper are:

If you use the convention-based mapping and a property is later renamed that becomes a runtime error and a common source of annoying bugs.
If you don’t use convention-based mapping (ie you explicitly map each property) then you are just using automapper to do your projection, which is unnecessary complexity.

For larger projects I like to implement explicit type mappers and register them in an IoC container. The advantages are: mappers are explicit. Easy to identify and test mappings the mappers can have their own dependencies. The consumers of the mappers don’t have to depend on the mappers dependencies.

=======================================================================================================================================

public interface IMapper { T2 Map(T1 input); }

public class MapAppleToOrange : IMapper { public MapAppleToOrange(/ opportunity for constructor injection here. Useful for mapping VMs -> persisted models / ISomeDependency dependency) {}

Orange Map(Apple input) { // map things here // make use of dependency } }

// usage

public class HomeController { // let the IoC container resolve the required mappers // note that HomeController does not depend on ISomeDependency public HomeController(IMapper mapper) {}

public ActionResult Index(Apple apple) 
{
   // do some mapping
   return View(mapper.Map(apple));
}

}