Rob Ashton's Blog(s)

Github Live

Mon, 07 May 2012 12:28:00 +0000

Visualize git 'pushes' live as they happen across Github.

I visited @cranialstrain in England this weekend, and he suggested we hack something together around the Github APIs in response to the Github data challenge .

Looking at the Event stream, I thought it would be interesting to visualise what was going on in live, in a web browser, and given all the toys I've written over the past year or so in NodeJS, it was fairly clear that a simple web server processing the events and broadcasting them to clients wouldn't take a lot of work to complete.

So, we ended up with Github Live, which looks something like this once you've left it running for five minutes (during the morning, so it's a bit quiet)

The server side

The server is using Node, which is operating as a basic static file server, and as a point of call for incoming sockets with socket.io.

It starts off every 10 seconds polling the Events API, storing the most recent timestamp so to avoid publishing duplicates (the events API doesn't have a "last retrieved id").

It attempts to throttle requests to the events API to avoid the amount of duplicate events being retrieved from the API (if it finds an overlap, it increases the time until next request by a second, and if it doesn't, it decreases by a second).

In hindsight, the hideous inline callbacks would be best replaced with a stream that did all this work, and just published events transparently to the consuming code.

The next job, once this has taken place is that a request is made to Github for each pull, asking for information about the repository (for the language), so the events being streamed to the clients can be augmented with this information.

This is another thing that should be dealt with by a stream rather than inline callback soup.

Oh well, it's only 200 lines of throwaway code, perhaps something to tidy up on a rainy day.

The client side

Because it's quick and easy, we're using HTML and CSS3 to do display and transformations.

As the events stream in from the server, some HTML is created for the event and it is put in an appropriate bucket (for the language being used).

The outside container has a CSS transition applied to it, and the transform 'scale' is set to fit all buckets into the same window periodically.

From this I have ascertained that

I dislike vendor prefixes
these are not as fast as I'd like
Webkit has some unusual glitches if you're not careful

I guess with some more work, these things could be worked around, there are some definite performance improvements that could be made client side here.

I'd quite like to give an SVG implementation a go, and see about the performance of that. Another project for a rainy weekend.

The code

The code in all its (raw) form can be found at https://github.com/robashton/githubfall

I'd be interested to see any obvious improvements made and pull requested in.

Anatomy of a 48 hour HTML5-JS Game

Mon, 23 Apr 2012 12:34:00 +0000

I participated in my second Ludum Dare 48 hour game-building compo this weekend, and this was my fourth attempt at building a game from scratch in 48 hours in HTML5/JS.

Each time I have approached the problem in a slightly different way, but I'm beginning to get the process sussed (with a few caveats) so let's break down my latest attempt (Web | Source) and see how it is put together.

Libraries + Frameworks

I've tried a couple of approaches for this so far, one is to start with a set of 'base code' (written in advance either extracted from another project, or written especially for the 48 hour compo).

I've ended up settling for a halfway house, I have some libraries that I really like from JS/HTML development in general, and some libraries/components that I keep around for convenience.

Third party

Underscore - this is a library containing a load of polyfills for doing common operations in JS (iteration, binding, etc) - my favourite method is probably _.extend, which I'll cover in a bit.

jQuery - we all know what this is - to be honest, I use it for DOM selection and the most basic of manipulation only - as such it's a bit heavy-weight for what I need. Still - it's familiar to me and allows me to get started - can't say wrong with that.

Hand-rolled components

Swallow - Packaging up a directory as a JSON file and downloading it all-as-one might not be the most elegant/modern way of dealing with multiple assets but it at least allows an easy deterministic way of dealing with dependencies, and means that I forego a lot of the issues with playing Audio in browsers (creating a new audio file with a URL means the browser re-downloading the asset!).

Base64 encoding all the binary assets and getting on with life means not faffing around with them during the compo - which is a good thing.

// Build script
swallow.build({
    in: 'assets',
    out: 'site/assets.json'
});

// Client code
GlobalResources.loadPackage('assets.json', function() {
    game.start();
});

GlobalResources.getImage('image.png');
GlobalResources.playSound('explosion');

Camera - Separation of in-game coordinates from the pixels being displayed on screen is pretty important because otherwise we limit ourselves to a specific resolution and aspect ratio, and prevent our application from being run at different resolutions on different screens.

The principle of this tiny piece of code is that rather than drawing using 'screen coordinates', we draw using 'world coordinates' and haven't got to any of the transformations ourselves in expensive JS.

For example,

context.fillRect(0, 0, 100, 100);

Would ordinarily fill a rectangle 100 pixels by 100 pixels at the top-left of the screen, but if we apply transforms to the underlying canvas as if we had a camera moving over it, using the following code

camera.lookAt(50, 50);
camera.zoomTo(100);
camera.fieldOfView(Math.PI / 4);

Then the same fillRect code will be drawing the same rectangle (roughly) in the middle of the screen.

The advantage of this code is that the same picture can be drawn whether the canvas is sized at 320x240, 640x480, 800x600 (and can even handle strange aspect ratios). In case of bad performance, the canvas size can be set to half the size of the actual display and upscaled automatically!

Eventable - I've found that messaging is the best way to keep the ability to crank out features without littering the codebase with conditionals and irrelevant code, I have a basic set of behaviours in an object called "Eventable" that looks like

on(event, callback, context)
onAny(callback, context)
off(event, callback, context)
offAny(callback, context)

I also have a basic Scene object, through which all entity events bubble up through for caretaker objects to deal with, consider the following scenario from my #LD23 game

EnemyFactory decides to spawn an Asteroid

EnemyFactory hooks 'Destroyed' event on Asteroid

   asteroid.on('Destroyed', this.onAsteroidDestroyed, this);

Asteroid goes off and does its thing
Player blasts away Asteroid
Asteroid raises event like so:
```
this.raise('Destroyed');
```
EnemyFactory removes the Asteroid from the scene
Also listening to events from the scene are
- ExplosionCreation
- ScoreKeeper
- SoundCreation

They hooked the events from the scene when they were added to it like so

scene.on('Destroyed', this.onEntityDestroyed, this);

And in their respective methods they get to do

onEntityDestroyed: function(data, sender) {
    this.createExplosion(sender.x, sender.y);
}

onEntityDestroyed: function(data, sender) {
    this.increaseScore(sender.getPoints() * this.currentLevel);
}

onEntityDestroyed: function(data, sender) {
    this.playSound('explosion', sender.x, sender.y);
}

Having the ability to slap on extra functionality without creating masses of extension code meant that throwing in power-ups was a simple matter of creating something to listen to destruction events and add new entities to the scene to represent as power-ups.

Keeping the UI updated looks something like this:

var Hud = function(scene) {
    scene.autoHook(this);
    this.score = $('#score');
    this.health = $('#health');
    this.energy = $('#energy');
};
Hud.prototype = {
    onScoreChanged: function(score) {
        this.score.text(score);
    },
    onHealthChanged: function(health, sender) {
        if(sender.id !== 'player') return;
        this.health.css('width', sender.percentageHealth() + '%');
    },
    onEnergyChanged: function(energy, sender) {
        if(sender.id !== 'player') return;
        this.energy.css('width', sender.percentageEnergy() + '%');
    }
}

Hmmm, tidy.

I choose not to publish this as a library, because this is something specific to the way I like to work and everybody is either using one that already exists or are capable of writing on themselves.

Patterns and Practises

Working in a single file

When working on my more long-term projects, I often use a dependency/module system like RequireJS to break up the project across multiple files (one-per-class type of thing)

When working on a 48 hour game jam, I find that just coding everything in a single file like a madman is really helpful providing I'm using a good text editor with the ability to search and jump around the document built in.

Working with "Classes"

I admit it, I'm a sucker for encapsulating state and behaviour into discrete little objects that I can create and throw around the place,

The thing is, because JS allows for duck-typing, this object flinging makes throwing things into a scene and performing operations on them pretty convenient.

For example, I have a scene object, which exposes the following methods, and at its most simplistic looks something like this:

add: function(entity) {
    this.entities[entity.id] = entity;
},
remove: function(entity) {
    delete this.entities[entity.id];
}
tick: function() {
    this.eachEntity(this.entityTick);
},
entityTick: function(entity) {
    if(entity.tick) entity.tick();
},
draw: function(context) {
    this.eachEntity(this.entityDraw, context);
},
entityDraw: function(entity, context) {
    if(entity.draw) entity.draw();
}

Now, in my long-term projects, drawing/logic/etc are just components that are attached to the entities, and the scene is certainly not responsible for this stuff - but for this kind of rapid-work project having something really simplistic really aids in the development process.

The important things of note, is above - we only care that an entity has a field called 'id', we don't care where it got it from - and if that entity has a draw method, we'll use it and if the entity has a tick method, we'll use that too.

I don't bother trying to emulate classic inheritance, even in something as simplistic as this it's not desirable (and leads to more complexity). I do however make judicious use of underscore's 'extend' method.

// Basic rendering functionality for a textured quad
var Quad = function() {
    // stuff
};

Quad.prototype = {
    draw: function(context) {}; // stuff
}

// A basic powerup which floats towards the player
var Powerup = function(image, x, y) {
    Quad.call(this, image, x, y);
    Eventable.call(this);
    this.id = IdGenerator.Next("powerup");
};
Powerup.prototype = {
    tick: function() {
        this.moveTowardsPlayer();
    },
    notifyCollision: function(other) {
        if(other.isPlayer())
            this.bestow();
    }
};
_.extend(Powerup.prototype, Quad.prototype, Eventable.prototype);

// An actual powerup
var DestructionFieldPickup = function(x, y) {
    Powerup.call(this, "destructionfield.png", x, y );

};
DestructionFieldPickup.prototype = {
    bestow: function() {
        this.scene.addEntity(new DestructionField(this.x, this.y));
    }
}
_.extend(DestructionFieldPickup.prototype, Powerup.prototype);

I'd usually shy away from such hierarchies, but in a 48 hour jam they're a really easy way of throwing functionality in with gay abandon (remember, I don't need to maintain this code, I don't need to remember that the Pickup somehow magically has an 'x' value 2 months later, I don't need to remember what fields have already been declared so not to overwrite them etc).

Working with such lightweight base components and with such explicit objects means that providing my codebase remains below 2000 lines of code (about the maximum deliverable for a solo 48 hour jam if I'm honest), I can keep it all in my head and not fuss around too much.

At least they're relatively small and (mostly) hide their data, and hold onto the functionality they expose in neat, readable blobs.

Assets

I'm not an artist, and I'm not a sound engineer either, I have found however that with Inkscape it is possible to create relatively non-sucky art with the combination of geometric shapes.

Including these is simple, as they're bundled up with Swallow - however, sounds are more tricky.

Sound on the internet SUCKS.

SUCKS. SUCKS SUCKS.

breath, basically you'll be fine if you use a combination of ogg vorbis and MP3, and don't worry about the older browsers.

In a little game like this, I don't worry about the cost and simply download both files all of the time (in swallow), I guess I could package them up individually and do a check on start-up, and perhaps a little library is warranted (either one on the internet or hand-rolled)

The code for playing a sound goes as follows therefore:

playSound: function(path) {
  var player = new Audio();
  if(player.canPlayType("audio/mpeg")) {
    player.src = "data:audio/mpeg;base64," + this.findData(path + '.mp3');
  } else {
    player.src = "data:audio/ogg;base64," + this.findData(path + '.ogg');
  }
  player.volume = 0.5;
  player.play();
}

This will work okay, as the data is cached (so no faffing with re-load pauses).

In Summary

In a 48 hour game jam, I've found that productivity is much more important than the long-term maintainability of the code, but this does not mean abandoning some sensible software practises, as short term maintainability is still important (keeping 2000 lines of procedural spaghetti code in your head isn't quite as easy...).

Any questions? The code is over here, and the above should help with the navigation a bit...

Asset packaging in browser based games

Fri, 30 Mar 2012 07:54:00 +0000

I'm redirecting you to http://altdevblogaday.com/2012/03/28/asset-packaging-in-browser-based-games/ with JavaScript, feel free to skip the setTimeout call I've used

Anti-templating languages

Wed, 28 Mar 2012 16:34:00 +0000

I don't really like templating languages or view engines (especially in JavaScript) - it's something I've been vocal on in person for a while now but never got around to writing about.

Things like this have been ranted on before by other people, but I want to share my particular dislike of the frameworks and technologies here, as well as present up the way I'm currently working.

Logic in your views

Now, we all know this isn't a good idea, but what do we really mean by this? What are the problems we're facing?

EJS

EJS is a view engine similar in nature to WebForms in ASP.NET, which means for .NET devs it's often reached at for its comfortable familiarity and lack of learning requirements.

Let's look at the default example on the EJS website to understand this

<ul>
<% for(var i=0; i < supplies.length; i++) {%>
   <li><%= supplies[i] %></li>
<% } %>
</ul>

EJS promises to "Clean the HTML out of your JavaScript", and the very first demo shows us an example of HTML which has been dirtied with JavaScript.

This is just moving the problem around, this is very much a case of "logic in the view" and it makes it difficult to maintain because it's difficult to read and it's hard to tell where the HTML begins and where the HTML ends.

Standard practise might be to do something like this:

<ul>
    <%= Helpers.RenderList(supplies) %>
</ul>

But this just means I've moved the HTML back into my JavaScript again. I guess what we're saying here, is that trying to arbitrarily separate 'view' from 'logic' in this manner is a fools errand, doomed to fail because all we're doing is moving the problem around.

Mustache

Mustache is a "logic-less" templating language (they say so on their site), there are other "logic-less" templating languages around too and they're all much of a muchness.

I quote:

We call it "logic-less" because there are no if statements, else clauses, or for loops. Instead there are only tags.

for-loop-replacement:


    <ul>
    {{#supplies}}
      <li>{{text}}</li>
    {{/supplies}}
    </ul>

if-else-statement-replacement


{{#shipped}}
    <li>This product shipped on {{shipdate}}</li>
{{/shipped}}
{{^shipped}}
    <li>This product has not yet shipped</li>
{{/shipped}

This is just a for loop and an if-else statement but with different syntax, pretending they're otherwise is doing a disservice to everybody who is going to be reading and writing this code.

Adding onto this, we of course also have the ability to seamlessly call methods from Mustache because we're always going to need to write code somewhere (Well, it's one way to keep logic out of the view by.. err calling logic from the view) (see above re: arbitrary separation).

{{#wrapped}}
  {{name}} is awesome.
{{/wrapped}}

{
  "name": "Rob",
  "wrapped": function() {
    return function(text) {
      return "<b>" + render(text) + "</b>"
    }
  }
}

This serves to just confuse though, as the fact a method is being called is hidden from us and means we're reduced to jumping between template and code to work out what is going on.

A shared concern

I have a problem that's shared across all of these solutions though, and that's the one of dealing with external designers.

I have only once worked in a situation where I was privileged enough to work with a designer who knew her HTML and Webforms syntax and could be taught more if needed because she was a permanent member of our team. (And that was only because I spent months campaigning to get somebody who knew what they were doing when it came to making things look pretty!)

I want to be delivered clean HTML which I can use in my application
I want to be able to integrate updates to that HTML if need be
Teaching a contractor how to use 'template language X' is a waste of my money
Finding a contractor who knows 'template langauge X' is a waste of my time

Here in lies a problem - the moment we go to any templating language/system that isn't just HTML, we have to transform what the designer has given us into that templating language - and then translate it back when patching in any amendments that might come as we continue developing.

Performance

Not only those points, but if we're truly going to have a logic-less templating language and we're using third party APIs in any way (whether they be third-third party, or just plain old third party) then in order to get the data into a shape fit for binding directly to your template, transforms must be done which means writing mapping code one way or another.

It's not healthy I tell you - if you're going to transform one set of data into other data that is an exact match of your view requirements, and then transform from that data into another set of data (your view) then you're paying a cost for this. (Throw in your favourite MVC/MVVM framework for JS and even more so, but that's another blog entry entirely).

So what do I like then?

Accept that a for loop is a for loop
Accept that an if statement is an if statement
Accept that you're always going to need some of these things in your applications
Accept that the above is best suited to being in a programming language of some sort

We have a great opportunity in JS, where we have a language that has been built almost for the primary purpose of interacting with the output that the user sees and where we have libraries whose sole purpose is the interaction with that output.

Use the force

Enter the anti-templating system "Plates" (there are others that are similar, but this is what I'm using at the moment, as it's isomorphic, fast with no-frills and hopefully will remain so - despite the "issue" reports asking for "nesting" or "collections" etc - as they're missing the point).

Rather than being a templating language, Plates merely binds data to HTML.

HTML!! You know - the stuff that you're going to give to the browser, the stuff which your designer gives you - the stuff that everybody on the internet and their pet animals know how to use.

Given some HTML:

<div id="test"></div>

And some model:

{ 
    "test": "hello"
}

Then

Plates.bind(html, model);

Combined with a bit of JavaScript this gives us enough power to do everything we'd want to do when it comes to taking some data and displaying it on a page. (Yes, it supports matching by class, yes it supports putting data into attributes on those elements, this is all trivial).

How does it all fit together?

Well, I tend to either keep the HTML snippets which I'm going to hydrate on the page itself in a hidden div, or if they're shared templates, as files which I can pull down with a HTTP GET (not rocket science really).

How do I deal with collections? Easy - I write a for loop. How do I deal with different paths? I want an 'if' statement.

But don't you end up with spaghetti code?

Well no - just because I'm not following an enforced and arbitrary separation of 'view' and 'logic' doesn't mean I'm throwing away sensible software practises.

It's just, that separation comes naturally on a case-by-case basis.

Sometimes I'll end up with some code that matches the model purely on convention and I can write

hydrateTemplate('source', 'target', data);

Sometimes I'll end up with builders that look like this

 startTemplateWithId('targetId')
    .withText('title', data.title)
    .withText('name', data.name)
    .withCollection('itemList', data.items, getItemHtml)
    into('placeholder');

Or similar (although in most simple cases this kind of over-blown code isn't needed).

My mark-up remains clean, my model remains clean, and the code that lies between is kept clean, tidy and to the point - not to mention re-usable where appropriate because it's just code.

When asked where this style fits in, I'd say it's essentially just MVP, with the line between V and P moving around to fit the situation.

Summary

Separation of view and logic isn't going to happen along any sort of neat line without silly amounts of abstraction; use something that allows you to have clean HTML and clean JavaScript with as little bullshit in-between as possible.

You'll be happier, your code will be happier and you'll find it is a lot easier to deliver a product when you stop arguing about whether you have enough separation in your abstractions.

Lessons learned building a multiplayer game in NodeJS and WebGL

Mon, 19 Mar 2012 12:00:00 +0000

I've uploaded Hoverbattles to its own server on EC2, and it has been running fine with an uptime of over 96 hours so far, and this is great!

http://hoverbattles.com

I've wanted to share a few of the mistakes/lessons learned writing and deploying a multiplayer game built entirely with JavaScript on top of NodeJS and WebGL for a while and this represents an opportune moment to do so.

I've gone with a brain-dump of various related learnings, as well as a couple of periphery items - first off, we'll go with the reason I couldn't keep Hoverbattles up on the old server.

Deploy long-running, processor-intensive node apps to decent hardware

If you're going to deploy a long running node application that is going to be running a constant load, don't deploy it to either:

A micro instance of EC2
A really low budget VPS

This might be plain old obvious to most people, but apparently not to me - I first deployed to my own VPS, and found that after it had been running for an hour or so it consumed all the memory on the server and fell over in a steaming heap.

I tried to find a memory leak in Hoverbattles itself, I changed to various versions of node, I re-built various components of the OS and nothing seemed to work. I wasn't able to reproduce the issue on my meaty laptop and I gave up for a while, as I was working on a new project.

Turns out that by running long running processes on a virtualised OS on oversubscribed hardware, the OS is lied to, or it lies to you and things don't work quite well. Some of the more knowledgeable types could probably tell us why - but the bottom line is you shouldn't be doing it.

Note: This probably doesn't go for simple websites built on top of node in express or whatever, I'm talking about long running applications that are doing processing almost constantly, like the server-side component to a 'realtime' multiplayer game.

If you're really going to share code between client + server, plan this accordingly

With Hoverbattles, it started off as a purely client-based game, with JavaScript files being included in the main HTML file and this worked great while I was experimenting with the WebGL and working out how everything was going to work.

I soon moved to having a server implementation running the logic, and ported everything across to CommonJS, using Stitch to package up all of the files so I could use them on the client.

This is actually problematic, as you don't want all the code on the client, and you don't want all the code on the server - with CommonJS you'll only get the code loaded on the server that is used there, but if you're stitching your entire /src folder, you're potentially also sending down code for your persistence, communication, 'secret sauce' stuff etc.

I ended up solving this in Hoverbattles by having a folder structure of:

 /shared
 /server
 /client

This is a bit hideous and arbitrary - and doesn't allow me to organise my codebase naturally along its logical borders, and it's for that reason in my latest projects I've switched across to using RequireJS.

By writing different entry points to the same code, and simply boot-strapping in various sub-systems and behaviours from those call-sites, you can naturally end up with a dependency chain that only includes code relevant to the platform for which it is targetted.

Avoid creating new objects in the main event loop

This is obvious too if you've been developing any sort of large scale JavaScript application, or you come from an unmanaged background where this is something you learn not to do from very early on - but I've been developing in a mostly managed world for a few years (C#) and creating objects doesn't carry with it the same overhead so you become quite cavalier to it.

I started profiling Hoverbattles a few weeks in and was surprised to find out that 70% of my CPU time was spent in a single method, that is:

vec3.create();

Vec3 is an object literal from glMatrix containing useful functions for manipulating and creating vectors on top of the typed arrays available in WebGL compatible browsers - these are cool for a number of reasons (performance oriented reasons mostly) and I didn't really think twice about my usage here.

Consider the following imaginary method called once a frame for each missile currently active in the scene.

calculateDistanceToTarget: function() { 
  var difference = vec3.create();
  var targetDestination = this.target.position;
  vec3.subtract(targetDestination, this.position, difference);
  return vec3.length(difference);
};

This is a really bad idea, and I had code lying around all over the place that would do this - create a temporary float array in order to perform some calculation and then carry on.

The answer was to create a buffer or two on start-up for each system that needed to do things like this, then the code looks something like this.

calculateDistanceToTarget: function() { 
  var targetDestination = this.target.position;
  vec3.subtract(targetDestination, this.position, this.sharedVec3);
  return vec3.length(this.sharedVec3);
};

This isn't too nice, but so long as you keep the use of this shared buffer to a single method (IE, write into it, then read out of it immediately) then there are no issues with multiple methods across the system using this data.

On that note though...

Private state should remain private

I got in the habit in Hoverbattles of being a bit cavalier about accessing state and simply doing direct property access across objects in order to perform calculations. In some cases this isn't a bad idea and keeps the code legible and fast. In most cases it's a lot of coupling added for little gain - especially if you start to write back to those fields later.

Just like in C#, property/field access is generally a bad idea, you should be asking objects questions which they can answer, and giving them extra information for those questions if they need it - and you should definitely be telling them to do things instead of taking that responsibility away from them.

The thing is, in most LOB apps this is really not that big a deal, CRUD is boring, the applications we build are boring, we can get away with this stuff. When you're dealing with a game world where dozens of things are going on 30 times a second, controlling access to state starts to become important. Lesson learned.

Consider instead our earlier example of:

calculateDistanceToTarget: function() { 
  var targetDestination = this.target.position;
  vec3.subtract(targetDestination, this.position, this.sharedVec3);
  return vec3.length(this.sharedVec3);
};

We could instead have:

calculateDistanceToTarget: function() {
    return this.target.distanceFrom(this.position);
};

This subtle switch in logic means we're no longer accessing the supposedly private state of another object, and I'm free to change it without worrying about the rest of my code breaking.

I'm also able to far easier control this behaviour when writing tests (target can be a fake target if I feel it necessary to stub out the real logic).

In my current projects I am being a lot more strict about state access - all state is technically public due to the nature of my JavaScript objects, but I don't give into temptation and touch it (that's will power yo').

Encapsulation is really important in a project that has this much "business logic", and having a sensible object model is a big part of this.

Push, don't pull - but sometimes pull

I also made the mistake in Hoverbattles of trying to build components to control every aspect of a particular behaviour - this involved pulling state from various places to work out whether A or B should happen, or whether to display X on the screen or not.

This didn't scale, I ended up having to pull state from three sources (which means asking the world for the entities concerned), and writing methods to pull that state on the components that those entities were build out of.

Turns out I ended up with a system in places that looks like classic Event Sourcing; You look at high score tables, persistence, particle systems and the HUD as views on top of the single source of truth and consider that you can build those views from events being raised in the game world. Suddenly it makes sense that all state in areas with high view subscription should be built from events raised by those entities.

Once I had that realisation, development got a lot easier, "Hey, I've been told to move, I'll raise an event with the relevant data, subscribe to it myself to update my own state and let everybody else do the same".

I didn't go overboard with this, in Hoverbattles there are only a few places where the above is true, and even then only just and in my latest code this is much more established pattern - the world has commands coming in via input or across the network, and raises events so everything that cares can be updated.

Sometimes it is just more appropriate to pull the state, especially if it's hard to raise an event without duplicating data (see above the cost of creating new objects), and the trick so far has been recognising that and trying not to overly homogenise.

Don't let network-code take over your domain

In Hoverbattles there was a real problem when it came to writing the network code, it ended up being far too pervasive and leaked into too many aspects of the logical entity code.

Some of this managed to be repaired before it became too much of a mess, the main realisation was that there were only a few classes of problem involved in most of the network code, that is:

User input, sent as commands to both the world and the server
Periodic sync, extracting state from objects and serializing across the wire to all clients
Protected code that can only be executed on the server, but the results of which need executing on both client and server

These are actually ordered in terms of difficulty:

User input can easily be sorted by sticking an intermediary between the actual input emitter and the game world (so, one object with some code in it).
Each component has the chance to serialize state and receive state by convention with methods called _in and _out
This one caused a lot of issues, trying to attach different components to entities depending on whether they were created on the server/client, etc

That last one was a bit of a doosy, I ended up with about 20 objects trying to juggle only the responsibility specific to the server or to the client, it looked something like this.

/client/
/client/firingbehaviour.js
/client/dyingbehaviour.js
/client/lockingbehaviour.js
/server/
/server/firingbehaviour.js
/server/dyingbehaviour.js
/server/lockingbehaviour.js
/shared/
/shared/firingbehaviour.js
/shared/dyingbehaviour.js
/shared/lockingbehaviour.js

This gave me hard to debug errors because the responsibilities for an entity's behaviour were spread all over the place, state was being mutated all over the shop and it was becoming hard not to duplicate code across the different environments.

All of this because I was full of pride and didn't want to write into my code anywhere the line of code:

if(Environment.IsServer)

As it felt wrong. I ended up with a compromise, which is I'd do that on the main entity object (which tends to bring together these different behavourial components).

Here's the thing - if I was using events to update my internal state, surely I could simply suppress the events on the client if the client didn't have permission to make that decision (for example, player health loss is a decision only the server can make - I don't want craft blowing up and being removed from the scene if they didn't actually die, it's a horrible visual artifact).

applyDamage: function(amount) {
  var newHealth = this.health - amount;
  this.raiseServerEvent('CraftDamaged', newHealth);
},
onCraftDamaged: function(newHealth) {
  this.health = newHealth;
  if(this.health < 0)
   this.raiseEvent('CraftDestroyed');
}

Regardless of client or server, this logic would get executed - but only on the server would the event actually get raised when a craft lost health, and that event would automatically be proxied to all the clients for the rest of the logic to be executed.

This leaves me with:

/craft/
/craft/firingbehaviour.js
/craft/dyingbehaviour.js
/craft/lockingbehaviour.js

Because this was back-patched in over a weaker system, there are some remnants of this change left over the Hoverbattles source, but it is a lesson I'm applying in the new game to really good effect.

The network code in the latest project consists of about three objects primarily just routing commands and events and it is most likely going to stay that way.

Arrays are mutable reference types

Duh. we all know that, why bother including it? Well - remember I said that creating new objects is expensive so I was sharing them? Yeah - well that can bite too, and given that this was one of my recurring bugs (my own stupidity granted) it's worth documenting.

In C#, typically you don't expose mutable reference types, you'd return an IEnumerable if you wanted to expose a collection from an object, which is a read-only collection fit for... well reading from the consuming code.

Can't quite pull off that trick in JavaScript (although the solution could exist in user-land it's a bit of a faff as it's not transparent).

The problem is, in a game like Hoverbattles - half of our state is in fact arrays of either length of '3', or length of '16' (vectors and matrices) and we have to be careful when receiving a vector or matrix that we don't own. Consider the following simplified code, which is reminiscent of an actual bug I had in Hoverbattles.

Player

moveLeft: function(amount) {
  this.position[0] -= amount;
  this.raiseEvent('Moved', this.position);
}

Enemy

onPlayerMoved: function(newPosition) {
  this.playerPosition = newPosition;
},
doSomeLogic: function() {
   // Some calculation that indirectly modifies the array
   this.playerPosition[0] += 5;
}

Okay, the above is quite obvious, but this kind of thing happened (in substantially more convoluted scenarios) with the outrageous result of player craft ending up where they should not be. (Especially in the network code where objects are receiving new state a lot of the time).

The answer is, if you're receiving an array from an event or command, to copy it over to your own internal value if you want to keep the state around for any length of time for future processing. (Ignore this at your peril unless you're smart and/or have lots of tests).

Push it to the GPU

Hoverbattles first particle system was written on the CPU, and looked something like this:

var ParticleEngine = function(count) {
  this.particles = new Array(count);
  for(var x = 0; x < count ; x++) {
    this.particles[x] = {
      x: 0,
      y: 0
      velx: 0,
      vely: 0
    }
  }
};

ParticleEngine.prototype = {
  update: function() {
    for(var x = 0; x < count ; x++) {
      this.particles[x].x += this.particles[x].velx;
      // etc
    }
  }
};

Yeah, this didn't go too well - I wanted... no, I needed many thousands of particles, and blocking the event loop on the browser by looping through large collections of objects is a big no no.

If you can push processing from the CPU on the browser, to the GPU using shaders, then you should, JavaScript is slow and not a suitable place to be playing with large loops of data.

Besides, GLSL is quite a pretty language to do it in:

void main(void){

    float age = (time - aCreationTime);
    vec3 position = aVertexPosition + (aVelocity * age);
    vColour = aColour;

    vec3 vectorToPoint = (position - vCamera);
    float distanceSquared = abs(dot(vectorToPoint, vectorToPoint));
    float scale = clamp(distanceSquared, 1.0, 10000.0);      

    life = 1.0 - (age / aLifetime);
    life = clamp(life, 0.0, 1.0);

    gl_PointSize = (aSize * maxsize) / (scale / 100.0);
    gl_Position =  uProjection * uView * vec4(position, 1.0);
}

Relax

Finally - something I covered previously - relax, there is no problem you cannot solve with a bit of patience, re-factoring, profiling, and debugging. :-)

Your container is not wanted here

Thu, 02 Feb 2012 12:34:00 +0000

Finding myself embroiled in yet another debate about IOC containers on Twitter I've decided to place my current thoughts here for posterity.

I don't really like using IOC containers

There, I said it. I really don't. I used to, I did - I thought they were an excellent way to manage those dependencies, to push the effort of lifetime and scope management into something that would automatically handle those things for me so I wouldn't have to think about them.

I thought they were an excellent way to bootstrap of my entire application from a single place, and have all the interfaces matched with their single implementations and pushed into the relevant consumers without having to think about it.

But you know what? Having your entire application sucked out of a black box and then writing rules for the exceptions to those wonderful conventions and then writing new conventions and interceptors and using all the "wonderful features" of the modern IOC container started to lead to developers spending more time debugging mysterious container issues and fighting odd/conflicting lifetime issues than writing code of real value.

Oh, you could easily dismiss this with "Oh, Rob doesn't know how to use a container properly", but you'd be missing the point, because even if I didn't (and I do by the way), it's irrelevant whether I do or not.

Missing the point? Missing the point because nearly every team using an IOC container IS doing it wrong, and they're doing it wrong because they are complicated and they give you a lot of "extensibility points" to make it easy to do things like interception, they make it easy to do things like per-request items, they make it easy to create singletons that aren't really singletons. and they make it easy to create lots of interfaces that get sucked into lots of classes (in the name of low coupling, and usually with the result of a total lack of any cohesiveness).

It's putting the cart before the horse

Learn to walk before you run, cart before horse, etc. The fundamental issue here, is that people are spending their times learning about IOC containers, gaining some level of test-ability because everything is an interface that talks to other interfaces via interfaces to interfaces. This is not to say that IOC containers cause this explicitly, because if you've already got a grasp of OO concepts then you aren't going to do too much damage here (except for hiding simple concepts like lifetime management up behind infrastructure that's a bit more future-career-proof).

My fundamental issue is that not enough time is being spent by developers learning how to just grow a testable and maintainable code-base. Throwing your lot in with a container with a centralized bootstrap process and claiming that's an advantage is missing out on a fundamental aspect of clean software development - that is, neat little packages that know how to bootstrap themselves and expose a sensible API for doing so - allowing them to be used across the code-base in an understandable and idiomatically crafted way.

Allowing your junior developers to "not worry about these things", because the almighty and all-knowing container will look after them and ensure that the code is testable, and that dependencies will just work automatically is simply shirking the responsibility of actually teaching those developers the useful and transferable skills that will help them deliver products across a multitude of languages and platforms. (EG. not just the two that come with a million IOC containers to choose from).

In GOOS we are extremely explicit about scope! Java is a block structured language with lexical scoping, closures and objects. Blocks and objects are scopes. Variables declared in a block and instance variables declared in an object are in a scope. There is no need to re-implement (badly) what the language (compiler and VM) already provide.

Nat Pryce

And this, is what I believe that it all boils down to.

You want per-request scoping? That's a "using statement" around the entry point to that request.
You want application-lifetime scoping? Just create the object on start-up and let it get cleaned up on application-close
You want something more fine-grained? That's just another using statement around the code concerned.

Objects still don't need to know about their own scoping, of course not, and we realized that with containers early on with the removal of attributes from most frameworks. But why make all the effort of pushing scoping into a framework when it is such an intrinsic part of your application and it's relatively trivial to manage anyway? Lifetime management is not an implementation detail to be pushed away into central infrastructure code, and nothing but trouble will be had from trying to work that way (nested sub-containers anyone? No - I thought not).

You want to talk about writing masses of boilerplate code? I have very little in the applications I'm actively developing now - each abstraction developed is responsible for its own set-up, and only exposes to the outside world any configuration needs that it might require and the public interfaces required to do its job. That code is written and tested as part of the code-base, is compile safe and is fast to bootstrap because "it's just code". Abstractions are built on top of other abstractions and are tested against other abstractions with appropriate levels of isolation depending on the test concerned and there are no problems here at all.

This approach does not preclude the injection of dependencies into say, the subsystem which might be created as a consequence of its construction, it merely hides that detail behind an appropriate API because the consumers of this package don't typically care about that construction.

You want to talk about managing deep or complex object graphs? That's not a problem - each package is only ever going to have a shallow object graph, because that's sensible software design - I don't have complicated object graphs because complicated object graphs tend to show themselves during testing and are very quickly turned into simple object graphs.

It's just software, and we should be spending more time learning how to deliver software and less time learning how to manipulate favourite container X.

What does it look like when I code?

Tue, 20 Dec 2011 17:47:00 +0000

Something like this

This is a time-lapse of me creating a game over 48 hours (a weekend), overall there is about 30 hours of screen-time packed into 3 minutes of video - it's cool to see how the game and code progress over those 30 hours.

What is interesting is how I always have the social elements open in a browser window on my left hand side, I'd never noticed it before - I don't think it slowed me down any - the pauses where I wasn't coding I was up making coffee or thinking about a problem

Still, it would be interesting to see how I'd get on without it if I was doing another of these - the next rendition of the competition is next March and I think I'll be doubling my efforts to create something cool - I might even go as far as to do a 3D effort in WebGL

Hoverbattles Released (and more)

Tue, 20 Dec 2011 15:28:00 +0000

I've gone off on a tangent recently and been playing around with games development as if I was 15 again

I think I'm going to make a game that I actually want people to play next, but first up - three things I've released recently

Hoverbattles

My flagship 'game', something I've learned a lot from these past couple of months - written with a NodeJS back-end, with WebGL front-end, the code is awful in places and I think I have a few memory leaks (or third party libs do!) but I'm pretty much done with this now.

Note: The server this is on is not brill, so there will be lag

This can be found at http://hoverbattles.com - fill your boots.

Source can be found at https://github.com/robashton/HoverBattles

Plane Thinking

I coded this over a few days as warm-up for LD22, the 48 hour games development challenge, this is plain old Canvas (although I was playing with using WebGL to do progressive enhancement it turns out that copying buffers between the two gets expensive quickly and I didn't take it much further

This should work in most browsers, I really should make the effort to get it working with touch controls as it would work well on iPad

This can be played at http://planethinking.heroku.com/

Source can be found at https://github.com/robashton/plane-thinking

You don't have to be alone

This is my entry to the 48 hour game development competition Ludum Dare - I don't really think of this as competing with anybody else, trying to build a game from scratch over 48 hours is mega hard and I'm really proud that I was able to pull it off with plot, sound, music and alternative ending galore

The code for this is ... suboptimal, if you play it on anything other than a really high end desktop computer in any other browser than Chrome then do so at your peril. I'll be learning from that in my next game and making an effort to keep those render calls down

The game can be found at: http://ld22-ashton.heroku.com/

The source can be found at: https://github.com/robashton/ld48_22

The point of all this

Games development is complicated, the code in Hoverbattles is probably the most complex domain I've ever worked on, and finding ways to keep the accidental complexity low whilst keeping the technical complexity low at the same time was a big challenge

I've learned a lot by stepping away from ordinary business app development, and I'll be taking that back to the workplace with me, as well as carrying on in this space - never before has there been a better time for aspiring games development to noodle on in their spare time.

This site? Yeah I've messed this up a bit - I need to rip all these posts out and deploy them as static content and set up some re-directs, a project for a rainy weekend when I don't want to play with games :-)

Merry Xmas

Rob

A relaxed attitude towards the pragmatic delivery of 'okay' software

Tue, 01 Nov 2011 11:37:00 +0000

A brief middle-of-project retrospective

I've not been coding in my spare time much the past few months, being seriously busy for a client - but the past couple of weeks has seen me pick up a project of mine that I started a few months ago (HoverBattles) and start pushing to some level of completion.

This has been an interesting project for me, not least of all because it's written entirely in Javascript (WebGL + JS, NodeJS and CouchDB) but because this time I made a real effort to drop any up-front 'zomg my code must be perfect' aspirations from the get go.

What does this mean? Well I pretty much decided that technical debt should not be something to be overly avoided, overly organised code-bases stifle creativity and I really just wanted to deliver something.

I also knew that as I learned more about effective JavaScript that my opinions on the 'best way to achieve things' would be changing about 20x more rapidly than it does when working in an environment I'm heavily used to, and trying to constantly maintain code at some mythical standard would prevent me from actually delivering

A couple of months later how does that leave me? Is the code-base a huge unmaintainable mess? I would say no - it is not.

There are messy bits in there but they're largely on the periphery. An avoidance of up-front framework/empire building has allowed me to be morally flexible with regards to where I put new code to Get Cool Stuff Done, and a healthy concern when going over pre-existing code to add something new has led to some easy wins when re-factoring around the pain points that naturally emerge.

I don't think that I've taken a cavalier approach to the project - at a macro level it's quite well structured, and for the most part there is a clean separation between the different sub-components that drive the system. In a code review there are bits I'd have to apologise for but even in projects with the greatest amount of technical debt avoidance this has been true.

There are some problems with the messaging/multi-player side of things, complex work-flows have emerged as a consequence of the individual components receiving some input, doing something and raising some output in orders that I did not pre-plan.

This has been made more complicated by the fact that some of these components only exist on the server, and yet the events they raise are still sent to the client because the client needs to react even if it is not the one doing the critical thinking.

This is an example of what I have come to classify as a real problem - that is, it is a problem that is naturally occurring and isn't one I have constructed to satisfy the whims of my inner Powerpoint Architect.

When I think of all the things that we work on in our line of business applications, the efforts we go to de-couple everything so it can be easily tested and maintained, the efforts we go to make sure we have the extensibility points and have our "what ifs" covered, I'm seeing a lot of that in a new light as this project goes on.

I don't think a lot of our 'units' in our LOB world are really units at all, they're fractions of 'units', and it's only as part of a more complex interaction that things get interesting and worth spending time fussing over.

Even those seeking to do more vertical testing of a unit within their system (across several internal components) aren't really testing anything meaningful, they're not really spending their time on anything really meaningful either - I wonder if we do a lot of this stuff just to make our jobs more interesting because LOB apps are at a micro-level... quite boring

Anyway I digress, a response could be that some of that rigidity and forward thinking is needed because we have more than one soul working on these projects and if everybody took the cavalier 'get it done' attitude we'd end up with a big mess right?

I'm not convinced - I think that if you have a team that can actually communicate and react to problems as they arise that a good momentum would still be possible, a lot of the technical solutions delivered in these LOB apps seem to exist as a way of avoiding the need for communication and I'm beginning to think of that as less okay than I did

I'm also beginning to think that a concentration on these things causes bigger balls to be dropped. There is little point in arguing over patterns if you're going to forget to apply sanitisation to user-provided input for example - or suitable defensive mechanisms against things that might go wrong (as dirty as that might make some of your code).

Going back to the whole CQRS thing as that was the topic of the last post, this ties in well - those things exist as solutions to complexity that already exists - not as ways of creating complexity that wasn't there before. Technical solutions should be avoided unless they're actually delivering the necessary value.

Random blathering I know, I'll actually start talking about the tech in the game soon I think, it's getting interesting and I think there are some things to say about it

CQRS is too complicated

Wed, 28 Sep 2011 19:45:00 +0000

Is something I hear all too often at conferences and on Twitter, and more often or not it is said because of either a basic misunderstanding of what CQRS is or is not - or perhaps because they've dipped their toes into the hyperactive DDDCQRS mailing list and been scared away by all the white coat discussion that goes on in there a lot of the time.

The other day, the sentiment was yet again voiced by somebody of whose opinion I respect on Twitter and I ended up in about five minutes writing a gist explaining why I didn't think this was the case (Writing 4000 word essays is an hour's work if I'm feeling ranty), I've tidied it up a bit and decided to throw it below as it works well in a blog entry.

A basic summary

At the highest level CQRS just means maintaining a happy division between the reads and writes across your system - that is, having the reads in your system executed in a thin clean manner appropriate to the views you want to retrieve (one model), and your writes going through all the crazy logic you need such as validation, updating queues, third party systems, processing business rules (another model)

Consider the traditional and very-tongue-in-cheek N-Tier architecture I have created here in powerpoint, seen in a million "architecture" presentations in ASP.NET webforms shops across the world:

Now modify it a little bit so that our reads haven't got to go through all that cruft, haven't got to somehow amene themselves to a bunch of "DAL objects" that are created with the very best intention of standardising our access to some form of database (and normally optimised for the write actions anyway).

We can instantly make our lives a lot easier by creating a pile of code optimised for creating views for our presentation layer, perhaps doing a bit of raw SQL or calling a sproc to generate the view for us. We can helpfully formalise this arrangement and for the most part set down a rule that the direction of travel down those two paths is one way (towards the DB for writes and away from the DB for reads). Funnily enough - most systems that do that BOL/BLL/DAL/OCKS stuff end up with something that looks like this anyway because it's too hard to do everything through a single model.

This is now a form of CQRS - at the highest level we've effectively split our system into two models and done something that's very similar to what we'd call CQS if we were doing it at the method level. This in itself should surely be enough to convince you that CQRS itself is not complicated and it might be a useful thing to look further into.

Of course, as you go further down the rabbit hole...

Some examples

CQRS can be achieved by using a document database like Raven or Couch - using your documents as a write store, using your indexes as a query store.

It can be achieved with your favourite ORM (Even better if you can actually use that O and that M and get some good old OO going) - if you want to use your objects for encapsulating business logic and go directly to the the queries to project the data you need for views (HQL, SQL directly, SPROCS, whatever) - from the same database even, providing this remains efficient enough for your needs. (Funnily enough, "our" collective weak attempts at creating domain models with NHibernate are what led to us re-discovering the need for two models in the first place in my opinion).

Of course you may well end up with two databases anyway, as trying to query a database comprised of tables that represent state in your "objects" can be pretty inefficient, with the read store updated from the write store using hooks in your write system to generate pre-calculated views or data that's more applicable to generating views - this is not a bad model and can work too, it's still CQRS.

CQRS gets the "complicated" label because people often associate it directly with event sourcing, which requires that little bit more of up-front development in order to get the level of elegance you won't find in the above scenarios. However, even event sourcing is really simple once you look at it - and is a natural progression from some of the other ways of "doing" CQRS - which can be a bit muddy (not that there is anything wrong with systems that are a bit muddy). Note that I'm not mentioning DDD here At All - which is where a lot of heavy learning lies, and nearly none of us do anyway.

Consider hooking those events in your system to manually flatten/re-arrange data into other stores as outlined above? Does that work for that one other store? How about a reporting store? How about full text search? What about integration with third party systems and the data they want to see from you? How about the boardroom reports your CEO now wants on his desk each morning before he starts his day?

Youch. Deciding your single source of truth is the already written state gives you an amount of inflexibility, which you may or may not be happy with up to a point.

Updating other views of this truth after small changes can be inefficient and awkward. Recovering after introducing any write bugs to the system can be expensive also. Hell - even changing your model can also be expensive as database migrations are hardly the easiest things if you're trying to work with multiple stores and layers all over the place. When your powerpoint presentations start looking like this you have complexity issues- and these complexity issues aren't caused by CQRS, they're caused by having complex powerpont presentations.

Moving to events and jumping through a few hoops to make this possible can open up a world of simplicity, and if it's not for you there are other options open to you. CQRS is not complicated - trying to shoehorn the responsibilities of read and write through a single model is complicated. Most of us realise that going through a standard "BLL, DAL, BOL, TLA, CRA, P) layer for both reads/writes is dumb, and CQRS is a good way of formalising this decision.

Another tdlr;

You can see that clearly there is a natural progression from the very basics to having the need to go for a full blown event sourcing system with publishers/subscribers/servers/eventual consistency once the complexity of trying to manage a more "simple" solution starts to overwhelm.

Unless you have that complexity and that need then of course trying to thrust an ivory tower designed architecture onto a system that doesn't need it is going to seem complicated. Hint: If your technical solution is more complicated than your original problem you're probably doing it wrong.